HK1232703B - Image processing device and image processing method - Google Patents

Description

Image processing device and image processing method

This application is a divisional application of the Chinese invention patent application filed by the same applicant on December 2, 2011, with application number 201180057796.9 (PCT/JP2011/077954) and invention name “Image processing device and image processing method”.

Technical Field

The present disclosure relates to an image processing device and an image processing method.

Background Art

H.264/AVC (a standard specification for image coding schemes) applies a deblocking filter to block boundaries, each containing 4×4 pixels, to prevent image quality degradation caused by block artifacts when encoding images. Deblocking filters require extensive processing, potentially accounting for up to 50% of the total processing required for image decoding.

The standardization work for High Efficiency Video Coding (HEVC), a next-generation image coding scheme, proposes applying a deblocking filter to blocks of 8×8 pixels or more, as per JCTVC-A119 (see Non-Patent Document 1). The technique proposed in JCTVC-A119 increases the block size to the minimum unit that allows the deblocking filter to be applied, allowing filtering to be performed in parallel on block boundaries in the same direction within a macroblock.

Citation List

Non-patent literature

Non-patent document 1: K. Ugur (Nokia), K. R. Andersson (LM Ericsson), A. Fuldseth (Tandberg Telecom), "JCTVC-A119: Video coding technology proposal by Tandberg, Nokia, and Ericsson", Documents of the first meeting of the Joint Collaborative Team on Video Coding(JCT-VC),Dresden,Germany,15-23April,2010.

Summary of the Invention

Even using the techniques proposed in JCTVC-A119, there remains a dependency between the processing of vertical block boundaries and horizontal block boundaries. Specifically, processing of a vertical boundary of one macroblock must wait until the horizontal boundary of an adjacent macroblock is processed. Processing of a horizontal boundary of one macroblock must wait until the vertical boundary of the same macroblock is processed. This technique only provides for a very limited degree of parallel processing of the deblocking filter. Consequently, it does not successfully address the issues of delay and reduced data rate caused by the large amount of processing required when applying the deblocking filter.

The technical object according to the present disclosure is to provide an image processing device and an image processing method capable of providing further parallel processing when applying a deblocking filter.

According to an embodiment of the present disclosure, there is provided an image processing apparatus including a decoding portion configured to decode an image from a coded stream, a horizontal filtering section configured to apply a deblocking filter to a vertical block boundary within an image to be decoded by the decoding portion, a vertical filtering section configured to apply a deblocking filter to a horizontal block boundary within the image to be decoded by the decoding portion, and a control section configured to cause the horizontal filtering section to filter a plurality of vertical block boundaries included in a processing unit including a plurality of coding units in parallel, and to cause the vertical filtering section to filter a plurality of horizontal block boundaries included in the processing unit in parallel.

Typically, the image processing device can be implemented as an image decoding device that decodes an image.

According to an embodiment of the present disclosure, there is provided an image processing method, including decoding an image from a coded stream, performing horizontal filtering to apply a deblocking filter to vertical block boundaries within the image to be decoded, performing vertical filtering to apply a deblocking filter to horizontal block boundaries within the image to be decoded, and controlling the horizontal filtering and the vertical filtering so as to filter a plurality of vertical block boundaries included in a processing unit including a plurality of coding units in parallel, and filter a plurality of horizontal block boundaries included in the processing unit in parallel.

According to an embodiment of the present disclosure, there is provided an image processing apparatus including a horizontal filtering unit configured to, when encoding an image to be encoded, apply a deblocking filter to a vertical block boundary within an image to be locally decoded, a vertical filtering unit configured to apply a deblocking filter to a horizontal block boundary within the image, a control section configured to cause the horizontal filtering unit to filter in parallel a plurality of vertical block boundaries included in a processing unit including a plurality of coding units, and to cause the vertical filtering unit to filter in parallel a plurality of horizontal block boundaries included in the processing unit, and an encoding unit configured to encode the image to be encoded using the images filtered by the horizontal filtering unit and the vertical filtering unit.

Typically, the image processing device can be implemented as an image encoding device that encodes an image.

According to an embodiment of the present disclosure, there is provided an image processing method, comprising performing horizontal filtering so as to apply a deblocking filter to vertical block boundaries within an image to be locally decoded when encoding an image to be encoded, performing vertical filtering so as to apply a deblocking filter to horizontal block boundaries within the image, controlling horizontal filtering and vertical filtering so as to filter in parallel a plurality of vertical block boundaries included in a processing unit including a plurality of coding units, and filtering in parallel a plurality of horizontal block boundaries included in the processing unit, and encoding the image to be encoded using the image filtered by means of horizontal filtering and vertical filtering.

As described above, the image processing apparatus and the image processing method according to the present disclosure also improve parallel processing when applying a deblocking filter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a block diagram showing an example of the structure of an image encoding device according to an embodiment.

FIG2 is a block diagram showing an example of the structure of an image decoding device according to the embodiment.

FIG. 3 is an explanatory diagram showing an example of neighboring pixels around a boundary.

FIG. 4 is an explanatory diagram illustrating reference pixels during filtering necessity determination processing according to the related art.

FIG. 5 is an explanatory diagram illustrating pixels updated by filtering processing.

FIG. 6 is an explanatory diagram illustrating identification of edges for explaining the embodiment.

FIG. 7 is an explanatory diagram illustrating parallel processing according to the prior art.

FIG. 8 is a first explanatory diagram illustrating dependency relationships between processes according to the prior art.

FIG. 9 is a second explanatory diagram illustrating dependency relationships between processes according to the prior art.

FIG. 10 is an explanatory diagram illustrating a processing sequence according to the prior art.

FIG. 11 is an explanatory diagram illustrating a processing sequence according to the first example.

FIG12 is a block diagram illustrating a detailed structure of the deblocking filter according to the first embodiment.

FIG13 is a block diagram illustrating a detailed structure of a determination section.

FIG. 14 is an explanatory diagram illustrating neighboring blocks around a slice boundary.

FIG. 15 is an explanatory diagram illustrating a first example of a processing sequence for each slice.

FIG. 16 is an explanatory diagram illustrating a second example of the processing sequence for each slice.

FIG. 17 is an explanatory diagram illustrating first and second examples of the determination technique provided by the modification.

FIG. 18 is an explanatory diagram illustrating third and fourth examples of the determination technique provided by the modification.

FIG. 19 is an explanatory diagram illustrating fifth and sixth examples of the determination technique provided by the modification.

FIG20 is a flowchart illustrating the processing flow of the deblocking filter according to the first example.

FIG21 is a flowchart illustrating the flow of filtering necessity determination processing.

FIG. 22 is an explanatory diagram illustrating a processing sequence according to the second example.

FIG23 is a block diagram illustrating a detailed structure of a deblocking filter according to the second example.

FIG24 is a flowchart illustrating the processing flow of the deblocking filter according to the second example.

FIG. 25 is an explanatory diagram illustrating the processing sequence of each CLU.

FIG26 is a flowchart illustrating the processing flow of each LCU.

FIG27 is an explanatory diagram illustrating the overview of the third example.

FIG28 is a block diagram illustrating a detailed structure of a deblocking filter according to the third example.

FIG. 29 is an explanatory diagram illustrating determination of weights of a weighted average.

FIG30 is an explanatory diagram illustrating an example of weights of a weighted average value.

FIG31 is an explanatory diagram illustrating output pixel values of a calculation section according to the third example.

FIG32 is an explanatory diagram illustrating a first example of a processing sequence for comparison.

FIG. 33 is an explanatory diagram illustrating a first example of a processing sequence provided by the third example.

FIG34 is an explanatory diagram illustrating a second example of the processing sequence for comparison.

FIG35 is an explanatory diagram illustrating a second example of the processing sequence provided by the third example.

FIG36 is a flowchart illustrating a first example of the processing flow of the deblocking filter according to the third example.

FIG37 is a flowchart illustrating the flow of the pixel value calculation processing shown in FIG36 .

FIG38 is an explanatory diagram illustrating a multi-view codec.

FIG39 is an explanatory diagram illustrating an image encoding process applied to a multi-view codec according to an embodiment.

FIG40 is an explanatory diagram illustrating an image decoding process applied to a multi-view codec according to an embodiment.

FIG41 is an explanatory diagram illustrating a scalable codec.

FIG42 is an explanatory diagram illustrating an image encoding process applied to a scalable codec according to an embodiment.

FIG43 is an explanatory diagram illustrating an image decoding process applied to a scalable codec according to an embodiment.

FIG44 is a block diagram illustrating a schematic structure of a television set.

FIG45 is a block diagram illustrating a schematic structure of a mobile phone.

FIG46 is a block diagram illustrating a schematic structure of a recording/reproducing apparatus.

FIG47 is a block diagram illustrating a schematic structure of an image pickup apparatus.

DETAILED DESCRIPTION

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that in the specification and the drawings, elements having substantially the same function and structure are denoted by the same reference numerals, and repeated descriptions are omitted.

Specific embodiments will be described in the following order.

1. Equipment Overview

1-1. Image Coding Device

1-2. Image Decoding Device

2. Existing Technology

2-1. Basic structure of deblocking filter

2-2. Dependencies between processes according to existing technologies

3. First embodiment

3-1. Deblocking filter structure example

3-2. Modification of Judgment Conditions

3-3. Processing Flow

4. Second embodiment

4-1. Deblocking filter structure example

4-2. Processing Flow

4-3. Processing Example for Each LCU

5. Third embodiment

5-1. Overview

5-2. Deblocking filter structure example

5-3. Processing sequence example

5-4. Processing Flow

6. Application of various codecs

6-1. Multi-view codec

6-2. Scalable Codec

7. Example Application

8. Summary

1. Equipment Overview

1 and 2 , the following describes an overview of devices to which the technology disclosed in this specification is applicable. The technology disclosed in this specification is applicable to an image encoding device and an image decoding device.

[1-1. Image Coding Device]

FIG1 is a block diagram showing an example of the structure of an image encoding device 10 according to an embodiment. Referring to FIG1 , the image encoding device 10 includes an A/D (Analog-Digital) conversion section 11, a rearrangement buffer 12, a subtraction section 13, an orthogonal transform section 14, a quantization section 15, a lossless encoding unit 16, an accumulation buffer 17, a rate control section 18, an inverse quantization section 21, an inverse orthogonal transform section 22, an addition section 23, a deblocking filter 24 a, a frame memory 25, a selector 26, an intra-frame prediction section 30, a motion estimation section 40, and a mode selection section 50.

The A/D conversion section 11 converts an image signal input in an analog format into image data in a digital format, and then outputs a series of digital image data to the reordering buffer 12 .

The reordering buffer 12 reorders images included in a series of image data input from the A/D conversion section 11. After reordering the images according to a GOP (Group of Pictures) structure corresponding to the encoding process, the reordering buffer 12 outputs the reordered image data to the subtraction section 13, the intra prediction section 30, and the motion estimation section 40.

The image data input from the reordering buffer 12 and the predicted image data selected by the mode selection section 50 described later are supplied to the subtraction section 13. The subtraction section 13 calculates prediction error data, which is the difference between the image data input from the reordering buffer 12 and the predicted image data input from the mode selection section 50, and outputs the calculated prediction error data to the orthogonal transform section 14.

The orthogonal transform section 14 performs an orthogonal transform on the prediction error data input from the subtraction section 13. The orthogonal transform performed by the orthogonal transform section 14 may be, for example, a discrete cosine transform (DCT) or a Karhunen-Loeve transform. The orthogonal transform section 14 outputs the transform coefficient data obtained by the orthogonal transform process to the quantization section 15.

The transform coefficient data input from the orthogonal transform section 14 and the rate control signal from the rate control section 18 described later are supplied to the quantization section 15. The quantization section 15 quantizes the transform coefficient data and outputs the quantized transform coefficient data (hereinafter referred to as quantized data) to the lossless encoding section 16 and the inverse quantization section 21. In addition, the quantization section 15 switches the quantization parameter (quantization scale) based on the rate control signal from the rate control section 18, thereby changing the bit rate of the quantized data to be input to the lossless encoding section 16.

The quantized data input from the quantization section 15 and the information on intra-frame prediction or inter-frame prediction generated by the intra-frame prediction section 30 or the motion estimation section 40 described later and selected by the mode selection section 50 are supplied to the lossless encoding section 16. The information on intra-frame prediction may include prediction mode information indicating the optimal intra-frame prediction mode for each block. In addition, the information on inter-frame prediction may include, for example, prediction mode information for predicting the motion vector of each block, differential motion vector information, reference image information, and the like.

The lossless encoding section 16 generates a coded stream by performing lossless encoding on the quantized data. The lossless encoding performed by the lossless encoding section 16 may be, for example, variable-length coding or arithmetic coding. Furthermore, the lossless encoding section 16 multiplexes the above-mentioned information regarding intra-frame prediction or inter-frame prediction into the header of the coded stream (e.g., block header, slice header, etc.). The lossless encoding section 16 then outputs the generated coded stream to the accumulation buffer 17.

The accumulation buffer 17 temporarily stores the encoded stream input from the lossless encoding section 16 using a storage medium such as a semiconductor memory. Then, the accumulation buffer 17 outputs the accumulated encoded stream at a rate corresponding to the frequency band of the transmission line (or the output line from the image encoding device 10).

The rate control section 18 monitors the free space of the accumulation buffer 17. Subsequently, the rate control section 18 generates a rate control signal according to the free space in the accumulation buffer 17, and outputs the generated rate control signal to the quantization section 15. For example, when there is not much free space in the accumulation buffer 17, the rate control section 18 generates a rate control signal to reduce the bit rate of the quantized data. Alternatively, for example, when the free space in the accumulation buffer 17 is sufficiently large, the rate control section 18 generates a rate control signal to increase the bit rate of the quantized data.

The inverse quantization section 21 performs inverse quantization processing on the quantized data input from the quantization section 15. Then, the inverse quantization section 21 outputs transform coefficient data obtained by the inverse quantization processing to the inverse orthogonal transform section 22.

The inverse orthogonal transform section 22 performs an inverse orthogonal transform process on the transform coefficient data input from the inverse quantization section 21 , thereby restoring the prediction error data.

The addition section 23 adds the restored prediction error data input from the inverse orthogonal transform section 22 and the predicted image data input from the mode selection section 50, thereby generating decoded image data. Subsequently, the addition section 23 outputs the generated decoded image data to the deblocking filter 24a and the frame memory 25.

The deblocking filter 24a performs filtering to reduce block distortion that occurs during image encoding. For example, the deblocking filter 24a determines the necessity of filtering for each block boundary of the decoded image data supplied from the addition section 23, and applies the deblocking filter to the boundary determined to require filtering. The deblocking filter 24a is also supplied with information used to determine the necessity of filtering (e.g., mode information, transform coefficient information, and motion vector information), as well as the decoded image data from the addition section 23. After filtering, block distortion is removed from the decoded image data, and the deblocking filter 24a outputs the decoded image data to the frame memory 25. The processing of the deblocking filter 24a will be described in detail later.

The frame memory 25 stores the decoded image data input from the adding section 23 and the decoded image data after filtering input from the deblocking filter 24 a using a storage medium.

The selector 26 reads the decoded image data before filtering to be used for intra-frame prediction from the frame memory 25, and supplies the read decoded image data as reference image data to the intra-frame prediction section 30. In addition, the selector 26 reads the decoded image data after filtering to be used for inter-frame prediction from the frame memory 25, and supplies the read decoded image data as reference image data to the motion estimation section 40.

The intra-frame prediction section 30 performs intra-frame prediction processing according to each intra-frame prediction mode based on the image data to be encoded input from the reordering buffer 12 and the decoded image data supplied via the selector 26. For example, the intra-frame prediction section 30 evaluates the prediction results of each intra-frame prediction mode using a predetermined cost function. The intra-frame prediction section 30 then selects the intra-frame prediction mode that minimizes the cost function value, that is, the intra-frame prediction mode that maximizes the compression ratio, as the optimal intra-frame prediction mode. Furthermore, the intra-frame prediction section 30 outputs prediction mode information indicating the optimal intra-frame prediction mode, the predicted image data, and information regarding the intra-frame prediction, such as the cost function value, to the mode selection section 50.

The motion estimation section 40 performs inter-frame prediction processing (inter-frame prediction processing) based on the encoding image data supplied from the reordering buffer 12 and the decoded image data supplied via the selector 26. For example, the motion estimation section 40 evaluates the prediction results of each prediction mode using a predetermined cost function. The motion estimation section 40 then selects the optimal prediction mode, that is, the prediction mode that minimizes the cost function value or maximizes the compression ratio. The motion estimation section 40 generates predicted image data according to the optimal prediction mode. The motion estimation section 40 outputs information about inter-frame prediction, such as prediction mode information indicating the optimal intra-frame prediction mode, predicted image data, and the cost function value, to the mode selection section 50.

The mode selection section 50 compares the cost function value related to intra-frame prediction input from the intra-frame prediction section 30 with the cost function value related to inter-frame prediction input from the motion estimation section 40. The mode selection section 50 then selects the prediction method with the smaller cost function value from the intra-frame prediction and the inter-frame prediction. If intra-frame prediction is selected, the mode selection section 50 outputs information about the intra-frame prediction to the lossless encoding section 16 and outputs predicted image data to the subtraction section 13 and the addition section 23. Furthermore, if inter-frame prediction is selected, the mode selection section 50 outputs the information about the inter-frame prediction described above to the lossless encoding section 16 and outputs predicted image data to the subtraction section 13 and the addition section 23.

[1-2. Image decoding device]

FIG2 is a block diagram showing a structural example of an image decoding device 60 according to an embodiment. Referring to FIG2 , the image decoding device 60 includes an accumulation buffer 61, a lossless decoding section 62, an inverse quantization section 63, an inverse orthogonal transform section 64, an addition section 65, a deblocking filter 24 b, a rearrangement buffer 67, a D/A (digital-to-analog) conversion section 68, a frame memory 69, selectors 70 and 71, an intra-frame prediction section 80, and a motion compensation section 90.

The accumulation buffer 61 temporarily stores the encoded stream input via the transmission line using a storage medium.

The lossless decoding section 62 decodes the coded stream input from the accumulation buffer 61 according to the encoding method used during encoding. Furthermore, the lossless decoding section 62 decodes information multiplexed into the header area of the coded stream. This information multiplexed into the header area of the coded stream may include, for example, information regarding intra-frame prediction and information regarding inter-frame prediction in a block header. The lossless decoding section 62 outputs the information regarding intra-frame prediction to the intra-frame prediction section 80. Furthermore, the lossless decoding section 62 outputs the information regarding inter-frame prediction to the motion compensation section 90.

The inverse quantization section 63 inversely quantizes the quantized data decoded by the lossless decoding section 62. The inverse orthogonal transform section 64 generates prediction error data by performing inverse orthogonal transform on the transform coefficient data input from the inverse quantization section 63 according to the orthogonal transform method used at the time of encoding. Subsequently, the inverse orthogonal transform section 64 outputs the generated prediction error data to the addition section 65.

The addition section 65 adds the prediction error data input from the inverse orthogonal transform section 64 and the predicted image data input from the selector 71, thereby generating decoded image data.

The deblocking filter 24b performs filtering to reduce block distortion that appears in the decoded image. For example, the deblocking filter 24b determines the necessity of filtering at each block boundary for the decoded image data input from the adding section 65, and applies the deblocking filter to the boundaries where filtering is determined to be necessary. The deblocking filter 24b is also supplied with information used to determine the necessity of filtering, as well as the decoded image data from the adding section 65. After filtering, block distortion is removed from the decoded image data, and the deblocking filter 24b outputs the decoded image data to the reordering buffer 67 and the frame memory 69. The processing of the deblocking filter 24b will be described in detail later.

The rearrangement buffer 67 generates a series of image data in time sequence by rearranging the image input from the deblocking filter 24b. Then, the rearrangement buffer 67 outputs the generated image data to the D/A conversion section 68.

The D/A conversion section 68 converts the image data in digital format input from the rearrangement buffer 67 into an image signal in analog format. Then, the D/A conversion section 68 causes the image to be displayed, for example, by outputting the analog image signal to a display (not shown) connected to the image decoding device 60.

The frame memory 69 stores the decoded image data before filtering input from the adding section 65 and the decoded image data after filtering input from the deblocking filter 24 b using a storage medium.

The selector 70 switches the output destination of the image data from the frame memory 69 between the intra prediction section 80 and the motion compensation section 90 for each block in the image according to the mode information obtained by the lossless decoding section 62. For example, when the intra prediction mode is specified, the selector 70 outputs the decoded image data before filtering supplied from the frame memory 69 as reference image data to the intra prediction section 80. On the other hand, when the inter prediction mode is specified, the selector 70 outputs the decoded image data after filtering supplied from the frame memory 69 as reference image data to the motion compensation section 90.

The selector 71 switches the output source of the predicted image data to be supplied to the addition section 65 between the intra prediction section 80 and the motion compensation section 90 for each block in the image in accordance with the mode information obtained by the lossless decoding section 62. For example, in the case where the intra prediction mode is specified, the selector 71 supplies the predicted image data output from the intra prediction section 80 to the addition section 65. In the case where the inter prediction mode is specified, the selector 71 supplies the predicted image data output from the motion compensation section 90 to the addition section 65.

The intra prediction section 80 performs in-screen prediction of pixel values based on the information on intra prediction input from the lossless decoding section 62 and the reference image data from the frame memory 69, thereby generating predicted image data. The intra prediction section 80 then outputs the generated predicted image data to the selector 71.

The motion compensation section 90 performs motion compensation processing based on the information on inter-frame prediction input from the lossless decoding section 62 and the reference image data from the frame memory 69 to generate predicted image data. The motion compensation section 90 then outputs the generated predicted image data to the selector 71.

<2. Prior Art>

[2-1. Basic structure of deblocking filter]

Generally, the process of using a deblocking filter in existing image coding schemes such as H.264/AVC or HEVC includes two processes: a filtering necessity determination process and a filtering process. For example, these two processes in HEVC are described below.

(1) Filtering requires judgment processing

The filtering need determination process determines whether it is necessary to apply a deblocking filter to each boundary of each block within the input image. Block boundaries include vertical boundaries between blocks that are horizontally adjacent to each other, and horizontal boundaries between blocks that are vertically adjacent to each other. JCTVC-A119 uses a block size of 8×8 pixels as the minimum processing unit. For example, a macroblock of 16×16 pixels includes 4 blocks of 8×8 pixels. This process is applied to one (left) vertical boundary and one (top) horizontal boundary of each block, that is, 4 boundaries plus 4 boundaries totaling 8 boundaries. This specification assumes that macroblocks, which are technical terms, include coding units (CUs) in the context of HEVC.

FIG3 is an illustrative diagram showing an example of pixels in two adjacent blocks (adjacent blocks) Ba and Bb around a boundary. The following example illustrates a vertical boundary, but the explanation is clearly applicable to horizontal boundaries. The example in FIG3 uses the symbol p _ij to represent pixels in block Ba. In this symbol, i represents a column index and j represents a row index. Starting from the column closest to the vertical boundary, the column index i is numbered sequentially (from right to left) as 0, 1, 2, and 3. From top to bottom, the row index j is numbered as 0, 1, 2, ..., 7. The left half of block Ba is omitted in the figure. The symbol q _kj is used to represent pixels in block Bb. In this symbol, k represents a column index and j represents a row index. Starting from the column closest to the vertical boundary, the column index k is numbered sequentially (from left to right) as 0, 1, 2, and 3. The right half of block Bb is omitted in the figure.

The necessity of applying the deblocking filter to the vertical boundary between blocks Ba and Bb shown in FIG. 3 can be determined using the following conditions.

Decision conditions for the luminance component (Luma)... If conditions A and B are both true, then apply the deblocking filter.

Condition A:

(A1) Block Ba or Bb enters intra prediction mode;

(A2) block Ba or Bb has a non-zero orthogonal transform coefficient; or

(A3) |MVAx-MVBx|≥4 or |MVAy-MVBy|≥4

Condition B:

|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |<β

Condition A3 assumes the motion vector of block Ba is (MVAx, MVAy) and the motion vector of block Bb is (MVBx, MVBy) using Qpel (1/4 pixel) precision. Condition B uses β as the edge determination threshold. The initial value of β is given according to the quantization parameter. The user can specify the value of β using parameters in the slice header.

Decision conditions for chroma components (Chroma) ... If condition A1 is true, then apply the deblocking filter.

Condition A1: Block Ba or Bb enters intra prediction mode.

As shown by the dotted boxes L3 and L6 in FIG4 , the filtering of general vertical boundaries requires determination processing (particularly according to determination condition B of the luminance component) with reference to pixels on the third and sixth rows (assuming the top row is the first row) in each block. Similarly, the filtering of horizontal boundaries requires determination processing with reference to pixels on the third and sixth columns (not shown in FIG4 ) in each block. The above determination conditions are used to determine the boundaries on which the deblocking filter needs to be applied to perform the filtering processing described below.

(2) Filtering

If it is determined that a deblocking filter needs to be applied to the boundary, the pixels on the left and right sides of the vertical boundary and the pixels above and below the horizontal boundary are filtered. Regarding the luminance component, the filter strength is switched between strong and weak filters according to the pixel value.

Filtering the luminance component

Select Strength... Select the filter strength for each row or column. If all of the following conditions C1-C3 are met, then a strong filter is selected. If any of the following conditions are not met, then a weak filter is selected.

(C1)d<(β>>2)

(C2)(|p _3j -p _0j |+|q _0j -q _3j |)<(β>>3)

(C3)|p _0j -q _0j |<((5t _C +1)>>1)

Where j represents the row index of the vertical boundary or the column index of the horizontal boundary. d = |p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |.

Weak filtering

△＝Clip(-t _C ,t _C ,(13(q _0j -p _0j )+4(q _1j -p _1j )-5(q _2j -p _2j )+16)>>5))

p _0j =Clip _0-255 (p _0j +△)

q _0j =Clip _0-255 (q _0j -△)

p _1j =Clip _0-255 (p _1j +△/2)

q _1j =Clip _0-255 (q _1j -△/2)

Strong filtering

p _0j ＝Clip _0-255 ((p _2j +2p _1j +2p _0j +2q _0j +q _1j +4)>>3)

q _0j ＝Clip _0-255 ((p _1j +2p _0j +2q _0j +2q _1j +q _2j +4)>>3)

p _1j ＝Clip _0-255 ((p _2j +p _1j +p _0j +q _0j +2)>>2)

q _1j ＝Clip _0-255 ((p _0j +q _0j +q _1j +q _2j +2)>>2)

p _2j ＝Clip _0-255 ((2p _3j +3p _2j +p _1j +p _0j +q _0j +4)>>3)

q _2j ＝Clip _0-255 ((p _0j +q _0j +q _1j +3q _2j +2q _3j +4)>>3)

Here, Clip(a,b,c) indicates a process of clamping the value c within the range of a≤c≤b, and Clip _0-255 (c) indicates a process of clamping the value c within the range of 0≤c≤255.

Filtering chroma components

△＝Clip(-t _C ,t _C ,((((q _0j -p _0j )<<2)+p _1j -q _1j +4)>>3))

p _0j =Clip _0-255 (p _0j +△)

q _0j =Clip _0-255 (q _0j -△)

As shown by the dashed boxes C6-C8 and C1-C3 in Figure 5, the filtering process for general vertical boundaries (especially the strong filtering for the luminance component) updates the pixel values on the first to third columns and the sixth to eighth columns in each block. Similarly, the filtering process for horizontal boundaries updates the pixel values on the first to third rows and the sixth to eighth rows in each block.

[2-2. Dependencies between Processes According to Conventional Technology]

For the sake of explanation, as shown in FIG6 , macroblocks MBx (MB0, MB1, ...) each having a size of 16×16 pixels include an upper left vertical boundary denoted as Vx,0, an upper middle vertical boundary denoted as Vx,1, a lower left vertical boundary denoted as Vx,2, a lower middle vertical boundary denoted as Vx,3, an upper left horizontal boundary denoted as Hx,0, an upper right horizontal boundary denoted as Hx,1, a left middle horizontal boundary denoted as Hx,2, and a right middle horizontal boundary denoted as Hx,3. Regarding boundary Z, for example, the filtering necessity determination process is denoted as J _Z , and the filtering process is denoted as F _Z .

The above-described prior art results in no dependencies between processes for boundaries in the same direction within a macroblock. Therefore, this technique enables parallel filtering of vertical and horizontal boundaries within a macroblock. For example, Figure 7 clearly shows that within macroblock MB0, there are no dependencies between the four filtering processes F _V0,0 , F _V0,1 , F _V0,2 , and F _V0,3 (no pixels are repeatedly updated), allowing the filtering processes to be performed in parallel.

However, the above-mentioned prior art leaves behind a dependency between the filtering process for vertical boundaries and the filtering necessity determination process for horizontal boundaries. This prior art also leaves behind a dependency between the filtering process for horizontal boundaries and the filtering necessity determination process for vertical boundaries. For example, if vertical boundaries are processed before horizontal boundaries, then the filtering necessity determination process for horizontal boundaries within a given macroblock must be performed after the filtering process for vertical boundaries is terminated. For example, Figure 8 shows that within macroblock MB0, the filtering necessity determination process J _H0,0 depends on the results of the filtering processes F _V0,0 and F _V0,1 , and the filtering necessity determination process J _H0,1 depends on the result of the filtering process F _V0,1 . Similarly, the filtering necessity determination process for vertical boundaries within a given macroblock must be performed after the filtering process for the horizontal boundaries of adjacent macroblocks is terminated. For example, FIG9 shows that the filtering necessity determination process J _V1,0 for macroblock MB1 depends on the results of the filtering processes F _H0,1 and F _H0,3 for macroblock MB0, and the filtering necessity determination process J _V1,2 for macroblock MB1 depends on the result of the filtering process F _H0,3 for macroblock MB0.

The existing technology involves dependencies between processes, so even with the technology proposed in JCTVC-A119, parallel processing of deblocking filters can be provided to a very limited extent.

FIG10 is an explanatory diagram illustrating a sequence of deblocking filter processing according to the prior art. This example assumes that the deblocking filter is fed with an image of size 32×32 pixels. The input image includes four macroblocks MB0-MB3, each of which has a size of 16×16 pixels.

In Figure 10, each dotted box represents a process that will be performed in parallel. For example, in the first step, filtering is performed in parallel on the four vertical boundaries of macroblock MB0: J _V0,0 , J _V0,1 , J _V0,2 , and J _V0,3 . In the second step, filtering is performed in parallel on the four vertical boundaries of macroblock MB0: F _V0,0 , F _V0,1 , F _V0,2 , and F _V0,3 . After the second step is completed, the third step performs filtering in parallel on the four horizontal boundaries of macroblock MB0: J _H0,0 , J _H0,1 , J _H0,2 , and J _H0,3 . The third step uses the pixel values after the filtering of the vertical boundaries in the second step to determine the need for filtering on the horizontal boundaries. In the fourth step, filtering is performed in parallel on the four horizontal boundaries of macroblock MB0: F _H0,0 , F _H0,1 , F _H0,2 , and F _H0,3 . After the fourth step is terminated, processing is continued for macroblock MB1 (steps 5 to 8). In the fifth step, the pixel values obtained after the filtering process for the horizontal boundary of macroblock MB0 in step 4 are used to determine the need for filtering for the vertical boundary of macroblock MB1. After the processing for macroblock MB1 is terminated, processing is continued for macroblock MB2 (steps 9 to 12). After the processing for macroblock MB2 is terminated, processing is continued for macroblock MB3 (steps 13 to 16).

This limited degree of parallel processing cannot satisfactorily solve the problem of delay or data rate reduction caused by the large amount of processing when applying the deblocking filter. The three embodiments described below further improve the parallel processing when applying the deblocking filter.

<3. First embodiment>

[3-1. Deblocking filter structure example]

The following describes exemplary structures of the deblocking filter 24a of the image encoding device 10 shown in FIG. 1 and the deblocking filter 24b of the image decoding device 60 shown in FIG. 2 according to the first embodiment. The structures of the deblocking filter 24a and the deblocking filter 24b may be identical to each other. In the following description, when there is no need to distinguish between the deblocking filter 24a and the deblocking filter 24b, they are collectively referred to as the deblocking filter 24.

(1) Dependencies between new processes

According to this embodiment, the processing using the deblocking filter 24 also includes two processes: filtering necessity determination processing and filtering processing. In the filtering necessity determination processing regarding either vertical or horizontal boundaries, the deblocking filter 24 uses the pixel values of the image input to the deblocking filter across multiple macroblocks for the determination. For example, if vertical boundaries are processed before horizontal boundaries, the deblocking filter 24 can perform filtering necessity determination processing on the vertical boundary of a particular block without having to wait for filtering processing on the horizontal boundary of an adjacent block. If horizontal boundaries are processed before vertical boundaries, the deblocking filter 24 can perform filtering necessity determination processing on the horizontal boundary of a particular block without having to wait for filtering processing on the horizontal boundary of an adjacent block. As a result, processing dependencies between macroblocks are reduced.

Reducing inter-macroblock processing dependencies enables parallel processing across multiple macroblocks within an image. For example, this enables parallel processing of the need for filtering across all vertical boundaries of all blocks within the input image. This also enables parallel processing of the need for filtering across all horizontal boundaries of all blocks within the input image.

Figure 11 is an explanatory diagram illustrating a processing sequence that can be used in this embodiment. This example also assumes that the deblocking filter is fed with an image of size 32x32 pixels. The input image includes four macroblocks MB0-MB3, each of size 16x16 pixels.

In Figure 11, each dotted box represents a process that will be performed in parallel. While the example in Figure 10 requires 16 processing steps for the processing sequence, the example in Figure 11 combines the same number of processes into four steps. The first step involves performing filtering in parallel on all vertical and horizontal boundaries of all macroblocks MB0-MB3, requiring the decision processing J _V0,0 to J _V3,3 and J _H0,0 to J _H3,3 . The second step involves performing filtering in parallel on all 16 vertical boundaries of all macroblocks MB0-MB3, requiring the decision processing F _V0,0 to F _V3,3 . The third step involves performing filtering in parallel on all horizontal boundaries of all macroblocks MB0-MB3, requiring the decision processing F _H0,0 to F _H3,3 . The fourth step involves performing filtering in parallel on all 16 horizontal boundaries of all macroblocks MB0-MB3, requiring the decision processing F _H0,0 to F _H3,3 . If horizontal boundaries are processed before vertical boundaries, the third and fourth steps can precede the first and second steps.

Figure 11 provides an example of maximizing parallelism (processing performed in parallel) by parallelizing processing in all macroblocks of a picture. However, this example is not limiting and processing may be parallelized in some macroblocks rather than in all macroblocks of a picture.

(2) Detailed structure of the deblocking filter

FIG12 is a block diagram illustrating a detailed structure of the deblocking filter 24 that performs the above-mentioned parallel processing according to the first embodiment. Referring to FIG12 , the deblocking filter 24 includes a vertical determination unit 110, a horizontal determination unit 114, a horizontal filtering unit 130, a vertical filtering unit 140, and a parallelization control unit 150.

(2-1) Vertical determination component

The vertical decision section 110 includes a plurality of vertical boundary decision sections 112-1 to 112-n. Each vertical boundary decision section 112 is supplied with an image input to the deblocking filter 24 and decision information for determining whether filtering is required.

The vertical boundary determination sections 112-1 to 112-n determine whether to apply a deblocking filter to a vertical boundary by utilizing pixel values of an image input to the deblocking filter 24 across a plurality of macroblocks within the image. Each vertical boundary determination section 112 supplies information indicating the determination result regarding each vertical boundary, such as binary information indicating a determination result that a value of "1" forces the application of a deblocking filter, to the horizontal filtering section 130.

(2-2) Horizontal filter component

The horizontal filtering section 130 includes a plurality of horizontal filtering sections 132-1 to 132-n. Each horizontal filtering section 132 is supplied with an input image and a decision result regarding each vertical boundary from the vertical decision section 110.

The decision result from the corresponding vertical boundary decision section 112 may indicate that a filter needs to be applied. In this case, each horizontal filtering component 132 applies a deblocking filter with respect to the vertical boundary to the elements on the left and right sides of the vertical boundary. Each horizontal filtering component 132 provides the filtered pixel value of the pixel to which the filter is applied and the pixel values of the input image with respect to the other pixels to the horizontal decision component 114 and the vertical filtering component 140.

(2-3) Level determination component

The horizontal decision section 114 includes a plurality of horizontal boundary decision portions 116-1 to 116-n. Each horizontal boundary decision portion 116 is supplied with pixel values after filtering by the horizontal filtering section 130 and decision information for determining whether filtering is required.

Horizontal boundary determination sections 116-1 to 116-n determine whether to apply a deblocking filter to a horizontal boundary across a plurality of macroblocks within an image using pixel values after filtering by horizontal filtering section 130. Each horizontal boundary determination section 116 supplies information indicating the determination result regarding each horizontal boundary to vertical filtering section 140.

(2-4) Vertical filter components

The vertical filtering section 140 includes a plurality of vertical filtering sections 142 - 1 to 142 - n. Each vertical filtering section 142 is supplied with pixel values after filtering by the horizontal filtering section 130 and a decision result from the horizontal decision section 114 regarding each horizontal boundary.

The decision result from the corresponding horizontal boundary decision section 116 may indicate that the filter needs to be applied. In this case, each vertical filter component 142 applies the deblocking filter with respect to the horizontal boundary to the elements above and below the horizontal boundary. Each vertical filter component 142 supplies the filtered pixel value to the pixel to which the filter is applied, and supplies the pixel value provided by the horizontal filter component 130 to the other pixels. The output from each vertical filter component 142 may constitute the output image from the deblocking filter 24.

(3) More detailed structure of the judgment part

13 is a block diagram illustrating a detailed structure of each of the vertical boundary decision section 112 and the horizontal boundary decision section 116. Referring to FIG13, each decision section includes a tap forming section 121, a calculation section 122, a threshold comparison section 123, a distortion evaluation section 124, and a filter decision section 125.

The tap formation section 121 obtains reference pixel values from the pixel values of two adjacent blocks straddling a boundary of interest in the input image and forms taps (a set of reference pixel values) that determine the determination condition B for the luminance component. For example, in blocks of 8×8 pixels, the focus may be on the vertical boundary. In this case, the tap formation section 121 forms taps based on the pixel values in the third and sixth rows of the two blocks on the left and right sides. If the focus is on the horizontal boundary, the tap formation section 121 may form taps based on the pixel values in the third and sixth columns of the two blocks on the top and bottom sides. The calculation section 122 assigns the taps formed by the tap formation section 121 to the left side of the determination expression in the determination condition B and calculates the edge value to be compared with the edge determination threshold β. The threshold comparison section 123 compares the value calculated by the calculation section 122 with the edge determination threshold β and outputs the comparison result to the filter determination section 125.

The distortion evaluation section 124 evaluates the determination condition A for the luminance component using the mode information (MB mode), transform coefficient information, and motion vector information supplied as determination information. The distortion evaluation section 124 outputs the evaluation result to the filter determination section 125. The distortion evaluation section 124 evaluates only the determination condition A1 for the chrominance component based on the mode information.

The filter determination section 125 determines whether to apply a deblocking filter to the boundary of interest based on the comparison result of the determination condition B supplied from the threshold comparison section 123 and the evaluation result of the determination condition A supplied from the distortion evaluation section 124. The filter determination section 125 outputs information indicating the determination result.

(4) Parallelization control part

The parallelization control section 150 shown in FIG. 12 controls the parallelism of the filtering necessity determination processing in the vertical determination section 110 and the horizontal determination section 114 , and the parallelism of the filtering processing in the horizontal filtering section 130 and the vertical filtering section 140 .

For example, the parallelization control part 150 can control the parallelism of the processing of each block according to the input image size. More specifically, if the input image size is large, the parallelization control part 150 increases the parallelism of the processing of each block. This can adaptively prevent the delay or reduction of the data rate caused by the processing amount increased accordingly with the image size. For example, the parallelization control part 150 can control the parallelism of the processing of each block according to the sequence parameter group, the picture parameter group, or the parameters contained in the slice header. This makes it possible to flexibly configure the parallelism according to the requirements of the user who develops the device. For example, the parallelism can be configured according to the restrictions on the installation environment, such as the number of processor cores or the number of software threads.

This embodiment enables parallel processing across macroblocks. This means that any processing sequence for each block within an image has no effect on the final output result. Therefore, the parallelization control section 150 can control the filtering need determination processing sequence in the vertical determination unit 110 and the horizontal determination unit 114, and the filtering processing sequence in the horizontal filtering unit 130 and the vertical filtering unit 140 on a block-by-block basis.

More specifically, the parallelization control section 150 can control the filtering process sequence based on the dependency of filtering processes between macroblocks. For example, according to conventional techniques, the dependency of processing between adjacent macroblocks around a slice boundary can delay the parallel processing of each slice within an image. However, according to this embodiment, the parallelization control section 150 can filter adjacent macroblocks around a slice boundary before other macroblocks.

For example, FIG14 illustrates eight macroblocks MB10-MB13 and MB20-MB23 around a slice boundary. Macroblocks MB10-MB13 belong to slice SL1. Macroblocks MB20-MB23 belong to slice SL2. With respect to these macroblocks, the filtering process for the horizontal boundary of macroblock MB20 in slice SL2 depends on the filtering process for the vertical boundary of macroblock MB12 in slice SL1. Similarly, the filtering process for the horizontal boundary of macroblock MB21 in slice SL2 depends on the filtering process for the vertical boundary of macroblock MB13 in slice SL1.

Under these conditions, according to the example in FIG. 15 , the parallelization control unit 150 prioritizes filtering the vertical boundaries of macroblocks MB12 and MB13 in the filtering process for slice SL1 over processing other boundaries. This prevents a significant delay from occurring in the filtering process for the horizontal boundaries of macroblocks MB20 and MB21 in the filtering process for slice SL2. The example in FIG. 16 initially performs filtering on the vertical boundaries of all macroblocks included in slice SL1 in parallel. Furthermore, in this case, no delay occurs in the filtering process for the horizontal boundaries of macroblocks MB20 and MB21 in slice SL2.

[3-2. Modification of Judgment Conditions]

As described above, similar to the prior art illustrated in FIG4 , each vertical boundary determination section 112 refers to the pixels corresponding to the third and sixth rows in a block and determines whether filtering is required for each vertical boundary of the block. Similarly, each horizontal boundary determination section 116 refers to the pixels corresponding to the third and sixth columns in a block and determines whether filtering is required for each horizontal boundary of the block. In this case, the structure according to this embodiment can be easily implemented without changing the determination conditions for the filtering necessity determination process provided in the prior art.

Each vertical boundary determination section 112 and each horizontal boundary determination section 116 can perform determinations using different determination conditions than those in the prior art. For example, each vertical boundary determination section 112 can refer to pixels corresponding to three or more columns in a block. Each horizontal boundary determination section 116 can refer to pixels corresponding to three or more columns in a block. Furthermore, each vertical boundary determination section 112 and each horizontal boundary determination section 116 can use different determination condition expressions than those in the prior art. Referring to Figures 17-19, six examples of determination techniques according to this embodiment are described below.

(1) First example

FIG17 is an explanatory diagram illustrating the first and second examples of the determination technique. In the first and second examples, the filtering necessity determination process regarding the vertical boundary (particularly, determination condition B is applied to the determination of the luminance component) refers to the pixels of all rows L1 to L8 from the first row to the eighth row in each block. The filtering necessity determination process regarding the horizontal boundary similarly refers to the pixels of all columns from the first column to the eighth column in each block.

As a first example, the determination condition of the luminance component can be defined as follows.

Decision conditions for the luminance component (Luma)... If both condition A and condition B are true, then apply the deblocking filter.

Condition A:

(A1) Block Ba or Bb enters intra prediction mode;

(A2) block Ba or Bb has a non-zero orthogonal transform coefficient; or

(A3)|MVAx-MVBx|≥4 or |MVAy-MVBy|≥4

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₇ -2p ₁₇ +p ₀₇ |+|q ₂₇ -2q ₁₇ +q ₀₇ |

iD ₁ =|p ₂₁ -2p ₁₁ +p ₀₁ |+|q ₂₁ -2q ₁₁ +q ₀₁ |+|p ₂₆ -2p ₁₆ +p ₀₆ |+|q ₂₆ -2q ₁₆ +q ₀₆ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |

iD ₃ =|p ₂₃ -2p ₁₃ +p ₀₃ |+|q ₂₃ -2q ₁₃ +q ₀₃ |+|p ₂₄ -2p ₁₄ +p ₀₄ |+|q ₂₄ -2q ₁₄ +q ₀₄ |

iD _ave =(iD ₀ +iD ₁ +iD ₂ +iD ₃ )>>2

Under this condition, iD _ave < β

The determination condition of the chrominance component can be the same as that of the prior art. A weighted average value can be calculated to calculate the average value iD _ave of the four determination parameters iD ₀ -iD ₃ .

(2) Second example

As a second example, the determination condition B of the luminance component can be defined as follows.

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₇ -2p ₁₇ +p ₀₇ |+|q ₂₇ -2q ₁₇ +q ₀₇ |

iD ₁ =|p ₂₁ -2p ₁₁ +p ₀₁ |+|q ₂₁ -2q ₁₁ +q ₀₁ |+|p ₂₆ -2p ₁₆ +p ₀₆ |+|q ₂₆ -2q ₁₆ +q ₀₆ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |

iD ₃ =|p ₂₃ -2p ₁₃ +p ₀₃ |+|q ₂₃ -2q ₁₃ +q ₀₃ |+|p ₂₄ -2p ₁₄ +p ₀₄ |+|q ₂₄ -2q ₁₄ +q ₀₄ |

According to this condition, iD ₀ <β, iD ₁ <β, iD ₂ <β and iD ₃ <β

The equations for calculating the four determination parameters iD ₀ -iD ₃ are the same as those in the first example. The applicable condition is not that all four determination parameters iD ₀ -iD ₃ are smaller than the edge determination threshold β, but that at least three, two, or one of the four determination parameters iD ₀ -iD ₃ are smaller than the edge determination threshold β.

(3) The third example

FIG18 is an explanatory diagram illustrating third and fourth examples of the determination technique. In the third and fourth examples, the filtering necessity determination process regarding vertical boundaries (particularly, determination condition B is applied to the luminance component) refers to pixels in four rows, L1, L3, L6, and L8, in each block. Similarly, the filtering necessity determination process regarding horizontal boundaries refers to pixels in four columns in each block.

As a third example, the determination condition of the luminance component may be defined as follows.

Decision conditions for the luminance component (Luma)... If conditions A and B are both true, then apply the deblocking filter.

Condition A:

(A1) Block Ba or Bb enters intra prediction mode;

(A2) block Ba or Bb has a non-zero orthogonal transform coefficient; or

(A3)|MVAx-MVBx|≥4 or |MVAy-MVBy|≥4

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₇ -2p ₁₇ +p ₀₇ |+|q ₂₇ -2q ₁₇ +q ₀₇ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |

iD _ave = (iD ₀ + iD ₂ ) >> 1

Under this condition, iD _ave < β

The determination condition of the chrominance component can be the same as that of the prior art. A weighted average value can be calculated to calculate the average value iD _ave of the two determination parameters iD ₀ and iD ₂ .

(4) The fourth example

Fourth Example The determination condition B of the luminance component can be defined as follows.

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₇ -2p ₁₇ +p ₀₇ |+|q ₂₇ -2q ₁₇ +q ₀₇ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |

Under this condition, iD ₀ <β and iD ₂ <β

The equations for calculating the two decision parameters iD ₀ and iD ₂ are the same as those in the third example. The applicable condition is not that both decision parameters iD ₀ and iD ₂ are less than the edge decision threshold β, but that either one of the two decision parameters iD ₀ and iD ₂ is less than the edge decision threshold β.

Although an example is described in which the first, third, sixth, and eighth rows (or columns) L1, L3, L6, and L8 in a block are referred to during determination, other combinations of rows and columns may be referred to.

(5) The fifth example

FIG19 is an explanatory diagram illustrating fifth and sixth examples of the determination technique. In the fifth and sixth examples, the determination process for vertical boundary filtering requires reference to pixels in four rows, L1, L3, L5, and L7, within each block. Similarly, the determination process for horizontal boundary filtering requires reference to pixels in four columns within each block.

The fifth example defines the determination condition of the luminance component as follows.

Decision conditions for the luminance component (Luma)... If conditions A and B are both true, then apply the deblocking filter.

Condition A:

(A1) Block Ba or Bb enters intra prediction mode;

(A2) block Ba or Bb has a non-zero orthogonal transform coefficient; or

(A3)|MVAx-MVBx|≥4 or |MVAy-MVBy|≥4

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₆ -2p ₁₆ +p ₀₆ |+|q ₂₆ -2q ₁₆ +q ₀₆ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₄ -2p ₁₄ +p ₀₄ |+|q ₂₄ -2q ₁₄ +q ₀₄ |

iD _ave = (iD ₀ + iD ₂ ) >> 1

Under this condition, iD _ave < β

The determination condition of the chrominance component can be the same as that of the prior art. A weighted average value can be calculated to calculate the average value iD _ave of the two determination parameters iD ₀ and iD ₂ .

(6) The sixth example

Sixth Example The determination condition B of the luminance component is defined as follows.

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₆ -2p ₁₆ +p ₀₆ |+|q ₂₆ -2q ₁₆ +q ₀₆ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₄ -2p ₁₄ +p ₀₄ |+|q ₂₄ -2q ₁₄ +q ₀₄ |

Under this condition, iD ₀ <β and iD ₂ <β

The equations for calculating the two decision parameters iD ₀ and iD ₂ are the same as those in the fifth example. The applicable condition is not that both decision parameters iD ₀ and iD ₂ are less than the edge decision threshold β, but that either one of the two decision parameters iD ₀ and iD ₂ is less than the edge decision threshold β.

Generally, increasing the number of rows and columns referenced during decision making improves decision accuracy. Thus, the first and second examples of referencing eight rows and eight columns minimize the likelihood of filtering blocks that were not initially targeted by the deblocking filter, and the likelihood of not filtering blocks that were initially targeted by the deblocking filter. This results in improved image quality for encoding and decoding. On the other hand, reducing the number of rows and columns referenced during decision making reduces processing costs. Because there is a tradeoff between image quality and processing costs, it is advantageous to adaptively select the number of rows and columns referenced during decision making, depending on the device's intended use or installation constraints. It is also advantageous to adaptively select the combination of rows and columns to be referenced.

As described in the first, third, and fifth examples, the average value iD _ave of the decision parameter and the edge decision threshold β can be compared to appropriately make decisions on a block basis without being excessively affected by parameter variations for each row or column.

[3-3. Processing Flow]

20 and 21 , the processing flow of the deblocking filter 24 will be described.

FIG20 is a flowchart illustrating the processing flow of the deblocking filter 24 according to the first embodiment. Referring to FIG20 , vertical boundary determination sections 112-1 to 112-n concurrently determine whether filtering is required for all vertical boundaries within a plurality of macroblocks included in the input image (step S110). Horizontal filtering components 132-1 to 132-n concurrently apply the deblocking filter to all vertical boundaries determined to require deblocking filtering in step S110 (step S120). Horizontal boundary determination sections 116-1 to 116-n concurrently determine whether filtering is required for all horizontal boundaries within a plurality of macroblocks included in the input image (step S130). Vertical filtering components 142-1 to 142-n concurrently apply the deblocking filter to all horizontal boundaries determined to require deblocking filtering in step S130 (step S140).

The above processing flow is merely an example. For example, the deblocking filter 24 may process two or more macroblocks in parallel. The processing sequence may be changed.

FIG21 is a flowchart illustrating the flow of the filtering necessity determination process corresponding to steps S110 and S130 in FIG20 . Referring to FIG21 , the distortion evaluation section 124 evaluates the boundary for distortion based on the mode information, transform coefficient information, and motion vector information (step S150). If the evaluation indicates the presence of distortion (determination condition A is true), the process proceeds to step S154. If the evaluation indicates the absence of distortion, the process proceeds to step S160 (step S152).

In step S154, the calculation section 122 calculates an edge value based on the reference pixel tap formed by the tap formation section 121 (step S154). The threshold comparison section 123 compares the calculated value with the edge determination threshold β (step S156). If the edge value is not less than the threshold β (determination condition B is true), the process proceeds to step S158. If the edge value is not less than the threshold β, the process proceeds to step S160.

At step S158, the filter determination section 125 determines to apply the deblocking filter to the determined boundary (step S158). At step S140, the filter determination section 125 determines not to apply the deblocking filter to the determined boundary (step S160).

<4. Second embodiment>

[4-1. Deblocking filter structure example]

An exemplary structure of the deblocking filter 24 according to the second embodiment is described below.

(1) Dependencies between new processes

According to this embodiment, the deblocking filter 24 determines whether filtering is necessary for each block's vertical boundary without waiting for the deblocking filter to be applied to other blocks in the macroblock to which the block belongs. The deblocking filter 24 also determines whether filtering is necessary for each block's horizontal boundary without waiting for the deblocking filter to be applied to other blocks in the macroblock to which the block belongs. This can reduce dependencies within the macroblock.

As described above, by reducing the dependency of processing, it is possible to parallelize the filtering necessity determination processing for vertical boundaries and horizontal boundaries in a macroblock.

22 is an explanatory diagram illustrating a first example of a processing sequence available on the deblocking filter 24. This example also assumes that the deblocking filter is fed with an image of size 32×32 pixels. The input image includes four macroblocks MB0-MB3, each of size 16×16 pixels.

In Figure 22, each dotted box represents a process to be performed in parallel. While the example in Figure 10 requires 16 processing steps for the processing sequence, the example in Figure 22 combines the same number of processes into 12 steps. The first step performs the filtering required decision processes J _V0,0 to J _V0,3 and J _H0,0 to J _H0,3 for the four vertical and four horizontal boundaries of macroblock MB0 in parallel. The second step performs the filtering required decision processes F _V0,0 to F _V0,3 for the four vertical boundaries of macroblock MB0 in parallel. The third step performs the filtering required decision processes J _V1,0 to J _V1,3 and J _H1,0 to J _H1,3 for the four vertical and four horizontal boundaries of macroblock MB1 in parallel. The fourth step performs the filtering required decision processes F _V1,0 to F _V1,3 for the four vertical boundaries of macroblock MB1 in parallel. The fifth step performs filtering processes F _H0,0 to F _H0,3 on the four horizontal boundaries of macroblock MB0 in parallel. The sixth step performs filtering necessity determination processes J _V2,0 to J _V2,3 and J _H2,0 to J _H2,3 on the four vertical and four horizontal boundaries of macroblock MB2 in parallel. The seventh step performs filtering processes F V2,0 to F V2,3 on the four vertical boundaries of macroblock MB2 in parallel. The eighth step performs filtering processes F _H1,0 to F _H1,3 on the four horizontal boundaries of macroblock MB1 in parallel. The ninth step performs filtering necessity determination processes _{J V3,0 to} J _V3,3 and J _H3,0 to J _H3,3 on the four vertical and four horizontal boundaries of macroblock MB3 in parallel. The tenth step performs filtering processes _{F V3,0} to F _V3,3 on the four vertical boundaries of macroblock MB3 in parallel. In the eleventh step, filtering processes F _H2,0 to F _H2,3 are performed in parallel on the four horizontal boundaries of macroblock MB2. In the twelfth step, filtering processes F _H3,0 to F _H3,3 are performed in parallel on the four horizontal boundaries of macroblock MB3. In this case, the deblocking filter 24 can process the entire input image using fewer processing steps than in the conventional art.

(2) Detailed structure of the deblocking filter

23 is a block diagram illustrating a detailed structure of the deblocking filter 24 that performs the above-mentioned parallel processing according to the second embodiment. Referring to FIG23 , the deblocking filter 24 includes a vertical determination unit 210, a horizontal determination unit 214, a horizontal filtering unit 130, a vertical filtering unit 140, and a parallelization control section 150.

(2-1) Vertical determination component

The vertical determination section 210 includes a plurality of vertical boundary determination sections 212-1 to 212-n. Each vertical boundary determination section 212 determines whether to apply a deblocking filter to the vertical boundary of each block without waiting for the deblocking filter to be applied to other blocks in the macroblock to which the block belongs. Each vertical boundary determination section 212 supplies information indicating the determination result regarding each vertical boundary, such as binary information indicating a determination result of forcing the application of the deblocking filter, to the horizontal filtering section 130.

(2-2) Level determination component

The horizontal determination section 214 includes a plurality of horizontal boundary determination sections 216-1 to 216-n. Each horizontal boundary determination section 216 determines whether to apply a deblocking filter to the horizontal boundary of each block without waiting for the deblocking filter to be applied to other blocks in the macroblock to which the block belongs. Each horizontal boundary determination section 216 supplies information indicating the determination result for each horizontal boundary to the vertical filtering section 140.

Furthermore, according to this embodiment, each vertical boundary determination section 212 and each horizontal boundary determination section 216 can determine whether filtering is required for each boundary by referring to pixels at various positions, similar to conventional techniques. In fact, each vertical boundary determination section 212 and each horizontal boundary determination section 216 can determine whether filtering is required for each boundary using the technique described in "3-2. Modification of Determination Conditions."

[3-2. Processing Flow]

FIG24 is a flowchart illustrating the processing flow of the deblocking filter 24 according to the second embodiment. Referring to FIG24 , vertical boundary determination sections 212-1 through 212-n concurrently determine whether filtering is required for all vertical boundaries within the macroblock of interest within the input image (step S202). Horizontal boundary determination sections 214-1 through 214-n concurrently determine whether filtering is required for all horizontal boundaries within the macroblock of interest (step S204). Steps S202 and S204 can also be performed in parallel.

The horizontal filtering sections 132 - 1 to 132 - n apply the deblocking filter in parallel to the vertical boundary in the macroblock of interest to which the deblocking filter is determined to be applied in step S202 (step S210 ).

The processing of step S220 is for the macroblock of interest in the most recent cycle. For the first macroblock of interest, the processing of step S220 can be skipped. Vertical filtering components 142-1 to 142-n apply the deblocking filter in parallel to step S204 in the most recent cycle, determining the horizontal boundary to which the deblocking filter should be applied (step S220).

If there is an unprocessed macroblock of interest in the input image, the processing of steps S202 to S220 is repeated for a new macroblock of interest (step S230).

If no unprocessed macroblock of interest remains, the vertical filtering sections 142 - 1 ˜ 142 - n apply deblocking filters in parallel to the macroblock of interest in the final cycle, and determine the horizontal boundary to which the deblocking filter needs to be applied (step S240 ).

The above-mentioned processing flow is also only an example. The parallelism and order of the processing can be changed. In addition, the parallelization control part 150 can control the parallelism and order of the processing.

[4-3. Example of processing per LCU]

As described above, the technology according to the various embodiments described in this specification can be provided in the form of processing based on HEVC coding units (CUs). According to HEVC, the coding unit with the maximum size is called the largest coding unit (LCU), for example, it can be selected as 64×64 pixels. The smallest optional CU size is 8×8 pixels. Typically, starting from the LCU on the upper left side of the image (or slice), the image is encoded and decoded corresponding to each LCU in a raster scan order. The following describes an example of processing corresponding to the LCU in the deblocking filter 24.

Fig. 25 is an explanatory diagram illustrating a processing sequence for each LCU according to the above-described second embodiment. This example assumes that the LCU size is 16×16 pixels and the CU size is 8×8 pixels.

Referring to FIG. 25 , the first stage in the upper left side of the figure indicates that filtering is completed for the (n-1)th LCU. The target pixels for filtering at vertical boundaries are shaded pixels. The target pixels for filtering at horizontal boundaries are solid pixels.

In Figure 25, the second processing on the upper right side and the third processing on the lower left side target the n-th LCU. Before the second processing, filtering needs to be determined for all vertical boundaries and horizontal boundaries belonging to the n-th LCU in parallel. That is, filtering needs to be determined for the boundaries of the CU belonging to the n-th LCU without waiting for the deblocking filter to be applied to other CUs in the n-th LCU. The second processing performs filtering in parallel on the vertical boundaries that belong to the n-th LCU and are determined to require the application of the deblocking filter. The second processing performs filtering in parallel on the horizontal boundaries that belong to the n-th LCU and are determined to require the application of the deblocking filter.

The fourth stage of processing on the lower right side of Figure 25 targets the (n+1)th LCU. In the fourth stage, after the boundaries of all CUs belonging to the (n+1)th LCU are subjected to filtering necessity determination processing in parallel, the vertical boundaries determined to require application of the deblocking filter are filtered in parallel.

Although this example assumes that the LCU size is 16×16 pixels, it can be set to 32×32 pixels or 64×64 pixels. The effect of shortening the processing time according to parallelization is further improved because increasing the size of the LCU to be selected also increases the number of vertical boundaries and horizontal boundaries belonging to one LCU.

FIG26 is a flowchart illustrating the processing flow of the deblocking filter 24 for each LCU.

26 , the vertical boundary determination sections 212-1 to 212-n concurrently determine whether filtering is required for all vertical boundaries within the LCU of interest in the input image (step S252). The horizontal boundary determination sections 216-1 to 216-n concurrently determine whether filtering is required for all horizontal boundaries within the LCU of interest (step S254). Steps S252 and S254 may also be performed in parallel.

The horizontal filtering sections 132 - 1 to 132 - n apply deblocking filters in parallel to the vertical boundaries in the LCU of interest determined to require application of the deblocking filters in step S252 (step S260 ).

The vertical filtering sections 142 - 1 to 142 - n apply the deblocking filter in parallel to the horizontal boundary in the LCU of interest determined to need application of the deblocking filter in step S254 (step S270 ).

If there is an unprocessed LCU left in the input image, the process in steps S252 to S270 is repeated for a new LCU of interest (step S280). If there is no unprocessed LCU, the process is terminated.

<5. Third embodiment>

[5-1. Overview]

The first and second embodiments modify the existing processing sequence of the deblocking filter to increase processing parallelism. Specifically, the first embodiment reduces processing dependencies by expanding the range of pixel values in the input image provided to the deblocking filter when determining the need for filtering. The third embodiment, described below, further enhances this inventive concept. The third embodiment further parallelizes processing by filtering the input pixel values provided to the deblocking filter during filtering of vertical and horizontal boundaries.

FIG27 is an explanatory diagram illustrating an overview of this embodiment. The lower left side of FIG27 shows the shape of an input pixel (also referred to as a reconstructed pixel) before being processed by the deblocking filter. This embodiment allows reference to the pixel values input to the deblocking filter for both the filtering necessity determination process for vertical and horizontal boundaries, as well as the filtering process for vertical boundaries and the filtering process for horizontal boundaries. This allows for resolution of the dependency between the two filtering necessity determination processes, as well as the dependency between the two filtering processes.

The filtering process for vertical boundaries and the filtering process for horizontal boundaries may update the values of repeated pixels. The solid pixels in FIG27 illustrate the locations of pixels that may be repeated. For pixels that are repeatedly updated by two filters operating in parallel, the deblocking filter according to this embodiment calculates a single output pixel value based on the two filter outputs.

[5-2. Deblocking filter structure example]

28 is a block diagram illustrating a detailed structure of the deblocking filter 24 according to the third embodiment. Referring to FIG28 , the deblocking filter 24 includes a line memory 308, a decision section 310, a horizontal filtering section 330, a vertical filtering section 340, a parallelization control section 150, and a calculation section 360.

Line memory 308 stores pixel values of the input image supplied to deblocking filter 24. Filtering processing in horizontal filtering section 330 and vertical filtering section 340 does not update the pixel values stored in line memory 308. Filtering performed by the following sections in decision section 310 requires decision processing to refer to the pixel values stored in line memory 308. The device includes another memory for a purpose different from the processing of deblocking filter 24. This memory can be reused (shared) as line memory 308.

The determination unit 310 includes vertical boundary determination sections 312-1 to 312-n and horizontal boundary determination sections 314-1 to 314-n. The vertical boundary determination section 312 and the horizontal boundary determination section 314 are supplied with the pixel values of the image input to the deblocking filter 24 stored in the line memory 308 and determination information for determining whether filtering is required.

The vertical boundary decision section 312 decides whether to apply the deblocking filter to each vertical boundary using the pixel values input to the deblocking filter 24. The vertical boundary decision section 312 outputs information indicating the decision result on each vertical boundary to the horizontal filtering section 330.

The horizontal boundary determination section 314 also determines whether to apply the deblocking filter to each horizontal boundary using the pixel values input to the deblocking filter 24. The horizontal boundary determination section 314 performs determination processing in parallel with the determination processing performed by the vertical boundary determination section 312. The horizontal boundary determination section 314 outputs information indicating the determination result regarding each horizontal boundary to the vertical filtering section 340.

Furthermore, according to this embodiment, each vertical boundary determination section 312 and each horizontal boundary determination section 314 may determine whether filtering is required for each boundary by referring to pixels at various positions, similar to the conventional technique. Alternatively, each vertical boundary determination section 312 and each horizontal boundary determination section 314 may determine whether filtering is required for each boundary using the technique described in "3-2. Modification of Determination Conditions."

Horizontal filtering section 330 includes horizontal filtering sections 332-1 to 332-n. Horizontal filtering section 332 is supplied with the input image value from line memory 308 and the decision result regarding each vertical boundary from decision section 310.

If the decision result from the vertical boundary decision section 312 indicates that a filter needs to be applied, the horizontal filtering component 332 applies a deblocking filter for the vertical boundary to the left and right pixels around the corresponding vertical boundary. The horizontal filtering component 332 outputs the filtered pixel value for the filtered pixel or the input pixel value for other pixels to the calculation section 360.

The vertical filtering section 340 includes vertical filtering sections 342-1 to 342-n. The vertical filtering section 342 is supplied with the input pixel value from the line memory 308 and the decision result from the decision section 310 regarding each horizontal boundary.

If the determination result from horizontal boundary determination section 314 indicates that a filter is required, vertical filtering section 342 applies a deblocking filter for the horizontal boundary to the upper and lower pixels surrounding the corresponding horizontal boundary. The filtering process of vertical filtering sections 342-1 through 342-n is performed in parallel with the filtering process of horizontal filtering sections 332-1 through 332-n. Vertical filtering section 342 outputs the filtered pixel value for the filtered pixel or the input pixel value for the other pixels to calculation section 360.

The calculation section 360 is supplied in parallel with the output pixel value from the horizontal filtering section 330 and the output pixel value from the vertical filtering section 340. In addition, the calculation section 360 is supplied with the determination results from the vertical boundary determination section 312 and the horizontal boundary determination section 314. In accordance with the determination results, the calculation section 360 calculates the output pixel values of the pixels filtered by the horizontal filtering section 330 and the vertical filtering section 340 based on the filter outputs from the horizontal filtering section 330 and the vertical filtering section 340.

According to this embodiment, for example, for pixels that are repeatedly filtered, the calculation section 360 calculates the average of the two filter outputs. The calculation section 360 may calculate a simple average of the two filter outputs. Alternatively, the calculation section 360 may calculate a weighted average of the two filter outputs. For example, the calculation section 360 may determine the weight for the weighted average of each pixel based on its distance from the vertical boundary and the horizontal boundary.

FIG29 is an explanatory diagram illustrating the determination of weights for the weighted average calculated by the calculation section 360. FIG29 shows the pixel of interest _PZ corresponding to one of the repetition positions illustrated in FIG27 in black. Three pixels corresponding to the distance DH exist between the pixel of interest _PZ and the nearest vertical boundary _VZ . Two pixels corresponding to the distance _DH exist between the pixel _of interest _PZ and the nearest horizontal boundary _HZ . Distance _DH is smaller than distance DH. In this case, the calculation section 360 may set the weight of the output of the deblocking filter applied to the horizontal boundary _HZ to be greater than the weight of the output of the deblocking filter applied to the vertical boundary _VZ . The example in _FIG29 assumes that the ratio of the filter output _Vout for the vertical boundary _VZ to the filter output _Hout for the horizontal boundary _HZ is 2:3.

As can be seen from Figure 29, calculating the weighted average of the outputs of the two filters can provide each pixel of interest with an output pixel value similar to the case of applying a two-dimensional filter with filter taps along the horizontal direction and filter taps along the vertical direction. Parallelizing the filtering process for vertical and horizontal boundaries can also appropriately reduce the block distortion that occurs at the vertical and horizontal boundaries. As another embodiment, the deblocking filter 24 may include a two-dimensional filter that simultaneously calculates horizontal filtering, vertical filtering, and a weighted average. However, in this case, the installation is extremely complicated because the filter coefficients need to be changed differently depending on the pixel. On the other hand, the third embodiment performs two one-dimensional filters in parallel and then calculates the weighted average. This can easily provide processing that is roughly the same as that of a two-dimensional filter while ensuring the functionality of the existing deblocking filter.

FIG30 is an explanatory diagram illustrating an example of the weights of the weighted average calculated based on the example in FIG29 . FIG30 shows 36 pixels (6×6) around the intersection of the vertical and horizontal boundaries. These pixels correspond to the repeated positions described above. For pixels located at equal distances from the vertical and horizontal boundaries, the ratio of the weight of the filter output V _out to the weight of the filter output H _out is 1:1 (2:2 or 3:3). For pixels closer to the vertical boundary, the weight of the filter output V _out is greater than the weight of the filter output H _out . For example, the weight ratio of pixel _P1 is V _out : H _out = 3:1. For pixels closer to the horizontal boundary, the weight of the filter output V _out is less than the weight of the filter output H _out . For example, the weight ratio of pixel _P2 is V _out : H _out = 1:3.

By changing the weight of the weighted average according to the distance between each pixel and the boundary, block distortion can be suppressed more effectively, thereby improving image quality.

The weights described above are merely examples. For example, instead of or in addition to the distance between each pixel and the boundary, the calculation section 360 may determine the weight of the weighted average of the pixels according to the edge strength of the vertical and horizontal boundaries corresponding to each pixel. The edge strength may be represented by a parameter such as the edge value calculated from the calculation section 122 as shown in Figure 13. In this case, the weight of the filter output for boundaries with stronger edges may be set to be greater than the weight of the filter output for boundaries with weaker edges. Changing the weight of the weighted average according to the edge strength can adaptively improve the effect of the deblocking filter on boundaries that significantly cause block distortion.

For pixels filtered by one of the horizontal filtering section 330 and the vertical filtering section 340, the calculation section 360 selects the output from the block actually filtered. For pixels not filtered by the horizontal filtering section 330 or the vertical filtering section 340, the calculation section 360 directly outputs the input pixel value to be output to the deblocking filter 24. The table in FIG31 lists the output pixel values from the calculation section 360 according to the results of the determination of whether filtering is required.

[5-3. Processing sequence example]

Two examples of processing sequences that can be used by the deblocking filter 24 according to this embodiment are described below. This example also assumes that the deblocking filter is fed with an image of size 32×32 pixels. The input image includes four macroblocks MB0-MB3, each of size 16×16 pixels.

(1) First example

For comparison purposes, FIG32 illustrates a processing sequence when a dependency still exists between filtering for vertical boundaries and filtering for horizontal boundaries. In FIG32 , the first step performs filtering necessity determination processes (J _V0,0 to J _V3,3 and J _H0,0 to J _H3,3 ) on all vertical boundaries and all horizontal boundaries of all four macroblocks MB0-MB3 in parallel. The second step performs filtering processes (F _V0,0 to F _V3,3) on the 16 vertical boundaries of the four macroblocks MB0-MB3. The third step performs filtering processes (F _H0,0 to F _H3,3) on the 16 horizontal boundaries of the four macroblocks MB0-MB3. The fourth step stores the pixel values after filtering for horizontal boundaries in the memory for the output from the deblocking filter 24.

FIG33 illustrates a first example of a processing sequence provided by this embodiment. In FIG33 , the first step performs filtering necessity determination processing (J _V0,0 to J _V3,3 and J _H0,0 to J _H3,3 ) on all vertical and horizontal boundaries of the four macroblocks MB0-MB3 in parallel. The second step performs filtering processing (F _V0,0 to F _V3,3 and F _H0,0 to F _H3,3 ) on all vertical and horizontal boundaries of the four macroblocks MB0-MB3 in parallel. In practice, the second step only filters boundaries determined to require filtering. The third step stores pixel values in the memory used for the output from the deblocking filter 24. For pixels filtered by the horizontal filtering component 330 and the vertical filtering component 340, a weighted average of the two filter outputs is calculated as the output pixel value.

(2) Second example

While the first example maximizes parallelism, the deblocking filter 24 according to the second example can also perform processing on a per-macroblock basis.

For comparison purposes, FIG34 illustrates the processing sequence for each macroblock when a dependency still exists between the filtering processes for vertical boundaries and the filtering processes for horizontal boundaries. The processing sequence in FIG34 is substantially the same as the processing sequence in FIG22 according to the first embodiment. FIG36 explicitly illustrates the four processing steps (the 6th, 10th, 14th, and 16th) that are omitted from FIG22 for simplicity, and that store pixel values in memory for output. The 16 processing steps, including these four, constitute the processing in FIG34.

FIG35 illustrates a second example of a processing sequence provided by this embodiment. In FIG35 , the first step performs filtering necessity determination processing (J _V0,0 to J _V0,3 and J _H0,0 to J _H0,3 ) on the four vertical and four horizontal boundaries of macroblock MB0 in parallel. The second step performs filtering processing (F _V0,0 to F _V0,3 and F _H0,0 to F _H0,3 ) on the four vertical and four horizontal boundaries of macroblock MB0 in parallel. The third step stores the pixel values of macroblock MB0 in a memory for the output from deblocking filter 24. For pixels repeatedly filtered by two filters, a weighted average of the two filter outputs is calculated as the output pixel value. Steps 4 through 6 process macroblock MB1 in a similar manner. Steps 7 through 9 process macroblock MB2 in a similar manner. Steps 10 through 12 process macroblock MB3 in a similar manner. The processing in FIG35 includes 12 fewer processing steps than the processing in FIG34 .

The third embodiment eliminates the dependency between the filtering processes for vertical boundaries and horizontal boundaries. Deblocking filter 24 can be processed using fewer processing steps than in the first and second embodiments. One advantage of allowing filtering to refer solely to the pixel inputs to the deblocking filter is that any arbitrary configuration of filter taps does not result in any dependency between the filtering processes for vertical boundaries and horizontal boundaries. By utilizing more pixels to form filter taps than in the prior art, the third embodiment can improve image quality. For example, the prior art uses filter taps of three pixels on each side of each boundary, as described above with reference to FIG7 . Even when filter taps of five or more pixels are used at each boundary, this embodiment does not result in any dependencies between the processes. By further reducing the block size, which serves as the processing unit for the deblocking filter, no dependencies between the processes arise.

In addition, in the third embodiment as well as the first and second embodiments, the parallelization control portion 150 can control the degree of parallelism and the order of processing in the deblocking filter 24 .

[5-4. Processing Flow]

Fig. 36 is a flowchart illustrating an example of the processing flow of the deblocking filter according to the third embodiment. Fig. 37 is a flowchart illustrating the flow of the pixel value calculation process shown in Fig. 36 .

36 , vertical boundary determination units 312-1 to 312-n concurrently determine whether filtering is required for all vertical boundaries within the input image or macroblock (step S302). Horizontal boundary determination units 314-1 to 314-n concurrently determine whether filtering is required for all horizontal boundaries within the input image or macroblock (step S304). Steps S302 and S304 can also be performed in parallel.

The horizontal filtering components 332-1 to 332-n apply the deblocking filters in parallel to the process performed at step S302, and determine all vertical boundaries to which the deblocking filters are to be applied (step S306). The vertical filtering components 342-1 to 342-n apply the deblocking filters in parallel to the process performed at step S304, and determine all horizontal boundaries to which the deblocking filters are to be applied (step S308). Steps S306 and S308 may also be performed in parallel.

The calculation section 360 then performs the pixel value calculation processing (step S310) as shown in Fig. 37. Referring to Fig. 37, the processing from step S314 to step S326 is then looped for each pixel to be processed (step S312).

In step S314, the calculation section 360 determines whether the two filters for the vertical boundary and the horizontal boundary have filtered the pixel of interest (step S314). If the two filters have filtered the pixel of interest, the process proceeds to step S322. If the two filters have not filtered the pixel of interest, the process proceeds to step S316.

In step S316, the calculation section 360 determines whether one of the two filters for the vertical boundary and the horizontal boundary has filtered the pixel of interest (step S316). If one of the two filters has filtered the pixel of interest, the process proceeds to step S320. If neither of the two filters has filtered the pixel of interest, the process proceeds to step S318.

At step S318, the calculation section 360 obtains the input pixel value to the deblocking filter 24 (step S318). At step S320, the calculation section 360 obtains the filter output from the filter that actually performs filtering on the pixel of interest (step S320).

In step S322, the calculation section 360 determines a weight value for calculating a weighted average of the filter outputs from the two filters related to the pixel of interest, based on the distance from the pixel of interest to the vertical and horizontal boundaries, or the edge strength of the vertical and horizontal boundaries corresponding to the pixel of interest (step S322). The calculation section 360 calculates the weighted average of the filter outputs from the two filters using the determined weights (step S324).

When the pixel value is obtained in step S318 or S320, or when the pixel value is calculated in step S324, the calculation section 360 stores the pixel value of the pixel of interest in the memory (step S326). When all pixels to be processed have been processed, the processing sequence shown in Figures 36 and 37 is terminated.

<6. Application of various codecs>

The technology according to the present disclosure is applicable to various codecs related to image encoding and decoding. An example of applying the technology according to the present disclosure to a multi-view codec and a scalable codec is described below.

[6-1. Multi-view codec]

A multi-view codec is an image coding scheme for encoding and decoding multi-viewpoint videos. FIG38 is an explanatory diagram illustrating a multi-view codec. FIG38 illustrates a frame sequence of three views captured at three viewpoints. Each view has a view ID (view_id). One of the views is designated as a base view. Views other than the base view are referred to as non-base views. The example in FIG38 shows a base view having a view ID of "0", and two non-base views having view IDs of "1" or "2". Encoding multi-view image data can compress the data size of the encoded stream as a whole by encoding each frame of the non-base view based on encoding information about each frame of the base view.

During the encoding and decoding processes according to the multi-view codec described above, a deblocking filter may be applied to each view. The deblocking filter applied to each view may be performed in parallel with horizontal filtering and vertical filtering, using the processing of multiple CUs comprising each view as a unit, according to the techniques disclosed herein. The processing unit may represent several CUs, LCUs, or images. Parameters for controlling parallel processing may be set for each view (such as those described in the second paragraph of the preceding "(4) Parallelization Control Section"). Parameters set for the base view may be reused for non-base views.

Horizontal filtering and vertical filtering can be performed in parallel within multiple views. The multiple views can share parameters for controlling parallel processing (such as the parameters described in the second paragraph of the "(4) Parallelization Control Section" above). It is advantageous to additionally specify a flag indicating whether the multiple views share the parameters.

Fig. 39 is an explanatory diagram illustrating an image encoding process applied to the above-described multi-view codec. Fig. 39 shows an example of the structure of a multi-view encoding device 710. The multi-view encoding device 710 includes a first encoding unit 720, a second encoding unit 730, and a multiplexing section 740.

The first encoding component 720 encodes the base view image to generate an encoded stream of the base view. The second encoding component 730 encodes the non-base view images to generate encoded streams of the non-base views. The multiplexing section 740 multiplexes the encoded stream of the base view generated by the first encoding component 720 with one or more encoded streams of the non-base views generated by the second encoding component 730 to generate a multiplexed stream of multiple views.

The first encoding section 720 and the second encoding section 730 illustrated in FIG39 are configured similarly to the image encoding device 10 according to the above-described embodiment. Applying a deblocking filter to a view enables parallel processing of horizontal and vertical filtering in units of processing including multiple CUs. Parameters controlling these processes can be inserted into the header area of the coded stream for each view, or into the common header area of the multiplexed stream.

Fig. 40 is an explanatory diagram illustrating an image decoding process applied to the multi-view codec described above. Fig. 40 shows an example of the structure of a multi-view decoding device 760. The multi-view decoding device 760 includes a demultiplexing section 770, a first decoding section 780, and a second decoding section 790.

The demultiplexing section 770 demultiplexes the multi-view multiplexed stream into a base view coded stream and one or more non-base view coded streams. The first decoding section 780 decodes the base view image from the base view coded stream. The second decoding section 790 decodes the non-base view images from the non-base view coded streams.

The first decoding section 780 and the second decoding section 790 illustrated in FIG40 are configured similarly to the image decoding device 60 according to the above-described embodiment. Applying a deblocking filter to a view enables parallelization of horizontal filtering and parallelization of vertical filtering in units of processing involving multiple CUs. Parameters controlling these processes can be obtained from the header area of the coded stream for each view, or from the common header area of the multiplexed stream.

[6-2. Scalable Codec]

Scalable codec is an image coding scheme that provides hierarchical coding. Figure 41 is an explanatory diagram illustrating the scalable codec. Figure 41 illustrates a frame sequence of three layers with different spatial resolutions, temporal resolutions, or image qualities. Each layer has a layer ID (layer_id). These layers include a base layer with the lowest resolution (or image quality). Each layer other than the base layer is called an enhancement layer. The example in Figure 41 shows a base layer with a layer ID of "0", and two enhancement layers with layer IDs of "1" or "2". Encoding multi-layer image data can compress the data size of the encoded stream as a whole by encoding each frame of the enhancement layer based on the encoding information about each frame of the base layer.

During the encoding and decoding processes according to the scalable codec described above, a deblocking filter may be applied to each layer. The deblocking filter applied to each layer may be performed in parallel, using the processing of multiple CUs per view as a unit, according to the techniques disclosed herein. The processing unit may represent several CUs, LCUs, or images. Parameters for controlling parallel processing (such as those described in the second paragraph of the "(4) Parallelization Control Section") may be set for each layer. Parameters set for base layer views may be reused for enhancement layers.

Horizontal filtering and vertical filtering can be performed in parallel within multiple layers. The multiple layers can share parameters for controlling parallel processing (such as the parameters described in the second paragraph of the previous "(4) Parallelization Control Section"). It is advantageous to additionally specify a flag indicating whether the multiple layers share the parameters.

Fig. 42 is an explanatory diagram illustrating the image encoding process applied to the above-mentioned scalable codec. Fig. 42 shows an example of the structure of a scalable encoding device 810. The scalable encoding device 810 includes a first encoding unit 820, a second encoding unit 830, and a multiplexing section 840.

The first encoding component 820 encodes the base layer image to generate a base layer coded stream. The second encoding component 830 encodes the enhancement layer image to generate a coded stream for the enhancement layer. The multiplexing section 840 multiplexes the base layer coded stream generated by the first encoding component 820 with one or more coded streams of the enhancement layer generated by the second encoding component 830, thereby generating a multi-layer multiplexed stream.

The first encoding section 820 and the second encoding section 830 illustrated in FIG42 are configured similarly to the image encoding device 10 of the above-described embodiment. Applying a deblocking filter to each layer enables parallelization of horizontal filtering and parallelization of vertical filtering in units of processing involving multiple CUs. Parameters controlling these processes can be inserted into the header area of the coded stream for each layer, or into the common header area of the multiplexed stream.

Fig. 43 is an explanatory diagram illustrating the image decoding process applied to the above-mentioned scalable codec. Fig. 43 shows an example of the structure of a scalable decoding device 860. The scalable decoding device 860 includes a demultiplexing section 870, a first decoding section 880, and a second decoding section 890.

The demultiplexing section 870 demultiplexes the multi-layer multiplexed stream into a base layer coded stream and one or more enhancement layer coded streams. The first decoding section 880 decodes the base layer image from the base layer coded stream. The second decoding section 890 decodes the enhancement layer image from the enhancement layer coded stream.

The first decoding section 880 and the second decoding section 890 illustrated in FIG43 are configured similarly to the image decoding device 60 according to the above-described embodiment. Applying a deblocking filter to each layer enables parallelization of horizontal filtering and vertical filtering in units of processing involving multiple CUs. Parameters controlling these processes can be obtained from the header area of the coded stream for each layer, or from the common header area of the multiplexed stream.

<7. Example Application>

The image encoding device 10 and the image decoding device 60 according to the above-described embodiment are applicable to various electronic devices, such as transmitters and receivers used for satellite broadcasting, cable broadcasting such as cable television, distribution over the Internet, distribution to terminals via cellular communications, etc.; recording devices that record images on media such as optical disks, magnetic disks, or flash memory; and reproduction devices that reproduce images from such storage media. Four exemplary applications are described below.

[7-1. First Example Application]

Figure 44 is a block diagram showing an example of a schematic structure of a television set to which the above-described embodiment is applicable. A television set 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing section 905, a display section 906, an audio signal processing section 907, a speaker 908, an external interface 909, a control section 910, a user interface 911, and a bus 912.

The tuner 902 extracts a signal of a desired channel from the broadcast signal received via the antenna 901 and demodulates the extracted signal. The tuner 902 then outputs the encoded bit stream obtained by demodulation to the demultiplexer 903. That is, the tuner 902 serves as a transmission device of the television 900 that receives the encoded stream in which the image is encoded.

The demultiplexer 903 separates the video stream and audio stream of the program to be viewed from the coded bit stream, and outputs each of the separated streams to the decoder 904. In addition, the demultiplexer 903 extracts auxiliary data such as an EPG (Electronic Program Guide) from the coded bit stream, and supplies the extracted data to the control section 910. In addition, if the coded bit stream is scrambled, the demultiplexer 903 can perform descrambling.

The decoder 904 decodes the video stream and the audio stream input from the demultiplexer 903. Then, the decoder 904 outputs the video data generated by the decoding process to the video signal processing section 905. In addition, the decoder 904 outputs the audio data generated by the decoding process to the audio signal processing section 907.

The video signal processing section 905 reproduces the video data input from the decoder 904 and causes the display section 906 to display the video. The video signal processing section 905 can also cause the display section 906 to display application screens supplied via the network. Furthermore, the video signal processing section 905 can perform additional processing on the video data, such as noise removal, according to settings. Furthermore, the video signal processing section 905 can generate GUI (Graphical User Interface) images, such as menus, buttons, and cursors, and overlay these generated images on the output image.

The display portion 906 is driven by the driving signal supplied from the video signal processing portion 905 and displays a video or image on a video screen of a display device (eg, a liquid crystal display, a plasma display, an OLED, etc.).

The audio signal processing section 907 performs reproduction processing such as D/A conversion and amplification on the audio data input from the decoder 904, and outputs audio from the speaker 908. In addition, the audio signal processing section 907 may perform additional processing such as noise removal on the audio data.

The external interface 909 is an interface between the television 900 and an external device or network. For example, the video stream and audio stream received by the external interface 909 can be decoded by the decoder 904. That is, the external interface 909 also serves as a transmission device of the television 900 for receiving a coded stream in which an image is coded.

The control section 910 includes a processor such as a CPU (Central Processing Unit) and a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The memory stores programs to be executed by the CPU, program data, EPG data, data obtained via a network, and the like. When the television set 900 is started, the CPU reads and executes the program stored in the CPU. The CPU controls the operation of the television set 900 according to an operation signal input from the user interface 911 by executing the program.

The user interface 911 is connected to the control unit 910. The user interface 911 includes buttons and switches used by the user to operate the television set 900, and a remote control signal receiving unit. The user interface 911 detects user operations using these components, generates operation signals, and outputs the generated operation signals to the control unit 910.

The bus 912 interconnects the tuner 902 , the demultiplexer 903 , the decoder 904 , the video signal processing section 905 , the audio signal processing section 907 , the external interface 909 , and the control section 910 .

In the television set 900 configured in this manner, the decoder 904 has the function of the image decoding device 60 according to the above embodiment. Therefore, also in the case of image decoding in the television set 900, the parallelism of the deblocking filter processing can be enhanced, thereby ensuring high-speed processing.

[7-2. Second Example Application]

45 is a block diagram showing an example of a schematic configuration of a mobile phone using the above-described embodiment. The mobile phone 920 includes an antenna 921, a communication section 922, an audio codec 923, a speaker 924, a microphone 925, a camera section 926, an image processing section 927, a demultiplexing section 928, a recording/reproducing section 929, a display section 930, a control section 931, an operation section 932, and a bus 933.

An antenna 921 is connected to the communication section 922. A speaker 924 and a microphone 925 are connected to the audio codec 923. An operation section 932 is connected to the control section 931. A bus 933 interconnects the communication section 922, the audio codec 923, the camera section 926, the image processing section 927, the demultiplexing section 928, the recording/reproducing section 929, the display section 930, and the control section 931.

The mobile phone 920 performs operations such as transmission/reception of audio signals, transmission/reception of e-mail or image data, video recording, and data recording in an audio communication mode, a data communication mode, a camera mode, and a video phone mode.

In audio communication mode, the analog audio signal generated by the microphone 925 is provided to the audio codec 923. The audio codec 923 converts the analog audio signal into audio data, performs A/D conversion and compression on the converted audio data. The audio codec 923 then outputs the compressed audio data to the communication section 922. The communication section 922 encodes and modulates the audio data to generate a transmission signal. The communication section 922 then transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication section 922 amplifies the wireless signal received via the antenna 921 and converts the frequency of the wireless signal to obtain a received signal. The communication section 922 then demodulates and decodes the received signal to generate audio data, and outputs the generated audio data to the audio codec 923. The audio codec 923 expands and D/A converts the audio data to generate an analog audio signal. The audio codec 923 then provides the generated audio signal to the speaker 924, so that the audio is output.

In data communication mode, the control section 931 generates text data constituting an email in response to user operations via the operating section 932. Furthermore, the control section 931 causes the text to be displayed on the display section 930. Furthermore, the control section 931 generates email data in response to a transmission instruction via the operating section 932 and outputs the generated email data to the communication section 922. The communication section 922 then encodes and modulates the email data to generate a transmission signal. The communication section 922 then transmits the generated transmission signal to a base station (not shown) via the antenna 921. Furthermore, the communication section 922 amplifies the wireless signal received via the antenna 921 and converts the frequency of the wireless signal to obtain a received signal. The communication section 922 then demodulates and decodes the received signal to recover the email content and outputs the recovered email data to the control section 931. The control section 931 causes the display section 930 to display the contents of the email and causes the email data to be stored in the storage medium of the recording/reproducing section 929.

The recording/reproducing section 929 includes any readable and writable storage medium, for example, a built-in storage medium such as a RAM, a flash memory, or an externally mounted storage medium such as a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, a USB memory, or a memory card.

Furthermore, in the camera mode, the camera section 926 captures an image of a subject, generates image data, and outputs the generated image data to the image processing section 927. The image processing section 927 encodes the image data input from the camera section 926 so that the encoded stream is stored in the storage medium of the recording/reproducing section 929.

Furthermore, in videophone mode, the demultiplexing section 928 multiplexes the video stream encoded by the image processing section 927 and the audio stream input from the audio codec 923, and outputs the multiplexed stream to the communication section 922. The communication section 922 encodes and modulates the streams to generate a transmission signal. The communication section 922 then transmits the generated transmission signal to a base station (not shown) via the antenna 921. Furthermore, the communication section 922 amplifies the wireless signal received via the antenna 921 and converts the frequency of the wireless signal to obtain a received signal. These transmission and received signals may include an encoded bit stream. The communication section 922 then demodulates and decodes the received signal to restore the streams, and outputs the restored streams to the demultiplexing section 928. The demultiplexing section 928 separates the video stream and the audio stream from the input stream, outputting the video stream to the image processing section 927 and the audio stream to the audio codec 923. The image processing section 927 decodes the video stream to generate video data. The video data is supplied to the display portion 930, so that a series of images are displayed using the display portion 930. The audio codec 923 expands and D/A converts the audio stream, generating an analog audio signal. Subsequently, the audio codec 923 supplies the generated audio signal to the speaker 924, so that the audio is output.

In the mobile phone 920 configured in this manner, the image processing section 927 has the functions of the image encoding device 10 and the image decoding device 60 according to the above-described embodiment. Therefore, even when encoding and decoding images in the mobile phone 920, the parallelism of the deblocking filter processing can be enhanced, thereby ensuring high-speed processing.

[7-3. Third Example Application]

FIG46 is a block diagram showing an example of a schematic structure of a recording/reproducing device using the above-described embodiment. The recording/reproducing device 940 encodes the audio and video data of a received broadcast program and records it on a recording medium. The recording/reproducing device 940 can also encode audio and video data obtained from another device and record it on a recording medium. Furthermore, the recording/reproducing device 940 reproduces the data recorded on the recording medium using a monitor or speakers in accordance with user instructions. In this case, the recording/reproducing device 940 decodes the audio and video data.

The recording/reproducing device 940 includes a tuner 941 , an external interface 942 , an encoder 943 , an HDD (Hard Disk Drive) 944 , an optical disc drive 945 , a selector 946 , a decoder 947 , an OSD (On Screen Display) 948 , a control section 949 , and a user interface 950 .

The tuner 941 extracts a signal of a desired channel from a broadcast signal received by an antenna (not shown), and demodulates the extracted signal. The tuner 941 then outputs a coded bit stream obtained by demodulation to the selector 946. That is, the tuner 941 serves as a transmission device of the recording/reproducing device 940.

The external interface 942 is an interface for connecting the recording/reproducing device 940 and an external device or a network. For example, the external interface 942 may be an IEEE 1394 interface, a network interface, a USB interface, a flash memory interface, or the like. For example, video data and audio data received using the external interface 942 are input to the encoder 943. In other words, the external interface 942 serves as a transmission device for the recording/reproducing device 940.

In the case where the video data and the audio data input from the external interface 942 are not encoded, the encoder 943 encodes the video data and the audio data, and then outputs the encoded bit stream to the selector 946 .

The HDD 944 records a coded bit stream, which is compressed content data of video or audio, various programs, and other data, in an internal hard disk. In addition, when reproducing video or audio, the HDD 944 reads this data from the hard disk.

The optical disc drive 945 records data on a mounted recording medium or reads data from a mounted recording medium. The recording medium mounted in the optical disc drive 945 may be a DVD disc (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD+, DVD+RW, etc.), a Blu-ray disc, or the like.

When recording video or audio, the selector 946 selects an encoded bit stream input from the tuner 941 or the encoder 943 and outputs the selected encoded bit stream to the HDD 944 or the optical disc drive 945. In addition, when reproducing video or audio, the selector 946 outputs the encoded bit stream input from the HDD 944 or the optical disc drive 945 to the decoder 947.

The decoder 947 decodes the encoded bit stream and generates video data and audio data. Then, the decoder 947 outputs the generated video data to the OSD 948. In addition, the decoder 904 outputs the generated audio data to an external speaker.

The OSD 948 reproduces the video data input from the decoder 947, thereby displaying the video. In addition, the OSD 948 may overlay an image of a GUI such as a menu, a button, a cursor, etc., on the displayed video.

The control section 949 includes a processor such as a CPU and a memory such as a RAM or ROM. The memory stores programs executed by the CPU, program data, and the like. When the recording/reproducing device 940 is activated, the CPU reads and executes the program stored in the memory. By executing the program, the CPU controls the operation of the recording/reproducing device 940 in accordance with operation signals input from the user interface 950.

The user interface 950 is connected to the control section 949. The user interface 950 includes buttons and switches used by the user to operate the recording/reproducing device 940, and a remote control signal receiving section. The user interface 950 detects the user's operation through these components, generates an operation signal, and outputs the generated operation signal to the control section 949.

In the recording/reproducing device 940 configured in this manner, the encoder 943 has the function of the image encoding device 10 according to the above-described embodiment. In addition, the decoder 947 has the function of the image decoding device 60 according to the above-described embodiment. Therefore, even when encoding and decoding images in the recording/reproducing device 940, the parallelism of the deblocking filter processing can be enhanced, thereby ensuring high-speed processing.

[7-4. Fourth Example Application]

Fig. 47 is a block diagram showing an example of a schematic configuration of an image pickup apparatus according to the above embodiment. The image pickup apparatus 960 captures an image of a subject, generates an image, encodes the image data, and records the image data in a recording medium.

The imaging apparatus 960 includes an optical component 961 , an imaging section 962 , a signal processing section 963 , an image processing section 964 , a display section 965 , an external interface 966 , a memory 967 , a media drive 968 , an OSD 969 , a control section 970 , a user interface 971 , and a bus 972 .

The optical component 961 is connected to the camera section 962. The camera section 962 is connected to the signal processing section 963. The display section 965 is connected to the image processing section 964. The user interface 971 is connected to the control section 970. The bus 972 interconnects the image processing section 964, the external interface 966, the memory 967, the media drive 968, the OSD 969, and the control section 970.

The optical components 961 include a focusing lens, an aperture stop mechanism, and other components. These components form an optical image of the subject on the imaging plane of the imaging section 962. The imaging section 962 includes an image sensor such as a CCD or CMOS sensor, and uses photoelectric conversion to convert the optical image formed on the imaging plane into an electrical image signal. The imaging section 962 then outputs the image signal to the signal processing section 963.

The signal processing section 963 performs various camera signal processing such as knee correction, gamma correction, color correction, etc. on the image signal input from the imaging section 962 . The signal processing section 963 outputs image data after the camera signal processing to the image processing section 964 .

The image processing section 964 encodes the image data input from the signal processing section 963 to generate encoded data. The image processing section 964 then outputs the generated encoded data to the external interface 966 or the media drive 968. Furthermore, the image processing section 964 decodes the encoded data input from the external interface 966 or the media drive 968 to generate image data. The image processing section 964 then outputs the generated image data to the display section 965. Furthermore, the image processing section 964 may output the image data input from the signal processing section 963 to the display section 965, causing an image to be displayed. Furthermore, the image processing section 964 may overlay display data obtained from the OSD 969 on the image to be output to the display section 965.

The OSD 969 generates images of the GUI, such as menus, buttons, cursors, and the like, and outputs the generated images to the image processing section 964 .

The external interface 966 is configured as, for example, a USB input/output terminal. For example, when printing an image, the external interface 966 connects the imaging device 960 and a printer. A drive may also be connected to the external interface 966 as appropriate. Removable media such as magnetic disks and optical disks can be installed in the drive, and programs read from the removable media can be installed in the imaging device 960. Furthermore, the external interface 966 can be configured as a network interface for connecting to a network such as a LAN or the Internet. In other words, the external interface 966 functions as a transmission device for the imaging device 960.

The recording medium to be mounted on the media drive 968 may be any readable and writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, a semiconductor memory, etc. In addition, the recording medium may be fixedly mounted on the media drive 968, constituting a non-removable storage section such as a built-in hard disk drive or an SSD (Solid State Drive).

The control section 970 includes a processor such as a CPU and a memory such as RAM or ROM. The memory stores programs to be executed by the CPU, program data, and the like. When the imaging device 960 is activated, the CPU reads and executes the program stored in the memory. By executing the program, the CPU controls the operation of the imaging device 960 in accordance with operation signals input from the user interface 971.

The user interface 971 is connected to the control section 970. The user interface 971 includes buttons, switches, and the like used by the user to operate the imaging device 960. The user interface 971 detects user operations via these constituent elements, generates an operation signal, and outputs the generated operation signal to the control section 970.

In the camera 960 configured in this manner, the image processing section 964 has the functions of the image encoding device 10 and the image decoding device 60 according to the above-described embodiment. Therefore, even when encoding and decoding images in the camera 960, the parallelism of the deblocking filter processing can be enhanced, thereby ensuring high-speed processing.

8. Summary

Referring to Figures 1-47, three embodiments of deblocking filters for an image encoding device 10 and an image decoding device 60 according to the present invention are described. These three embodiments alleviate the dependencies inherent in deblocking filter processing in the prior art. This improves the parallelism of processing when applying the deblocking filter. As a result, delays or data rate reductions caused by the large processing volume of the deblocking filter can be avoided, thereby ensuring high-speed processing. The parallelism and order of deblocking filter processing can be flexibly set according to various conditions such as image size and installation environment.

According to the first embodiment, when determining whether filtering is required for either a vertical boundary or a horizontal boundary, pixel values of the input image provided to the deblocking filter are referenced across multiple macroblocks within the image. This reduces processing dependencies between macroblocks or coding units. Therefore, it is possible to perform parallel filtering requirement determination processing across multiple macroblocks, or even all macroblocks within the image (if parallelism is maximized).

According to the second embodiment, whether filtering is required for each block's vertical and horizontal boundaries is determined without waiting for the deblocking filter to be applied to other blocks in the macroblock to which the block belongs. This reduces the processing dependencies between vertical and horizontal boundaries within a macroblock or coding unit. Consequently, filtering requirement determination processing for vertical and horizontal boundaries can be performed in parallel within a macroblock.

According to the third embodiment, the filtering processes for vertical and horizontal boundaries filter the pixels input to the deblocking filter. This structure allows the filtering processes for vertical and horizontal boundaries to be performed in parallel. This further accelerates the deblocking filter processing. For pixels updated by the two parallel filtering processes, the output pixel value is calculated based on the two filter outputs. Parallelizing the two filtering processes can also appropriately reduce block distortion at vertical and horizontal boundaries. The output pixel value can be calculated as a weighted average of the two filter outputs. This allows the deblocking filter to more effectively eliminate block distortion, thereby further improving image quality.

This specification primarily describes an example in which filtering for vertical boundaries precedes filtering for horizontal boundaries. Furthermore, the aforementioned effects of the techniques disclosed herein also apply to cases in which filtering for horizontal boundaries precedes filtering for vertical boundaries. The size of the deblocking filter processing unit or macroblock may be determined differently from that described in this specification. A feasible technique is to omit the filtering necessity determination process and parallelize the application of the deblocking filter to vertical and horizontal boundaries.

The technology for transmitting information for parallelizing the deblocking filter process from the encoder to the decoder is not limited to the technology for multiplexing the information into the coded stream header. For example, the information may not be multiplexed into the coded bit stream, but may be transmitted or recorded as independent data related to the coded bit stream. The term "association" means ensuring the possibility of associating an image (or a part of an image, such as a slice or a block) contained in the bit stream with the information corresponding to the image. That is, the information can be transmitted via a transmission path different from the transmission path used for the image (or bit stream). The information can be recorded on a recording medium different from the recording medium used for the image (or bit stream) (or a different recording area on the same recording medium). The information and the image (or bit stream) can be associated with each other based on any unit such as multiple frames, a frame, or a part of a frame.

The preferred embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is certainly not limited to the above examples. Those skilled in the art may derive various changes and modifications within the scope of the appended claims, and it should be understood that these changes and modifications are naturally within the technical scope of the present invention.

This specification refers to filtering for vertical boundaries as "horizontal filtering," and filtering for horizontal boundaries as "vertical filtering." Typically, filtering for vertical boundaries utilizes horizontally arranged filter taps. Filtering for horizontal boundaries utilizes vertically arranged filter taps. Therefore, the above terminology is used for filtering.

Reference Signs List

10,60 Image processing equipment

112,212 First judgment part (vertical boundary judgment part)

116,216 Second judgment part (horizontal boundary judgment part)

132 first filtering part (horizontal filtering component)

142 Second filtering part (vertical filtering component)

150 Parallelization Control Section

Claims (16)

1. An image processing apparatus, comprising: The decoding unit decodes the bitstream to generate an image; A horizontal filtering unit, spanning multiple vertical boundaries, applies the horizontal deblocking filter in a state where there is no processing dependency between the horizontal deblocking filters targeting the multiple vertical boundaries and the vertical deblocking filters in the vertical direction. The vertical boundaries are the vertical boundaries between blocks within the image generated by the decoding unit, and the horizontal deblocking filter targets pixels located near the vertical boundaries. A vertical filtering section, spanning multiple horizontal boundaries, applies a vertical deblocking filter. The horizontal boundaries are horizontal boundaries between blocks within an image that include pixels to which the horizontal deblocking filter has been applied by the horizontal filtering section. The vertical deblocking filter targets pixels located near the horizontal boundaries. 2. The image processing apparatus according to claim 1, wherein The horizontal filtering unit applies the horizontal deblocking filter to pixels located near all vertical boundaries of the image generated by the decoding unit. 3. The image processing apparatus according to claim 2, wherein The vertical filtering unit applies the vertical deblocking filter to pixels located near all horizontal boundaries of an image in which the deblocking filter is determined to be applied. 4. The image processing apparatus according to claim 3, wherein The horizontal filtering unit applies the horizontal deblocking filter in a manner that ensures that the horizontal deblocking filters for the multiple adjacent vertical boundaries between blocks do not depend on each other, so that there is no dependency between the processing of the horizontal deblocking filters and the processing of the vertical deblocking filters among the multiple blocks that are the processing targets of the horizontal deblocking filters. The vertical filtering unit applies the vertical deblocking filter in such a way that the vertical deblocking filters for the multiple adjacent horizontal boundaries between blocks do not depend on each other. 5. The image processing apparatus according to claim 4, wherein The horizontal filtering section applies the horizontal deblocking filters to each other using non-repeating tap lengths for the horizontal deblocking filters of the plurality of adjacent vertical boundaries between blocks. The vertical filtering section applies the vertical deblocking filter to each other using non-repeating tap lengths for the vertical deblocking filters of the plurality of adjacent horizontal boundaries between blocks. 6. The image processing apparatus according to claim 1, wherein The horizontal filtering section applies the horizontal deblocking filter in parallel. The vertical filtering section applies the vertical deblocking filter in parallel. 7. The image processing apparatus according to claim 1, wherein The block size is 8×8. The length of the vertical boundary is 8 pixels. The length of the horizontal boundary is 8 pixels. 8. The image processing apparatus according to claim 1, wherein The decoding unit decodes the bitstream according to each encoding unit divided from the largest encoding unit to generate an image. 9. An image processing method, comprising: The decoding step involves decoding the bitstream to generate an image; A horizontal filtering step, spanning multiple vertical boundaries, applies the horizontal deblocking filter in a state where there is no processing dependency between the horizontal deblocking filters applied to the multiple vertical boundaries and the vertical deblocking filters applied to the vertical boundaries. The vertical boundaries are the vertical boundaries between blocks within the image generated by the decoding step, and the horizontal deblocking filter targets pixels located near the vertical boundaries. The vertical filtering step applies a vertical deblocking filter across multiple horizontal boundaries, whereby the horizontal boundaries are horizontal boundaries between blocks within an image that include pixels to which the horizontal deblocking filter has been applied in the horizontal filtering step, and the vertical deblocking filter targets pixels located near the horizontal boundaries. 10. The image processing method according to claim 9, wherein In the horizontal filtering step, the horizontal deblocking filter is applied to pixels within the image generated by the decoding step that are located near all the vertical boundaries of the image to which the horizontal deblocking filter is determined to be applied. 11. The image processing method according to claim 10, wherein In the vertical filtering step, the vertical deblocking filter is applied to pixels within the image of the image to which the deblocking filter has been applied in the horizontal filtering step, which are located near all the horizontal boundaries of the image that are determined to be to be applied to the vertical deblocking filter. 12. The image processing method according to claim 11, wherein... In the horizontal filtering step, the horizontal deblocking filters are applied in such a way that the horizontal deblocking filters for the multiple adjacent vertical boundaries between blocks do not depend on each other, so that there is no dependency between the multiple blocks that are the processing objects of the horizontal deblocking filters and the processing associated with the vertical deblocking filters. In the vertical filtering step, the vertical deblocking filter is applied in such a way that the vertical deblocking filters for the multiple adjacent horizontal boundaries between blocks do not depend on each other. 13. The image processing method according to claim 12, wherein... In the horizontal filtering step, the horizontal deblocking filters for the multiple adjacent vertical boundaries between blocks are applied to each other using non-repeating tap lengths. In the vertical filtering step, the vertical deblocking filters for the multiple adjacent horizontal boundaries between blocks are applied to each other using non-repeating tap lengths. 14. The image processing method according to claim 9, wherein In the horizontal filtering step, the horizontal deblocking filter is applied in parallel. In the vertical filtering step, the vertical deblocking filter is applied in parallel. 15. The image processing method according to claim 9, wherein The block size is 8×8. The length of the vertical boundary is 8 pixels. The length of the horizontal boundary is 8 pixels. 16. The image processing method according to claim 9, wherein In the decoding step, the bitstream is decoded according to each encoding unit divided from the largest encoding unit to generate an image.