HK1233094A - Image processing device and image processing method - Google Patents

Description

Image processing apparatus and image processing method

The present application is a divisional application of a chinese invention patent application having an application number of 201180057796.9(PCT/JP2011/077954) and an invention name of "image processing apparatus and image processing method", which is filed on 12/2/2011 by the same applicant.

Technical Field

The present disclosure relates to an image processing apparatus and an image processing method.

Background

h.264/AVC (one of the standard specifications of image coding schemes), a deblocking filter is applied to a block boundary in units of blocks each containing 4 × 4 pixels in order to avoid image quality degradation caused by block distortion when coding an image. Deblocking filters require a significant amount of processing, which can account for 50% of the total processing in image decoding.

Standardization work for High Efficiency Video Coding (HEVC) (next generation image coding scheme) has proposed applying a deblocking filter in units of blocks each containing 8 × 8 pixels or more, in accordance with JCTVC-a119 (see non-patent document 1). The technique proposed in JCTVC-a119 increases the block size to the minimum unit that allows the application of a deblocking filter to perform filtering processing in parallel for the same direction block boundaries within one macro block.

CITATION LIST

Non-patent document

Non-patent document 1: ugur (Nokia), K.R.Andersson (LM Ericsson), A.Fuldseth (Tandberg Telecom), "JCTVC-A119: Video Coding technology pro + y Tandberg, Nokia, and Ericsson", Documents of the first meeting of the Joint Video Coding technique on Video Coding (JCT-VC), Dresden, Germany,15-23April, 2010.

Disclosure of Invention

Even with the technique proposed in JCTVC-A119, there is still a dependency between the processing of vertical block boundaries and the processing of horizontal block boundaries. Specifically, the processing of the vertical boundary of one macroblock waits until the processing of the horizontal boundary of an adjacent macroblock is performed. The processing of the horizontal boundary of one macroblock is waited until the processing of the vertical boundary of the same macroblock is performed. The above-described techniques provide only a very limited degree of parallel processing of the deblocking filter. Thus, the above-described techniques cannot successfully solve the problems of delay and reduction in data rate caused by a large amount of processing when the deblocking filter is applied.

It is an object of the technique according to the present disclosure to provide an image processing apparatus and an image processing method capable of providing further parallel processing when a deblocking filter is applied.

According to an embodiment of the present disclosure, there is provided an image processing apparatus including a decoding section configured to decode an image from an encoded stream, a horizontal filtering section configured to apply a deblocking filter to a vertical block boundary within the image to be decoded by the decoding section, a vertical filtering section configured to apply a deblocking filter to a horizontal block boundary within the image to be decoded by the decoding section, and a control section configured to cause the horizontal filtering section to filter, in parallel, a plurality of vertical block boundaries included in a processing unit including a plurality of coding units, and to cause the vertical filtering section to filter, in parallel, the plurality of horizontal block boundaries included in the processing unit.

Typically, the image processing apparatus may be implemented as an image decoding apparatus that decodes an image.

According to an embodiment of the present disclosure, there is provided an image processing method including decoding an image from an encoded stream, performing horizontal filtering to apply a deblocking filter to a vertical block boundary within the image to be decoded, performing vertical filtering to apply a deblocking filter to a horizontal block boundary within the image to be decoded, and controlling the horizontal filtering and the vertical filtering so as to filter a plurality of vertical block boundaries included in a processing unit including a plurality of coding units in parallel, and filter a plurality of horizontal block boundaries included in the processing unit in parallel.

According to an embodiment of the present disclosure, there is provided an image processing apparatus including a horizontal filtering section configured to apply a deblocking filter to a vertical block boundary within an image to be locally decoded when encoding the image to be encoded, a vertical filtering section configured to apply the deblocking filter to the horizontal block boundary within the image, a control section configured to cause the horizontal filtering section to filter in parallel a plurality of vertical block boundaries included in a processing unit including a plurality of coding units, and cause the vertical filtering section to filter in parallel a plurality of horizontal block boundaries included in the processing unit, and an encoding section configured to encode the image to be encoded using the image filtered by the horizontal filtering section and the vertical filtering section.

Typically, the image processing apparatus may be implemented as an image encoding apparatus that encodes an image.

According to an embodiment of the present disclosure, there is provided an image processing method including performing horizontal filtering to apply a deblocking filter to a vertical block boundary within an image to be locally decoded when the image to be encoded is encoded, performing vertical filtering to apply a deblocking filter to a horizontal block boundary within the image, controlling the horizontal filtering and the vertical filtering to filter a plurality of vertical block boundaries included in a processing unit including a plurality of coding units in parallel, and filtering a plurality of horizontal block boundaries included in the processing unit in parallel, and encoding the image to be encoded using the image filtered by the horizontal filtering and the vertical filtering.

As described above, the image processing apparatus and the image processing method according to the present disclosure also improve parallel processing when the deblocking filter is applied.

Drawings

Fig. 1 is a block diagram showing an example of the configuration of an image encoding apparatus according to the embodiment.

Fig. 2 is a block diagram showing an example of the configuration of an image decoding apparatus according to the embodiment.

Fig. 3 is an explanatory diagram showing an example of adjacent pixels around the boundary.

Fig. 4 is an explanatory diagram illustrating a reference pixel during a filtering need determination process according to the related art.

Fig. 5 is an explanatory diagram illustrating a pixel updated with the filtering process.

Fig. 6 is an explanatory diagram illustrating identification of an edge for explaining the embodiment.

Fig. 7 is an explanatory diagram illustrating parallel processing according to the related art.

Fig. 8 is a first explanatory diagram illustrating a dependency relationship between processes according to the related art.

Fig. 9 is a second explanatory diagram illustrating a dependency relationship between processes according to the related art.

Fig. 10 is an explanatory diagram illustrating a processing sequence according to the related art.

Fig. 11 is an explanatory diagram illustrating a processing sequence according to the first example.

Fig. 12 is a block diagram illustrating a detailed structure of a deblocking filter according to the first embodiment.

Fig. 13 is a block diagram illustrating a detailed structure of the determination section.

Fig. 14 is an explanatory diagram illustrating neighboring blocks around a slice boundary.

Fig. 15 is an explanatory diagram illustrating a first example of a processing sequence of each slice.

Fig. 16 is an explanatory diagram illustrating a second example of the processing sequence of each slice.

Fig. 17 is an explanatory diagram illustrating first and second examples of the determination technique provided by the modification.

Fig. 18 is an explanatory diagram illustrating third and fourth examples of the determination technique provided by the modification.

Fig. 19 is an explanatory diagram illustrating fifth and sixth examples of the determination technique provided by the modification.

Fig. 20 is a flowchart illustrating the flow of processing of the deblocking filter according to the first example.

Fig. 21 is a flowchart illustrating the flow of the filtering necessity determination process.

Fig. 22 is an explanatory diagram illustrating a processing sequence according to the second example.

Fig. 23 is a block diagram illustrating a detailed structure of a deblocking filter according to the second example.

Fig. 24 is a flowchart illustrating the processing flow of the deblocking filter according to the second example.

Fig. 25 is an explanatory diagram illustrating a processing sequence of each CLU.

FIG. 26 is a flow chart illustrating the process flow for each LCU.

Fig. 27 is an explanatory diagram illustrating an overview of the third example.

Fig. 28 is a block diagram illustrating a detailed structure of a deblocking filter according to the third example.

Fig. 29 is an explanatory diagram illustrating determination of the weight of the weighted average.

Fig. 30 is an explanatory diagram illustrating an example of the weight of the weighted average.

Fig. 31 is an explanatory diagram illustrating an output pixel value of the calculation section according to the third example.

Fig. 32 is an explanatory diagram illustrating a first example of a processing sequence for comparison.

Fig. 33 is an explanatory diagram illustrating a first example of the processing sequence provided by the third example.

Fig. 34 is an explanatory diagram illustrating a second example of the processing sequence for comparison.

Fig. 35 is an explanatory diagram illustrating a second example of the processing sequence provided by the third example.

Fig. 36 is a flowchart illustrating a first example of the processing flow of the deblocking filter according to the third example.

Fig. 37 is a flowchart illustrating the flow of the pixel value calculation process shown in fig. 36.

Fig. 38 is an explanatory diagram illustrating a multi-view codec.

Fig. 39 is an explanatory diagram illustrating an image encoding process applied to a multi-view codec according to an embodiment.

Fig. 40 is an explanatory diagram illustrating an image decoding process applied to a multi-view codec according to an embodiment.

Fig. 41 is an explanatory diagram illustrating an extensible codec.

Fig. 42 is an explanatory diagram illustrating an image encoding process applied to a scalable codec according to an embodiment.

Fig. 43 is an explanatory diagram illustrating an image decoding process applied to a scalable codec according to an embodiment.

Fig. 44 is a block diagram illustrating a schematic structure of a television set.

Fig. 45 is a block diagram illustrating a schematic structure of a mobile phone.

Fig. 46 is a block diagram illustrating a schematic structure of a recording/reproducing apparatus.

Fig. 47 is a block diagram illustrating a schematic configuration of the image pickup apparatus.

Detailed Description

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that in the specification and the drawings, elements having substantially the same function and structure are denoted by the same reference numerals, and repeated description is omitted.

The embodiments will be described in the following order.

1. Overview of the devices

1-1. image encoding apparatus

1-2 image decoding apparatus

2. Description of the Prior Art

2-1 basic structure of deblocking filter

2-2. dependency between Processes according to the prior art

3. First embodiment

3-1. deblocking Filter architecture example

3-2 variation of determination conditions

3-3. treatment process

4. Second embodiment

4-1. deblocking Filter architecture example

4-2. treatment process

4-3. processing examples for each LCU

5. Third embodiment

5-1. overview

5-2. deblocking Filter architecture example

5-3 example processing sequence

5-4. treatment process

6. Applications to various codecs

6-1. multiview codec

6-2. scalable codec

7. Illustrative applications

8. Summary of the invention

1. Overview of the devices

Referring to fig. 1 and 2, an overview of an apparatus to which the technology disclosed in the present specification is applied is described below. The technique disclosed in the present specification is applicable to an image encoding apparatus and an image decoding apparatus.

[1-1. image encoding apparatus ]

Fig. 1 is a block diagram showing an example of the structure of an image encoding apparatus 10 according to the embodiment. Referring to fig. 1, the image encoding apparatus 10 includes an a/D (analog-to-digital) conversion section 11, a rearrangement buffer 12, a subtraction section 13, an orthogonal transformation section 14, a quantization section 15, a lossless encoding section 16, an accumulation buffer 17, a rate control section 18, an inverse quantization section 21, an inverse orthogonal transformation section 22, an addition section 23, a deblocking filter 24a, a frame memory 25, a selector 26, an intra prediction section 30, a motion estimation section 40, and a mode selection section 50.

The a/D conversion section 11 converts an image signal input in an analog format into image data in a digital format, and then outputs a series of digital image data to the rearrangement buffer 12.

The rearrangement buffer 12 rearranges an image included in a series of image data input from the a/D conversion section 11. After rearranging the images in accordance with the GOP (group of pictures) structure corresponding to the encoding process, the rearrangement buffer 12 outputs the rearranged image data to the subtraction section 13, the intra prediction section 30, and the motion estimation section 40.

The image data input from the rearrangement buffer 12 and the predicted image data selected by the mode selecting portion 50 described later are supplied to the subtracting portion 13. The subtracting section 13 calculates prediction error data, which is a difference between the image data input from the rearrangement buffer 12 and the predicted image data input from the mode selecting section 50, and outputs the calculated prediction error data to the orthogonal transform section 14.

The orthogonal transformation section 14 orthogonally transforms the prediction error data input from the subtraction section 13. The orthogonal transform performed by the orthogonal transform section 14 may be, for example, Discrete Cosine Transform (DCT) or Karhunen-Loeve transform. The orthogonal transformation section 14 outputs the transformation coefficient data obtained by the orthogonal transformation process to the quantization section 15.

The transform coefficient data input from the orthogonal transform section 14 and a rate control signal from a rate control section 18 described later are supplied to the quantization section 15. The quantization section 15 quantizes the transform coefficient data, and outputs the quantized transform coefficient data (hereinafter referred to as quantized data) to the lossless coding section 16 and the inverse quantization section 21. In addition, the quantization section 15 switches the quantization parameter (quantization scale) in accordance with the rate control signal from the rate control section 18, thereby changing the bit rate of the quantized data to be input to the lossless coding section 16.

The quantized data input from the quantization section 15, and information on intra prediction or inter prediction, which will be described later, generated by the intra prediction section 30 or the motion estimation section 40 and selected by the mode selection section 50 are supplied to the lossless encoding section 16. The information regarding intra prediction may include prediction mode information indicating an optimal intra prediction mode for each block. In addition, the information regarding inter prediction may include, for example, prediction mode information for predicting a motion vector of each block, differential motion vector information, reference picture information, and the like.

The lossless encoding section 16 generates an encoded stream by performing lossless encoding processing on the quantized data. The lossless coding of the lossless coding section 16 may be, for example, variable length coding or arithmetic coding. Further, the lossless encoding section 16 multiplexes the above-mentioned information on intra prediction or information on inter prediction into a header (e.g., a block header, a slice header, etc.) of the encoded stream. Subsequently, the lossless coding section 16 outputs the generated coded stream to the accumulation buffer 17.

The accumulation buffer 17 temporarily holds the encoded stream input from the lossless encoding section 16 using a storage medium such as a semiconductor memory. Subsequently, the accumulation buffer 17 outputs the accumulated encoded stream at a rate corresponding to the frequency band of the transmission line (or the output line from the image encoding device 10).

The rate control section 18 monitors the free space of the accumulation buffer 17. Subsequently, the rate control section 18 generates a rate control signal in accordance with the free space on the accumulation buffer 17, and outputs the generated rate control signal to the quantization section 15. For example, when there is not a large amount of free space on the accumulation buffer 17, the rate control section 18 generates a rate control signal that reduces the bit rate of the quantized data. In addition, for example, when the free space on the accumulation buffer 17 is sufficiently large, the rate control section 18 generates a rate control signal that increases the bit rate of the quantized data.

The inverse quantization section 21 performs inverse quantization processing on the quantized data input from the quantization section 15. Subsequently, the inverse quantization section 21 outputs the transform coefficient data obtained by the inverse quantization process to the inverse orthogonal transform section 22.

The inverse orthogonal transform part 22 performs inverse orthogonal transform processing on the transform coefficient data input from the inverse quantization part 21, thereby restoring prediction error data. Subsequently, the inverse orthogonal transform section 22 outputs the restored prediction error data to the addition section 23.

The addition section 23 adds the restored prediction error data input from the inverse orthogonal transform section 22 and the predicted image data input from the mode selection section 50, thereby generating decoded image data. Subsequently, the addition section 23 outputs the generated decoded image data to the deblocking filter 24a and the frame memory 25.

The deblocking filter 24a performs a filtering process to reduce block distortion occurring during image coding. For example, the deblocking filter 24a decides the necessity of filtering for each block boundary of the decoded image data supplied from the addition section 23, and applies the deblocking filter to the boundary determined to require a filter. The deblocking filter 24a is also supplied with information (for example, mode information, transform coefficient information, and motion vector information) for deciding the necessity of filtering, and decoded image data from the addition section 23. After the filtering, the block distortion is removed from the decoded image data, and the deblocking filter 24a outputs the decoded image data to the frame memory 25. The process of the deblocking filter 24a will be described later in detail.

The frame memory 25 holds the decoded image data input from the addition section 23 and the filtered decoded image data input from the deblocking filter 24a using a storage medium.

The selector 26 reads the decoded image data before filtering to be used for intra prediction from the frame memory 25, and supplies the read decoded image data as reference image data to the intra prediction section 30. In addition, the selector 26 reads the filtered decoded image data to be used for inter prediction from the frame memory 25 and supplies the read decoded image data as reference image data to the motion estimation section 40.

The intra prediction section 30 performs intra prediction processing in accordance with each intra prediction mode based on the image data to be encoded input from the rearrangement buffer 12 and the decoded image data supplied via the selector 26. For example, the intra prediction section 30 evaluates the prediction result of each intra prediction mode using a predetermined cost function. Subsequently, the intra prediction section 30 selects, as the optimum intra prediction mode, the intra prediction mode that minimizes the cost function value, that is, the intra prediction mode that maximizes the compression ratio. Further, the intra prediction section 30 outputs prediction mode information indicating the optimum intra prediction mode, prediction image data, and information on intra prediction such as a cost function value to the mode selection section 50.

The motion estimation section 40 performs an inter-frame prediction process (prediction process between frames) based on the image data for encoding supplied from the rearrangement buffer 12 and the decoded image data supplied via the selector 26. For example, the motion estimation section 40 evaluates the prediction result for each prediction mode using a predetermined cost function. Subsequently, the motion estimation section 40 selects the optimal prediction mode, i.e., the prediction mode in which the cost function value is minimized or the compression ratio is maximized. The motion estimation section 40 generates predicted image data in accordance with the optimum prediction mode. The motion estimation section 40 outputs information on inter prediction such as prediction mode information indicating an optimal intra prediction mode, prediction image data, and a cost function value to the mode selection section 50.

The mode selection part 50 compares the cost function value related to intra prediction input from the intra prediction part 30 with the cost function value related to inter prediction input from the motion estimation part 40. Subsequently, the mode selection section 50 selects a prediction method having a smaller cost function value from the intra prediction and the inter prediction. In the case where intra prediction is selected, the mode selection portion 50 outputs information on intra prediction to the lossless encoding section 16, and in addition, outputs prediction image data to the subtraction portion 13 and the addition portion 23. Further, in the case where inter prediction is selected, the mode selection portion 50 outputs the above-described information on inter prediction to the lossless encoding section 16, and also outputs predicted image data to the subtraction portion 13 and the addition portion 23.

[1-2. image decoding apparatus ]

Fig. 2 is a block diagram showing a configuration example of the image decoding apparatus 60 according to the embodiment. Referring to fig. 2, the image decoding apparatus 60 includes an accumulation buffer 61, a lossless decoding portion 62, an inverse quantization portion 63, an inverse orthogonal transform portion 64, an addition portion 65, a deblocking filter 24b, a rearrangement buffer 67, a D/a (digital-analog) conversion portion 68, a frame memory 69, selectors 70 and 71, an intra prediction portion 80, and a motion compensation portion 90.

The accumulation buffer 61 temporarily holds the encoded stream input via the transmission line using a storage medium.

The lossless decoding section 62 decodes the encoded stream input from the accumulation buffer 61 in accordance with the encoding method used at the time of encoding. In addition, the lossless decoding section 62 decodes the information multiplexed to the header area of the encoded stream. The information multiplexed into the header area of the encoded stream may include, for example, information on intra prediction and information on inter prediction in a block header. The lossless decoding section 62 outputs information on intra prediction to the intra prediction section 80. In addition, the lossless decoding section 62 outputs information on inter prediction to the motion compensation section 90.

The inverse quantization part 63 inversely quantizes the quantized data that has been decoded by the lossless decoding part 62. The inverse orthogonal transform part 64 generates prediction error data by performing inverse orthogonal transform on the transform coefficient data input from the inverse quantization part 63 in accordance with the orthogonal transform method used at the time of encoding. Subsequently, the inverse orthogonal transform section 64 outputs the generated prediction error data to the addition section 65.

The addition section 65 adds the prediction error data input from the inverse orthogonal transform section 64 and the predicted image data input from the selector 71, thereby generating decoded image data. Subsequently, the addition section 65 outputs the generated decoded image data to the deblocking filter 24b and the frame memory 69.

The deblocking filter 24b performs a filtering process to reduce block distortion occurring on the decoded image. The deblocking filter 24b determines the necessity of filtering at each block boundary, for example, with respect to the decoded image data input from the addition section 65, and applies the deblocking filter to the boundary determined to require a filter. The deblocking filter 24b is also supplied with information for determination of the necessity of filtering, and the decoded image data from the addition section 65. After the filtering, the block distortion is removed from the decoded image data, and the deblocking filter 24b outputs the decoded image data to the rearrangement buffer 67 and the frame memory 69. The process of the deblocking filter 24b will be described later in detail.

The rearrangement buffer 67 generates a series of image data in time order by rearranging the image input from the deblocking filter 24 b. Subsequently, the rearrangement buffer 67 outputs the generated image data to the D/a conversion portion 68.

The D/a conversion section 68 converts the image data in the digital format input from the rearrangement buffer 67 into an image signal in an analog format. Subsequently, the D/a conversion section 68 causes an image to be displayed, for example, by outputting the analog image signal to a display (not shown) connected to the image decoding apparatus 60.

The frame memory 69 holds the decoded image data before filtering input from the addition section 65 and the decoded image data after filtering input from the deblocking filter 24b with a storage medium.

The selector 70 switches the output destination of the image data from the frame memory 69 between the intra prediction section 80 and the motion compensation section 90 for each block in the image in accordance with the mode information obtained by the lossless decoding section 62. For example, in the case where the intra prediction mode is designated, the selector 70 outputs the pre-filtering decoded image data supplied from the frame memory 69 to the intra prediction section 80 as reference image data. In addition, in the case where the inter prediction mode is specified, the selector 70 outputs the filtered decoded image data supplied from the frame memory 69 to the motion compensation section 90 as reference image data.

The selector 71 switches the output source of the predicted image data to be supplied to the addition section 65 between the intra prediction section 80 and the motion compensation section 90 for each block in the image in accordance with the mode information obtained by the lossless decoding section 62. For example, in the case where the intra prediction mode is specified, the selector 71 supplies the prediction image data output from the intra prediction portion 80 to the addition portion 65. In the case where the inter prediction mode is specified, the selector 71 supplies the predicted image data output from the motion compensation section 90 to the addition section 65.

The intra prediction section 80 performs in-picture (in screen) prediction of pixel values based on the information on intra prediction input from the lossless decoding section 62 and the reference image data from the frame memory 69, generating predicted image data. Subsequently, the intra prediction section 80 outputs the generated prediction image data to the selector 71.

The motion compensation section 90 performs motion compensation processing based on the information on inter prediction input from the lossless decoding section 62 and the reference image data from the frame memory 69, generating predicted image data. Subsequently, the motion compensation section 90 outputs the generated prediction image data to the selector 71.

<2. Prior Art >

[2-1. basic Structure of deblocking Filter ]

Generally, a process using a deblocking filter in an existing image coding scheme such as h.264/AVC or HEVC includes two processes, i.e., filtering requires a decision process and a filtering process. For example, the two processes in HEVC are explained below.

(1) Filtering need determination processing

The filtering need decision process decides whether or not a deblocking filter needs to be applied to each boundary of the individual blocks within the input image. The block boundaries include vertical boundaries between respective blocks that are horizontally adjacent to each other, and horizontal boundaries between respective blocks that are vertically adjacent to each other. JCTVC-a119 uses a block size of 8 × 8 pixels as a minimum processing unit. For example, a macroblock of 16 × 16 pixels includes 4 blocks of 8 × 8 pixels. This process is applied for one (left) vertical boundary and one (upper) horizontal boundary per block, i.e. 4 boundaries plus 4 boundaries add up to 8 boundaries. This specification assumes that a macroblock as a technical term includes a Coding Unit (CU) in the context of HEVC.

Fig. 3 is an explanatory diagram showing an example of pixels in two blocks (adjacent blocks) Ba and Bb adjacent to each other around the boundary. The following illustrates a vertical boundary, and the description is obviously applicable to a horizontal boundary. Example in FIG. 3 utilizesSymbol p_ijRepresenting the pixels in block Ba. In this symbol, i denotes a column index and j denotes a row index. Starting with the column closest to the vertical boundary, the column indices i are numbered sequentially (from right to left) as 0, 1, 2, and 3. From top to bottom, the row index j is numbered 0, 1, 2, …, 7. The left half of block Ba is omitted from the figure. Symbol q_kjFor representing the pixels in the block Bb. In this symbol, k denotes a column index and j denotes a row index. Starting from the column closest to the vertical boundary, the column indices k are numbered sequentially (left to right) as 0, 1, 2 and 3. The right half of block Bb is omitted from the figure.

The necessity of applying the deblocking filter to the vertical boundary between the blocks Ba and Bb shown in fig. 3 can be determined using the following conditions.

Decision condition … for the Luma component (Luma) if both conditions a and B are true, then a deblocking filter is applied.

Condition a:

(A1) the block Ba or Bb enters an intra-frame prediction mode;

(A2) block Ba or Bb has non-zero orthogonal transform coefficients; or

(A3) | MVAx-MVBx | ≧ 4 or | MVAy-MVBy | ≧ 4

Condition B:

|p₂₂-2p₁₂+p₀₂|+|q₂₂-2q₁₂+q₀₂|+|p₂₅-2p₁₅+p₀₅|+|q₂₅-2q₁₅+q₀₅|<β

the condition a3 assumes that the motion vector of the block Ba is (MVAx, MVAy) and the motion vector of the block Bb is (MVBx, MVBy) in terms of Qpel (1/4 pixels) precision. Condition B uses β as an edge determination threshold. According to the quantization parameter, an initial value of β is given. The user may specify the value of β using parameters within the slice header.

Decision condition … for the Chroma component (Chroma) applies a deblocking filter if condition a1 is true.

Condition a 1: the block Ba or Bb enters the intra prediction mode.

As shown by the broken line boxes L3 and L6 in fig. 4, filtering for a general vertical boundary requires a decision process (particularly in terms of a decision condition B of a luminance component) to refer to pixels on the third and sixth lines (assuming that the uppermost line is the first line) in each block. Similarly, filtering of the horizontal boundary requires that the decision process reference pixels on the third and sixth columns in each block (not shown in fig. 4). The above-described decision condition is used to determine the boundary to which the deblocking filter needs to be applied to perform the filtering process described below.

(2) Filtering process

If it is determined that the deblocking filter needs to be applied to the boundary, a filtering process is performed on pixels on the left and right sides of the vertical boundary, and pixels on the upper and lower sides of the horizontal boundary. With regard to the luminance component, the filter strength is switched between the strong filter and the weak filter in accordance with the pixel value.

Filtering luminance components

The selection strength … is per row or per column selection filter strength. A strong filter is selected if the following conditions C1-C3 are all satisfied. A weak filter is selected if any of the following conditions is not met.

(C1)d<(β>>2)

(C2)(|p_3j-p_0j|+|q_0j-q_3j|)<(β>>3)

(C3)|p_0j-q_0j|<((5t_C+1)>>1)

Where j represents the row index of the vertical boundary, or the column index of the horizontal boundary. d ═ p₂₂-2p₁₂+p₀₂|+|q₂₂-2q₁₂+q₀₂|+|p₂₅-2p₁₅+p₀₅|+|q₂₅-2q₁₅+q₀₅|。

Weak filtering

△＝Clip(-t_C,t_C,(13(q_0j-p_0j)+4(q_1j-p_1j)-5(q_2j-p_2j)+16)>>5))

p_0j＝Clip_0-255(p_0j+△)

q_0j＝Clip_0-255(q_0j-△)

p_1j＝Clip_0-255(p_1j+△/2)

q_1j＝Clip_0-255(q_1j-△/2)

Strong filtering

p_0j＝Clip_0-255((p_2j+2p_1j+2p_0j+2q_0j+q_1j+4)>>3)

q_0j＝Clip_0-255((p_1j+2p_0j+2q_0j+2q_1j+q_2j+4)>>3)

p_1j＝Clip_0-255((p_2j+p_1j+p_0j+q_0j+2)>>2)

q_1j＝Clip_0-255((p_0j+q_0j+q_1j+q_2j+2)>>2)

p_2j＝Clip_0-255((2p_3j+3p_2j+p_1j+p_0j+q_0j+4)>>3)

q_2j＝Clip_0-255((p_0j+q_0j+q_1j+3q_2j+2q_3j+4)>>3)

Wherein Clip (a, b, c) represents a process of clamping a value c within a range a ≦ c ≦ b, Clip_0-255(c) Representing a process of clamping the value c within the range 0 ≦ c ≦ 255.

Filtering chrominance components

△＝Clip(-t_C,t_C,((((q_0j-p_0j)<<2)+p_1j-q_1j+4)>>3))

p_0j＝Clip_0-255(p_0j+△)

q_0j＝Clip_0-255(q_0j-△)

As shown by the broken-line boxes C6-C8 and C1-C3 in fig. 5, the pixel values on the first column through the third column, and the sixth column through the eighth column in each block are updated for the filtering processing of a general vertical boundary (especially for strong filtering of a luminance component). Similarly, the filtering process for the horizontal boundary updates the pixel values on the first to third rows, and the sixth to eighth rows in each block.

[2-2. dependency between Processes according to Prior Art ]

For the sake of illustration, as shown in fig. 6, macroblocks MBx (MB0, MB1 …) each having a size of 16 × 16 pixels include an upper left vertical boundary represented as Vx,0, an upper middle vertical boundary represented as Vx,1, a lower left vertical boundary represented as Vx,2, an upper middle vertical boundary represented as Vx,3, an upper left horizontal boundary represented as Hx,0, an upper right horizontal boundary represented as Hx,1, a lower left horizontal boundary represented as Hx,2, and a lower right horizontal boundary represented as Hx,3_ZAnd the filtering process is denoted as F_Z。

The above-described prior art results in that within one macro-block, there is no dependency between the processes with respect to the same-direction boundaries. Thus, the technique enables parallel filtering of vertical and horizontal boundaries within a macroblock. For example, fig. 7 clearly shows that within macroblock MB0, within 4 filtering processes F_V0,0,F_V0,1,F_V0,2And F_V0,3In between, there is no dependency relationship (no pixel is repeatedly updated), and the filtering process can be performed in parallel.

However, the above-described prior art leaves behind the need to determine the dependency between the filtering process for vertical boundaries and the filtering process for horizontal boundaries. This prior art also leaves the dependency between the filtering process for horizontal boundaries and the filtering process for vertical boundaries to be determined. For example, if a vertical boundary is processed before a horizontal boundary, then filtering a horizontal boundary within a given macroblock after terminating the filtering process for the vertical boundary requires a decision process. For example, fig. 8 shows that in the macroblock MB0, the filtering requirement determination process J_H0,0Dependent on the filtering process F_V0,0And F_V0,1As a result of (1), filtering requires decision processing J_H0,1Dependent on the filtering process F_V0,1The result of (1). Similarly, filtering vertical boundaries within a given macroblock after the filtering process for the horizontal boundaries of neighboring macroblocks has terminated requires a decision process. For example, fig. 9 shows the filtering necessity determination process J for the macroblock MB1_V1,0Dependent on the filtering process F for macroblock MB0_H0,1And F_H0,3As a result, the decision process J is required for the filtering of the macroblock MB1_V1,2Dependent on the filtering process F for macroblock MB0_H0,3The result of (1).

The related art relates to the dependency relationship between processes, and thus even with the technique proposed in JCTVC-a119, it is possible to provide parallel processing of a deblocking filter to a very limited extent.

Fig. 10 is an explanatory diagram illustrating a sequence of a deblocking filter process according to the related art. This example assumes that the deblocking filter is supplied with an image of size 32 × 32 pixels. The input image includes 4 macroblocks MB0-MB3, each having a size of 16 × 16 pixels.

In fig. 10, each of the dashed boxes represents processing to be performed in parallel. For example, the first step of filtering 4 vertical boundaries in macroblock MB0 in parallel requires decision process J_V0,0,J_V0,1,J_V0,2And J_V0,3. The second step performs the filtering process F in parallel on the 4 vertical boundaries in the macroblock MB0_V0,0,F_V0,1,F_V0,2F_V0,3. After the second step has terminated, the third step performs filtering in parallel on the 4 horizontal boundaries in macroblock MB0, requiring decision process J_H0,0,J_H0,1,J_H0,2And J_H0,3. The third step uses the pixel values after the filtering processing with respect to the vertical boundary in the second step for the filtering of the horizontal boundary requiring decision processing. The fourth step performs the filtering process F on the 4 horizontal boundaries in the macroblock MB0 in parallel_H0,0,F_H0,1,F_H0,2And F_H0,3. After the fourth step is terminated, the processing for the macro block MB1 is successively performed (fifth step to eighth step). The fifth step uses the pixel values after the filtering process of the fourth step with respect to the horizontal boundary of the macroblock MB0 for the filtering of the vertical boundary of the macroblock MB1 requiring the decision process. After the processing for the macroblock MB1 is terminated, the processing for the macroblock MB2 is successively performed (ninth step to twelfth step). After the process for the macroblock MB2 is terminated, the process for the macroblock MB3 is successively performed (thirteenth step to sixteenth step).

Such parallel processing of limited extent cannot satisfactorily solve the problem of delay or reduction in data rate caused by a large amount of processing when the deblocking filter is applied. The 3 embodiments described below further improve the parallel processing when applying the deblocking filter.

<3 > first embodiment

[3-1. deblocking Filter Structure example ]

An exemplary configuration of the deblocking filter 24a of the image encoding apparatus 10 shown in fig. 1 and the deblocking filter 24b of the image decoding apparatus 60 shown in fig. 2 according to the first embodiment is explained below. The structures of the deblocking filter 24a and the deblocking filter 24b may be identical to each other. In the following description, when it is not necessary to distinguish the deblocking filter 24a and the deblocking filter 24b from each other, they are collectively referred to as a deblocking filter 24.

(1) Dependencies between new processes

According to the present embodiment, the processing using the deblocking filter 24 also includes two kinds of processing, i.e., filtering requiring decision processing and filtering processing. In the filtering-requiring decision process with respect to one of the vertical boundary and the horizontal boundary, the deblocking filter 24 uses pixel values of an image input to the deblocking filter for the decision across a plurality of macroblocks. For example, if the vertical boundary is processed before the horizontal boundary, deblocking filter 24 may filter the vertical boundary of a particular block requiring decision processing without waiting for filtering processing for the horizontal boundaries of neighboring blocks. Deblocking filter 24 may filter the horizontal boundaries of a particular block requiring a decision process if the horizontal boundaries are processed before the vertical boundaries, without having to wait for a filtering process for the horizontal boundaries of neighboring blocks. As a result, processing dependencies between macroblocks will be mitigated.

Reducing the dependency of processing between macroblocks enables parallel processing between macroblocks in an image. This enables, for example, filtering in parallel for the vertical boundaries of all blocks within the input image, requiring decision processing. This also enables filtering to be performed in parallel for the horizontal boundaries of all blocks within the input image, requiring decision processing.

Fig. 11 is an explanatory diagram illustrating a processing sequence that can be used in the present embodiment. This example also assumes that the deblocking filter is supplied with an image of size 32 × 32 pixels. The input image comprises 4 macroblocks MB0-MB3 each of size 16 × 16 pixels.

In fig. 11, each of the dashed boxes represents processing to be performed in parallel. Although the example in fig. 10 requires 16 processing steps for the processing sequence, the example in fig. 11 groups the same number of processes into 4 processing steps. The first step of filtering all the vertical boundaries and all the horizontal boundaries of all the macroblocks MB0-MB3 in parallel requires a decision process J_V0,0～J_V3,3And J_H0,0～J_H3,3. The second step performs in parallel a filtering process F on the 16 vertical boundaries of all macroblocks MB0-MB3_V0,0～F_V3,3. Third step for all macroblocks MB0-MB3With horizontal boundaries, performing filtering in parallel requires a decision process F_H0,0～F_H3,3. The fourth step performs the filtering process F in parallel on the 16 horizontal boundaries of all macroblocks MB0-MB3_H0,0～F_H3,3. If the horizontal boundary is processed before the vertical boundary, the third and fourth steps may precede the first and second steps.

Fig. 11 provides an example in which the parallelism (parallel processing) is maximized by parallelizing the processing in all macroblocks in the image. However, without being limited to this example, the processing may be made parallel within some macroblocks, rather than all macroblocks in the image.

(2) Detailed structure of deblocking filter

Fig. 12 is a block diagram illustrating a detailed structure of the deblocking filter 24 that performs the above-described parallel processing according to the first embodiment. Referring to fig. 12, the deblocking filter 24 includes a vertical decision section 110, a horizontal decision section 114, a horizontal filtering section 130, a vertical filtering section 140, and a parallelization control section 150.

(2-1) vertical judging Member

The vertical decision section 110 includes a plurality of vertical boundary decision sections 112-1 to 112-n. Each vertical boundary decision section 112 is supplied with the image input to the deblocking filter 24, and decision information for deciding whether filtering is required.

The vertical boundary decision sections 112-1 to 112-n decide whether or not to apply a deblocking filter to a vertical boundary by using pixel values of an image input to the deblocking filter 24 across a plurality of macroblocks within the image. Each vertical boundary decision section 112 supplies information indicating the decision result regarding each vertical boundary, such as binary information indicating the decision result of application of a "1" forced deblocking filter, to the horizontal filtering section 130.

(2-2) horizontal Filter Unit

The horizontal filtering part 130 includes a plurality of horizontal filtering parts 132-1 to 132-n. Each horizontal filtering section 132 is supplied with the input image, and the determination result for each vertical boundary from the vertical determination section 110.

The decision result from the corresponding vertical boundary decision section 112 may indicate that a filter needs to be applied. In this case, each horizontal filtering section 132 applies the deblocking filter with respect to the vertical boundary to the elements on the left and right sides of the vertical boundary. Each horizontal filtering section 132 supplies the filtered pixel value with respect to the pixel to which the filter is applied and the pixel value of the input image with respect to the other pixels to the horizontal decision section 114 and the vertical filtering section 140.

(2-3) level determination section

The horizontal determination section 114 includes a plurality of horizontal boundary determination sections 116-1 to 116-n. Each horizontal boundary determining section 116 is supplied with pixel values after filtering by the horizontal filtering section 130, and determination information for determining whether filtering is necessary.

The horizontal boundary decision sections 116-1 to 116-n decide whether or not to apply a deblocking filter to the horizontal boundary by using pixel values after filtering by the horizontal filtering section 130 across a plurality of macroblocks within the image. Each horizontal boundary determining section 116 supplies information indicating the determination result regarding each horizontal boundary to the vertical filtering section 140.

(2-4) vertical Filter Member

The vertical filtering part 140 includes a plurality of vertical filtering parts 142-1 to 142-n. Each vertical filtering section 142 is supplied with the pixel value after filtering by the horizontal filtering section 130, and the determination result for each horizontal boundary from the horizontal determination section 114.

The decision result from the corresponding horizontal boundary decision section 116 may indicate that the filter needs to be applied. In this case, each vertical filtering section 142 applies the deblocking filter with respect to the horizontal boundary to the elements on the upper and lower sides of the horizontal boundary. Each vertical filtering section 142 supplies the filtered pixel value to the pixel to which the filter is applied, and supplies the pixel value supplied from the horizontal filtering section 130 to the other pixels. The output from each vertical filtering element 142 may constitute an output image from deblocking filter 24.

(3) More detailed structure of the decision section

Fig. 13 is a block diagram illustrating a detailed structure of each of the vertical boundary determining section 112 and the horizontal boundary determining section 116. Referring to fig. 13, each decision section includes a tap constituting section 121, a calculating section 122, a threshold value comparing section 123, a distortion evaluating section 124, and a filter decision section 125.

The tap constituting section 121 obtains reference pixel values from pixel values of two blocks adjacent across a boundary of interest in an input image, and constitutes a tap (a set of reference pixel values) that determines the determination condition B of the luminance component described above. For example, in a block of all 8 × 8 pixels in size, the vertical boundary may be of interest. In this case, the tap constituting section 121 constitutes a tap from the pixel values of the third and sixth rows belonging to the two blocks on the left and right sides. If the horizontal boundary is focused, the tap constituting section 121 may constitute a tap from the pixel values of the third column and the sixth column belonging to the two blocks on the upper and lower sides. The calculation section 122 assigns the tap constructed by the tap construction section 121 to the left side of the determination expression in the determination condition B, and calculates an edge value to be compared with the edge determination threshold β. The threshold value comparing section 123 compares the value calculated by the calculating section 122 and the edge decision threshold value β, and outputs the comparison result to the filter deciding section 125.

The distortion evaluation section 124 evaluates the above-described decision condition a for the luminance component using the mode information (MB mode), the transform coefficient information, and the motion vector information supplied as decision information. The distortion evaluation section 124 outputs the evaluation result to the filter decision section 125. The distortion evaluation section 124 evaluates only the decision condition a1 for the chrominance component based on the mode information.

The filtering decision section 125 decides whether or not to apply a deblocking filter to the boundary of interest based on the comparison result of the decision condition B supplied from the threshold value comparison section 123 and the evaluation result of the decision condition a supplied from the distortion evaluation section 124. The filter decision section 125 outputs information indicating the decision result.

(4) Parallelization control section

The parallelization control section 150 shown in fig. 12 controls the parallelism of the filtering-requiring determination process in the vertical determination section 110 and the horizontal determination section 114, and the parallelism of the filtering process in the horizontal filtering section 130 and the vertical filtering section 140.

For example, the parallelization control part 150 may control the parallelism of the processing of each block according to the input image size. More specifically, if the input image size is large, the parallelization control section 150 increases the degree of parallelism of the processing of each block. This can adaptively prevent a delay or a decrease in data rate caused by a processing amount increased corresponding to an image size. For example, the parallelization control part 150 may control the parallelism of the processing of each block according to a sequence parameter set, a picture parameter set, or a parameter included in a slice header. This enables flexibility in configuring the degree of parallelism as required by the user of the development device. For example, the parallelism may be configured according to restrictions on the installation environment, such as the number of processor cores, or the number of software threads.

This embodiment enables parallel processing between macroblocks. This means that any sequence of processing for individual blocks within the image has no effect on the final output result. Thus, the parallelization control section 150 can control the filtering-need determination processing sequence in the vertical determination section 110 and the horizontal determination section 114, and the filtering processing sequence in the horizontal filtering section 130 and the vertical filtering section 140 on a block basis.

More specifically, the parallelization control section 150 can control the filter processing sequence in accordance with the filter processing dependency relationship between macroblocks. For example, according to the prior art, processing dependencies between neighboring macroblocks around a slice boundary can delay parallel processing for each slice within an image. However, the parallelization control section 150 according to the present embodiment can perform the filtering process on the neighboring macroblocks around the slice boundary before other macroblocks.

For example, FIG. 14 illustrates 8 macroblocks MB10-MB13 and MB20-MB23 around a slice boundary. Macroblocks MB10-MB13 belong to slice SL 1. Macroblocks MB20-MB23 belong to slice SL 2. With respect to these macroblocks, the filtering process for the horizontal boundary on the macroblock MB20 in the slice SL2 depends on the filtering process for the vertical boundary on the macroblock MB12 in the slice SL 1. Similarly, the filtering process for the horizontal boundary on the macroblock MB21 in the slice SL2 depends on the filtering process for the vertical boundary on the macroblock MB13 in the slice SL 1.

Under these conditions, according to the example in fig. 15, the parallelization control section 150 performs the filtering process on the vertical boundaries of the macroblocks MB12 and MB13 in the filtering process on the slice SL1 in preference to the processing on the other boundaries. As a result, a large delay is prevented from occurring in the filtering process for the horizontal boundaries of the macroblocks MB20 and MB21 in the filtering process with respect to the slice SL 2. The example in fig. 16 initially performs the filtering process in parallel on the vertical boundaries of all the macroblocks included in the slice SL 1. Also in this case, in the filtering process for the horizontal boundaries of the macroblocks MB20 and MB21 in the slice SL2, no delay occurs.

[3-2. modified judgment Condition ]

As described above, similarly to the related art illustrated in fig. 4, each vertical boundary determining section 112 refers to pixels corresponding to the third and sixth lines in a block and determines whether filtering is required for the vertical boundary of each block. Likewise, each horizontal boundary determination section 116 refers to pixels corresponding to the third column and the sixth column in the block, and determines whether filtering is required for the horizontal boundary of each block. In this case, the configuration according to the present embodiment can be easily realized without changing the determination condition of the filtering necessity determination process provided by the existing apparatus.

Each of the vertical boundary determining section 112 and each of the horizontal boundary determining section 116 can make a determination using a determination condition different from that of the related art. For example, each vertical boundary determination section 112 may refer to pixels corresponding to 3 or more columns in the block. Each horizontal boundary determining part 116 may refer to pixels corresponding to 3 or more columns in the block. In addition, each of the vertical boundary determining section 112 and each of the horizontal boundary determining section 116 may use a determination condition expression different from the related art. Referring to fig. 17 to 19, 6 examples of the determination technique according to the present embodiment are explained below.

(1) First example

Fig. 17 is an explanatory diagram illustrating first and second examples of the determination technique. In the first and second examples, the filtering regarding the vertical boundary requires a decision process (in particular, the decision condition B is used for the decision of the luminance component) to refer to pixels of all lines L1-L8 from the first line to the eighth line in each block. Filtering with respect to the horizontal boundary requires that the decision process also refer to pixels of all columns from the first column to the eighth column in each block.

The first example may define the decision condition of the luminance component as follows.

Decision condition … for the Luma component (Luma) if both condition a and condition B are true, then a deblocking filter is applied.

Condition a:

(A1) the block Ba or Bb enters intra prediction mode;

(A2) block Ba or Bb has non-zero orthogonal transform coefficients; or

(A3) | MVAx-MVBx | ≧ 4 or | MVAy-MVBy | ≧ 4

Condition B:

iD₀＝|p₂₀-2p₁₀+p₀₀|+|q₂₀-2q₁₀+q₀₀|+|p₂₇-2p₁₇+p₀₇|+|q₂₇-2q₁₇+q₀₇|

iD₁＝|p₂₁-2p₁₁+p₀₁|+|q₂₁-2q₁₁+q₀₁|+|p₂₆-2p₁₆+p₀₆|+|q₂₆-2q₁₆+q₀₆|

iD₂＝|p₂₂-2p₁₂+p₀₂|+|q₂₂-2q₁₂+q₀₂|+|p₂₅-2p₁₅+p₀₅|+|q₂₅-2q₁₅+q₀₅|

iD₃＝|p₂₃-2p₁₃+p₀₃|+|q₂₃-2q₁₃+q₀₃|+|p₂₄-2p₁₄+p₀₄|+|q₂₄-2q₁₄+q₀₄|

iD_ave＝(iD₀+iD₁+iD₂+iD₃)>>2

under this condition, iD_ave<β

The determination conditions for the chrominance components may be the same as in the above-described prior art. A weighted average can be calculated to calculate the 4 decision parameters iD₀～iD₃Average value of (ii)_ave。

(2) Second example

The second example may define the determination condition B of the luminance component as follows.

Condition B:

iD₀＝|p₂₀-2p₁₀+p₀₀|+|q₂₀-2q₁₀+q₀₀|+|p₂₇-2p₁₇+p₀₇|+|q₂₇-2q₁₇+q₀₇|

iD₁＝|p₂₁-2p₁₁+p₀₁|+|q₂₁-2q₁₁+q₀₁|+|p₂₆-2p₁₆+p₀₆|+|q₂₆-2q₁₆+q₀₆|

iD₂＝|p₂₂-2p₁₂+p₀₂|+|q₂₂-2q₁₂+q₀₂|+|p₂₅-2p₁₅+p₀₅|+|q₂₅-2q₁₅+q₀₅|

iD₃＝|p₂₃-2p₁₃+p₀₃|+|q₂₃-2q₁₃+q₀₃|+|p₂₄-2p₁₄+p₀₄|+|q₂₄-2q₁₄+q₀₄|

according to this condition, iD₀<β，iD₁<β，iD₂<β and iD₃<β

Calculate 4 decision parameters iD₀～iD₃Is the same as that of the first example. The applicable conditions are not all 4 decision parameters iD₀～iD₃Are all less than the edge decision threshold β, but are 4 decision parameters iD₀～iD₃At least 3, 2, or 1 is less than the edge determination threshold β.

(3) Third example

Fig. 18 is an explanatory diagram illustrating third and fourth examples of the determination technique. In the third and fourth examples, the filtering regarding the vertical boundary requires a decision process (in particular, the decision condition B is used for the decision of the luminance component) referring to the pixels of the 4 lines L1, L3, L6, and L8 in each block. Filtering with respect to horizontal boundaries requires that the decision process also references the 4 columns of pixels in each block.

The third example may define the determination condition of the luminance component as follows.

Decision condition … for the Luma component (Luma) if both conditions a and B are true, then a deblocking filter is applied.

Condition a:

(A1) the block Ba or Bb enters an intra-frame prediction mode;

(A2) block Ba or Bb has non-zero orthogonal transform coefficients; or

(A3) | MVAx-MVBx | ≧ 4 or | MVAy-MVBy | ≧ 4

Condition B:

iD₀＝|p₂₀-2p₁₀+p₀₀|+|q₂₀-2q₁₀+q₀₀|+|p₂₇-2p₁₇+p₀₇|+|q₂₇-2q₁₇+q₀₇|

iD₂＝|p₂₂-2p₁₂+p₀₂|+|q₂₂-2q₁₂+q₀₂|+|p₂₅-2p₁₅+p₀₅|+|q₂₅-2q₁₅+q₀₅|

iD_ave＝(iD₀+iD₂)>>1

under this condition, iD_ave<β

The determination conditions for the chrominance components may be the same as in the above-described prior art. A weighted average can be calculated to calculate the 2 decision parameters iD₀And iD₂Average value of (ii)_ave。

(4) Fourth example

The fourth example may define the determination condition B of the luminance component as follows.

Condition B:

iD₀＝|p₂₀-2p₁₀+p₀₀|+|q₂₀-2q₁₀+q₀₀|+|p₂₇-2p₁₇+p₀₇|+|q₂₇-2q₁₇+q₀₇|

iD₂＝|p₂₂-2p₁₂+p₀₂|+|q₂₂-2q₁₂+q₀₂|+|p₂₅-2p₁₅+p₀₅|+|q₂₅-2q₁₅+q₀₅|

under this condition, iD₀<β and iD₂<β

Calculate 2 decision parameters iD₀And iD₂Equation (2) andthe three examples are identical. The applicable condition is not 2 decision parameters iD₀And iD₂Are all less than the edge decision threshold β, but 2 decision parameters iD₀And iD₂Is less than the edge determination threshold β.

Although the examples of the first, third, sixth, and eighth rows (or columns) L1, L3, L6, and L8 in the reference block during the determination are described, other combinations of rows and columns may be referred to.

(5) Fifth example

Fig. 19 is an explanatory diagram illustrating fifth and sixth examples of the determination technique. In the fifth and sixth examples, the filtering for the vertical boundary requires the decision processing to refer to the pixels of the 4 lines L1, L3, L5, and L7 in each block. Filtering for horizontal boundaries requires that the decision process also references the 4 columns of pixels in each block.

The fifth example defines the determination condition of the luminance component as follows.

Decision condition … for the Luma component (Luma) if both conditions a and B are true, then a deblocking filter is applied.

Condition a:

(A1) the block Ba or Bb enters an intra-frame prediction mode;

(A2) block Ba or Bb has non-zero orthogonal transform coefficients; or

(A3) | MVAx-MVBx | ≧ 4 or | MVAy-MVBy | ≧ 4

Condition B:

iD₀＝|p₂₀-2p₁₀+p₀₀|+|q₂₀-2q₁₀+q₀₀|+|p₂₆-2p₁₆+p₀₆|+|q₂₆-2q₁₆+q₀₆|

iD₂＝|p₂₂-2p₁₂+p₀₂|+|q₂₂-2q₁₂+q₀₂|+|p₂₄-2p₁₄+p₀₄|+|q₂₄-2q₁₄+q₀₄|

iD_ave＝(iD₀+iD₂)>>1

under this condition, iD_ave<β

The determination conditions for the chrominance components may be the same as in the above-described prior art. A weighted average can be calculated to calculate the 2 decision parameters iD₀And iD₂Average value of (ii)_ave。

(6) Sixth example

The sixth example defines the determination condition B of the luminance component as follows.

Condition B:

iD₀＝|p₂₀-2p₁₀+p₀₀|+|q₂₀-2q₁₀+q₀₀|+|p₂₆-2p₁₆+p₀₆|+|q₂₆-2q₁₆+q₀₆|

iD₂＝|p₂₂-2p₁₂+p₀₂|+|q₂₂-2q₁₂+q₀₂|+|p₂₄-2p₁₄+p₀₄|+|q₂₄-2q₁₄+q₀₄|

under this condition, iD₀<β and iD₂<β

Calculate 2 decision parameters iD₀And iD₂Is the same as that of the fifth example. The applicable condition is not 2 decision parameters iD₀And iD₂Are all less than the edge decision threshold β, but 2 decision parameters iD₀And iD₂Is less than the edge determination threshold β.

In general, increasing the number of rows and columns to be referred to in determination can improve the determination accuracy. Thus, referring to the first and second examples of 8 rows and 8 columns, the possibility of filtering blocks that are not originally the target of the deblocking filter to be applied, and the possibility of not filtering blocks that are originally the target of the deblocking filter to be applied, can be minimized. The result will be an improvement in the quality of the image to be encoded and decoded. On the other hand, reducing the number of rows and columns to be referred to in determination can reduce processing cost. Since there is a trade-off between image quality and processing cost, it is advantageous to adaptively select the number of rows and columns to be referred to in the determination according to the use of the device or the restriction on installation. It is also advantageous to adaptively select the combination of rows and columns to be parameterized.

As described in the first, third, and fifth examples, the average value iD of the determination parameter may be compared_aveAnd an edge decision threshold β so that the decision is properly made on a block basis without being unduly affected by parameter variations per row or column.

[3-3. treatment procedure ]

Referring to fig. 20 and 21, the flow of processing of the deblocking filter 24 is explained.

Fig. 20 is a flowchart illustrating the flow of processing of deblocking filter 24 according to the first embodiment. Referring to FIG. 20, the vertical boundary decision sections 112-1 to 112-n decide in parallel whether or not filtering is required for all vertical boundaries in a plurality of macroblocks included in an input image (step S110). The horizontal filtering sections 132-1 to 132-n apply the deblocking filter in parallel to all the vertical boundaries determined at step S110 to which the deblocking filter needs to be applied (step S120). The horizontal boundary decision sections 116-1 to 116-n decide whether or not filtering is required in parallel for all horizontal boundaries in a plurality of macroblocks included in the input image (step S130). The vertical filtering sections 142-1 to 142-n apply the deblocking filter in parallel to all the horizontal boundaries determined at step S130 to which the deblocking filter needs to be applied (step S140).

The above process flow is merely an example. For example, deblocking filter 24 may process two or more macroblocks in parallel. The sequence of processing may be changed.

Fig. 21 is a flowchart illustrating a processing flow of the filtering necessity determination process corresponding to steps S110 and S130 in fig. 20. Referring to fig. 21, the distortion evaluation section 124 evaluates the boundary for distortion based on the mode information, the transform coefficient information, and the motion vector information (step S150). If the evaluation indicates that there is distortion (decision condition a is true), the process proceeds to step S154. If the evaluation indicates that there is no distortion, the process proceeds to step S160 (step S152).

In step S154, the calculation section 122 calculates an edge value from the reference pixel tap constituted by the tap constituting section 121 (step S154). The threshold value comparing section 123 compares the calculated value with the edge determination threshold value β (step S156). If the edge value is not less than the threshold value β (determination condition B is true), the process advances to step S158. If the edge value is not less than the threshold value β, the process advances to step S160.

In step S158, the filter decision section 125 determines to apply a deblocking filter to the determined boundary (step S158). In step S140, the filter decision section 125 determines not to apply the deblocking filter to the determined boundary (step S160).

<4. second embodiment >

[4-1. deblocking Filter Structure example ]

An exemplary structure of the deblocking filter 24 according to the second embodiment is described below.

(1) Dependencies between new processes

According to the present embodiment, the deblocking filter 24 performs filtering requiring decision processing for the vertical boundary of each block without waiting for the application of the deblocking filter to other blocks in the macroblock to which the block belongs. The deblocking filter 24 performs filtering for the horizontal boundary of each block requiring decision processing without waiting for the application of the deblocking filter to other blocks in the macroblock to which the block belongs. This can reduce the dependency of processing within the macro block.

As described above, processing dependency is reduced, and it is possible to parallelize the filtering determination processing for the vertical boundary and the horizontal boundary in the macroblock.

Fig. 22 is an explanatory diagram illustrating a first example of processing sequences available at the deblocking filter 24. The example also assumes that the deblocking filter is supplied with an image of size 32 × 32 pixels. The input image includes 4 macroblocks MB0-MB3 each of 16 × 16 pixels in size.

In fig. 22, each of the dashed boxes represents processing to be performed in parallel. Although the example in fig. 10 requires 16 processing steps for the processing sequence, the example in fig. 22 groups the same number of processes into 12 processing steps. The first step, performing in parallel the filtering of 4 vertical and 4 horizontal boundaries of the macroblock MB0, requires a decision process J_V0,0～J_V0,3And J_H0,0～J_H0,3. The second step performs in parallel the filtering process F for the 4 vertical boundaries in the macroblock MB0_V0,0～F_V0,3. The third step performs filtering for 4 vertical boundaries and 4 horizontal boundaries of the macroblock MB1 in parallel, requiring decision processing J_V1,0～J_V1,3And J_H1,0～J_H1,3. The fourth step performs in parallel the filtering process F for the 4 vertical boundaries in the macroblock MB1_V1,0～F_V1,3. The fifth step performs in parallel the filtering process F for the 4 horizontal boundaries in the macroblock MB0_H0,0～F_H0,3. The sixth step performs filtering-need decision processing J for 4 vertical boundaries and 4 horizontal boundaries of the macroblock MB2 in parallel_V2,0～J_V2,3And J_H2,0～J_H2,3. The seventh step performs in parallel the filtering process F for the 4 vertical boundaries in the macroblock MB2_V2,0～F_V2,3. The eighth step performs the filtering process F for 4 horizontal boundaries in the macroblock MB1 in parallel_H1,0～F_H1,3. The ninth step performs filtering-need decision processing J for 4 vertical boundaries and 4 horizontal boundaries of the macroblock MB3 in parallel_V3,0～J_V3,3And J_H3,0～J_H3,3. The tenth step performs in parallel the filtering process F for the 4 vertical boundaries in the macroblock MB3_V3,0～F_V3,3. The eleventh step performs filter processing F for 4 horizontal boundaries in the macroblock MB2 in parallel_H2,0～F_H2,3. The twelfth step performs in parallel the filtering processing F for the 4 horizontal boundaries in the macroblock MB3_H3,0～F_H3,3. In this case, the deblocking filter 24 can perform processing on the entire input image with fewer processing steps than in the related art.

(2) Detailed structure of deblocking filter

Fig. 23 is a block diagram illustrating a detailed structure of the deblocking filter 24 that performs the above-described parallel processing according to the second embodiment. Referring to fig. 23, the deblocking filter 24 includes a vertical decision section 210, a horizontal decision section 214, a horizontal filtering section 130, a vertical filtering section 140, and a parallelization control section 150.

(2-1) vertical judging Member

The vertical decision section 210 includes a plurality of vertical boundary decision sections 212-1 to 212-n. Each vertical boundary decision section 212 decides whether or not to apply a deblocking filter to the vertical boundary of each block without waiting for the deblocking filter to be applied to other blocks in the macroblock to which the block belongs. Each vertical boundary decision section 212 supplies information indicating the decision result regarding each vertical boundary, such as binary information indicating the decision result of application of a "1" forced deblocking filter, to the horizontal filtering section 130.

(2-2) level determination section

The horizontal determination section 214 includes a plurality of horizontal boundary determination sections 216-1 to 216-n. Each horizontal boundary decision section 216 decides whether or not to apply a deblocking filter to the horizontal boundary of each block without waiting for the deblocking filter to be applied to other blocks in the macroblock to which the block belongs. Each horizontal boundary determining section 216 supplies information indicating the determination result regarding each horizontal boundary to the vertical filtering section 140.

In addition, according to the present embodiment, each vertical boundary determining section 212 and each horizontal boundary determining section 216 can determine whether or not filtering is required for each boundary by referring to pixels at respective positions similarly to the related art. In fact, each vertical boundary decision section 212 and each horizontal boundary decision section 216 may decide whether or not filtering is required for each boundary in accordance with the technique described in "3-2. decision condition modification".

[3-2. treatment procedure ]

Fig. 24 is a flowchart illustrating the flow of processing of the deblocking filter 24 according to the second embodiment. Referring to FIG. 24, the vertical boundary decision sections 212-1 to 212-n decide in parallel whether or not filtering is required for all vertical boundaries in a macroblock of interest contained in an input image (step S202). The horizontal boundary decision sections 214-1 to 214-n decide whether or not filtering is required in parallel for all horizontal boundaries included in the macroblock of interest (step S204). Steps S202 and S204 may also be performed in parallel.

The horizontal filtering sections 132-1 to 132-n apply deblocking filters in parallel to determine vertical boundaries in the macroblock of interest to which deblocking filters need to be applied in step S202 (step S210).

The process of step S220 is directed to the macroblock of interest in the latest cycle. The process at step S220 may be skipped for the first macroblock of interest. The vertical filtering sections 142-1 to 142-n apply deblocking filters in parallel to step S204 in the most recent loop, determining the horizontal boundaries to which deblocking filters need to be applied (step S220).

If there is a macroblock of interest in the input image that is not processed, the processing of steps S202-S220 is repeated for a macroblock of new interest (step S230).

If no unprocessed macroblocks of interest remain, then the vertical filtering components 142-1-142-n apply a deblocking filter in parallel to the last cycle of macroblocks of interest, determining the horizontal boundaries at which the deblocking filter needs to be applied (step S240).

The flow of the above-described processing is also merely an example. The parallelism and order of processing may be varied. In addition, the parallelization control part 150 can control the parallelism and the order of processing.

[4-3. examples of treatment of each LCU ]

As described above, the techniques according to the various embodiments described in this specification may be provided in the form of a process based on HEVC Coding Units (CUs). According to HEVC, the coding unit with the largest size is called the Largest Coding Unit (LCU), which may be chosen to be 64 × 64 pixels, for example. The minimum optional CU size is 8 × 8 pixels. Generally, an image is encoded and decoded corresponding to each LCU in a raster scan order, starting from an LCU at the upper left side of the image (or slice). An example of processing corresponding to an LCU in the deblocking filter 24 is explained below.

Fig. 25 is an explanatory diagram illustrating a processing sequence of each LCU according to the above-described second embodiment. This example assumes an LCU size of 16 × 16 pixels and a CU size of 8 × 8 pixels.

Referring to fig. 25, the first stage, shown at the upper left side of the figure, indicates that filtering for the LCU is complete up to the (n-1) th LCU. The target pixel for filtering of the vertical boundary is a shadow pixel. The target pixel for filtering of the horizontal boundary is a solid pixel.

In fig. 25, the second process on the upper right side and the third process on the lower left side target the nth LCU. Performing filtering in parallel requires a decision process for all vertical and horizontal boundaries belonging to the nth LCU before the second process. That is, performing filtering for the boundary of CUs belonging to the nth LCU requires a decision process without waiting for applying a deblocking filter to other CUs in the nth LCU. The second process performs the filtering process in parallel for the vertical boundary belonging to the nth LCU and determined to require application of a deblocking filter. The second process performs the filtering process in parallel for the horizontal boundaries belonging to the nth LCU and determined to require application of a deblocking filter.

The processing at the fourth stage on the lower right of fig. 25 targets the (n +1) th LCU. In the fourth stage, after the filtering necessity decision process is performed in parallel on the boundaries of all CUs belonging to the (n +1) th LCU, the filtering process is performed in parallel on the vertical boundaries determined to require application of the deblocking filter.

Although this example assumes an LCU size of 16 × 16 pixels, it may be set to 32 × 32 pixels or 64 × 64 pixels. The effect of shortening the processing time in accordance with the parallelization is further improved because increasing the size of the LCU to be selected also increases the number of vertical and horizontal boundaries belonging to one LCU.

Fig. 26 is a flowchart illustrating the process flow of deblocking filter 24 for each LCU.

Referring to FIG. 26, the vertical boundary decision sections 212-1 to 212-n decide in parallel whether or not filtering is required for all vertical boundaries in the LCU of interest contained in the input image (step S252). The horizontal boundary decision sections 216-1 to 216-n decide whether or not filtering is required in parallel for all horizontal boundaries included in the LCU of interest (step S254). Steps S252 and S254 may also be performed in parallel.

The horizontal filtering sections 132-1 to 132-n apply deblocking filters in parallel to the vertical boundaries in the LCU of interest for which application of a deblocking filter is determined to be necessary in step S252 (step S260).

The vertical filtering units 142-1 to 142-n apply deblocking filters in parallel to the horizontal boundaries in the LCU of interest for which application of a deblocking filter is determined to be necessary in step S254 (step S270).

If an unprocessed LCU remains in the input image, the processing at steps S252-S270 is repeated for the LCU of new interest (step S280). If there are no unprocessed LCUs, processing terminates.

<5. third embodiment >

[5-1. overview ]

The first and second embodiments change the existing processing sequence of the deblocking filter to improve the parallelism of the processing. In particular, the first embodiment reduces the processing dependency by expanding the range of pixel values of the input image supplied to the deblocking filter when the necessity of filtering is determined. The third embodiment described below enhances the inventive concept. The third embodiment further parallelizes the processing by filtering the input pixel values supplied to the deblocking filter in the filtering processing for the vertical boundary and the horizontal boundary.

Fig. 27 is an explanatory diagram illustrating an overview of the present embodiment. In the lower left side of fig. 27, a shape representing an input pixel (also referred to as a reconstructed pixel) before being processed by the deblocking filter is shown. The present embodiment allows the pixel values of the input deblocking filter to be referred from the filtering need decision process with respect to the vertical boundary and the horizontal boundary, and the filtering process with respect to the vertical boundary and the filtering process with respect to the horizontal boundary. Thus, this enables to solve the dependency relationship between the two filtering need determination processes and the dependency relationship between the two filtering processes.

The filtering process for the vertical boundary and the filtering process for the horizontal boundary may update the values of the repeated pixels. The solid pixels in fig. 27 illustrate the locations of pixels that may be repeated. The deblocking filter according to the present embodiment calculates one output pixel value from two filter outputs, as for a pixel repeatedly updated by two filters operating in parallel.

[5-2. deblocking Filter Structure example ]

Fig. 28 is a block diagram illustrating a detailed structure of deblocking filter 24 according to the third embodiment. Referring to fig. 28, deblocking filter 24 includes a line memory 308, a decision section 310, a horizontal filtering section 330, a vertical filtering section 340, a parallelization control section 150, and a calculation section 360.

Line memory 308 holds pixel values of the input image that are provided to deblocking filter 24. The filtering processes in the horizontal filtering section 330 and the vertical filtering section 340 do not update the pixel values held in the line memory 308. The filtering performed by the following section in the decision section 310 requires the decision process to refer to the pixel values held in the line memory 308. The arrangement includes another memory for a purpose different from the processing of the deblocking filter 24. This memory may be reused (shared) as row memory 308.

The decision section 310 includes vertical boundary decision sections 312-1 to 312-n, and horizontal boundary decision sections 314-1 to 314-n. The vertical boundary decision section 312 and the horizontal boundary decision section 314 are supplied with pixel values of the image input to the deblocking filter 24 stored in the line memory 308, and decision information for deciding whether filtering is necessary.

The vertical boundary decision section 312 decides whether or not to apply a deblocking filter to each vertical boundary using the pixel values input to the deblocking filter 24. The vertical boundary decision section 312 outputs information indicating the decision result regarding each vertical boundary to the horizontal filtering section 330.

The horizontal boundary decision section 314 also uses the pixel values input to the deblocking filter 24 to decide whether or not to apply a deblocking filter to each horizontal boundary. The horizontal boundary determining section 314 performs the determination process in parallel with the determination process performed by the vertical boundary determining section 312. The horizontal boundary decision section 314 outputs information indicating the decision result regarding each horizontal boundary to the vertical filtering section 340.

In addition, according to the present embodiment, each vertical boundary decision section 312 and each horizontal boundary decision section 314 can decide whether or not filtering is required for each boundary by referring to pixels at respective positions similarly to the related art. Each vertical boundary decision section 312 and each horizontal boundary decision section 314 may instead decide for each boundary whether filtering is required or not in accordance with the technique described in "3-2. decision condition variation".

The horizontal filtering part 330 includes horizontal filtering parts 332-1 to 332-n. The horizontal filtering section 332 is supplied with the input image values from the line memory 308, and the determination result for each vertical boundary from the determination section 310.

If the decision result from the vertical boundary decision section 312 indicates that the filter needs to be applied, the horizontal filtering section 332 applies a deblocking filter with respect to the vertical boundary to the left and right pixels around the corresponding vertical boundary. The horizontal filtering section 332 outputs the filtered pixel value with respect to the filtered pixel, or the input pixel value with respect to the other pixels, to the calculating section 360.

The vertical filtering part 340 includes vertical filtering parts 342-1 to 342-n. The vertical filtering section 342 is supplied with the input pixel values from the line memory 308, and the determination result for each horizontal boundary from the determination section 310.

If the decision result from the horizontal boundary decision section 314 indicates that the filter needs to be applied, the vertical filtering section 342 applies a deblocking filter with respect to the horizontal boundary to the upper and lower pixels around the corresponding horizontal boundary. The filtering process of the vertical filtering units 342-1 to 342-n is performed in parallel with the filtering process of the horizontal filtering units 332-1 to 332-n. The vertical filtering section 342 outputs the filtered pixel value in the case of the filtered pixel, or the input pixel value in the case of the other pixel, to the calculation section 360.

The calculation section 360 is supplied with the output pixel value from the horizontal filtering section 330 and the output pixel value from the vertical filtering section 340 in parallel. Further, the calculation section 360 is supplied with the determination results from the vertical boundary determination section 312 and the horizontal boundary determination section 314. According to the determination result, the calculation section 360 calculates the output pixel values of the pixels filtered from the horizontal filtering section 330 and the vertical filtering section 340 based on the filter outputs from the horizontal filtering section 330 and the vertical filtering section 340.

According to the present embodiment, for example, for the repeatedly filtered pixels, the calculation section 360 calculates the average value of the two filter outputs. The calculation section 360 may calculate a simple average of the two filter outputs. The calculation portion 360 may instead calculate a weighted average of the two filter outputs. For example, the calculation section 360 may determine the weight for the weighted average of the pixels in accordance with the distance of each pixel to the vertical boundary and to the horizontal boundary.

FIG. 29 is an explanatory diagram illustrating determination of the weight for the weighted average calculated with the calculating section 360. Fig. 29 shows the pixel of interest P corresponding to one of the repetition positions illustrated in fig. 27 in black_Z. At the pixel of interest P_ZAnd nearest vertical boundary V_ZIs present at a distance D from_VCorresponding to 3 pixels. At the pixel of interest P_ZAnd nearest horizontal boundary H_ZIs present at a distance D from_HCorresponding to 2 pixels. Distance D_HLess than distance D_V. In this case, the calculation section 360 may apply the self-application to the horizontal boundary H_ZIs set to be greater than the weight from the output applied to the vertical boundary V_ZWeight of the output of the deblocking filter of (1). The example in fig. 29 assumes that the vertical boundary V is concerned_ZIs output V of the filter_outAnd with respect to the horizontal boundary H_ZIs output H of the filter_outThe ratio of the ratio is 2: 3.

As can be seen from fig. 29, the weighted average of the two filter outputs is calculated so that an output pixel value similar to the case of applying 1 two-dimensional filter having a filter tap in the horizontal direction and a filter tap in the vertical direction can be provided to each pixel of interest. Parallelizing the filtering process for the vertical and horizontal boundaries may also appropriately reduce the block distortion occurring at the vertical and horizontal boundaries. As another embodiment, deblocking filter 24 may comprise 1 two-dimensional filter that simultaneously calculates horizontal filtering, vertical filtering, and a weighted average. In this case, however, the installation is extremely complicated because the filter coefficients need to be changed differently corresponding to the pixels. On the other hand, the third embodiment performs 2 one-dimensional filtering in parallel, and then calculates a weighted average. This can easily provide substantially the same processing as a two-dimensional filter while ensuring the function of an existing deblocking filter.

FIG. 30 is an explanatory diagram illustrating an example of the weight of the weighted average calculated according to the example in FIG. 29 FIG. 30 shows 36 pixels (6 × 6) around the intersection of the vertical boundary and the horizontal boundary, which correspond to the repetition positions described above_outWeight of and filter output H_outIs 1:1(2:2 or 3: 3). For pixels closer to the vertical boundary, the filter output V_outIs weighted more than the filter output H_outThe weight of (c). For example, the pixel P₁Is V_out:H_out3: 1. For pixels closer to the horizontal boundary, the filter output V_outIs less than the filter output H_outThe weight of (c). For example, the pixel P₂Is V_out:H_out＝1:3。

By changing the weight of the weighted average value according to the distance between each pixel and the boundary, block distortion can be more effectively suppressed, and image quality can be improved.

The weights described above are merely examples. For example, the calculation section 360 may determine the weight of the weighted average of the pixels in accordance with the edge intensities of the vertical and horizontal boundaries corresponding to each pixel, instead of or in addition to the distance between each pixel and the boundary. The edge strength may be expressed by a parameter such as an edge value calculated from the calculation section 122 as shown in fig. 13. In this case, the weight of the filter output with respect to the boundary having the stronger edge may be set to be greater than the weight of the filter output with respect to the boundary having the weaker edge. Varying the weight of the weighted average, in terms of edge strength, can adaptively improve the effect of the deblocking filter on boundaries that significantly contribute to block distortion.

The calculating part 360 selects an output from the block actually filtered in terms of the pixel filtered by one of the horizontal filtering part 330 and the vertical filtering part 340. As for the pixels not filtered by the horizontal filtering section 330 or the vertical filtering section 340, the calculating section 360 directly outputs the input pixel values to be output to the deblocking filter 24. The table in fig. 31 lists output pixel values from the calculation section 360 according to the result of the determination of whether filtering is required.

[5-3. example of treatment sequence ]

Two examples of processing sequences that can be utilized by the deblocking filter 24 according to the present embodiment are described below. This example also assumes that the deblocking filter is supplied with an image of size 32 × 32 pixels. The input image includes 4 macroblocks MB0-MB3 each of 16 × 16 pixels in size.

(1) First example

For comparison, fig. 32 illustrates a processing sequence when there is still a dependency relationship between the filter processing for the vertical boundary and the filter processing for the horizontal boundary. In fig. 32, the first step of performing filtering in parallel for all vertical boundaries and all horizontal boundaries of all 4 macroblocks MB0-MB3 requires a decision process J_V0,0～J_V3,3And J_H0,0～J_H3,3. The second step performs a filtering process F of 16 vertical boundaries of 4 macroblocks MB0-MB3_V0,0～F_V3,3. The third step performs a filtering process F of 16 horizontal boundaries of 4 macroblocks MB0-MB3_H0,0～F_H3,3. The fourth step stores the pixel values after the filtering process for the horizontal boundary in a memory for output from the deblocking filter 24.

Fig. 33 illustrates a first example of the processing sequence provided by the present embodiment. In fig. 33, the first step of performing filtering for all vertical boundaries and all horizontal boundaries of 4 macroblocks MB0-MB3 in parallel requires a decision process J_V0,0～J_V3,3And J_H0,0～J_H3,3. The second step performs in parallel the filtering process F for all the vertical boundaries and all the horizontal boundaries of the 4 macroblocks MB0-MB3_V0,0～F_V3,3And F_H0,0～F_H3,3. In practice, the second step filters only the boundaries that determine the need for filtering. The third step stores the pixel values in memory for output from deblocking filter 24. As for the pixels filtered by the horizontal filtering section 330 and the vertical filtering section 340, a weighted average of the outputs of the two filters can be calculated as an output pixel value.

(2) Second example

Although the first example maximizes the parallelism, the deblocking filter 24 according to the second example is also capable of performing processing for each macroblock.

For comparison, fig. 34 illustrates a processing sequence for each macroblock when there is still a dependency between the filtering processing for the vertical boundary and the filtering processing for the horizontal boundary. The processing sequence in fig. 34 is substantially the same as that in fig. 22 according to the first embodiment. Fig. 36 explicitly shows 4 processing steps (6 th, 10 th, 14 th, and 16 th) that save the pixel values in the memory for output, and are omitted from fig. 22 for simplicity. The 16 process steps including these 4 process steps constitute the process in fig. 34.

Fig. 35 illustrates a second example of the processing sequence provided by the present embodiment. In fig. 35, the first step performs filtering-need determination processing J for 4 vertical boundaries and 4 horizontal boundaries of a macroblock MB0 in parallel_V0,0～J_V0,3And J_H0,0～J_H0,3. The second step performs in parallel a filtering process F for 4 vertical boundaries and 4 horizontal boundaries of the macroblock MB0_V0,0～F_V0,3And F_H0,0～F_H0,3. The third step stores the pixel values of macroblock MB0 in memory for output from deblocking filter 24. For pixels that are repeatedly filtered by both filters, a weighted average of the outputs of both filters may be calculated as the output pixel value. The fourth step to the sixth step similarly process the macroblock MB 1. The seventh step to the ninth step similarly process the macroblock MB 2. The tenth step to the twelfth step similarly process the macroblock MB 3. The process in fig. 35 includes 12 fewer process steps than the steps of the process in fig. 34.

The third embodiment eliminates the dependency relationship between the filter processing for the vertical boundary and the filter processing for the horizontal boundary. The processing of the deblocking filter 24 can be performed with fewer processing steps than in the first and second embodiments. One of the advantages of allowing the filtering process to refer only to the pixel input to the deblocking filter is that the arbitrary structure of the filter taps does not result in any dependency between the filtering process for vertical boundaries and the filtering process for horizontal boundaries. The third embodiment can improve the image quality by constituting the filter tap with more pixels than in the prior art. For example, the prior art uses 3-pixel filter taps for each side of the respective boundary, as described above with reference to fig. 7. Even if filter taps of 5 pixels or more are used at each boundary, the present embodiment does not cause any dependency between processes. By further reducing the block size as a processing unit of the deblocking filter, no dependency occurs even between processes.

In addition, in the third embodiment and the first and second embodiments, the parallelization control section 150 can control the parallelism and the order of processing in the deblocking filter 24.

[5-4. treatment procedure ]

Fig. 36 is a flowchart illustrating an example of the processing flow of the deblocking filter according to the third embodiment. Fig. 37 is a flowchart illustrating the flow of the pixel value calculation process shown in fig. 36.

Referring to FIG. 36, the vertical boundary decision sections 312-1 to 312-n decide whether or not filtering is required in parallel for all vertical boundaries within an input image or macroblock (step S302). The horizontal boundary decision sections 314-1 to 314-n decide whether or not filtering is required in parallel for all horizontal boundaries within the input image or macroblock (step S304). Steps S302 and S304 may also be performed in parallel.

The horizontal filtering components 332-1 to 332-n apply the deblocking filter in parallel to all vertical boundaries where the deblocking filter needs to be applied at step S302 (step S306). The vertical filtering components 342-1 to 342-n apply deblocking filters in parallel to all horizontal boundaries where deblocking filters need to be applied at step S304 (step S308). Steps S306 and S308 may also be performed in parallel.

The calculating section 360 then performs a pixel value calculating process as shown in fig. 37 (step S310). Referring to fig. 37, the processing from step S314 to step S326 is then looped for each pixel to be processed (step S312).

In step S314, the calculation portion 360 determines whether or not the two filters for the vertical boundary and the horizontal boundary have filtered the pixel of interest (step S314). If both filters have filtered the pixel of interest, the process proceeds to step S322. If the two filters have not filtered the pixel of interest, the process proceeds to step S316.

In step S316, the calculation portion 360 determines whether one of the two filters for the vertical boundary and the horizontal boundary has filtered the pixel of interest (step S316). If one of the two filters has filtered the pixel of interest, the process proceeds to step S320. If neither of the two filters has filtered the pixel of interest, the process proceeds to step S318.

In step S318, the calculation section 360 obtains an input pixel value to the deblocking filter 24 (step S318). In step S320, the calculation portion 360 obtains a filter output from a filter that actually filters the pixel of interest (step S320).

In step S322, the calculation section 360 determines a weight value for calculating a weighted average of filter outputs from two filters related to the pixel of interest in accordance with the distance from the pixel of interest to the vertical boundary and the horizontal boundary, or the edge intensity corresponding to the vertical boundary and the horizontal boundary of the pixel of interest (step S322). The calculating section 360 calculates a weighted average of the filter outputs from the two filters using the determined weights (step S324).

The calculation section 360 saves the pixel value of the pixel of interest in the memory while obtaining the pixel value at step S318 or S320 or calculating the pixel value at step S324 (step S326). When all the pixels to be processed have been processed, the processing sequence as shown in fig. 36 and 37 is terminated.

<6. applications to various codecs >

The techniques according to the present disclosure are applicable to various codecs related to image encoding and decoding. An example of applying the technology according to the present disclosure to a multi-view codec and an extensible codec is described below.

[6-1. multiview codec ]

The multi-view codec is an image coding scheme for coding and decoding multi-view video. Fig. 38 is an explanatory diagram illustrating a multi-view codec. Fig. 38 illustrates a frame sequence of 3 views taken at 3 viewpoints. Each view is provided with a view ID (view _ ID). One of the views is designated as the base view. Views other than the base view are referred to as non-base views. The example in fig. 38 represents a basic view with a view ID of "0", and two non-basic views with a view ID of "1" or "2". Encoding multi-view image data can compress the data size of an encoded stream as a whole by encoding frames of a non-base view according to encoding information on the frames of the base view.

During the encoding process and the decoding process according to the multi-view codec described above, a deblocking filter may be applied to each view. Applying a deblocking filter to each view may, in accordance with the techniques of this disclosure, parallelize horizontal filtering and vertical filtering in units of processing that includes multiple CUs for each view. The above processing units may represent several CUs, LCUs, or images. Parameters that control parallel processing, such as those described in paragraph 2 of the previous "(4) parallelization control section", may be set for each view. The parameters set for the base view may be reused for the non-base views.

Horizontal filtering and vertical filtering may be parallelized across multiple views. The multiple views may share parameters that control parallel processing (such as the parameters described in paragraph 2 of the previous "(4) parallelization control section"). It is advantageous to additionally specify a flag indicating whether the parameter is shared by a plurality of views.

Fig. 39 is an explanatory diagram illustrating an image encoding process applied to the above-described multi-view codec. Fig. 39 shows an example of the structure of the multi-view encoding apparatus 710. The multi-view encoding apparatus 710 includes a first encoding part 720, a second encoding part 730, and a multiplexing part 740.

The first encoding section 720 encodes the base view image, generating an encoded stream of the base view. The second encoding part 730 encodes the non-base view image, resulting in an encoded stream of the non-base view. The multiplexing section 740 multiplexes the encoded stream of the base view generated from the first encoding section 720 and one or more encoded streams of non-base views generated from the second encoding section 730, thereby generating a multiplexed stream of multiple views.

The first encoding section 720 and the second encoding section 730 illustrated in fig. 39 are constituted similarly to the image encoding apparatus 10 according to the above-described embodiment. Applying a deblocking filter to a view enables parallel horizontal filtering and vertical filtering in units of processing that includes multiple CUs. The parameters controlling these processes may be inserted in a header area of the encoded stream of each view or in a common header area in the multiplexed stream.

Fig. 40 is an explanatory diagram illustrating an image decoding process applied to the multi-view codec described above. Fig. 40 shows an example of the structure of the multi-view decoding apparatus 760. The multi-view decoding apparatus 760 includes a demultiplexing part 770, a first decoding part 780, and a second decoding part 790.

The demultiplexing part 770 demultiplexes the multi-view multiplexed stream into a base view encoded stream and one or more non-base view encoded streams. The first decoding part 780 decodes a base view image from the encoded stream of the base view. The second decoding part 790 decodes the non-base view image from the encoded stream of the non-base view.

The first decoding part 780 and the second decoding part 790 illustrated in fig. 40 are constituted similarly to the image decoding apparatus 60 according to the above-described embodiment. Applying a deblocking filter to a view enables parallel horizontal filtering and parallel vertical filtering in units of processing that includes multiple CUs. The parameters controlling these processes may be obtained from a header area of the encoded stream of each view or from a common header area of the multiplexed stream.

[6-2. Expandable codec ]

Scalable codecs are image coding schemes that provide hierarchical coding. Fig. 41 is an explanatory diagram illustrating an extensible codec. Fig. 41 illustrates a frame sequence of 3 layers different in spatial resolution, temporal resolution, or image quality. Each layer has a layer ID (layer _ ID). These layers include the base layer with the lowest resolution (or image quality). The layers other than the base layer are referred to as enhancement layers. The example in fig. 41 represents a base layer having a layer ID of "0", and two enhancement layers having layer IDs of "1" or "2". The multi-layer image data may be encoded by compressing the data size of the encoded stream as a whole by encoding frames of the enhanced layer according to encoding information about frames of the base layer.

During the encoding process and the decoding process according to the above-described scalable codec, a deblocking filter may be applied to each layer. Applying a deblocking filter to each layer may, in accordance with the techniques of this disclosure, parallelize horizontal and vertical filtering in units of processing that includes multiple CUs for each view. The above processing units may represent several CUs, LCUs, or images. Parameters to control parallel processing, such as those described in paragraph 2 of the foregoing "(4) parallelization control section", may be set for each layer. The parameters set for the base layer view can be reused for the enhancement layer.

Horizontal filtering and vertical filtering may be made in parallel within multiple layers. The multiple layers may share parameters that control parallel processing, such as the parameters described in paragraph 2 of the previous "(4) parallelization control section". It is advantageous to additionally specify a flag indicating whether the multiple layers share the parameter.

Fig. 42 is an explanatory diagram illustrating an image encoding process applied to the above-described scalable codec. Fig. 42 shows an example of the structure of the scalable encoding device 810. The scalable encoding apparatus 810 includes a first encoding part 820, a second encoding part 830, and a multiplexing part 840.

The first encoding section 820 encodes the base layer image, generating an encoded stream of the base layer. The second encoding part 830 encodes the enhanced layer image, generating an encoded stream of the enhanced layer. The multiplexing section 840 multiplexes the encoded stream of the base layer generated from the first encoding section 820 and one or more encoded streams of the enhancement layer generated from the second encoding section 830, thereby generating a multi-layered multiplexed stream.

The first encoding section 820 and the second encoding section 830 illustrated in fig. 42 are constituted similarly to the image encoding apparatus 10 of the above-described embodiment. Applying a deblocking filter to each layer enables parallel horizontal filtering and parallel vertical filtering in units of processing containing a plurality of CUs. The parameters controlling these processes may be inserted in a header area of the encoded stream of each layer or in a common header area in the multiplexed stream.

Fig. 43 is an explanatory diagram illustrating an image decoding process applied to the above-described scalable codec. Fig. 43 shows an example of the structure of the scalable decoding apparatus 860. The scalable decoding apparatus 860 includes a demultiplexing part 870, a first decoding part 880, and a second decoding part 890.

The demultiplexing section 870 demultiplexes the multi-layered multiplexed stream into a coded stream of the base layer and coded streams of one or more enhancement layers. The first decoding section 880 decodes the base layer image from the encoded stream of the base layer. The second decoding section 890 decodes the enhancement layer picture from the encoded stream of the enhancement layer.

The first decoding portion 880 and the second decoding portion 890 illustrated in fig. 43 are constituted similarly to the image decoding apparatus 60 according to the above-described embodiment. Applying a deblocking filter to each layer enables parallel horizontal filtering and parallel vertical filtering in units of processing containing a plurality of CUs. The parameters controlling these processes may be obtained from a header area of the encoded stream of each layer or from a common header area of the multiplexed stream.

<7. exemplary applications >

The image encoding device 10 and the image decoding device 60 according to the above-described embodiments are applicable to various electric appliances such as a transmitter and a receiver for satellite broadcasting, cable broadcasting such as cable television, distribution over the internet, distribution to terminals by cellular communication, and the like, a recording device that records an image in a medium such as an optical disk, a magnetic disk, or a flash memory, a reproducing device that reproduces an image from such a storage medium, and the like. The following describes 4 exemplary applications.

[7-1. first exemplary application ]

Fig. 44 is a block diagram showing an example of a schematic configuration of a television to which the above-described embodiment is applied. The television set 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing section 905, a display section 906, an audio signal processing section 907, a speaker 908, an external interface 909, a control section 910, a user interface 911, and a bus 912.

The tuner 902 extracts a signal of a desired channel from a broadcast signal received via the antenna 901, and demodulates the extracted signal. Subsequently, the tuner 902 outputs the coded bit stream obtained by demodulation to the demultiplexer 903. That is, the tuner 902 functions as a transmission device of the television set 900 that receives the encoded stream in which the image is encoded, of the television set 900.

The demultiplexer 903 separates a video stream and an audio stream of a program to be viewed from the encoded bit stream, and outputs each of the separated streams to the decoder 904. In addition, the demultiplexer 903 extracts auxiliary data such as an EPG (electronic program guide) from the encoded bitstream and supplies the extracted data to the control section 910. In addition, the demultiplexer 903 may perform descrambling in the case where the encoded bit stream is scrambled.

The decoder 904 decodes the video stream and the audio stream input from the demultiplexer 903. Subsequently, the decoder 904 outputs the video data generated by the decoding process to the video signal processing section 905. In addition, the decoder 904 outputs the audio data generated by the decoding process to the audio signal processing section 907.

The video signal processing section 905 reproduces the video data input from the decoder 904, causing the display section 906 to display the video. The video signal processing section 905 can also cause the display section 906 to display an application screen supplied through a network. Further, the video signal processing section 905 can perform additional processing such as noise removal on the video data in accordance with the setting. Further, the video signal processing section 905 may generate an image of GUI (graphical user interface) such as a menu, a button, a cursor, or the like, and superimpose the generated image on the output image.

The display section 906 is driven by a drive signal supplied from the video signal processing section 905 to display a video or an image on a video screen of a display device (e.g., a liquid crystal display, a plasma display, an OLED, or the like).

The audio signal processing section 907 performs reproduction processing such as D/a conversion and amplification on the audio data input from the decoder 904, and outputs audio from the speaker 908. In addition, the audio signal processing section 907 may perform additional processing such as noise removal on the audio data.

The external interface 909 is an interface between the television 900 and an external device or a network. For example, a video stream and an audio stream received by the external interface 909 may be decoded by the decoder 904. That is, the external interface 909 also serves as transmission means of the television set 900 for receiving an encoded stream in which an image is encoded.

The control section 910 includes a processor such as a CPU (central processing unit), and a memory such as a RAM (random access memory), a ROM (read only memory). The memory stores programs to be executed by the CPU, program data, EPG data, data obtained through a network, and the like. When the television set 900 is started, the CPU reads and executes the program stored in the CPU. The CPU controls the operation of the television 900 by executing a program in accordance with an operation signal input from the user interface 911.

The user interface 911 is connected to the control section 910. The user interface 911 includes buttons and switches used by the user to operate the television 900, and a receiving section for remote control signals. The user interface 911 detects the operation of the user by these constituent elements, generates an operation signal, and outputs the generated operation signal to the control section 910.

A bus 912 interconnects the tuner 902, the demultiplexer 903, the decoder 904, the video signal processing section 905, the audio signal processing section 907, the external interface 909, and the control section 910.

In the television set 900 configured in this manner, the decoder 904 has the function of the image decoding apparatus 60 according to the above-described embodiment. Thus, also in the case of image decoding in the television 900, the parallelism of the deblocking filter process can be enhanced, thereby ensuring high-speed processing.

[7-2. second exemplary application ]

Fig. 45 is a block diagram showing an example of the schematic configuration of the mobile phone employing the above-described embodiment. The mobile phone 920 includes an antenna 921, a communication portion 922, an audio codec 923, a speaker 924, a microphone 925, a camera portion 926, an image processing portion 927, a demultiplexing portion 928, a recording/reproducing portion 929, a display portion 930, a control portion 931, an operation portion 932, and a bus 933.

The antenna 921 is connected to the communication section 922. A speaker 924 and a microphone 925 are connected to the audio codec 923. The operation section 932 is connected to the control section 931. A bus 933 interconnects the communication portion 922, the audio codec 923, the camera portion 926, the image processing portion 927, the demultiplexing portion 928, the recording/reproducing portion 929, the display portion 930, and the control portion 931.

The mobile phone 920 performs operations such as transmission/reception of audio signals, transmission/reception of e-mail or image data, image capturing, recording of data, and the like in accordance with modes including an audio communication mode, a data communication mode, a camera shooting mode, and a video phone mode.

In the audio communication mode, an analog audio signal generated by the microphone 925 is provided to the audio codec 923. The audio codec 923 converts the analog audio signal into audio data, and a/D converts and compresses the converted audio data. Subsequently, the audio codec 923 outputs the compressed audio data to the communication section 922. The communication section 922 encodes and modulates audio data, thereby generating a transmission signal. Subsequently, the communication section 922 transmits the generated transmission signal to a base station (not shown) through the antenna 921. In addition, the communication section 922 amplifies a wireless signal received via the antenna 921, converts the frequency of the wireless signal, and thereby obtains a received signal. Subsequently, the communication section 922 demodulates and decodes the received signal, generates audio data, and outputs the generated audio data to the audio codec 923. The audio codec 923 expands and D/a-converts the audio data, thereby generating an analog audio signal. Subsequently, the audio codec 923 supplies the generated audio signal to the speaker 924, thereby causing audio to be output.

In addition, in the data communication mode, the control section 931 generates text data constituting an electronic mail in accordance with an operation by the user via the operation section 932. Further, the control part 931 causes text to be displayed on the display part 930. Further, the control section 931 generates email data in accordance with a transmission instruction by the operation section 932, and outputs the generated email data to the communication section 922. Subsequently, the communication section 922 encodes and modulates the email data, thereby generating a transmission signal. Then, the communication section 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication section 922 amplifies a wireless signal received via the antenna 921, converts the frequency of the wireless signal, and thereby obtains a received signal. Subsequently, the communication section 922 demodulates and decodes the received signal, restores the email, and outputs the restored email data to the control section 931. The control portion 931 causes the display portion 930 to display the contents of the electronic mail and, in addition, causes the electronic mail data to be saved in the storage medium of the recording/reproducing portion 929.

The recording/reproducing portion 929 includes any readable and writable storage medium. For example, a storage medium is a built-in storage medium such as RAM, flash memory, or the like, or an externally mounted storage medium such as a hard disk, magnetic disk, magneto-optical disk, USB memory, memory card, or the like.

Further, in the image capturing mode, the camera section 926 captures an image of a subject, generates image data, and outputs the generated image data to the image processing section 927. The image processing section 927 encodes the image data input from the camera section 926 such that the encoded stream is stored in the storage medium of the recording/reproducing section 929.

Further, in the video phone mode, the demultiplexing section 928 multiplexes the video stream encoded by the image processing section 927 and the audio stream input from the audio codec 923, and outputs the multiplexed stream to the communication section 922. The communication section 922 encodes and modulates the stream, generating a transmission signal. Subsequently, the communication section 922 transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication section 922 amplifies a wireless signal received via the antenna 921, converts the frequency of the wireless signal, and thereby obtains a received signal. These transmission and reception signals may comprise coded bit streams. Subsequently, the communication section 922 demodulates and decodes the received signal, restores the stream, and outputs the restored stream to the demultiplexing section 928. The demultiplexing section 928 separates a video stream and an audio stream from the input stream, outputs the video stream to the image processing section 927, and outputs the audio stream to the audio codec 923. The image processing portion 927 decodes the video stream, thereby generating video data. The video data is supplied to the display section 930 so that a series of images are displayed using the display section 930. The audio codec 923 expands and D/a-converts the audio stream, generating an analog audio signal. Subsequently, the audio codec 923 supplies the generated audio signal to the speaker 924, thereby causing audio to be output.

In the mobile phone 920 configured in this way, the image processing portion 927 has the functions of the image encoding apparatus 10 and the image decoding apparatus 60 according to the above-described embodiments. Therefore, even when an image is encoded and decoded in mobile phone 920, the degree of parallelism of the deblocking filter process can be enhanced, and high-speed processing can be ensured.

[7-3. third exemplary application ]

Fig. 46 is a block diagram showing an example of the schematic configuration of a recording/reproducing apparatus employing the above-described embodiment. The recording/reproducing apparatus 940 encodes audio data and video data of the received broadcast program and records in a recording medium. The recording/reproducing apparatus 940 may also encode audio data and video data obtained from another apparatus and record in a recording medium. Further, the recording/reproducing apparatus 940 reproduces data recorded in the recording medium using a monitor or a speaker according to an instruction of a user. At this time, the recording/reproducing apparatus 940 decodes the audio data and the video data.

The recording/reproducing apparatus 940 includes a tuner 941, an external interface 942, an encoder 943, an HDD (hard disk drive) 944, an optical disk drive 945, a selector 946, a decoder 947, an OSD (on screen display) 948, a control portion 949, and a user interface 950.

The tuner 941 extracts a signal of a desired channel from a broadcast signal received via an antenna (not shown), and demodulates the extracted signal. Subsequently, the tuner 941 outputs the encoded bit stream obtained by demodulation to the selector 946. That is, the tuner 941 serves as a transmission device of the recording/reproducing apparatus 940.

The external interface 942 is an interface for connecting the recording/reproducing apparatus 940 and an external apparatus or network. For example, the external interface 942 may be an IEEE 1394 interface, a network interface, a USB interface, a flash memory interface, or the like. For example, video data and audio data received using the external interface 942 are input to the encoder 943. That is, the external interface 942 serves as a transmission means of the recording/reproducing apparatus 940.

In the case where the video data and the audio data input from the external interface 942 are not encoded, the encoder 943 encodes the video data and the audio data. Subsequently, the encoder 943 outputs the encoded bit stream to the selector 946.

The HDD 944 records an encoded bitstream, which is compressed content data of video or audio, various programs, and other various items of data, in an internal hard disk. In addition, when reproducing video or audio, the HDD 944 reads out these data from the hard disk.

The optical disc drive 945 records data in a mounted recording medium or reads data from the mounted recording medium. The recording medium mounted in the optical disk drive 945 may be a DVD optical disk (DVD-video, DVD-RAM, DVD-R, DVD-RW, DVD + RW, etc.), a blu-ray (registered trademark) optical disk, or the like.

When recording video or audio, the selector 946 selects an encoded bitstream input from the tuner 941 or the encoder 943 and outputs the selected encoded bitstream to the HDD 944 or the optical disk drive 945. In addition, when reproducing video or audio, the selector 946 outputs the encoded bit stream input from the HDD 944 or the optical disk drive 945 to the decoder 947.

The decoder 947 decodes the encoded bit stream, generating video data and audio data. Then, the decoder 947 outputs the generated video data to the OSD 948. In addition, the decoder 904 outputs the generated audio data to an external speaker.

The OSD 948 reproduces the video data input from the decoder 947, thereby displaying video. In addition, the OSD 948 may overlay an image of the GUI, such as a menu, a button, a cursor, etc., on the displayed video.

The control portion 949 includes a processor such as a CPU, and a memory such as a RAM or a ROM. The memory stores programs executed by the CPU, program data, and the like. When the recording/reproducing apparatus 940 is started, the CPU reads and executes the program stored in the memory. The CPU controls the operation of the recording/reproducing apparatus 940 according to an operation signal input from the user interface 950 by executing a program.

The user interface 950 is connected to the control portion 949. The user interface 950 includes buttons and switches used by the user to operate the recording/reproducing apparatus 940, and a receiving section of a remote control signal. The user interface 950 detects operations by the user via these constituent elements, generates an operation signal, and outputs the generated operation signal to the control portion 949.

In the recording/reproducing apparatus 940 configured in this way, the encoder 943 has the functions of the image encoding apparatus 10 according to the above-described embodiment. In addition, the decoder 947 has the functions of the image decoding apparatus 60 according to the above-described embodiment. Thus, in the case of encoding and decoding an image in the recording/reproducing apparatus 940, it is also possible to enhance the parallelism of the deblocking filter process, thereby ensuring high-speed processing.

[7-4. fourth exemplary application ]

Fig. 47 is a block diagram showing an example of a schematic configuration of an image pickup apparatus employing the above-described embodiment. The image pickup apparatus 960 picks up an image of a subject, generates an image, encodes image data, and records the image data in a recording medium.

The image pickup apparatus 960 includes an optical member 961, an image pickup section 962, a signal processing section 963, an image processing section 964, a display section 965, an external interface 966, a memory 967, a media drive 968, an OSD 969, a control section 970, a user interface 971, and a bus 972.

The optical member 961 is connected to the image pickup section 962. The image pickup section 962 is connected to the signal processing section 963. The display section 965 is connected to the image processing section 964. The user interface 971 is connected to the control section 970. The bus 972 interconnects the image processing portion 964, the external interface 966, the memory 967, the media drive 968, the OSD 969, and the control portion 970.

The optical member 961 includes a focus lens, an aperture stop mechanism, and the like. The optical member 961 forms an optical image of a subject on an imaging surface of the imaging section 962. The image pickup section 962 includes an image sensor such as a CCD, a CMOS, or the like, and converts an optical image formed on an image pickup surface into an image signal as an electric signal by photoelectric conversion. Subsequently, the image pickup section 962 outputs the image signal to the signal processing section 963.

The signal processing section 963 performs various camera signal processes such as knee correction, γ correction, color correction, and the like on the image signal input from the image pickup section 962. The signal processing section 963 outputs the image data after the camera signal processing to the image processing section 964.

The image processing section 964 encodes the image data input from the signal processing section 963, generating encoded data. Subsequently, the image processing portion 964 outputs the generated encoded data to the external interface 966 or the media drive 968. In addition, the image processing portion 964 decodes encoded data input from the external interface 966 or the media drive 968, thereby generating image data. Subsequently, the image processing section 964 outputs the generated image data to the display section 965. In addition, the image processing section 964 may output the image data input from the signal processing section 963 to the display section 965, thereby causing the image to be displayed. Further, the image processing section 964 may superimpose display data obtained from the OSD 969 on an image to be output to the display section 965.

The OSD 969 generates an image of the GUI such as a menu, a button, a cursor, and the like, and outputs the generated image to the image processing section 964.

The external interface 966 is configured as a USB input/output terminal, for example. The external interface 966 connects the image pickup apparatus 960 and a printer when printing an image, for example. In addition, a drive is connected to the external interface 966 as appropriate. A removable medium such as a magnetic disk, an optical disk, or the like is mounted in the drive, and a program read from the removable medium may be installed in the image pickup apparatus 960. Further, the external interface 966 may be configured as a network interface to be connected to a network such as a LAN, the internet, or the like. That is, the external interface 966 functions as a transmission means of the image pickup apparatus 960.

The recording medium to be mounted on the media drive 968 may be any readable and writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, a semiconductor memory, or the like. In addition, the recording medium may be fixedly mounted on the media drive 968, constituting a non-portable storage section such as a built-in hard disk drive or an SSD (solid state drive).

The control section 970 includes a processor such as a CPU, and a memory such as a RAM or a ROM. The memory stores programs to be executed by the CPU, program data, and the like. When the image pickup apparatus 960 is started, the CPU reads and executes the program stored in the memory. The CPU controls the operation of the image pickup apparatus 960 in accordance with an operation signal input from the user interface 971 by executing a program.

The user interface 971 is connected to the control section 970. The user interface 971 includes buttons, switches, and the like used by the user to operate the image pickup apparatus 960. The user interface 971 detects an operation by a user via these constituent elements, generates an operation signal, and outputs the generated operation signal to the control section 970.

In the image pickup apparatus 960 configured in this way, the image processing section 964 has the functions of the image encoding apparatus 10 and the image decoding apparatus 60 according to the above-described embodiments. Thus, also in the case of encoding and decoding an image in the image pickup apparatus 960, the degree of parallelism of the deblocking filter process can be enhanced, thereby ensuring high-speed processing.

<8. summary >

Referring to fig. 1 to 47, 3 embodiments of the deblocking filter of the image encoding apparatus 10 and the image decoding apparatus 60 according to the embodiments are explained. These 3 embodiments mitigate the dependency of the deblocking filter process inherent in the prior art. This can improve the parallelism of processing when the deblocking filter is applied. As a result, a delay or a reduction in data rate caused by a large processing amount of the deblocking filter can be avoided, thereby ensuring high-speed processing. The parallelism and the order of the deblocking filter process can be flexibly set according to various conditions such as an image size or an installation environment.

According to the first embodiment, when it is decided whether or not filtering is required for one of a vertical boundary and a horizontal boundary, pixel values of an input image supplied to a deblocking filter are referred across a plurality of macroblocks within the image. As a result, the processing dependency between macroblocks or coding units will be reduced. Thus, being able to span multiple macroblocks, or all macroblocks within a picture (if parallelism is maximized), parallel filtering requires decision processing.

According to the second embodiment, a decision is made as to whether filtering is required for the vertical boundary and the horizontal boundary of each block without waiting for applying a deblocking filter to other blocks in the macroblock to which the block belongs. This mitigates the processing dependency between the vertical and horizontal boundaries in a macroblock or coding unit. Thus, filtering for a vertical boundary and a horizontal boundary in parallel can require decision processing in a macroblock.

According to the third embodiment, the filtering process for the vertical boundary and the horizontal boundary filters the pixels input to the deblocking filter. This structure makes it possible to make the filtering processes for the vertical boundary and the horizontal boundary parallel to each other. This may further speed up the processing of the deblocking filter. With pixels updated with two filtering processes performed in parallel, an output pixel value is calculated from the two filter outputs. Paralleling the two filtering processes may also suitably reduce block distortion that occurs at vertical and horizontal boundaries. The output pixel value may be calculated as a weighted average of the two filter outputs. This may allow the deblocking filter to more effectively remove block distortion, thereby further improving image quality.

This specification mainly describes an example in which the filtering processing for a vertical boundary precedes the filtering processing for a horizontal boundary. In addition, the above-described effects of the technique according to the present disclosure are also applicable to the case where the filter processing for the horizontal boundary precedes the filter processing for the vertical boundary. The size of the deblocking filter processing units or macroblocks can be determined differently than described in the specification. One possible technique may omit the filtering need decision process and parallelize the application of the deblocking filter for vertical and horizontal boundaries.

The technique of transmitting information for parallelization of deblocking filter processing from the encoding side to the decoding side is not limited to the technique of multiplexing the information into the header of the encoded stream. For example, the information may not be multiplexed into the encoded bitstream, but may be transmitted or recorded as separate data associated with the encoded bitstream. The term "associated" means ensuring the possibility of relating an image (or a part of an image, such as a slice or a block) contained in a bitstream to information corresponding to the image. That is, the information may be transmitted through a transmission path different from a transmission path for an image (or a bit stream). The information may be recorded on a different recording medium (or a different recording area on the same recording medium) from the recording medium for the image (or the bitstream). The information and the image (or the bit stream) may be related to each other according to any unit such as a plurality of frames, one frame, a part of one frame.

Preferred embodiments of the present invention are described above with reference to the drawings, however, the present invention is of course not limited to the above examples. Those skilled in the art can derive various changes and modifications within the scope of the appended claims, and it should be understood that these changes and modifications are naturally within the technical scope of the present invention.

The present specification expresses the filtering process for the vertical boundary as "horizontal filtering", and expresses the filtering process for the horizontal boundary as "vertical filtering". Typically, the filtering process for vertical boundaries utilizes horizontally arranged filter taps. The filtering process for the horizontal boundary utilizes vertically arranged filter taps. Therefore, for the filtering process, the above terms are used.

List of reference numerals

10,60 image processing apparatus

112,212 first decision section (vertical boundary decision section)

116,216 second decision section (horizontal boundary decision section)

132 first filter part (horizontal filter component)

142 second first filter part (vertical filter component)

150 parallelization control section

Claims (16)

1. An image processing apparatus comprising:

a horizontal filtering unit that applies a horizontal deblocking filter to a plurality of vertical boundaries, which are vertical boundaries between blocks in an image to be locally decoded during encoding, in a state where there is no processing dependency associated with the vertical deblocking filter between the horizontal deblocking filters for the vertical boundaries, the horizontal deblocking filter being targeted for pixels located near the vertical boundaries;

a vertical filtering unit that applies a vertical deblocking filter across a plurality of horizontal boundaries between blocks in an image including pixels to which the horizontal deblocking filter is applied by the horizontal filtering unit, the vertical deblocking filter being applied to pixels located near the horizontal boundaries; and

and a coding unit that codes an image using the image to which the vertical deblocking filter is applied by the vertical filtering unit.

2. The image processing apparatus according to claim 1, wherein

The horizontal filtering unit applies the horizontal deblocking filter to pixels located in the vicinity of all vertical boundaries determined to be applied with the horizontal deblocking filter in a picture of an image to be locally decoded at the time of encoding.

3. The image processing apparatus according to claim 2, wherein

The vertical filtering unit applies the vertical deblocking filter to pixels located in the vicinity of all horizontal boundaries determined to be applied with the vertical deblocking filter in a picture of an image to which the deblocking filter is applied by the horizontal filtering unit.

4. The image processing apparatus according to claim 3, wherein

The horizontal filtering unit applies the horizontal deblocking filter in a state where the dependency relationship of the processing accompanying the vertical deblocking filter does not exist between the plurality of blocks to be processed by the horizontal deblocking filter so that the horizontal deblocking filters do not have a dependency relationship with each other with respect to the plurality of vertical boundaries adjacent to each other between the blocks,

the vertical filtering unit applies the vertical deblocking filter so that the vertical deblocking filters do not have a dependency relationship with each other with respect to a plurality of horizontal boundaries adjacent to each other between blocks.

5. The image processing apparatus according to claim 4, wherein

The horizontal filtering unit applies the horizontal deblocking filter to the plurality of vertical boundaries adjacent to each other between blocks using a tap length that does not overlap with each other,

the vertical filtering unit applies the vertical deblocking filter to the plurality of horizontal boundaries adjacent to each other between blocks using a tap length that does not overlap with each other.

6. The image processing apparatus according to claim 1, wherein

The horizontal filtering section applies the horizontal directional deblocking filters in parallel,

the vertical filtering section applies the deblocking filters in the vertical direction in parallel.

7. The image processing apparatus according to claim 1, wherein

The block size of the block is 8 x 8,

the length of the vertical boundary is 8 pixels,

the length of the horizontal boundary is 8 pixels.

8. The image processing apparatus according to claim 1, wherein

The encoding unit encodes the image for each of the coding units divided from the maximum coding unit.

9. An image processing method comprising:

a horizontal filtering step of applying a horizontal deblocking filter across a plurality of vertical boundaries, the vertical boundaries being vertical boundaries between blocks within an image to be locally decoded at the time of encoding, in a state where there is no dependency of processing with respect to a vertical deblocking filter in a horizontal direction with respect to the plurality of vertical boundaries, the horizontal deblocking filter being targeted for pixels located in the vicinity of the vertical boundaries;

a vertical filtering step of applying a vertical-direction deblocking filter across a plurality of horizontal boundaries between blocks within an image including pixels to which the horizontal-direction deblocking filter is applied by the horizontal filtering step, the vertical-direction deblocking filter being targeted for pixels located in the vicinity of the horizontal boundaries; and

and a coding step of coding the image using the image to which the vertical deblocking filter is applied in the vertical filtering step.

10. The image processing method according to claim 9, wherein

In the horizontal filtering step, the horizontal deblocking filter is applied to pixels located in the vicinity of all vertical boundaries of the image determined to be locally decoded at the time of encoding, to which the horizontal deblocking filter is applied.

11. The image processing method according to claim 10, wherein

In the vertical filtering step, the vertical deblocking filter is applied to pixels located in the vicinity of all horizontal boundaries determined to be applied with the vertical deblocking filter in the picture of the image to which the deblocking filter is applied in the horizontal filtering step.

12. The image processing method according to claim 11, wherein

In the horizontal filtering step, the horizontal deblocking filter is applied in a state where the dependency of the processing accompanying the vertical deblocking filter does not exist between the plurality of blocks that are processing targets of the horizontal deblocking filter, in such a manner that the horizontal deblocking filters do not generate the dependency with each other for the plurality of vertical boundaries adjacent to each other between the blocks,

in the vertical filtering step, the vertical deblocking filter is applied in such a manner that the vertical deblocking filters for a plurality of the horizontal boundaries adjacent between blocks do not produce a dependency relationship with each other.

13. The image processing method according to claim 12, wherein

Applying, in the horizontal filtering step, the horizontal deblocking filters using tap lengths that do not overlap with each other at the horizontal deblocking filters for a plurality of the vertical boundaries adjacent between blocks,

in the vertical filtering step, the vertical deblocking filters are applied using non-repeating tap lengths from one another at the vertical deblocking filters for a plurality of the horizontal boundaries adjacent between blocks.

14. The image processing method according to claim 9, wherein

In the horizontal filtering step, the horizontal directional deblocking filters are applied in parallel,

in the vertical filtering step, the deblocking filters in the vertical direction are applied in parallel.

15. The image processing method according to claim 9, wherein

The block size of the block is 8 x 8,

the length of the vertical boundary is 8 pixels,

the length of the horizontal boundary is 8 pixels.

16. The image processing method according to claim 9, wherein

In the encoding step, the image is encoded for each of the coding units divided from the maximum coding unit.