HK1229110B - Image processing device and image processing method - Google Patents

Description

Image processing device and image processing method

This application is a divisional application of the Chinese invention patent application filed by the same applicant with application number 201180057815.8 (PCT/JP2011/077953) filed on December 2, 2011, and with the invention name “Image processing device and image processing method”.

Technical Field

The present disclosure relates to an image processing device and an image processing method.

Background Art

H.264/AVC, a standard for image coding schemes, applies a deblocking filter to block boundaries, for example, in units of 4×4 pixels, in order to prevent image quality degradation due to block distortion during image coding. The deblocking filter requires a large amount of processing and can account for, for example, 50% of the total processing load in image decoding.

JCTVC-A119 (see Non-Patent Document 1), a standard for High Efficiency Video Coding (HEVC) as a next-generation image coding system, proposes applying a deblocking filter to each block containing 8×8 pixels or more. The technology proposed in JCTVC-A119 increases the block size, the minimum unit for applying the deblocking filter, so that filtering is performed in parallel across block boundaries in the same direction within a macroblock.

Reference List

Non-patent literature

Non-patent literature 1: K. Ugur (Nokia), K. R. Andersson (LM Ericsson), A. Fuldseth (Tandberg Telecom), "JCTVC-A119: Video coding technology proposal by Tandberg, Nokia, and Ericsson", Documents of the first meeting of the Joint Collaborative Team on Video Coding (JCT-VC), Dresden, Germany, April 15-23, 2010.

Summary of the Invention

Technical issues

However, the technology proposed in JCTVC-A119 does not resolve the dependency between processing at vertical block boundaries and processing at horizontal block boundaries. It remains difficult to enable parallel processing at block boundaries in different directions within a macroblock (or coding unit), as well as parallel processing between macroblocks. Consequently, the technology described above fails to successfully address the issues of delay and data rate reduction caused by the large amount of processing required when a deblocking filter is applied.

The technology according to the present disclosure aims to provide an image processing device and an image processing method capable of providing parallel processing when a deblocking filter is applied.

Solutions to the Problem

According to an embodiment of the present disclosure, there is provided an image processing apparatus comprising: a decoding unit configured to decode an image from a coded stream; a determination unit configured to perform determination processing to determine whether a deblocking filter is applicable to adjacent blocks that are adjacent across block boundaries within the image decoded by the decoding unit; a filtering unit configured to apply a deblocking filter to the adjacent blocks that the determination unit has determined as applicable to the deblocking filter; and a control unit configured to cause the determination unit to perform determination processing for vertical block boundaries and horizontal block boundaries using pixels of adjacent blocks of a reconstructed image as reference pixels.

The image processing device may typically be implemented as an image decoding device for decoding an image.

In addition, according to an embodiment of the present disclosure, there is provided an image processing method, comprising: decoding an image from a coded stream; performing a determination process to determine whether a deblocking filter is applicable to adjacent blocks that are adjacent across block boundaries within the decoded image; applying a deblocking filter to adjacent blocks that have been determined to be applicable to the deblocking filter in the determination process; and controlling the determination process in the following manner: using pixels of adjacent blocks of a reconstructed image as reference pixels, a determination process is performed for vertical block boundaries and horizontal block boundaries.

In addition, according to an embodiment of the present disclosure, there is provided an image processing device, comprising: a determination unit configured to perform determination processing to determine whether a deblocking filter is applicable to adjacent blocks adjacent across block boundaries within a locally decoded image when the image to be encoded is encoded; a filtering unit configured to apply a deblocking filter to the adjacent blocks that the determination unit has determined to be applicable for the deblocking filter; a control unit configured to cause the determination unit to perform determination processing for vertical block boundaries and horizontal block boundaries using pixels of adjacent blocks of a reconstructed image as reference pixels; and an encoding unit configured to encode the image to be encoded using the image filtered by the filtering unit.

The image processing apparatus may typically be implemented as an image encoding apparatus for encoding an image.

According to an embodiment of the present disclosure, there is provided an image processing method, comprising: performing a determination process to determine whether a deblocking filter is applicable to adjacent blocks that are adjacent across block boundaries within a locally decoded image when the image to be encoded is encoded; applying the deblocking filter to the adjacent blocks that have been determined to be applicable to the deblocking filter in the determination process; controlling the determination process in the following manner: using pixels of adjacent blocks of a reconstructed image as reference pixels, performing a determination process for vertical block boundaries and horizontal block boundaries; and encoding the image to be encoded using the image filtered by the deblocking filter.

Advantageous Effects of the Invention

As described above, according to the image processing apparatus and the image processing method of the present disclosure, parallel processing is further improved when the deblocking filter is applied.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an example of the configuration of an image encoding device according to an embodiment.

FIG. 2 is a block diagram showing an example of the configuration of an image decoding device according to an embodiment.

FIG. 3 is an explanatory diagram showing an example of adjacent pixels around a boundary.

FIG. 4 is an explanatory diagram showing reference pixels in filtering necessity determination processing according to the related art.

FIG. 5 is an explanatory diagram showing pixels updated by the filtering process.

FIG. 6 is an explanatory diagram showing identification of edges for explaining the embodiment.

FIG. 7 is an explanatory diagram showing parallel processing according to the related art.

FIG. 8 is a first explanatory diagram illustrating dependencies between processes according to the related art.

FIG. 9 is a second explanatory diagram illustrating dependencies between processes according to the related art.

FIG. 10 is an explanatory diagram showing the sequence of processing according to the related art.

FIG. 11 is a first explanatory diagram showing reference pixels in filtering necessity determination processing according to the first working example.

FIG. 12 is a second explanatory diagram illustrating reference pixels in the filtering necessity determination process according to the first working example.

FIG. 13 is an explanatory diagram showing a first example of the processing sequence.

FIG. 14 is an explanatory diagram showing a second example of the processing sequence.

FIG15 is a block diagram showing a detailed configuration of a deblocking filter according to the first working example.

FIG16 is a block diagram showing a detailed configuration of a determination section.

FIG. 17 is an explanatory diagram showing adjacent blocks around a slice boundary.

FIG. 18 is an explanatory diagram showing a first example of the order of processing for each slice.

FIG. 19 is an explanatory diagram showing a second example of the order of processing for each slice.

FIG. 20 is a flowchart illustrating a first example of a processing flow of a deblocking filter according to an embodiment.

FIG. 21 is a flowchart illustrating a second example of the processing flow of the deblocking filter according to the embodiment.

FIG. 22 is a flowchart illustrating the flow of filtering necessity determination processing according to an embodiment.

FIG23 is a block diagram showing a detailed configuration of a deblocking filter according to the second working example.

FIG. 24 is an explanatory diagram showing first and second examples of the determination technique provided by the second working example.

FIG. 25 is an explanatory diagram showing third and fourth examples of the determination technique provided by the second working example.

FIG. 26 is an explanatory diagram showing fifth and sixth examples of the determination technique provided by the second working example.

FIG. 27 is an explanatory diagram showing the processing order for each LCU.

FIG28 is a flowchart showing the flow of processing for each LCU.

FIG29 is an explanatory diagram showing an outline of the third operation example.

FIG30 is a block diagram showing a detailed configuration of a deblocking filter according to the third working example.

FIG. 31 is an explanatory diagram showing determination of weights for weighted averaging.

FIG. 32 is an explanatory diagram showing an example of weighting for weighted averaging.

FIG33 is an explanatory diagram showing output pixel values from the calculation section according to the third working example.

FIG. 34 is an explanatory diagram showing a first example of a processing sequence for comparison.

FIG35 is an explanatory diagram showing a first example of a processing sequence provided by the third operation example.

FIG. 36 is an explanatory diagram showing a second example of the processing sequence for comparison.

FIG37 is an explanatory diagram showing a second example of the processing sequence provided by the third operation example.

FIG38 is a flowchart showing a first example of the processing flow of the deblocking filter according to the third working example.

FIG39 is a flowchart showing the flow of the pixel value calculation processing shown in FIG38 .

FIG40 is an explanatory diagram showing a multiview codec.

FIG. 41 is an explanatory diagram showing an image encoding process according to an embodiment applied to a multi-view codec.

FIG42 is an explanatory diagram showing an image decoding process according to an embodiment applied to a multi-view codec.

FIG43 is an explanatory diagram showing a scalable codec.

FIG44 is an explanatory diagram showing an image encoding process according to an embodiment applied to a scalable codec.

FIG45 is an explanatory diagram showing an image decoding process according to an embodiment applied to a scalable codec.

FIG46 is a block diagram showing a schematic configuration of a television device.

FIG47 is a block diagram showing an outline configuration of a mobile phone.

FIG48 is a block diagram showing an outline configuration of a recording/reproducing apparatus.

FIG49 is a block diagram showing an outline configuration of an image capturing device.

Reference Signs List

10, 60 Image processing equipment

112-1 to 112-n, 212-1 to 212-n First determination unit (vertical boundary determination unit)

114-1 to 114-n, 214-1 to 214-n Second determination unit (horizontal boundary determination unit)

132-1 to 132-n, 332-1 to 332-n First filter section (horizontal filter section)

142-1 to 142-n, 342-1 to 342-n Second filter section (vertical filter section)

150 Parallelization Control Unit

208 lines of memory (memory)

360 Computing Department

Description of the embodiments

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that in this specification and the accompanying drawings, elements having substantially the same function and structure are labeled with the same reference numerals, and repeated explanations are omitted.

The description of the embodiments will be given in the following order.

1. Device Overview

11. Image coding equipment

12. Image decoding equipment

2. Existing Technology

21. Basic configuration of deblocking filter

22. Dependencies between processes according to the prior art

3. First working example

31. Deblocking filter configuration example

32. Processing

4. Second working example

41. Deblocking filter configuration example

42. Processing

43. Processing example for each LCU

5. Third working example

51. Summary

52. Deblocking filter configuration example

53. Processing Order Example

54.Processing Flow

6. Application of various codecs

61.Multi-view codec

62. Scalable Codec

7. Sample Application

8. Summary

1. Device Overview

1 and 2 , the following describes an overview of a device to which the technology disclosed in this specification is applicable. The technology disclosed in this specification can be applied to, for example, an image encoding device and an image decoding device.

[11. Image Coding Device]

FIG1 is a block diagram showing an example of the configuration of an image encoding device according to an embodiment. Referring to FIG1 , the image encoding device 10 includes an A/D (analog to digital) conversion section 11, a reordering buffer 12, a subtraction section 13, an orthogonal transform section 14, a quantization section 15, a lossless encoding section 16, an accumulation buffer 17, a rate control section 18, an inverse quantization section 21, an inverse orthogonal transform section 22, an addition section 23, a deblocking filter 24a, a frame memory 25, a selector 26, an intra-frame prediction section 30, a motion estimation section 40, and a mode selection section 50.

The A/D conversion section 11 converts an image signal input in an analog format into image data in a digital format, and outputs a series of digital image data to the reordering buffer 12 .

The reordering buffer 12 reorders the images included in the series of image data input from the A/D conversion section 11. After reordering the images according to the GOP (Group of Pictures) structure of the encoding process, the reordering buffer 12 outputs the reordered image data to the subtraction section 13, the intra prediction section 30, and the motion estimation section 40.

The image data input from the reordering buffer 12 and the predicted image data selected by the mode selection section 50 described later are supplied to the subtraction section 13. The subtraction section 13 calculates predicted error data, which is the difference between the image data input from the reordering buffer 12 and the predicted image data input from the mode selection section 50, and outputs the calculated predicted error data to the orthogonal transformation section 14.

The orthogonal transform section 14 performs an orthogonal transform on the predicted error data input from the subtraction section 13. The orthogonal transform performed by the orthogonal transform section 14 may be, for example, a discrete cosine transform (DCT) or a Karhunen-Loeve transform. The orthogonal transform section 14 outputs the transform coefficient data obtained by the orthogonal transform process to the quantization section 15.

The transform coefficient data input from the orthogonal transform section 14 and the rate control signal from the rate control section 18 described later are supplied to the quantization section 15. The quantization section 15 quantizes the transform coefficient data and outputs the quantized transform coefficient data (hereinafter referred to as quantized data) to the lossless encoding section 16 and the inverse quantization section 21. In addition, the quantization section 15 switches the quantization parameter (quantization scale) based on the rate control signal from the rate control section 18, thereby changing the bit rate of the quantized data to be input to the lossless encoding section 16.

The quantized data input from the quantization unit 15 and information on intra prediction or inter prediction generated by the intra prediction unit 30 or the motion estimation unit 40 and selected by the mode selection unit 50, as described later, are supplied to the lossless encoding unit 16. The information on intra prediction may include, for example, prediction mode information indicating the optimal intra prediction mode for each block. Furthermore, the information on inter prediction may include, for example, prediction mode information for motion vector prediction for each block, differential motion vector information, reference image information, and the like.

The lossless encoding unit 16 generates a coded stream by performing lossless encoding on the quantized data. The lossless encoding performed by the lossless encoding unit 16 may be, for example, variable length coding or arithmetic coding. Furthermore, the lossless encoding unit 16 multiplexes the aforementioned information regarding intra-frame prediction or inter-frame prediction into the header of the coded stream (e.g., a block header, a slice header, etc.). The lossless encoding unit 16 then outputs the generated coded stream to the accumulation buffer 17.

The accumulation buffer 17 temporarily stores the encoded stream input from the lossless encoding section 16 using a storage medium such as a semiconductor memory. The accumulation buffer 17 then outputs the accumulated encoded stream at a code rate corresponding to the frequency band of the transmission line (or output line from the image encoding device 10).

The rate control unit 18 monitors the available space in the accumulation buffer 17. It then generates a rate control signal based on the available space in the accumulation buffer 17 and outputs the generated rate control signal to the quantization unit 15. For example, when there is not much available space in the accumulation buffer 17, the rate control unit 18 generates a rate control signal that reduces the bit rate of the quantized data. Alternatively, when there is sufficient available space in the accumulation buffer 17, the rate control unit 18 generates a rate control signal that increases the bit rate of the quantized data.

The inverse quantization section 21 performs an inverse quantization process on the quantized data input from the quantization section 15. Then, the inverse quantization section 21 outputs the transform coefficient data obtained by the inverse quantization process to the inverse orthogonal transform section 22.

The inverse orthogonal transform section 22 performs an inverse orthogonal transform process on the transform coefficient data input from the inverse quantization section 21 , thereby restoring the predicted error data.

The addition unit 23 generates decoded image data by adding the restored predicted error data input from the inverse orthogonal transform unit 22 and the predicted image data input from the mode selection unit 50. The addition unit 23 then outputs the generated decoded image data to the deblocking filter 24a and the frame memory 25.

The deblocking filter 24a performs filtering processing to reduce block distortion that occurs during image encoding. For example, the deblocking filter 24a determines the necessity of filtering for each block boundary of the decoded image data supplied from the addition unit 23, and applies deblocking filtering to the boundaries determined to require filtering. The deblocking filter 24a is also supplied with information used to determine the necessity of filtering (e.g., mode information, transform coefficient information, and motion vector information), as well as the image data decoded from the addition unit 23. After filtering, block distortion is eliminated from the decoded image data, and the deblocking filter 24a outputs the decoded image data to the frame memory 25. The processing of the deblocking filter 24a will be described in detail later.

The frame memory 25 stores the decoded image data input from the adding unit 23 and the decoded image data after filtering input from the deblocking filter 24 a using a storage medium.

The selector 26 reads out the decoded image data before filtering for intra-frame prediction from the frame memory 25, and supplies the decoded image data thus read out as reference image data to the intra-frame prediction unit 30. Furthermore, the selector 26 reads out the decoded image data after filtering for inter-frame prediction from the frame memory 25, and supplies the decoded image data thus read out as reference image data to the motion estimation unit 40.

The intra-frame prediction unit 30 performs intra-frame prediction processing for each intra-frame prediction mode based on the image data to be encoded input from the reordering buffer 12 and the decoded image data provided via the selector 26. For example, the intra-frame prediction unit 30 uses a predetermined cost function to evaluate the prediction results of each intra-frame prediction mode. The intra-frame prediction unit 30 then selects the intra-frame prediction mode with the smallest cost function value, that is, the intra-frame prediction mode with the highest compression rate, as the optimal intra-frame prediction mode. Furthermore, the intra-frame prediction unit 30 outputs prediction mode information indicating the optimal intra-frame prediction mode, the predicted image data, and information related to the intra-frame prediction, such as the cost function value, to the mode selection unit 50.

The motion estimation unit 40 performs inter-frame prediction processing (prediction processing between frames) based on the image data for encoding provided from the reordering buffer 12 and the decoded image data provided via the selector 26. For example, the motion estimation unit 40 uses a predetermined cost function to evaluate the prediction results of each prediction mode. The motion estimation unit 40 then selects the optimal prediction mode, that is, the prediction mode that minimizes the cost function value or maximizes the compression rate. The motion estimation unit 40 generates predicted image data based on the optimal prediction mode. The motion estimation unit 40 outputs information related to inter-frame prediction, such as prediction mode information indicating the optimal intra-frame prediction mode, predicted image data, and the cost function value, to the mode selection unit 50.

The mode selection unit 50 compares the cost function value related to intra-frame prediction input from the intra-frame prediction unit 30 with the cost function value related to inter-frame prediction input from the motion estimation unit 40. The mode selection unit 50 then selects the prediction method with the smaller cost function value from the intra-frame prediction and inter-frame prediction methods. If intra-frame prediction is selected, the mode selection unit 50 outputs information about the intra-frame prediction to the lossless encoding unit 16 and outputs the predicted image data to the subtraction unit 13 and the addition unit 23. If inter-frame prediction is selected, the mode selection unit 50 outputs information about the inter-frame prediction to the lossless encoding unit 16 and outputs the predicted image data to the subtraction unit 13 and the addition unit 23.

[12. Image decoding device]

2 is a block diagram showing an example of the configuration of an image decoding device 60 according to an embodiment. Referring to FIG2 , the image decoding device 60 includes an accumulation buffer 61, a lossless decoding section 62, an inverse quantization section 63, an inverse orthogonal transform section 64, an addition section 65, a deblocking filter 24 b, a rearrangement buffer 67, a D/A (digital to analog) conversion section 68, a frame memory 69, selectors 70 and 71, an intra prediction section 80, and a motion compensation section 90.

The accumulation buffer 61 temporarily stores the encoded stream input via the transmission line using a storage medium.

The lossless decoding unit 62 decodes the coded stream input from the accumulation buffer 61 according to the coding method used at the time of encoding. Furthermore, the lossless decoding unit 62 decodes information multiplexed into the header region of the coding process. This information multiplexed into the header region of the coding process may include, for example, information regarding intra-frame prediction and information regarding inter-frame prediction in the block header. The lossless decoding unit 62 outputs the intra-frame prediction information to the intra-frame prediction unit 80. Furthermore, the lossless decoding unit 62 outputs the inter-frame prediction information to the motion compensation unit 90.

The inverse quantization section 63 inversely quantizes the quantized data decoded by the lossless decoding section 62. The inverse orthogonal transform section 64 generates predicted error data by performing an inverse orthogonal transform on the transform coefficient data input from the inverse quantization section 63 according to the orthogonal transform method used in encoding. The inverse orthogonal transform section 64 then outputs the generated predicted error data to the addition section 65.

The addition unit 65 generates decoded image data by adding the predicted error data input from the inverse orthogonal transform unit 64 and the predicted image data input from the selector 71. The addition unit 65 then outputs the generated decoded image data to the deblocking filter 24b and the frame memory 69.

The deblocking filter 24b performs filtering processing to reduce block distortion that appears in the decoded image. For example, the deblocking filter 24b determines the necessity of filtering at each block boundary of the decoded image data input from the addition unit 65, and applies the deblocking filter to the boundaries determined to require filtering. The deblocking filter 24b is also supplied with information used to determine the necessity of filtering, as well as the decoded image data from the addition unit 65. After filtering, block distortion is eliminated from the decoded image data, and the deblocking filter 24b outputs the decoded image data to the reordering buffer 67 and the frame memory 69. The processing of the deblocking filter 24b will be described in detail later.

The rearrangement buffer 67 rearranges the image input from the deblocking filter 24 b to generate a time series of image data, and then outputs the generated image data to the D/A conversion unit 68 .

The D/A converter 68 converts the digital image data input from the rearrangement buffer 67 into an analog image signal. The D/A converter 68 then displays the image by outputting the analog image signal to a display (not shown) connected to the image decoding device 60, for example.

The frame memory 69 stores the decoded image data before filtering input from the addition unit 65 and the decoded image data after filtering input from the deblocking filter 24 b using a storage medium.

The selector 70 switches the output destination of the image data from the frame memory 69 between the intra prediction unit 80 and the motion compensation unit 90 for each block in the image, based on the mode information obtained from the lossless decoding unit 62. For example, when the intra prediction mode is specified, the selector 70 outputs the decoded image data before filtering supplied from the frame memory 69 to the intra prediction unit 80 as reference image data. Alternatively, when the inter prediction mode is specified, the selector 70 outputs the decoded image data after filtering supplied from the frame memory 69 to the motion compensation unit 90 as reference image data.

The selector 71 switches the output source of the predicted image data to be supplied to the addition section 65 between the intra prediction section 80 and the motion compensation section 90 for each block in the image, based on the mode information acquired by the lossless decoding section 62. For example, when the intra prediction mode is specified, the selector 71 supplies the predicted image data output from the intra prediction section 80 to the addition section 65. When the inter prediction mode is specified, the selector 71 supplies the predicted image data output from the motion compensation section 90 to the addition section 65.

The intra prediction unit 80 performs intra-screen prediction of pixel values based on the information on intra prediction input from the lossless decoding unit 62 and the reference image data from the frame memory 69, and generates predicted image data. The intra prediction unit 80 then outputs the generated predicted image data to the selector 71.

The motion compensation section 90 performs motion compensation processing based on the information on inter-frame prediction input from the lossless decoding section 62 and the reference image data from the frame memory 69 and generates predicted image data. The motion compensation section 90 then outputs the generated predicted image data to the selector 71.

<2. Prior Art>

[2-1. Basic configuration of deblocking filter]

Generally speaking, the deblocking filter used in existing image coding systems such as H.264/AVC or HEVC includes two types of processing: filtering requirement determination processing and filtering processing. The following describes these two types of processing in HEVC, for example.

(1) Filtering requires certain processing

The filtering needs to determine whether the deblocking filter needs to be applied to each boundary of the block within the input image. Block boundaries include vertical boundaries between blocks that are horizontally adjacent to each other, and horizontal boundaries between blocks that are vertically adjacent to each other. JCTVC-A119 uses a block size of 8×8 pixels as the minimum processing unit. For example, a macroblock of 16×16 pixels contains 4 blocks of 8×8 pixels. For each block, the processing is applied to one (left) vertical boundary and one (top) horizontal boundary, that is, 4 boundaries plus 4 boundaries equals 8 boundaries in total. The specification assumes that macroblocks, which are technical terms, are included in coding units (CUs) in the environment of HEVC.

FIG3 is an illustrative diagram showing an example of pixels in two blocks (adjacent blocks) Ba and Bb adjacent to each other around a boundary. The vertical boundary is described below as an example, and the description is obviously also applicable to horizontal boundaries. The example in FIG3 uses the symbol p _ij to represent the pixels in block Ba. In this symbol, i represents the column index and j represents the row index. The column index i is numbered 0, 1, 2 and 3 (from right to left) from the column closest to the vertical boundary. The row index j is numbered 0, 1, 2 ... 7 from top to bottom. The left half of block Ba is omitted from the figure. The symbol q _kj is used to represent the pixels in block Bb. In this symbol, k represents the column index and j represents the row index. The column index k is numbered 0, 1, 2 and 3 (from left to right) from the column closest to the vertical boundary. The right half of block Bb is omitted from the figure.

The necessity of applying the deblocking filter to the vertical boundary between blocks Ba and Bb shown in FIG. 3 can be determined using the following conditions.

Determination conditions for the Luma component ... If both conditions A and B are true, the deblocking filter is applied.

Condition A:

(A1) Block Ba or Bb enters intra prediction mode;

(A2) block Ba or Bb has a non-zero orthogonal transform coefficient; or

(A3) |MVAx-MVBx|≥4 or |MVAy-MVBy|≥4

Condition B:

|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |<β

Condition A3 assumes that the motion vector for block Ba is (MVAx, MVAy) and the motion vector for block Bb is (MVBx, MVBy) based on Qpel (1/4 pixel) accuracy. Condition B uses β as the edge determination threshold. The initial value of β is given by the quantization parameter. The value of β can be specified by the user using a parameter in the slice header.

Determination conditions for the chroma component...If condition A1 is true, the deblocking filter is applied.

Condition A1: Block Ba or Bb enters intra prediction mode.

As shown in the dashed boxes L3 and L6 in FIG4 , filtering of a typical vertical boundary requires determining the processing (especially under determination condition B for the luminance component) with reference to pixels in the third and sixth rows (assuming the top row is the first) in each block. Similarly, filtering of a horizontal boundary requires determining the processing with reference to pixels in the third and sixth columns in each block (not shown in FIG4 ). The above determination conditions are used to determine the need to apply the deblocking filter to the boundary, and the filtering process described below is performed on that boundary.

(2) Filtering

If it is determined that a deblocking filter needs to be applied to a boundary, filtering is performed on pixels to the right and left of the vertical boundary and on pixels above and below the horizontal boundary. For the luminance component, the filter strength is switched between a strong filter and a weak filter according to the pixel value.

Filter the luminance component

Select Strength... Selects the filter strength for each row or column. If all of the following conditions C1 to C3 are met, a strong filter is selected. If any of these conditions are not met, a weak filter is selected.

(C1)d<(β>>2)

(C2)(|p _3j -p _0j |+|q _0j -q _3j |)<(β>>3)

(C3)|p _0j -q _0j |<((5t _C +1)>>1)

Where j represents the row index for the vertical boundary or the column index for the horizontal boundary. d = |p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |

Weak filter

△＝Clip(-t _C ,t _C ,(13(q _0j -p _0j )+4(q _1j -p _1j )-5(q _2j -p _2j )+16)>>5))

p _0j =Clip _0-255 (p _0j +△)

q _0j =Clip _0-255 (q _0j -△)

p _1j =Clip _0-255 (p _1j +△/2)

q _1j =Clip _0-255 (q _1j -△/2)

Strong filter

p _0j ＝Clip _0-255 ((p _2j +2p _1j +2p _0j +2q _0j +q _1j +4)>>3)

q _0j ＝Clip _0-255 ((p _1j +2p _0j +2q _0j +2q _1j +q _2j +4)>>3)

p _1j ＝Clip _0-255 ((p _2j +p _1j +p _0j +q _0j +2)>>2)

q _1j ＝Clip _0-255 ((p _0j +q _0j +q _1j +q _2j +2)>>2)

p _2j ＝Clip _0-255 ((2p _3j +3p _2j +p _1j +p _0j +q _0j +4)>>3)

q _2j ＝Clip _0-255 ((p _0j +q _0j +q _1j +3q _2j +2q _3j +4)>>3)

Here, Clip (a, b, c) indicates a process of clipping the value c to within the range of a≤c≤b, and Clip _0-255 (c) indicates a process of clipping the value c to within the range of 0≤c≤255.

Filtering the chroma component

△＝Clip(-t _C , t _C , ((((q _0j -p _0j )<<2)+p _1j -q _1j +4)>>3))

p _0j =Clip _0-255 (p _0j +△)

q _0j =Clip _0-255 (q _0j -△)

As shown in the dashed boxes C6 to C8 and C1 to C3 in Figure 5, the filtering process at the general vertical boundaries (especially the strong filter for the luminance component) updates the pixel values of the first to third and sixth to eighth columns in each block. Similarly, the filtering process at the horizontal boundaries updates the pixel values of the first to third and sixth to eighth rows in each block.

[22. Dependencies between processes according to prior art]

For the purpose of explanation, as shown in FIG6 , each macroblock MBx (MB0, MB1, . . . ) having a size of 16×16 pixels includes an upper left vertical boundary denoted as Vx,0, an upper middle vertical boundary denoted as Vx,1, a lower left vertical boundary denoted as Vx,2, a lower middle vertical boundary denoted as Vx,3, an upper left horizontal boundary denoted as Hx,0, an upper right horizontal boundary denoted as Hx,1, a left middle horizontal boundary denoted as Hx,2, and a right middle horizontal boundary denoted as Hx,3. Regarding the boundary Z, for example, the filtering process required is determined as J _Z and the filtering process is denoted as F _Z.

The above-described prior art does not create dependencies between processing for boundaries in the same direction within a macroblock. Therefore, this technique can, for example, perform parallel filtering on vertical and horizontal boundaries within a macroblock. As an example, FIG7 illustrates that within macroblock MB0, there are no dependencies between the four filtering processes F _V0,0 , F _V0,1 , F _V0,2 , and F _V0,3 (no pixels are redundantly updated), and the filtering processes can be performed in parallel.

However, the above-mentioned prior art leaves a dependency between the filtering process for vertical boundaries and the filtering necessity determination process for horizontal boundaries. The prior art also leaves a dependency between the filtering process for horizontal boundaries and the filtering necessity determination process for vertical boundaries. For example, if vertical boundaries are processed before horizontal boundaries, then the filtering necessity determination process for horizontal boundaries within a given macroblock must be performed after the vertical boundary filtering process is terminated. As an example, Figure 8 shows that within macroblock MB0, the filtering necessity determination process J _H0,0 depends on the results of filtering processes F _V0,0 and F _V0,1 , and the filtering necessity determination process J _H0,1 depends on the result of filtering process F _V0,1 . Similarly, the filtering necessity determination process for vertical boundaries within a given macroblock must be performed after the filtering process for the horizontal boundaries of adjacent macroblocks is terminated. As an example, FIG9 shows that the filtering need determination process J _V1,0 for macroblock MB1 depends on the results of the filtering processes F _H0,1 and F _H0,3 for macroblock MB0, and the filtering need determination process J _V1,2 for macroblock MB1 depends on the result of the filtering process F _H0,3 for macroblock MB0.

The existing technology includes dependencies between processes, and thus even using the technology proposed in JCTVC-A119, only very limited parallel processing of deblocking filtering can be provided.

FIG10 is an explanatory diagram showing the order of processing of a deblocking filter according to the prior art. The example assumes that the deblocking filter is provided with an image having a size of 32×32 pixels. The input image contains four macroblocks MB0 to MB3, each having a size of 16×16 pixels.

In Figure 10, each dotted box represents a process to be performed in parallel. For example, the first step involves performing filtering on the four vertical boundaries of macroblock MB0 in parallel, requiring the determination of the processes J _V0,0 , J _V0,1 , J _V0,2 , and J _V0,3 . The second step involves performing filtering on the four vertical boundaries of macroblock MB0 in parallel, requiring the determination of the processes F _V0,0 , F _V0,1 , F _V0,2 , and F _V0,3 . After the second step terminates, the third step involves performing filtering on the four horizontal boundaries of macroblock MB0 in parallel, requiring the determination of the processes J _H0,0 , J _H01 , J _H02 , and J _H0,3 . The fourth step involves performing filtering on the four horizontal boundaries of macroblock MB0 in parallel, requiring the determination of the processes F _H0,0 , F _H01 , F _H02 , and F _H0,3 . After the fourth step terminates, the processes for macroblock MB1 are performed sequentially (steps 5 to 8). After the processing of macroblock MB1 terminates, the processes for macroblock MB2 are performed sequentially (steps 9 to 12). After the processing of the macroblock MB2 is terminated, the processing of the macroblock MB3 is successively performed (the thirteenth to sixteenth steps).

Such limited parallel processing cannot satisfactorily solve the problem of delay or data bit rate degradation caused by large processing load when applying a deblocking filter. When the definition is applied, the following three working examples further improve parallel processing.

<3. First Working Example>

[3-1. Deblocking filter configuration example]

The following describes example configurations of the deblocking filter 24a used in the image encoding device 10 shown in FIG. 1 and the deblocking filter 24b used in the image decoding device 60 shown in FIG. 2 according to the first working example. The configurations of the deblocking filter 24a and the deblocking filter 24b can be identical. In the following description, when there is no need to distinguish between them, the deblocking filter 24a and the deblocking filter 24b are also collectively referred to as the deblocking filter 24.

(1) Dependencies between new processes

According to the working example, the processing using the deblocking filter 24 also includes two types of processing, namely, filtering need determination processing and filtering processing. However, the deblocking filter 24 uses the value of the reference pixel, which is different from the prior art, to determine whether deblocking filtering is applicable to the vertical boundary and the horizontal boundary. More specifically, for the determination of the vertical boundary, the deblocking filter 24 uses a reference pixel, that is, a pixel included in the pixels of the adjacent blocks around the vertical boundary and belonging to a row to which deblocking filtering is not applicable for the horizontal boundary. For the determination of the horizontal boundary, the deblocking filter 24 uses another reference pixel, that is, a pixel included in the pixels of the adjacent blocks around the horizontal boundary and belonging to a row to which deblocking filtering is not applicable for the vertical boundary. In addition, in the following description, the deblocking filter 24 performs processing based on a block size of 8×8 pixels as a processing unit, for example.

FIG11 is an explanatory diagram showing reference pixels in the process of determining the need for filtering performed by the deblocking filter 24 on vertical boundaries. Referring to FIG11 , the macroblock MB0 has a size of 16×16 pixels. The deblocking filter 24 uses reference pixels belonging to at least one of the fourth and fifth rows (L4 and L5) of each block to determine whether to apply the filter to the four vertical boundaries of the macroblock MB0. Deblocking filtering for horizontal boundaries is not applied to these two rows (see FIG9 ). This configuration eliminates the dependency between the filtering process for horizontal boundaries and the process of determining the need for filtering for vertical boundaries.

FIG12 is an explanatory diagram illustrating reference pixels used in the process of determining the need for filtering of horizontal boundaries by the deblocking filter 24. FIG12 also illustrates macroblock MB0. The deblocking filter 24 uses reference pixels belonging to at least one of the fourth and fifth columns (C4 and C5) of each block to determine whether filtering is applicable to the four horizontal boundaries of macroblock MB0. Deblocking filtering of vertical boundaries is not applied to these two columns (see FIG7 or 8). This configuration eliminates the dependency between the filtering process for vertical boundaries and the process of determining the need for filtering of horizontal boundaries.

Eliminating dependencies between processes thus enables parallelization of the filtering need determination process for vertical and horizontal boundaries within a macroblock. Processing between macroblocks can be parallelized. The filtering need determination process can be performed in parallel for vertical and horizontal boundaries of all macroblocks in the input image.

13 is an explanatory diagram showing a first example of a processing sequence that can be used in the deblocking filter 24. The example also assumes that the deblocking filter is provided with an image having a size of 32×32 pixels. The input image contains four macroblocks MB0 to MB3, each having a size of 16×16 pixels.

In Figure 13, each dotted box represents a process to be performed in parallel. The example in Figure 10 requires 16 processing steps for a series of processing, while the example in Figure 13 aggregates the same number of processing steps into three. The first step involves performing filtering in parallel on all vertical and horizontal boundaries of all macroblocks MB0 to MB3, requiring the determination of processes J _V0,0 to J _V3,3 and J _H0,0 to J _H3,3 . The second step involves performing filtering processes F _V0,0 to F _V3,3 in parallel on the 16 vertical boundaries of all macroblocks MB0 to MB3. The third step involves performing filtering processes F _H0,0 and F _H3,3 in parallel on the 16 horizontal boundaries of all macroblocks MB0 to MB3. The second and third steps can be performed in reverse order.

The example in Fig. 13 maximizes the degree of parallelism (the number of processes performed in parallel) based on parallel processing between macroblocks. However, according to the example in Fig. 14 , the deblocking filter 24 can perform processing for each macroblock.

The example in Figure 14 aggregates the same number of processes shown in Figures 10 and 13 into 12 processing steps. The first step performs filtering in parallel on the four vertical and four horizontal boundaries in macroblock MB0, requiring the determination of the processes J _V0,0 through J _V0,3 and J _H0,0 through J _H0,3 . The second step performs filtering in parallel on the four vertical boundaries in macroblock MB0, requiring the determination of the processes F _V0,0 through F _V0,3 . The third step performs filtering in parallel on the four vertical and four horizontal boundaries in macroblock MB1, requiring the determination of the processes J _V1,0 through J _V1,3 and J _H1,0 through J _H1,3 . The fourth step performs filtering in parallel on the four vertical boundaries in macroblock MB1, requiring the determination of the processes F _V1,0 through F _V1,3 . The fifth step performs filtering in parallel on the four horizontal boundaries in macroblock MB0, requiring the determination of the processes F _H0,0 through F _H0,3 . The sixth step performs filtering in parallel on the four vertical and four horizontal boundaries of macroblock MB2, requiring the determination of J _V2,0 through J _V2,3 and J _H2,0 through J _H2,3 . The seventh step performs filtering in parallel on the four vertical boundaries of macroblock MB2, requiring the determination of F _V2,0 through F _V2,3 . The eighth step performs filtering in parallel on the four horizontal boundaries of macroblock MB1, requiring the determination of F _H1,0 through F _H1,3 . The ninth step performs filtering in parallel on the four vertical and four horizontal boundaries of macroblock MB3, requiring the determination of J _V3,0 through J _V3,3 and J _H3,0 through J _H3,3 . The tenth step performs filtering in parallel on the four vertical boundaries of macroblock MB3, requiring the determination of F _V3,0 through F _V3,3 . The eleventh step performs filtering in parallel on the four horizontal boundaries of macroblock MB2, requiring the determination of F _H2,0 through F _H2,3 . The twelfth step performs filtering processes F _H3,0 to F _H3,3 in parallel on four horizontal boundaries in the macroblock MB3. In this case, although the degree of parallelism is lower than the example of FIG13 , the deblocking filter 24 can perform processing on the entire input image using fewer processing steps than in the prior art.

(2) Basic configuration of deblocking filter

15 is a block diagram showing a detailed configuration of the deblocking filter 24 according to the first working example for performing the above-described parallel processing.

(2-1) Determine the block

The determination block 110 includes vertical boundary determination sections 112-1 to 112-n and horizontal boundary determination sections 114-1 to 114-n. The vertical boundary determination section 112 and the horizontal boundary determination section 114 are supplied with an image input to the deblocking filter 24 and determination information used to determine the necessity of filtering.

The vertical boundary determination section 112 determines whether to apply the deblocking filter to the vertical boundary using the pixel values of the reference pixels belonging to the rows to which the deblocking filter for the horizontal boundary is not applied as shown in FIG11. In this example, the pixel values of the reference pixels are input to the deblocking filter 24. The vertical boundary determination section 112 outputs information indicating the determination result regarding each vertical boundary (e.g., binary information indicating that a value of "1" indicates a determination result that the deblocking filter needs to be applied) to the horizontal filtering block 130.

The horizontal boundary determination unit 114 determines whether to apply the deblocking filter to the horizontal boundary using the pixel values of the reference pixels belonging to the columns to which the deblocking filter for the vertical boundary is not applied, as shown in FIG12. In this example, the pixel values of the reference pixels are also input to the deblocking filter 24. The determination process performed by each horizontal boundary determination unit 114 is performed in parallel with the determination process performed by each vertical boundary determination unit 112. The horizontal boundary determination unit 114 outputs information indicating the determination result regarding each horizontal boundary to the vertical filtering block 140.

16 is a block diagram showing a detailed configuration of each vertical boundary determination section 112 and horizontal boundary determination section 114. Referring to FIG16 , each determination section includes a tap formation section 121, a calculation section 122, a threshold comparison section 123, a distortion evaluation section 124, and a filter determination section 125.

The tap formation unit 121 obtains reference pixel values from the pixel values of two adjacent blocks around the boundary of interest in the input image and forms taps (a set of reference pixel values) for determining the determination condition B for the above-mentioned luminance component. For example, the vertical boundary of each block having a size of 8×8 pixels can be used for attention. In this case, the tap formation unit 121 forms taps based on the pixel values of the fourth and/or fifth rows belonging to the two blocks on the left and right. If the horizontal boundary is of interest, the tap formation unit 121 forms taps based on the pixel values of the fourth and/or fifth columns belonging to the two blocks above and below. The calculation unit 122 assigns the taps formed by the tap formation unit 121 to the left-hand side of the determination expression in the determination condition B and calculates the edge value to be compared with the edge determination threshold β. The threshold comparison unit 123 compares the value calculated by the calculation unit 122 with the edge determination threshold β and outputs the comparison result to the filter determination unit 125.

The distortion evaluation unit 124 evaluates the determination condition A for the luminance component using the mode information (MB mode), transform coefficient information, and motion vector information provided as the determination information. The distortion evaluation unit 124 outputs the evaluation result to the filter determination unit 125. Based on the mode information, the distortion evaluation unit 124 evaluates only the determination condition A1 for the chrominance component.

The filter determination unit 125 determines whether to apply the deblocking filter to the attention boundary based on the comparison result of the determination condition B supplied from the threshold comparison unit 123 and the evaluation result of the determination condition A supplied from the distortion evaluation unit 124. The filter determination unit 125 outputs information indicating the determination result.

(2-2) Horizontal filter block

15 , the configuration of the deblocking filter 24 will be further described. The horizontal filtering block 130 includes horizontal filtering sections 132 - 1 to 132 - n. The horizontal filtering section 132 is supplied with an input image and a determination result from the determination block 110 regarding each vertical boundary.

If the determination result from vertical boundary determination unit 112 indicates that a filter is required, horizontal filtering unit 132 applies a deblocking filter to the pixels surrounding the corresponding vertical boundary. For the filtered pixels, horizontal filtering unit 132 outputs the filtered pixel values to vertical filtering block 140. For the other pixels, horizontal filtering unit 132 outputs the pixel values of the input image to vertical filtering block 140.

(2-3) Vertical filter block

The vertical filtering block 140 includes vertical filtering sections 142-1 to 142-n. The vertical filtering section 142 is supplied with an input image and a determination result from the determination block 110 regarding each horizontal boundary.

If the determination result from horizontal boundary determination unit 114 indicates that a filter needs to be applied, vertical filtering unit 142 applies deblocking filtering for the horizontal boundary to the upper and lower pixels surrounding the corresponding horizontal boundary. For the filtered pixels, vertical filtering unit 142 outputs the filtered pixel values, and for the other pixels, vertical filtering unit 142 outputs the pixel values provided by horizontal filtering block 130. The output from each vertical filtering unit 142 may be included in the output image from deblocking filter 24.

(2-4) Parallelization Control Unit

The parallelization control section 150 controls the parallelism of the filtering necessity determination process in the determination block 110 and the parallelism of the filtering processes in the horizontal filtering block 130 and the vertical filtering block 140 .

For example, the parallelization control unit 150 can control the degree of parallelism for processing each block based on the input image size. More specifically, if the input image size is relatively large, the parallelization control unit 150 increases the degree of parallelism for processing each block. This can adaptively prevent delays or data bitrate degradation caused by the increased processing volume based on image size. For example, the parallelization control unit 150 can control the degree of parallelism for processing each block based on a sequence parameter set, a picture parameter set, or parameters contained in a slice header. This allows for flexible configuration of the degree of parallelism based on the requirements of the user developing the device. For example, the degree of parallelism can be configured based on installation environment constraints such as the number of processor cores or the number of software threads.

The working example enables parallelization of processing across macroblocks. This demonstrates that the order in which blocks are processed within an image has no effect on the final output result. Therefore, the parallelization control unit 150 can control the order in which filtering needs to be determined in the determination block 110, and the order in which filtering is performed in the horizontal filter block 130 and the vertical filter block 140 on a block-by-block basis.

More specifically, the parallelization control unit 150 can control the order of filtering based on the dependencies between the filtering processes between macroblocks. According to conventional techniques, for example, the dependencies between adjacent macroblocks near slice boundaries can delay the parallel processing of each slice within an image. However, the parallelization control unit 150 according to the working example can perform filtering on adjacent macroblocks near slice boundaries before other macroblocks.

For example, FIG17 shows eight macroblocks MB10 to MB13 and MB20 to MB23 around a slice boundary. Macroblocks MB10 to MB13 belong to slice SL1. Macroblocks MB20 to MB23 belong to slice SL2. With respect to these macroblocks, the filtering process for the horizontal boundary of macroblock MB20 in slice SL2 depends on the filtering process for the vertical boundary of macroblock MB12 in slice SL1. Similarly, the filtering process for the horizontal boundary of macroblock MB21 in slice SL2 depends on the filtering process for the vertical boundary of macroblock MB13 in slice SL1.

Under these conditions, according to the example of FIG. 18 , the parallelization control unit 150 prioritizes filtering the vertical boundary between macroblocks MB12 and MB13 over other boundaries in the filtering process for slice SL1. This prevents significant delays in filtering the horizontal boundary between macroblocks MB20 and MB21 during the filtering process for slice SL2. The example of FIG. 19 initially performs filtering on vertical boundaries in parallel for all macroblocks included in slice SL1. Furthermore, in this case, no delay occurs in filtering the horizontal boundary between macroblocks MB20 and MB21 in slice SL2.

[3-2. Processing Flow]

20 to 22 , the processing flow of the deblocking filter 24 will be described.

(1) First Scenario

Fig. 20 is a flowchart showing an example of a process flow of the deblocking filter 24 according to the first scenario. The first scenario corresponds to an example of a large degree of parallelism as shown in Fig. 13 .

20 , vertical boundary determination units 112-1 to 112-n concurrently determine whether filtering is required for the vertical boundaries of all macroblocks included in the input image (step S102). Horizontal boundary determination units 114-1 to 114-n concurrently determine whether filtering is required for the horizontal boundaries of all macroblocks included in the input image (step S104). Steps S102 and S104 are also performed in parallel.

The horizontal filtering units 132-1 to 132-n apply deblocking filters in parallel to all vertical boundaries determined to require deblocking filters in step S102 (step S110). The vertical filtering units 142-1 to 142-n apply deblocking filters in parallel to all horizontal boundaries determined to require deblocking filters in step S104 (step S120).

(2) Second Scenario

Fig. 21 is a flowchart showing an example of a process flow of the deblocking filter 24 according to the second scenario. The second scenario corresponds to an example of a small degree of parallelism as shown in Fig. 14 .

21 , vertical boundary determination units 112-1 to 112-n concurrently determine whether filtering is required for all vertical boundaries of the macroblock of interest within the input image (step S202). Horizontal boundary determination units 114-1 to 114-n concurrently determine whether filtering is required for all horizontal boundaries of the macroblock of interest (step S204). Steps S202 and S204 are also performed in parallel.

The horizontal filtering units 132 - 1 to 132 - n apply the deblocking filter in parallel to the vertical boundary in the target macroblock determined in step S202 as requiring application of the deblocking filter (step S210 ).

The processing of step S220 is for the macroblock of interest in the previous loop. The processing of step S220 can be skipped for the first macroblock of interest. Vertical flattening filter units 142-1 to 142-n apply deblocking filters in parallel to the horizontal boundaries determined in step S204 in the previous loop as requiring deblocking filters (step S220).

If there is still a macroblock of interest that has not been processed in the input image, the process of steps S202 to S220 is repeated for the newly-imported macroblock (step S230).

If there is no unprocessed macroblock of interest, the vertical filtering units 142 - 1 to 142 - n determine that the macroblock of interest in the final loop is a horizontal boundary requiring application of the deblocking filter, and apply the deblocking filter in parallel (step S240 ). The process then terminates.

Although two typical cases of parallel processing in units of images and macroblocks have been described, these two cases are merely examples for illustration. The processing of the deblocking filter 24 can be parallelized in various units, such as a given number of macroblocks (two or four macroblocks), or a group of horizontally or vertically arranged blocks.

(3) Filtering needs to be determined

FIG. 22 is a flowchart showing the flow of the filtering necessity determination process corresponding to steps S102 and S104 of FIG. 20 and steps S202 and S204 of FIG. 21 .

22 , the distortion evaluator 124 evaluates the distortion of each boundary based on the mode information, transform coefficient information, and motion vector information (step S130). If the evaluation result indicates that distortion exists (determination condition A is true), the process proceeds to step S134. If the evaluation result indicates that distortion does not exist, the process proceeds to step S140 (step S132).

In step S134, the calculation unit 122 calculates an edge value based on the reference pixel constructed by the tap construction unit 121 (step S134). The threshold comparison unit 123 compares the calculated value with the edge determination threshold β (step S136). If the edge value is less than the threshold β (determination condition B is true), the process proceeds to step S138. If the edge value is not less than the threshold β, the process proceeds to step S140.

In step S138, the filter determination section 125 determines to apply the deblocking filter to the boundary to be determined (step S138). In step S140, the filter determination section 125 determines not to apply the deblocking filter to the boundary to be determined (step S140).

<4. Second Working Example>

The first working example performs filtering necessity determination processing on a given block using pixel values of pixels that have not been updated by filtering processing on other blocks. In contrast, the second working example described below provides a memory for storing pixel values input to the deblocking filter, thereby eliminating limitations on filtering necessity determination processing and enabling the use of a wider variety of determination conditions.

[4-1. Deblocking filter configuration example]

(1) Description of each part

23 is a block diagram showing a detailed configuration of the deblocking filter 24 according to the second working example. Referring to FIG23 , the deblocking filter 24 includes a line memory 208 , a determination block 210 , a horizontal filtering block 130 , a vertical filtering block 140 , and a parallelization control section 150 .

Line memory 208 stores pixel values of the input image to be supplied to deblocking filter 24. Filtering processing in horizontal filter block 130 and vertical filter block 140 does not update pixel values stored in line memory 208. Filtering necessity determination processing performed by various components in determination block 210, described below, references pixel values stored in line memory 208. The device includes another memory for a purpose other than processing by deblocking filter 24. This memory can be reused (shared) as line memory 208.

The determination block 210 includes vertical boundary determination units 212-1 to 212-n and horizontal boundary determination units 214-1 to 214-n. The vertical boundary determination unit 212 and the horizontal boundary determination unit 214 are supplied with pixel values of an image stored in the line memory 208 for input to the deblocking filter 24 and determination information used to determine the need for filtering.

The vertical boundary determination section 212 determines whether to apply the deblocking filter to each vertical boundary using the pixel values input to the deblocking filter 24. The vertical boundary determination section 212 outputs a determination result indicating information about each vertical boundary to the horizontal filtering block 130.

The horizontal boundary determination section 214 also determines whether to apply the deblocking filter to each horizontal boundary using the pixel values input to the deblocking filter 24. The horizontal boundary determination section 214 performs determination processing in parallel with the determination processing performed by the vertical boundary determination section 212. The horizontal boundary determination section 214 outputs information indicating the determination result regarding each horizontal boundary to the vertical filtering block 140.

(2) Various determination conditions

Similar to the prior art shown in FIG4 , vertical boundary determination unit 212 according to the working example can refer to the pixels in the third and sixth rows of a block to determine the necessity of vertical boundary filtering for each block. However, in this case, the pixel values to be referenced are stored in line memory 208 and are part of the image input to deblocking filter 24. Similarly, horizontal boundary determination unit 214 can refer to the pixels in the third and sixth columns of a block to determine the necessity of horizontal boundary filtering for each block. In this case, the configuration according to the working example can be easily provided without changing the determination conditions for the filtering necessity determination process installed in the existing device.

Vertical boundary determination unit 212 may refer to pixels in more than three rows of a block during its determination. Similarly, horizontal boundary determination unit 214 may refer to pixels in more than three columns of a block during its determination. Vertical boundary determination unit 212 and horizontal boundary determination unit 214 may use determination conditional expressions that differ from those used in the prior art. Six examples of determination techniques according to a working example will be described below with reference to Figures 24 to 26 .

(2-1) First Example

FIG24 is an explanatory diagram illustrating first and second examples of the determination technique. In the first and second examples, the filtering necessity determination process for the vertical boundary (particularly, the determination using determination condition B for the luminance component) refers to the pixels of all the first to eighth rows L1 to L8 in each block. The filtering necessity determination process for the horizontal boundary also refers to the pixels of all the first to eighth columns in each block.

First Example The determination condition for the luminance component may be defined as follows.

Determination conditions for the Luma component ... If both conditions A and B are true, the deblocking filter is applied.

Condition A:

(A1) Block Ba or Bb enters intra prediction mode;

(A2) block Ba or Bb has a non-zero orthogonal transform coefficient; or

(A3) |MVAx-MVBx|≥4 or |MVAy-MVBy|≥4

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₇ -2p ₁₇ +p ₀₇ |+|q ₂₇ -2q ₁₇ +q ₀₇ |

iD ₁ =|p ₂₁ -2p ₁₁ +p ₀₁ |+|q ₂₁ -2q ₁₁ +q ₀₁ |+|p ₂₆ -2p ₁₆ +p ₀₆ |+|q ₂₆ -2q ₁₆ +q ₀₆ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |

iD ₃ =|p ₂₃ -2p ₁₃ +p ₀₃ |+|q ₂₃ -2q ₁₃ +q ₀₃ |+|p ₂₄ -2p ₁₄ +p ₀₄ |+|q ₂₄ -2q ₁₄ +q ₀₄ |

iD _ave =(iD ₀ +iD ₁ +iD ₂ +iD ₃ )>>2

Under this condition, iD _ave < β

The determination conditions for the chrominance component may be the same as those in the prior art described above. A weighted average may be calculated to calculate the average iD _ave of the four determination parameters iD ₀ to iD ₃ .

(2-2) Second Example

Second Example The determination condition B for the luminance component may be defined as follows.

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₇ -2p ₁₇ +p ₀₇ |+|q ₂₇ -2q ₁₇ +q ₀₇ |

iD ₁ =|p ₂₁ -2p ₁₁ +p ₀₁ |+|q ₂₁ -2q ₁₁ +q ₀₁ |+|p ₂₆ -2p ₁₆ +p ₀₆ |+|q ₂₆ -2q ₁₆ +q ₀₆ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |

iD ₃ =|p ₂₃ -2p ₁₃ +p ₀₃ |+|q ₂₃ -2q ₁₃ +q ₀₃ |+|p ₂₄ -2p ₁₄ +p ₀₄ |+|q ₂₄ -2q ₁₄ +q ₀₄ |

Under this condition, iD ₀ <β, and iD ₁ <β, and iD ₂ <β, and iD ₃ <β

The equations for calculating the four determination parameters iD ₀ to iD ₃ are the same as those in the first example. The applicable condition is that not all of the four determination parameters iD ₀ to iD ₃ are used, but at least three, two, or one of them is smaller than the edge determination threshold β.

(2-3) Third Example

FIG25 is an explanatory diagram illustrating third and fourth examples of the determination technique. In the third and fourth examples, the filtering necessity determination process for vertical boundaries (particularly, the determination using determination condition B for the luminance component) refers to pixels in four rows, L1, L3, L6, and L8, in each block. The filtering necessity determination process for horizontal boundaries also refers to pixels in four columns, in each block.

Third Example The determination condition for the luminance component may be defined as follows.

Determination conditions for the Luma component ... If both conditions A and B are true, the deblocking filter is applied.

Condition A:

(A1) Block Ba or Bb enters intra prediction mode;

(A2) block Ba or Bb has a non-zero orthogonal transform coefficient; or

(A3) |MVAx-MVBx|≥4 or |MVAy-MVBy|≥4

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₇ -2p ₁₇ +p ₀₇ |+|q ₂₇ -2q ₁₇ +q ₀₇ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |

iD _ave = (iD ₀ + iD ₂ ) >> 1

Under this condition, iD _ave < β

The determination conditions for the chrominance component may be the same as those in the prior art. A weighted average may be calculated to calculate the average iD _ave of the two determination parameters iD ₀ and iD ₂ .

(2-4) Fourth Example

Fourth Example The determination condition B for the luminance component may be defined as follows.

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₇ -2p ₁₇ +p ₀₇ |+|q ₂₇ -2q ₁₇ +q ₀₇ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₅ -2p ₁₅ +p ₀₅ |+|q ₂₅ -2q ₁₅ +q ₀₅ |

Under this condition, iD ₀ <β, and iD ₂ <β

The equations for calculating the two determination parameters iD ₀ and iD ₂ are the same as those in the third example. The applicable condition is that not all of the two determination parameters iD ₀ and iD ₂ but one of them is smaller than the edge determination threshold β.

Although an example has been described in which the first, third, sixth, and eighth rows (or columns) L1, L3, L6, and L8 in a block are referred to in the determination, other combinations of rows or columns may be referred to.

(2-5) Fifth Example

FIG26 is an explanatory diagram illustrating fifth and sixth examples of the determination technique. In the fifth and sixth examples, the determination process for filtering the vertical boundary requires reference to pixels in four rows (L1, L3, L5, and L7) in each block. The determination process for filtering the horizontal boundary also requires reference to pixels in four columns in each block.

Fifth Example The determination condition for the luminance component may be defined as follows.

Determination conditions for the Luma component ... If both conditions A and B are true, the deblocking filter is applied.

Condition A:

(A1) Block Ba or Bb enters intra prediction mode;

(A2) block Ba or Bb has a non-zero orthogonal transform coefficient; or

(A3) |MVAx-MVBx|≥4 or |MVAy-MVBy|≥4

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₆ -2p ₁₆ +p ₀₆ |+|q ₂₆ -2q ₁₆ +q ₀₆ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₄ -2p ₁₄ +p ₀₄ |+|q ₂₄ -2q ₁₄ +q ₀₄ |

iD _ave = (iD ₀ + iD ₂ ) >> 1

Under this condition, iD _ave < β

The determination conditions for the chrominance component may be the same as those in the prior art. A weighted average may be calculated to calculate the average iD _ave of the two determination parameters iD ₀ and iD ₂ .

(2-6) Sixth Example

Sixth Example The determination condition B for the luminance component may be defined as follows.

Condition B:

iD ₀ =|p ₂₀ -2p ₁₀ +p ₀₀ |+|q ₂₀ -2q ₁₀ +q ₀₀ |+|p ₂₆ -2p ₁₆ +p ₀₆ |+|q ₂₆ -2q ₁₆ +q ₀₆ |

iD ₂ =|p ₂₂ -2p ₁₂ +p ₀₂ |+|q ₂₂ -2q ₁₂ +q ₀₂ |+|p ₂₄ -2p ₁₄ +p ₀₄ |+|q ₂₄ -2q ₁₄ +q ₀₄ |

Under this condition, iD ₀ <β, and iD ₂ <β

The equations for calculating the two determination parameters iD ₀ and iD ₂ are the same as those in the fifth example. The applicable condition is that not all but one of the two determination parameters iD ₀ and iD ₂ is smaller than the edge determination threshold β.

Generally speaking, increasing the number of rows and columns referenced for determination improves determination accuracy. Therefore, the first and second examples of referencing eight rows and columns can minimize the possibility of filtering blocks that are not originally suitable for deblocking filtering, as well as the possibility of filtering blocks that are not originally suitable for deblocking filtering. This results in improved quality of the image to be encoded and decoded. On the other hand, reducing the number of rows and columns referenced for determination can reduce processing costs. Because there is a trade-off between image quality and processing costs, it is advantageous to be able to adaptively select the number of rows and columns referenced for determination, depending on the limitations of device usage or installation. It is also advantageous to be able to adaptively select a combination of rows and columns to be referenced.

As explained in the first, third, and fifth examples, the average value iD _ave of the determination parameters can be compared with the edge determination threshold β to appropriately perform determination on a block basis without excessive influence of parameter variations for each row or column.

[4-2. Processing Flow]

According to the second working example and the first working example, the deblocking filter 24 can operate with different degrees of parallelism.

In the first scenario using a high degree of parallelism, vertical boundary determination units 212-1 through 212-n determine in parallel whether filtering is required for all vertical boundaries of macroblocks included in the input image. Horizontal boundary determination units 214-1 through 214-n determine in parallel whether filtering is required for all horizontal boundaries of macroblocks included in the input image. Determination of vertical and horizontal boundaries is also performed in parallel. Horizontal filtering units 132-1 through 132-n and vertical filtering units 142-1 through 142-n apply deblocking filters to the vertical and horizontal boundaries determined to require deblocking filters (see FIG. 20 ).

In the second scenario using a low degree of parallelism, vertical boundary determination units 212-1 through 212-n concurrently determine whether filtering is required for all vertical boundaries of a macroblock of interest within the input image. Horizontal boundary determination units 214-1 through 214-n concurrently determine whether filtering is required for all horizontal boundaries within the macroblock of interest. The vertical and horizontal boundary determination processes are also performed in parallel. Horizontal filtering units 132-1 through 132-n then concurrently apply deblocking filters to the vertical boundaries determined to require deblocking filters. Vertical filtering units 142-1 through 142-n concurrently apply deblocking filters to the horizontal boundaries determined to require deblocking filters. This process is repeated for all macroblocks within the input image (see FIG. 21 ).

These two cases are merely examples for illustration. Furthermore, in the second working example, the processing of the deblocking filter 24 can be parallelized in various units, such as a given number of macroblocks (two or four macroblocks) or a group of horizontally or vertically positioned blocks. Furthermore, in both the second working example and the first working example, the parallelization control unit 150 can control the degree of parallelism and order of processing in the deblocking filter 24.

[4-3. Processing example for each LCU

As already mentioned, the technology according to the various working examples described in this specification can be provided as processing based on HEVC coding units (CUs). According to HEVC, the coding unit with the largest size is called the largest coding unit (LCU), which can be selected as 64×64 pixels, for example. The minimum selectable CU size is 8×8 pixels. Generally, an image is encoded and decoded corresponding to each LCU in a raster scan order from the upper left of the picture (or slice). An example of processing corresponding to the LCU in the deblocking filter 24 is described below.

Fig. 27 is an explanatory diagram showing the processing order for each LCU according to the second working example described above. This example assumes that the LCU size is 16×16 pixels and the CU size is 8×8 pixels.

Referring to Figure 27, the first stage is shown in the upper left of the figure, indicating that the filtering of LCUs has been completed up to the (n-1)th LCU. The shaded pixels are the objects of filtering as vertical boundaries. The padded pixels are the objects of filtering as horizontal boundaries.

The processing for the second stage in the upper right of Figure 27 and the processing for the third stage in the lower left are for the nth LCU. Before the second stage, the pixel values input to the deblocking filter 24 are used to perform filtering in parallel on all vertical and horizontal boundaries belonging to the nth LCU to determine the need for filtering. In the second stage, the pixel values input to the deblocking filter 24 are used to perform filtering in parallel on the vertical boundaries belonging to the nth LCU and determined to require the application of the deblocking filter. The pixel values processed in the second stage are then used to perform filtering in parallel on the horizontal boundaries belonging to the nth LCU and determined to require the application of the deblocking filter.

The fourth stage of processing in the lower right corner of Figure 27 is for the (n+1)th LCU. In the fourth stage, after the filtering necessity determination process is performed in parallel on all vertical and horizontal boundaries belonging to the (n+1)th LCU, the filtering process is performed in parallel on the vertical boundaries determined to require the application of the deblocking filter.

Although the LCU size is assumed to be 16×16 pixels in this example, it can also be set to 32×32 or 64×64 pixels. Since increasing the size of the LCU to be selected also increases the number of vertical and horizontal boundaries belonging to one LCU, the effect of shortening the processing time due to parallelization is further improved.

FIG28 is a flowchart showing the processing flow of the deblocking filter 24 for each LCU.

28 , the vertical boundary determination units 212-1 to 212-n concurrently determine whether filtering is required for all vertical boundaries of the LCU of interest contained in the input image (step S252). The horizontal boundary determination units 214-1 to 214-n concurrently determine whether filtering is required for all horizontal boundaries of the LCU of interest (step S254). Steps S252 and S254 are also performed in parallel.

The horizontal filtering units 132 - 1 to 132 - n apply the deblocking filter in parallel to the vertical boundary in the focus LCU determined in step S252 to require application of the deblocking filter (step S260 ).

The vertical filtering units 142 - 1 to 142 - n apply the deblocking filter in parallel to the horizontal boundary in the target LCU determined in step S254 to require application of the deblocking filter (step S270 ).

If there is an unprocessed LCU of interest in the input image (step S280), the process of steps S252 to S270 is repeated for the newly-processed LCU of interest. If there is no unprocessed LCU, the process is terminated.

<5. Third Working Example>

[5-1. Summary]

According to the second working example, the filtering process for vertical and horizontal boundaries requires determination of the pixel values input to the deblocking filter, thereby eliminating the dependency between the filtering process and enabling parallelization of the determination process. The third working example described below applies the concepts of the second working example to the filtering process. The filtering process for vertical and horizontal boundaries also filters the pixel values input to the deblocking filter.

FIG29 is an explanatory diagram showing an overview of this working example. The lower left portion of FIG29 shows the shape of an input pixel (also referred to as a reconstructed pixel) before being deblocked. This working example enables the pixel values referenced for deblocking filtering to be determined not only from the filtering needs for vertical and horizontal boundaries, but also from the pixel values referenced for the filtering process for vertical boundaries and the filtering process for horizontal boundaries. Thus, the dependency between the two filtering processes is eliminated. The two filtering processes are performed in parallel.

The filtering process for vertical boundaries and the filtering process for horizontal boundaries can update the values of repeated pixels. The padded pixels in Figure 29 illustrate the locations of possible repeated pixels. The deblocking filter according to the working example calculates a single output pixel value based on the two filter outputs for pixels that are repeatedly updated by two filters running in parallel.

[5-2. Deblocking filter configuration example]

FIG30 is a block diagram illustrating the detailed configuration of the deblocking filter 24 according to the third working example. Referring to FIG30 , the deblocking filter 24 includes a line memory 208, a determination block 210, a horizontal filtering block 330, a vertical filtering block 340, a parallelization control unit 150, and a calculation unit 360. The determination block 210 includes vertical boundary determination units 212-1 through 212-n and horizontal boundary determination units 214-1 through 214-n. The vertical boundary determination unit 212 and the horizontal boundary determination unit 214 can determine the necessity of boundary filtering based on the various determination conditions described in the second working example.

The horizontal filtering block 330 includes horizontal filtering sections 332 - 1 to 332 - n. The horizontal filtering section 332 is supplied with the input image value from the line memory 208 and the determination result from the determination block 210 regarding each vertical boundary.

If the determination result from the vertical boundary determination unit 212 indicates that a filter is required, the horizontal filtering unit 332 applies a deblocking filter for the vertical boundary to the left and right pixels surrounding the corresponding vertical boundary. For the filtered pixels, the horizontal filtering unit 332 outputs the filtered pixel values to the calculation unit 360, while for the other pixels, the horizontal filtering unit 332 outputs the input pixel values to the calculation unit 360.

The vertical filtering block 340 includes vertical filtering sections 342-1 to 342-n. The vertical filtering section 342 is supplied with the input pixel value from the line memory 208 and the determination result from the determination block 210 regarding each horizontal boundary.

If the determination result from the horizontal boundary determination unit 214 indicates that a filter is required, the vertical filter unit 342 applies a deblocking filter for the horizontal boundary to the upper and lower pixels surrounding the corresponding horizontal boundary. The filtering process by the vertical filter units 342-1 through 342-n is performed in parallel with the filtering process by the horizontal filter units 332-1 through 332-n. For the filtered pixels, the vertical filter unit 342 outputs the filtered pixel values to the calculation unit 360, while for the other pixels, the vertical filter unit 342 outputs the input pixel values to the calculation unit 360.

The calculation section 360 is supplied in parallel with the output pixel values from the horizontal filtering block 330 and the output pixel values from the vertical filtering block 340. Furthermore, the calculation section 360 is supplied with the determination results from the vertical boundary determination section 212 and the horizontal boundary determination section 214. Based on the determination results, the calculation section 360 calculates output pixel values for the pixels filtered by the horizontal filtering block 330 and the vertical filtering block 340, based on the filter outputs from the horizontal filtering block 330 and the vertical filtering block 340.

According to a working example, for example, the calculation unit 360 calculates the average of the two filter outputs for the repeatedly filtered pixels. The calculation unit 360 may calculate a simple average of the two filter outputs. Alternatively, the calculation unit 360 may calculate a weighted average of the two filter outputs. For example, the calculation unit 360 may determine the weight for the weighted average of the pixels based on the distance from each pixel to the vertical boundary and the distance from the horizontal boundary.

FIG31 is an explanatory diagram illustrating the determination of weights for the weighted average calculated by the calculation unit 360. FIG31 shows that the pixel of interest _PZ is black, corresponding to one of the repetition positions shown in FIG29 . The distance _DD between the pixel of interest _PZ and the nearest vertical boundary _VZ is three pixels. The distance DH between the pixel of interest _PZ and the nearest horizontal boundary _HZ is two pixels. Distance _DH is smaller _than distance _DD . In this case, the calculation unit 360 may set the weighting of the output of the deblocking filter applied to the horizontal boundary _HZ to be greater than the weighting of the output of the deblocking filter applied to the vertical boundary _VZ . The example of FIG31 assumes that the ratio of the filter output _Vout for the vertical boundary _VZ to the filter output _Hout for the horizontal boundary _HZ is 2:3.

As shown in Figure 31, calculating the weighted average of the outputs of the two filters can provide each pixel of interest with an output pixel value similar to that of a two-dimensional filter with filter taps along the horizontal direction and filter taps along the vertical direction. Parallelizing the filtering process for vertical and horizontal boundaries can also appropriately reduce the block distortion that occurs at the vertical and horizontal boundaries. As another working example, the deblocking filter 24 can include a two-dimensional filter that simultaneously calculates horizontal filtering, vertical filtering, and weighted averaging. However, in this case, the installation is very complicated because the filter coefficients need to be changed in various ways corresponding to the pixels. On the other hand, the third working example executes two one-dimensional filters in parallel and then calculates the weighted average. This can easily provide processing that is essentially equivalent to a two-dimensional filter and ensures the functionality of the existing deblocking filter.

FIG32 is an explanatory diagram showing an example of weights for weighted averaging calculated based on the example of FIG31 . FIG32 shows 36 pixels (6×6) around one intersection between a vertical boundary and a horizontal boundary. The pixels correspond to the repeated positions described above. For pixels located equidistant from the vertical boundary and the horizontal boundary, the ratio of the weight of the filter output V _out to the weight of the filter output H _out is 1:1 (2:2 or 3:3). For pixels closer to the vertical boundary, the weight of the filter output V _out is greater than the weight of the filter output H _out . For example, the ratio of the weight of pixel P ₁ is V _out : H _out = 3:1. For pixels closer to the horizontal boundary, the weight of the filter output V _out is less than the weight of the filter output H _out . For example, the ratio of the weight of pixel P ₂ is V _out : H _out = 1:3.

By changing the weight used for weighted averaging according to the distance between each pixel and the boundary, block distortion can be more effectively suppressed and image quality can be improved.

The above weights are merely examples. For example, the calculation unit 360 may determine the weights for the weighted average of the pixels based on the edge strength corresponding to the vertical and horizontal boundaries of each pixel, instead of or in addition to the distance between each pixel and the boundary. The edge strength may be represented by a parameter such as the edge value calculated by the calculation unit 122 as shown in FIG16 . In this case, the weight of the filter output for boundaries with stronger edges may be set to be greater than the weight of the filter output for boundaries with weaker edges. By varying the weighted average according to the edge strength, the effect of the deblocking filter on boundaries that significantly cause block distortion can be adaptively improved.

The calculation unit 360 selects the output from the block actually filtered for pixels filtered by one of the horizontal filtering block 330 and the vertical filtering block 340. For pixels not filtered by the horizontal filtering block 330 or the vertical filtering block 340, the calculation unit 360 directly outputs the input pixel value to be output to the deblocking filter 24. The table of FIG33 lists the output pixel values from the calculation unit 360 according to the result of determining whether filtering is required.

[5-3. Processing order example]

Two examples of processing sequences that can be used for the deblocking filter 24 according to a working example are described below. The examples also assume that the deblocking filter is provided with an image of size 32×32 pixels. The input image contains four macroblocks MB0 to MB3, each of size 16×16 pixels.

(1) First Example

For comparison, FIG34 illustrates a processing sequence that maintains a dependency between filtering processes for vertical boundaries and filtering processes for horizontal boundaries. The processing sequence of FIG34 is essentially equivalent to the processing sequence of FIG13 according to the first working example. In FIG34 , the first step involves performing filtering operations J _V0,0 to J _V3,3 and J _H0,0 to J _H3,3 in parallel on all vertical boundaries and all horizontal boundaries of all four macroblocks MB0 to MB3. The second step involves performing filtering operations F V0,0 to F V3,3 on the 16 vertical boundaries of the four macroblocks MB0 to MB3. The third step involves performing filtering operations _{F H0,0 to} F _H3,3 on the 16 horizontal boundaries _of the four macroblocks MB0 to MB3. The fourth step (omitted from FIG13 ) stores the pixel values after filtering of the horizontal boundaries in the memory used for the output from the deblocking filter 24.

FIG35 illustrates a first example of the processing sequence provided by the working example. In FIG35 , the first step performs filtering in parallel on all vertical boundaries and all horizontal boundaries of the four macroblocks MB0 to MB3, determining that filtering is required (J _V0,0 to J _V3,3 and J _H0,0 to J _H3,3 ). The second step performs filtering in parallel on all vertical and horizontal boundaries of the four macroblocks MB0 to MB3, determining that filtering is required (F _V0,0 to F _V3,3 and F _H0,0 to F _H3,3 ). In practice, the second step only filters boundaries determined to require filtering. The third step stores pixel values in the memory used for outputs from the deblocking filter 24. For pixels filtered by the horizontal filter block 330 and the vertical filter block 340, a weighted average of the two filter outputs can be calculated as the output pixel value.

(2) Second Example

The first example maximizes the degree of parallelism, but the deblocking filter 24 according to the second example is also capable of performing processing for each macroblock.

For comparison, Figure 36 shows the processing sequence for each macroblock when the dependency between filtering for vertical and horizontal boundaries is maintained. The processing sequence in Figure 36 is essentially equivalent to the processing sequence in Figure 14 according to the first working example. Figure 36 explicitly shows four processing steps (steps 6, 10, 14, and 16) that store pixel values in output memory and are omitted from Figure 14 for simplicity. The processing in Figure 36 consists of 16 processing steps, including these four steps.

FIG37 illustrates a second example of the processing sequence provided by the working example. In FIG37 , the first step involves performing filtering operations J _V0,0 through J _V0,3 and J _H0,0 through J _H0,3 in parallel on the four vertical and four horizontal boundaries of macroblock MB0. The second step involves performing filtering operations F _V0,0 through F _V0,3 and F _H0,0 through F _H0,3 in parallel on the four vertical and four horizontal boundaries of macroblock MB0. The third step stores the pixel values of macroblock MB0 in the memory used for the output from deblocking filter 24. For pixels repeatedly filtered by two filters, a weighted average of the two filter outputs is calculated as the output pixel value. Steps 4 through 6 process macroblock MB1 in a similar manner. Steps 7 through 9 process macroblock MB2 in a similar manner. Steps 10 through 12 process macroblock MB3 in a similar manner. The process in FIG37 includes 12 processing steps, which is fewer than the process in FIG36 .

The third working example eliminates the dependency between the filtering process for the vertical boundary and the filtering process for the horizontal boundary. The processing of the deblocking filter 24 can be performed using fewer processing steps than for the first and second working examples. One advantage of allowing the filtering process to refer only to the pixels input to the deblocking filter is that any configuration of the filter taps will not create a dependency between the filtering process for the vertical boundary and the filtering process for the horizontal boundary. By configuring the filter taps using more pixels than the prior art, the third working example can improve image quality. For example, as illustrated in reference FIG7 , the prior art uses filter taps of three pixels on each side of each boundary. Even if filter taps of more than 5 pixels are used for each boundary, the working example will not create a dependency between the processing. Even if the block size as the processing unit of the deblocking filter is further reduced, no dependency will be created between the processing.

In addition, in the third operation example and the first and second operation examples, the parallelization control unit 150 can control the degree of parallelism and the order of processing in the deblocking filter 24 .

[5-4. Processing Flow]

Fig. 38 is a flowchart showing an example of a process flow for a deblocking filter according to the third working example. Fig. 39 is a flowchart showing a flow of a pixel value calculation process shown in Fig. 38 .

38 , the vertical boundary determination units 212-1 to 212-n concurrently determine whether filtering is required for all vertical boundaries within the input image or macroblock (step S302). The horizontal boundary determination units 214-1 to 214-n concurrently determine whether filtering is required for all horizontal boundaries within the input image or macroblock (step S304). Steps S302 and S304 are also performed in parallel.

The horizontal filtering units 332-1 to 332-n apply deblocking filters in parallel to all vertical boundaries determined to require deblocking filters in step S302 (step S306). The vertical filtering units 342-1 to 342-n apply deblocking filters in parallel to all horizontal boundaries determined to require deblocking filters in step S304 (step S308). Steps S306 and S308 are also performed in parallel.

The calculation section 360 then performs the pixel value calculation process (step S310) shown in Fig. 39. Referring to Fig. 39, the process from step S314 to step S326 loops for each pixel to be processed (step S312).

In step S314, the calculation unit 360 determines whether the two filters for the vertical boundary and the horizontal boundary have filtered the pixel of interest (step S314). If the two filters have filtered the pixel of interest, the process proceeds to step S322. If the two filters have not filtered the pixel of interest, the process proceeds to step S316.

In step S316, the calculation unit 360 determines whether the pixel of interest has been filtered by one of the two filters for the vertical boundary and the horizontal boundary (step S316). If the pixel of interest has been filtered by one of the two filters, the process proceeds to step S320. If the pixel of interest has not been filtered by either filter, the process proceeds to step S318.

In step S318, the calculation section 360 obtains the input pixel value to the deblocking filter 24 (step S318). In step S320, the calculation section 360 obtains the filter output from the filter that actually filters the pixel of interest (step S320).

In step S322, the calculation unit 360 determines a weight value for calculating a weighted average of the filter outputs from the two filters with respect to the pixel of interest based on the distance from the pixel of interest to the vertical and horizontal boundaries, or the edge strength of the vertical and horizontal boundaries corresponding to the pixel of interest (step S322). The calculation unit 360 calculates the weighted average of the filter outputs from the two filters using the determined weight (step S324).

The calculation section 360 stores the pixel value of the pixel of interest in the memory, wherein the pixel value is acquired in step S318 or S320 or calculated in step S324 (step S326). When the processing is performed on all the pixels to be processed, the series of processing shown in Figures 38 and 39 is terminated.

<6. Application to various codecs>

The technology according to the present disclosure can be applied to various codecs related to image encoding and decoding. An example of applying the technology according to the present disclosure to a multi-view codec and a scalable codec is described below.

[6-1. Multi-view codec]

A multi-view codec is an image encoding system that encodes and decodes multi-view videos. Figure 40 is an explanatory diagram showing a multi-view codec. Figure 40 shows a sequence of frames of three views captured at three observation points. Each view is provided with a view ID (view_id). One of these views is designated as a base view. Views other than the base view are referred to as non-base views. The example of Figure 40 shows a base view with a view ID of "0" and two non-base views with a view ID of "1" or "2". By encoding the frames of the non-base views based on the encoding information about the frames of the base view, encoding the multi-view image data can compress the data size of the encoded stream as a whole.

In the encoding and decoding processes of the multi-view codec described above, a deblocking filter can be applied to each view. When a deblocking filter is applied to each view, the filtering need determination process for each view at vertical block boundaries and horizontal block boundaries can be parallelized based on the technology according to the present disclosure. The filtering need determination process and the filtering process for each view can be parallelized. Parameters controlling the filtering need determination process or the filtering process (such as the one described in the previous paragraph 0094) can be provided for each view. The parameters provided to the base view can be reused for non-base views.

The filtering needs determination process or the filtering process can be parallelized on multiple views. The views can share parameters that control the filtering needs determination process or the filtering process (such as the one described in the previous paragraph 0094). It may be advantageous to additionally specify a flag indicating whether the views share parameters.

Fig. 41 is an explanatory diagram showing an image encoding process applicable to the multi-view codec described above. Fig. 41 shows the configuration of a multi-view encoding device 710 as an example. The multi-view encoding device 710 includes a first encoding section 720, a second encoding section 730, and a multiplexing section 740.

The first encoding unit 720 encodes the base view image and generates a base view coded stream. The second encoding unit 730 encodes the non-base view images and generates non-base view coded streams. The multiplexing unit 740 multiplexes the base view coded stream generated by the first encoding unit 720 and one or more non-base view coded streams generated by the second encoding unit 730 to generate a multi-view multiplexed stream.

The first encoding unit 720 and the second encoding unit 730 shown in FIG41 are configured similarly to the image encoding device 10 according to the above-described embodiment. Applying a deblocking filter to each view enables parallelization of the filtering necessity determination process for vertical and horizontal block boundaries, or parallelization of the filtering necessity determination process and the filtering process. Parameters controlling these processes can be inserted into the header region of the coded stream for each view, or into the common header region of the multiplexed stream.

Fig. 42 is an explanatory diagram showing an image decoding process applicable to the multi-view codec described above. Fig. 42 shows the configuration of a multi-view decoding device 760 as an example. The multi-view decoding device 760 includes a demultiplexing section 770, a first decoding section 780, and a second decoding section 790.

The demultiplexing unit 770 demultiplexes the multi-view multiplexed stream into a base view coded stream and one or more non-base view coded streams. The first decoding unit 780 decodes the base view image from the base view coded stream. The second decoding unit 790 decodes the non-base view images from the non-base view coded streams.

The first decoding unit 780 and the second decoding unit 790 shown in FIG42 are configured similarly to the image decoding apparatus 60 according to the above embodiment. Applying a deblocking filter to each view enables parallelization of the filtering necessity determination process for vertical and horizontal block boundaries, or parallelization of the filtering necessity determination process and the filtering process. Parameters controlling these processes can be obtained from the header area of the encoding process for each view, or from a common header area in the multiplexed stream.

[6-2. Scalable Codec]

A scalable codec is an image coding system that provides layered coding. FIG43 is an explanatory diagram showing a scalable codec. FIG43 shows a frame sequence of three layers for different spatial resolutions, temporal resolutions, or image qualities. Each layer is provided with a layer ID (layer_id). These layers include a base layer with the lowest resolution (or image quality). Layers other than the base layer are called enhancement layers. The example of FIG43 shows a base layer with a layer ID of "0" and two enhancement layers with a layer ID of "1" or "2". By encoding the frames of the enhancement layer based on the coding information about the frames of the base layer, encoding the multi-layer image data can compress the data size of the coded stream as a whole.

In the encoding and decoding processes of the scalable codec described above, deblocking filtering can be applied to each layer. When deblocking filtering is applied to each layer, the filtering need determination process for each layer at vertical block boundaries and horizontal block boundaries can be parallelized based on the technology disclosed herein. The filtering need determination process and the filtering process for each layer can be parallelized. Parameters controlling the filtering need determination process or the filtering process (such as the one described in the previous paragraph 0094) can be provided for each layer. The parameters provided to the base layer can be reused for the enhancement layer.

The filtering needs determination process or the filtering process can be parallelized on multiple layers. Multiple layers can share parameters that control the filtering needs determination process or the filtering process (such as the one described in the previous paragraph 0094). It may be advantageous to additionally specify a flag indicating whether the layer shares the parameter.

Fig. 44 is an explanatory diagram showing an image encoding process applicable to the above-mentioned scalable codec. Fig. 44 shows the configuration of a scalable encoding device 810 as an example. The scalable encoding device 810 includes a first encoding unit 820, a second encoding unit 830, and a multiplexing unit 840.

The first encoding unit 820 encodes the base layer image and generates a coded stream of the base layer. The second encoding unit 830 encodes the enhancement layer image and generates a coded stream of the enhancement layer. The multiplexing unit 840 multiplexes the coded stream of the base layer generated by the first encoding unit 820 and one or more coded streams of the enhancement layer generated by the second encoding unit 830 to generate a multi-layer multiplexed stream.

The first encoding unit 820 and the second encoding unit 830 shown in FIG44 are configured similarly to the image encoding device 10 according to the above-described embodiment. Applying a deblocking filter to each layer can parallelize the filtering necessity determination process for vertical block boundaries and horizontal block boundaries, or the filtering necessity determination process and the filtering process. Parameters controlling these processes can be inserted into the header area of the coded stream for each layer, or into the common header area of the multiplexed stream.

Fig. 45 is an explanatory diagram showing an image decoding process applicable to the scalable codec. Fig. 45 shows an example of a configuration of a scalable decoding device 860. The scalable decoding device 860 includes a demultiplexing unit 870, a first decoding unit 880, and a second decoding unit 890.

The demultiplexing unit 870 demultiplexes the multiplexed streams into a base layer coded stream and one or more enhancement layer coded streams. The first decoding unit 880 decodes the base layer image from the base layer coded stream. The second decoding unit 890 decodes the enhancement layer image from the enhancement layer coded stream.

The first decoding unit 880 and the second decoding unit 890 shown in FIG45 are configured similarly to the image decoding device 60 according to the above embodiment. Applying a deblocking filter to each layer can parallelize the filtering necessity determination process for vertical block boundaries and horizontal block boundaries, or the filtering necessity determination process and the filtering process. Parameters controlling these processes can be obtained from the header area of the coded stream of each layer or from the common header area in the multiplexed stream.

<7. Sample Application>

The image encoding device 10 and image decoding device 60 according to the above-described embodiments can be applied to various electronic devices, such as transmitters and receivers for satellite broadcasting, cable broadcasting such as cable television, distribution on the Internet, distribution to terminals via cellular communications, etc.; recording devices that record images on media such as optical disks, magnetic disks, or flash memory; and reproduction devices that reproduce images from such storage media. Four example applications are described below.

[71. First Example Application]

46 is a block diagram showing an example of a schematic configuration of a television using the above-described embodiment. A television 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, an external interface 909, a control unit 910, a user interface 911, and a bus 912.

The tuner 902 extracts a signal of a desired channel from a broadcast signal received via the antenna 901 and demodulates the extracted signal. The tuner 902 then outputs a coded bit stream obtained by demodulation to the demultiplexer 903. The tuner 902 serves as a transmission device of the television 900 for receiving a coded stream in which an image is coded.

The demultiplexer 903 separates the video stream and audio stream of the program to be viewed from the encoded bit stream, and outputs each separated stream to the decoder 904. In addition, the demultiplexer 903 extracts auxiliary data such as EPG (Electronic Program Guide) from the encoded bit stream, and supplies the extracted data to the control section 910. In addition, in the case where the encoded bit stream is scrambled, the demultiplexer 903 can perform descrambling.

The decoder 904 decodes the video stream and the audio stream input from the demultiplexer 903. The decoder 904 then outputs the video data generated by the decoding process to the video signal processing section 905. In addition, the decoder 904 outputs the audio data generated by the decoding process to the audio signal processing section 907.

The video signal processing unit 905 reproduces the video data input from the decoder 904 and causes the display unit 906 to display the video. The video signal processing unit 905 may also cause the display unit 906 to display an application screen provided via the network. Furthermore, the video signal processing unit 905 may perform additional processing, such as noise removal on the video data, according to settings. Furthermore, the video signal processing unit 905 may generate images of GUIs (graphical user interfaces), such as menus, buttons, and cursors, and superimpose these images on the output image.

The display unit 906 is driven by a driving signal provided by the video signal processing unit 905 and displays a video or image on a video screen of a display device (eg, a liquid crystal display, a plasma display, an OLED, etc.).

The audio signal processing section 907 performs reproduction processing such as D/A conversion and amplification on the audio data input from the decoder 904, and outputs audio from the speaker 908. In addition, the audio signal processing section 907 may perform additional processing such as noise removal on the audio data.

The external interface 909 is an interface for connecting the television 900 to an external appliance or a network. For example, a video stream or an audio stream received via the external interface 909 can be decoded by the decoder 904. That is, the external interface 909 also serves as a transmission device of the television 900 for receiving a coded stream in which an image is coded.

The control unit 910 includes a processor such as a CPU (Central Processing Unit) and memory such as RAM (Random Access Memory) and ROM (Read Only Memory). The memory stores programs to be executed by the CPU, program data, EPG data, data acquired via a network, and the like. For example, when the television 900 is turned on, the program stored in the memory is read and executed by the CPU. The CPU controls the operation of the television 900 by, for example, executing the program in response to an operation signal input from the user interface 911.

The user interface 911 is connected to the control unit 910. The user interface 911 includes, for example, buttons and switches used by the user to operate the television 900, and a receiver for remote control signals. The user interface 911 detects user operations via these components, generates operation signals, and outputs the generated operation signals to the control unit 910.

The bus 912 connects the tuner 902 , the demultiplexer 903 , the decoder 904 , the video signal processing section 905 , the audio signal processing section 907 , the external interface 909 , and the control section 910 to one another.

In the television 900 configured in this manner, the decoder 904 has the function of the image decoding device 60 according to the above embodiment. Thus, also in the case of image decoding in the television 900, the parallelism of the deblocking filter processing can be enhanced and high-speed processing can be ensured.

[7-2. Second Example Application]

47 is a block diagram showing an example of a schematic configuration of a mobile phone employing the above-described embodiment. The mobile phone 920 includes an antenna 921, a communication unit 922, an audio codec 923, a speaker 924, a microphone 925, a camera unit 926, an image processing unit 927, a demultiplexing unit 928, a recording/reproducing unit 929, a display unit 930, a control unit 931, an operation unit 932, and a bus 933.

The antenna 921 is connected to the communication section 922. The speaker 924 and the microphone 925 are connected to the audio codec 923. The operation section 932 is connected to the control section 931. The bus 933 connects the communication section 922, the audio codec 923, the camera section 926, the image processing section 927, the demultiplexing section 928, the recording/reproducing section 929, the display section 930, and the control section 931 to each other.

The mobile phone 920 performs operations such as transmission/reception of audio signals, transmission/reception of e-mail or image data, image capture, recording of data, etc. in various operation modes including an audio communication mode, a data communication mode, an image capturing mode, and a video phone mode.

In the audio communication mode, the analog audio signal generated by the microphone 925 is provided to the audio codec 923. The audio codec 923 converts the analog audio signal into audio data, and performs A/D conversion and compression on the converted audio data. Then, the audio codec 923 outputs the compressed audio data to the communication unit 922. The communication unit 922 encodes and modulates the audio data and generates a transmission signal. Then, the communication unit 922 sends the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies the wireless signal received via the antenna 921, converts the frequency of the wireless signal, and obtains the received signal. Then, the communication unit 922 demodulates and decodes the received signal and generates audio data, and outputs the generated audio data to the audio codec 923. The audio codec 923 expands the audio data and performs D/A conversion and generates an analog audio signal. Then, the audio codec 923 provides the generated audio signal to the speaker 924 and outputs the audio.

In data communication mode, the control unit 931 generates text data constituting an email, for example, based on user operations via the operation unit 932. Furthermore, the control unit 931 causes the text to be displayed on the display unit 930. Furthermore, based on a user transmission instruction via the operation unit 932, the control unit 931 generates email data and outputs the generated email data to the communication unit 922. The communication unit 922 then encodes and modulates the email data and generates a transmission signal. The communication unit 922 then transmits the generated transmission signal to a base station (not shown) via the antenna 921. Furthermore, the communication unit 922 amplifies the wireless signal received via the antenna 921, converts the frequency of the wireless signal, and acquires the received signal. The communication unit 922 then demodulates and decodes the received signal to recover the email data and outputs the recovered email data to the control unit 931. The control unit 931 causes the display unit 930 to display the contents of the email and also stores the email data on the storage medium of the recording/reproducing unit 929.

The recording/reproducing unit 929 includes any readable and writable storage medium. For example, the storage medium may be a built-in storage medium such as RAM, flash memory, or an externally loaded storage medium such as a hard disk, magnetic disk, optical disk, optical disc, USB memory, or memory card.

Furthermore, in the image capture mode, the camera section 926 captures an image of a subject, generates image data, and outputs the generated image data to the image processing section 927. The image processing section 927 encodes the image data input from the camera section 926 and causes the encoded stream to be stored in the storage medium of the recording/reproducing section 929.

In addition, in the video phone mode, the demultiplexing unit 928 multiplexes the video stream encoded by the image processing unit 927 and the audio stream input from the audio codec 923, and outputs the multiplexed stream to the communication unit 922. The communication unit 922 encodes and modulates the stream and generates a transmission signal. The communication unit 922 then sends the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies the wireless signal received via the antenna 921, converts the frequency of the wireless signal, and obtains the received signal. These transmission signals and received signals may include an encoded bit stream. The communication unit 922 then demodulates and decodes the received signal, restores the stream, and outputs the restored stream to the demultiplexing unit 928. The demultiplexing unit 928 separates the video stream and the audio stream from the input stream, outputs the video stream to the image processing unit 927, and outputs the audio stream to the audio codec 923. The image processing unit 927 decodes the video stream and generates video data. The video data is supplied to the display unit 930, and a series of images are displayed on the display unit 930. The audio codec 923 expands and D/A converts the audio stream and generates an analog audio signal. The audio codec 923 then supplies the generated audio signal to the speaker 924 and outputs the audio.

In the mobile phone 920 configured in this manner, the image processing unit 927 has the functions of the image encoding device 10 and the image decoding device 60 according to the above-described embodiment. Thus, even in the case of image decoding and encoding in the mobile phone 920, the parallelism of the deblocking filter processing can be enhanced, and high-speed processing can be ensured.

[7-3. Third Example Application]

Figure 48 is a block diagram showing an example of the schematic configuration of a recording/reproducing device using the above-described embodiment. The recording/reproducing device 940, for example, encodes the audio data and video data of a received broadcast program and records them on a recording medium. The recording/reproducing device 940 can also encode audio data and video data obtained from another device and record them on a recording medium. In addition, the recording/reproducing device 940, for example, reproduces the data recorded on the recording medium using a monitor or speaker in accordance with user instructions. In this case, the recording/reproducing device 940 decodes the audio data and video data.

The recording/reproducing device 940 includes a tuner 941, an external interface 942, an encoder 943, an HDD (Hard Disk Drive) 944, a disk drive 945, a selector 946, a decoder 947, an OSD (On-Screen Display) 948, a control section 949, and a user interface 950.

The tuner 941 extracts a signal of a desired channel from a broadcast signal received via an antenna (not shown) and demodulates the extracted signal. The tuner 941 then outputs a coded bit stream obtained by demodulation to the selector 946. That is, the tuner 941 functions as a transmission device of the recording/reproducing device 940.

The external interface 942 is an interface for connecting the recording/reproducing device 940 to an external appliance or a network. For example, the external interface 942 may be an IEEE 1394 interface, a network interface, a USB interface, a flash memory interface, or the like. For example, video data and audio data received by the external interface 942 are input to the encoder 943. In other words, the external interface 942 functions as a transmission device for the recording/reproducing device 940.

In the case where the video data and the audio data input from the external interface 942 are not encoded, the encoder 943 encodes the video data and the audio data, and then outputs the encoded bit stream to the selector 946 .

The HDD 944 records the encoded bit stream, various programs, and other items of data as compressed video or audio content data on an internal hard disk. In addition, the HDD 944 reads these items of data from the hard disk when reproducing video or audio.

The disk drive 945 records or reads data on a loaded recording medium. The recording medium loaded in the disk drive 945 may be, for example, a DVD disk (DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD+, DVD+RW, etc.), a Blu-ray (registered trademark) disk, or the like.

The selector 946 selects an encoded bit stream input from the tuner 941 or the encoder 943 when recording video or audio, and outputs the selected encoded bit stream to the HDD 944 or the disk drive 945. In addition, the selector 946 outputs the encoded bit stream input from the HDD 944 or the disk drive 945 to the decoder 947 when reproducing video or audio.

The decoder 947 decodes the encoded bit stream and generates video data and audio data. The decoder 947 then outputs the generated video data to the OSD 948. In addition, the decoder 904 outputs the generated audio data to an external speaker.

The OSD 948 reproduces the video data input from the decoder 947 and displays the video. In addition, the OSD 948 may superimpose an image of a GUI such as a menu, a button, a cursor, or the like on the displayed video, for example.

The control unit 949 includes a processor such as a CPU and a memory such as RAM or ROM. The memory stores programs executed by the CPU, program data, and the like. For example, when the recording/reproducing device 940 is activated, the program stored in the memory is read and executed by the CPU. The CPU controls the operation of the recording/reproducing device 940 by, for example, executing the program in response to an operation signal input from the user interface 950.

The user interface 950 is connected to the control unit 949. The user interface 950 includes, for example, buttons and switches used by the user to operate the recording/reproducing device 940, and a receiving unit for remote control signals. The user interface 950 detects user operations via these components, generates operation signals, and outputs the generated operation signals to the control unit 949.

In the recording/reproducing device 940 configured in this manner, the encoder 943 has the function of the image encoding device 10 according to the above-described embodiment. In addition, the decoder 947 has the function of the image decoding device 60 according to the above-described embodiment. Thus, even in the case of image decoding and encoding in the recording/reproducing device 940, it is possible to enhance the parallelism of the deblocking filter processing and ensure high-speed processing.

[7-4. Fourth Example Application]

Fig. 49 is a block diagram showing an example of the schematic configuration of an image capturing device employing the above-described embodiment. The image capturing device 960 captures an image of a subject, generates an image, encodes the image data, and records the image data on a recording medium.

The image capture device 960 includes an optical block 961, an image capture section 962, a signal processing section 963, an image processing section 964, a display section 965, an external interface 966, a memory 967, a media drive 968, an OSD 969, a control section 970, a user interface 971 and a bus 972.

The optical block 961 is connected to the image capturing section 962. The image capturing section 962 is connected to the signal processing section 963. The display section 965 is connected to the image processing section 964. The user interface 971 is connected to the control section 970. The bus 972 interconnects the image processing section 964, the external interface 966, the memory 967, the media drive 968, the OSD 969, and the control section 970.

The optical block 961 includes a focusing lens, an aperture stop mechanism, and other components. The optical block 961 forms an optical image of the subject on the image capture surface of the image capture unit 962. The image capture unit 962 includes an image sensor such as a CCD or CMOS sensor, and converts the optical image formed on the image capture surface into an electrical image signal through photoelectric conversion. The image capture unit 962 then outputs the image signal to the signal processing unit 963.

The signal processing section 963 performs various camera signal processes such as knee correction, gamma correction, color correction, etc. on the image signal input from the image capturing section 962 . The signal processing section 963 outputs image data after the camera signal process to the image processing section 964 .

The image processing unit 964 encodes the image data input from the signal processing unit 963 and generates encoded data. The image processing unit 964 then transmits the generated encoded data to the external interface 966 or the media drive 968. In addition, the image processing unit 964 decodes the encoded data input from the external interface 966 or the media drive 968 and generates image data. The image processing unit 964 then outputs the generated image data to the display unit 965. In addition, the image processing unit 964 may output the image data input from the signal processing unit 963 to the display unit 965 and display the image. In addition, the image processing unit 964 may superimpose the data for display obtained from the OSD 969 on the image output to the display unit 965.

The OSD 969 generates an image of a GUI such as a menu, a button, a cursor, and the like, and outputs the generated image to the image processing section 964 .

The external interface 966 is configured as, for example, a USB input/output terminal. For example, when printing an image, the external interface 966 connects the image capture device 960 to a printer. Furthermore, a drive is connected to the external interface 966 as needed. For example, removable media such as a magnetic disk or optical disk is loaded into the drive, and a program read from the removable media can be installed in the image capture device 960. Furthermore, the external interface 966 can be configured as a network interface connected to a network such as a LAN or the Internet. In other words, the external interface 966 functions as a transmission device for the image capture device 960.

The recording medium loaded in the media drive 968 may be, for example, any readable and writable removable medium, such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory. Alternatively, the recording medium may be fixedly loaded in the media drive 968 configured as a non-portable storage unit such as a built-in hard disk drive or an SSD (solid state drive).

The control unit 970 includes a processor such as a CPU and a memory such as RAM or ROM. The memory stores programs executed by the CPU, program data, and the like. For example, when the image capture device 960 is activated, the program stored in the memory is read and executed by the CPU. The CPU controls the operation of the image capture device 960 by, for example, executing the program in response to an operation signal input from the user interface 971.

The user interface 971 is connected to the control section 970. The user interface 971 includes, for example, buttons, switches, and the like for the user to operate the image capturing device 960. The user interface 971 detects user operations via these components, generates an operation signal, and outputs the generated operation signal to the control section 970.

In the image capturing device 960 configured in this manner, the image processing section 964 has the functions of the image encoding device 10 and the image decoding device 60 according to the above-described embodiment. Thus, in the case of image decoding and encoding in the image capturing device 960, it is possible to enhance the parallelism of the deblocking filter processing and ensure high-speed processing.

8. Summary

With reference to Figures 1 to 49, three working examples of the deblocking filter for the image encoding device 10 and the image decoding device 60 according to the embodiment have been described. The three working examples use the pixel values input to the deblocking filter to determine whether it is necessary to apply the deblocking filter for vertical boundaries and horizontal boundaries. The determination process can be performed in parallel without relying on the filtering process results. The dependency of the processing between macroblocks is eliminated, and the processing of the macroblocks can be parallelized. As a result, delays or data bit rate degradation caused by the large amount of processing of the deblocking filter can be avoided, and high-speed processing can be guaranteed. The degree of parallelism and order of the deblocking filter processing can be flexibly configured according to various conditions such as image size or installation environment.

According to the first working example, whether to apply a deblocking filter to a vertical boundary is determined using the pixel values of pixels belonging to a row to which the definition of a horizontal boundary does not apply. Whether to apply a deblocking filter to a horizontal boundary is determined using the pixel values of pixels belonging to a row to which the definition of a vertical boundary does not apply. Filtering a block does not update the pixel values used to determine that filtering is required for another block. Even if filtering is required for a given block, the pixel values input to the deblocking filter do not need to be stored in an additional memory after filtering another block. This can save hardware costs required for device installation.

According to the second working example, a memory that is not updated by the filtering process stores pixel values input to the deblocking filter. The filtering process needs to determine the reference input pixel values. In this case, the filtering process needs to determine the reference pixels whose positions are not restricted. This enables the flexible determination conditions to be appropriately used for various purposes, such as more accurately determining the necessity of filtering or determining the necessity with reduced processing costs.

According to the third working example, for the filter processing of vertical boundaries and horizontal boundaries, the pixels input to the deblocking filter are filtered. This configuration can parallelize the filter processing for vertical boundaries and horizontal boundaries with each other. This can further accelerate the processing of the deblocking filter. For pixels updated by two parallel filter processes, the output pixel value is calculated based on the two filter outputs. Parallelizing the two filter processes can also appropriately reduce the block distortion that occurs at the vertical and horizontal boundaries. The output pixel value can be calculated as a weighted average of the two filter outputs. This can allow the deblocking filter to more effectively eliminate block distortion and further improve image quality.

This specification primarily describes an example in which filtering of vertical boundaries precedes filtering of horizontal boundaries. However, the aforementioned effects of the techniques disclosed herein can also be achieved when filtering of horizontal boundaries precedes filtering of vertical boundaries. The deblocking filter processing unit, or macroblock, may differ from the size described in this specification. Available techniques can omit the filtering requirement determination process and parallelize the application of deblocking filters to vertical and horizontal boundaries.

The technology for sending information for parallelization of deblocking filtering processing from the encoding side to the decoding side is not limited to the technology of multiplexing the information into the header of the coded stream. For example, the information may not be multiplexed into the coded bit stream, but it can be sent or recorded as separate data associated with the coded bit stream. The term "association" indicates the possibility of guaranteeing the linking of the image (or part of the image such as a slice or block) contained in the bit stream with the information corresponding to the image. That is, the information can be sent on a transmission path different from that used for the image (or bit stream). The information can be recorded on a recording medium different from that used for the image (or bit stream) (or a different recording area of the same recording medium). The information and the image (or bit stream) can be associated with each other based on arbitrary units such as multiple frames, one frame, or a part of a frame.

Although the preferred embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is of course not limited to the above examples. Those skilled in the art may find various changes and improvements within the scope of the appended claims, which should be understood to naturally fall within the technical scope of the present invention.

This specification refers to filtering for vertical boundaries as "horizontal filtering" and filtering for horizontal boundaries as "vertical filtering." Generally speaking, filtering for vertical boundaries uses horizontally arranged filter taps. Filtering for horizontal boundaries uses vertically arranged filter taps. Therefore, the above nomenclature is used for the filtering processes.

Claims (8)

1. An image processing apparatus, comprising: The horizontal determination unit is configured to perform horizontal filtering determination processing across multiple vertical block boundaries between multiple blocks; The horizontal determination control unit is configured to control the horizontal determination unit to perform horizontal filtering determination processes in parallel on all vertical block boundaries in the maximum coding unit or picture within the decoded image obtained by local decoding in the encoding process, wherein each horizontal filtering determination process determines whether to apply a horizontal deblocking filter to pixels near the vertical block boundaries. A horizontal filtering unit is configured to apply the horizontal deblocking filter in parallel across multiple vertical block boundaries determined by the horizontal determination unit to be suitable for the horizontal deblocking filter within the decoded image, and to generate a first filtered image. The vertical determination unit is configured to perform vertical filtering determination processing across multiple horizontal block boundaries between multiple blocks; A vertical determination control unit is configured to control the vertical determination unit to perform vertical filtering determination processes in parallel on all horizontal block boundaries in the maximum coding unit or drawing within the decoded image, wherein each vertical filtering determination process determines whether a vertical deblocking filter is applied to pixels near the horizontal block boundaries. A vertical filtering unit is configured to apply a vertical deblocking filter in parallel across multiple horizontal block boundaries determined by the vertical determination unit to be suitable for the vertical deblocking filter within a first filtered image including pixels to which the horizontal filtering unit has applied the deblocking filter, and to generate a second filtered image; and The encoding unit is configured to encode the image using the second filtered image. 2. The image processing apparatus according to claim 1, wherein, The horizontal filtering section applies the horizontal deblocking filter to the maximum encoding unit or all vertical block boundaries in the drawing. The vertical filtering section applies the vertical deblocking filter to the maximum encoding unit or all horizontal block boundaries in the drawing. 3. The image processing apparatus according to claim 2, wherein, The horizontal filtering unit applies the horizontal deblocking filter in a manner that the horizontal deblocking filters for the multiple adjacent vertical block boundaries between blocks do not depend on each other. The vertical filtering unit applies the vertical deblocking filter in a manner that the vertical deblocking filters for the multiple adjacent horizontal block boundaries between blocks do not depend on each other. 4. The image processing apparatus according to claim 3, wherein, The encoding unit encodes the image according to the encoding units divided from the largest encoding unit. 5. An image processing method, comprising: Perform horizontal filtering determination processing across multiple vertical block boundaries between multiple blocks; The horizontal filtering determination process is controlled in the following manner: horizontal filtering determination processes are performed in parallel on all vertical block boundaries in the largest coding unit or picture within the decoded image obtained by local decoding in the encoding process, and each horizontal filtering determination process determines whether to apply a horizontal deblocking filter to pixels near the vertical block boundaries. Within the decoded image, the horizontal deblocking filter is applied in parallel across multiple vertical block boundaries that are determined to be suitable for the horizontal deblocking filter, and a first filtered image is generated. Perform vertical filtering to determine the boundaries of multiple horizontal blocks across multiple blocks; The vertical filtering determination process is controlled in the following manner: the vertical filtering determination process is performed in parallel on all horizontal block boundaries in the largest coding unit or drawing within the decoded image, and each vertical filtering determination process determines whether to apply a vertical deblocking filter to pixels near the horizontal block boundaries. Within the first filtered image, which includes pixels to which the horizontal deblocking filter has been applied, a vertical deblocking filter is applied in parallel across multiple horizontal block boundaries determined to be to which the vertical deblocking filter is applied, and a second filtered image is generated; and The encoding unit encodes the image using the second filtered image. 6. The image processing method according to claim 5, wherein, The horizontal deblocking filter is applied to the maximum encoding unit or all vertical block boundaries in the drawing. The vertical deblocking filter is applied to the maximum encoding unit or all horizontal block boundaries in the drawing. 7. The image processing method according to claim 6, wherein, The horizontal deblocking filter is applied to multiple adjacent vertical block boundaries in a manner that does not create a dependency between them. The vertical deblocking filter is applied to multiple adjacent horizontal block boundaries in such a way that the vertical deblocking filters for multiple adjacent horizontal block boundaries do not depend on each other. 8. The image processing method according to claim 7, wherein, The image is encoded according to the encoding units segmented from the largest encoding unit.