HK1243572B - Image processing device and method - Google Patents

Description

Image processing device and method

This application is a divisional application of the invention patent application with application number 201380010481.8, application date February 28, 2013, and name “Image Processing Device and Method”.

Technical Field

The present disclosure relates to an image processing apparatus and method.

Background Art

In H.264/AVC (Advanced Video Coding), one of the standard specifications for video coding schemes, each of the High Profile and higher profiles allows image data to be quantized using a quantization step size that differs for each component of an orthogonal transform coefficient. The quantization step size for each component of an orthogonal transform coefficient can be set based on a reference step value and a quantization matrix (also known as a scaling list) defined by a size equal to the unit of the orthogonal transform.

Specified values for the quantization matrix are prepared for each prediction mode (intra-frame prediction mode, inter-frame prediction mode) and for each transform unit size (4×4, 8×8). In addition, the user is allowed to specify a unique quantization matrix that is different from the values specified in the sequence parameter set or picture parameter set. When no quantization matrix is used, the quantization step size used for quantization has the same value for all components.

In HEVC (High Efficiency Video Coding), which is being standardized as a next-generation video coding scheme and the successor to H.264/AVC, the concept of coding units (CUs) corresponding to traditional macroblocks has been introduced (see, for example, NPL 1). The range of coding unit sizes is specified by a set of values in the sequence parameter set, which are powers of 2 and are called the largest coding unit (LCU) and the smallest coding unit (SCU). In addition, the split_flag is used to specify the specific coding unit size within the range specified by the LCU and SCU.

In HEVC, one coding unit can be divided into one or more orthogonal transform units or one or more transform units (TUs). Available transform unit sizes are any one of 4×4, 8×8, 16×16, and 32×32.

At the same time, for the purpose of reducing the amount of code during transmission, the DC component (also referred to as a direct current component) of the quantization matrix (scaling list) is transmitted as data different from its AC component (also referred to as an alternating current component). Specifically, the DC component of the scaling list is transmitted as a DC coefficient (also referred to as a direct current coefficient) different from the AC coefficient (also referred to as an alternating current coefficient), which is the AC component of the scaling list.

To reduce the amount of coding of the DC coefficient during transmission, it has been proposed to subtract a constant (eg, 8) from the value of the DC coefficient and encode the resulting value (scaling_list_dc_coef_minus8) using signed exponential Golomb coding (see, eg, NPL 1).

Reference List

Non-patent literature

NPL 1: Benjamin Bross, Fraunhofer HHI, Woo-Jin Han, Gachon University, Jens-Rainer Ohm, RWTH Aachen, Gary J.Sullivan, Microsoft, Thomas Wiegand, Fraunhofer HHI/TU Berlin, JCTVC-H1003, "High Efficiency Video Coding (HEVC) text specification draft 6", Joint Collaborative Team on Video Coding(JCT-VC)ofITU-T SG16WP3and ISO/IEC JTC1/SC29/WG11 7th Meeting:Geneva,CH,November 21-30, 2011

Summary of the Invention

Technical issues

However, there is a concern that while the above approach facilitates processing, it will not provide sufficient compression efficiency.

The present disclosure has been made in view of the above circumstances, and an object of the present disclosure is to enable suppression of an increase in the encoding amount of a scaling list.

Solution to the problem

One aspect of the present disclosure provides an image processing device, including: a setting unit, configured to set a replacement difference coefficient, which is the difference between a replacement coefficient and a coefficient located at the beginning of a quantization matrix, the size of the quantization matrix being limited to not more than a sending size, the sending size being the maximum size allowed in sending, the replacement coefficient being used to replace the coefficient located at the beginning of an up-converted quantization matrix, the up-converted quantization matrix being obtained by up-converting the quantization matrix into a size identical to a block size serving as a unit for performing dequantization processing; a quantization unit, configured to quantize an image to generate quantized data; and a sending unit, configured to send encoded data obtained by encoding the quantized data generated by the quantization unit, replacement coefficient data obtained by encoding the replacement coefficient, and replacement difference coefficient data obtained by encoding the replacement difference coefficient set by the setting unit.

The setting unit may set a difference between the replacement coefficient and an initial value set for the quantization matrix.

The setting unit may set a difference coefficient that is a difference between coefficients of the quantization matrix, and the transmitting unit may transmit difference coefficient data obtained by encoding the difference coefficient set by the setting unit.

The sending unit may send the replacement coefficient data and the replacement difference coefficient data together.

The sending unit may send the replacement coefficient data and the replacement difference coefficient data in the order of replacement coefficient data and replacement difference coefficient data.

The quantization unit may quantize the image using a quantization matrix or an up-converted quantization matrix.

The coding unit as a unit of processing for performing the coding process and the transform unit as a unit of processing for performing the transform process may have a hierarchical structure, and the image processing device may further include a coding unit configured to encode the quantized data generated by the quantization unit.

One aspect of the present disclosure also provides an image processing method, including: setting a replacement difference coefficient, which is the difference between the replacement coefficient and the coefficient located at the beginning of a quantization matrix, the size of the quantization matrix is limited to not more than a sending size, which is the maximum size allowed in the transmission, the replacement coefficient is used to replace the coefficient located at the beginning of the up-converted quantization matrix, and the up-converted quantization matrix is obtained by up-converting the quantization matrix into a size that is the same as the block size as the unit for performing dequantization processing; quantizing the image to generate quantized data; and sending the encoded data obtained by encoding the generated quantized data, the replacement coefficient data obtained by encoding the replacement coefficient, and the replacement difference coefficient data obtained by encoding the set replacement difference coefficient.

In one aspect of the present disclosure, a replacement difference coefficient is set, which is the difference between the replacement coefficient and the coefficient located at the beginning of a quantization matrix, the size of the quantization matrix is limited to be no larger than a transmission size, which is the maximum size allowed in transmission, the replacement coefficient is used to replace the coefficient located at the beginning of the up-converted quantization matrix, and the up-converted quantization matrix is obtained by up-converting the quantization matrix into a size that is the same as a block size as a unit for performing dequantization processing; the image is quantized to generate quantized data; and the encoded data obtained by encoding the generated quantized data, the replacement coefficient data obtained by encoding the replacement coefficient, and the replacement difference coefficient data obtained by encoding the set replacement difference coefficient are transmitted.

Advantageous Effects of the Invention

According to the present disclosure, it is possible to process an image and, in particular, suppress an increase in the amount of coding of a quantization matrix.

BRIEF DESCRIPTION OF THE DRAWINGS

[ Fig. 1] Fig. 1 is a diagram showing an example of a zoom list.

[ Fig. 2] Fig. 2 is a diagram showing an example of up-conversion.

[Fig. 3] Fig. 3 is a diagram showing an example of how a scaling list is used in a decoder.

[Fig. 4] Fig. 4 is a diagram showing an example of encoding of a scaling list.

[Fig. 5] Fig. 5 is a diagram showing an example of encoding of a scaling list using the present technology.

[ Fig. 6] Fig. 6 is a diagram showing an example of an exponential Golomb code.

[ Fig. 7] Fig. 7 includes diagrams showing examples of syntax representing a zoom list.

[ Fig. 8] Fig. 8 is a diagram showing an example of the syntax of a default matrix.

[ Fig. 9] Fig. 9 includes diagrams showing examples of the semantics of a default matrix.

[ Fig. 10] Fig. 10 is a diagram showing an example of the syntax of a zoom list.

[ Fig. 11] Fig. 11 is a diagram showing an example of the syntax of a zoom list using the present technology.

[ Fig. 12] Fig. 12 includes diagrams showing an example of the syntax of a zoom list in the related art.

[Fig. 13] Fig. 13 is a diagram showing an example of the syntax of a zoom list.

[Figure 14] Figure 14 is a block diagram showing an example of the main configuration of an image encoding device.

[Figure 15] Figure 15 is a block diagram showing an example of the main configuration of an orthogonal transform/quantization unit.

[Fig. 16] Fig. 16 is a block diagram showing an example of the main configuration of a matrix processing unit.

[Fig. 17] Fig. 17 is a diagram showing an example of downsampling.

[Fig. 18] Fig. 18 is a diagram showing an example of removal of overlapping portions.

[Fig. 19] Fig. 19 is a block diagram showing an example of the main configuration of a DPCM unit.

[Figure 20] Figure 20 is a flowchart showing an example of the process of quantization matrix encoding processing.

[Fig. 21] Fig. 21 is a flowchart showing an example of the flow of DPCM processing.

[Figure 22] Figure 22 is a block diagram showing an example of the main configuration of an image decoding device.

[Figure 23] Figure 23 is a block diagram showing an example of the main configuration of a dequantization/inverse orthogonal transform unit.

[Figure 24] Figure 24 is a block diagram showing an example of the main configuration of a matrix generating unit.

[Fig. 25] Fig. 25 is a diagram showing an example of nearest neighbor interpolation processing.

[Fig. 26] Fig. 26 is a block diagram showing an example of the main configuration of an inverse DPCM unit.

[Figure 27] Figure 27 is a flowchart showing an example of the flow of matrix generation processing.

[Figure 28] Figure 28 is a flowchart showing an example of the process of residual signal decoding processing.

[Figure 29] Figure 29 is a flowchart showing an example of the process of inverse DPCM processing.

[Fig. 30] Fig. 30 is a diagram showing another example of the syntax of a zoom list.

[Figure 31] Figure 31 is a block diagram showing another example configuration of the DPCM unit.

[Figure 32] Figure 32 is a flowchart showing another example of the process of DPCM processing.

[Figure 33] Figure 33 is a block diagram showing another example configuration of an inverse DPCM unit.

[Figure 34] Figure 34 is a flowchart showing another example of the process of inverse DPCM processing.

[Fig. 35] Fig. 35 is a diagram showing another example of the syntax of a zoom list.

[Figure 36] Figure 36 is a flowchart showing another example of the process of inverse DPCM processing.

[Fig. 37] Fig. 37 is a diagram showing another example of the syntax of a zoom list.

[Figure 38] Figure 38 is a block diagram showing another example configuration of the DPCM unit.

[Fig. 39] Fig. 39 is a flowchart showing another example of DPCM processing.

[Figure 40] Figure 40 is a block diagram showing another example configuration of an inverse DPCM unit.

[Figure 41] Figure 41 is a flowchart showing another example of the process of inverse DPCM processing.

[Figure 42] Figure 42 is a flowchart continuing from Figure 41, showing another example of the process of inverse DPCM processing.

[Fig. 43] Fig. 43 includes diagrams showing another example of syntax representing a zoom list.

[Fig. 44] Fig. 44 includes diagrams showing another example of the syntax of a zoom list.

[Fig. 45] Fig. 45 includes diagrams showing another example of syntax representing a zoom list.

[Figure 46] Figure 46 is a diagram showing an example of a multi-view image encoding scheme.

[Figure 47] Figure 47 is a schematic diagram showing an example of the main configuration of a multi-view image encoding device to which the present technology is applied.

[Figure 48] Figure 48 is a schematic diagram showing an example of the main configuration of a multi-view image decoding device to which the present technology is applied.

[Figure 49] Figure 49 is a diagram showing an example of a layered image coding scheme.

[Figure 50] Figure 50 is a schematic diagram showing an example of the main configuration of a layered image encoding device to which the present technology is applied.

[Figure 51] Figure 51 is a schematic diagram showing an example of the main configuration of a layered image decoding device to which the present technology is applied.

[Figure 52] Figure 52 is a block diagram showing an example of the main configuration of a computer.

[Figure 53] Figure 53 is a block diagram showing an example of the main configuration of a television device.

[Figure 54] Figure 54 is a block diagram showing an example of the main configuration of a mobile terminal device.

[Figure 55] Figure 55 is a block diagram showing an example of the main configuration of a recording/reproducing device.

[Figure 56] Figure 56 is a block diagram showing an example of the main configuration of an imaging device.

[Figure 57] Figure 57 is a block diagram showing an example of the use of scalable coding.

[Figure 58] Figure 58 is a block diagram showing another example of the use of scalable coding.

[Figure 59] Figure 59 is a block diagram showing another example of the use of scalable coding.

DETAILED DESCRIPTION

Modes for carrying out the present disclosure (hereinafter, referred to as embodiments) will be described below. In this regard, the description will be made in the following order.

1. First embodiment (exemplary application of the present technology)

2. Second Embodiment (Image Encoding Device, Image Decoding Device: First Method)

3. Third Embodiment (Image Encoding Device, Image Decoding Device: Second Method)

4. Fourth Embodiment (Image Encoding Device, Image Decoding Device: Third Method)

5. Fifth Embodiment (Image Encoding Device, Image Decoding Device: Fourth Method)

6. Sixth Embodiment (Image Encoding Device, Image Decoding Device: Other Methods)

7. Seventh Embodiment (Multi-view Image Encoding Device, Multi-view Image Decoding Device)

8. Eighth Embodiment (Layered Image Coding Apparatus, Layered Image Decoding Apparatus)

9. Ninth embodiment (Computer)

10. Sample Application

11. Example Application of Scalable Coding

<1. First embodiment>

In this embodiment, a description will be given of exemplary applications of the present technology, which will be described in detail in the second embodiment and later embodiments of the present technology.

<1-1. Exemplary Applications of the Present Technology>

First, an exemplary example to which the present technology is applicable will be described. The present technology is a technology related to encoding and decoding of a scaling list used in quantization and dequantization processing performed when image data is encoded and decoded.

The encoding and decoding of image data may include quantization and dequantization of coefficient data. This quantization and dequantization is performed in units of blocks of a predetermined size, and a scaling list (or quantization matrix) having a size corresponding to the block size is used. For example, in HEVC (High Efficiency Video Coding), quantization (or dequantization) is performed in sizes such as 4×4, 8×8, 16×16, and 32×32. In HEVC, quantization matrices having sizes of 4×4 and 8×8 can be prepared.

FIG1 shows an example of an 8×8 scaling list. As shown in FIG1 , the scaling list includes a DC coefficient and an AC coefficient. The DC coefficient including one value is the (0,0) coefficient of the quantization matrix and corresponds to the DC coefficient of the discrete cosine transform (DCT). The AC coefficient is the coefficient of the quantization matrix other than the (0,0) coefficient and corresponds to the coefficient of the DCT other than the DC coefficient. Note that, as shown in FIG1 , the AC coefficient is represented by a matrix. That is, the AC coefficient also includes the (0,0) coefficient (hereinafter also referred to as the AC coefficient (0,0)), and when used for quantization/dequantization, the (0,0) coefficient located at the beginning of the quantization matrix is replaced by the DC coefficient. Therefore, the DC coefficient is also called a replacement coefficient. In the example shown in FIG1 , the AC coefficients form an 8×8 matrix.

Furthermore, in HEVC, an up-converted version (up-conversion) of the 8×8 quantization matrix is used for 16×16 or 32×32 quantization (or dequantization).

FIG2 illustrates an example of upconversion of an 8×8 scaling list to a 16×16 scaling list. As shown in FIG2 , the scaling list is upconverted using, for example, a nearest neighbor interpolation process. Details of the nearest neighbor interpolation process will be described below with reference to, for example, FIG25 . As shown in FIG2 , upconversion is performed on the AC coefficients of the scaling list. The (0,0) coefficient among the upconverted AC coefficients is then replaced by the DC coefficient.

Two types of 8×8 scaling lists are prepared, namely, an 8×8 scaling list for up-conversion to 16×16 ("8×8 for 16×16") and an 8×8 scaling list for up-conversion to 32×32 ("8×8 for 32×32").

The scaling list used for quantization during encoding (using the encoder) is also used for dequantization during decoding (using the decoder). In other words, the scaling list is sent from the encoding side (encoder) to the decoding side (decoder). Figure 3 shows an example of how the scaling list is sent.

3, two types of 8x8 scaling lists are sent, namely, an 8x8 scaling list for up-conversion to 16x16 size and an 8x8 scaling list for up-conversion to 32x32 size. Although not shown in the figure, a 4x4 scaling list is also sent.

The AC coefficients of the 8×8 scaling list for up-conversion to 16×16 size, which have been sent in the above manner, are up-converted to 16×16 size using the above-mentioned nearest neighbor interpolation process on the decoding side (decoder), and are used for dequantization of blocks with 16×16 size after the (0,0) coefficient is replaced by the DC coefficient.

Similarly, the AC coefficients of the 8×8 scaling list for up-conversion to 32×32 size, which have been sent in the above manner, are also up-converted to 32×32 size using the above-mentioned nearest neighbor interpolation process on the decoding side (decoder), and are used for dequantization of blocks with 32×32 size after the (0,0) coefficient is replaced by the DC coefficient.

<1-2. Scaling List Encoding>

Transmitting a scaling list in this manner will increase the amount of code required. Therefore, to minimize the reduction in coding efficiency, a method is used to encode the scaling list to reduce the amount of code required. Figure 4 shows an example of scaling list encoding. Specifically, an 8×8 scaling list is transmitted as follows.

In the case of up-conversion from an 8×8 matrix to a 16×16 matrix:

(1) The difference between the (0,0) coefficient of the 8×8 matrix (that is, the AC coefficient (0,0)) and a predetermined initial value “8” is obtained.

(2) Differences between coefficients of an 8×8 matrix (that is, AC coefficients) (adjacent coefficients in a sequence of coefficients arranged in a one-dimensional manner in scanning order) are obtained.

(3) The difference between the (0,0) coefficient of the 16×16 matrix (that is, the DC coefficient) and a predetermined initial value “8” is obtained.

(4) The differences obtained in (1) and (2) and the difference obtained in (3) are sent separately.

In the case of up-conversion from an 8×8 matrix to a 32×32 matrix:

(1) The difference between the (0,0) coefficient of the 8×8 matrix (that is, the AC coefficient (0,0)) and a predetermined initial value “8” is obtained.

(2) Differences between coefficients of an 8×8 matrix (that is, AC coefficients) (adjacent coefficients in a sequence of coefficients arranged in a one-dimensional manner in scanning order) are obtained.

(3) The difference between the (0,0) coefficient of the 32×32 matrix (that is, the DC coefficient) and a predetermined initial value “8” is obtained.

(4) The differences obtained in (1) and (2) and the difference obtained in (3) are sent separately.

However, in the above method, these differences are encoded using signed exponential Golomb coding and transmitted in (4). As described above, the difference obtained in (1) is the difference between the AC coefficient (0, 0) and the initial value "8". Therefore, there is a concern that if the value of the AC coefficient (0, 0) is not close to the initial value "8", the amount of code may increase.

For example, in FIG4 , the value of the AC coefficient (0,0) is "12," and the value "4" is encoded using signed exponential Golomb coding and transmitted as the difference obtained in (1). That is, 7 bits are required to transmit the difference obtained in (1), and the encoding efficiency may decrease accordingly. If the value of the difference obtained in (1) increases, the encoding efficiency may further decrease. The same is true for the case of an 8×8 scaling list for up-conversion to a 16×16 size and an 8×8 scaling list for up-conversion to a 32×32 size.

At the same time, the energy of DCT coefficients is generally concentrated in the DC coefficient and adjacent low-order coefficients. Therefore, the quantization matrix usually also has small values for the DC coefficient and adjacent coefficients. In addition, if significantly different values are used for each frequency, quantization errors may be subjectively perceived. In order to suppress this visual degradation of image quality, continuous values are used for the DC coefficient and adjacent coefficients.

The (0,1) coefficient, (1.0) coefficient, and (1.1) coefficient obtained after up-conversion correspond to the AC coefficient (0,0) before up-conversion. In addition, the (0,0) coefficient obtained after up-conversion corresponds to the DC coefficient.

Therefore, in the scaling list, the value of the AC coefficient (0, 0) and the value of the DC coefficient are usually close to each other. For example, the MPEG2, AVC, and HEVC default matrices adopt values with this relationship. In the example shown in FIG4, the value of the DC coefficient is the same as the value of the AC coefficient (0, 0), that is, "12". Therefore, the difference obtained in (3) (that is, the difference between the DC coefficient and the initial value "8") is also "4".

That is, obtaining the difference between each of the DC coefficient and AC coefficient (0, 0) whose values are close to each other and the initial value may increase the difference between them and may also cause redundancy. It can be said that there is a risk of further reducing encoding efficiency.

To solve this problem, instead of using the method shown in Figure 4, the following method is used to send the zoom list. Figure 5 shows an example of this method.

In the case of up-conversion from an 8×8 matrix to a 16×16 matrix:

(1) Obtain the difference between the (0,0) coefficient of the 8×8 matrix (that is, the AC coefficient (0,0)) and the (0,0) coefficient of the 16×16 matrix (that is, the DC coefficient).

(2) Differences between coefficients of an 8×8 matrix (that is, AC coefficients) (adjacent coefficients in a sequence of coefficients arranged in a one-dimensional manner in scanning order) are obtained.

(3) The difference between the (0,0) coefficient of the 16×16 matrix (that is, the DC coefficient) and a predetermined initial value “8” is obtained.

(4) The differences obtained in (1) to (3) are transmitted in common.

In the case of up-conversion from an 8×8 matrix to a 32×32 matrix:

(1) Obtain the difference between the (0,0) coefficient of the 8×8 matrix (that is, the AC coefficient (0,0)) and the (0,0) coefficient of the 32×32 matrix (that is, the DC coefficient).

(2) Differences between coefficients of an 8×8 matrix (that is, AC coefficients) (adjacent coefficients in a sequence of coefficients arranged in a one-dimensional manner in scanning order) are obtained.

(3) The difference between the (0,0) coefficient of the 32×32 matrix (that is, the DC coefficient) and a predetermined initial value “8” is obtained.

(4) The differences obtained in (1) to (3) are transmitted in common.

Similar to the method shown in FIG. 4 , in (4) the differences are encoded using exponential Golomb coding and transmitted as exponential Golomb codes.

At the destination to which these differences are sent as exponential Golomb codes, when the exponential Golomb codes are received, the received exponential Golomb codes are decoded to obtain the respective differences, and the reverse processing to that in (1) to (3) above is performed on the obtained differences to determine the respective coefficients (DC coefficient and AC coefficient).

<1-3. Exemplary Features of the Present Technology>

Exemplary features of the present technology related to the above-described transmission method will now be described.

<1-3-1. DPCM between AC coefficient (0,0) and DC coefficient>

The scaling list is encoded using differential pulse code modulation (DPCM) and transmitted. In the example shown in FIG4 , the AC coefficient and the DC coefficient are DPCM-encoded separately, while in the example shown in FIG5 , according to one of the features of the present technology, the difference between the AC coefficient (0,0) and the DC coefficient (also referred to as the replacement difference coefficient) is determined and transmitted.

As described above, the AC coefficient (0, 0) and the DC coefficient generally take values close to each other. Therefore, the difference between the AC coefficient (0, 0) and the DC coefficient is likely to be smaller than the difference between the AC coefficient (0, 0) and the initial value "8". In other words, using this technique, transmitting the replacement difference coefficient as the difference between the AC coefficient (0, 0) and the DC coefficient is more likely to reduce the amount of code.

For example, in the example shown in FIG. 5 , the value of the difference obtained in ( 1 ) is “0”.

FIG6 is a table showing an example of signed exponential Golomb coding. As indicated in the table shown in FIG6, the exponential Golomb code for the value "4" has a code length of 7 bits, while the exponential Golomb code for the value "0" has a code length of 1 bit. That is, compared with the method shown in FIG4, the method shown in FIG5 can reduce the amount of coding by 6 bits.

Typically, transmitting an 8×8 quantization matrix requires approximately 100 to 200 bits of code. Therefore, 6 bits account for approximately 6% of the total. Reducing the code size by 6% in advanced syntax can be a significant effect.

<1-3-2. Joint Transmission of DC Coefficient and AC Coefficient>

FIG7 shows an example of the syntax of the scaling list. The example shown in Part A of FIG7 shows the syntax of the example shown in FIG4. Specifically, after the difference between the AC coefficient (0, 0) and the initial value "8" and the AC coefficient difference (scaling_list_delta_coef) are transmitted, the difference between the DC coefficient and the initial value "8" (scaling_list_dc_coef_minus8) is separately transmitted.

In contrast, one of the characteristics of the present technology is that the difference between the DC coefficient and the AC coefficient (0, 0), as well as the AC coefficient difference, are arranged in this order and transmitted together. Specifically, as shown in FIG5 , after the DC coefficient and the AC coefficient arranged in a predetermined scanning order are arranged in a one-dimensional manner and the difference between the DC coefficient and the initial value "8" is determined, the difference between adjacent coefficients in the coefficient sequence is determined. Furthermore, the resulting differences (coefficient differences) are arranged in a one-dimensional manner in the obtained order and transmitted together.

The syntax in this case is shown in the example in Part B of Figure 7. Specifically, initially, the difference between the DC coefficient and the initial value "8" (scaling_list_dc_coef_minus8) is transmitted, and then the difference between the DC coefficient and the AC coefficient (0, 0) and the difference between the AC coefficients (scaling_list_delta_coef) are transmitted. In other words, the DC coefficient and the AC coefficient are encoded and transmitted together.

In this way, the common transmission of the differences arranged in the order in which they were obtained enables the decoding side (decoder) to which the differences are transmitted to decode these differences in the order in which they were transmitted and obtain the individual coefficients. In other words, the DPCM-encoded scaling list can be easily decoded. More specifically, the processing load can be reduced. In addition, the rearrangement of the differences is no longer required, resulting in a reduction in buffer capacity. In addition, the individual differences can be decoded in the order in which they were provided, resulting in a suppression of an increase in processing time.

<1-3-3. Sending the default matrix>

FIG8 is a diagram showing an example of a syntax for sending a default matrix. In the related art, as shown in FIG8 , the initial coefficient (that is, the DC coefficient) is sent as "0" to send information indicating that the default matrix is used. In other words, the value of the difference between the DC coefficient and the initial value "8" (scaling_list_dc_coef_minus8) is "-8". However, as shown in FIG6 , the exponential Golomb code for the value "-8" has a code length of 9 bits. In other words, there is a concern that the coding efficiency may be significantly reduced. Generally, it is desired that the number of bits of the high-level syntax be as small as possible. In addition, as shown in FIG8 , due to the increased complexity of the syntax, the processing load may increase.

To address these issues, the initial coefficients are not set to "0," but rather the semantics of scaling_list_pred_matrix_id_delta are modified. More specifically, the semantics of scaling_list_pred_matrix_id_delta are modified from that shown in Part A of FIG. 9 to that shown in Part B of FIG. That is, in the related art, as shown in Part A of FIG. 9 , a value equal to "0" indicates reference to the previous matrix (MatrixID-1). Instead, as shown in Part B of FIG. 9 , a value of "0" in scaling_list_pred_matrix_id_delta means reference to the default matrix.

Therefore, the code length of the exponential Golomb code used to transmit information indicating the use of the default matrix can be equal to 1 bit, and the reduction in coding efficiency can be suppressed. In addition, in the related art, the scaling list requires the syntax shown in parts A and B of Figure 10. This syntax can be simplified as in the example shown in Figure 11. In other words, the processing load involved in encoding and decoding the scaling list can be reduced.

<1-4. Features of Syntax Using the Present Technology>

The syntax will be described more specifically.

In the example of the related art shown in parts A and B of FIG10, the default determination needs to be performed twice, namely, scaling_list_dc_coef_minus8 and scaling_list_delta_coef. In addition, for scaling_list_delta_coef, the determination is performed in a "for" loop, and when useDefaultScalingMatrixFlag = 1, the loop is exited. In addition, an intermediate flag called "stopNow" is required, and because of this condition, there is also a branch such as substituting nextCoef into the value of scalingList. In this way, the syntax of the related art includes complex processing.

In the present technology, accordingly, as in the example shown in FIG. 11 , the DC coefficient calculated from scaling_list_dc_coef_minus8 is substituted into nextCoef to set the initial value of scaling_list_delta_coef as the DC coefficient.

In addition, in the semantics, the value of scaling_list_pred_matrix_id_delta represented by “+1” in the related art remains unchanged, and a value “0” is used as a special value.

That is, in the related art, when ScalingList[0][2] is to be decoded (matrixId=2), if scaling_list_pred_matrix_id_delta=0, matrixId=2 is obtained from refMatrixId=matrixId-(1+scaling_list_pred_matrix_id_delta). Therefore, refMatrixId=1 is obtained, and the value of ScalingList[0][1] is copied.

In contrast, in the present technology, refMatrixId=matrixId-scaling_list_pred_matrix_id_delta is set. When ScalingList[0][2] is to be decoded (matrixId=2), if ScalingList[0][1] is to be copied (or if refMatrixId=1 is to be obtained), scaling_list_pred_matrix_id_delta=1 may be set.

Therefore, as shown in FIG11, the number of lines of the syntax of the scaling list can be significantly reduced. In addition, two variables that are to be included as intermediate data, namely UseDefaultScalingMatrix and stopNow, can be omitted. In addition, the branch formed in the "for" loop as shown in FIG10 is no longer necessary. Therefore, the processing load involved in encoding and decoding the scaling list can be reduced.

<1-5. Processing Unit Implementing the Present Technology>

When the present technology is applied to the transmission of a scaling list, the scaling list is encoded and decoded in the above-described manner. Specifically, the image encoding device 10 described below with reference to FIG14 encodes the scaling list and transmits the encoded scaling list, and the image decoding device 300 described below with reference to FIG22 receives the encoded scaling list and decodes the encoded scaling list.

The scaling list is encoded by the matrix processing unit 150 ( FIG. 15 ) in the orthogonal transform/quantization unit 14 ( FIG. 14 ) of the image encoding device 10. More specifically, the scaling list is encoded by the DPCM unit 192 and the exp-G unit 193 (both the DPCM unit 192 and the exp-G unit 193 are shown in FIG. 16 ) in the entropy encoding unit 164 ( FIG. 16 ) in the matrix processing unit 150. That is, the DPCM unit 192 determines the difference between the coefficients (DC coefficient and AC coefficient) of the scaling list, and the exp-G unit 193 encodes each difference using exponential Golomb coding.

In order to encode the scaling list using the present technology as described above, the DPCM unit 192 may have, for example, the example configuration shown in FIG19 and may perform DPCM processing as in the example shown in FIG21. In addition, semantics as in the example shown in part C of FIG44 or part C of FIG45 may be used.

In other words, only the DPCM unit 192 and the exp-G unit 193 may be required to implement encoding of the scaling list using the present technology, and other components having any configuration may be used as needed. Required configurations such as a processing unit for up-converting the scaling list and a processing unit for performing quantization using the scaling list may be provided according to the embodiment.

The scaling list is decoded by the matrix generation unit 410 ( FIG. 23 ) in the dequantization/inverse orthogonal transform unit 313 ( FIG. 22 ) of the image decoding device 300. More specifically, the scaling list is decoded by the exp-G unit 551 and the inverse DPCM unit 552 ( FIG. 24 ) in the entropy decoding unit 533 ( FIG. 24 ) in the matrix generation unit 410. That is, the exp-G unit 551 decodes the Golomb code to obtain differences, and the inverse DPCM unit 552 determines the coefficients (DC coefficient and AC coefficient) of the scaling list from the differences.

In order to decode the encoded scaling list using the present technology as described above, the inverse DPCM unit 552 may have the example configuration shown in FIG26, for example, and may perform inverse DPCM processing as in the example shown in FIG29. In addition, the same semantics as in the examples shown in part C of FIG44 or part C of FIG45 may be used.

In other words, only the exp-G unit 551 and the inverse DPCM unit 552 may be required to implement decoding of the scaling list using the present technology, and other components having any configuration may be used as needed. Required configurations such as a processing unit for up-converting the scaling list and a processing unit for performing dequantization using the scaling list may be provided according to the embodiment.

Various embodiments to which the present technology is applied will be described below for a more detailed description of the present technology.

<2. Second embodiment>

<2-1. Syntax: First Method>

(1) Syntax of related technologies

First, FIG12 shows an example of the syntax of a quantization matrix (or scaling list) in related art. In actual use, the difference matrix between the scaling list and its prediction matrix is usually transmitted rather than the scaling list. Therefore, in the following description of the syntax, etc., it is assumed that the description of the scaling list can also be applied to the difference matrix.

Part A of FIG. 12 shows the syntax of the zoom list data (zoom list data syntax), and part B of FIG. 12 shows the syntax of the zoom list (zoom list syntax).

(1-1) Scaling List Data Syntax

As shown in part A of Figure 12, the syntax of the scaling list data stipulates: reading a flag indicating whether a scaling list is provided (scaling_list_present_flag), a flag indicating whether the current mode is a copy mode (scaling_list_pred_mode_flag), information indicating which scaling list is referenced in the copy mode (scaling_list_pred_matrix_id_delta), etc.

(1-2) Scaling List Syntax

As shown in part B of FIG. 12 , the syntax of the scaling list stipulates that a DC coefficient (scaling_list_dc_coef_minus8) from which a constant (eg, 8) is subtracted, a difference between AC coefficients (scaling_list_delta_coef), etc. are read, and the DC coefficient and AC coefficient are restored.

However, there is a concern that while the above-described syntax facilitates processing, it will not provide sufficient compression efficiency of the DC coefficient.

Therefore, in order to achieve sufficient compression efficiency for the DC coefficient (also referred to as the DC coefficient), which is a coefficient of the DC component (direct current component), the difference between the DC coefficient and another coefficient is determined, and the difference is transmitted instead of the DC coefficient. In other words, the difference is information used to calculate the DC coefficient and, in other words, is essentially equivalent to the DC coefficient. However, the difference is usually smaller than the DC coefficient. Therefore, transmitting the difference instead of the DC coefficient can lead to a reduction in the amount of code.

In the following description, for convenience of description, the scaling list (quantization matrix) has a size of 8 × 8. A specific example of the above-mentioned method of transmitting the difference between the DC coefficient and another coefficient instead of the DC coefficient will be described below.

(2) Syntax of the first method

For example, 65 coefficients may be transmitted using DPCM (Differential Pulse Code Modulation), where the DC coefficient is considered to be the element at the beginning of an 8x8 matrix (AC coefficients) (first approach).

That is, first, the difference between a predetermined constant and the DC coefficient is calculated, and this difference is used as the initial coefficient of the DPCM data. Then, the difference between the DC coefficient and the initial AC coefficient is calculated, and this difference is used as the second coefficient of the DPCM data. Then, the difference between the initial AC coefficient and the second AC coefficient is calculated, and this difference is used as the third coefficient of the DPCM data. Subsequently, the difference from the previous AC coefficient is calculated, and this difference is used as the fourth coefficient of the DPCM data. Subsequent coefficients of the DPCM data are determined in a manner similar to that described above. The coefficients of the DPCM data generated in this manner are sequentially transmitted, starting with the initial coefficient.

Therefore, when the values of the (0,0) coefficient (AC coefficient) and the DC coefficient of the 8×8 matrix are close to each other, the compression ratio can be improved. By implementing the above-mentioned first method, the image encoding device can process the DC coefficient in a manner similar to that of the AC coefficient (alternating current coefficient), where the AC coefficient is the coefficient of the AC component (also called the alternating current component). Note that in order to implement the above-mentioned first method, the image decoding device to which the above-mentioned coefficients are sent needs to specially process only the initial coefficients. Specifically, the image decoding device needs to extract the DC coefficient from the AC coefficients.

Figure 13 shows the syntax of the scaling list in the above case. In the example shown in Figure 13, 65 differences between coefficients (scaling_list_delta_coef) are read, and among the coefficients (nextcoef) determined from the differences, the coefficient at the beginning (nextcoef) is used as the DC coefficient (scaling_list_dc_coef), while the other coefficients are used as AC coefficients (ScalingList[i]).

An image encoding device that implements the syntax of the above-mentioned first method will be described below.

<2-2. Image Coding Device>

Figure 14 is a block diagram showing an example configuration of an image encoding device 10 according to an embodiment of the present disclosure. The image encoding device 10 shown in Figure 14 is an image processing device to which the present technology is applied, and the image processing device is configured to encode input image data and output the encoded image data. Referring to Figure 14, the image encoding device 10 includes an A/D (analog-to-digital) conversion unit 11 (A/D), a rearrangement buffer 12, a subtraction unit 13, an orthogonal transform/quantization unit 14, a lossless encoding unit 16, an accumulation buffer 17, a rate control unit 18, a dequantization unit 21, an inverse orthogonal transform unit 22, an adder unit 23, a deblocking filter 24, a frame memory 25, a selector 26, an intra-frame prediction unit 30, a motion search unit 40, and a mode selection unit 50.

The A/D conversion unit 11 converts an image signal input in an analog form into image data in a digital form, and outputs a sequence of digital image data to the rearrangement buffer 12 .

The rearrangement buffer 12 rearranges images included in the image data sequence input from the A/D conversion unit 11. After rearranging the images according to a GOP (Group of Pictures) structure for encoding processing, the rearrangement buffer 12 outputs the image data whose images have been rearranged to the subtraction unit 13, the intra prediction unit 30, and the motion search unit 40.

The image data input from the rearrangement buffer 12 and the predicted image data selected by the mode selection unit 50 are supplied to the subtraction unit 13, which will be described below. The subtraction unit 13 calculates prediction error data representing the difference between the image data input from the rearrangement buffer 12 and the predicted image data input from the mode selection unit 50, and outputs the calculated prediction error data to the orthogonal transform/quantization unit 14.

The orthogonal transform/quantization unit 14 performs orthogonal transform and quantization on the prediction error data input from the subtraction unit 13, and outputs quantized transform coefficient data (hereinafter referred to as quantized data) to the lossless encoding unit 16 and the dequantization unit 21. The bit rate of the quantized data output from the orthogonal transform/quantization unit 14 is controlled according to the rate control signal supplied from the rate control unit 18. The detailed configuration of the orthogonal transform/quantization unit 14 will be further described below.

The lossless encoding unit 16 is provided with the quantized data input from the orthogonal transform/quantization unit 14, information for generating a scaling list (or quantization matrix) on the decoding side, and information on intra prediction or inter prediction selected by the mode selection unit 50. The information on intra prediction may include, for example, prediction mode information indicating the optimal intra prediction mode for each block. In addition, the information on inter prediction may include, for example, prediction mode information for block-by-block prediction of a motion vector, differential motion vector information, reference image information, etc. Furthermore, the information for generating a scaling list on the decoding side may include identification information indicating the maximum size of the scaling list to be transmitted (or the difference matrix between the scaling list (quantization matrix) and its prediction matrix).

The lossless coding unit 16 performs lossless coding on the quantized data to generate a coded stream. The lossless coding performed by the lossless coding unit 16 may be, for example, variable length coding, arithmetic coding, or the like. Furthermore, the lossless coding unit 16 multiplexes information used to generate a scaling list into the header of the coded stream (e.g., a sequence parameter set and a picture parameter set). The lossless coding unit 16 also multiplexes the aforementioned information regarding intra-frame prediction or inter-frame prediction into the header of the coded stream. Thereafter, the lossless coding unit 16 outputs the generated coded stream to the accumulation buffer 17.

The accumulation buffer 17 uses a storage medium such as a semiconductor memory to temporarily accumulate the encoded stream input from the lossless encoding unit 16. Thereafter, the accumulation buffer 17 outputs the accumulated encoded stream at a rate corresponding to the bandwidth of the transmission path (or output line of the image encoding device 10).

The rate control unit 18 monitors the accumulation buffer 17 to check for capacity availability. The rate control unit 18 generates a rate control signal according to the available capacity of the accumulation buffer 17, and outputs the generated rate control signal to the orthogonal transform/quantization unit 14. For example, when the available capacity of the accumulation buffer 17 is low, the rate control unit 18 generates a rate control signal for reducing the bit rate of the quantized data. Alternatively, for example, when the available capacity of the accumulation buffer 17 is sufficiently high, the rate control unit 18 generates a rate control signal for increasing the bit rate of the quantized data.

The dequantization unit 21 performs a dequantization process on the quantized data input from the orthogonal transform/quantization unit 14. Thereafter, the dequantization unit 21 outputs the transform coefficient data obtained by the dequantization process to the inverse orthogonal transform unit 22.

The inverse orthogonal transform unit 22 performs an inverse orthogonal transform process on the transform coefficient data input from the dequantization unit 21 to restore the prediction error data. Thereafter, the inverse orthogonal transform unit 22 outputs the restored prediction error data to the adder unit 23.

The adder unit 23 adds together the restored prediction error data input from the inverse orthogonal transform unit 22 and the predicted image data input from the mode selection unit 50 to generate decoded image data. Thereafter, the adder unit 23 outputs the generated decoded image data to the deblocking filter 24 and the frame memory 25.

The deblocking filter 24 performs a filtering process for reducing block artifacts caused by encoding of an image. The deblocking filter 24 filters the decoded image data input from the adder unit 23 to remove (or at least reduce) block artifacts, and outputs the filtered decoded image data to the frame memory 25.

The frame memory 25 stores the decoded image data input from the adder unit 23 and the filtered decoded image data input from the deblocking filter 24 using a storage medium.

The selector 26 reads decoded image data to be filtered for intra prediction from the frame memory 25, and supplies the read decoded image data as reference image data to the intra prediction unit 30. The selector 26 also reads decoded image data after filtering for inter prediction from the frame memory 25, and supplies the read decoded image data as reference image data to the motion search unit 40.

The intra-frame prediction unit 30 performs intra-frame prediction processing in each intra-frame prediction mode based on the image data to be encoded input from the rearrangement buffer 12 and the decoded image data provided via the selector 26. For example, the intra-frame prediction unit 30 evaluates the prediction results obtained in each intra-frame prediction mode using a predetermined cost function. The intra-frame prediction unit 30 then selects the intra-frame prediction mode that minimizes the cost function value (that is, the intra-frame prediction mode that provides the highest compression ratio) as the optimal intra-frame prediction mode. In addition, the intra-frame prediction unit 30 outputs prediction mode information indicating the optimal intra-frame prediction mode, the predicted image data, and information about the intra-frame prediction (such as the cost function value) to the mode selection unit 50.

The motion search unit 40 performs an inter-frame prediction process (or inter-frame prediction process) based on the image data to be encoded input from the rearrangement buffer 12 and the decoded image data supplied via the selector 26. For example, the motion search unit 40 evaluates the prediction results obtained in each prediction mode using a predetermined cost function. The motion search unit 40 then selects the prediction mode that minimizes the cost function value (that is, the prediction mode that provides the highest compression ratio) as the optimal prediction mode. In addition, the motion search unit 40 generates predicted image data based on the optimal prediction mode. The motion search unit 40 outputs information on inter-frame prediction, including prediction mode information indicating the selected optimal prediction mode, predicted image data, and information on inter-frame prediction such as the cost function value, to the mode selection unit 50.

The mode selection unit 50 compares the cost function value for intra prediction input from the intra prediction unit 30 with the cost function value for inter prediction input from the motion search unit 40. The mode selection unit 50 then selects the prediction technique having the smaller cost function value of the cost function values for intra prediction and inter prediction. If intra prediction is selected, the mode selection unit 50 outputs information about intra prediction to the lossless encoding unit 16 and also outputs predicted image data to the subtraction unit 13 and the adder unit 23. Alternatively, if inter prediction is selected, the mode selection unit 50 outputs the above-mentioned information about inter prediction to the lossless encoding unit 16 and also outputs predicted image data to the subtraction unit 13 and the adder unit 23.

<2-3. Example Configuration of Orthogonal Transform/Quantization Unit>

15 is a block diagram illustrating an example of a detailed configuration of the orthogonal transform/quantization unit 14 of the image encoding device 10 shown in FIG 14. Referring to FIG 15 , the orthogonal transform/quantization unit 14 includes a selection unit 110, an orthogonal transform unit 120, a quantization unit 130, a scaling list buffer 140, and a matrix processing unit 150.

(1) Select unit

The selection unit 110 selects a transform unit (TU) for orthogonal transform of the image data to be encoded from among a plurality of transform units having different sizes. Examples of possible sizes of the transform unit that can be selected by the selection unit 110 include 4×4 and 8×8 for H.264/AVC (Advanced Video Coding), and 4×4, 8×8, 16×16, and 32×32 for HEVC (High Efficiency Video Coding). The selection unit 110 can select the transform unit based on, for example, the size or quality of the image to be encoded, the performance of the image encoding device 10, etc. The selection of the transform unit by the selection unit 110 can be manually adjusted by the user who develops the image encoding device 10. Thereafter, the selection unit 110 outputs information specifying the size of the selected transform unit to the orthogonal transform unit 120, the quantization unit 130, the lossless encoding unit 16, and the dequantization unit 21.

(2) Orthogonal Transformation Unit

The orthogonal transform unit 120 performs an orthogonal transform on the image data (that is, the prediction error data) supplied from the subtraction unit 13 in units of the transform unit selected by the selection unit 110. The orthogonal transform performed by the orthogonal transform unit 120 may be, for example, a discrete cosine transform (DCT), a Karhunen-Loève transform, or the like. Thereafter, the orthogonal transform unit 120 outputs the transform coefficient data obtained through the orthogonal transform process to the quantization unit 130.

(3) Quantization unit

The quantization unit 130 quantizes the transform coefficient data generated by the orthogonal transform unit 120 by using the scaling list corresponding to the transform unit selected by the selection unit 110. In addition, the quantization unit 130 switches the quantization step size according to the rate control signal provided from the rate control unit 18 to change the bit rate of the quantized data to be output.

In addition, the quantization unit 130 causes groups of scaling lists corresponding to the plurality of transform units selectable by the selection unit 110 to be stored in the scaling list buffer 140. For example, in HEVC, if there are four possible sizes of transform units, namely, 4×4, 8×8, 16×16, and 32×32, four groups of scaling lists corresponding to the four sizes, respectively, may be stored in the scaling list buffer 140. Note that if a designated scaling list is used for a given size, only a flag indicating that the designated scaling list is used (a scaling list defined by the user is not used) may be stored in the scaling list buffer 140 in association with the given size.

Generally, a set of scaling lists that can be used by the quantization unit 130 may be set for each sequence of the coded stream. In addition, the quantization unit 130 may update the set of scaling lists set for each sequence on a picture-by-picture basis. Information for controlling the setting and updating of the set of scaling lists may be inserted into, for example, a sequence parameter set and a picture parameter set.

(4) Scaling List Buffer

The scaling list buffer 140 temporarily stores a set of scaling lists respectively corresponding to a plurality of transform units selectable by the selection unit 110 using a storage medium such as a semiconductor memory. When the matrix processing unit 150 performs the processing described below, the set of scaling lists stored in the scaling list buffer 140 is referred to.

(5) Matrix processing unit

The matrix processing unit 150 encodes the scaling list to be used for encoding (quantization). Thereafter, the encoded data of the scaling list generated by the matrix processing unit 150 (hereinafter referred to as encoded scaling list data) is output to the lossless encoding unit 16 and can be inserted into the header of the encoded stream.

<2-4. Detailed Example Configuration of Matrix Processing Unit>

16 is a block diagram showing a more detailed configuration example of the matrix processing unit 150. Referring to FIG16, the matrix processing unit 150 includes a prediction unit 161, a difference matrix generation unit 162, a difference matrix size conversion unit 163, an entropy encoding unit 164, a decoding unit 165, and an output unit 166.

(1) Prediction unit

The prediction unit 161 generates a prediction matrix. As shown in FIG16 , the prediction unit 161 includes a copy unit 171 and a prediction matrix generation unit 172.

In copy mode, the copy unit 171 copies the previously transmitted scaling list and uses the copied quantization matrix as a prediction matrix (or a scaling list for predicting the orthogonal transform unit to be processed). More specifically, the copy unit 171 obtains the size and list ID (ListID) of the previously transmitted scaling list from the storage unit 202 in the decoding unit 165. The size is information indicating the size of the scaling list (ranging from 4×4 to 32×32, for example). The list ID is information indicating the type of prediction error data to be quantized.

For example, the list ID includes identification information indicating whether the prediction error data to be quantized is prediction error data of a luminance component generated using a prediction image subjected to intra prediction (Intra Luma), prediction error data of a color difference component (Cr) generated using a prediction image subjected to intra prediction (Intra Cr), prediction error data of a color difference component (Cb) generated using a prediction image subjected to intra prediction (Intra Cb), or prediction error data of a luminance component generated using a prediction image subjected to inter prediction (Inter Luma).

The copying unit 171 selects a previously transmitted scaling list of the same size as the scaling list input to the matrix processing unit 150 (the scaling list of the orthogonal transform unit to be processed) as the scaling list to be copied, and provides the list ID of the scaling list to be copied to the output unit 166 to output the list ID to the device outside the matrix processing unit 150 (the lossless encoding unit 16 and the dequantization unit 21). In other words, in this case, only the list ID is transmitted to the decoding side (or included in the encoded data) as information indicating the prediction matrix generated by copying the previously transmitted scaling list. Therefore, the image encoding device 10 can suppress an increase in the encoding amount of the scaling list.

In addition, in normal mode, the prediction matrix generation unit 172 obtains the previously transmitted scaling list from the storage unit 202 in the decoding unit 165 and uses the scaling list to generate a prediction matrix (or predict the scaling list of the orthogonal transform unit to be processed). The prediction matrix generation unit 172 provides the generated prediction matrix to the difference matrix generation unit 162.

(2) Difference matrix generation unit

The difference matrix generation unit 162 generates a difference matrix (residual matrix) which is a difference between the prediction matrix supplied from the prediction unit 161 (prediction matrix generation unit 172) and the scaling list input to the matrix processing unit 150. As shown in FIG16 , the difference matrix generation unit 162 includes a prediction matrix size conversion unit 181, a calculation unit 182, and a quantization unit 183.

The prediction matrix size transforming unit 181 transforms (hereinafter also referred to as converting) the size of the prediction matrix supplied from the prediction matrix generating unit 172 so that the size of the prediction matrix matches the size of the scaling list input to the matrix processing unit 150 .

For example, if the size of the prediction matrix is larger than the size of the scaling list, the prediction matrix size conversion unit 181 down-converts (hereinafter also referred to as down-conversion) the prediction matrix. More specifically, for example, when the prediction matrix has a size of 16×16 and the scaling list has a size of 8×8, the prediction matrix size conversion unit 181 down-converts the prediction matrix into an 8×8 prediction matrix. Note that any method for down-conversion can be used. For example, the prediction matrix size conversion unit 181 can reduce the number of elements in the prediction matrix by using a filter (by calculation) (hereinafter also referred to as downsampling). Alternatively, the prediction matrix size conversion unit 181 can also reduce the number of elements in the prediction matrix without using a filter (hereinafter also referred to as subsampling) by removing some elements (for example, only even-numbered elements among the two-dimensional elements (in FIG. 17 , solid black elements)), for example, as shown in FIG.

In addition, for example, if the size of the prediction matrix is smaller than the size of the scaling list, the prediction matrix size conversion unit 181 converts the prediction matrix upward (hereinafter also referred to as up-conversion). More specifically, for example, when the prediction matrix has an 8×8 size and the scaling list has a 16×16 size, the prediction matrix size conversion unit 181 converts the prediction matrix upward into a 16×16 prediction matrix. Note that any method for up-conversion can be used. For example, the prediction matrix size conversion unit 181 may increase the number of elements in the prediction matrix by using a filter (by calculation) (hereinafter also referred to as upsampling). Alternatively, the prediction matrix size conversion unit 181 may also increase the number of elements in the prediction matrix by, for example, copying each element in the prediction matrix without using a filter (hereinafter also referred to as inverse subsampling).

The prediction matrix size conversion unit 181 supplies the prediction matrix whose size has been matched with the size of the scaling list to the calculation unit 182 .

The calculation unit 182 subtracts the scaling list input to the matrix processing unit 150 from the prediction matrix supplied from the prediction matrix size conversion unit 181 and generates a difference matrix (residual matrix). The calculation unit 182 supplies the calculated difference matrix to the quantization unit 183.

The quantization unit 183 quantizes the difference matrix supplied from the calculation unit 182. The quantization unit 183 supplies the quantized difference matrix to the difference matrix size conversion unit 163. The quantization unit 183 also supplies information used for quantization (such as a quantization parameter) to the output unit 166 to output the information to a device (lossless encoding unit 16 and dequantization unit 21) outside the matrix processing unit 150. Note that the quantization unit 183 may be omitted (that is, it may not be necessary to perform quantization of the difference matrix).

(3) Difference Matrix Size Transformation Unit

If necessary, the difference matrix size conversion unit 163 converts the size of the difference matrix (quantized data) provided from the difference matrix generation unit 162 (quantization unit 183) into a size smaller than or equal to the maximum size allowed for transmission (hereinafter also referred to as the transmission size). This maximum size can have any selectable value and is, for example, 8×8.

The encoded data output from the image encoding device 10 is transmitted to an image decoding device corresponding to the image encoding device 10 via, for example, a transmission path or a storage medium, and is decoded by the image decoding device. The image encoding device 10 is provided with an upper limit (maximum size) on the size of the difference matrix (quantized data) during such transmission or in the encoded data output from the image encoding device 10.

If the size of the difference matrix is larger than the maximum size, the difference matrix size conversion unit 163 down-converts the difference matrix so that the size of the difference matrix becomes smaller than or equal to the maximum size.

Note that, similar to the down-conversion of the prediction matrix described above, the difference matrix can be down-converted using any method. For example, down-sampling can be performed using a filter or the like, or sub-sampling including removing elements can be performed.

In addition, the difference matrix after down-conversion can have any size less than the maximum size. However, in general, the larger the difference between the size before and after conversion, the larger the error becomes. Therefore, it is desirable that the difference matrix be down-converted to the maximum size.

The difference matrix size transforming unit 163 supplies the down-converted difference matrix to the entropy encoding unit 164. Note that if the size of the difference matrix is smaller than the maximum size, the above-described down-conversion is unnecessary, and therefore, the difference matrix size transforming unit 163 supplies the difference matrix input thereto as it is to the entropy encoding unit 164 (that is, down-conversion of the difference matrix is omitted).

(4) Entropy coding unit

The entropy encoding unit 164 encodes the difference matrix (quantized data) supplied from the difference matrix size transform unit 163 using a predetermined method. As shown in FIG16 , the entropy encoding unit 164 includes an overlap determination unit (135 degree unit) 191, a DPCM (Differential Pulse Code Modulation) unit 192, and an exp-G unit 193.

The overlap determination unit 191 determines the symmetry of the difference matrix provided from the difference matrix size conversion unit 163. For example, as shown in FIG18 , if the residual (difference matrix) represents a 135-degree symmetric matrix, the overlap determination unit 191 removes the data (matrix elements) of the symmetric portion, which is the overlap data. If the residual matrix does not represent a 135-degree symmetric matrix, the overlap determination unit 191 omits the removal of the data (matrix elements). The overlap determination unit 191 provides the data of the difference matrix, from which the symmetric portion (if necessary) has been removed, to the DPCM unit 192.

The DPCM unit 192 performs DPCM encoding on the data of the difference matrix from which the symmetric portion (if necessary) has been removed, supplied from the overlap determination unit 191 , and generates DPCM data. The DPCM unit 192 supplies the generated DPCM data to the exp-G unit 193 .

The exp-G unit 193 encodes the DPCM data supplied from the DPCM unit 192 using a signed or unsigned exponential Golomb code (hereinafter also referred to as an exponential Golomb code). The exp-G unit 193 supplies the encoding result to the decoding unit 165 and the output unit 166.

(5) Decoding unit

The decoding unit 165 restores the scaling list from the data provided by the exp-G unit 193. The decoding unit 165 provides information on the restored scaling list to the prediction unit 161 as the previously transmitted scaling list.

As shown in FIG. 16 , the decoding unit 165 includes a scaling list restoring unit 201 and a storage unit 202 .

The scaling list restoration unit 201 decodes the exponential Golomb code supplied from the entropy encoding unit 164 (exp-G unit 193) to restore the scaling list to be input to the matrix processing unit 150. For example, the scaling list restoration unit 201 decodes the exponential Golomb code using a method corresponding to the encoding method of the entropy encoding unit 164, and obtains a difference matrix by performing a transform inverse to the size transform performed by the difference matrix size transform unit 163 and performing dequantization corresponding to the quantization performed by the quantization unit 183. The scaling list restoration unit 201 also subtracts the obtained difference matrix from the prediction matrix to restore the scaling list.

The zoom list restoring unit 201 supplies the restored zoom list to the storage unit 202 for storage in association with the size and list ID of the zoom list.

The storage unit 202 stores information about the scaling list provided from the scaling list restoration unit 201. The information about the scaling list stored in the storage unit 202 is used to generate a prediction matrix for another orthogonal transform unit processed at a later time. That is, the storage unit 202 provides the stored information about the scaling list to the prediction unit 161 as information about the previously transmitted scaling list.

Note that instead of storing information about the scaling list restored in the above manner, the storage unit 202 may store the scaling list input to the matrix processing unit 150 in association with its size and list ID. In this case, the scaling list restoration unit 201 may be omitted.

(6) Output unit

The output unit 166 outputs the supplied various types of information to a device outside the matrix processing unit 150. For example, in the copy mode, the output unit 166 supplies the list ID of the prediction matrix supplied from the copy unit 171 to the lossless encoding unit 16 and the dequantization unit 21. In addition, for example, in the normal mode, the output unit 166 supplies the exponential Golomb code supplied from the exp-G unit 193 and the quantization parameter supplied from the quantization unit 183 to the lossless encoding unit 16 and the dequantization unit 21.

The output unit 166 also provides identification information indicating the maximum size (transmission size) allowed in the transmission of the scaling list (or the difference matrix between the scaling list and its prediction matrix) to the lossless encoding unit 16 as information for generating the scaling list on the decoding side. As described above, the lossless encoding unit 16 creates a coded stream including information for generating the scaling list, and provides the coded stream to the decoding side. The identification information indicating the transmission size can be pre-specified by the level, specification, etc. In this case, the information about the transmission size is shared in advance between the device on the encoding side and the device on the decoding side. Therefore, the transmission of the above-mentioned identification information can be omitted.

<2-5. Detailed Example Configuration of DPCM Unit>

19 is a block diagram showing an example of a more detailed configuration of the DPCM unit 192. Referring to FIG19, the DPCM unit 192 includes a DC coefficient encoding unit 211 and an AC coefficient DPCM unit 212.

The DC coefficient encoding unit 211 obtains a DC coefficient from the coefficients provided by the overlap determination unit 191, subtracts the value of the DC coefficient from a predetermined initial value (e.g., 8) to determine a difference value, and uses the difference value as an initial (i=0) difference value (scaling_list_delta_coef). The DC coefficient encoding unit 211 provides the calculated difference value (scaling_list_delta_coef (i=0)) to the exp-G unit 193 as the initial coefficient of the scaling list corresponding to the attention area being processed.

The AC coefficient DPCM unit 212 obtains an AC coefficient from the coefficients provided by the overlap determination unit 191, and subtracts the value of the AC coefficient from the coefficient of the previous process to determine a difference value (scaling_list_delta_coef(i>0)). The AC coefficient DPCM unit 212 provides the determined difference value (scaling_list_delta_coef(i>0)) to the exp-G unit 193 as the coefficient of the scaling list corresponding to the target area being processed. Note that when i=1, the previous coefficient is represented by i=0. Therefore, the "DC coefficient" is the coefficient of the previous process.

In this manner, the DPCM unit 192 can transmit the DC coefficient as an element located at the beginning of the scaling list (AC coefficient), thereby improving the encoding efficiency of the scaling list.

<2-6. Flow of Quantization Matrix Coding Processing>

Next, an example of the flow of the quantization matrix encoding process performed by the matrix processing unit 150 shown in FIG. 16 will be described with reference to the flowchart shown in FIG. 20 .

When the quantization matrix encoding process starts, in step S101 , the prediction unit 161 acquires a scaling list (or quantization matrix) for a current region (also referred to as a focus region) which is an orthogonal transform unit to be processed.

In step S102, the prediction unit 161 determines whether the current mode is the copy mode. If it is determined that the current mode is not the copy mode, the prediction unit 161 advances the process to step S103.

In step S103 , the prediction matrix generation unit 172 acquires the previously transmitted scaling list from the storage unit 202 , and generates a prediction matrix using the scaling list.

In step S104, the prediction matrix size conversion unit 181 determines whether the size of the prediction matrix generated in step S103 is different from the size of the scaling list for the current region (region of interest) acquired in step S101. If it is determined that the sizes are different, the prediction matrix size conversion unit 181 advances the process to step S105.

In step S105 , the prediction matrix size conversion unit 181 converts the size of the prediction matrix generated in step S103 into the size of the scaling list for the current region acquired in step S101 .

When the process of step S105 is completed, the prediction matrix size conversion unit 181 advances the process to step S106. If it is determined in step S104 that the size of the prediction matrix is the same as the size of the scaling list, the prediction matrix size conversion unit 181 advances the process to step S106 while skipping the process of step S105 (or not executing the process of step S105).

In step S106 , the calculation unit 182 subtracts the scaling list from the prediction matrix to calculate a difference matrix between the prediction matrix and the scaling list.

In step S107, the quantization unit 183 quantizes the difference matrix generated in step S106. Note that this processing can be omitted.

In step S108, the difference matrix size conversion unit 163 determines whether the size of the quantized difference matrix is larger than the transmission size (the maximum size allowed in transmission). If it is determined that the size of the quantized difference matrix is larger than the transmission size, the difference matrix size conversion unit 163 advances the process to step S109 and down-converts the difference matrix to the transmission size or smaller.

When the process of step S109 is completed, the difference matrix size conversion unit 163 advances the process to step S110. In addition, if it is determined in step S108 that the size of the quantized difference matrix is less than or equal to the transmission size, the difference matrix size conversion unit 163 advances the process to step S110 while skipping the process of step S109 (or not executing the process of step S109).

In step S110, the overlap determination unit 191 determines whether the quantized difference matrix has 135-degree symmetry. If it is determined that the quantized difference matrix has 135-degree symmetry, the overlap determination unit 191 advances the process to step S111.

In step S111, the overlap determination unit 191 removes the overlapping portion (overlapping data) in the quantized difference matrix. After the overlapping data is removed, the overlap determination unit 191 advances the process to step S112.

In addition, if it is determined in step S110 that the quantized difference matrix does not have 135-degree symmetry, the overlap determination unit 191 advances the process to step S112 while skipping the process of step S111 (or not performing the process of step S111 ).

In step S112 , the DPCM unit 192 performs DPCM encoding on the difference matrix from which the overlapping portion has been removed if necessary.

In step S113, the exp-G unit 193 determines whether the DPCM data generated in step S112 has a positive sign or a negative sign. If it is determined that the sign is included, the exp-G unit 193 advances the process to step S114.

In step S114, the exp-G unit 193 encodes the DPCM data using signed exponential Golomb coding. The output unit 166 outputs the generated exponential Golomb code to the lossless coding unit 16 and the dequantization unit 21. When the process of step S114 is completed, the exp-G unit 193 advances the process to step S116.

In addition, if it is determined in step S113 that the symbol is not included, the exp-G unit 193 advances the process to step S115 .

In step S115, the exp-G unit 193 encodes the DPCM data using unsigned exponential Golomb coding. The output unit 166 outputs the generated exponential Golomb code to the lossless coding unit 16 and the dequantization unit 21. When the process of step S115 is completed, the exp-G unit 193 advances the process to step S116.

If the current mode is determined to be the copy mode in step S102, the copy unit 171 copies the previously transmitted scaling list and uses the copied scaling list as the prediction matrix. The output unit 166 outputs the list ID corresponding to the prediction matrix to the lossless encoding unit 16 and the dequantization unit 21 as information indicating the prediction matrix. The copy unit 171 then advances the process to step S116.

In step S116, the zoom list restoring unit 201 restores the zoom list. In step S117, the storage unit 202 stores the zoom list restored in step S116.

When the process of step S117 is completed, the matrix processing unit 150 ends the quantization matrix encoding process.

<2-7. DPCM Processing Flow>

Next, an example of the flow of the DPCM process performed in step S112 in FIG. 20 will be described with reference to the flowchart shown in FIG. 21 .

When the DPCM process starts, in step S131, the DC coefficient encoding unit 211 determines the difference between the DC coefficient and a constant. In step S132, the AC coefficient DPCM unit 212 determines the difference between the DC coefficient and the initial AC coefficient.

In step S133, the AC coefficient DPCM unit 212 determines whether all AC coefficients have been processed. If it is determined that there is an unprocessed AC coefficient, the AC coefficient DPCM unit 212 advances the process to step S134.

In step S134, the AC coefficient DPCM unit 212 changes the processing target to the next AC coefficient. In step S135, the AC coefficient DPCM unit 212 determines the difference between the previously processed AC coefficient and the current AC coefficient being processed. When the processing of step S135 is completed, the AC coefficient DPCM unit 212 returns the processing to step S133.

In this manner, the AC coefficient DPCM unit 212 repeatedly performs the processing of steps S133 to S135 as long as it is determined in step S133 that there is an unprocessed AC coefficient. If it is determined in step S133 that there is no unprocessed AC coefficient, the AC coefficient DPCM unit 212 ends the DPCM processing and returns the processing to FIG. 20 .

As described above, the difference between the DC coefficient and the AC coefficient located at the beginning among the AC coefficients is determined and sent to the image decoding device instead of the DC coefficient. Therefore, the image encoding device 10 can suppress an increase in the encoding amount of the scaling list.

Next, an example configuration of an image decoding device according to an embodiment of the present disclosure will be described.

<2-8. Image Decoding Device>

FIG22 is a block diagram showing an example configuration of an image decoding device 300 according to an embodiment of the present disclosure. The image decoding device 300 shown in FIG22 is an image processing device to which the present technology is applied, and is configured to decode the encoded data generated by the image encoding device 10. Referring to FIG22 , the image decoding device 300 includes an accumulation buffer 311, a lossless decoding unit 312, a dequantization/inverse orthogonal transform unit 313, an adder unit 315, a deblocking filter 316, a rearrangement buffer 317, a D/A (digital-to-analog) conversion unit 318, a frame memory 319, selectors 320 and 321, an intra-frame prediction unit 330, and a motion compensation unit 340.

The accumulation buffer 311 temporarily accumulates the encoded stream input via the transmission path using a storage medium.

The lossless decoding unit 312 decodes the encoded stream input from the accumulation buffer 311 according to the encoding scheme used for encoding. The lossless decoding unit 312 also decodes the information multiplexed in the header area of the encoded stream. The information multiplexed in the header area of the encoded stream may include, for example, information for generating the above-mentioned scaling list and information about intra-frame prediction and information about inter-frame prediction contained in the block header. The lossless decoding unit 312 outputs the decoded quantized data and the information for generating the scaling list to the dequantization/inverse orthogonal transform unit 313. The lossless decoding unit 312 also outputs information about intra-frame prediction to the intra-frame prediction unit 330. The lossless decoding unit 312 also outputs information about inter-frame prediction to the motion compensation unit 340.

The dequantization/inverse orthogonal transform unit 313 performs dequantization and inverse orthogonal transform on the quantized data input from the lossless decoding unit 312 to generate prediction error data. Thereafter, the dequantization/inverse orthogonal transform unit 313 outputs the generated prediction error data to the adder unit 315.

The adder unit 315 adds together the prediction error data input from the dequantization/inverse orthogonal transform unit 313 and the predicted image data input from the selector 321 to generate decoded image data. Thereafter, the adder unit 315 outputs the generated decoded image data to the deblocking filter 316 and the frame memory 319.

The deblocking filter 316 filters the decoded image data input from the adder unit 315 to remove block artifacts, and outputs the filtered decoded image data to the rearrangement buffer 317 and the frame memory 319 .

The rearrangement buffer 317 rearranges the image input from the deblocking filter 316 to generate a time series image data sequence. Thereafter, the rearrangement buffer 317 outputs the generated image data to the D/A conversion unit 318.

The D/A conversion unit 318 converts the digital image data input from the rearrangement buffer 317 into an analog image signal. Thereafter, the D/A conversion unit 318 outputs the analog image signal to, for example, a display (not shown) connected to the image decoding device 300 to display the image.

The frame memory 319 stores the decoded image data to be filtered input from the adder unit 31 and the decoded image data after filtering input from the deblocking filter 316 using a storage medium.

The selector 320 switches the destination to which the image data supplied from the frame memory 319 is to be output between the intra prediction unit 330 and the motion compensation unit 340 for each block in the image, based on the mode information acquired by the lossless decoding unit 312. For example, if the intra prediction mode is specified, the selector 320 outputs the decoded image data to be filtered supplied from the frame memory 319 to the intra prediction unit 330 as reference image data. Alternatively, if the inter prediction mode is specified, the selector 320 outputs the filtered decoded image data supplied from the frame memory 319 to the motion compensation unit 340 as reference image data.

The selector 321 switches the source from which the predicted image data to be supplied to the adder unit 315 is output between the intra prediction unit 330 and the motion compensation unit 340 for each block in the image, according to the mode information acquired by the lossless decoding unit 312. For example, if the intra prediction mode is specified, the selector 321 supplies the predicted image data output from the intra prediction unit 330 to the adder unit 315. If the inter prediction mode is specified, the selector 321 supplies the predicted image data output from the motion compensation unit 340 to the adder unit 315.

The intra prediction unit 330 performs intra-screen prediction of pixel values based on the information on intra prediction input from the lossless decoding unit 312 and the reference image data provided from the frame memory 319, and generates predicted image data. Thereafter, the intra prediction unit 330 outputs the generated predicted image data to the selector 321.

The motion compensation unit 340 performs motion compensation processing based on the information on inter prediction input from the lossless decoding unit 312 and the reference image data supplied from the frame memory 319 and generates predicted image data. Thereafter, the motion compensation unit 340 outputs the generated predicted image data to the selector 321.

<2-9. Example Configuration of Dequantization/Inverse Orthogonal Transform Unit>

23 is a block diagram showing an example of the main configuration of the dequantization/inverse orthogonal transform unit 313 of the image decoding device 300 shown in FIG 22. Referring to FIG 23, the dequantization/inverse orthogonal transform unit 313 includes a matrix generation unit 410, a selection unit 430, a dequantization unit 440, and an inverse orthogonal transform unit 450.

(1) Matrix generation unit

The matrix generation unit 410 decodes the encoded scaling list data extracted from the bitstream and provided by the lossless decoding unit 312 and generates a scaling list. The matrix generation unit 410 provides the generated scaling list to the dequantization unit 440.

(2) Select unit

The selection unit 430 selects a transform unit (TU) for inverse orthogonal transform of the image data to be decoded from a plurality of transform units having different sizes. Examples of possible sizes of the transform unit that can be selected by the selection unit 430 include 4×4 and 8×8 for H.264/AVC, and 4×4, 8×8, 16×16, and 32×32 for HEVC. The selection unit 430 can select the transform unit based on, for example, the LCU, SCU, and split_flag included in the header of the coded stream. Thereafter, the selection unit 430 outputs information specifying the size of the selected transform unit to the dequantization unit 440 and the inverse orthogonal transform unit 450.

(3) Dequantization unit

The dequantization unit 440 dequantizes the transform coefficient data quantized when encoding the image by using the scaling list of the transform unit selected by the selection unit 430. Thereafter, the dequantization unit 440 outputs the dequantized transform coefficient data to the inverse orthogonal transform unit 450.

(4) Inverse orthogonal transformation unit

The inverse orthogonal transform unit 450 performs an inverse orthogonal transform on the transform coefficient data dequantized by the dequantization unit 440 in units of the selected transform unit according to the orthogonal transform scheme used for encoding to generate prediction error data. Thereafter, the inverse orthogonal transform unit 450 outputs the generated prediction error data to the adder unit 315.

<2-10. Detailed Example Configuration of Matrix Generation Unit>

24 is a block diagram illustrating an example of a detailed configuration of the matrix generation unit 410 shown in FIG 23. Referring to FIG 24, the matrix generation unit 410 includes a parameter analysis unit 531, a prediction unit 532, an entropy decoding unit 533, a scaling list restoration unit 534, an output unit 535, and a storage unit 536.

(1) Parameter analysis unit

The parameter analysis unit 531 analyzes various flags and parameters regarding the scaling list provided from the lossless decoding unit 312. In addition, based on the analysis result, the parameter analysis unit 531 provides various information provided from the lossless decoding unit 312 (such as encoded data of the difference matrix) to the prediction unit 532 or the entropy decoding unit 533.

For example, if pred_mode is equal to 0, the parameter analysis unit 531 determines that the current mode is the copy mode and provides pred_matrix_id_delta to the copy unit 541. In addition, for example, if pred_mode is equal to 1, the parameter analysis unit 531 determines that the current mode is the full scan mode (normal mode) and provides pred_matrix_id_delta and pred_size_id_delta to the prediction matrix generation unit 542.

In addition, for example, if residual_flag is true, the parameter analysis unit 531 supplies the encoded data (exponential Golomb code) of the scaling list supplied from the lossless decoding unit 312 to the exp-G unit 551 of the entropy decoding unit 533. The parameter analysis unit 531 also supplies residual_symmetry_flag to the exp-G unit 551.

In addition, the parameter analyzing unit 531 supplies residual_down_sampling_flag to the difference matrix size transforming unit 562 of the scaling list restoring unit 534 .

(2) Prediction unit

The prediction unit 532 generates a prediction matrix according to the control of the parameter analysis unit 531. As shown in FIG24, the prediction unit 532 includes a copy unit 541 and a prediction matrix generation unit 542.

In the copy mode, the copy unit 541 copies the previously transmitted scaling list and uses the copied scaling list as the prediction matrix. More specifically, the copy unit 541 reads the previously transmitted scaling list corresponding to pred_matrix_id_delta and having the same size as the scaling list for the current region from the storage unit 536, uses the read scaling list as the predicted image, and provides the predicted image to the output unit 535.

In normal mode, the prediction matrix generation unit 542 generates (or predicts) a prediction matrix using a previously transmitted scaling list. More specifically, the prediction matrix generation unit 542 reads the previously transmitted scaling list corresponding to pred_matrix_id_delta and pred_size_id_delta from the storage unit 536 and generates a prediction matrix using the read scaling list. In other words, the prediction matrix generation unit 542 generates a prediction matrix similar to the prediction matrix generated by the prediction matrix generation unit 172 ( FIG. 16 ) of the image encoding device 10. The prediction matrix generation unit 542 provides the generated prediction matrix to the prediction matrix size conversion unit 561 of the scaling list restoration unit 534.

(3) Entropy decoding unit

The entropy decoding unit 533 restores the difference matrix from the exponential Golomb code supplied from the parameter analyzing unit 531. As shown in FIG.

The exp-G unit 551 decodes the signed or unsigned exponential Golomb code (hereinafter also referred to as exponential Golomb decoding) to restore DPCM data and provides the restored DPCM data to the inverse DPCM unit 552 together with residual_symmetry_flag.

The inverse DPCM unit 552 performs DPCM decoding on the data from which the overlapping portion has been removed to generate residual data from the DPCM data. The inverse DPCM unit 552 supplies the generated residual data to the inverse overlap determination unit 553 together with residual_symmetry_flag.

If residual_symmetry_flag is true, that is, if the residual data is the remainder of a 135-degree symmetric matrix from which the data (matrix elements) of the overlapping symmetric portion have been removed, the inverse overlap determination unit 553 restores the data of the symmetric portion. In other words, the difference matrix of the 135-degree symmetric matrix is restored. Note that if residual_symmetry_flag is not true, that is, if the residual data represents a matrix that is not a 135-degree symmetric matrix, the inverse overlap determination unit 553 uses the residual data as the difference matrix without restoring the data of the symmetric portion. The inverse overlap determination unit 553 provides the difference matrix restored in the above manner to the scaling list restoration unit 534 (difference matrix size conversion unit 562).

(4) Zoom list recovery unit

The scaling list restoration unit 534 restores the scaling list. As shown in FIG24 , the scaling list restoration unit 534 includes a prediction matrix size transform unit 561 , a difference matrix size transform unit 562 , a dequantization unit 563 , and a calculation unit 564 .

If the size of the prediction matrix provided from the prediction unit 532 (prediction matrix generation unit 542) is different from the size of the scaling list for the current region to be restored, the prediction matrix size transformation unit 561 converts the size of the prediction matrix.

For example, if the size of the prediction matrix is larger than the size of the scaling list, the prediction matrix size conversion unit 561 down-converts the prediction matrix. Alternatively, if the size of the prediction matrix is smaller than the size of the scaling list, the prediction matrix size conversion unit 561 up-converts the prediction matrix. The same method as that used by the prediction matrix size conversion unit 181 ( FIG. 16 ) of the image encoding device 10 is selected as the conversion method.

The prediction matrix size conversion unit 561 supplies the prediction matrix whose size has been matched with the size of the scaling list to the calculation unit 564 .

If residual_down_sampling_flag is true, that is, if the size of the transmitted difference matrix is smaller than the size of the current region to be quantized, the difference matrix size conversion unit 562 up-converts the difference matrix to increase the size of the difference matrix to a size corresponding to the current region to be dequantized. Any method for up-conversion can be used. For example, a method corresponding to the down-conversion method performed by the difference matrix size conversion unit 163 ( FIG. 16 ) of the image encoding device 10 can be used.

For example, if the difference matrix size conversion unit 163 has downsampled the difference matrix, the difference matrix size conversion unit 562 may upsample the difference matrix. Alternatively, if the difference matrix size conversion unit 163 has subsampled the difference matrix, the difference matrix size conversion unit 562 may perform inverse subsampling on the difference matrix.

For example, the difference matrix size transforming unit 562 may perform a nearest neighbor interpolation process (Nearest Neighbor) as shown in FIG25 instead of the usual linear interpolation. The nearest neighbor interpolation process can reduce the storage capacity.

Therefore, even if a large-sized scaling list is not transmitted, there is no need to store data obtained after upsampling for upsampling from a small-sized scaling list. In addition, when data involved in calculations during upsampling is stored, an intermediate buffer or the like is not required.

Note that if residual_down_sampling_flag is not true, that is, if the difference matrix is sent with the same size as when used for quantization processing, the difference matrix size transformation unit 562 omits up-conversion of the difference matrix (or may up-convert the difference matrix by a factor of 1).

The difference matrix size transforming unit 562 supplies the difference matrix up-converted in the above-described manner to the dequantizing unit 563 as needed.

The dequantization unit 563 dequantizes the supplied difference matrix (quantized data) using a method corresponding to the method of quantization performed by the quantization unit 183 ( FIG. 16 ) of the image encoding device 10, and supplies the dequantized difference matrix to the calculation unit 564. Note that if the quantization unit 183 is omitted, that is, if the difference matrix supplied from the difference matrix size transform unit 562 is not quantized data, the dequantization unit 563 may be omitted.

The calculation unit 564 adds together the prediction matrix provided from the prediction matrix size conversion unit 561 and the difference matrix provided from the dequantization unit 563 and restores the scaling list for the current region. The calculation unit 564 provides the restored scaling list to the output unit 535 and the storage unit 536.

(5) Output unit

The output unit 535 outputs the provided information to a device outside the matrix generation unit 410. For example, in the copy mode, the output unit 535 provides the prediction matrix provided from the copy unit 541 to the dequantization unit 440 as a scaling list for the current region. In addition, for example, in the normal mode, the output unit 535 provides the scaling list for the current region provided from the scaling list restoration unit 534 (calculation unit 564) to the dequantization unit 440.

(6) Storage unit

The storage unit 536 stores the scaling list provided from the scaling list restoration unit 534 (calculation unit 564), along with the size and list ID of the scaling list. The information about the scaling list stored in the storage unit 536 is used to generate a prediction matrix for another orthogonal transform unit processed later. In other words, the storage unit 536 provides the stored information about the scaling list to the prediction unit 532 as information about the previously transmitted scaling list.

<2-11. Detailed Example Configuration of Inverse DPCM Unit>

26 is a block diagram showing an example of a detailed configuration of the inverse DPCM unit 552 shown in FIG 24. Referring to FIG 26 , the inverse DPCM unit 552 includes an initial setting unit 571, a DPCM decoding unit 572, and a DC coefficient extraction unit 573.

The initial setting unit 571 acquires sizeID and MatrixID, and sets various variables to initial values. The initial setting unit 571 supplies the acquired and set information to the DPCM decoding unit 572.

The DPCM decoding unit 572 determines each coefficient (DC coefficient and AC coefficient) from the difference (scaling_list_delta_coef) between the DC coefficient and the AC coefficient using the initial setting etc. supplied from the initial setting unit 571. The DPCM decoding unit 572 supplies the determined coefficients to the DC coefficient extraction unit 573 (ScalingList[i]).

The DC coefficient extraction unit 573 extracts the DC coefficient from the coefficients (ScalingList[i]) provided by the DPCM decoding unit 572. The DC coefficient is located at the beginning of the AC coefficients. In other words, the initial coefficient (ScalingList[0]) among the coefficients provided by the DPCM decoding unit 572 is the DC coefficient. The DC coefficient extraction unit 573 extracts the coefficient at the beginning as the DC coefficient and outputs the extracted coefficient to the inverse overlap determination unit 553 (DC_coef). The DC coefficient extraction unit 573 outputs the remaining coefficients (ScalingList[i] (i>0)) to the inverse overlap determination unit 553 as AC coefficients.

Therefore, the inverse DPCM unit 552 can perform correct DPCM decoding and can obtain the DC coefficient and the AC coefficient. That is, the image decoding device 300 can suppress an increase in the encoding amount of the scaling list.

<2-12. Flow of Quantization Matrix Decoding Processing>

An example of the flow of the quantization matrix decoding process performed by the matrix generation unit 410 having the above-described configuration will be described with reference to the flowchart shown in FIG. 27 .

When the quantization matrix decoding process starts, in step S301 , the parameter analysis unit 531 reads the quantized values ( Qscale0 to Qscale3 ) of regions 0 to 3 .

In step S302, the parameter analysis unit 531 reads pred_mode. In step S303, the parameter analysis unit 531 determines whether pred_mode is equal to 0. If it is determined that pred_mode is equal to 0, the parameter analysis unit 531 determines that the current mode is the copy mode, and advances the process to step S304.

In step S304, the parameter analysis unit 531 reads pred_matrix_id_delta. In step S305, the copy unit 541 copies the transmitted scaling list and uses the copied scaling list as the prediction matrix. In copy mode, the prediction matrix is output as the scaling list for the current region. When the processing of step S305 is completed, the copy unit 541 ends the quantization matrix decoding process.

In addition, if it is determined in step S303 that pred_mode is not equal to 0, the parameter analysis unit 531 determines that the current mode is the full scan mode (normal mode), and advances the process to step S306.

In step S306, the parameter analysis unit 531 reads pred_matrix_id_delta, pred_size_id_delta, and residual_flag. In step S307, the prediction matrix generation unit 542 generates a prediction matrix from the transmitted scaling list.

In step S308, the parameter analysis unit 531 determines whether residual_flag is true. If it is determined that residual_flag is not true, there is no residual matrix, and the prediction matrix generated in step S307 is output as the scaling list for the current region. Therefore, in this case, the parameter analysis unit 531 ends the quantization matrix decoding process.

In addition, if it is determined in step S308 that residual_flag is true, the parameter analysis unit 531 advances the process to step S309 .

In step S309 , the parameter analysis unit 531 reads residual_down_sampling_flag and residual_symmetry_flag.

In step S310 , the exp-G unit 551 and the inverse DPCM unit 552 decode the exponential Golomb code of the residual matrix and generate residual data.

In step S311, the inverse overlap determination unit 553 determines whether residual_symmetry_flag is true. If it is determined that residual_symmetry_flag is true, the inverse overlap determination unit 553 advances the process to step S312 and restores the removed overlapping portion of the residual data (or performs inverse symmetry processing). When the difference matrix is generated as a 135-degree symmetric matrix in the above manner, the inverse overlap determination unit 553 advances the process to step S313.

In addition, if it is determined in step S311 that residual_symmetry_flag is not true (or if the residual data is such a difference matrix that the difference matrix is not a 135-degree symmetric matrix), the inverse overlap determination unit 553 advances the processing to step S313 while skipping the processing of step S312 (or not performing inverse symmetry processing).

In step S313, the difference matrix size conversion unit 562 determines whether residual_down_sampling_flag is true. If residual_down_sampling_flag is true, the difference matrix size conversion unit 562 advances the process to step S314 and up-converts the difference matrix to a size corresponding to the current region to be dequantized. After the difference matrix is up-converted, the difference matrix size conversion unit 562 advances the process to step S315.

In addition, if it is determined in step S313 that residual_down_sampling_flag is not true, the difference matrix size transform unit 562 advances the process to step S315 while skipping the process of step S314 (or not up-converting the difference matrix).

In step S315, the calculation unit 564 adds the difference matrix and the prediction matrix to generate a scaling list for the current region. When the process of step S315 is completed, the quantization matrix decoding process ends.

<2-13. Flow of Residual Signal Decoding Processing>

Next, an example of the flow of the residual signal decoding process performed in step S310 in FIG. 27 will be described with reference to the flowchart shown in FIG. 28 .

When the residual signal decoding process starts, in step S331 , the exp-G unit 551 decodes the supplied exponential Golomb code.

In step S332 , the inverse DPCM unit 552 performs inverse DPCM processing on the DPCM data obtained by decoding by the exp-G unit 551 .

When the inverse DPCM process is completed, the inverse DPCM unit 552 ends the residual signal decoding process and returns the process to FIG. 27 .

<2-14. Flow of Inverse DPCM Processing>

Next, an example of the flow of the inverse DPCM process performed in step S332 in FIG. 28 will be described with reference to the flowchart shown in FIG. 29 .

When the inverse DPCM process starts, in step S351 , the initial setting unit 571 acquires sizeID and MatrixID.

In step S352 , the initial setting unit 571 sets coefNum as follows.

coefNum=min((1<<(4+(sizeID<<1))),65)

In step S353 , the initial setting unit 571 sets the variable i and the variable nextcoef as follows.

i＝0

nextcoef=8

In step S354, the DPCM decoding unit 572 determines whether the variable i < coefNum. If the variable i is smaller than coefNum, the initial setting unit 571 advances the process to step S355.

In step S355 , the DPCM decoding unit 572 reads the DPCM data of the coefficient (scaling_list_delta_coef).

In step S356 , the DPCM decoding unit 572 determines nextcoef as follows using the read DPCM data, and also determines scalingList[i].

nextcoef=(nextcoef+scaling_list_delta_coef+256)%256

scalingList[i] = nextcoef

In step S357, the DC coefficient extraction unit 573 determines whether sizeID is greater than 1 and whether the variable i is equal to 0 (that is, the coefficient at the beginning). If it is determined that sizeID is greater than 1 and the variable i represents the coefficient at the beginning, the DC coefficient extraction unit 573 advances the process to step S358 and uses the coefficient as the DC coefficient (DC_coef=nextcoef). When the process of step S358 is completed, the DC coefficient extraction unit 573 advances the process to step S360.

If, in step S357, it is determined that sizeID is less than or equal to 1 or that the variable i does not represent the first coefficient, the DC coefficient extraction unit 573 advances the process to step S359 and changes the variable i by one for each coefficient since the DC coefficient has already been extracted (ScalingList[(i-(sizeID)>1)?1;0]=nextcoef). If the process of step S359 is complete, the DC coefficient extraction unit 573 advances the process to step S360.

In step S360 , the DPCM decoding unit 572 increments the variable i to change the processing target to the subsequent coefficient, and then returns the process to step S354 .

In step S354, the processing of steps S354 to S360 is repeatedly performed until it is determined that the variable i is greater than or equal to coefNum. If it is determined in step S354 that the variable i is greater than or equal to coefNum, the DPCM decoding unit 572 ends the inverse DPCM processing and returns the processing to FIG.

Therefore, the difference between the DC coefficient and the AC coefficient located at the beginning of the AC coefficients can be correctly decoded.Therefore, the image decoding device 300 can suppress an increase in the encoding amount of the scaling list.

<3. Third embodiment>

<3-1. Syntax: Second Method>

Another method for transmitting the difference between the DC coefficient and another coefficient instead of the DC coefficient may be, for example, transmitting the difference between the DC coefficient and the (0,0) component of the 8×8 matrix as DPCM data different from the DPCM data of the 8×8 matrix (second method). For example, after the DPCM data of the 8×8 matrix is transmitted, the difference between the DC coefficient and the (0,0) component of the 8×8 matrix may be transmitted.

Therefore, similarly to the first method, when the value of the (0,0) coefficient (AC coefficient) and the value of the DC coefficient of the 8×8 matrix are close to each other, the compression ratio can be further improved.

FIG30 shows the syntax of the scaling list in the second method. In the example shown in FIG30 , 64 differences between coefficients are read (scaling_list_delta_coef). Finally, the difference between the DC coefficient and the (0,0) coefficient (AC coefficient) is read (scaling_list_dc_coef_delta), and the DC coefficient is determined from this difference.

In the second method, accordingly, the syntax for decoding AC coefficients may be similar to the syntax of the related art shown in FIG12. That is, the syntax of the second method may be obtained by slightly modifying the example of the related art, and the syntax of the second method may be more feasible than the syntax of the first method.

However, the first method allows the image decoding apparatus to restore the DC coefficient when the image decoding apparatus receives the initial coefficient, while the second method does not allow the image decoding apparatus to obtain the DC coefficient until the image decoding apparatus has received all coefficients and decompressed all DPCM data.

An image encoding device that implements the syntax of the above-mentioned second method will be described below.

<3-2. Detailed Example Configuration of DPCM Unit>

In the second method, the image encoding device 10 has a configuration substantially similar to that of the first method described above. Specifically, the image encoding device 10 has the same configuration as that shown in FIG14 . Furthermore, the orthogonal transform/quantization unit 14 has the same configuration as that shown in FIG15 . Furthermore, the matrix processing unit 150 has the same configuration as that shown in FIG16 .

An example configuration of the DPCM unit 192 in the second example is shown in Fig. 31. As shown in Fig. 31, in the second example, the DPCM unit 192 includes an AC coefficient buffer 611, an AC coefficient encoding unit 612, an AC coefficient DPCM unit 613, and a DC coefficient DPCM unit 614.

The AC coefficient buffer 611 stores the initial AC coefficient (that is, the (0, 0) coefficient) supplied from the overlap determination unit 191. The AC coefficient buffer 611 supplies the stored initial AC coefficient (AC coefficient (0, 0)) to the DC coefficient DPCM unit 614 at a predetermined timing after all AC coefficients have been subjected to DPCM processing, or in response to a request.

The AC coefficient encoding unit 612 obtains the initial AC coefficient (AC coefficient (0, 0)) supplied from the overlap determination unit 191 and subtracts the value of the initial AC coefficient from a constant (e.g., 8). The AC coefficient encoding unit 612 supplies the subtraction result (difference) to the exp-G unit 193 as the initial coefficient (scaling_list_delta_coef (i=0)) of the DPCM data of the AC coefficient.

The AC coefficient DPCM unit 613 obtains the AC coefficient supplied from the overlap determination unit 191, determines a difference (DPCM) from the previous AC coefficient for each of the second and subsequent AC coefficients, and supplies the determined difference to the exp-G unit 193 as DPCM data (scaling_list_delta_coef (i=1 to 63)).

The DC coefficient DPCM unit 614 acquires the DC coefficient supplied from the overlap determination unit 191. The DC coefficient DPCM unit 614 also acquires the initial AC coefficient (AC coefficient (0, 0)) stored in the AC coefficient buffer 611. The DC coefficient DPCM unit 614 subtracts the initial AC coefficient (AC coefficient (0, 0)) from the DC coefficient to determine the difference therebetween, and supplies the determined difference to the exp-G unit 193 as DPCM data (scaling_list_dc_coef_delta) of the DC coefficient.

As described above, in the second method, the difference between the DC coefficient and another coefficient (the initial AC coefficient) is determined. Then, after transmitting the DPCM data (scaling_list_delta_coef) of the AC coefficient as the difference between the AC coefficients, the difference is transmitted as the DPCM data (scaling_list_dc_coef_delta) of the DC coefficient that is different from the DPCM data of the AC coefficient. Thus, similar to the first method, the image encoding device 10 can improve the encoding efficiency of the scaling list.

<3-3. DPCM Processing Flow>

Furthermore, in the second method, the image encoding device 10 performs the quantization matrix encoding process in a manner similar to that in the first method described with reference to the flowchart shown in FIG. 20 .

An example of the flow of the DPCM process in the second method performed in step S112 in FIG. 20 will be described with reference to the flowchart shown in FIG. 32 .

When the DPCM process starts, in step S401 , the AC coefficient buffer 611 stores initial AC coefficients.

In step S402 , the AC coefficient encoding unit 612 subtracts the initial AC coefficient from a predetermined constant (eg, 8) to determine a difference therebetween (initial DPCM data).

The AC coefficient DPCM unit 613 performs the processing of steps S403 to S405 in a manner similar to the processing of steps S133 to S135 in Figure 21. That is, the processing of steps S403 to S405 is repeatedly performed to generate DPCM data of all AC coefficients (differences from the previous AC coefficient).

If it is determined in step S403 that all the AC coefficients have been processed (that is, if there is no unprocessed AC coefficient), the AC coefficient DPCM unit 613 advances the process to step S406 .

In step S406 , the DC coefficient DPCM unit 614 subtracts the initial AC coefficient saved in step S401 from the DC coefficient to determine the difference therebetween (DPCM data of the DC coefficient).

When the process of step S406 is completed, the DC coefficient DPCM unit 614 ends the DPCM process and returns the process to FIG. 20 .

Therefore, the difference between the DC coefficient and another coefficient is also determined and sent to the image decoding device as DPCM data.Therefore, the image encoding device 10 can suppress an increase in the encoding amount of the scaling list.

<3-4. Detailed Example Configuration of Inverse DPCM Unit>

In the second method, the image decoding device 300 has a configuration substantially similar to that in the first method. Specifically, in the second method, the image decoding device 300 also has the same configuration as the example shown in FIG22. In addition, the dequantization/inverse orthogonal transform unit 313 has the same configuration as the example shown in FIG23. Furthermore, the matrix generation unit 410 has the same configuration as the example shown in FIG24.

33 is a block diagram showing an example of a detailed configuration of the inverse DPCM unit 552 shown in FIG24 in the second method. Referring to FIG33 , the inverse DPCM unit 552 includes an initial setting unit 621, an AC coefficient DPCM decoding unit 622, an AC coefficient buffer 623, and a DC coefficient DPCM decoding unit 624.

The initial setting unit 621 acquires sizeID and MatrixID, and sets various variables to initial values. The initial setting unit 621 supplies the acquired and set information to the AC coefficient DPCM decoding unit 622.

The AC coefficient DPCM decoding unit 622 obtains the DPCM data (scaling_list_delta_coef) of the AC coefficients provided by the exp-G unit 551. The AC coefficient DPCM decoding unit 622 decodes the obtained DPCM data of the AC coefficients using the initial settings provided by the initial setting unit 621, etc., to determine the AC coefficients. The AC coefficient DPCM decoding unit 622 provides the determined AC coefficients (ScalingList[i]) to the inverse overlap determination unit 553. The AC coefficient DPCM decoding unit 622 also provides the initial AC coefficient (ScalingList[0], that is, the AC coefficient (0,0)) among the determined AC coefficients to the AC coefficient buffer 623 for storage.

The AC coefficient buffer 623 stores the initial AC coefficient (ScalingList[0], that is, AC coefficient (0, 0)) provided from the AC coefficient DPCM decoding unit 622. The AC coefficient buffer 623 provides the initial AC coefficient (ScalingList[0], that is, AC coefficient (0, 0)) to the DC coefficient DPCM decoding unit 624 at a predetermined timing or in response to a request.

The DC coefficient DPCM decoding unit 624 obtains the DPCM data (scaling_list_dc_coef_delta) of the DC coefficient provided by the exp-G unit 551. The DC coefficient DPCM decoding unit 624 also obtains the initial AC coefficient (ScalingList[0], that is, the AC coefficient (0,0)) stored in the AC coefficient buffer 623. The DC coefficient DPCM decoding unit 624 decodes the DPCM data of the DC coefficient using the initial AC coefficient to determine the DC coefficient. The DC coefficient DPCM decoding unit 624 provides the determined DC coefficient (DC_coef) to the inverse overlap determination unit 553.

Therefore, the inverse DPCM unit 552 can perform correct DPCM decoding and can obtain the DC coefficient and the AC coefficient. That is, the image decoding device 300 can suppress an increase in the encoding amount of the scaling list.

<3-5. Flow of Inverse DPCM Processing>

In addition, in the second method, the image decoding device 300 performs quantization matrix decoding processing in a manner similar to that in the first method described above with reference to the flowchart shown in FIG27. Similarly, the image decoding device 300 performs residual signal decoding processing in a manner similar to that in the first method described above with reference to the flowchart shown in FIG28.

An example of the flow of the inverse DPCM process performed by the inverse DPCM unit 552 will be described with reference to the flowchart shown in FIG. 34 .

When the inverse DPCM process starts, in step S421 , the initial setting unit 621 acquires sizeID and MatrixID.

In step S422 , the initial setting unit 621 sets coefNum as follows.

coefNum=min((1<<(4+(sizeID<<1))),64)

In step S423 , the initial setting unit 621 sets the variable i and the variable nextcoef as follows.

i＝0

nextcoef=8

In step S424, the DPCM decoding unit 572 determines whether the variable i<coefNum. If the variable i is smaller than coefNum, the initial setting unit 621 advances the process to step S425.

In step S425 , the AC coefficient DPCM decoding unit 622 reads the DPCM data (scaling_list_delta_coef) of the AC coefficient.

In step S426 , the AC coefficient DPCM decoding unit 622 determines nextcoef as follows using the read DPCM data, and also determines scalingList[i].

nextcoef=(nextcoef+scaling_list_delta_coef+256)%256

scalingList[i] = nextcoef

Note that the calculated initial AC coefficient (ScalingList[0], that is, AC coefficient (0,0)) is stored in the AC coefficient buffer 623.

In step S427 , the AC coefficient DPCM decoding unit 622 increments the variable i to change the target to be processed to the subsequent coefficient, and then returns the process to step S424 .

In step S424 , the processing of steps S424 to S427 is repeatedly performed until it is determined that the variable i is greater than or equal to coefNum. If it is determined in step S424 that the variable i is greater than or equal to coefNum, the AC coefficient DPCM decoding unit 622 advances the processing to step 428 .

In step S428 , the DC coefficient DPCM decoding unit 624 determines whether sizeID is greater than 1. If it is determined that sizeID is greater than 1, the DC coefficient DPCM decoding unit 624 advances the process to step S429 and reads the DPCM data of the DC coefficient (scaling_list_dc_coef_delta).

In step S430 , the DC coefficient DPCM decoding unit 624 acquires the initial AC coefficient (ScalingList[0], that is, AC coefficient (0,0)) stored in the AC coefficient buffer 623 , and decodes the DPCM data of the DC coefficient (DC_coef) using the initial AC coefficient as follows.

DC_coef=scaling_list_dc_coef_delta+ScalingList[0]

When the DC coefficient (DC_coef) is obtained, the DC coefficient DPCM decoding unit 624 ends the inverse DPCM process and returns the process to FIG. 28 .

In addition, if it is determined in step S428 that sizeID is less than or equal to 1, the DC coefficient DPCM decoding unit 624 ends the inverse DPCM process and returns the process to FIG. 28 .

Therefore, the difference between the DC coefficient and the AC coefficient located at the beginning of the AC coefficients can be correctly decoded.Therefore, the image decoding device 300 can suppress an increase in the encoding amount of the scaling list.

<4. Fourth embodiment>

<4-1. Syntax: Third Method>

In the above-mentioned second method, the DC coefficient may also be limited to a value smaller than the initial AC coefficient (AC coefficient (0,0)) (third method).

This ensures that the DPCM data for the DC coefficient (that is, the difference obtained by subtracting the initial AC coefficient from the DC coefficient) can be positive. Therefore, this DPCM data can be encoded using an unsigned exponential Golomb code. Thus, the third method prevents the DC coefficient from being larger than the initial AC coefficient, but reduces the amount of code compared to the first and second methods.

Fig. 35 shows the syntax of the scaling list in the third method. As shown in Fig. 35 , in this case, DPCM data of the DC coefficient (scaling_list_dc_coef_delta) is limited to positive values.

The syntax of the third method described above can be implemented by the image encoding device 10 similar to the image encoding device in the second method. However, in the third method, the exp-G unit 193 can encode the DPCM data of the DC coefficient using an unsigned exponential Golomb code. Note that the image encoding device 10 can perform processing such as quantization matrix encoding processing and DPCM processing in a manner similar to that in the second method.

In addition, the syntax of the third method can be implemented in a manner similar to that in the second method by the image decoding device 300. In addition, the image decoding device 300 can perform quantization matrix decoding processing in a manner similar to that in the second method.

<4-2. Flow of Inverse DPCM Processing>

An example of the flow of the inverse DPCM process performed by the inverse DPCM unit 552 will be described with reference to the flowchart shown in FIG. 36 .

The processing of steps S451 to S459 is performed in a manner similar to the processing of steps S421 to S429 in FIG. 34 .

In step S460 , the DC coefficient DPCM decoding unit 624 acquires the initial AC coefficient (ScalingList[0], that is, AC coefficient (0,0)) stored in the AC coefficient buffer 623 , and decodes the DPCM data of the DC coefficient (DC_coef) using the initial AC coefficient as follows.

DC_coef=ScalingList[0]-scaling_list_dc_coef_delta

When the DC coefficient (DC_coef) is obtained, the DC coefficient DPCM decoding unit 624 ends the inverse DPCM process and returns the process to FIG. 28 .

In addition, if it is determined in step S458 that sizeID is less than or equal to 1, the DC coefficient DPCM decoding unit 624 ends the inverse DPCM process and returns the process to FIG. 28 .

Therefore, the difference between the DC coefficient and the AC coefficient located at the beginning of the AC coefficients can be correctly decoded.Therefore, the image decoding device 300 can suppress an increase in the encoding amount of the scaling list.

<5. Fifth embodiment>

<5-1. Grammar: Fourth Method>

Another method for transmitting the difference between the DC coefficient and another coefficient instead of the DC coefficient is, for example, to collect only the DC coefficients of a plurality of scaling lists and perform DPCM by obtaining the difference between the DC coefficient and the AC coefficient of each scaling list (fourth method). In this case, the DPCM data of the DC coefficient is a collection of data for a plurality of scaling lists and is transmitted as data different from the DPCM data of the AC coefficient of each scaling list.

Therefore, for example, when there is a correlation between the DC coefficients of the scaling lists (MatrixID), the compression ratio can be further improved.

FIG37 shows the syntax of the DC coefficient of the scaling list in the fourth method. In this case, as shown in the example shown in FIG37, since the DC coefficient is processed in a cycle different from the cycle of the AC coefficients of each scaling list, the processing for the AC coefficient and the processing for the DC coefficient need to be independent of each other.

This ensures that more various methods for scaling list encoding and decoding processes can be implemented, but the complexity of DPCM processing and inverse DPCM processing may increase. For example, a process for copying only AC coefficients in copy mode and making the value of DC coefficients different can be easily implemented.

The number of scaling lists whose DC coefficients are processed together is arbitrary.

<5-2. Detailed Example Configuration of DPCM Unit>

In the fourth method, the image encoding device 10 has a configuration substantially similar to that of the first method described above. Specifically, the image encoding device 10 has the same configuration as that shown in FIG14 . Furthermore, the orthogonal transform/quantization unit 14 has the same configuration as that shown in FIG15 . Furthermore, the matrix processing unit 150 has the same configuration as that shown in FIG16 .

An example configuration of the DPCM unit 192 in the fourth method is shown in Fig. 38. As shown in Fig. 38, in this case, the DPCM unit 192 includes an AC coefficient DPCM unit 631, a DC coefficient buffer 632, and a DC coefficient DPCM unit 633.

The AC coefficient DPCM unit 631 performs DPCM processing on each AC coefficient of each scaling list provided from the overlap determination unit 191. Specifically, the AC coefficient DPCM unit 631 subtracts the initial AC coefficient from a predetermined constant (e.g., 8) for each scaling list, and subtracts the AC coefficient being processed (current AC coefficient) from the previous AC coefficient. The AC coefficient DPCM unit 631 provides the DPCM data (scaling_list_delta_coef) generated for each scaling list to the exp-G unit 193.

The DC coefficient buffer 632 stores the DC coefficient of each scaling list supplied from the overlap determination unit 191. The DC coefficient buffer 632 supplies the stored DC coefficient to the DC coefficient DPCM unit 633 at a predetermined timing or in response to a request.

The DC coefficient DPCM unit 633 obtains the DC coefficient accumulated in the DC coefficient buffer 632. The DC coefficient DPCM unit 633 determines DPCM data for the obtained DC coefficient. Specifically, the DC coefficient DPCM unit 633 subtracts the initial DC coefficient from a predetermined constant (e.g., 8) and subtracts the DC coefficient being processed (the current DC coefficient) from the previous DC coefficient. The DC coefficient DPCM unit 633 provides the generated DPCM data (scaling_list_delta_coef) to the exp-G unit 193.

Therefore, the image encoding device 10 can improve the encoding efficiency of the scaling list.

<5-3. DPCM Processing Flow>

Furthermore, in the fourth method, the image encoding device 10 performs the quantization matrix encoding process in a manner similar to that in the first method described above with reference to the flowchart shown in FIG. 20 .

An example of the flow of the DPCM process in the fourth method performed in step S112 in FIG. 20 will be described with reference to the flowchart shown in FIG. 39 .

The AC coefficient DPCM unit 631 performs the processing of steps S481 to S485 in a manner similar to the processing of steps S401 to S405 in FIG. 32 (the processing in the second method).

If it is determined in step S483 that all the AC coefficients have been processed, the AC coefficient DPCM unit 631 advances the process to step S486 .

In step S486, the AC coefficient DPCM unit 631 determines whether all scaling lists (or difference matrices) in which the DC coefficient is commonly DPCM-encoded have been processed. If it is determined that there is an unprocessed scaling list (or difference matrix), the AC coefficient DPCM unit 631 returns the process to step S481.

If it is determined in step S486 that all scaling lists (or difference matrices) have been processed, the AC coefficient DPCM unit 631 advances the process to step S487.

The DC coefficient DPCM unit 633 performs the processes of steps S487 to S491 on the DC coefficient stored in the DC coefficient buffer 632 in a manner similar to the processes of steps S481 to S485 .

If it is determined in step S489 that all the DC coefficients stored in the DC coefficient buffer 632 have been processed, the DC coefficient DPCM unit 633 ends the DPCM processing and returns the processing to FIG. 20 .

By performing the DPCM process in the above-described manner, the image encoding device 10 can improve the encoding efficiency of the scaling list.

<5-4. Detailed Example Configuration of Inverse DPCM Unit>

The image decoding device 300 in the fourth method has a configuration that is basically similar to that in the first method. Specifically, in the fourth method, the image decoding device 300 also has the same configuration as the example shown in FIG22. In addition, the dequantization/inverse orthogonal transformation unit 313 has the same configuration as the example shown in FIG23. In addition, the matrix generation unit 410 has the same configuration as the example shown in FIG24.

40 is a block diagram showing an example of a detailed configuration of the inverse DPCM unit 552 shown in FIG24 in the fourth method.

The initial setting unit 641 acquires sizeID and MatrixID and sets various variables to initial values. The initial setting unit 641 supplies the acquired and set information to the AC coefficient DPCM decoding unit 642 and the DC coefficient DPCM decoding unit 643.

The AC coefficient DPCM decoding unit 642 acquires the DPCM data (scaling_list_delta_coef(ac)) of the AC coefficient supplied from the exp-G unit 551. The AC coefficient DPCM decoding unit 642 decodes the acquired DPCM data of the AC coefficient using the initial settings supplied from the initial setting unit 641, and determines the AC coefficient. The AC coefficient DPCM decoding unit 642 supplies the determined AC coefficient (ScalingList[i]) to the inverse overlap determination unit 553. The AC coefficient DPCM decoding unit 642 performs the above-described processing on a plurality of scaling lists.

The DC coefficient DPCM decoding unit 643 obtains the DPCM data of the DC coefficient (scaling_list_delta_coef(dc)) provided from the exp-G unit 551. The DC coefficient DPCM decoding unit 643 decodes the obtained DPCM data of the DC coefficient using the initial settings provided from the initial setting unit 641, and determines the DC coefficient of each scaling list. The DC coefficient DPCM decoding unit 643 provides the determined DC coefficient (scaling_list_dc_coef) to the inverse overlap determination unit 553.

Therefore, the inverse DPCM unit 552 can perform correct DPCM decoding and can obtain the DC coefficient and the AC coefficient. That is, the image decoding device 300 can suppress an increase in the encoding amount of the scaling list.

<5-5. Flow of Inverse DPCM Processing>

In addition, in the fourth method, the image decoding device 300 performs a quantization matrix decoding process in a manner similar to that in the first method described above with reference to the flowchart shown in FIG27. Similarly, the image decoding device 300 performs a residual signal decoding process in a manner similar to that in the first method described above with reference to the flowchart shown in FIG28.

An example of the flow of the inverse DPCM process performed by the inverse DPCM unit 552 will be described with reference to the flowcharts shown in FIG. 41 and FIG. 42 .

When the inverse DPCM process starts, the initial setting unit 641 and the AC coefficient DPCM decoding unit 642 perform the processes of steps S511 to S517 in a manner similar to that in the processes of steps S421 to S427 in FIG. 34 .

If it is determined in step S514 that the variable i is greater than or equal to coefNum, the AC coefficient DPCM decoding unit 642 advances the process to step S518 .

In step S518, the AC coefficient DPCM decoding unit 642 determines whether all scaling lists (difference matrices) in which the DC coefficient is commonly subjected to DPCM processing have been processed. If it is determined that there is an unprocessed scaling list (difference matrix), the AC coefficient DPCM decoding unit 642 returns the process to step S511 and repeatedly performs the subsequent processing.

In addition, if it is determined that there is no unprocessed scaling list (difference matrix), the AC coefficient DPCM decoding unit 642 advances the process to FIG. 42 .

In step S521 in FIG. 42 , the initial setting unit 641 sets sizeID and a variable nextcoef as follows.

sizeID=2

nextcoef=8

In addition, in step S522 , the initial setting unit 641 sets MatrixID as follows.

MatrixID=0

In step S523 , the DC coefficient DPCM decoding unit 643 determines whether sizeID<4. If it is determined that sizeID is smaller than 4, the DC coefficient DPCM decoding unit 643 advances the process to step S524 .

In step S524, the DC coefficient DPCM decoding unit 643 determines whether MatrixID<(sizeID==3)?2:6 is satisfied. If it is determined that MatrixID<(sizeID==3)?2:6 is satisfied, the DC coefficient DPCM decoding unit 643 advances the process to step S525.

In step S525 , the DC coefficient DPCM decoding unit 643 reads the DPCM data (scaling_list_delta_coef) of the DC coefficient.

In step S526 , the DC coefficient DPCM decoding unit 643 determines nextcoef as follows using the read DPCM data, and also determines scaling_dc_coef.

nextcoef=(nextcoef+scaling_list_delta_coef+256)%256

scaling_dc_coef[sizeID-2][MatrixID]=nextcoef

In step S527 , the DC coefficient DPCM decoding unit 643 increments the MatrixID to change the processing target to the subsequent DC coefficient (the subsequent scaling list or residual matrix), and then returns the process to step S524 .

If it is determined in step S524 that MatrixID<(sizeID==3)?2:6 is not satisfied, the DC coefficient DPCM decoding unit 643 advances the process to step S528.

In step S528 , the DC coefficient DPCM decoding unit 643 increments the sizeID to change the processing target to the subsequent DC coefficient (the subsequent scaling list or residual matrix), and then returns the process to step S523 .

If it is determined in step S523 that sizeID is greater than or equal to 4, the DC coefficient DPCM decoding unit 643 ends the inverse DPCM process and returns the process to FIG. 28 .

Therefore, the difference between the DC coefficients can be correctly decoded.Therefore, the image decoding device 300 can suppress an increase in the encoding amount of the scaling list.

<6. Sixth embodiment>

<6-1. Other syntax: First example>

FIG43 shows another example of the syntax of the scaling list. This figure corresponds to FIG12. In the example shown in FIG12, the initial value of nextcoef is set to a predetermined constant (e.g., 8). Alternatively, as shown in FIG43, the initial value of nextcoef can be overwritten by the DPCM data of the DC coefficient (scaling_list_dc_coef_minus8).

Therefore, the encoding amount of the initial AC coefficient (AC coefficient (0, 0)) in the 16×16 scaling list and the 32×32 scaling list can be reduced.

<6-2. Other syntax: Second example>

FIG44 shows another example of the syntax of the zoom list. This figure corresponds to FIG12.

In the example shown in Figure 12, when the value of scaling_list_pred_matrix_id_delta, which is information specifying a reference destination in copy mode, is "0", a scaling list that is one scaling list earlier than the current scaling list being processed is referenced, and when the value of scaling_list_pred_matrix_id_delta is "1", a scaling list that is two scaling lists earlier than the current scaling list being processed is referenced.

In contrast, in the example shown in FIG. 44 , as shown in part C of FIG. 44 , when the value of scaling_list_pred_matrix_id_delta, which is information specifying a reference destination in copy mode, is “0,” the default scaling list is referenced, and when the value of scaling_list_pred_matrix_id_delta is “1,” the previous scaling list is referenced.

In this way, modifying the semantics of scaling_list_pred_matrix_id_delta can simplify the syntax in the manner shown in part B of FIG. 44 , and can reduce the load of the DPCM process and the inverse DPCM process.

<6-3. Other syntax: third example>

FIG45 shows another example of the syntax of the zoom list. This figure corresponds to FIG12.

In the example shown in FIG. 45 , both the example shown in FIG. 43 described above and the example shown in FIG. 44 described above are used.

In the example shown in FIG45 , the amount of coding of the initial AC coefficient (AC coefficient (0,0)) in the 16×16 scaling list and the 32×32 scaling list can be reduced accordingly. In addition, the syntax can be simplified, and the load of the DPCM process and the inverse DPCM process can be reduced.

In the above embodiments, the value of the predetermined constant is arbitrary. In addition, the size of the zoom list is also arbitrary.

In addition, although a description has been given above of the size conversion processing of the scaling list, the prediction matrix, or the difference matrix therebetween, the size conversion processing may be processing for actually generating a matrix whose size has been converted, or may be processing for setting how each element in the matrix is read from the memory without actually generating the data of the matrix (reading control of the matrix data).

In the above-mentioned resizing process, each element in the matrix whose size has been transformed is composed of any element in the matrix whose size has not yet been transformed. That is, by reading the elements in the matrix whose size has not yet been transformed stored in the memory using a certain method (such as, reading some elements in the matrix or reading an element multiple times), the matrix whose size has been transformed can be generated. In other words, a method for reading each element (or performing read control of matrix data) is defined to basically achieve the above-mentioned resizing. This method can eliminate the processing such as writing the matrix data whose size has been transformed to the memory. In addition, the reading of the matrix data whose size has been transformed basically depends on how to perform nearest neighbor interpolation, etc. Therefore, the resizing can be achieved through a relatively low-load processing (such as, selecting a suitable option from a plurality of options prepared in advance). Therefore, the above-mentioned method can reduce the load of resizing.

That is, the above-mentioned size conversion processing includes processing for actually generating matrix data whose size has been converted, and also includes reading control of the matrix data.

Note that although the above description is made in the case where the difference matrix is encoded and transmitted, this is merely illustrative, and the scaling list may be encoded and transmitted. In other words, the AC coefficients and DC coefficients of the scaling list described above as coefficients to be processed may be the AC coefficients and DC coefficients of the difference matrix between the scaling list and the prediction matrix.

In addition, the encoding amount of information about parameters, flags, etc. of the scaling list (such as the size and list ID of the scaling list) can be reduced by, for example, obtaining the difference between the information and previously transmitted information and transmitting the difference.

In addition, although the foregoing description has been made in the case where a large-sized quantization matrix or difference matrix is down-converted and transmitted, this is merely illustrative, and the quantization matrix or difference matrix may be transmitted without being down-converted while the size of the quantization matrix used for quantization remains unchanged.

The present technique can be applied to any type of image encoding and decoding involving quantization and dequantization.

In addition, the present technology can also be applied to, for example, image encoding devices and image decoding devices for receiving image information (bitstream) compressed using orthogonal transform (such as discrete cosine transform) and motion compensation (such as MPEG or H.26x) via network media (such as satellite broadcasting, cable television, the Internet, or mobile phones). The present technology can also be applied to image encoding devices and image decoding devices for processing on storage media (such as optical disks, magnetic disks, and flash memory). In addition, the present technology can also be applied to quantization devices and dequantization devices included in the above-mentioned image encoding devices and image decoding devices.

<7. Seventh embodiment>

<Application to Multi-view Image Coding and Multi-view Image Decoding>

The above series of processing can be applied to multi-view image encoding and multi-view image decoding. Fig. 46 shows an example of a multi-view image encoding scheme.

As shown in FIG46 , a multi-view image includes images from multiple viewpoints (or views). The multiple views in the multi-view image include base views, each of which is encoded and decoded using its image without using images of another view, and non-base views, each of which is encoded and decoded using images of another view. Each non-base view can be encoded and decoded using an image of the base view or using images of any other non-base view.

When the multi-viewpoint images shown in FIG46 are to be encoded and decoded, the image of each view is encoded and decoded. The method described above in the previous embodiment can be applied to the encoding and decoding of each view. This can suppress the degradation of the image quality of each view.

In addition, the flags and parameters used in the methods described in the previous embodiments can be shared in encoding and decoding of each view, which can suppress a decrease in encoding efficiency.

More specifically, for example, information about the scaling list (eg, parameters, flags, etc.) may be shared in encoding and decoding of each view.

Of course, any other required information may be shared in the encoding and decoding of each view.

For example, when a scaling list or information about a scaling list included in a sequence parameter set (SPS) or a picture parameter set (PPS) is to be transmitted, if these parameter sets (SPS and PPS) are shared between views, the scaling list or information about the scaling list is also shared accordingly. This can suppress a reduction in coding efficiency.

In addition, the matrix elements in the scaling list (or quantization matrix) of the base view can be changed according to the difference value between views. In addition, offset values used to adjust the matrix elements of non-base views with respect to the matrix elements in the scaling list (quantization matrix) of the base view can be transmitted. This can suppress the increase in the amount of code.

For example, the zoom list for each view may be transmitted separately in advance. When the zoom list is to be changed for each view, only information indicating the difference from the corresponding zoom list in the previously transmitted zoom list may be transmitted. The information indicating the difference is arbitrary and may be, for example, information in units of 4×4 or 8×8 or a difference in matrices.

Note that if a zoom list or information about a zoom list is shared between views but an SPS or PPS is not shared, it may be possible to refer to the SPS or PPS of other views (that is, use the zoom list or information about the zoom list of other views).

Furthermore, if such a multi-view image is represented as an image having a YUV image and a depth image (depth) corresponding to the amount of difference between views as components, an independent scaling list or information about the scaling list of the image for each component (Y, U, V, and depth) may be used.

For example, since the depth image (depth) is an edge image, no scaling list is required. Therefore, although the SPS or PPS specifies the use of a scaling list, the scaling list may not be applied to the depth image (or a scaling list with the same (or flat) matrix elements may be applied).

<Multi-viewpoint Image Coding Device>

47 is a diagram showing a multi-view image encoding device for performing the above-described multi-view image encoding operation. As shown in FIG47 , a multi-view image encoding device 700 includes an encoding unit 701 , an encoding unit 702 , and a multiplexing unit 703 .

Coding unit 701 encodes images of a base view and generates an encoded base view image stream. Coding unit 702 encodes images of non-base views and generates an encoded non-base view image stream. Multiplexing unit 703 multiplexes the encoded base view image stream generated by coding unit 701 and the encoded non-base view image stream generated by coding unit 702 and generates an encoded multi-viewpoint image stream.

The image encoding device 10 ( FIG. 14 ) can be used for each of the encoding units 701 and 702 of the multi-view image encoding device 700 . That is, an increase in the amount of code for the scaling list in encoding each view can be suppressed, and a decrease in the image quality of each view can be suppressed. Furthermore, the encoding units 701 and 702 can perform processing (such as quantization and dequantization) using the same flags or parameters (that is, the flags and parameters can be shared). Therefore, a decrease in encoding efficiency can be suppressed.

<Multi-viewpoint image decoding device>

48 is a diagram showing a multi-view image decoding device for performing the above-described multi-view image decoding operation. As shown in FIG48 , a multi-view image decoding device 710 includes a demultiplexing unit 711 , a decoding unit 712 , and a decoding unit 713 .

The demultiplexing unit 711 demultiplexes the coded multi-view image stream, which is multiplexed with the coded base view image stream and the coded non-base view image stream, and extracts the coded base view image stream and the coded non-base view image stream. The decoding unit 712 decodes the coded base view image stream extracted by the demultiplexing unit 711 and obtains the image of the base view. The decoding unit 713 decodes the coded non-base view image stream extracted by the demultiplexing unit 711 and obtains the image of the non-base view.

The image decoding device 300 ( FIG. 22 ) can be used in each of the decoding units 712 and 713 of the multi-view image decoding device 710 . That is, an increase in the amount of code required for scaling lists in decoding each view can be suppressed, and a decrease in image quality for each view can be suppressed. Furthermore, the decoding units 712 and 713 can perform processing (such as quantization and dequantization) using the same flags and parameters (that is, the flags and parameters can be shared). Therefore, a decrease in coding efficiency can be suppressed.

<8. Eighth embodiment>

<Apply to layered image coding and layered image decoding>

The above series of processing can be applied to layered image coding and layered image decoding (scalable coding and scalable decoding). Fig. 49 shows an example of a layered image coding scheme.

Layered image coding (scalable coding) is a process of dividing an image into multiple layers (layers) to provide scalability functions for image data with respect to predetermined parameters, and encoding each layer. Layered image decoding (scalable decoding) is a decoding process corresponding to layered image coding.

As shown in FIG49 , in image layering, a single image is divided into multiple sub-images (or layers) using a predetermined parameter with a scalability function as a reference. That is, the image (or layered image) decomposed into layers includes multiple layered (or layer) images having different values for the predetermined parameter. The multiple layers in the layered image include base layers, each of which is encoded and decoded using its images without using images of another layer, and non-base layers (also called enhancement layers), each of which is encoded and decoded using images of another layer. Each non-base layer can be encoded and decoded using images of the base layer or using images of any other non-base layer.

Typically, each non-base layer includes data of a difference image (difference data) between its image and an image of another layer in order to reduce redundancy. For example, when one image is decomposed into two layers (i.e., a base layer and a non-base layer (also referred to as an enhancement layer)), an image with lower quality than the original image can be obtained using only the data of the base layer, and the original image (i.e., an image with higher quality) can be obtained by combining the data of the base layer and the data of the non-base layer.

The image layering performed in the above manner facilitates obtaining images of various qualities depending on the situation. This ensures that image compression information can be transmitted from the server according to the capabilities of the terminal and network without performing decoding. For example, only the image compression information about the base layer is transmitted to a terminal with low processing power (such as a mobile phone) to reproduce a moving image with low spatial and temporal resolution or low quality, and in addition to the image compression information about the base layer, the image compression information about the enhancement layer is also transmitted to a terminal with high processing power (such as a television set or personal computer) to reproduce a moving image with high spatial and temporal resolution or high quality.

When the layered images in the example shown in FIG49 are to be encoded and decoded, the image of each layer is encoded and decoded. The method described above in each of the previous embodiments can be applied to the encoding and decoding of each layer. This can suppress the degradation of the image quality of each layer.

In addition, the flags and parameters used in the method described in each of the previous embodiments can be shared in encoding and decoding of each layer. This can suppress a decrease in encoding efficiency.

More specifically, for example, information about the scaling list (eg, parameters, flags, etc.) may be shared in encoding and decoding of each layer.

Of course, any other required information may be shared in the encoding and decoding of each layer.

Examples of layered images include images layered according to spatial resolution (also referred to as spatial resolution scalability) (spatial scalability). In layered images with spatial resolution scalability, the resolution of the image differs for each layer. For example, the layer with the lowest spatial resolution is designated as the base layer, and the layer with a higher resolution than the base layer is designated as a non-base layer (enhancement layer).

The image data of a non-base layer (enhancement layer) can be data independent of other layers, and similar to the base layer, an image having the same resolution as that of the layer can be obtained using only the image data. However, typically, the image data of a non-base layer (enhancement layer) is data corresponding to a difference image between the image of the layer and the image of another layer (e.g., a layer one layer lower than the layer). In this case, an image having the same resolution as that of the base layer is obtained using only the image data of the base layer, while an image having the same resolution as that of the non-base layer (enhancement layer) is obtained by combining the image data of the non-base layer (enhancement layer) and the image data of another layer (e.g., a layer one layer lower than the layer). This can suppress redundancy in image data between layers.

In the above-described layered images with spatial resolution scalability, the image resolution varies for each layer. Therefore, the resolution of the processing units used for encoding and decoding each layer also varies. Therefore, if the scaling list (quantization matrix) is shared between encoding and decoding of each layer, it is possible to up-convert the scaling list (quantization matrix) according to the resolution ratio of each layer.

For example, assume that the image of the base layer has a 2K resolution (e.g., 1920×1080) and the image of the non-base layer (enhancement layer) has a 4K resolution (e.g., 3840×2160). In this case, for example, the 16×16 size of the image of the base layer (2K image) corresponds to the 32×32 size of the image of the non-base layer (4K image). The scaling list (quantization matrix) is appropriately up-converted according to the resolution ratio.

For example, a 4×4 quantization matrix used for quantization and dequantization of the base layer is up-converted to 8×8 and used in quantization and dequantization of non-base layers. Similarly, an 8×8 scaling list of the base layer is up-converted to 16×16 in non-base layers. Similarly, a quantization matrix up-converted to 16×16 and used in the base layer is up-converted to 32×32 in non-base layers.

Note that the parameters that provide scalability are not limited to spatial resolution, and examples of parameters may include temporal resolution (temporal scalability). In layered images with temporal resolution scalability, the frame rate of the image differs for each layer. Other examples include bit depth scalability, in which the bit depth of the image data differs for each layer; and chroma scalability, in which the format of the component differs for each layer.

Other examples include SNR scalability, where the signal-to-noise ratio (SNR) of the image is different for each layer.

In view of the improvement of image quality, the lower the signal-to-noise ratio of the image, the smaller the quantization error. To this end, in SNR scalability, different scaling lists (non-common scaling lists) are used for quantization and dequantization of each layer according to the signal-to-noise ratio in a desired manner. For this reason, as described above, if the scaling list is shared between layers, an offset value for adjusting the matrix elements of the enhancement layer for the matrix elements in the scaling list of the base layer may be sent. More specifically, information indicating the difference between the common scaling list and the scaling list actually used may be sent on a layer-by-layer basis. For example, information indicating the difference may be sent in a sequence parameter set (SPS) or a picture parameter set (PPS) for each layer. The information indicating the difference is arbitrary. For example, the information may be a matrix having elements representing the difference between corresponding elements in two scaling lists, or may be a function indicating the difference.

<Layered Image Coding Apparatus>

50 is a diagram showing a hierarchical image encoding apparatus for performing the above-described hierarchical image encoding operation. As shown in FIG50 , the hierarchical image encoding apparatus 720 includes an encoding unit 721, an encoding unit 722, and a multiplexing unit 723.

Coding unit 721 encodes images on the base layer and generates a coded base layer image stream. Coding unit 722 encodes images on non-base layers and generates a coded non-base layer image stream. Multiplexing unit 723 multiplexes the coded base layer image stream generated by coding unit 721 and the coded non-base layer image stream generated by coding unit 722 and generates a coded layered image stream.

The image encoding device 10 ( FIG. 14 ) can be used for each of the encoding units 721 and 722 of the layered image encoding device 720 . That is, an increase in the amount of code required for scaling lists in encoding each layer can be suppressed, and a decrease in image quality for each layer can be suppressed. Furthermore, the encoding units 721 and 722 can perform processing (such as quantization and dequantization) using the same flags or parameters (that is, the flags and parameters can be shared). Therefore, a decrease in encoding efficiency can be suppressed.

<Layered Image Decoding Device>

51 is a diagram showing a layered image decoding apparatus for performing the above-described layered image decoding operation. As shown in FIG51 , the layered image decoding apparatus 730 includes a demultiplexing unit 731 , a decoding unit 732 , and a decoding unit 733 .

Demultiplexing unit 731 demultiplexes the coded layer image stream, which is multiplexed with the coded base layer image stream and the coded non-base layer image stream, and extracts the coded base layer image stream and the coded non-base layer image stream. Decoding unit 732 decodes the coded base layer image stream extracted by demultiplexing unit 731 and obtains the base layer image. Decoding unit 733 decodes the coded non-base layer image stream extracted by demultiplexing unit 731 and obtains the non-base layer image.

The image decoding device 300 ( FIG. 22 ) can be used in each of the decoding units 732 and 733 of the layered image decoding device 730 . That is, an increase in the amount of code required for scaling lists during decoding of each layer can be suppressed, and a decrease in image quality for each layer can be suppressed. Furthermore, the decoding units 712 and 713 can perform processing (such as quantization and dequantization) using the same flags or parameters (that is, the flags and parameters can be shared). Consequently, a decrease in coding efficiency can be suppressed.

<9. Ninth embodiment>

<Computer>

The above series of processing can be executed by hardware or by software. In this case, the series of processing can be implemented as a computer as shown in Figure 52, for example.

52 , a CPU (Central Processing Unit) 801 in a computer 800 executes various processing operations according to a program stored in a ROM (Read Only Memory) 802 or a program loaded from a storage unit 813 into a RAM (Random Access Memory) 803. The RAM 803 also stores data and the like required for the CPU 801 to execute various processing operations as needed.

The CPU 801, the ROM 802, and the RAM 803 are connected to one another via a bus 804. An input/output interface 810 is also connected to the bus 804.

The input/output interface 810 is connected to an input unit 811, an output unit 812, a storage unit 813, and a communication unit 814. The input unit 811 includes a keyboard, a mouse, a touch panel, an input terminal, and the like. The output unit 812 includes desired output devices (such as speakers and displays, the displays including CRT (cathode ray tube), LCD (liquid crystal display), and OELD (organic electroluminescent display)), output terminals, and the like. The storage unit 813 includes: a desired storage medium, such as a hard disk or a flash memory; and a control unit that controls the input and output of the storage medium. The communication unit 814 includes desired wired or wireless communication devices, such as a modem, a LAN interface, a USB (universal serial bus) device, and a Bluetooth (registered trademark) device. The communication unit 814 performs communication processing with other communication devices via a network (including, for example, the Internet).

If necessary, a drive 815 is also connected to the input/output interface 810. A removable medium 821 (such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory) is placed in the drive 815 as needed. The drive 815 reads computer programs, data, and the like from the removable medium 821 placed therein according to, for example, the control of the CPU 801. The read data and computer program are provided to, for example, the RAM 803. If necessary, the computer program read from the removable medium 821 is further installed in the storage unit 813.

When the above-described series of processes is executed by software, a program constituting the software is installed from a network or a recording medium.

As shown in FIG52 , examples of recording media include removable media 821 that are distributed separately from the device body to provide the program to the user, such as a magnetic disk (including a floppy disk), an optical disk (including a CD-ROM (Compact Disc - Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory having the program recorded thereon. Other examples of recording media include devices that are distributed to the user in a manner previously included in the device body, such as a ROM 802 having the program recorded thereon and a hard disk included in the storage unit 813.

Note that the program executed by the computer 800 may be a program that performs processing operations in a time-series manner in the order described herein, or may be a program that performs processing operations in parallel or at necessary timing such as when called.

In addition, as used herein, steps describing a program stored in a recording medium naturally include processing operations performed in a time-series manner in the order described and processing operations performed in parallel or individually (and not necessarily in a time-series manner).

In addition, as used herein, the term "system" refers to a group of constituent elements (devices, modules (components), etc.), regardless of whether all constituent elements are housed in the same housing. Therefore, a plurality of devices housed in different housings and connected via a network and a single device including a plurality of modules housed in a single housing are defined as a system.

In addition, the configuration described above as a single device (or processing unit) may be divided into multiple devices (or processing units). Conversely, the configuration described above as multiple devices (or processing units) may be combined into a single device (or processing unit). In addition, of course, configurations other than the above configurations may be added to the configuration of each device (or each processing unit). In addition, if the devices (or processing units) have substantially the same configuration and/or operation with respect to the entire system, a portion of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or another processing unit). In other words, the embodiments of the present technology are not limited to the previous embodiments, and various modifications may be made without departing from the scope of the present technology.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to the examples disclosed herein. It is obvious that a person having common sense in the technical field of the present disclosure can implement various changes or modifications without departing from the scope of the technical concept defined in the claims, and it should be understood that such changes or modifications naturally also fall within the technical scope of the present disclosure.

For example, the present technology may be implemented using a cloud computing configuration in which a plurality of devices are shared via a network and cooperate to process a single function.

In addition, each step shown in the above flowchart may be performed by a single device or performed by a plurality of devices in a shared manner.

In addition, if a single step includes a plurality of processes, the plurality of processes included in the single step may be performed by a single device or performed by a plurality of devices in a shared manner.

The image encoding device 10 ( FIG. 14 ) and the image decoding device 300 ( FIG. 22 ) according to the aforementioned embodiments can be applied to various electronic devices, such as transmitters or receivers for transmitting data via satellite broadcasting, cable broadcasting (such as cable TV), or the Internet, or for transmitting data to or from a terminal via cellular communication, recording devices for recording images on media (such as optical disks, magnetic disks, and flash memory), and reproduction devices for reproducing images from such storage media. Four exemplary applications will be described below.

<10. Sample Application>

<First Example Application: Television Receiver>

53 shows an example of a schematic configuration of a television apparatus to which the foregoing embodiment is applied. A television apparatus 900 includes an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, a video signal processing unit 905, a display unit 906, an audio signal processing unit 907, a speaker 908, an external interface 909, a control unit 910, a user interface 911, and a bus 912.

The tuner 902 extracts a signal in a desired channel from a broadcast signal received via the antenna 901 and demodulates the extracted signal. The tuner 902 then outputs a coded bit stream obtained by demodulation to the demultiplexer 903. In other words, the tuner 902 functions as a transmitting unit in the television apparatus 900 for receiving a coded stream including coded images.

The demultiplexer 903 demultiplexes the encoded bit stream into a video stream and an audio stream of a program to be viewed, and outputs the demultiplexed streams to the decoder 904. The demultiplexer 903 also extracts auxiliary data such as an EPG (Electronic Program Guide) from the encoded bit stream, and supplies the extracted data to the control unit 910. Note that if the encoded bit stream has been scrambled, the demultiplexer 903 can also descramble the encoded bit stream.

The decoder 904 decodes the video stream and audio stream input from the demultiplexer 903. The decoder 904 then outputs video data generated by the decoding process to the video signal processing unit 905. The decoder 904 also outputs audio data generated by the decoding process to the audio signal processing unit 907.

The video signal processing unit 905 reproduces the video data input from the decoder 904 and displays the video on the display unit 906. The video signal processing unit 905 may also display an application screen provided via a network on the display unit 906. The video signal processing unit 905 may also perform other processing on the video data, such as noise removal, according to the settings. In addition, the video signal processing unit 905 may also generate a GUI (Graphical User Interface) image (such as a menu, button, or cursor) and superimpose the generated image on the output image.

The display unit 906 is driven by a driving signal supplied from the video signal processing unit 905 and displays a video or image on a video surface of a display device such as a liquid crystal display, a plasma display, or an OELD (organic electroluminescent display) (organic EL display).

The audio signal processing unit 907 performs reproduction processing such as D/A conversion and amplification on the audio data input from the decoder 904, and causes audio to be output from the speaker 908. The audio signal processing unit 907 may also perform other processing such as noise removal on the audio data.

The external interface 909 is an interface for connecting the television apparatus 900 to an external device or a network. For example, a video stream or an audio stream received via the external interface 909 can be decoded by the decoder 904. In other words, the external interface 909 also serves as a transmission unit in the television apparatus 900 for receiving an encoded stream including an encoded image.

The control unit 910 includes a processor (such as a CPU) and a memory (such as a RAM and a ROM). The memory stores programs to be executed by the CPU, program data, EPG data, data obtained via a network, etc. When, for example, the television device 900 is started, the program stored in the memory is read and executed by the CPU. The CPU executes the program to control the operation of the television device 900 according to an operation signal input from, for example, the user interface 911.

The user interface 911 is connected to the control unit 910. The user interface 911 includes, for example, buttons and switches for allowing the user to operate the television device 900, a receiving unit for remote control signals, etc. The user interface 911 detects the user's operation via the above components to generate an operation signal, and outputs the generated operation signal to the control unit 910.

The bus 912 is for connecting the tuner 902 , the demultiplexer 903 , the decoder 904 , the video signal processing unit 905 , the audio signal processing unit 907 , the external interface 909 , and the control unit 910 to one another.

In the television device 900 having the above configuration, the decoder 904 has the function of the image decoding apparatus 300 ( FIG. 22 ) according to the previous embodiment. Therefore, the television device 900 can suppress an increase in the encoding amount of the scaling list.

<Second Example Application: Mobile Phone>

54 shows an example of a schematic configuration of a mobile phone to which the foregoing embodiment is applied. The mobile phone 920 includes an antenna 921, a communication unit 922, an audio codec 923, a speaker 924, a microphone 925, a camera unit 926, an image processing unit 927, a multiplexing/demultiplexing unit 928, a recording/reproducing unit 929, a display unit 930, a control unit 931, an operation unit 932, and a bus 933.

The antenna 921 is connected to the communication unit 922. The speaker 924 and the microphone 925 are connected to the audio codec 923. The operation unit 932 is connected to the control unit 931. The bus 933 is used to connect the communication unit 922, the audio codec 923, the camera unit 926, the image processing unit 927, the multiplexing/demultiplexing unit 928, the recording/reproducing unit 929, the display unit 930, and the control unit 931 to each other.

The mobile phone 920 performs operations in various operation modes including a voice call mode, a data communication mode, an image capturing mode, and a video call mode, such as sending and receiving audio signals, sending and receiving emails or image data, capturing images, and recording data.

In voice call mode, the analog audio signal generated by the microphone 925 is provided to the audio codec 923. The audio codec 923 converts the analog audio signal into audio data and performs A/D conversion and compression on the converted audio data. The audio codec 923 then outputs the compressed audio data to the communication unit 922. The communication unit 922 encodes and modulates the audio data and generates a transmission signal. The communication unit 922 then transmits the generated transmission signal to a base station (not shown) via the antenna 921. In addition, the communication unit 922 amplifies the radio signal received via the antenna 921 and performs frequency conversion on the amplified signal to obtain a received signal. The communication unit 922 then demodulates and decodes the received signal to generate audio data and outputs the generated audio data to the audio codec 923. The audio codec 923 scales the audio data and performs D/A conversion to generate an analog audio signal. The audio codec 923 then provides the generated audio signal to the speaker 924 so that audio is output.

In data communication mode, for example, the control unit 931 generates text data forming an email based on a user operation via the operation unit 932. Furthermore, the control unit 931 causes the text to be displayed on the display unit 930. Furthermore, the control unit 931 generates email data based on a transmission instruction given by the user via the operation unit 932, and outputs the generated email data to the communication unit 922. The communication unit 922 encodes and modulates the email data to generate a transmission signal. The communication unit 922 then transmits the generated transmission signal to a base station (not shown) via the antenna 921. Furthermore, the communication unit 922 amplifies the radio signal received via the antenna 921 and performs frequency conversion on the amplified signal to obtain a received signal. The communication unit 922 then demodulates and decodes the received signal to recover the email data, and outputs the recovered email data to the control unit 931. The control unit 931 causes the content of the email to be displayed on the display unit 930, and also causes the email data to be stored in the storage medium of the recording/reproducing unit 929.

The recording/reproducing unit 929 includes a desired readable/writable storage medium. The storage medium may be, for example, a built-in storage medium such as RAM or flash memory or an external storage medium such as a hard disk, a magnetic disk, a magneto-optical disk, an optical disk, a USB memory, or a memory card.

In addition, in the image capturing mode, for example, the camera unit 926 captures an image of an object to generate image data, and outputs the generated image data to the image processing unit 927. The image processing unit 927 encodes the image data input from the capturing unit 926 and causes the encoded stream to be stored in the storage medium of the recording/reproducing unit 929.

In addition, in videophone mode, for example, the multiplexing/demultiplexing unit 928 multiplexes the video stream encoded by the image processing unit 927 and the audio stream input from the audio codec 923, and outputs the multiplexed stream to the communication unit 922. The communication unit 922 encodes and modulates the multiplexed stream to generate a transmission signal. The communication unit 922 then transmits the generated transmission signal to a base station (not shown) via the antenna 921. The communication unit 922 also amplifies the radio signal received via the antenna 921 and performs frequency conversion on the amplified signal to obtain a received signal. The transmission signal and the received signal may include an encoded bit stream. The communication unit 922 demodulates and decodes the received signal to restore the stream and outputs the restored stream to the multiplexing/demultiplexing unit 928. The multiplexing/demultiplexing unit 928 then demultiplexes the input stream into a video stream and an audio stream, and outputs the video stream and the audio stream to the image processing unit 927 and the audio codec 923, respectively. The image processing unit 927 decodes the video stream to generate video data. The video data is provided to the display unit 930, and a series of images are displayed by the display unit 930. The audio codec 923 expands the audio stream and performs D/A conversion to generate an analog audio signal. The audio codec 923 then provides the generated audio signal to the speaker 924 so that the audio is output.

In the mobile phone 920 having the above configuration, the image processing unit 927 has the functions of the image encoding device 10 ( FIG. 14 ) and the image decoding device 300 ( FIG. 22 ) according to the previous embodiment. Therefore, the mobile phone 920 can suppress an increase in the amount of code for the scaling list.

In addition, although a description has been given of, for example, a mobile phone 920, similar to the mobile phone 920, the image encoding device and the image decoding device to which the present technology is applied can be used for any device having imaging functions and communication functions similar to those of the mobile phone 920, such as a PDA (personal digital assistant), a smart phone, a UMPC (ultra mobile personal computer), a netbook, or a notebook personal computer.

<Third Example Application: Recording/Reproducing Device>

FIG55 shows an example of a schematic configuration of a recording/reproducing device to which the preceding embodiments are applied. The recording/reproducing device 940 encodes, for example, audio data and video data of a received broadcast program and records the encoded audio data and video data on a recording medium. Alternatively, the recording/reproducing device 940 can also encode audio data and video data obtained from, for example, another device and record the encoded audio data and video data on a recording medium. Furthermore, the recording/reproducing device 940 reproduces, for example, data recorded on a recording medium using a monitor and speakers in accordance with user instructions. In this case, the recording/reproducing device 940 decodes the audio data and video data.

The recording/reproducing device 940 includes a tuner 941 , an external interface 942 , an encoder 943 , an HDD (Hard Disk Drive) 944 , a disk drive 945 , a selector 946 , a decoder 947 , an OSD (On Screen Display) 948 , a control unit 949 , and a user interface 950 .

The tuner 941 extracts a signal in a desired channel from a broadcast signal received via an antenna (not shown) and demodulates the extracted signal. The tuner 941 then outputs a coded bit stream obtained by demodulation to the selector 946. In other words, the tuner 941 functions as a transmitting unit in the recording/reproducing device 940.

The external interface 942 is an interface for connecting the recording/reproducing apparatus 940 to an external device or a network. The external interface 942 may be, for example, an IEEE 1394 interface, a network interface, a USB interface, a flash memory interface, or the like. For example, video data and audio data received via the external interface 942 are input to the encoder 943. In other words, the external interface 942 functions as a transmission unit in the recording/reproducing apparatus 940.

If the video data and the audio data input from the external interface 942 are not encoded, the encoder 943 encodes the video data and the audio data input from the external interface 942. The encoder 943 then outputs the encoded bit stream to the selector 946.

The HDD 944 records a coded bit stream including compressed content data such as video and audio, various programs, and other data on an internal hard disk. In addition, when reproducing video and audio, the HDD 944 reads the above data from the hard disk.

The disk drive 945 records data on a recording medium placed therein and reads data from the recording medium placed therein. The recording medium placed in the disk drive 945 may be, for example, a DVD disk (such as DVD-Video, DVD-RAM, DVD-R, DVD-RW, DVD+R, or DVD+RW) or a Blu-ray (registered trademark) disk.

When recording video and audio, the selector 946 selects an encoded bit stream input from the tuner 941 or the encoder 943 and outputs the selected encoded bit stream to the HDD 944 or the disk drive 945. When reproducing video and audio, the selector 946 outputs the encoded bit stream input from the HDD 944 or the disk drive 945 to the decoder 947.

The decoder 947 decodes the encoded bit stream to generate video data and audio data. The decoder 947 then outputs the generated video data to the OSD 948. The decoder 904 also outputs the generated audio data to an external speaker.

The OSD 948 reproduces the video data input from the decoder 947 and displays the video. In addition, the OSD 948 may also superimpose a GUI image such as a menu, a button, or a cursor on the video to be displayed.

The control unit 949 includes a processor (such as a CPU) and a memory (such as a RAM and a ROM). The memory stores programs to be executed by the CPU, program data, etc. When the recording/reproducing device 940 is started, for example, the program stored in the memory is read and executed by the CPU. The CPU executes the program based on an operation signal input from the user interface 950, for example, to control the operation of the recording/reproducing device 940.

The user interface 950 is connected to the control unit 949. The user interface 950 includes, for example, buttons and switches for allowing the user to operate the recording/reproducing device 940, a receiving unit for remote control signals, etc. The user interface 950 detects the user's operation via the above components to generate an operation signal, and outputs the generated operation signal to the control unit 949.

In the recording/reproducing device 940 having the above configuration, the encoder 943 has the functions of the image encoding device 10 ( FIG. 14 ) according to the previous embodiment. Furthermore, the decoder 947 has the functions of the image decoding device 300 ( FIG. 22 ) according to the previous embodiment. Therefore, the recording/reproducing device 940 can suppress an increase in the amount of code for the scaling list.

<Fourth Example Application: Imaging Device>

Fig. 56 shows an example of a schematic configuration of an imaging device to which the foregoing embodiment is applied. The imaging device 960 captures an image of an object to generate image data, encodes the image data, and records the encoded image data on a recording medium.

The imaging device 960 includes an optical block 961 , an imaging unit 962 , a signal processing unit 963 , an image processing unit 964 , a display unit 965 , an external interface 966 , a memory 967 , a media drive 968 , an OSD 969 , a control unit 970 , a user interface 971 , and a bus 972 .

The optical block 961 is connected to the imaging unit 962. The imaging unit 962 is connected to the signal processing unit 963. The display unit 965 is connected to the image processing unit 964. The user interface 971 is connected to the control unit 970. The bus 972 is used to connect the image processing unit 964, the external interface 966, the memory 967, the media drive 968, the OSD 969, and the control unit 970 to each other.

The optical block 961 includes a focusing lens, an aperture mechanism, and the like. The optical block 961 forms an optical image of an object on an imaging surface of the imaging unit 962. The imaging unit 962 includes an image sensor (such as a CCD or CMOS image sensor) and converts the optical image formed on the imaging surface into an image signal serving as an electrical signal by performing photoelectric conversion. The imaging unit 962 then outputs the image signal to the signal processing unit 963.

The signal processing unit 963 performs various camera signal processing operations such as knee correction, gamma correction, and color correction on the image signal input from the imaging unit 962. The signal processing unit 963 outputs the image data subjected to the camera signal processing operation to the image processing unit 964.

The image processing unit 964 encodes the image data input from the signal processing unit 963 to generate encoded data. The image processing unit 964 then outputs the generated encoded data to the external interface 966 or the media drive 968. In addition, the image processing unit 964 decodes the encoded data input from the external interface 966 or the media drive 968 to generate image data. The image processing unit 964 then outputs the generated image data to the display unit 965. In addition, the image processing unit 964 can also output the image data input from the signal processing unit 963 to the display unit 965 so that an image is displayed. In addition, the image processing unit 964 can also superimpose display data obtained from the OSD 969 on the image to be output to the display unit 965.

The OSD 969 generates a GUI image such as a menu, a button, or a cursor, and outputs the generated image to the image processing unit 964 .

The external interface 966 is formed as, for example, a USB input/output terminal. When printing an image, the external interface 966 connects the imaging device 960 to a printer, for example. If necessary, a drive is also connected to the external interface 966. A removable medium (such as a magnetic disk or an optical disk) is placed in the drive, and a program read from the removable medium can be installed in the imaging device 960. In addition, the external interface 966 can also be formed as a network interface to connect to a network (such as a LAN or the Internet). In other words, the external interface 966 serves as a sending unit in the imaging device 960.

The recording medium to be placed in the media drive 968 may be, for example, any readable/writable removable medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory. Alternatively, the recording medium may be fixedly connected to the media drive 968 and may be formed as a built-in hard disk drive or a non-portable storage unit such as an SSD (Solid State Drive).

The control unit 970 includes a processor (such as a CPU) and a memory (such as a RAM and a ROM). The memory stores programs to be executed by the CPU, program data, etc. When the imaging device 960 is turned on, for example, the program stored in the memory is read and executed by the CPU. The CPU executes the program based on an operation signal input from the user interface 971, for example, to control the operation of the imaging device 960.

The user interface 971 is connected to the control unit 970. The user interface 971 includes, for example, buttons, switches, etc. for allowing the user to operate the imaging device 960. The user interface 971 detects user operations via the above components to generate operation signals and outputs the generated operation signals to the control unit 970.

In the imaging device 960 having the above configuration, the image processing unit 964 has the functions of the image encoding device 10 ( FIG. 14 ) and the image decoding device 300 ( FIG. 22 ) according to the previous embodiment. Therefore, the imaging device 960 can suppress an increase in the encoding amount of the scaling list.

<7. Example Application of Scalable Coding>

<First System>

Next, a specific example of the use of scalable coded data that has been coded using scalable coding (layered (image) coding) will be described. Scalable coding can be used for selection of data to be transmitted, for example, as shown in the example shown in FIG.

In the data sending system 1000 shown in Figure 57, the distribution server 1002 reads the scalable coded data stored in the scalable coded data storage unit 1001, and distributes the scalable coded data to terminal devices (such as a personal computer 1004, an AV device 1005, a tablet device 1006, and a mobile phone 1007) via a network 1003.

In this case, the distribution server 1002 selects coded data with desired quality based on the performance of the terminal device, the communication environment, etc., and sends the selected coded data. Even if the distribution server 1002 sends data with a quality higher than the required level, the terminal device may not always obtain a high-quality image, and may cause a delay or overflow. In addition, such data may occupy a communication bandwidth exceeding the required level, or may increase the load on the terminal device exceeding the required level. Conversely, even if the distribution server 1002 sends data with a quality lower than the required level, the terminal device may not necessarily obtain an image with sufficient quality. Therefore, the distribution server 1002 reads the scalable coded data stored in the scalable coded data storage unit 1001 as coded data with a quality suitable for the performance of the terminal device, the communication environment, etc., as needed, and sends the read coded data.

For example, it is assumed that the scalable coded data storage unit 1001 stores scalable coded data (BL+EL) 1011 on which scalable coding has been performed. The scalable coded data (BL+EL) 1011 is coded data including a base layer and an enhancement layer, and is decoded to obtain an image of the base layer and an image of the enhancement layer.

The distribution server 1002 selects an appropriate layer based on the performance of the terminal device transmitting the data, the communication environment, etc., and reads the data of the layer. For example, the distribution server 1002 reads high-quality scalable coded data (BL+EL) 1011 from the scalable coded data storage unit 1001 and transmits the read scalable coded data (BL+EL) 1011 as is to the personal computer 1004 or tablet device 1006 having high processing power. In contrast, for example, the distribution server 1002 extracts the data of the base layer from the scalable coded data (BL+EL) 1011 and transmits the extracted base layer data to the AV device 1005 and mobile phone 1007 having low processing power as scalable coded data (BL) 1012 having the same content as the scalable coded data (BL+EL) 1011 but lower quality than the scalable coded data (BL+EL) 1011.

Using scalable coded data in this manner facilitates adjustment of the data volume, thereby suppressing the occurrence of delays or overflows and suppressing unnecessary increases in the load on terminal devices or communication media. Furthermore, the scalable coded data (BL+EL) 1011 has reduced redundancy between layers and therefore has a smaller amount of data than data having data of each layer coded separately. Therefore, the storage area of the scalable coded data storage unit 1001 can be used more efficiently.

Note that, since various devices (such as personal computer 1004, AV device 1005, tablet device 1006 and mobile phone 1007) can be used as terminal devices, the hardware performance of the terminal device is different for each device. In addition, since various applications can be executed by the terminal device, the software capabilities of the application can be different. In addition, the network 1003 used as a communication medium can be implemented as any communication line network (the communication line network can be a wired communication line network, a wireless communication line network or the two, such as the Internet and a LAN (local area network)), and has various data transmission capabilities. This performance and ability can be different according to other communications, etc.

Therefore, before data transmission begins, the distribution server 1002 can communicate with the terminal device to which the data is to be transmitted, and can obtain information about the capabilities of the terminal device (such as the hardware performance of the terminal device or the performance of the application (software) executed by the terminal device) and information about the communication environment (such as the available bandwidth of the network 1003). In addition, the distribution server 1002 can select an appropriate layer based on the obtained information.

Note that the layer can be extracted by the terminal device. For example, the personal computer 1004 can decode the transmitted scalable coded data (BL+EL) 1011 and display the image of the base layer or the image of the enhancement layer. Alternatively, for example, the personal computer 1004 can extract the scalable coded data (BL) 1012 of the base layer from the transmitted scalable coded data (BL+EL) 1011, store the extracted scalable coded data (BL) 1012, transmit the extracted scalable coded data (BL) 1012 to another device, or decode the extracted scalable coded data (BL) 1012 to display the image of the base layer.

Of course, the number of scalable coded data storage units 1001, the number of distribution servers 1002, the number of networks 1003, and the number of terminal devices are arbitrary. Furthermore, while the description has been given of an example in which the distribution server 1002 transmits data to a terminal device, the example used is not limited to this example. The data transmission system 1000 can be used in any system that selects an appropriate layer based on the capabilities of the terminal device, the communication environment, etc. when transmitting coded data encoded using scalable coding to the terminal device.

In addition, in a manner similar to the application to hierarchical encoding and hierarchical decoding described above with reference to Figures 49 to 51, the present technology can also be applied to the data sending system 1000 shown in the above-mentioned Figure 57, thereby achieving advantages similar to those described above with reference to Figures 49 to 51.

<Second System>

For example, as shown in the example shown in FIG. 58 , scalable coding may also be used for transmission via multiple communication media.

In the data transmission system 1100 shown in FIG58 , a broadcasting station 1101 transmits scalable coded data (BL) 1121 of a base layer via a terrestrial broadcast 1111. The broadcasting station 1101 also transmits (e.g., packages and transmits) scalable coded data (EL) 1122 of an enhancement layer via a desired network 1112 (the network 1112 is formed of a communication network, which may be a wired communication network, a wireless communication network, or both).

The terminal device 1102 has a function of receiving a terrestrial broadcast 1111 from the broadcasting station 1101, and receives scalable coded data (BL) 1121 of a base layer via the terrestrial broadcast 1111. The terminal device 1102 also has a communication function of performing communication via a network 1112, and receives scalable coded data (EL) 1122 of an enhancement layer transmitted via the network 1112.

The terminal device 1102 decodes the scalable coded data (BL) 1121 of the base layer obtained via the terrestrial broadcast 1111 according to, for example, user instructions, to obtain an image of the base layer, stores the scalable coded data (BL) 1121, or transmits the scalable coded data (BL) 1121 to another device.

In addition, the terminal device 1102 combines the scalable coded data (BL) 1121 of the base layer obtained via the terrestrial broadcast 1111 with the scalable coded data (EL) 1122 of the enhancement layer obtained via the network 1112 according to, for example, a user instruction to obtain scalable coded data (BL+EL), decodes the scalable coded data (BL+EL) to obtain an image of the enhancement layer, stores the scalable coded data (BL+EL), or transmits the scalable coded data (BL+EL) to another device.

As described above, scalable coded data can be transmitted via, for example, a different communication medium for each layer. Therefore, the load can be distributed and delay or overflow can be prevented from occurring.

In addition, the communication medium to be used for transmission can be selected for each layer according to the situation. For example, the scalable coded data (BL) 1121 of the base layer having a relatively large amount of data can be transmitted via a communication medium having a large bandwidth, and the scalable coded data (EL) 1122 of the enhancement layer having a relatively small amount of data can be transmitted via a communication medium having a narrow bandwidth. Alternatively, for example, the communication medium for transmitting the scalable coded data (EL) 1122 of the enhancement layer can be switched between the network 1112 and the terrestrial broadcast 1111 according to the available bandwidth of the network 1112. Naturally, the above situation is similarly applied to the data of any layer.

Control performed in the above-described manner can further suppress an increase in the load of data transmission.

Of course, the number of layers is arbitrary, and the number of communication media used for transmission is also arbitrary. Furthermore, the number of terminal devices 1102 to which data is to be distributed is also arbitrary. Furthermore, while the description has been given using the example of broadcasting from broadcast station 1101, the example used is not limited to this. The data transmission system 1100 can be used in any system that divides data encoded using scalable coding into multiple segments in layers and transmits the data segments over multiple lines.

In addition, in a manner similar to the application to hierarchical encoding and hierarchical decoding described above with reference to Figures 49 to 51, the present technology can also be applied to the data sending system 1100 shown in the above-mentioned Figure 58, thereby achieving advantages similar to those described above with reference to Figures 49 to 51.

<Third System>

For example, as shown in the example shown in FIG. 59 , scalable coding may also be used for storage of encoded data.

In the imaging system 1200 shown in Figure 59, the imaging device 1201 performs scalable coding on image data obtained by capturing an image of an object 1211, and provides the obtained data to the scalable coded data storage device 1202 as scalable coded data (BL+EL) 1221.

The scalable coded data storage device 1202 stores the scalable coded data (BL+EL) 1221 supplied from the imaging device 1201 at a quality appropriate to the situation. For example, during normal time, the scalable coded data storage device 1202 extracts the data of the base layer from the scalable coded data (BL+EL) 1221 and stores the extracted data of the base layer as scalable coded data (BL) 1222 of the base layer having low quality and a small amount of data. In contrast, during a time of interest, for example, the scalable coded data storage device 1202 stores the scalable coded data (BL+EL) 1221 having high quality and a large amount of data as is.

Therefore, the scalable coded data storage device 1202 can store images with high quality only when necessary. This can suppress the increase in data volume while suppressing the decrease in the value of the image due to the decrease in quality, and can improve the use efficiency of the storage area.

For example, assume that imaging device 1201 is a security camera. If the object to be monitored (e.g., an intruder) does not appear in the captured image (normal time), the captured image may not have important content. Therefore, priority is given to reducing the amount of data, and the image data (scalable coded data) of the image is stored at low quality. In contrast, if the object to be monitored appears as object 1211 in the captured image (attention time), the captured image may have important content. Therefore, priority is given to image quality, and the image data (scalable coded data) of the image is stored at high quality.

Note that the normal time or the attention time may be determined by analyzing an image, for example, by the scalable coded data storage device 1202. Alternatively, the imaging device 1201 may determine the normal time or the attention time and may send the determination result to the scalable coded data storage device 1202.

Note that the determination of the usual time or the attention time can be based on any criteria, and the image on which this determination is based can have any content. Of course, conditions other than the content of the image can be used as the determination criteria. The state can change based on, for example, the volume or waveform of the recorded audio, or can change at intervals of a predetermined time period. Alternatively, the state can change based on external instructions (such as user instructions).

In addition, although the description has been given of an example of changing between two states (i.e., normal time and attention time), the number of states is arbitrary, and state changes may be performed between more than two states (such as normal time, attention time, more attention time, and serious attention time). Note that the upper limit number of states to be changed depends on the number of layers of scalable coded data.

Furthermore, the imaging device 1201 may be configured to determine the number of scalable coded layers according to the state. For example, during normal time, the imaging device 1201 may generate scalable coded data (BL) 1222 of a base layer having low quality and a small amount of data, and provide the generated scalable coded data (BL) 1222 to the scalable coded data storage device 1202. Furthermore, for example, during the time of interest, the imaging device 1201 may generate scalable coded data (BL+EL) 1221 of a base layer having high quality and a large amount of data, and provide the generated scalable coded data (BL+EL) 1221 to the scalable coded data storage device 1202.

Although a security camera has been described as an example, the imaging system 1200 may be used in any application and may be used in applications other than security cameras.

In addition, in a manner similar to the application to hierarchical encoding and hierarchical decoding described above with reference to Figures 49 to 51, the present technology can also be applied to the imaging system 1200 shown in the above-mentioned Figure 59, thereby achieving advantages similar to those described above with reference to Figures 49 to 51.

Note that this technology can also be applied to HTTP streaming, such as MPEG DASH, where appropriate encoded data is selected from multiple pieces of encoded data prepared in advance and having different resolutions, and the selected appropriate encoded data is used in units of segments. In other words, information about encoding and decoding can also be shared between multiple pieces of encoded data.

It goes without saying that the image encoding device and the image decoding device to which the present technology is applied can also be applied to devices or systems other than the above-mentioned devices.

Note that an example has been described here in which the quantization matrix (or the coefficients used to form the quantization matrix) is transmitted from the encoding side to the decoding side. A technique for transmitting the quantization matrix can be to transmit or record the quantization matrix as separate data associated with the encoded bitstream, rather than multiplexing the quantization parameters into the encoded bitstream. As used herein, the term "associated" means that when an image (which may be a portion of an image, such as a slice or block) included in the bitstream is decoded, the image can be linked to information corresponding to the image. In other words, the information can be transmitted over a transmission path different from that used to transmit the image (or bitstream). Alternatively, the information can be recorded on a recording medium different from that used to record the image (or bitstream) (or in a different recording area on the same recording medium). Furthermore, the information and the image (or bitstream) can be associated with each other in arbitrary units, such as multiple frames, a single frame, or a portion of a frame.

Label list

10 image encoding device, 14 orthogonal transform/quantization unit, 16 lossless encoding unit, 150 matrix processing unit, 192 DPCM unit, 211 DC coefficient encoding unit, 212 AC coefficient DPCM unit, 300 image decoding device, 312 lossless decoding unit, 313 dequantization/inverse orthogonal transform unit, 410 matrix generating unit, 552 inverse DPCM unit, 571 initial setting unit, 572 DPCM decoding unit, 573 DC coefficient extraction unit, 611 AC coefficient buffer, 612 AC coefficient encoding unit, 613 AC coefficient DPCM unit, 614 DC coefficient DPCM unit, 621 initial setting unit, 622 AC coefficient DPCM decoding unit, 623 AC coefficient buffer, 624 DC coefficient DPCM decoding unit, 631 AC coefficient DPCM unit, 632 DC coefficient buffer, 633 DC coefficient DPCM unit, 641 initial setting unit, 642 AC coefficient DPCM decoding unit, 643 DC coefficient DPCM decoding unit.

Claims (19)

1. An image processing apparatus, comprising: The setting unit sets a replacement difference coefficient, which is the difference between the replacement coefficient and the coefficient at the beginning of the first quantization matrix of the first size. The replacement coefficient is used when replacing the coefficient at the beginning of the second quantization matrix of the second size obtained by upconverting the first quantization matrix of the first size. A quantization unit quantizes transform coefficient data to generate quantized data, the transform coefficient data being obtained by performing an orthogonal transform on the image; and The encoding unit generates encoded data by encoding the quantized data generated by the quantization unit, the encoded data including the substitution difference coefficients set by the setting unit. 2. The image processing apparatus as claimed in claim 1, wherein The setting unit sets the difference coefficients, which are the differences between the coefficients of the first quantization matrix of the first size, respectively. The encoding unit generates the encoded data, which includes the difference coefficients set by the setting unit. 3. The image processing apparatus as claimed in claim 2, wherein The encoding unit generates the encoded data of the syntax, which is a group of difference coefficients that combines the substitution difference coefficients and the difference coefficients. 4. The image processing apparatus as claimed in claim 3, wherein The encoding unit encodes the data according to the order of the substitution difference coefficients and the difference coefficients, generating the encoded data with the difference coefficient group as the syntax. 5. The image processing apparatus as claimed in claim 4, wherein The setting unit sets an initial difference coefficient as the difference between the replacement coefficient and the initial value of the coefficient set for the quantization matrix. The encoding unit generates the encoded data including the initial difference coefficients set by the setting unit. 6. The image processing apparatus of claim 5, wherein The encoding unit encodes the data in the order of the initial difference coefficients and the difference coefficient group to generate the encoded data with the initial difference coefficients and the difference coefficient group as the syntax. 7. The image processing apparatus of claim 6, wherein The encoding unit performs exponential Golomb encoding according to the order of the initial difference coefficients and the difference coefficient group, generating the encoded data with the exponential Golomb encoding of the initial difference coefficients and the exponential Golomb encoding of the difference coefficient group as the syntax. 8. The image processing apparatus of claim 7, wherein The image processing device further includes an orthogonal transformation unit, which performs an orthogonal transformation on the image to generate the transformation coefficient data. The quantization unit quantizes the transformation coefficient data generated by the orthogonal transformation unit. 9. The image processing apparatus of claim 8, wherein The orthogonal transformation unit generates the transformation coefficient data by performing orthogonal transformations using a transformation unit of the second size. The quantization unit uses a second quantization matrix of the second size to quantize the transformation coefficient data generated by the orthogonal transformation unit. 10. An image processing method, wherein, A substitution difference coefficient is set, which is the difference between the substitution coefficient and the coefficient at the beginning of the first quantization matrix of the first size. The substitution coefficient is used when replacing the coefficient at the beginning of the second quantization matrix of the second size obtained by up-converting the first quantization matrix of the first size. The transform coefficient data is quantized to generate quantized data, which is obtained by performing an orthogonal transform on the image; Encoded data is generated by encoding the generated quantized data, and the encoded data includes the set substitution difference coefficient. 11. The image processing method as described in claim 10, wherein... Set the difference coefficients, which are the differences between the coefficients of the first quantization matrix of the first size, to represent the differences between each other. Generate the encoded data, which includes the difference coefficients set. 12. The image processing method as described in claim 11, wherein... Generate the coded data of the syntax by combining the substitution difference coefficients and the difference coefficients into a group of difference coefficients. 13. The image processing method as described in claim 12, wherein... The encoding is performed according to the order of the replacement difference coefficients and the difference coefficients to generate the encoded data with the difference coefficient group as the syntax. 14. The image processing method as described in claim 13, wherein... Set an initial difference coefficient as the difference between the replacement coefficient and the initial values of the coefficients set for the quantization matrix. Generate the encoded data, including the initial difference coefficients set. 15. The image processing method as described in claim 14, wherein... Encode the data in the order of the initial difference coefficients and the difference coefficient group to generate the encoded data with the initial difference coefficients and the difference coefficient group as the syntax. 16. The image processing method as described in claim 15, wherein... Perform exponential Golomb encoding according to the order of the initial difference coefficients and the difference coefficient group to generate the encoded data with the exponential Golomb encoding of the initial difference coefficients and the exponential Golomb encoding of the difference coefficient group as the syntax. 17. The image processing method as described in claim 16, wherein... The image is subjected to an orthogonal transformation to generate the transformation coefficient data. The generated transformation coefficient data is then quantized. 18. The image processing method as described in claim 17, wherein... The transformation coefficient data is generated by performing an orthogonal transformation using a second-sized transformation unit. The generated transform coefficient data is quantized using a second quantization matrix of the second size. 19. A computer-readable recording medium having a program recorded thereon that enables a computer to function as a unit: The setting unit sets a replacement difference coefficient, which is the difference between the replacement coefficient and the coefficient at the beginning of the first quantization matrix of the first size. The replacement coefficient is used when replacing the coefficient at the beginning of the second quantization matrix of the second size obtained by upconverting the first quantization matrix of the first size. A quantization unit quantizes transform coefficient data to generate quantized data, the transform coefficient data being obtained by performing an orthogonal transform on the image; and The encoding unit generates encoded data by encoding the quantized data generated by the quantization unit, the encoded data including the substitution difference coefficients set by the setting unit.