JP2000350204A - Hierarchical encoding method and image encoding / decoding device - Google Patents

Abstract

(57) [Summary] In a conventional data partitioning method of MPEG2, in order to identify a division point of a base layer,
The value specified in the priority breakpoint was used. Therefore, the decryption device
If the right breakpoint cannot be identified, the data-partitioned stream cannot be reproduced. A DCT coefficient sequence is divided for each macroblock into a base layer that is a low-frequency component and an enhancement layer that is a high-frequency component, and an EOB code is inserted as a division point after the low-frequency component of each DCT coefficient of the base layer. I do. [Effect] Even if the MPEG2 decoding apparatus does not support data partitioning, decoding can be performed only with the base layer.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION In coding using DCT coefficients such as MPEG, hierarchical coding by a data partition that divides the DCT coefficients into a base layer composed of low-frequency components and an enhanced layer composed of high-frequency components, and in particular, hierarchical coding TECHNICAL FIELD The present invention relates to a hierarchical coding method that can decode even a decoding device that does not support encoding using only a base layer.

[0002] Further, the present invention relates to an image encoding device and an image decoding device capable of selecting the reproduction quality of an image according to charging and access right by using hierarchical encoding.

[0003]

2. Description of the Related Art Conventionally, ITU-T Recommendation H.264 has been adopted. 262 (M
In PEG2), as shown in FIG. 3, the DC is changed according to the branch point specified in the priority breakpoint included in the elementary stream of MPEG2.
DCT coefficient pairs of the T coefficient block (run, level
A method for dividing l) into two partitions is defined.

[0004] Further, when selling conventionally digitized image data, two types of sample image having inferior image quality and encrypted formal image data are recorded in the same CDROM. The purchaser determines the purchase of the image data by looking at the sample image. If the user wants to purchase the image data, he obtains the desired image data by purchasing an encryption key for decrypting the encrypted image data.

[0005]

In the above prior art, in dividing a continuous DCT coefficient pair up to the EOB in block units, the DCT coefficient pair is simply divided at a designated division point, so that there is no EOB at the division point. Priority break specified for each slice to identify the division point
Point information is required, and priority b
A decoder that does not perform hierarchical decoding by recognizing a breakpoint has a problem that hierarchical decoding cannot be performed.

[0006] In the above-mentioned other prior art, two image data, that is, a sample image having inferior image quality and encrypted image data are converted into C image data.
Since the data is recorded on the same medium such as a DROM, there is a problem that the data actually used can be stored in only one half of a recording medium.

An object of the present invention is to provide a priority br
An object of the present invention is to reproduce an image using only information of a base layer without performing special processing in an image decoder that does not recognize an eakpoint.

Another object of the present invention is to enable the use of low-quality image data and high-quality image data without increasing the amount of image data, and to realize the use of image data according to the authority of the user. It is in.

[0009]

According to the DCT coding employed in MPEG, an image is decomposed into blocks of 8 × 8 pixels, and each block is subjected to a transform process (DCT) into a frequency spectrum. The converted two-dimensional signal composed of the 8 × 8 DCT transform coefficients is converted into a one-dimensional signal by zigzag scanning, and the data amount is reduced by Huffman coding. An EOB is inserted at the end of the one-dimensional signal composed of one block. In Huffman coding, since most of the coefficient values obtained by DCT become 0, the number of non-zero AC components (Level) and the number of zero-value AC components (R
un) is a variable length coding represented by a coefficient pair.

[0010] In order to achieve the above object, in a hierarchical coding for dividing a one-dimensional signal subjected to Huffman coding into a base layer composed of low-frequency components and an enhancement layer composed of high-frequency components, division is performed. By newly inserting the EOB into the divided point of the base layer, the base layer has the same structure as the one-dimensional signal before division, and an image can be reproduced even if only the base layer is input to the image decoding device.

For example, in the case of a transmitting device and a receiving device connected by a network via a router using an MPEG2 elementary stream, the transmitting device encodes an image and converts it into an elementary stream, By encoding, the enhancement layer is inserted using a user-defined field in the elementary stream, and transmitted as one stream to the network.
An intermediate router controls whether or not the user-defined field is discarded. If the user-defined field is detected, the receiving side performs hierarchical decoding processing and then reproduces an image by the image decoding unit. If the user-defined field has been discarded, the received stream can be directly input to the image decoding means to reproduce the image.

[0012] In order to achieve the above object, by encrypting hierarchically coded enhancement layer data, base layer data is used if there is no encryption key for decrypting the encrypted enhancement layer. Thus, a low-quality image can be reproduced.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing a first embodiment of the present invention, FIG. 2 is a flowchart showing the operation of the first embodiment of the present invention, and FIG. FIG. 4 is a diagram illustrating a partitioning method, and FIG. 4 is a diagram illustrating a data partitioning method according to the present invention.
Is a stream structure of MPEG2, and FIG.
FIG. 7 shows an example of a stream structure according to the present invention, and FIG. 8 shows an example of a stream structure according to the present invention.

In this embodiment, an example in which hierarchical encoding is performed on an MPEG2 elementary stream will be described.

The data partitioning method described in the specification as MPEG2 uses Pr Pr as shown in FIG.
The picture data is divided into the base data and the enhanced data at the location specified in the ioority breakpoint (1 in this example). In the present invention, as shown in FIG. 4, EO is added to the base data after data partitioning.
B is additionally inserted.

FIG. 5 shows the structure of an elementary stream of MPEG2. FIG. 6 shows an example of the elementary stream. One stream is from the sequence header to the end of the sequence. The stream can be repeated. In the stream, frame data in which a picture header, picture coding, and picture data are grouped repeatedly exists. User data can be omitted and can be used freely.

In the embodiment according to the present invention, as shown in FIG. 7, after data partitioning, enhanced data is placed as user data before picture data.
As a result, the correspondence between the picture data and the enhanced data can be established. As shown in FIG. 8, the enhanced data is stored as one of the user data following the user data start code and the enhanced data identification code. In a conventional MPEG2 decoder in which user data is ignored, enhanced data existing as user data is ignored, and decoding is performed using only base data.

In FIG. 1, reference numeral 10 denotes an MPEG2 elementary stream, 11 denotes picture extraction means for extracting picture data from the elementary stream, 12 denotes data other than the picture data divided by the picture extraction means 11, and 13 denotes picture extraction. The picture data extracted by the means 11 is 14
DCT coefficient extracting means for extracting a CT coefficient;
Data other than the DCT coefficient extracted by the coefficient extracting means, 16 is the DCT coefficient extracted by the DCT coefficient extracting means 14.
T coefficient data, 17 denotes DCT dividing means for dividing the DCT coefficient data 16, 18 denotes base DCT coefficient data of low frequency components divided by the DCT dividing means 17, 19 denotes DC
Enhanced DCT coefficient data of high-frequency components divided by the T dividing means 17, temporary storage means 20 for temporarily storing input data, 21 EOB inserting means for inserting an EOB code into base DCT coefficient data 18, Reference numeral 22 denotes a hierarchically encoded output for outputting a hierarchically encoded stream; 23, an EOB code; and 29, a selection unit for switching input data.

Next, the operation of each unit in FIG. 1 will be described with reference to the flowchart in FIG. The picture data 13 is extracted (101) from the input elementary stream 10 by the picture extracting means 11 (101), and the data 12 other than the picture data is output (102) to the hierarchical coded output 22.

The picture data 13 is divided (105) into DCT coefficient data 16 and data 15 other than the DCT coefficients by the DCT coefficient extracting means 14 unless it is the last block (103) of the picture data block.

The data 15 other than the DCT coefficient data is stored (106) in the temporary storage means 20.

The DCT coefficient data 16 is divided by the DCT dividing means 17 into base DCT coefficient data 18 and enhancement coefficient data 19 at a designated coefficient length (117).

The base coefficient data 18 is stored in the temporary storage 2
0 is stored (111), and finally (109), the EOB code 23 is stored (110) in the temporary storage means 20 by the EOB insertion means.

The enhancement coefficient data 19 is output to the hierarchical output 22 as user data (108). When the block of one picture data 13 ends (103), the data stored in the temporary storage means 20 is output (104) by the retrofit encoding output 22.

Next, a processing program for hierarchical encoding according to the present invention will be described with reference to the PAD diagram of FIG.

This program shows a function for dividing one block obtained by DCT-transforming 8 × 8 pixels into a base layer and an enhancement layer at a division point BP.

The variable i is cleared to 0, and the variable length code of the DCT coefficient is sequentially input to k (201) until the variable i reaches the set division point BP (200).
The input k is determined to be EOB (202), and
If so, the EOB is output to the base layer and the enhancement layer (203), and the function ends (204). k is EOB
If not, Run and Level are obtained from the variable length code k (205), and Run is added to the variable i (206).
Further, k is output to the base layer (207).

If the variable i is equal to or larger than BP, the EOB is output to the base layer (208).

The variable i is substituted for the variable j, and the variable length code of the DCT coefficient is sequentially input to k (210) until the variable j becomes 64 or more (209). It is determined whether the input k is EOB (211).
Is output to the enhancement layer (212), and the function ends (2).
13). If k is not EOB, R from variable length code k
Un and Level are obtained (214), and Ru is set to the variable j.
n is added (215). Further, k is output to the enhancement layer (216).

Hereinafter, a second embodiment of the present invention will be described with reference to the drawings.

FIG. 9 is a block diagram showing a second embodiment.
FIG. 10 is a flowchart showing the operation of the second embodiment.

In this embodiment, a hierarchical decoding process for converting a hierarchically encoded stream into an original MPEG2 elementary stream will be described.

In FIG. 9, reference numeral 31 denotes an enhancement extracting means for extracting enhanced data; 32, a temporary storage means for temporarily storing the enhanced data; 33, a DCT combining means for combining base and enhanced DCT coefficient data;
Is picture data, 36 is base DCT coefficient data,
Reference numeral 37 denotes EOB deletion means for deleting the EOB, and reference numeral 41 denotes base data.

Next, the operation of each unit in FIG. 9 will be described with reference to the flowchart in FIG.

From the input hierarchically encoded stream 22 (120), enhanced data is extracted by the enhanced extracting means 31 (122), and the enhanced data is stored in the temporary storage means 32 as the enhanced DCT coefficient data 19 (123). . On the other hand, the base data 41 other than the enhanced data is divided into picture data 35 and data other than the picture data by the picture extracting means 11 (124), and the data other than the picture data is output as the elementary stream 10 (128).

From the picture data 35, the DCT coefficient 36 is extracted (125) by the DCT coefficient extracting means 14,
The elements other than the CT coefficient 36 are output (128) as the elementary stream 10. Unnecessary EOB is deleted from the DCT coefficient 36 by the EOB deletion means 37 (126). The base DCT coefficient data 18 is stored in the temporary storage
Are combined (127) with the enhanced DCT coefficient data 19 stored in the DCT coefficient data 16 before hierarchical coding.
Is restored. The restored DCT coefficient data 16 is output (128) as the elementary stream 10.

Next, a processing program for hierarchical decoding according to the present invention will be described with reference to the PAD diagram of FIG.

This program shows a function for inputting base layer and enhanced layer data obtained by dividing one block obtained by subjecting an 8 × 8 pixel to DCT transformation at a division point BP, converting the data into data before hierarchical coding, and outputting the data.

The variable i is cleared to 0, and the variable length code of the DCT coefficient is sequentially input from the base layer to k (221) until the variable i becomes 64 or more (22).
0). It is determined whether the input k is EOB (222),
If EOB, k is discarded (223), and loop (22)
Exit from 0). If k is not EOB, Run and Level are obtained from the variable length code k (225), Run is added to the variable i (226), and k is output (227).

Next, the variable i is substituted into the variable j, and the variable j becomes 6
The variable-length code of the DCT coefficient of the base layer is sequentially input to k (230) until the number becomes 4 or more (2).
29). It is determined whether the input k is an EOB (231). If the input k is the EOB, k is output (232) and the function is terminated (233). If k is not EOB, Run and Level are obtained from the variable length code k (234), Run is added to the variable j (235), and k is output (236).

Hereinafter, a third embodiment of the present invention will be described with reference to the drawings. In the present embodiment, as an example, an image encoding apparatus that encodes the compression-encoded MPEG2 standard stream shown in FIG. 6 into the hierarchically encoded stream shown in FIG. 8 will be described. In the present embodiment, the hierarchical coding
Although the hierarchical encoding method shown in the first embodiment has been described as an example, the hierarchical encoding method is not limited, and hierarchical encoding methods such as spatial scalability, SNR scalability, and temporal scalability can be used.

In this embodiment, the conventional hierarchical coding is the method shown in FIG. 3, and the hierarchical coding according to the present invention is the method shown in FIG.

FIG. 13 is a block diagram showing the second embodiment, and FIG. 14 is a flowchart showing the operation of the second embodiment.

In FIG. 13, reference numeral 43 denotes image data before compression; image encoding means for compressing 44 images; 40, an MPEG2 elementary stream before hierarchical encoding; 41, a hierarchical code for performing hierarchical encoding according to the present invention. Encoding means, 42 is a conventional hierarchical encoding means for performing conventional hierarchical encoding, 47 is a hierarchical encoding flag for selecting a hierarchical encoding method, 48 is a hierarchical code selecting means for selecting a hierarchical encoding means, and 49 is a hierarchical encoding means. Encoded base data, 50 is hierarchically encoded enhanced data, 51 is stream multiplexing means for multiplexing hierarchically encoded enhanced data and base data, and 52 is encryption encoding means for encrypting enhanced data. , 53 are encryption keys, 54 is a stream in which enhanced data and base data are multiplexed, and 55 is a layer-encoded stream. Selection means for selecting a layer before encoding of the input stream, 56 selection flag for selecting the hierarchical coding, 57 output stream, 58 is an image encoding device.

Next, the operation of each unit in FIG. 13 will be described with reference to the flowchart of FIG.

The image data 43 input to the image encoding device 58 is compression-encoded (14
0) and outputs the elementary stream 40. The elementary stream 40 is selected by the selection unit 55 (14
1) If the selection flag 56 does not select the hierarchical encoding, the image encoding device 58 outputs (142) the output stream 57 as it is. When the selection flag 56 selects the hierarchical coding, the hierarchical code selecting means 48 converts the input stream 40 into the hierarchical coding means 41 (144) according to the present invention or the conventional hierarchical coding according to the hierarchical coding flag 47 (143). The data is input to the means 42 (145), subjected to hierarchical coding, and output as base data 49 and enhanced data 50. If there is an encryption key 53 (146), the enhanced data 50 is encrypted (147). The stream multiplexing means multiplexes (148) the output of the base data 49 and the output of the encryption coding means 52 into the stream format shown in FIG.
49).

Hereinafter, a fourth embodiment of the present invention will be described with reference to the drawings. In the present embodiment, an image decoding apparatus that decodes the hierarchically encoded stream illustrated in FIG. 8 into an original stream and reproduces an image will be described as an example. In the present embodiment, a method for automatically detecting the hierarchical encoding method is described, but it is of course possible to specify from outside. Further, in the present embodiment, the hierarchical coding method shown in the first embodiment is taken as an example of the hierarchical coding, but the present invention is not limited to the hierarchical coding method, and hierarchical codes such as spatial scalability, SNR scalability, and temporal scalability are used. Conversion methods are also available.

In this embodiment, the conventional hierarchical coding is the method shown in FIG. 3, and the hierarchical coding according to the present invention is the method shown in FIG.

FIG. 14 is a block diagram showing a second embodiment.

In FIG. 14, reference numeral 75 denotes an image decoding device;
0 is a hierarchically encoded input stream to be input to the image encoding apparatus, 61 is a stream separating unit that extracts enhanced data from the stream, 62 is base data obtained by removing enhanced data from the hierarchically encoded stream, and 63 is a stream. Enhancement data, 64 is an encryption / decryption means for decrypting the encrypted enhanced data, 65 is an encryption key for decrypting the encryption, 66 is the decrypted enhanced data, and 74 is the input enhanced data. An encoding detecting means for analyzing and identifying a hierarchical coding method, 67 is a hierarchical decoding means for decoding the hierarchical code according to the method of the present invention, 68 is a hierarchically decoded stream, and 69 is a conventional method for decoding the hierarchical code. A conventional hierarchical decoding means for converting, a hierarchically decoded stream 70, an input 71 for selecting an input, and Force selecting means, 72 selection flag for selecting the output 73 is hierarchical decoded elementary stream, the image decoding means for decoding the image 76, 77 are decoded image data.

Next, the operation of each section in FIG. 15 will be described with reference to the flowchart of FIG.

The input stream 60 input (160) to the image decoding device 75 outputs the base data 62 (16) according to the selection flag 72 by the selection means 71.
3) It is selected whether the image is reproduced by the image decoding means 76 of the stream 68 or the stream 70 subjected to the hierarchical decoding (161).

In the case of hierarchical decoding, the enhancement data 63 is extracted by the stream separation means 61 (164). If there is no enhanced data (162), selecting means 7
According to 1, the input stream 60 is output as it is as the output stream 73 (163). Enhanced data 63
Decrypts the encryption using the encryption key 65 by the decryption means (166). If the enhancement data 63 is not encrypted (165), it is output as it is as the enhancement data 66. If the encryption key 65 does not match (167), the enhanced data 63 is discarded, and the base data 62 is output as the output stream 73 (163).
Is done. The decrypted enhanced data 66 is identified by the encoding detecting means 74 as (168) the conventional hierarchical encoding (170) or the hierarchical encoding according to the present invention (169), and the corresponding hierarchical decoding means 67 or the conventional hierarchical encoding (169) is used. Is input to the hierarchical decoding means 69. If the enhancement data 66 is input, the hierarchical decoding means 67
, A hierarchical decoding process is performed, and the hierarchically decoded stream 68 is decoded into an image by the image decoding means 76 (171). If there is an input of the enhancement data 66, the conventional hierarchical decoding means 69 performs a hierarchical decoding process together with the input of the base data 62, and decodes the hierarchically decoded stream 70 into an image by the image decoding means 76 (1).
71).

Hereinafter, a fifth embodiment of the present invention will be described with reference to the drawings. In this embodiment, as an example, a connected transmitting device and receiving device are connected via a router device on a network, and a stream that is hierarchically coded on the transmitting device side is decoded into an original stream on the receiving device side. An image encoding / decoding device that reproduces an image is described below.
The router device deletes the enhancement layer according to the condition. The layer coding means uses the method shown in the first embodiment of the present invention.

In FIG. 17, reference numeral 80 denotes an image decoding means;
81 is a layer coding unit for performing layer coding, 82 is a transmission unit connected to a network, 84 is a network, 85
Is a router installed in the middle of the network, 86 is a receiving means, 87 is a hierarchical decoding means, and 88 is an image decoding means.

Next, the operation of each unit in FIG. 17 will be described.

The image data encoded by the image encoding means 80 is hierarchically encoded into an enhancement layer and a base layer by the hierarchical encoding means, and transmitted to the network 84 by the transmission means 82 as a stream. The stream transmitted from the transmitting device 83 is received by the receiving means 86 of the receiving device 89. The received stream is hierarchically decoded by the hierarchical decoding means 87 into the original encoded image data based on the hierarchically encoded enhancement layer and base layer, and is reproduced as an image by the image decoding means. The router device 85 on the network 84
The enhancement layer transmitted by the transmission device 83 is discarded according to conditions. When the enhanced layer is discarded by the router device, in the receiving device 89, only the base layer is received by the receiving unit 86, and is directly input to the image encoding unit 88 to be reproduced as an image only by the base layer.

In the above embodiment, an example in which an MPEG2 elementary stream is used has been described. However, the present invention is of course applicable to other encoding methods employing a compression encoding algorithm using DCT.

[0060]

According to the present invention, even if the data of the enhanced layer of the divided high frequency components is lost, the MPEG2
Since the elementary stream structure is not broken, an image can be normally reproduced only with the base layer data.

Further, by using the user-defined fields in the MPEG2 elementary stream for the enhancement layer data, even when a decoder that does not support hierarchical coding is used, the user-defined fields are ignored. An image can be reproduced using only low frequency components.

Further, by encrypting the data of the enhancement layer, the original high-quality image cannot be reproduced without the encryption key.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a first embodiment.

FIG. 2 is a flowchart showing the operation of the first embodiment.

FIG. 3 shows a conventional data partitioning method.

FIG. 4 is a data partitioning method according to the present invention.

FIG. 5 is a stream structure of MPEG2.

FIG. 6 shows an example of an MPEG2 stream.

FIG. 7 shows a stream structure according to the present invention.

FIG. 8 shows an example of a stream according to the present invention.

FIG. 9 is a PAD diagram showing the operation of hierarchical coding.

FIG. 10 is a block diagram showing a second embodiment.

FIG. 11 is a flowchart showing the operation of the second embodiment.

FIG. 12 is a PAD diagram showing the operation of hierarchical decoding.

FIG. 13 is a block diagram showing a third embodiment.

FIG. 14 is a flowchart showing the operation of the third embodiment.

FIG. 15 is a block diagram showing a fourth embodiment.

FIG. 16 is a flowchart showing the operation of the fourth embodiment.

FIG. 17 is a block diagram showing a fifth embodiment.

[Explanation of symbols]

10 ... MPEG stream, 11 ... Stream analysis means, 13 ... Picture data, 14 ... DCT coefficient extraction means, 16 ... DCT coefficient data, 17 ... DCT division means,
18 ... base data, 19 ... enhanced data, 20 ...
Temporary storage means, 21... EOB insertion means, 22... Hierarchically encoded stream, 29.

Claims (11)

[Claims] 1. An MPEG (Moving Picture)
DCTs (Discs) such as Experts Group
In encoding using the Rete Cosine Transform, a continuous DCT coefficient pair (run, l) up to EOB in each macroblock is executed.
evel) is divided by a designated DCT coefficient pair, and in data partitioning for storing the divided DCT coefficient pairs in different partitions, an EOB (End Of Block) is inserted at the division point, and the DCT up to the division point is inserted. The coefficient pair and the inserted EOB are stored in the base partition, and the DCT after the division point is stored.
A hierarchical coding method, wherein the last EOB following a coefficient pair and a DCT coefficient pair is stored in an enhanced partition. 2. An image encoding apparatus, comprising means for hierarchically encoding input data by the hierarchical encoding method according to claim 1. 3. The hierarchical encoding device according to claim 2, wherein
A hierarchical encoding method according to the conventional MPEG2 standard specification;
An image coding apparatus comprising means for selecting the hierarchical coding method according to claim 1. 4. The hierarchical encoding apparatus according to claim 2, wherein
An image coding apparatus comprising means for selecting whether or not to perform hierarchical coding using the hierarchical coding method according to claim 1. 5. An image comprising base data having the most basic data and a difference between the base data and the original image data.
An image coding apparatus, characterized in that a means for encrypting enhanced data is provided in hierarchical coding for dividing into more than one enhanced data. 6. A storage medium for storing the base data according to claim 4 and the encrypted enhanced data. 7. An image decoding apparatus, comprising: a hierarchical decoding means for decoding data hierarchically encoded by the hierarchical encoding method according to claim 1 into data before hierarchical encoding. 8. The hierarchical decoding device according to claim 7, wherein:
An image decoding apparatus comprising: means for detecting a hierarchically encoded method; and performing hierarchical decoding according to the detected hierarchically encoded method. 9. The hierarchical decoding apparatus according to claim 7, wherein:
An image decoding apparatus, comprising: means for switching whether to perform hierarchical decoding on an input hierarchically coded signal. 10. Hierarchical coding for dividing an image into base data having the most basic data and two or more enhanced data consisting of a difference between the base data and the original image data. An image decoding apparatus, comprising: means for decrypting encrypted enhanced data. 11. An image decoding apparatus according to claim 10, wherein the base data is output when the encrypted enhanced data cannot be decrypted.