HK1260971B - Image decoding, method, image encoding method, image decoding device, image encoding device, and image encoding/decoding device - Google Patents

Description

Image decoding method, image encoding method, image decoding device, image encoding device and image encoding and decoding device

This application is a divisional application of Chinese patent 201380044373.2, entitled “Image decoding method, image encoding method, image decoding device, image encoding device, and image encoding and decoding device”.

Technical Field

The present invention relates to an image encoding method and an image decoding method.

Background Art

Most current standard video coding algorithms are based on hybrid video coding. Hybrid video coding uses various reversible and irreversible compression methods to achieve the desired compression gain. Hybrid video coding forms the basis of ISO/IEC standards (MPEG-1, MPEG-2, and MPEG-4, among others), as well as ITU-T standards (H.26x, including H.261 and H.263).

The latest video coding standard is called H.264/MPEG-4 Advanced Video Coding (AVC). It was standardized by a collaborative group of the JVT (Joint Coding Team), ITU-T, and the ISO/IEC MPEG group.

Furthermore, with the goal of improving the efficiency of high-resolution video coding, the Joint Collaborative Team on Video Coding (JCT-VC) is researching a video coding standard called HEVC (High-Efficiency Video Coding).

Prior art literature

Non-patent literature

Non-patent document 1: "Wavefront Parallel Processing for HEVC Encoding and Decoding" by C. Gordon et al., no. JCTVC-F274-v2, from the Meeting in Torino, July 2011

Non-Patent Document 2: “Tiles” by A. Fuldsetal., no. JCTVC-F355-v1, from the Meeting in Torino, July 2011

Non-patent document 3: JCTVC-J1003_d7, “High efficiency video coding (HEVC) text specification draft 8” of July 2012

Summary of the Invention

Problems to be solved by the invention

In such image encoding and decoding methods, it is desired to improve efficiency by utilizing both parallel processing and slice dependency.

Therefore, an object of the present invention is to provide an image encoding method or an image decoding method that can improve efficiency when utilizing both parallel processing and dependent slices.

Means for solving problems

An image decoding method according to one embodiment of the present invention decodes a bit stream including a coded signal, wherein the coded signal is a signal obtained by dividing an image into a plurality of slices and encoding each of the plurality of slices, wherein each slice includes a plurality of coding units. The image decoding method includes a decoding step of decoding the coded signal, wherein the plurality of slices are either normal slices or dependent slices, wherein the normal slice is a slice in which information included in a slice header is used in another slice, and the dependent slice is a slice in which information included in a slice header of another slice is used during decoding. The image includes a plurality of rows, each of which includes a plurality of coding units. When the normal slice starts at a position other than the beginning of the first row, the second row following the first row does not start at a dependent slice.

Furthermore, these general or specific aspects may be implemented by systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or by any combination of systems, methods, integrated circuits, computer programs, and recording media.

Effects of the Invention

The present invention can provide an image encoding method or an image decoding method that can improve efficiency when utilizing both parallel processing and dependent slices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a block diagram of an image encoding apparatus according to an embodiment.

FIG2 is a block diagram of an image decoding device according to an embodiment.

FIG. 3A is a schematic diagram for explaining WPP according to the embodiment.

FIG3B is a schematic diagram for explaining dependent slices within a WPP according to an embodiment.

FIG4A is a schematic diagram for explaining dependent slices when WPP according to the embodiment is not used.

FIG4B is a schematic diagram for explaining dependent slices when WPP according to the embodiment is used.

FIG5 is a diagram showing a slice header of an entropy slice or a dependency slice according to an embodiment.

FIG. 6 is a diagram showing an example of a slice structure that is not permitted when WPP is used according to the embodiment.

FIG. 7 is a diagram showing an example of a permitted slice structure when WPP is used according to an embodiment.

FIG8 is a schematic diagram showing the CABAC initialization process according to the embodiment.

FIG9 is a flowchart of a process for determining a CABAC initialization method for a dependent slice corresponding to characteristics of a preceding slice according to an embodiment.

FIG. 10 is a diagram showing an example of a slice structure according to the embodiment.

FIG. 11 is a diagram showing an example of a slice structure according to the embodiment.

FIG12 is a diagram showing a syntax example of a slice header according to the first embodiment.

FIG13 is a flowchart showing a decision process of the slice-dependent CABAC initialization method according to the first embodiment.

FIG. 14 is a diagram showing an example of a picture divided into slices according to the second embodiment.

FIG15 is a flowchart showing a decision process of the CABAC initialization method according to the second embodiment.

FIG. 16 is a diagram showing an example of a picture divided into slices according to the second embodiment.

FIG. 17A is a diagram showing an example of a slice structure that is not permitted according to the second embodiment.

FIG. 17B is a diagram showing an example of a permitted slice structure according to the second embodiment.

FIG. 17C is a diagram showing an example of a permitted slice structure according to the second embodiment.

FIG17D is a diagram showing an example of a permitted slice structure according to the second embodiment.

FIG. 18 is a diagram showing an example of a picture divided into slices according to the second embodiment.

FIG. 19 is a diagram showing an example of a picture divided into slices according to the third embodiment.

FIG20 is a diagram showing the overall structure of a content supply system that implements content distribution services.

FIG21 is a diagram showing the overall configuration of a system for digital broadcasting.

FIG22 is a block diagram showing a structural example of a television set.

FIG23 is a block diagram showing a configuration example of an information reproducing/recording unit that reads and writes information on a recording medium such as an optical disc.

FIG. 24 is a diagram showing a structural example of a recording medium as an optical disc.

FIG25A is a diagram showing an example of a mobile phone.

FIG25B is a block diagram showing a configuration example of a mobile phone.

FIG26 is a diagram showing the structure of multiplexed data.

FIG27 is a diagram schematically showing how each stream is multiplexed in the multiplexed data.

FIG28 is a diagram showing in more detail how the video stream is stored in a PES packet sequence.

FIG29 is a diagram showing the structure of TS packets and source packets of multiplexed data.

FIG30 is a diagram showing the data structure of PMT.

FIG31 is a diagram showing the internal structure of multiplexed data information.

FIG32 is a diagram showing the internal structure of stream attribute information.

FIG33 is a diagram showing the steps of recognizing image data.

FIG34 is a block diagram showing a configuration example of an integrated circuit for realizing the moving picture encoding method and the moving picture decoding method according to each embodiment.

FIG35 is a diagram showing a configuration for switching the driving frequency.

FIG. 36 is a diagram showing the procedure of recognizing image data and switching the driving frequency.

FIG. 37 is a diagram showing an example of a lookup table in which the standard of image data and the driving frequency are associated with each other.

FIG38A is a diagram showing an example of a structure in which modules of a signal processing unit are shared.

FIG38B is a diagram showing another example of a structure in which modules of a signal processing unit are shared.

DETAILED DESCRIPTION

(Knowledge underlying the present invention)

The present inventors have discovered that the following problems may occur regarding the image encoding method and the image decoding method described in the "Background Art" column.

First, the HEVC image encoding device and image decoding device will be described.

The image signal input to the image encoding device includes a plurality of images, each of which is called a frame (picture). Each frame includes a plurality of pixels arranged in a two-dimensional array. In all the above-mentioned standards based on hybrid image coding, each image frame is divided into a plurality of blocks, each of which contains a plurality of pixels. The size of the block changes, for example, depending on the content of the image. In addition, different encoding methods can be used for each block. For example, in HEVC, the maximum size of the block is 64×64 pixels. This maximum size is called a maximum coding unit (LCU). The LCU can be divided into 4 coding units (CUs) in summary.

In H.264/MPEG-4 AVC, encoding is performed in units of macroblocks (usually 16×16 pixel blocks). These macroblocks are sometimes divided into subblocks.

Typically, the encoding process in hybrid video coding includes spatial and/or temporal prediction. Specifically, each block to be coded is predicted using spatially or temporally adjacent blocks, i.e., using an already coded video frame. Next, the difference between the coded block and the predicted result, or residual block, is calculated. This residual block is then transformed from the spatial (pixel) domain to the frequency domain. The purpose of this transformation is to reduce the correlation between the input blocks.

Next, the transform coefficients obtained through the transformation are quantized. This quantization is an irreversible compression. Furthermore, the quantized coefficients are reversibly compressed through entropy coding. Furthermore, auxiliary information required for reconstructing the coded image signal is encoded and output along with the coded image signal. This information includes, for example, information related to spatial prediction, temporal prediction, and/or quantization.

FIG. 1 is a diagram showing an example of an image encoding device 100 compliant with H.264/MPEG-4 AVC and/or HEVC.

Subtractor 105 calculates a residual signal 106 (residual block) that is the difference between the encoding target block of input image signal 101 and the corresponding prediction signal 181 (prediction block). This prediction signal 181 is generated by temporal prediction or spatial prediction by prediction unit 180. The prediction type used in the prediction may change for each frame or each block. Predicting blocks and/or frames using temporal prediction is called inter-frame coding, while predicting blocks and/or frames using spatial prediction is called intra-frame coding.

The prediction signal using temporal prediction is derived using coded and decoded images stored in memory. The prediction signal using spatial prediction is derived using boundary pixel values between coded and decoded adjacent blocks stored in memory. Furthermore, the number of intra-frame prediction directions is determined by the size of the coding unit.

Residual signal 106 is also called a prediction error or prediction residual. Transformation unit 110 transforms residual signal 106 to generate transform coefficients 111. Quantization unit 120 quantizes transform coefficients 111 to generate quantized coefficients 121. Entropy coding unit 190 entropy codes quantized coefficients 121 to further reduce the amount of stored data and enable reversible transmission. Entropy coding is, for example, variable-length coding. The length of a codeword is determined based on the probability of occurrence.

Through the above-described processing, a coded signal 191 (coded bit stream) is generated.

The image coding apparatus 100 also includes a decoding unit for obtaining a decoded image signal (reconstructed image signal). Specifically, the inversion unit 130 generates a residual signal 131 by inverse quantizing and inverse transforming the quantized coefficients 121. This residual signal 131 is strictly different from the original residual signal 106 due to the influence of quantization error, also known as quantization noise.

Next, adder 140 adds residual signal 131 to prediction signal 181 to generate decoded image signal 141. Thus, to maintain compatibility between the image encoding apparatus and the image decoding apparatus, both the image encoding apparatus and the image decoding apparatus generate prediction signal 181 using encoded and decoded image signals.

Furthermore, quantization adds quantization noise to the decoded image signal 141. Since encoding is performed on a block-by-block basis, the added noise often varies from block to block. Consequently, block boundaries in the decoded image signal become noticeable, particularly when strong quantization is used. This block noise degrades image quality in human perception. To reduce this block noise, the deblocking filter 150 performs deblocking filtering on the decoded image signal 141.

For example, in the deblocking filtering process of H.264/MPEG-4 AVC, a filtering process appropriate for each region is selected. For example, when block noise is high, a stronger (narrowband) low-pass filter is used, and when block noise is low, a weaker (wideband) low-pass filter is used. The strength of this low-pass filter is determined based on the prediction signal 181 and the residual signal 131. This deblocking filtering process smoothes the edges of blocks. This improves the subjective image quality of the decoded image signal. In addition, the filtered image is used for motion compensation prediction of the next image. Therefore, this filtering process also reduces the prediction error, thereby improving coding efficiency.

Adaptive loop filter 160 generates decoded image signal 161 by performing sample adaptive offset (SAO) and/or adaptive loop filtering on decoded image signal 151 after deblocking filtering. As described above, deblocking filtering improves subjective image quality. Meanwhile, sample adaptive offset (SAO) and adaptive loop filter (ALF) aim to improve pixel-by-pixel reliability (objective quality).

SAO is a process that adds an offset value to pixels based on proximity. ALF is used to compensate for image distortion caused by compression. For example, ALF is a Wiener filter with filter coefficients determined to minimize the mean square error (MSE) between the decoded image signal 151 and the input image signal 101. For example, ALF coefficients are calculated and transmitted on a frame-by-frame basis. ALF can also be applied to the entire frame (image) or to a local area (block). Furthermore, auxiliary information indicating the area to be filtered can be transmitted on a block-by-block, frame-by-frame, or quadtree-by-quadtree basis.

In order to decode an inter-frame coded block, a portion of the coded and decoded image must be stored in advance in the reference frame buffer 170. The reference frame buffer 170 stores the decoded image signal 161 as a decoded image signal 171. The prediction unit 180 performs inter-frame prediction using motion-compensated prediction. Specifically, first, the motion estimator searches for a block that is most similar to the target block among the blocks included in the coded and decoded image frames. This similar block is used as the prediction signal 181. The relative offset (motion) between the target block and the similar block is sent to the image decoding device as motion data. This motion data is, for example, a three-dimensional motion vector included in the auxiliary information provided together with the coded image data. Here, the so-called three-dimensional includes two spatial dimensions and one temporal dimension.

Furthermore, to optimize prediction accuracy, motion vectors with spatial sub-pixel resolution, such as half-pixel resolution or quarter-pixel resolution, may be used. Motion vectors with spatial sub-pixel resolution represent spatial locations within the decoded frame where no actual pixel values exist, i.e., sub-pixel locations. Therefore, spatial interpolation of pixel values is required for motion-compensated prediction. This interpolation process is implemented, for example, by an interpolation filter (included in the prediction unit 180 shown in FIG1 ).

In both intra-frame and inter-frame coding modes, the residual signal 106, which is the difference between the input image signal 101 and the prediction signal 181, is transformed and quantized to generate quantization coefficients 121. Generally, the transform unit 110 uses an orthogonal transform such as a two-dimensional discrete cosine transform (DCT) or its integer version for this transformation. This effectively reduces correlation in natural images. Furthermore, since low-frequency components are generally more important for image quality than high-frequency components, more bits are used for low-frequency components than for high-frequency components.

The entropy coding unit 190 transforms the quantized coefficients 121 of the two-dimensional matrix into a one-dimensional matrix. Typically, a so-called zigzag scan is used. In a zigzag scan, the two-dimensional matrix is scanned in a predetermined order, from the DC coefficient at the upper left corner to the AC coefficient at the lower right corner. Generally, energy is concentrated in the upper left portion of the two-dimensional array of coefficients corresponding to low frequencies, so if a zigzag scan is performed, the values in the latter half tend to become zero. Therefore, by using run-length coding as part of entropy coding or as a pre-processing step, efficient coding can be achieved.

H.264/MPEG-4 AVC and HEVC use multiple types of entropy coding. Some syntax elements are coded with fixed length, but almost all syntax elements are variable length coded. In particular, context-adaptive variable length coding is used for encoding the prediction residual, while various integer codes are used for encoding other syntax elements. Context-adaptive arithmetic coding (CABAC) is also sometimes used.

Variable-length coding allows for reversible compression of encoded bitstreams. However, because codewords are variable length, they must be decoded continuously. Specifically, without uniquely indicating the position of the first codeword (starting point) at which entropy coding is restarted (initialized) or decoding, subsequent codewords cannot be encoded or decoded before the preceding codeword has been encoded or decoded.

The bitstream is encoded into a single codeword using arithmetic coding based on a specified probability model. In the case of CABAC, the specified probability model is determined by the content of the video sequence. Therefore, the longer the bitstream to be encoded, the more efficient the arithmetic coding and CABAC are. In other words, CABAC applied to the bitstream is more efficient for larger blocks. CABAC is restarted at the beginning of each sequence. Specifically, the probability model is initialized to a predetermined or specified value at the beginning of each video sequence.

H.264/MPEG-4, H.264/MPEG-4 AVC, and HEVC have two functional layers: the Video Coding Layer (VCL) and the Network Abstraction Layer (NAL). The video coding layer provides coding functions. The NAL encapsulates information elements into standard units called NAL units, depending on their purpose, such as transmission across channels and storage in storage devices. These information elements include, for example, information required for encoding prediction error signals and decoding image signals. Information required for decoding image signals includes prediction type, quantization parameters, motion vectors, and so on.

NAL units include VCL NAL units that contain compressed video data and related information, non-VCL units that encapsulate additional data such as parameter sets associated with the entire video sequence, and supplementary extension information (SEI) that provides additional information that can be used to improve decoding accuracy.

For example, non-VCL units include parameter sets. A parameter set is a collection of multiple parameters related to the encoding and decoding of a specific video sequence. For example, a parameter set includes a sequence parameter set (SPS), which contains parameters related to the encoding and decoding of the entire video sequence (picture sequence).

The sequence parameter set has a syntax structure containing syntax elements. The referenced picture parameter set (PPS) is specified by the syntax element pic_parameter_set_id, included in each slice header. Furthermore, the referenced SPS is specified by the syntax element seq_parameter_set_id, included in the PPS. In this way, the syntax elements included in the SPS are applied to the entire coded video sequence.

The PPS is a parameter set that defines the parameters used for encoding and decoding a single picture in a video sequence. The PPS has a syntax structure consisting of syntax elements. The referenced picture parameter set (PPS) is specified using the syntax element pic_parameter_set_id contained in each slice header. In this way, the syntax elements contained in the SPS are applied to the entire coded picture.

Therefore, it is easier to continue tracking the SPS than the PPS because the PPS varies for each picture, while the SPS is constant for the entire video sequence, which may reach several minutes or hours.

The VPS is a top-level parameter that contains information about multiple video sequences. This information includes things like the bit rate and the temporal layering structure of the video sequence. It also includes information about inter-layer dependencies (dependencies between different video sequences). Therefore, the VPS can be considered information about multiple video sequences, providing a general overview of each video sequence.

FIG. 2 is a block diagram showing an example of an image decoding device 200 compliant with the H.264/MPEG-4 AVC or HEVC video coding standard.

The coded signal 201 (bitstream) input to the image decoding device 200 is sent to the entropy decoding unit 290. The entropy decoding unit 290 decodes the coded signal 201 to obtain information elements necessary for decoding, such as quantized coefficients, motion data, and prediction mode. Furthermore, the entropy decoding unit 290 inversely scans the obtained quantized coefficients to obtain a two-dimensional array, thereby generating quantized coefficients 291, which are then output to the inversion unit 230.

The inversion unit 230 generates a residual signal 231 by inverse quantizing and inverse transforming the quantized coefficients 291. The residual signal 231 corresponds to a difference obtained by subtracting a prediction signal from an input image signal having no quantization noise and no error and input to the image encoding apparatus.

The prediction unit 280 uses temporal prediction or spatial prediction to generate a prediction signal 281. Typically, the decoded information element also includes information necessary for prediction, such as prediction type, in the case of intra prediction, and information necessary for prediction, such as motion data, in the case of motion compensation prediction.

Adder 240 generates a decoded image signal 241 by adding spatial domain residual signal 231 to prediction signal 281 generated by prediction unit 280. Deblocking filter 250 generates decoded image signal 251 by performing deblocking filtering on decoded image signal 241. Adaptive loop filter 260 generates decoded image signal 261 by performing sample adaptive offset processing and adaptive loop filtering on decoded image signal 251. Decoded image signal 261 is output as a display image and stored as decoded image signal 271 in reference frame buffer 270. Decoded image signal 271 is used for temporal or spatial prediction of subsequent blocks or images.

Compared to H.264/MPEG-4 AVC, HEVC features a highly parallel processing capability that facilitates encoding and decoding. Like H.264/MPEG-4 AVC, HEVC can divide a frame into multiple slices. Each slice consists of multiple LCUs, which are consecutively scanned. In H.264/MPEG-4 AVC, slices can be decoded individually, without performing spatial prediction across slices. This enables parallel processing on a slice-by-slice basis.

However, since slices have relatively large headers and there is no dependency between slices, compression efficiency is reduced. Furthermore, CABAC encoding loses efficiency when performed on smaller data blocks.

To address this issue, wavefront parallel processing (WPP) has been proposed to enable more efficient parallel processing. In WPP, the processed probability model of the second LCU in the previous row is used as the CABAC probability model for resetting the first LCU (the first LCU) in each LCU row (hereinafter simply referred to as a "row") of a picture. This maintains inter-block dependencies and enables parallel decoding of multiple LCU rows. Furthermore, the processing of each row is delayed by two LCUs relative to the previous row.

Furthermore, information indicating a starting point, where decoding of an LCU line is to be started, is included in the slice header and signaled.

Another method for improving parallelization is the use of tiles. A frame (picture) is divided into multiple tiles. Each tile is rectangular and contains multiple LCUs. The boundaries between tiles are set to divide the picture into an array. Furthermore, the tiles are processed in raster scan order.

Furthermore, all dependencies are lost at tile boundaries. Entropy coding methods such as CABAC are also reset at the beginning of each tile. Furthermore, only deblocking filtering and sample adaptive offset processing are applied across tile boundaries. This allows multiple tiles to be encoded or decoded in parallel. Details on tiles are described in Non-Patent Documents 2 and 3.

Furthermore, to make the slice concept more suitable for parallelization than the original purpose of slices in H.264/MPEG-4 AVC, which was error resilience, the concepts of dependent slices and entropy slices were proposed. Specifically, HEVC uses three slices: normal slices, dependent slices, and entropy slices.

Normal slices are slices already known from H.264/MPEG-4 AVC. Spatial prediction cannot be performed between normal slices. In other words, prediction cannot be performed across slice boundaries. In other words, normal slices are encoded without reference to other slices. CABAC restarts at the beginning of each slice to enable decoding of each slice.

Furthermore, a normal slice is used at the beginning of a frame. That is, each frame must start with a normal slice. A normal slice has a header containing parameters required for decoding the slice data.

An entropy slice is a slice that allows spatial prediction between a parent slice and an entropy slice. Here, a parent slice is, for example, a normal slice immediately preceding an entropy slice. The parent slice and entropy slice are analyzed independently.

Furthermore, slice data parsing is performed independently for parent slices and entropy slices, excluding the syntax elements in the slice header. Specifically, CABAC decoding of entropy slices requires the syntax elements contained in the parent slice's slice header. For example, these syntax elements include switching information indicating whether the slice data contains filter parameters. If filter parameters are present in the slice data, the CABAC decoder extracts this information. If not, the CABAC decoder does not assume filter data. In other words, after parsing the slice header of a normal slice, the CABAC decoder can process the parent slice and entropy slice in parallel.

However, the mother slice may also be a normal slice, for example, which is required in the reconstruction of the pixel values of the entropy slice. In addition, CABAC is also restarted at the beginning of the slice so that the entropy slice can be analyzed independently.

In entropy slices, a shorter slice header than that of normal slices can be used. The slice header includes a subset of coding parameters related to the information sent in the normal slice header. Information not included in the entropy slice header is copied from the header of the parent slice.

Dependency slices are similar to entropy slices without CABAC restart. Restarting CABAC includes initializing the context table (probability table) to default values and performing a termination process (termination process) in the arithmetic coding or arithmetic decoding process.

The parent slice header is used to parse and/or decode dependent slices. Therefore, neither the parent slice nor the dependent slice can be parsed, and therefore, the dependent slice cannot be decoded without the parent slice. A parent slice is typically the preceding slice of a dependent slice in coding order and includes a complete slice header. This also applies to the parent slice of an entropy slice.

Generally speaking, an entropy slice can be considered as a header parameter that depends on other slices, so the present invention can be applied to both dependent slices and entropy slices.

As described above, dependent slices and entropy slices use the slice header of the immediately preceding slice in the encoding order of the slices (information not included in the header of the dependent slice). This rule is used in a general manner. It is recognized that the parent slice on which the object dependent slice depends is referenceable. This reference includes the use of spatial prediction between slices and shared CABAC states. Dependent slices use the CABAC context table generated at the end of the immediately preceding slice. In this way, dependent slices do not initialize the CABAC table to the default value and continue to use the already created table. In addition, entropy slices and dependent slices are described in Non-Patent Document 3 (for example, see "dependent_slice_flag" on page 73).

When WPP is used, if the dependent slice starts at the beginning of an LCU row and the slice including the upper right LCU of the leading LCU indicates that it can be referenced, the dependent slice uses the CABAC context table of the upper right LCU.

HEVC provides some profiles. Profiles include settings for image encoding and decoding devices used in specific applications. For example, "main profiles" only include normal slices and dependent slices, but not entropy slices.

As described above, coded slices are encapsulated into NAL units, which are then encapsulated into, for example, Real-Time Protocol (RTP), and finally into Internet Protocol (IP) packets. This or another protocol stack enables the transmission of coded video over packet-oriented networks such as the Internet or a dedicated network.

Typically, a network includes at least one router, constructed from specialized hardware operating at ultra-high speeds. A router receives IP packets, parses the packet header, and forwards the appropriate packet to its destination. Because a router must handle communications from numerous sources, its control logic must be as simple as possible. Since a router determines the route for forwarding an IP packet, it must verify the destination address field in the IP header. To support Quality of Service (QoS), intelligent (media-aware) routers additionally verify specialized fields within network protocol headers, such as the IP header, RTP header, and NALU header.

As can be seen from the above discussion of video coding, different types of slices, such as dependent slices and entropy slices, defined for parallel processing, have different importance in terms of image quality degradation in the event of data loss. Dependent slices cannot be parsed and decoded without parent slices. This is because the entropy coding and decoding units cannot be restarted at the beginning of a dependent slice. Therefore, parent slices are more important in reconstructing an image or video.

In HEVC, dependency slices and entropy slices incorporate inter-slice dependencies (intra-frame dependencies) as a supplementary aspect of dependencies. This dependency is not the only dependency within a frame.

Furthermore, since slices are processed in parallel for each tile, the contexts of the arithmetic coding and decoding units are determined by default settings or based on the encoded or decoded slices. However, since header dependencies differ from the dependencies of arithmetic coding initialization, this may violate the purpose of the parallel processing and slice dependency mechanisms, resulting in delays and increased complexity.

Dependent slices can be used together with parallel processing tools such as WPP and tiles. Furthermore, by using dependent slices, a wavefront (substream) that can reduce transmission delay can be generated without causing coding loss.

Furthermore, since CABAC is not restarted in dependent slices, it is possible to use dependent slices as the starting point of a CABAC substream. Furthermore, since it indicates the starting point of an independent parsing, information indicating the starting point can also be included in the bitstream and transmitted. In particular, when two or more CABAC substreams are encapsulated as normal slices or dependent slices, the starting point is explicitly signaled using the number of bytes of each substream. Here, a substream represents a portion of a stream that can be parsed separately using a starting point. Furthermore, since each dependent slice requires a NAL unit header, a dependent slice can be used as a "marker" for the starting point. That is, it is possible to signal the starting point of such a marker.

The method of explicitly signaling the start point and the method of marking the start point via dependent slices can be used simultaneously. Here, it is necessary to be able to determine the start point of each NAL unit (the beginning of each NAL header). Any method can be used for this determination. For example, the following two methods can be used.

The first method is to insert a 3-byte start code at the beginning of each NAL header. The second method is to package each NAL unit into a different packet. In addition, due to the dependency of slices, the size of the slice header can also be reduced.

These methods enable CABAC parsing of entropy slices in parallel. This is because CABAC is always restarted at the beginning of the entropy slice. Parallel CABAC parsing after continuous pixel construction can overcome obstacles in CABAC parallel processing. Specifically, the WPP parallelization tool can be used to implement the decoding process of each LCU row on a single processing core. Furthermore, the allocation of LCU rows to each core can also be different. For example, two rows can be allocated to one core, or one row can be allocated to two cores.

Figure 3A shows a picture 300 assigned to multiple rows. Each row includes multiple largest coding units (LCUs). Rows 301 (Wavefront 1) and 302 (Wavefront 2) are processed in parallel. As shown by the arrows in the CABAC states (CABACstates) in Figure 3A, processing of row 302 begins after the first two LCUs in row 301 are decoded. Furthermore, the CABAC states after encoding or decoding the first two LCUs in row 301 are used when initializing CABAC in row 302. This allows processing of row 302 to begin after processing of the first two LCUs in row 301 is completed. In other words, there is a delay of two LCUs between the two processing cores.

Figure 3B shows an example of using dependent slices using WPP. The picture 310 shown in Figure 3B includes rows 311 to 314. Row 311 (Wavefront 1), row 312 (Wavefront 2), and row 313 (Wavefront 3) are processed using different cores.

Dependent slices form a WPP that can improve latency. Dependent slices do not have a complete slice header. Furthermore, if the start point is known (or the start point of the dependent slice is known using the rules described above), the dependent slice can be decoded independently of other slices. Furthermore, dependent slices can form a WPP without coding loss, making it suitable for low-latency applications.

In the typical case of encapsulating substreams (LCU rows) into slices, reliable parallel entropy encoding and decoding requires a clear start point to be inserted into the slice header. Therefore, the slice is not ready for transmission until the last substream of the slice is fully encoded. Furthermore, the slice header is not completed until all substreams in the slice are encoded. In other words, the beginning of the slice cannot be transmitted via RTP/IP layer packet fragments until the entire slice is processed.

However, when using dependent slices, since they can be used as starting point markers, explicit signaling of the starting point is unnecessary. Consequently, a normal slice can be divided into many dependent slices without coding loss. Furthermore, dependent slices can be transmitted immediately after encoding of the encapsulated substreams is complete (or even earlier in the case of packet fragmentation).

Furthermore, dependent slices do not reduce the dependency of spatial prediction. Furthermore, dependent slices do not reduce the parsing dependency. This is because the CABAC state of the preceding slice is usually required for parsing the object-dependent slice.

If dependent slices are not permitted, each LCU row can be used as a slice. This structure will improve transmission delay, but at the same time will cause a large coding loss as mentioned above.

Consider the case where an entire frame (picture) is encapsulated into a single slice. In this case, to enable parallel parsing, the slice header must signal the start point of the substream (LCU row). This results in transmission delay at the frame level. Specifically, the header must be corrected after the entire frame is encoded. Encapsulating the entire picture into a single slice does not itself increase transmission delay. For example, it is possible to start transmitting part of a slice before encoding is complete. However, when using WPP, the slice header must be corrected later to indicate the start point. Consequently, transmission of the entire slice is delayed.

In this way, using dependent slices can reduce latency. As shown in Figure 3B, picture 310 is divided into line 311, which is a normal slice, and lines 312, 313, and 314, which are dependent slices. When each line is a dependent slice, the transmission of one line can be delayed without coding loss. This is because dependent slices do not reduce spatial dependencies and do not restart the CABAC engine.

Figures 4A and 4B illustrate another example of CABAC initialization. Figure 4A illustrates CABAC initialization without using WPP. Furthermore, neither WPP nor tiles are used. Furthermore, the use of both normal slices and dependent slices is permitted.

The dependent slice (3) copies the header from the normal slice (2). That is, the normal slice (2) is the parent slice of the dependent slice (3). The dependent slice (3) uses the context table generated at the end of the normal slice (2). The dependent slice (3) is not the normal slice (1) but depends on the normal slice (2). That is, there is no spatial prediction between the normal slice (1) and the dependent slice (3).

Fig. 4B is a diagram showing the initialization of CABAC when WPP is used. It is permitted to use normal slices, dependent slices, and WPP together.

The dependent slice (3) copies the header of the normal slice (2). It is assumed that the dependent slice (3) uses the context table generated at the end of the second LCU of the normal slice (1). However, since slice (2) is a normal slice, the second LCU of slice (1) cannot be referenced. In other words, slice (1) is not the preceding slice immediately preceding the dependent slice in the coding order and is therefore not referenced.

However, slice (2) is used as a reference slice for slices (3) and (4). That is, when decoding slice (3) begins, the CABAC state must be initialized to the default value (indicated by the dotted arrow in FIG4B ). The dependent slice (4) uses the CABAC state after the second LCU in the upper right corner, which complies with the WPP conditions described above (indicated by the solid arrow).

5 is a diagram showing an example of the syntax of a slice header according to the current HEVC reference model (HM8.0). The slice header 320 includes a syntax element dependent_slice_flag indicating whether the target slice is a dependent slice or a normal slice.

As shown in row 321 of Figure 5 , when dependent_slice_flag is equal to 0, the header contains slice header information. In other words, the slice has a complete header. Otherwise, the header does not contain slice header information. In other words, as described above, dependent slices and entropy slices do not have complete slice headers and refer to the header of the preceding normal slice.

To support parallel processing, a start point is subsequently signaled. Even without restarting the entropy coding or decoding units, this start point can be used to independently and concurrently decode portions of the video stream (substream) between these start points. As mentioned above, start points are also marked for dependent slices, normal slices, and entropy slices.

HEVC has several parallel processing tools. As mentioned above, these tools are WPP, dependency slices, entropy slices, and tiles. However, since these tools are not always interchangeable, there are limitations on their combined use. Generally speaking, tiles and slices are used together.

However, among the main attributes, there are restrictions that require that a slice must be split into an integer number of tiles or more, and that a tile must be split into an integer number of slices or more. Typically, these restrictions apply to specific attributes (or specific levels of attributes). The purpose of these restrictions is to reduce hardware execution complexity.

When the entropy_coding_sync_enabled_flag of the PPS is equal to 1 (i.e., WPP is used) and the first coding block included in the slice is not the first coding block of the first coding tree block of the row consisting of the coding tree blocks of the tile, the condition for the bitstream to conform to the standard is that the last coding block of the slice belongs to the same coding tree block row as the first coding block of the slice. The coding tree represents the structure of the LCU and the LCU is further divided into 4 blocks. That is, when WPP is possible and the slice does not start from the beginning of the target LCU row, the slice must end at the end of the target LCU row or before it. In addition, not only the parallel processing means but also the details of the HEVC syntax are recorded in non-patent document 3.

This restriction is explained using Figure 6. Picture 330 shown in Figure 6 includes slices 331, 332, and 333, which are normal slices. Slices 331 and 332 are contained in a single LCU row. Slice 333 spans multiple LCU rows (three in this example) and is therefore not permitted. According to the above restriction, slice 333 must end at the end of the first LCU row.

FIG7 shows a picture 340 with a permitted slice structure when using WPP. Picture 340 includes slices 341, 342, and 343 as normal slices, and slice 344 as a dependent slice. These slices 341, 342, and 343 are included in the first LCU row. Slice 344 includes the following two rows.

Since slice 344 is a dependent slice, CABAC initialization for slice 344 depends on other slices 341, 342, and/or 343. If either slice 342 or 343 is a normal slice, as shown in FIG7 , slice 344 is initialized to the default CABAC state. Otherwise, the WPP table is used. Specifically, the processed CABAC state of the second LCU in the LCU row above the target row is used for initialization.

In this example, as described in FIG. 4B and in the above-mentioned description of the associated CABAC initialization, CABAC for the dependent slice 344 is initialized using a preset default CABAC state.

In this way, CABAC initialization is based on multiple preceding slices. Consequently, the processing, and in particular, the parsing, of the target slice depends on multiple other slices. Specifically, the type of preceding slice determines whether the CABAC context should be initialized with the default value or the WPP value. Thus, whether the preceding slice can be used determines the initialization method used for the target slice. This requires a fairly complex sequence of processing, which is described in detail below.

Since the first slice 341 has at least two LCUs, it is possible to refer to the CABAC state after encoding or decoding the first two LCUs.

Furthermore, if slice 342 or slice 343 is missing, slice 344 cannot be correctly decoded. This is because the type of slice 342 or 343 is unknown, making CABAC initialization impossible. In other words, even if information about only the two preceding slices is missing and slice 344 is correctly acquired, CABAC initialization for slice 344 cannot be performed, so the data for the correctly acquired slice 344 is discarded. Therefore, error concealment is required for slice 344. Consequently, incomplete error concealment can lead to distortion and degraded image quality.

In this case, almost all syntax elements in the slice header (primarily those controlling the switching of specific filtering operations, etc.) must be determined for all slices included in the frame. While some syntax elements may change for each slice, the control parameters determined for the entire frame are maintained throughout almost all processing performed by the image encoding device. Therefore, the following error concealment method can be used. This method only requires information about whether the lost slice is a dependent slice or a normal slice.

Furthermore, if packets arrive out of order, decoding latency can worsen. Specifically, if packets are expected to be rearranged, decoding latency can worsen. This contradicts the fundamental purpose of WPP, which is to provide ultra-low latency through dependent slicing.

FIG8 is a diagram illustrating another example of CABAC initialization processing. FIG8 illustrates the slice structure shown in FIG7 . The picture 350 shown in FIG8 includes slices 351 and 354. Slice 351 is a normal slice, the first slice in a frame, and includes four LCUs. At the beginning of the frame, i.e., at the beginning of slice 351, CABAC is initialized to a default state value (zero state). Alternatively, multiple default states may exist, in which case one default state is selected from the multiple default states. The default state is a specified value of the probability model for arithmetic coding.

Even if data for dependent slice 354 is obtained, if data for slices 342 and 343 (see FIG7 ) is missing due to loss or error, slice 354 cannot be decoded. This is because, as mentioned above, the CABAC engine cannot be initialized without information on slices 342 and 343.

9 is a flowchart of the process of determining the initialization method performed when the dependent slice 354 is obtained. In other words, this flowchart shows the method of dependency on two or more slices in CABAC initialization.

Assume that the following conditions are set for slice (4) (dependent slice 354): WPP is enabled. The dependent_slice_enabled_flag of the SPS is set to 1. The position of slice (4) satisfies (Equation 1).

slice_address % numLCUinRow=0 ･･･(Formula 1)

Here, "%" represents the modulo operation (the remainder after integer division). The parameter numLCUinRow represents the number of LCUs per row of picture 350. Therefore, the condition in (Equation 1) is satisfied at the beginning of a row. The parameter numLCUinRow can be derived based on the SPS settings.

First, it is determined whether the slice (4) is a dependent slice (S101). If the slice (4) is not a dependent slice (No in S101), default initialization is performed.

As shown in FIG8 , if slice (4) is a dependent slice (“Yes” in S101 ), i is set to 3 ( S102 ). That is, slice (3) immediately before slice (4) is set as slice i.

Next, it is determined whether slice i starts from the row above slice (4) (S103). Here, since i is set to 3, slice i is slice (3) immediately before the dependent slice (slice (4)) to be processed.

When the slice i does not start from the upper row of the slice (4) (No in S103), WPP is initialized (initialization using the WPP table) (S107).

On the other hand, when the slice i starts from the upper row of the slice (4) (Yes in S103), that is, in the case shown in FIG. 8 , it is determined whether the slice i is a dependent slice (S104).

If slice i is not a dependent slice (No in S104), the start position of slice i is analyzed. Specifically, it is determined whether slice_address % numLCUinRow is less than 2 (S106). In other words, it is determined whether the start position of slice i is the beginning of a row or the second LCU.

If slice_address % numLCUinRow is less than 2 (Yes in S106 ), WPP is initialized ( S107 ). On the other hand, if slice_address % numLCUinRow is greater than or equal to 2 (No in S106 ), default initialization is performed ( S108 ).

If slice i is a dependent slice (Yes in S104 ), the start position of slice i is analyzed. Specifically, a determination is made as to whether slice_address % numLCUinRow is less than 3 ( S105 ). In other words, a determination is made as to whether the start position of slice i is the beginning of a row, the second LCU, or the third LCU.

If slice_address % numLCUinRow is less than 3 ("Yes" in S105), WPP is initialized (S107). On the other hand, if slice_address % numLCUinRow is greater than or equal to 3 ("No" in S105), initialization is not performed, and index i is decremented by 1 (S109). That is, in this example, slice (2) two slices before the target slice (slice (4)) is set as slice i. Then, the processing from step S103 onward is performed on slice (2). Furthermore, if the same determination is made for slice (2), slice (1) is then set as slice i.

FIG10 shows a picture 360. Picture 360 includes five slices 361 to 365. Slice 361 is a normal slice, encompassing the entire first row. Slice 362 is a dependent slice, encompassing the entire second row. The third row includes dependent slices 363 and 364. Slice 365 is a dependent slice, encompassing the entire fourth row.

The following discusses each case where the slice 364 is lost or delayed, where the slice 364 is a dependent slice, and where the slice 364 is a normal slice.

If slice 364 is lost, the image decoding device cannot determine the type of slice 364. If the lost slice 364 is a dependent slice, decoding of slice 365 and subsequent slices can continue with slight errors during the reconstruction process. This is because, as described using Figures 8 and 9, slice 365 uses the CABAC state of the second LCU of slice 363. Therefore, no errors occur during the CABAC initialization process. However, since slice 365 uses spatial prediction based on slice 364, errors may occur during the pixel reconstruction process.

On the other hand, if the lost slice 364 is a normal slice, slice 365 cannot be decoded. This is because the syntax elements may use information from the slice header of the lost slice 364. In other words, normal slice 364 is the parent slice of dependent slice 365, and parsing and decoding dependent slice 365 requires information about the parent slice.

Without knowing the slice type of lost slice 364, the image decoding device discards decodable slice 365 to avoid possible decoding errors if lost slice 364 is a normal slice. Discarding slice 365 even if the data for slice 365 can be correctly retrieved is inefficient. Furthermore, all dependent slices following slice 365 must also be discarded.

If slice 364 is a normal slice, the CABAC engine is initialized to the default CABAC value for decoding slice 365 (see the case of "No" in S101 of FIG9 ). Consequently, slice 365 is not dependent on slice 363. Furthermore, spatial prediction between slices 363 and 365 is not performed. Thus, since CABAC is initialized to the default value at the beginning of slice 365, dependent slice 365 behaves similarly to a normal slice.

However, a normal slice has a complete slice header. On the other hand, slice 365 has only a shorter slice header, dependent on the parameters set in the slice header of the preceding normal slice. In other words, while there is an advantage in reducing the header size when slice 365 is a dependent slice, this advantage is not significant. On the other hand, if slice 365 is a normal slice, it can be decoded. Thus, in this case, setting slice 365 as a normal slice can be considered to be more advantageous than setting it as a dependent slice.

However, in WPP, the goal of slice dependency is not to ensure robustness against loss, but rather to enable WPP operation at ultra-low latency. On the other hand, in ultra-low-latency applications across networks, such as real-time applications, packet loss and packet reordering are to be expected. In such cases, as long as slice 364 is eventually available, slice 365 can be decoded. However, this will at least result in increased latency and packet loss. Therefore, in environments with high loss, WPP will operate suboptimally.

Fig. 11 is a diagram illustrating another issue related to CABAC initialization when WPP is used, and shows a picture 370. The picture 370 includes four slices 371-374.

Slice 371 is a normal slice, and slice 372 is a dependent slice. Here, slice 371 has at least two LCUs. The first row of picture 370 includes slices 371 and 372. The second row of picture 370 includes slices 373 and 374 as dependent slices.

In this case, it is assumed that the image coding device utilizes at least two processor cores. In other words, when using WPP, the image coding device encodes and parses two LCU rows in parallel. As a result, slice 373 becomes available long before slice 372 becomes available.

However, since CABAC initialization for slice 373 depends on slice 372, decoding of slice 373 cannot begin. Consequently, the delay between lines in encoding or decoding cannot be reduced to less than the entire length of a single LCU line. This contradicts the purpose of WPP, which aims to minimize the delay to within two LCUs.

The following describes the parallel processing of encoding and transmitting slices as shown in Figure 11. A processor core or two processing units, such as a processor, simultaneously encode the first slice of each row (slice 371 and slice 373). When encoding is complete, encoded slices 371 and 373 are encapsulated into packets with packet numbers (packet_id) 0 and 4, respectively. Here, packet number 4 is selected to ensure smaller numbers for slice 372 and possible additional NALUs.

When encoding of slice 372 is completed, slice 372 is encapsulated and transmitted as a packet of packet number 1. In addition, two NAL units with corresponding packet numbers 2 and 3 and dummy (padding) data are generated so that missing packet numbers 2 and 3 are not judged as missing packets.

In HEVC, this is achieved by using the filler_data SEI message or a specific NAL unit type that is guaranteed to be used for filler data. In this way, when the packet ID needs to be increased one by one for each NAL unit, a filler NALU is used to fill the difference.

The initialization of the target row depends on the second LCU in the row above it. Furthermore, if slices are inserted after the second LCU, this can affect the decision making for CABAC initialization, causing a problem. Based on this analysis and problem, the present invention provides a method for deriving a more efficient relationship between WPP and the use of dependent slices. To maintain WPP efficiency, it is important to avoid situations where the CABAC initialization of one row depends on another row.

An image decoding method according to one embodiment of the present invention decodes a bit stream including a coded signal, wherein the coded signal is a signal obtained by dividing an image into a plurality of slices and encoding each of the plurality of slices, wherein each slice includes a plurality of coding units. The image decoding method includes a decoding step of decoding the coded signal, wherein the plurality of slices are either normal slices or dependent slices, wherein the normal slice is a slice in which information included in a slice header is used in another slice, and the dependent slice is a slice in which information included in a slice header of another slice is used during decoding. The image includes a plurality of rows, each of which includes a plurality of coding units. When the normal slice starts at a position other than the beginning of the first row, the second row following the first row does not start at a dependent slice.

This can prevent the first slice in the second row from referring to slices other than the first slice in the first row, thereby improving efficiency when utilizing both parallel processing and dependent slices.

For example, in the decoding step, the first and second lines may be decoded in parallel, and when decoding of the second line is started, the second line may be decoded without referring to division information indicating a slice structure in the first line.

For example, in the decoding step, arithmetic decoding of the second line may be initialized using a context obtained by arithmetically decoding the second coding unit of the first line.

For example, the image decoding method may further include a step of acquiring information indicating whether the slice is a normal slice or a dependent slice from a slice header.

For example, the leading slice of the image may be a normal slice, and all other slices may be dependent slices.

For example, each slice may include all of one or more rows.

For example, arithmetic decoding of the dependent slice may be initialized using a context of a parent slice whose slice header is used in the dependent slice.

For example, the image decoding method may further include a step of acquiring, from the bitstream, a restriction indicator indicating that partitioning of the picture is restricted when the dependent slice is valid.

Furthermore, an image coding method according to one embodiment of the present invention generates a bit stream by dividing an image into a plurality of slices and encoding the slices, wherein each slice includes a plurality of coding units. The image coding method includes: a dividing step of dividing the image into the plurality of slices; and an encoding step of encoding the plurality of divided slices, wherein the plurality of slices are either normal slices or dependent slices, the normal slice being a slice in which information included in a slice header is used in another slice, and the dependent slice being a slice in which information included in a slice header of another slice is used during decoding. The image includes a plurality of rows, each of which includes a plurality of coding units. In the dividing step, the image is divided into the plurality of slices so that, when the normal slice starts at a position other than the beginning of the first row, the second row following the first row does not start at a dependent slice.

This can prevent the first slice in the second row from referring to slices other than the first slice in the first row, thereby improving efficiency when utilizing both parallel processing and dependent slices.

For example, in the segmentation step, the image may be segmented into the plurality of slices so that, when the first row and the second row are decoded in parallel in the image decoding device, the second row can be decoded without referring to the segmentation information indicating the slice structure of the first row when the image decoding device starts decoding the second row.

For example, in the encoding step, the arithmetic coding of the second line may be initialized using a context in which the second coding unit of the first line is arithmetically coded.

For example, the image encoding method may further include the step of embedding information indicating whether the slice is a normal slice or a dependent slice in the slice header.

For example, the leading slice of the image may be a normal slice, and all other slices may be dependent slices.

For example, each slice may include all of one or more rows.

For example, the arithmetic coding of the dependent slice may be initialized using the context of the parent slice used in the dependent slice by using the slice header.

For example, the image encoding method may further include a step of embedding a restriction indicator indicating that partitioning of a picture is restricted in the bitstream when the dependent slice is valid.

Furthermore, an image decoding device according to one embodiment of the present invention decodes a bit stream including a coded signal, wherein the coded signal is a signal obtained by dividing an image into a plurality of slices and encoding each of the plurality of slices, wherein each slice includes a plurality of coding units. The image decoding device includes a decoding unit that decodes the coded signal, wherein the plurality of slices are either normal slices or dependent slices, wherein the normal slice is a slice in which information included in a slice header is used in another slice, and the dependent slice is a slice in which information included in a slice header of another slice is used during decoding. The image includes a plurality of rows, each of the plurality of rows includes a plurality of coding units, and when the normal slice starts at a position other than the beginning of the first row, the second row following the first row does not start at a dependent slice.

This can prevent the first slice of the second row from referring to slices other than the first slice of the first row, thereby making it possible to utilize both parallel processing and efficiency when interoperating with dependent slices.

Furthermore, an image coding device according to one embodiment of the present invention generates a bit stream by dividing an image into a plurality of slices and encoding the slices, wherein each of the plurality of slices includes a plurality of coding units. The image coding device includes: a dividing unit that divides the image into the plurality of slices; and an encoding unit that encodes the plurality of divided slices, wherein the plurality of slices are either normal slices or dependent slices, wherein the normal slice is a slice in which information included in a slice header is used in another slice, and the dependent slice is a slice in which information included in a slice header of another slice is used during decoding. The image includes a plurality of rows, each of which includes a plurality of coding units. The dividing unit divides the image into the plurality of slices so that, when a normal slice starts at a position other than the beginning of a first row, a second row to be encoded subsequent to the first row does not start at a dependent slice.

This can prevent the first slice in the second row from referring to slices other than the first slice in the first row, thereby improving efficiency when utilizing both parallel processing and dependent slices.

Furthermore, an image coding and decoding device according to one aspect of the present invention includes the image coding device and the image decoding device.

In addition, the embodiments described below are all specific examples of the present invention. The numerical values, shapes, materials, components, configuration positions and connection methods of the components, steps, and the order of the steps shown in the following embodiments are examples and are not intended to limit the scope of the invention. In addition, regarding the components of the following embodiments that are not described in the independent claims representing the superordinate concepts, they are described as arbitrary components.

(Implementation Method 1)

In the image encoding method and the image decoding method according to the first embodiment, an indicator that explicitly indicates the initialization of CABAC is added.

12 is a diagram showing the syntax of a slice header according to Embodiment 1. A slice header 380 includes a new line 381 having a new syntax element "entropy_default_initialization_flag".

The entropy_default_initialization_flag, when set to a specified value, indicates that the slice's CABAC is initialized to the CABAC default (prescribed) value. This flag is a one-bit indicator with a first value, such as "1," indicating that the slice is initialized to the default CABAC value, and a second value, such as "0," indicating that the slice is initialized using a different method. The values "1" and "0" can be interchanged.

The "other method" for determining initialization may also be a method based on a predetermined method such as initialization of the value of the preceding slice. However, the "other method" may also include another determination process similar to the process shown in FIG9 , thereby possibly also deriving an initialization method using the default CABAC value.

An image decoding device according to this embodiment decodes a bitstream of a coded video sequence including image slices at least partially arithmetically coded. The image decoding device comprises: an analyzing unit that extracts an initialization indicator from the bitstream data of the slices, indicating whether a probability model for arithmetic decoding of the slices should be initialized with a predetermined value; a control unit that controls whether the probability model for arithmetic decoding should be initialized with the predetermined value according to the initialization indicator; and an arithmetic decoding unit that decodes the slices by performing arithmetic decoding.

For example, arithmetic coding may be context-adaptive arithmetic coding as defined by HEVC, but the present invention is not limited thereto.

The so-called predetermined value is a default value known by the image coding apparatus and the image decoding apparatus, and does not change depending on the content being coded.

The initialization indicator preferably refers to a 1-bit flag, in which "1" indicates that the probability model of arithmetic decoding is initialized with a specified value, and "0" indicates that the probability model of arithmetic decoding is initialized with another method.

This indicator is required only when the target slice is a dependent slice. This is because, in the case of normal slices, the CABAC default value is used for initialization (see the case of "No" in S101 of FIG9 ). Therefore, by confirming the condition "dependent_slice_flag == 1", it is first analyzed whether the target slice is a dependent slice.

Furthermore, an initialization indicator (flag) is advantageous when performing parallel processing of a slice with other slices. For example, parallel processing can also be WPP. Therefore, the syntax of the slice header shown in FIG12 includes the initialization indicator entropy_default_initialization_flag only when the condition entropy_coding_sync_enabled_flag == 1 is true.

Furthermore, the initialization indicator is only appropriate when the slice begins at the beginning of an LCU row. This is because parallel processing is only possible at this point, so immediate initialization of CABAC is required. This is expressed in the syntax shown in Figure 12 by the condition slice_address % PicWidthInCtbsY == 0.

As described above, the syntax element "slice_address" indicates the start of a slice using an offset included in the bitstream. "PicWidthInCtbsY" indicates the width of a frame using the number of coding tree block units (LCUs).

The logical product of the three conditions mentioned above is used in the determination as shown in line 381. That is, entropy_default_initialization_flag is passed to explicitly signal the initialization method only when the following (Equation 2) is true.

dependent_slice_flag==1 && entropy_coding_sync_enabled_flag==1 &&slice_address % PicWidthInCtbsY==0 ･･･ (Formula 2)

When (Equation 2) is not true, initialization is performed based on a common method, that is, based on the WPP rule.

Specifically, the image encoding method and image decoding method according to this embodiment divide image slices into coding units corresponding to pixel blocks of the image. A parser extracts the initialization indicator included in the header data only when the slice is a dependent slice. The arithmetic decoding unit of the dependent slice is initialized based on the context of the arithmetic decoding unit of the parent slice corresponding to each dependent slice.

Alternatively, the analysis unit may extract the initialization indicator of the header data only when parallel decoding of the lines formed by the coding unit is possible.

That is, according to this embodiment, a slice of an image is divided into coding units corresponding to pixel blocks of the image, and the analysis unit extracts the initialization indicator of the header data only when the slice starts at the beginning of a line consisting of coding unit blocks of the image.

FIG13 is a flowchart of a method for determining CABAC initialization for slices according to this embodiment. FIG13 assumes the case of picture 350 shown in FIG8 . If it is assumed that slice (4) (slice 354) and slice (1) (slice 351) are analyzed in parallel, the following determination is made.

First, it is determined whether slice (4) is a dependent slice (S111). If slice (4) is a dependent slice and satisfies the other conditions (row parallel processing is performed and the slice starts at the beginning of the LCU row) ("Yes" in S111), the initialization indicator "entropy_default_initialization_flag" is set to determine the initialization execution method (S112).

If entropy_default_initialization_flag indicates that default initialization is used (No in S112), default initialization is used (S114). On the other hand, if entropy_default_initialization_flag indicates that default initialization is not used (Yes in S112), initialization of the WPP that refers to the previous slice is used (S113).

In addition, this embodiment is not limited to signaling the initialization indicator in the slice header, and the same indicator may be embedded in another data structure, such as an additional extension information message.

(Implementation Method 2)

Embodiment 1 enables efficient parallel LCU row processing for WPP and dependent slices. Furthermore, new syntax elements are incorporated into the slice header. To avoid the addition of new syntax elements, the initialization rules can be modified to achieve independence of CABAC initialization for slices being processed in parallel.

In this embodiment, the definition of dependent slices and the operations of the image encoding and decoding devices for dependent slices are modified. This can be achieved by restricting the bitstream standard.

Specifically, an image decoding device according to this embodiment decodes a bitstream of a coded video sequence comprising a plurality of image slices that are divided into a plurality of coding units and at least partially arithmetically coded. The image decoding device includes a parsing unit that extracts the first and second rows of coding units from the bitstream. The coding units of the first and second rows are assigned to the slices so that when the arithmetic decoding unit for the second slice of the second row is initialized, the partitioning information of the first slice of the first row is not referenced. The starting position of the first slice of the first row is a predetermined number of coding units later than the second slice of the second row. The image decoding device further includes an arithmetic decoding unit that decodes the first and second slices by at least partially performing arithmetic decoding on the first and second slices in parallel.

14 is a diagram illustrating the functions of this embodiment, showing a picture 390 divided into a plurality of slices. The picture 390 includes four slices: a normal slice 391 , a normal slice 392 , a dependent slice 393 , and a normal slice 394 .

Three slices 391, 392, and 393 are included in the first row of the coding unit (LCU). Slice 394 includes the entire second and third rows.

The first example of restrictions on the use of slicing and parallel processing of rows is that "when entropy_code_sync_enabled_flag is equal to 1 and dependent_slice_enabled_flag is equal to 1, a slice can usually only start at the beginning of a coding tree block row." In addition, the flags of both entropy_code_sync_enabled_flag and dependent_slice_enabled_flag are included in the picture parameter set. In addition, the coding tree block (CTB) and the largest coding unit (LCU) are the same unit. The standard text (see non-patent document 3) uses CTB. In addition, the previous version of the standard text used LCU, but the current version uses CTB.

When a normal slice begins only at the beginning of a coding unit line (LCU line), dependent slices following that normal slice in other lines can always refer to the CABAC state of the normal slice. Here, the CABAC state refers to the CABAC state after WPP processing of the first LCU or the first two LCUs. Furthermore, since the headers of dependent slices depend on the headers of the preceding normal slices, if normal slice 394 is lost, the dependent slices must be discarded.

Thus, in the first example restriction, a normal slice always begins at the beginning of an LCU row. In other words, the first slice of an LCU row is a normal slice, and the remaining slices are dependent slices. In other words, a normal slice is only identified as the first slice of an LCU. Furthermore, slices other than the first slice of an LCU row are always dependent slices.

The restriction in the first example above does not need to be strict. In order to enable the initial application of WPP, it is sufficient that at least one or two LCUs of the previous normal slice can be used by the dependent slice.

Alternatively, the second example can be applied as another restriction (rule). In the second example, a normal slice does not begin after the second coding tree block in a coding tree block row. Since a normal slice must begin at the beginning of an LCU row, for example, as shown in FIG14 , the second slice 392 is not allowed to be set as a normal slice.

The first slice may start before the second coding unit in the first row. Alternatively, the first slice may be a normal slice, and the second slice may be a dependent slice using the normal slice header. Alternatively, the first slice may start at the beginning of the first row.

Fig. 15 is a flowchart of the process of determining the CABAC initialization method when the above-mentioned rule is set. Here, the example shown in Fig. 8 is used for explanation.

First, it is determined whether slice (4) is a dependent slice (S111). If slice (4) is a dependent slice ("Yes" in S111), WPP is initialized (S113). On the other hand, if slice (4) is not a dependent slice ("No" in S111), default initialization is performed (S114).

Thus, the image coding method according to this embodiment uses a context-adaptive entropy coding unit and is applied to a picture frame divided into at least two parts. The at least two parts are a first part and a second part that can be at least partially encoded and decoded in parallel.

According to this embodiment, when the first portion of a substream is divided into slices, the context table initialization for the second portion of the stream is determined using a method that is independent of the division of the first portion. For example, since WPP is performed on a row-by-row basis (on an LCU row-by-LCU row basis), a portion of the stream may correspond to an LCU row.

The present invention is not limited to the limitations exemplified above. Alternatively, the limitations described above may be formulated in a different manner. Another example of limitations is described below.

When a normal slice satisfies the following condition (Equation 3), a slice starting at the head of a subsequent LCU row is not a dependent slice.

slice_adress % PicWidthInCtbsY>1 ･･･(Formula 3)

For simplification, the above conditions can also be expressed as the following (Formula 4).

slice_adress % PicWidthInCtbsY !=0 ･･･(Formula 4)

Here, "!=" indicates inequality. These restrictions apply when entropy_coding_sync_enabled_flag is equal to 1, that is, when parallel processing of LCU rows is enabled. Furthermore, "slice_address" indicates the position of the slice starting within the bitstream, and the parameter "PicWidthInCtbsY" indicates the width of the picture (frame) within the LCU (coding tree block).

That is, if a normal slice does not start at the beginning of a line, the slice starting in the line immediately following it is not a dependent slice (Case 3). This condition eliminates the need to wait for decoding of the slice in the second line before parsing (decoding) a normal slice at a certain position in the first line.

That is, if a normal slice starts at a point other than the beginning of the first row, the second row following the first row does not start with a dependent slice. In other words, if at least one of the second or subsequent slices in the first row is a normal slice, the first slice in the second row is a normal slice.

The influence of the restriction in the third example will be described using Fig. 16. The picture 400 shown in Fig. 16 includes three slices 401 to 403 included in the first row. Of these three slices, the first two slices 401 and 402 are normal slices, and the third slice 403 is a dependent slice.

According to the above conditions, the fourth slice 404 cannot be set as a dependent slice. In FIG16 , this is indicated by a cross mark being added to the slice 404 .

Thus, the bitstream may also contain normal slices and dependent slices, the decoding of which is based on parameters signaled in the slice header of the normal slice. In the case where the normal slice starts after the beginning of the LCU row, the next LCU row does not start with a dependent slice.

In addition, a specific example is described using Figures 17A to 17D. For example, as shown in Figure 17A, if there is a normal slice (3) in the first row, the first slice (4) in the second row cannot be set as a dependent slice. In addition, if at least one of slices (2) and (3) is a normal slice, slice (4) cannot be set as a dependent slice. Therefore, as shown in Figure 17B, slice (4) needs to be set as a normal slice. In addition, in the third example, pictures such as those shown in Figures 17C and 17D are also recognized.

In the first example, the images shown in Figures 17A, 17B, and 17D are not recognized, but the image shown in Figure 17C is recognized. In the second example, the images shown in Figures 17A and 17B are not recognized, but the images shown in Figures 17C and 17D are recognized.

The restriction of the fourth example will be described using Fig. 18. When entropy_coding_sync_enabled_flag is equal to 1 and dependent_slice_enabled_flag is equal to 1, normal slices other than the first slice in the frame are not recognized (fourth example).

Specifically, when parallel processing is possible and dependent slices are enabled, only the normal slice is recognized as the first slice in the frame. That is, all slices in the frame, except the first slice, are dependent slices. In other words, the first slice of the image is the normal slice, and all other slices are dependent slices.

Picture 410 shown in Figure 18 includes five slices 411 to 415. Slices 411, 412, and 415 are normal slices, while slices 413 and 414 are dependent slices. Due to the restrictions in Example 4 above, normal slices 412 and 415 other than the first normal slice 411 are not recognized. In other words, slices 412 and 415 must be dependent slices. Furthermore, in Example 4, only the picture shown in Figure 17D is visible among the pictures shown in Figures 17A to 17D.

Furthermore, using the fourth example restriction has drawbacks in robustness against packet loss. Normal slices are generally used to reduce dependency and error propagation in lossy environments. Furthermore, in frames where only the first slice is a normal slice, there is a risk that all slices will not be decoded if the first slice cannot be decoded.

As another restriction, the following restriction may be applied: If a slice (normal or dependent) starts in the middle of an LCU row (i.e., at a position different from the start of the row), the next coding unit row does not start in the dependent slice (Case 5).

Furthermore, as those skilled in the art will appreciate, the plurality of restrictions described herein can be arbitrarily combined. In other words, the first to fifth examples described above can be combined and applied.

Furthermore, another example of a restriction is given below. When entropy_coding_sync_enabled_flag is equal to 1, a single LCU line cannot be split into slices (Example 6). When this restriction is applied, slices 412 and 413 are not recognized in the slice structure shown in Figure 18. In other words, when parallel processing of coding unit lines is possible, slices are only recognized to include one entire coding unit line or multiple entire coding unit lines.

Thus, the bitstream contains normal slices and dependent slices. The decoding of normal slices and dependent slices is based on parameters signaled by the slice header of the normal slice. After only the first slice in the image is used as a normal slice and the remaining slices are used as dependent slices, the image is divided into slices.

Furthermore, each slice includes all m coding unit lines. Here, m is an integer greater than or equal to 1. That is, each slice includes all one or more lines.

Furthermore, in addition to or in lieu of the application of the above restrictions, when any of the dependency slices, WPPs, and tiles are valid, an indicator indicating the above restrictions may be embedded in the bitstream. For example, the indicator may be embedded in the SPS or PPS. Alternatively, the indicator may be embedded in another message such as an SEI message or any video usability information (VUI) message.

Based on this indicator, the image decoding device understands the restrictions that apply. For example, this restriction is to only accept normal slices at the beginning of an LCU row (WPP) or the beginning of a tile. This is merely an example of a restriction; any of the above restrictions, a combination of these restrictions, or additional restrictions not explicitly stated may also apply.

For example, the indicator may be a one-bit flag indicating whether a specified constraint applies. Furthermore, multiple selectable constraints may be provided, and information indicating the selected constraint may be included in the bitstream and signaled to the image decoding device. In other words, rather than explicitly restricting usage as in the above example, the image decoding device may be informed of the constraint being applied by the image encoding device. Thus, any of the above examples of constraints may be applied.

Thus, the image decoding method according to one aspect of the present invention includes the step of obtaining a restriction indicator indicating that picture segmentation is restricted from a bitstream when a dependent slice is enabled. Furthermore, the image coding method according to one aspect of the present invention includes the step of embedding the restriction indicator indicating that picture segmentation is restricted into the bitstream when a dependent slice is enabled.

In addition, whether to add an indicator may not be determined based on whether the WPP, tile, or dependent slice is enabled.

Furthermore, an image decoding method according to one aspect of the present invention is an image decoding method for decoding a bitstream containing a coded signal obtained by dividing an image into a plurality of slices, each containing a plurality of coding units (LCUs), and including a decoding step for decoding the coded signal. Furthermore, an image coding method according to one aspect of the present invention is an image coding method for generating a bitstream by dividing an image into a plurality of slices, each containing a plurality of coding units (LCUs), and encoding the slices. The method includes a dividing step for dividing the image into a plurality of slices and an encoding step for encoding the plurality of slices.

Furthermore, the multiple slices are each a normal slice and a dependent slice. A normal slice is a slice whose information contained in its slice header may be used in another slice. A dependent slice is a slice whose decoding uses information contained in the slice header of another slice. Here, the so-called other slice is, for example, a normal slice that is located immediately before and closest to the dependent slice.

Furthermore, in the decoding step, arithmetic decoding of the dependent slice is initialized using the context of the mother slice in which the slice header is used in the dependent slice. Furthermore, in the encoding step, arithmetic coding of the dependent slice is initialized using the context of the mother slice in which the slice header is used in the dependent slice.

Furthermore, the image includes a plurality of lines, each of which includes a plurality of coding units.

Furthermore, in the segmentation step, the image is segmented into a plurality of tiles and into a plurality of slices to satisfy one or more of the above constraints.

Furthermore, in the decoding step, the first and second lines may be decoded in parallel, and when decoding of the second line is started, the second line may be decoded without referring to the partitioning information indicating the slice structure of the first line. Furthermore, in the partitioning step, the image may be partitioned into a plurality of slices so that when the image decoding device decodes the first and second lines in parallel, the image decoding device can decode the second line without referring to the partitioning information indicating the slice structure of the first line when decoding of the second line is started.

Here, the partition information is information indicating, for example, the position of a slice (the beginning position) or the position of a slice header. The image decoding apparatus determines the processing of the CABAC initialization method described above simply by referring to the partition information.

Furthermore, the so-called parallel decoding is, for example, the aforementioned WPP. Specifically, in the decoding step, arithmetic decoding of the second line is initialized using the context obtained by arithmetically decoding the second coding unit of the first line. Furthermore, in the encoding step, arithmetic coding of the second line is initialized using the context obtained by arithmetically encoding the second coding unit of the first line.

Furthermore, as described above, the slice header includes information indicating whether the slice is a normal slice or a dependent slice (dependent_slice_flag). Specifically, the image decoding method includes the step of obtaining information indicating whether the slice is a normal slice or a dependent slice from the slice header. Furthermore, the image encoding method includes the step of embedding information indicating whether the slice is a normal slice or a dependent slice within the slice header.

As described above, this embodiment can prevent slice-dependent processing from being delayed by more than two or three coding units by taking CABAC initialization of preceding slices into account during parallel processing. This allows efficient parallel processing of line encoding, decoding, and parsing.

The present invention is not limited to the implementation of the method for limiting slicing. In addition, the above-mentioned limitation may be associated with the slices for which the CABAC context can be obtained.

(Implementation Method 3)

In this embodiment, the CABAC initialization method for dependent slices during WPP processing is changed. Specifically, the allocation rule for parent slices of dependent slices is changed.

For example, a rule is set such that, regardless of the partitioning of the slice into LCU rows (and/or the type of the subsequent slice), dependent slices always obtain the slice header and CABAC context from the same slice.

The picture 420 shown in FIG19 includes slices 421 to 424. In the current HEVC, the parent slice of the dependent slice 424 is the slice 422. That is, the slice header of the dependent slice 424 is obtained from the slice 422 which is the preceding and nearest normal slice.

As described using FIG9 , CABAC initialization may be performed for dependent slices using the normal slice at the beginning of the preceding LCU line. However, if slice 422 is lost, CABAC initialization can be performed for slice 424. However, since the slice header information is missing, slice 424 cannot be decoded.

In contrast, in this embodiment, a dependent slice has the nearest normal slice starting from the same row as or before the row containing the dependent slice as its parent slice. In accordance with this rule, in this embodiment, as shown in FIG19 , the parent slice of slice 424 is set to slice 421. Furthermore, dependent slice 424 uses the slice header of slice 421 and the CABAC state of slice 421 for CABAC initialization.

Furthermore, the slice dependency is set so that the arithmetic decoding unit of each dependent slice is initialized based on the context of the arithmetic decoding unit of the parent slice.

Furthermore, information indicating the CABAC context table used in slice initialization may be explicitly signaled in the SEI message. In other words, all initial values that are expected to be used in initializing the CABAC engine may be explicitly signaled in the SEI message.

In addition, the "slice (normal slice or dependent slice)" used in the above description may also be called a "slice segment (normal slice segment or dependent slice segment)". In this case, a unit including one or more continuous slice segments is called a "slice". Specifically, one slice includes one normal slice segment and one or more continuous dependent slice segments following the normal slice segment. That is, when a normal slice segment is followed by a normal slice segment, the slice only includes the normal slice segment. In addition, when one or more continuous dependent slice segments follow a normal slice segment, the slice includes the normal slice segment and the one or more dependent slice segments. That is, the normal slice segment to the dependent slice segment immediately before the next normal slice segment are included in one slice.

In addition, when such a definition is used, the third example of the above-mentioned restriction of LCU rows and slices can be said to correspond to the following definition.

When entropy_coding_sync_enabled_flag is equal to 1 and the first coded tree block (LCU) contained in the slice is not the first coded tree block of a coded tree block row, the standard for the bitstream requires that the last coded tree block contained in the above slice belongs to the same coded tree block row as the first coded tree block contained in the slice.

Here, the case where the first CTB in a slice is not the first CTB in a CTB row refers to the case where a slice segment normally starts at a position other than the beginning of a CTB row. Furthermore, the case where the last CTB in a slice belongs to the same CTB row as the first CTB in the slice corresponds to the case where the next row does not start in a dependent slice.

For example, in the example shown in FIG17B , one slice is composed of slice segments (1) and (2) (recorded as slice (1) and slice (2) in FIG17B . The same description is used below), one slice is composed of slice segment (3), and one slice is composed of slice segments (4) and (5). A slice in which the first coding tree block of a slice is not the first coding tree block of a coding tree block row is a slice composed only of slice segment (3). The last coding tree block of the slice belongs to the same coding tree block row (row 1) as the first coding tree block of the slice. Therefore, the structure shown in FIG17B is recognized.

On the other hand, in the example shown in FIG17A , a slice is composed of slice segments (3) to (5). The first coding tree block of the slice (the first coding tree block of slice segment (3)) and the last coding tree block of the slice (the last coding tree block of slice segment (5)) belong to different coding tree block rows. Therefore, the structure shown in FIG17A is not recognized.

Furthermore, in FIG17C , slice segments (1) to (3) constitute one slice, and slice segments (4) to (5) constitute one slice. Furthermore, in FIG17D , slice segments (1) to (5) constitute one slice. That is, in FIG17C and FIG17D , there is no slice whose first coding tree block is not the first coding tree block in a coding tree block row, that is, there is no slice starting midway in a row. Therefore, the structures shown in FIG17C and FIG17D are recognized.

In addition, when entropy_coding_sync_enabled_flag is equal to 1 and the first coded tree block (LCU) contained in the slice segment is not the first coded tree block of a coded tree block row, the standard for the bitstream requires that the last coded tree block contained in the above slice segment belongs to the same coded tree block row as the first coded tree block contained in the slice segment.

As mentioned above, the image encoding method and the image decoding method according to the embodiments have been described, but the present invention is not limited to these embodiments.

The above-described image encoding method and image decoding method are implemented by an image encoding device and an image decoding device. The structures of the image encoding device and the image decoding device are, for example, the same as those shown in Figures 1 and 2 , and the characteristic steps included in the above-described image encoding method and image decoding method are performed by a processing unit shown in Figures 1 and 2 or a processing unit not shown.

Furthermore, the processing units included in the image encoding apparatus and the image decoding apparatus according to the above-described embodiments are typically implemented as integrated circuits (LSIs). These units may be implemented individually on a single chip, or partially or entirely on a single chip.

Furthermore, integrated circuits are not limited to LSIs and can also be implemented using dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI fabrication, or reconfigurable processors that can reconfigure the connections and settings of circuit cells within the LSI, can also be used.

In each of the above embodiments, each component may be formed of dedicated hardware or implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

In other words, the image encoding and decoding devices include a control circuit and a storage device electrically connected to (and accessible from) the control circuit. The control circuit includes at least one of dedicated hardware and a program execution unit. Furthermore, if the control circuit includes a program execution unit, the storage device stores a software program executed by the program execution unit.

Furthermore, the present invention may be the software program or a non-transitory computer-readable recording medium recording the program. Furthermore, the program may be distributed via a transmission medium such as the Internet.

In addition, all the numbers used in the above description are given as examples in order to specifically describe the present invention, and the present invention is not limited to the exemplified numbers.

Furthermore, the division of functional blocks in the block diagram is merely an example. It is also possible to implement multiple functional blocks as a single functional block, to divide a single functional block into multiple blocks, or to transfer some functions to other functional blocks. Furthermore, it is also possible to process the functions of multiple functional blocks having similar functions in parallel or in a time-sharing manner using a single piece of hardware or software.

The order in which the steps included in the above-described image encoding method or image decoding method are executed is provided as an example for the purpose of specifically explaining the present invention, and an order other than the above-described order may be employed. Furthermore, some of the above-described steps may be executed simultaneously (in parallel) with other steps.

While the image encoding and decoding devices according to one or more embodiments of the present invention have been described above based on embodiments, the present invention is not limited to these embodiments. Various modifications conceived by those skilled in the art to these embodiments, or combinations of components from different embodiments, may also fall within the scope of one or more embodiments of the present invention, without departing from the spirit of the present invention.

(Implementation Method 4)

By recording a program for implementing the structure of the moving picture encoding method (image encoding method) or moving picture decoding method (image decoding method) described in each of the above embodiments onto a storage medium, the processing described in each of the above embodiments can be easily implemented on a standalone computer system. The storage medium may be a magnetic disk, an optical disk, a magneto-optical disk, an IC card, a semiconductor memory, or any other medium capable of recording a program.

Furthermore, here, we will describe an application example of the moving picture encoding method (image encoding method) and moving picture decoding method (image decoding method) described in the above embodiments, and a system using the same. This system is characterized by including an image encoding and decoding device comprising an image encoding device using the image encoding method and an image decoding device using the image decoding method. Other system configurations may be modified as appropriate depending on the circumstances.

Fig. 20 shows the overall configuration of a content providing system ex100 that implements content distribution services. The area where communication services are provided is divided into cells of desired sizes, and base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations, are installed in each cell.

The content providing system ex100 is connected to the Internet ex101 via the Internet service provider ex102, the telephone network ex104, and base stations ex107 to ex110 to various devices such as a computer ex111, a PDA (Personal Digital Assistant) ex112, a camera ex113, a mobile phone ex114, and a game console ex115.

However, the content delivery system ex100 is not limited to the configuration shown in FIG20 ; certain components may be combined and connected. Furthermore, each device may be directly connected to the telephone network ex104 without intermediary of the fixed wireless base stations ex107 to ex110. Furthermore, each device may be directly connected to another via short-range wireless communication or the like.

The camera ex113 is a device capable of capturing moving images, such as a digital video camera. The camera ex116 is a device capable of capturing both still and moving images, such as a digital camera. Furthermore, the mobile phone ex114 may be any type of mobile phone, such as a GSM (Global System for Mobile Communications) phone, a CDMA (Code Division Multiple Access) phone, a W-CDMA (Wideband-Code Division Multiple Access) phone, a LTE (Long Term Evolution) phone, a HSPA (High Speed Packet Access) phone, or a PHS (Personal Handyphone System) phone.

In the content delivery system ex100, live broadcasts and other content can be performed by connecting a camera ex113 or other device to the streaming server ex103 via the base station ex109 and the telephone network ex104. During live broadcasts, content captured by a user using the camera ex113 (e.g., footage of a concert) is encoded as described in the above embodiments (functioning as an image encoding device according to one embodiment of the present invention) and transmitted to the streaming server ex103. The streaming server ex103 streams the transmitted content data to requesting clients. Clients include the computer ex111, PDA ex112, camera ex113, mobile phone ex114, and game console ex115, all capable of decoding the encoded data. Each device that receives the distributed data decodes and reproduces the data (functioning as an image decoding device according to one embodiment of the present invention).

Furthermore, the encoding of captured data can be performed by either the camera ex113 or the streaming server ex103, which performs data transmission, or they can be shared. Similarly, the decoding of distributed data can be performed by either the client or the streaming server ex103, or they can be shared. Furthermore, this is not limited to the camera ex113; still image and/or moving image data captured by the camera ex116 can also be transmitted to the streaming server ex103 via the computer ex111. In this case, the encoding process can be performed by any of the camera ex116, the computer ex111, or the streaming server ex103, or they can be shared.

These encoding and decoding processes are typically handled by the LSI ex500 included in the computer ex111 or various devices. The LSI ex500 can be a single chip or a structure composed of multiple chips. Alternatively, video encoding and decoding software can be installed on a recording medium (such as a CD-ROM, floppy disk, or hard disk) readable by the computer ex111 or other devices, and the encoding and decoding processes can be performed using this software. Furthermore, if the mobile phone ex114 is equipped with a camera, video data captured by the camera can be transmitted. In this case, the video data is encoded by the LSI ex500 included in the mobile phone ex114.

Alternatively, the streaming server ex103 may be a plurality of servers or a plurality of computers, and may process, record, and distribute data in a distributed manner.

As described above, in the content delivery system ex100, clients can receive and play back encoded data. Thus, in the content delivery system ex100, clients can receive, decode, and play back information sent by users in real time, enabling personal broadcasting even for users without special rights or equipment.

Furthermore, the example is not limited to the content delivery system ex100. As shown in FIG21 , at least one of the moving picture encoding device (image encoding device) or moving picture decoding device (image decoding device) described in the above embodiments can also be incorporated into a digital broadcasting system ex200. Specifically, at a broadcasting station ex201, multiplexed data, obtained by multiplexing music data onto video data, is transmitted via radio waves to a communication or broadcasting satellite ex202. This video data is encoded using the moving picture encoding methods described in the above embodiments (i.e., data encoded by the image encoding device according to one embodiment of the present invention). The broadcasting satellite ex202, receiving this data, emits broadcast radio waves. A household antenna ex204 capable of receiving satellite broadcasts receives these radio waves, and a device such as a television (receiver) ex300 or a set-top box (STB) ex217 decodes and reproduces the received multiplexed data (i.e., functions as an image decoding device according to one embodiment of the present invention).

Alternatively, the moving image decoding device or moving image encoding device described in the above embodiments can be installed in a reader/recorder ex218 that reads and decodes multiplexed data recorded on a recording medium ex215, such as a DVD or BD, or encodes video data and then multiplexes it with a music signal, as appropriate, before writing it to the recording medium ex215. In this case, the reproduced video signal can be displayed on a monitor ex219, and the video signal can be reproduced in another device or system via the recording medium ex215 on which the multiplexed data is recorded. Alternatively, the moving image decoding device can be installed in a set-top box ex217 connected to a cable TV cable ex203 or a satellite/terrestrial broadcast antenna ex204, and displayed on a television monitor ex219. In this case, the moving image decoding device can be installed in a television set instead of in a set-top box.

FIG22 shows a television (receiver) ex300 that uses the moving picture decoding method and moving picture encoding method described in the above embodiments. The television ex300 includes a tuner ex301 that receives or outputs multiplexed data in which audio data is multiplexed with video data via an antenna ex204 or a cable ex203 that receives the broadcast, a modulation/demodulation unit ex302 that demodulates or modulates the received multiplexed data into coded data for external transmission, and a multiplexing/demultiplexing unit ex303 that separates the demodulated multiplexed data into video data and audio data, or multiplexes the video data and audio data coded by a signal processing unit ex306.

The television ex300 also includes a signal processing unit ex306 including an audio signal processing unit ex304 and a video signal processing unit ex305 that decode audio data and video data, respectively, or encode their respective information (i.e., functioning as an image encoding device or image decoding device according to one embodiment of the present invention); an output unit ex309 including a speaker ex307 that outputs decoded audio signals and a display unit ex308 such as a monitor that displays decoded video signals. Furthermore, the television ex300 includes an interface unit ex317 including an operation input unit ex312 that receives user input. Furthermore, the television ex300 includes a control unit ex310 that collectively controls various components and a power supply circuit unit ex311 that supplies power to various components. The interface unit ex317 may also include, in addition to the operation input unit ex312, a bridge unit ex313 for connecting to external devices such as a reader/writer ex218, a slot unit ex314 for inserting a recording medium ex216 such as an SD card, a drive ex315 for connecting to an external recording medium such as a hard disk, and a modem ex316 for connecting to the telephone network. The recording medium ex216 is capable of electrically recording information using a nonvolatile/volatile semiconductor memory element. The various components of the television ex300 are interconnected via a synchronous bus.

First, the structure of the television ex300 that decodes and reproduces multiplexed data received from the outside via an antenna ex204 or other means will be described. The television ex300 receives user operations from a remote control ex220 or other means. Under the control of a control unit ex310, which includes a CPU or other means, the multiplexed data demodulated by the modulation/demodulation unit ex302 is demultiplexed by the multiplexing/demultiplexing unit ex303. Furthermore, the television ex300 decodes the demultiplexed audio data by the audio signal processing unit ex304 and decodes the demultiplexed video data by the video signal processing unit ex305 using the decoding methods described in the above embodiments. The decoded audio and video signals are output externally from the output unit ex309, respectively. During output, these signals can be temporarily stored in buffers ex318 and ex319, etc., to synchronize the audio and video signals for playback. Alternatively, the television ex300 can read the encoded multiplexed data from recording media ex215 and ex216, such as magnetic or optical disks or SD cards, rather than from a broadcast source. Next, the structure of the television ex300 that encodes audio and video signals, transmits them externally, or writes them to a recording medium will be described. The television ex300 receives user operations from the remote control ex220, etc., and under the control of the control unit ex310, the audio signal processing unit ex304 encodes the audio signal, and the video signal processing unit ex305 encodes the video signal using the encoding methods described in the above embodiments. The encoded audio and video signals are multiplexed by the multiplexing/demultiplexing unit ex303 and output externally. During multiplexing, these signals can be temporarily stored in buffers ex320, ex321, etc., to synchronize the audio and video signals for playback. Buffers ex318, ex319, ex320, and ex321 can be provided in multiple locations as shown, or a single or more buffers can be shared. Furthermore, data can be stored in buffers other than those shown, for example, between the modulation/demodulation unit ex302 or the multiplexing/demultiplexing unit ex303, to prevent system overflow or underflow.

Furthermore, in addition to receiving audio and video data from broadcasts or recording media, the television ex300 may also include a mechanism for accepting AV input from a microphone or camera and encoding the data obtained therefrom. While the television ex300 is described here as being capable of the aforementioned encoding, multiplexing, and external output, it may also be capable only of the aforementioned reception, decoding, and external output, without being able to perform these processes.

In addition, when the multiplexed data is read from or written into the recording medium by the reader/recorder ex218, the above-mentioned decoding processing or encoding processing can be performed by either the TV ex300 or the reader/recorder ex218, or the TV ex300 and the reader/recorder ex218 can share the processing.

As an example, Figure 23 shows the structure of an information reproduction/recording unit ex400 for reading or writing data from an optical disc. The information reproduction/recording unit ex400 includes the following units ex401, ex402, ex403, ex404, ex405, ex406, and ex407. The optical head ex401 writes information by irradiating a laser spot onto the recording surface of the recording medium ex215, which is an optical disc, and reads the information by detecting light reflected from the recording surface of the recording medium ex215. The modulation recording unit ex402 electrically drives the semiconductor laser built into the optical head ex401, modulating the laser light according to the recorded data. The reproduction demodulation unit ex403 amplifies the reproduction signal obtained by electrically detecting the light reflected from the recording surface by the photodetector built into the optical head ex401, separates and demodulates the signal components recorded on the recording medium ex215, and reproduces the desired information. The buffer ex404 temporarily stores information to be recorded on the recording medium ex215 and information to be reproduced from the recording medium ex215. The disk motor ex405 rotates the recording medium ex215. The servo control unit ex406 controls the rotation of the disk motor ex405 while moving the optical head ex401 to a predetermined information track and tracking the laser spot. The system control unit ex407 controls the overall information reproducing/recording unit ex400. The system control unit ex407 utilizes the various information stored in the buffer ex404, generates and appends new information as needed, and coordinates the operation of the modulation/recording unit ex402, the reproduction/demodulation unit ex403, and the servo control unit ex406 to record and reproduce information via the optical head ex401. The system control unit ex407, comprised of, for example, a microprocessor, executes these read/write programs to perform these processes.

The above description assumes that the optical head ex401 irradiates a laser spot, but a configuration in which high-density recording is performed using near-field light may also be employed.

Figure 24 shows a schematic diagram of a recording medium ex215, which is an optical disc. Guide grooves (grooves) are formed in a spiral pattern on the recording surface of the recording medium ex215. Address information indicating the absolute position on the disc is pre-recorded in the information track ex230 by changes in the groove shape. This address information includes information for identifying the position of a recording block ex231, which is the unit of recorded data. By reproducing the information track ex230 in a recording and reproducing device and reading the address information, the recording block can be identified. Furthermore, the recording medium ex215 includes a data recording area ex233, an inner circumference area ex232, and an outer circumference area ex234. The data recording area ex233 is used for recording user data. The inner circumference area ex232 and outer circumference area ex234, located on the inner or outer circumference of the data recording area ex233, are used for specific purposes other than recording user data. The information recording/reproducing unit ex400 reads and writes encoded audio data, video data, or encoded data multiplexed with these data, to the data recording area ex233 of the recording medium ex215.

The above description uses single-layer optical discs such as DVDs and BDs as examples, but the present invention is not limited to these. Optical discs with multi-layer structures and recording capabilities other than the surface are also possible. In addition, optical discs with a structure that records information using light of different wavelengths and colors in the same place on the disc, or layers that record different information from various angles, etc., can also be used to perform multi-dimensional recording/reproduction.

Furthermore, in the digital broadcasting system ex200, a car ex210 equipped with an antenna ex205 can receive data from a satellite ex202 or the like and reproduce moving images on a display device such as a car navigation system ex211 in the car ex210. The car navigation system ex211 can be configured by adding a GPS receiver to the configuration shown in FIG22 , for example. Similar configurations can also be used in the computer ex111 and the mobile phone ex114.

FIG25A shows a mobile phone ex114 that uses the moving picture decoding method and moving picture encoding method described in the above embodiments. The mobile phone ex114 includes an antenna ex350 for transmitting and receiving radio waves with the base station ex110, a camera unit ex365 capable of capturing both video and still images, and a display unit ex358, such as a liquid crystal display, that displays decoded data, including images captured by the camera unit ex365 and images received by the antenna ex350. The mobile phone ex114 also includes a main unit including an operation key unit ex366, an audio output unit ex357 including a speaker for audio output, an audio input unit ex356 including a microphone for audio input, a memory unit ex367 for storing encoded or decoded data, such as captured video, still images, recorded audio, or received video, still images, and emails, and a slot unit ex364 that serves as an interface with a recording medium that also stores data.

Next, an example configuration of the mobile phone ex114 will be described using FIG25B . The mobile phone ex114 includes a main control unit ex360 that collectively controls various components of the main body including the display unit ex358 and the operation key unit ex366. The power supply circuit unit ex361, the operation input control unit ex362, the video signal processing unit ex355, the camera interface unit ex363, the LCD (Liquid Crystal Display) control unit ex359, the modulation/demodulation unit ex352, the multiplexing/demultiplexing unit ex353, the audio signal processing unit ex354, the slot unit ex364, and the memory unit ex367 are interconnected via a bus ex370.

When the user ends a call and turns on the power button, the power circuit unit ex361 supplies power from the battery pack to various components, thereby activating the mobile phone ex114 to be operational.

Under the control of the main control unit ex360, which includes a CPU, ROM, and RAM, the mobile phone ex114, in voice call mode, receives a sound signal collected by the sound input unit ex356 and converts it into a digital sound signal through the sound signal processing unit ex354. This signal is then subjected to spread spectrum processing by the modulation/demodulation unit ex352. The signal is then subjected to digital-to-analog conversion and frequency conversion by the transmission/reception unit ex351 before being transmitted via the antenna ex350. Furthermore, in voice call mode, the mobile phone ex114 amplifies the received data received by the antenna ex350, performs frequency conversion and analog-to-digital conversion, performs despread spectrum processing by the modulation/demodulation unit ex352, converts the signal into analog sound data by the sound signal processing unit ex354, and outputs the signal via the sound output unit ex357.

Furthermore, when sending an email in data communication mode, the text data of the email, input through the operation keys ex366 and other functions on the main unit, is sent to the main control unit ex360 via the operation input control unit ex362. The main control unit ex360 then performs spectrum diffusion processing on the text data using the modulation/demodulation unit ex352. The transmission/reception unit ex351 then performs digital-to-analog conversion and frequency conversion, before transmitting the data via the antenna ex350 to the base station ex110. When receiving an email, the received data is processed substantially inversely to the aforementioned processing and then output to the display unit ex358.

In data communication mode, when transmitting video, still images, or both video and audio, the video signal processing unit ex355 compresses and encodes the video signal supplied from the camera unit ex365 using the moving image encoding method described in the aforementioned embodiments (i.e., it functions as an image encoding device in accordance with one embodiment of the present invention) and transmits the encoded video data to the multiplexing/demultiplexing unit ex353. Furthermore, the audio signal processing unit ex354 encodes the audio signal collected by the audio input unit ex356 during the capture of video or still images by the camera unit ex365 and transmits the encoded audio data to the multiplexing/demultiplexing unit ex353.

The multiplexing/demultiplexing unit ex353 multiplexes the coded video data supplied by the video signal processing unit ex355 and the coded audio data supplied by the audio signal processing unit ex354 in a prescribed manner. The resulting multiplexed data is subjected to spectrum spreading processing by the modulation/demodulation unit (modulation/demodulation circuit unit) ex352. After digital-to-analog conversion and frequency conversion are performed by the transmitting/receiving unit ex351, the data is transmitted via the antenna ex350.

When receiving data of a moving image file linking to a homepage, or an email with attached video or audio, during data communication mode, the multiplexing/demultiplexing unit ex353 demultiplexes the multiplexed data received via the antenna ex350 into a bit stream of video data and a bit stream of audio data. The multiplexing/demultiplexing unit ex353 then decodes the multiplexed data, separating it into a video data bit stream and an audio data bit stream. The multiplexing/demultiplexing unit ex353 then supplies the encoded video data to the video signal processing unit ex355 and the encoded audio data to the audio signal processing unit ex354 via the synchronous bus ex370. The video signal processing unit ex355 decodes the video signal using a moving image decoding method corresponding to the moving image encoding method described in the above embodiments (i.e., it functions as an image decoding device according to one embodiment of the present invention). For example, the video or still image contained in the moving image file linking to a homepage is displayed on the display unit ex358 via the LCD control unit ex359. The audio signal processing unit ex354 decodes the audio signal, outputting the audio through the audio output unit ex357.

Furthermore, similar to the television ex300, terminals such as the mobile phone ex114 can be implemented in three different configurations: a transmitting terminal with both an encoder and a decoder, a transmitting terminal with only an encoder, and a receiving terminal with only a decoder. Furthermore, the digital broadcasting system ex200 has been described as transmitting and receiving multiplexed data in which music data, etc., is multiplexed with video data. However, data such as text data associated with the video may also be multiplexed in addition to audio data, or the video data itself may be used instead of the multiplexed data.

In this way, the motion image encoding method or motion image decoding method shown in the above-mentioned embodiments can be used in any of the above-mentioned devices and systems, and in this way, the effects described in the above-mentioned embodiments can be obtained.

In addition, the present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the scope of the present invention.

(Implementation method 5)

Video data may be generated by switching between the moving picture encoding method or apparatus described in each of the above embodiments and a moving picture encoding method or apparatus conforming to a different standard such as MPEG-2, MPEG4-AVC, or VC-1 as needed.

When multiple video data are generated according to different standards, a decoding method corresponding to each standard must be selected during decoding. However, since it is not possible to identify which standard the video data to be decoded complies with, it is difficult to select an appropriate decoding method.

To address this issue, multiplexed data, which multiplexes audio data and other data with video data, is structured to include identification information indicating the standard to which the video data conforms. The following describes the specific structure of multiplexed data, including video data generated by the moving image encoding methods or apparatuses described in the above embodiments. The multiplexed data is a digital stream in the form of an MPEG-2 transport stream.

Figure 26 shows the structure of multiplexed data. As shown in Figure 26, the multiplexed data is obtained by multiplexing one or more of a video stream, an audio stream, a presentation graphics stream (PG), and an interactive graphics stream. The video stream represents the main and secondary images of a movie, the audio stream (IG) represents the main audio portion of the movie and the secondary audio mixed with the main audio, and the presentation graphics stream represents the movie's subtitles. Here, the main image refers to the normal image displayed on the screen, and the secondary image is an image displayed on a smaller screen within the main image. Furthermore, the interactive graphics stream represents the dialog screen created by arranging GUI components on the screen. The video stream is encoded using the moving image encoding methods or devices described in the above embodiments, or using conventional moving image encoding methods or devices based on standards such as MPEG-2, MPEG4-AVC, and VC-1. The audio stream is encoded using methods such as Dolby AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD, or Linear PCM.

Each stream included in the multiplexed data is identified by a PID. For example, the video stream used in the movie's image is assigned 0x1011, the audio stream is assigned 0x1100 to 0x111F, the presentation graphics are assigned 0x1200 to 0x121F, the interactive graphics stream is assigned 0x1400 to 0x141F, the video stream used in the movie's secondary image is assigned 0x1B00 to 0x1B1F, and the audio stream used for the secondary audio mixed with the primary audio is assigned 0x1A00 to 0x1A1F.

Figure 27 schematically illustrates how multiplexed data is multiplexed. First, the video stream ex235, consisting of multiple video frames, and the audio stream ex238, consisting of multiple audio frames, are converted into PES packet sequences ex236 and ex239, respectively, and then into TS packets ex237 and ex240. Similarly, the presentation graphics stream ex241 and interactive graphics data ex244 are converted into PES packet sequences ex242 and ex245, respectively, and then into TS packets ex243 and ex246. Multiplexed data ex247 is constructed by multiplexing these TS packets into a single stream.

Figure 28 shows in more detail how a video stream is stored in a PES packet sequence. The first segment of Figure 28 shows the video frame sequence of the video stream. The second segment shows the PES packet sequence. As shown by arrows yy1, yy2, yy3, and yy4 in Figure 28, the multiple I-pictures, B-pictures, and P-pictures that form Video Presentation Units in the video stream are segmented and stored in the payload of PES packets. Each PES packet has a PES header, which stores the PTS (Presentation Time Stamp) representing the display time of the picture and the DTS (Decoding Time Stamp) representing the decoding time of the picture.

Figure 29 shows the format of the TS packets ultimately written into the multiplexed data. A TS packet is a fixed-length 188-byte packet consisting of a 4-byte TS header containing information such as the stream PID identifying information and a 184-byte TS payload storing the data. The aforementioned PES packets are segmented and stored within the TS payload. On BD-ROMs, a 4-byte TP_Extra_Header is added to the TS packet, forming a 192-byte source packet that is written into the multiplexed data. The TP_Extra_Header contains information such as the ATS (Arrival_Time_Stamp). The ATS indicates the start time of transfer of the TS packet to the decoder's PID filter. In the multiplexed data, the source packets are arranged as shown in the lower section of Figure 29. The numbers, which increment from the beginning of the multiplexed data, are called SPNs (Source Packet Numbers).

Furthermore, in the TS packets included in the multiplexed data, in addition to the various streams such as images, audio, and subtitles, there are also PAT (Program Association Table), PMT (Program Map Table), PCR (Program Clock Reference), etc. The PAT indicates the PID of the PMT used in the multiplexed data, and the PID of the PAT itself is registered as 0. The PMT has the PID of each stream such as images, audio, and subtitles included in the multiplexed data, as well as attribute information of the stream corresponding to each PID, and also has various descriptors about the multiplexed data. The descriptors include copy control information that indicates whether copying of the multiplexed data is permitted or not. In order to synchronize the ATC (Arrival Time Clock) which is the time axis of the ATS and the STC (System Time Clock) which is the time axis of the PTS and DTS, the PCR has information on the STC time corresponding to the ATS to which the PCR packet is transferred to the decoder.

Figure 30 is a diagram illustrating the data structure of the PMT in detail. At the beginning of the PMT, there is a PMT header that describes the length of the data contained in the PMT, among other things. Following this header, there are multiple descriptors for the multiplexed data. The aforementioned copy control information, etc., is recorded as descriptors. Following these descriptors, there is stream information for each stream contained in the multiplexed data. The stream information consists of stream descriptors that record the stream type (which identifies the compression codec used for the stream), the stream PID, and stream attribute information (frame rate, aspect ratio, etc.). The stream descriptors contain the number of streams present in the multiplexed data.

When recording on a recording medium or the like, the multiplexed data is recorded together with a multiplexed data information file.

As shown in FIG31 , the multiplexed data information file is management information of the multiplexed data, corresponds one-to-one to the multiplexed data, and is composed of multiplexed data information, stream attribute information, and entry mapping.

As shown in Figure 31, the multiplexed data information consists of a system rate, a playback start time, and a playback end time. The system rate indicates the maximum transfer rate of the multiplexed data to the PID filter of the system target decoder (described later). The interval between ATSs included in the multiplexed data is set to be less than the system rate. The playback start time is the PTS of the first video frame of the multiplexed data, and the playback end time is set to the PTS of the last video frame of the multiplexed data plus the playback interval of one frame.

As shown in Figure 32, stream attribute information is registered for each PID, including attribute information about each stream included in the multiplexed data. Attribute information includes information specific to the video stream, audio stream, presentation graphics stream, and interactive graphics stream. Video stream attribute information includes information such as the codec used to compress the video stream, the resolution, aspect ratio, and frame rate of the individual image data that make up the video stream. Audio stream attribute information includes information such as the codec used to compress the audio stream, the number of channels included in the audio stream, the corresponding language, and the sampling frequency. This information is used for initializing the decoder before playback by the player.

In this embodiment, the stream type contained in the PMT in the multiplexed data is used. Furthermore, when multiplexed data is recorded on a recording medium, video stream attribute information contained in the multiplexed data information is used. Specifically, in the motion image encoding method or apparatus described in each of the above embodiments, a step or unit is provided for setting the stream type or video stream attribute information contained in the PMT to unique information indicating that the stream type or video stream attribute information is generated by the motion image encoding method or apparatus described in each of the above embodiments. This structure enables the distinction between image data generated by the motion image encoding method or apparatus described in each of the above embodiments and image data compliant with other standards.

FIG33 shows the steps of the moving picture decoding method according to this embodiment. In step exS100, the stream type contained in the PMT or the video stream attribute information contained in the multiplexed data is obtained from the multiplexed data. Next, in step exS101, a determination is made as to whether the stream type or video stream attribute information indicates multiplexed data generated using the moving picture encoding method or apparatus described in the above embodiments. If the stream type or video stream attribute information indicates multiplexed data generated using the moving picture encoding method or apparatus described in the above embodiments, decoding is performed in step exS102 using the moving picture decoding method described in the above embodiments. If the stream type or video stream attribute information indicates multiplexed data compliant with conventional standards such as MPEG-2, MPEG4-AVC, or VC-1, decoding is performed in step exS103 using a moving picture decoding method compliant with conventional standards.

By setting new inherent values in the stream type or video stream attribute information, it is possible to determine whether decoding is possible using the moving picture decoding methods or apparatuses described in the above-mentioned embodiments during decoding. Therefore, even when multiplexed data based on different standards is input, the appropriate decoding method or apparatus can be selected, enabling error-free decoding. Furthermore, the moving picture encoding method or apparatus, or moving picture decoding method or apparatus described in this embodiment can be used in any of the above-mentioned devices or systems.

(Implementation Method 6)

The moving picture encoding method and apparatus, and the moving picture decoding method and apparatus described in the above embodiments can typically be implemented using an LSI (Low-Level Integrated Circuit). As an example, FIG34 shows the structure of a single-chip LSI ex500. LSI ex500 includes the following units ex501, ex502, ex503, ex504, ex505, ex506, ex507, ex508, and ex509, which are connected via a bus ex510. The power supply circuit ex505 supplies power to each unit when the power is on, activating the unit to enable operation.

For example, when performing encoding processing, the LSI ex500 receives AV signals from the microphone ex117 and camera ex113 via the AV I/O ex509 under the control of the control unit ex501, which includes the CPU ex502, the memory controller ex503, the stream controller ex504, and the drive frequency control unit ex512. The input AV signals are temporarily stored in an external memory ex511, such as an SDRAM. Under the control of the control unit ex501, the stored data is appropriately divided into multiple passes based on the processing volume and processing speed, and then sent to the signal processing unit ex507, which encodes the audio signal and/or the video signal. The video signal encoding process is the same as that described in the above embodiments. The signal processing unit ex507 also multiplexes the encoded audio data and the encoded video data, as appropriate, and outputs the data externally via the stream I/O ex506. This output bit stream is then transmitted to the base station ex107 or written to the recording medium ex215. In addition, during multiplexing, data can be temporarily stored in the buffer ex508 to synchronize them.

In the above description, the memory ex511 is described as being external to the LSI ex500, but it may also be included within the LSI ex500. The number of buffers ex508 is not limited to one; multiple buffers may be provided. Furthermore, the LSI ex500 may be implemented as a single chip or as multiple chips.

In the above description, the control unit ex510 is assumed to include the CPU ex502, the memory controller ex503, the flow controller ex504, the drive frequency control unit ex512, and the like. However, the structure of the control unit ex510 is not limited to this. For example, the signal processing unit ex507 may also include a CPU. By also including a CPU within the signal processing unit ex507, processing speed can be further improved. As another example, the CPU ex502 may include the signal processing unit ex507, or a portion thereof, such as a sound signal processing unit. In this case, the control unit ex501 includes the CPU ex502 and the signal processing unit ex507 or a portion thereof.

Although the term "LSI" is used here, it may also be called "IC," "system LSI," "super LSI," or "ultra LSI" depending on the degree of integration.

Furthermore, integrated circuit implementation is not limited to LSIs and can also be implemented using dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI fabrication, or reconfigurable processors that can reconfigure the connections and settings of circuit cells within the LSI, can also be used. Such programmable logic devices typically execute the moving picture encoding and decoding methods described in the above-mentioned embodiments by downloading a program comprising software or firmware or reading it from a memory or the like.

Furthermore, if semiconductor technology or other derivative technologies lead to the emergence of integrated circuit technology that replaces LSI, it is natural that this technology can also be used to integrate functional modules. This could be applied to biotechnology, for example.

(Implementation Method 7)

When decoding video data generated using the moving image encoding methods or devices described in the above embodiments, the processing load is expected to increase compared to decoding video data compliant with conventional standards such as MPEG-2, MPEG4-AVC, and VC-1. Therefore, the LSI ex500 must be set to a higher drive frequency than the CPU ex502 drive frequency used when decoding video data compliant with conventional standards. However, setting the drive frequency higher increases power consumption.

To solve this problem, a motion picture decoding device such as the television ex300 or LSI ex500 employs a structure that identifies the standard to which the image data conforms and switches the drive frequency according to the standard. FIG35 shows the structure ex800 of this embodiment. The drive frequency switching unit ex803 sets the drive frequency to a high level when the image data is generated using the motion picture encoding method or device shown in the above embodiments. Furthermore, the decoding processing unit ex801, which executes the motion picture decoding method shown in the above embodiments, is instructed to decode the image data. On the other hand, when the image data conforms to a conventional standard, the drive frequency is set to a lower level than when the image data is generated using the motion picture encoding method or device shown in the above embodiments. Furthermore, the decoding processing unit ex802, which conforms to the conventional standard, is instructed to decode the image data.

More specifically, the driving frequency switching unit ex803 is comprised of the CPU ex502 and the driving frequency control unit ex512 in Figure 34 . Furthermore, the decoding processing unit ex801, which executes the moving image decoding methods described in the above embodiments, and the decoding processing unit ex802, which conforms to conventional standards, correspond to the signal processing unit ex507 in Figure 34 . The CPU ex502 identifies the standard to which the video data conforms. Furthermore, the driving frequency control unit ex512 sets the driving frequency based on a signal from the CPU ex502. Furthermore, the signal processing unit ex507 decodes the video data based on a signal from the CPU ex502. It is conceivable that the identification information described in Embodiment 5, for example, could be used to identify the video data. The identification information is not limited to that described in Embodiment 5; any information capable of identifying the standard to which the video data conforms may be used. For example, if the standard to which the video data conforms can be identified based on an external signal that identifies whether the video data is intended for a television or a disc, such identification can also be performed based on such an external signal. Furthermore, the CPU ex502 can select the driving frequency using a lookup table that associates the image data standard with the driving frequency, as shown in FIG37 . The lookup table is stored in advance in the buffer ex508 or in the internal memory of the LSI, and the CPU ex502 can select the driving frequency by referring to the lookup table.

FIG36 shows the steps for implementing the method of this embodiment. First, in step exS200, the signal processing unit ex507 obtains identification information from the multiplexed data. Next, in step exS201, the CPU ex502 determines, based on the identification information, whether the video data is generated using the encoding method or apparatus described in the above embodiments. If the video data is generated using the encoding method or apparatus described in the above embodiments, in step exS202, the CPU ex502 sends a signal to the drive frequency control unit ex512 to set the drive frequency high. The drive frequency control unit ex512 then sets the drive frequency to a high level. On the other hand, if the video data is compliant with conventional standards such as MPEG-2, MPEG4-AVC, or VC-1, in step exS203, the CPU ex502 sends a signal to the drive frequency control unit ex512 to set the drive frequency low. The drive frequency control unit ex512 then sets the drive frequency to a lower level than when the video data is generated using the encoding method or apparatus described in the above embodiments.

Furthermore, by changing the voltage applied to the LSI ex500 or a device including the LSI ex500 in conjunction with the switching of the drive frequency, it is possible to further enhance power conservation. For example, when the drive frequency is set to a low level, it is possible to also set the voltage applied to the LSI ex500 or a device including the LSI ex500 to a lower level than when the drive frequency is set to a high level.

Furthermore, the drive frequency setting method is not limited to the above-described setting method, as long as the drive frequency is set higher when the decoding processing volume is large and lower when the decoding processing volume is small. For example, if the processing volume of decoding video data compliant with the MPEG4-AVC standard is greater than the processing volume of decoding video data generated by the moving picture encoding method or apparatus described in each of the above embodiments, the drive frequency setting may be reversed.

Furthermore, the method for setting the drive frequency is not limited to a configuration in which the drive frequency is low. For example, it is also possible to set the voltage applied to the LSI ex500 or a device including the LSI ex500 to a high value when the identification information indicates video data generated by the moving image coding method or apparatus described in the above embodiments, and to set the voltage applied to the LSI ex500 or a device including the LSI ex500 to a low value when the identification information indicates video data compliant with conventional standards such as MPEG-2, MPEG4-AVC, or VC-1. Furthermore, as another example, it is also possible to not stop the drive of the CPU ex502 when the identification information indicates video data generated by the moving image coding method or apparatus described in the above embodiments, but to suspend the drive of the CPU ex502 when the identification information indicates video data compliant with conventional standards such as MPEG-2, MPEG4-AVC, or VC-1, as there is sufficient processing time. It is also possible to suspend the drive of the CPU ex502 when the identification information indicates video data generated by the moving image coding method or apparatus described in the above embodiments, as long as there is sufficient processing time. In this case, it is conceivable to set the pause time shorter than when the video data is displayed in accordance with conventional standards such as MPEG-2, MPEG4-AVC, and VC-1.

In this way, by switching the driving frequency according to the standard to which the video data conforms, power saving can be achieved. In addition, when the LSI ex500 or a device including the LSI ex500 is driven by a battery, the battery life can be extended along with power saving.

(Implementation Method 8)

Televisions, mobile phones, and other devices and systems sometimes receive input with multiple video data streams conforming to different standards. To decode these video data streams, the signal processing unit ex507 of the LSI ex500 must support multiple standards. However, using separate signal processing units ex507 for each standard increases the circuit size of the LSI ex500 and its cost.

To address this issue, a configuration is adopted in which the decoding processing unit used to execute the moving picture decoding methods described in the above embodiments shares some of its content with decoding processing units compliant with conventional standards such as MPEG-2, MPEG4-AVC, and VC-1. Figure ex900 in Figure 38A illustrates this configuration example. For example, the moving picture decoding methods described in the above embodiments and the moving picture decoding methods compliant with the MPEG4-AVC standard share some processing content, such as entropy coding, inverse quantization, deblocking filters, and motion compensation. A possible configuration is to share a decoding processing unit ex902 compliant with the MPEG4-AVC standard for these shared processing contents, while using a dedicated decoding processing unit ex901 for other processing contents unique to one embodiment of the present invention and not compliant with the MPEG4-AVC standard. In particular, the present invention features picture segmentation processing. Therefore, for example, a dedicated decoding processing unit ex901 may be used for picture segmentation processing, while a shared decoding processing unit may be used for any or all of the other processing functions, including inverse quantization, entropy decoding, deblocking filters, and motion compensation. Regarding the sharing of decoding processing units, a structure may be adopted in which a decoding processing unit for executing the moving picture decoding method shown in each of the above embodiments is shared for common processing contents, and a dedicated decoding processing unit is used for processing contents unique to the MPEG4-AVC standard.

Furthermore, ex1000 in Figure 38B illustrates another example of partially sharing processing. This example employs a configuration using a dedicated decoding processing unit ex1001 corresponding to processing specific to one embodiment of the present invention, a dedicated decoding processing unit ex1002 corresponding to processing specific to other conventional standards, and a shared decoding processing unit ex1003 corresponding to processing common to the moving picture decoding method in one embodiment of the present invention and moving picture decoding methods in other conventional standards. Dedicated decoding processing units ex1001 and ex1002 are not necessarily specialized for processing specific to one embodiment of the present invention or other conventional standards; they can be configured to perform other general-purpose processing. Furthermore, the configuration of this embodiment can also be implemented in LSI ex500.

In this way, by sharing the decoding processing unit for the common processing contents between the moving picture decoding method according to one embodiment of the present invention and the conventional standard moving picture decoding method, the circuit scale of the LSI can be reduced and the cost can be reduced.

Industrial applicability

The present invention can be applied to image encoding methods, image decoding methods, image encoding devices, and image decoding devices. Furthermore, the present invention can be used in high-resolution information display devices or imaging devices such as televisions, digital video recorders, car navigation systems, mobile phones, digital cameras, and digital video cameras that include image encoding devices.

Label Description

100 Image Coding Device, 101 Input Image Signal, 105 Subtractor, 106 Residual Signal, 110 Transformation Unit, 111 Transform Coefficient, 120 Quantization Unit, 121, 291 Quantization Coefficient, 130, 230 Inversion Unit, 131, 231 Residual Signal, 140, 240 Adder, 141, 151, 161, 171, 241, 251, 261, 271 Decoded Image Signal, 150, 250 Deblocking Filter, 160, 260 Adaptive Loop Filter, 170, 270 Reference Frame Buffer, 180, 280 Prediction Unit, 181, 281 Prediction Signal, 190 Entropy Coding Unit, 191, 201 Encoded Signal, 200 Image Decoding Device, 290 Entropy decoding unit, 300, 310, 330, 340, 350, 360, 370, 390, 400, 410, 420 picture, 301, 302, 311, 312, 313, 314, 321, 381 line, 320, 380 slice header, 331, 332, 333, 341, 342, 343, 344, 351, 354, 361, 362, 363, 364, 365, 371, 372, 373, 374, 391, 392, 393, 394, 401, 402, 403, 404, 411, 412, 413, 414, 415, 421, 422, 423, 424 slice

Claims (15)

1. An image encoding method, wherein the image comprises a first maximum coding unit (LCU) row and a second LCU row following the first LCU row, the method comprising: It has been determined that parallel wavefront processing is feasible; and The first LCU line and the second LCU line are split such that they contain a normal slice and a group of dependency slices, wherein the normal slice is not located at the beginning of the first LCU line, and the group of dependency slices consists of individual dependency slices encoded using information from the normal slice. In this case, based on wavefront parallel processing, it is possible to determine and usually the position of the slice is not at the beginning of the first LCU line, and perform the splitting of the first LCU line and the second LCU line so that the entire dependent slice group is included in the first LCU line. 2. The image encoding method of claim 1, further comprising, for each slice of the general slice and dependent slice group, generating information indicating whether each slice is a general slice or a dependent slice. 3. The image encoding method of claim 1 or 2, further comprising encoding an indicator into a bitstream to indicate that the wavefront parallel processing is feasible. 4. The image encoding method as described in claim 1 or 2, further comprising encoding the first LCU line and the second LCU line in parallel using wavefront parallel processing. 5. The image encoding method of claim 4, further comprising encoding the second LCU row without referring to the type or position of any slice in the first LCU row. 6. An image encoding apparatus, wherein the image comprises a first maximum coding unit (LCU) row and a second LCU row following the first LCU row, the apparatus comprising: The circuit is configured as follows: It has been determined that parallel wavefront processing is feasible; and The first LCU line and the second LCU line are split such that they contain a normal slice and a group of dependency slices, wherein the normal slice is not located at the beginning of the first LCU line, and the group of dependency slices consists of individual dependency slices encoded using information from the normal slice. In this case, based on wavefront parallel processing, it is possible to determine and usually the position of the slice is not at the beginning of the first LCU line, and perform the splitting of the first LCU line and the second LCU line so that the entire dependent slice group is included in the first LCU line. 7. The image encoding apparatus of claim 6, wherein the circuitry is further configured to generate information indicating whether each slice is a normal slice or a dependent slice for each slice of the normal slice and dependent slice group. 8. The image encoding apparatus of claim 6 or 7, wherein the circuitry is further configured to encode an indicator into a bitstream to indicate that the wavefront parallel processing is feasible. 9. The image encoding apparatus of claim 6 or 7, wherein the circuit is configured to encode the first LCU line and the second LCU line in parallel using the wavefront parallel processing. 10. The image encoding apparatus of claim 9, wherein the circuitry is configured to encode the second LCU row without referring to the type or position of a slice in the first LCU row. 11. A persistent computer-readable medium storing a computer-executable program that, when executed, causes a processor to encode an image comprising a first maximum coding unit (LCU) line and a second LCU line following the first LCU line, the processor being caused to: It has been determined that parallel wavefront processing is feasible; and The first LCU line and the second LCU line are split such that they contain a normal slice and a group of dependency slices, wherein the normal slice is not located at the beginning of the first LCU line, and the group of dependency slices consists of individual dependency slices encoded using information from the normal slice. In this case, based on wavefront parallel processing, it is possible to determine and usually the position of the slice is not at the beginning of the first LCU line, and perform the splitting of the first LCU line and the second LCU line so that the entire dependent slice group is included in the first LCU line. 12. The persistent computer-readable medium of claim 11, wherein the processor is further configured to generate information indicating whether each slice is a normal slice or a dependent slice for each slice of the general slice and dependent slice group. 13. The permanent computer-readable medium of claim 11 or 12, wherein the processor is further configured to encode an indicator into a bitstream to indicate that the wavefront parallel processing is feasible. 14. The permanent computer-readable medium of claim 11 or 12, wherein the processor is further configured to encode the first LCU line and the second LCU line in parallel using wavefront parallel processing. 15. The persistent computer-readable medium of claim 14, wherein the processor is further configured to encode the second LCU line without referring to the type or position of a slice in the first LCU line.