HK1235953B - Dynamic image encoding device and method, dynamic image decoding device and method - Google Patents

Description

Moving picture encoding device and method thereof, moving picture decoding device and method thereof

This application is a divisional application of the invention patent application with application number 201180047148.5, application date July 21, 2011, and invention name “Motion image encoding device, motion image decoding device, motion image encoding method and motion image decoding method”.

Technical Field

The present invention relates to a moving picture encoding device, a moving picture decoding device, a moving picture encoding method, and a moving picture decoding method used in image compression encoding technology, compressed image data transmission technology, and the like.

Background Art

For example, in international standard image coding methods such as MPEG (Moving Picture Experts Group) and "ITU-T H.26x", a method is adopted in which block data (hereinafter referred to as "macroblock") that summarizes a luminance signal of 16×16 pixels and a color difference signal of 8×8 pixels corresponding to the luminance signal of 16×16 pixels is treated as a unit and compressed based on motion compensation technology and orthogonal transformation/transform coefficient quantization technology.

In the motion compensation process in the moving picture encoding device and the moving picture decoding device, a motion vector is detected and a predicted image is generated in units of macroblocks with reference to a previous or next picture.

At this time, a picture obtained by performing inter-picture prediction coding with reference to only one picture is called a P picture, and a picture obtained by performing inter-picture prediction coding with reference to two pictures at the same time is called a B picture.

In AVC/H.264 (ISO/IEC 14496-10 | ITU-T H.264), which is an international standard, a coding mode called direct mode can be selected when encoding a B picture (for example, see Non-Patent Document 1).

That is, it is possible to select: in the macroblock of the encoding object, there is no encoding data of the motion vector, and the encoding mode of the motion vector of the macroblock of the encoding object is generated by using the motion vectors of the macroblocks of other encoded pictures and the motion vectors of the surrounding macroblocks through prescribed calculation processing.

There are two types of direct modes: a temporal direct mode and a spatial direct mode.

In the temporal direct mode, a motion vector of a macroblock to be coded is generated by referring to a motion vector of another coded picture and performing a motion vector scaling process according to the time difference between the coded picture and the coded picture.

In the spatial direct mode, the motion vector of the current macroblock is generated based on the motion vectors of at least one already encoded macroblock located around the current macroblock, with reference to the motion vectors.

In this direct mode, by using "direct_spatial_mv_pred_flag" which is a flag set in a slice header, either the temporal direct mode or the spatial direct mode can be selected in slice units.

In direct mode, the mode in which transform coefficients are not encoded is called skip mode. Hereinafter, when referring to direct mode, skip mode is also included.

Here, FIG11 is a schematic diagram illustrating a method of generating a motion vector by the temporal direct mode.

In FIG11 , “P” indicates a P picture, and “B” indicates a B picture.

In addition, the numbers 0-3 represent the display order of the pictures, indicating that the images are displayed at time T0, T1, T2, and T3.

It is assumed that the encoding process of the pictures is performed in the order of P0, P3, B1, and B2.

For example, assume a case where macroblock MB1 in picture B2 is encoded by the temporal direct mode.

In this case, the motion vector MV of the macroblock MB2 spatially co-located with the macroblock MB1 is used, which is the motion vector of the picture P3 that is the closest to the picture B2 among the coded pictures that are behind the picture B2 on the time axis.

This motion vector MV refers to the picture P0, and motion vectors MVL0 and MVL1 used when encoding the macroblock MB1 are obtained by the following equation (1).

FIG12 is a schematic diagram illustrating a method of generating a motion vector by a spatial direct mode.

In FIG12 , the current MB (currentMB) indicates a macroblock to be encoded.

At this time, if the motion vector of the encoded macroblock A on the left of the macroblock to be encoded is set to MVa, the motion vector of the encoded macroblock B above the macroblock to be encoded is set to MVb, and the motion vector of the encoded macroblock C above the right of the macroblock to be encoded is set to MVc, then the motion vector MV of the macroblock to be encoded can be calculated by finding the median (central value) of these motion vectors MVa, MVb, and MVc as shown in the following formula (2).

MV=median(MVa, MVb, MVc) (2)

In the spatial direct mode, motion vectors are calculated separately for the front and rear directions, but both can be calculated using the above-described method.

Furthermore, the reference image used for generating the predicted image is managed as a reference image list for each vector used for reference. When two vectors are used, the reference image lists are referred to as list 0 and list 1.

Reference image lists are stored sequentially from temporally closest reference images. Typically, list 0 represents forward reference images, and list 1 represents backward reference images. However, list 1 may represent forward reference images, list 0 may represent backward reference images, or both list 0 and list 1 may represent forward and backward reference images. Furthermore, the order of arrangement does not necessarily need to be from temporally closest reference images forward.

For example, the following Non-Patent Document 1 describes that a reference image list can be rearranged for each slice.

[Non-Patent Document 1] MPEG-4 AVC (ISO/IEC 14496-10)/ITU-T H.264 Standard

Summary of the Invention

Conventional image coding devices are configured as described above. By referring to the "direct_spatial_mv_pred_flag" flag set in the slice header, it is possible to switch between temporal direct mode and spatial direct mode on a slice-by-slice basis. However, switching between temporal direct mode and spatial direct mode is not possible on a macroblock-by-macroblock basis. Therefore, even if the optimal direct mode for a macroblock belonging to a slice is, for example, spatial direct mode, if the direct mode corresponding to that slice is determined to be temporal direct mode, temporal direct mode must be used for that macroblock, and the optimal direct mode cannot be selected. In this case, since the optimal direct mode cannot be selected, unnecessary motion vectors must be encoded, resulting in an increase in the amount of code.

The present invention is completed to solve the above-mentioned problems, and its purpose is to obtain a motion image encoding device, motion image decoding device, motion image encoding method and motion image decoding method that can select the optimal direct mode according to a specified block unit and reduce the amount of code.

The moving picture coding apparatus of the present invention comprises: a coding control unit that determines a maximum size of a coding block serving as a processing unit when performing prediction processing, determines an upper limit number of hierarchical divisions of the maximum-sized coding block, and selects a coding mode that determines a coding method for each coding block from one or more available coding modes; and a block division unit that divides an input image into predetermined coding blocks and hierarchically divides the coding blocks until the upper limit number of hierarchical divisions determined by the coding control unit is reached; when the coding control unit selects a direct mode inter-frame coding mode as the coding mode corresponding to the coding blocks divided by the block division unit, a motion compensation prediction unit that selects a motion vector suitable for generating a predicted image from one or more selectable motion vectors, performs motion compensation prediction processing on the coding block using the motion vector to generate a predicted image, and outputs index information indicating the motion vector to a variable length coding unit, which performs variable length coding on the index information.

According to the present invention, the present invention comprises: an encoding control unit that determines a maximum size of a coding block serving as a processing unit when performing prediction processing, determines an upper limit number of hierarchical divisions of the maximum-sized coding block, and selects a coding mode that determines a coding method for each coding block from one or more available coding modes; and a block division unit that divides an input image into predetermined coding blocks and hierarchically divides the coding blocks until the upper limit number of hierarchical divisions determined by the encoding control unit is reached. When the encoding control unit selects a direct mode inter-frame coding mode as the coding mode corresponding to the coding blocks divided by the block division unit, a motion compensation prediction unit selects a motion vector suitable for generating a predicted image from one or more selectable motion vectors, performs motion compensation prediction processing on the coding block using the motion vector to generate a predicted image, and outputs index information indicating the motion vector to a variable length encoding unit. The variable length encoding unit performs variable length coding on the index information. This achieves the effect of being able to select the optimal direct mode for each predetermined block, thereby reducing the amount of code.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a diagram showing the structure of a moving picture encoding apparatus according to Embodiment 1 of the present invention.

FIG2 is a diagram showing the structure of the motion compensation prediction unit 5 of the moving picture encoding device according to the first embodiment of the present invention.

FIG. 3 is a diagram showing the configuration of the direct vector generation unit 23 constituting the motion compensation prediction unit 5 .

FIG. 4 is a diagram showing the configuration of the direct vector determination unit 33 constituting the direct vector generation unit 23 .

FIG5 is a diagram showing the structure of the moving picture decoding apparatus according to Embodiment 1 of the present invention.

FIG6 is a diagram showing the structure of the motion compensation prediction unit 54 of the moving picture decoding device according to Embodiment 1 of the present invention.

FIG7 is a flowchart showing the processing contents of the moving picture encoding apparatus according to Embodiment 1 of the present invention.

FIG8 is a flowchart showing the processing contents of the moving picture decoding apparatus according to Embodiment 1 of the present invention.

FIG. 9 is an explanatory diagram showing a case where a coding block of a maximum size is hierarchically divided into a plurality of coding blocks.

FIG10(a) is a diagram showing the distribution of partitions after division, and FIG10(b) is an explanatory diagram showing, in a quad-tree graph, the allocation of coding modes m( ^Bn ) to partitions after hierarchical division.

FIG. 11 is a schematic diagram illustrating a method of generating a motion vector by the temporal direct mode.

FIG12 is a schematic diagram illustrating a method of generating a motion vector by a spatial direct mode.

FIG13 is a schematic diagram illustrating a method of generating a space direct vector from the median prediction candidates A1-An, B1-Bn, C, D, and E. FIG.

FIG. 14 is a schematic diagram illustrating a method of generating a space direct vector by scaling according to a distance in a time direction.

FIG. 15 is an explanatory diagram showing an example of calculation of an evaluation value using the similarity between a forward prediction image and a backward prediction image.

FIG. 16 is an explanatory diagram showing an evaluation formula using a variance value (variance) of a motion vector.

FIG17 is an explanatory diagram showing space vectors MV_A, MV_B, MV_C, and time vectors MV_1 to MV_8.

FIG18 is an explanatory diagram showing the generation of one candidate vector from a plurality of encoded vectors.

FIG. 19 is an explanatory diagram showing an example of calculating the evaluation value SAD by combining only temporally preceding images.

FIG. 20 is an explanatory diagram illustrating a search for an image similar to an L-shaped template.

FIG21 is an explanatory diagram showing an example in which the size of the coding block ^Bn is ^Ln = ^kMn .

FIG. 22 is an explanatory diagram showing an example of division such as (Ln ⁺¹ , Mn ⁺¹ ) = ( ^Ln /2, ^Mn /2).

FIG. 23 is an explanatory diagram showing an example in which one of the divisions in FIG. 21 or FIG. 22 can be selected.

FIG. 24 is an explanatory diagram showing an example in which the transform block size unit has a hierarchical structure.

FIG25 is a diagram showing the structure of the motion compensation prediction unit 5 of the moving picture encoding device according to Embodiment 3 of the present invention.

FIG26 is a diagram showing the structure of the direct vector generation unit 25 constituting the motion compensation prediction unit 5 .

FIG. 27 is a diagram showing the configuration of the initial vector generating section 34 constituting the direct vector generating section 25 .

FIG. 28 is a diagram showing the configuration of the initial vector determination unit 73 constituting the initial vector generation unit 34 .

FIG29 is a diagram showing the structure of the motion compensation prediction unit 54 of the moving picture decoding device according to Embodiment 3 of the present invention.

FIG30 is an explanatory diagram showing a motion vector search process.

FIG31 is a diagram showing the structure of the motion compensation prediction unit 5 of the moving picture encoding device according to the fourth embodiment of the present invention.

FIG32 is a diagram showing the structure of the motion compensation prediction unit 54 of the moving picture decoding device according to Embodiment 4 of the present invention.

FIG. 33 is an explanatory diagram showing direct vector candidate indices describing selectable motion vectors and index information indicating the motion vectors.

FIG34 is an explanatory diagram showing an example of encoding only index information of one vector.

FIG35 is a diagram showing the structure of the direct vector generation unit 26 constituting the motion compensation prediction unit 5. As shown in FIG35 , the direct vector generation unit 26 includes a plurality of motion compensation prediction units 5;

FIG36 is a diagram showing the structure of the motion compensation prediction unit 5 of the moving picture encoding device according to the fifth embodiment of the present invention.

FIG37 is a diagram showing the structure of the direct vector generation unit 27 constituting the motion compensation prediction unit 5. As shown in FIG37 , the direct vector generation unit 27 includes ...

FIG38 is a diagram showing the structure of the motion compensation prediction unit 54 of the moving picture decoding device according to Embodiment 5 of the present invention.

FIG39 is a diagram showing the structure of the direct vector generation unit 26 constituting the motion compensation prediction unit 5. As shown in FIG39 , the direct vector generation unit 26 includes a plurality of motion compensation prediction units 5;

FIG40 is an explanatory diagram showing the correlation with surrounding blocks.

FIG. 41 is an explanatory diagram showing a list of one or more selectable motion vectors for each block size of a coding block.

FIG. 42 is an explanatory diagram showing an example of a list in which the maximum block size is "128".

FIG. 43 is an explanatory diagram showing a list of one or more selectable motion vectors for each division pattern of a coding block.

FIG44 is a flowchart showing a process of transmitting list information in the moving picture encoding device.

FIG45 is a flowchart showing a process of receiving list information in the moving picture decoding device.

FIG46 is an explanatory diagram showing an example in which a change flag “ON” and list information indicating the list after the change are encoded in order to change “temporal” in the list from selectable to unselectable.

FIG. 47 is an explanatory diagram showing an example in which the currently held list is changed because the change flag is “ON”.

FIG. 48 is an explanatory diagram showing an example in which a change flag is prepared for each block size and only list information related to the block size for which the selectable motion vector is changed is encoded.

FIG49 is an explanatory diagram showing an example in which an inter-coded block is searched from a target block, and all vectors included in the block are regarded as spatial vector candidates.

(Explanation of Reference Numerals)

1: Coding control unit (coding control unit); 2: Block division unit (block division unit); 3: Switch (intra-frame prediction unit, motion compensation prediction unit); 4: Intra-frame prediction unit (intra-frame prediction unit); 5: Motion compensation prediction unit (motion compensation prediction unit); 6: Subtraction unit (differential image generation unit); 7: Transformation and quantization unit (image compression unit); 8: Inverse quantization and inverse transformation unit; 9: Addition unit; 10: Intra-frame prediction memory; 11: Loop filter unit; 12: Motion compensation prediction frame memory; 13: Variable length coding unit (variable length coding unit); 21: Switch; 22: Motion vector search unit; 23: Direct vector generation unit; 24: Motion compensation processing unit; 25, 26, 27: Direct vector generation unit; 31: Spatial direct vector generation unit; 32: Temporal direct vector generation unit; 33: Direct vector determination unit; 34, 36, 37: Initial vector generation unit; 35: Motion compensation processing unit Motion vector search unit; 41: Motion compensation unit; 42: Similarity calculation unit; 43: Direct vector selection unit; 51: Variable length decoding unit (variable length decoding unit); 52: Switch (intra-frame prediction unit, motion compensation prediction unit); 53: Intra-frame prediction unit (intra-frame prediction unit); 54: Motion compensation prediction unit (motion compensation prediction unit); 55: Inverse quantization/inverse transform unit (differential image generation unit); 56: Addition unit (decoded image generation unit); 57: Memory for intra-frame prediction; 58: Loop filter unit; 59: Motion compensation prediction frame memory; 61: Switch; 62: Direct vector generation unit; 63: Motion compensation processing unit; 64, 65, 66: Direct vector generation unit; 71: Spatial vector generation unit; 72: Temporal vector generation unit; 73: Initial vector determination unit; 81: Motion compensation unit; 82: Similarity calculation unit; 83: Initial vector determination unit; 91: Switch.

DETAILED DESCRIPTION

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Implementation method 1.

In this embodiment 1, a motion image encoding device is described which inputs each frame image of an image, performs motion compensation prediction between adjacent frames to obtain a prediction differential signal, and performs variable-length coding on the prediction differential signal through orthogonal transformation and quantization, performs compression processing, and then generates a bit stream, and a motion image decoding device which decodes the bit stream output from the motion image encoding device.

The moving picture encoding apparatus of this first embodiment is characterized in that it divides the video signal into regions of various sizes to perform intra-frame and inter-frame adaptive encoding in response to local changes in the spatial and temporal directions of the video signal.

In general, image signals have the characteristic of locally varying signal complexity across space and time. When viewed spatially, within a given image frame, there are graphics with uniform signal characteristics across relatively wide image regions, such as the sky and walls, but there are also graphics with complex texture patterns within smaller image regions, such as people and paintings with fine textures.

When observed in time, the local time-direction graphics of the sky and the wall change little, but the moving people and objects change greatly in time because their outlines perform rigid and non-rigid motion in time.

In the encoding process, a prediction differential signal with low signal power and entropy is generated through temporal and spatial prediction, thereby reducing the overall code amount. If the parameters used for prediction can be evenly applied to the largest possible image signal area, the code amount of the parameter can be reduced.

On the other hand, if the same prediction parameters are applied to an image signal pattern that varies greatly temporally and spatially, prediction errors increase, making it impossible to reduce the amount of code of the prediction difference signal.

Therefore, for image signal patterns that vary greatly temporally and spatially, it is preferable to reduce the area to be predicted, thereby reducing the power and entropy of the prediction difference signal even if the amount of data of the parameters used for prediction is increased.

In order to perform such encoding adapted to the general properties of image signals, the moving picture encoding apparatus of this first embodiment hierarchically divides the image signal into regions based on a predetermined maximum block size, and performs prediction processing and prediction difference encoding processing on each divided region.

The motion image encoding device of this embodiment 1 processes, as image signals, color image signals of arbitrary color spaces such as YUV signals composed of a luminance signal and two color difference signals, and RGB signals output from a digital camera element, as well as arbitrary image signals whose image frames are composed of horizontal and vertical two-dimensional columns of digital samples (pixels), such as monochrome image signals and infrared image signals.

The grayscale of each pixel can be 8 bits, 10 bits, 12 bits, etc.

However, in the following description, unless otherwise specified, the input video signal is assumed to be a YUV signal, and is a 4:2:0 format signal in which the two color difference components U and V are subsampled relative to the luminance component Y.

Furthermore, the unit of processed data corresponding to each image frame is referred to as a "picture." In this first embodiment, a "picture" is described as a signal of an image frame that has been sequentially scanned (progressive scanning). However, if the image signal is an interlaced signal, a "picture" may also be a field image signal that constitutes a video frame.

FIG1 is a diagram showing the structure of a moving picture encoding apparatus according to Embodiment 1 of the present invention.

In Figure 1, the encoding control unit 1 implements the following processing: determines the maximum size of the coding block that serves as the processing unit when performing motion compensation prediction processing (inter-frame prediction processing) or intra-frame prediction processing (intra-frame prediction processing), and determines the upper limit of the number of layers when the maximum-sized coding block is hierarchically divided.

The encoding control unit 1 also performs processing to select a coding mode suitable for each hierarchically divided coding block from one or more available coding modes (one or more intra-frame coding modes and one or more inter-frame coding modes (including the inter-frame coding mode of the direct mode)). Furthermore, the encoding control unit 1 constitutes an encoding control unit.

The block division unit 2 performs the following processing: upon receiving an image signal representing an input image, the block division unit 2 divides the input image represented by the image signal into coding blocks of the maximum size determined by the coding control unit 1, and further divides the coding blocks hierarchically until the upper limit of the number of layers determined by the coding control unit 1 is reached. The block division unit 2 also constitutes a block division unit.

The switching switch 3 implements the following processing: if the encoding mode selected by the encoding control unit 1 is the intra-frame encoding mode, the encoding block divided by the block division unit 2 is output to the intra-frame prediction unit 4; if the encoding mode selected by the encoding control unit 1 is the inter-frame encoding mode, the encoding block divided by the block division unit 2 is output to the motion compensation prediction unit 5.

Upon receiving the coding block divided by the block division unit 2 from the switch 3 , the intra prediction unit 4 performs intra prediction processing on the coding block using the intra prediction parameters output from the encoding control unit 1 to generate a predicted image.

The switch 3 and the intra prediction unit 4 constitute an intra prediction unit.

The motion compensation prediction unit 5 performs the following processing: when the inter-frame coding mode of the direct mode is selected by the coding control unit 1 as the coding mode suitable for the coding block divided by the block division unit 2, a spatial direct vector of the spatial direct mode is generated based on the motion vectors of the coded blocks located around the coding block, and a temporal direct vector of the temporal direct mode is generated based on the motion vectors of the coded pictures that can be referenced by the coding block. From the spatial direct vector or the temporal direct vector, the direct vector with a higher correlation between the reference images is selected, and the motion compensation prediction processing is performed on the coding block using the direct vector to generate a predicted image.

In addition, the motion compensation prediction unit 5 implements the following processing: when an inter-frame coding mode other than the direct mode is selected by the coding control unit 1 as a coding mode suitable for the coding block divided by the block division unit 2, a motion vector is searched from the coding block and the reference image stored in the motion compensation prediction frame memory 12, and using the motion vector, motion compensation prediction processing is implemented for the coding block to generate a predicted image.

The switch 3 and the motion compensation prediction unit 5 constitute a motion compensation prediction unit.

The subtraction unit 6 generates a differential image (= coded block - predicted image) by subtracting the predicted image generated by the intra prediction unit 4 or the motion compensation prediction unit 5 from the coded block divided by the block division unit 2. The subtraction unit 6 constitutes differential image generation means.

The transform/quantization unit 7 performs a transform process (e.g., a DCT (discrete cosine transform) or an orthogonal transform such as a KL transform with a pre-designed basis for a specific learning series) on the difference image generated by the subtraction unit 6, using the transform block size included in the predicted difference coding parameters output from the coding control unit 1 as a unit. Furthermore, the transform coefficients of the difference image are quantized using the quantization parameters included in the predicted difference coding parameters, and the quantized transform coefficients are output as compressed data of the difference image. Furthermore, the transform/quantization unit 7 constitutes image compression means.

The inverse quantization and inverse transformation unit 8 performs the following processing: using the quantization parameter included in the predicted differential coding parameter output from the coding control unit 1, the compressed data output from the transformation and quantization unit 7 is inverse quantized, and the inverse transformation processing (for example, inverse DCT (inverse discrete cosine transform), inverse KL transform and other inverse transformation processing) of the inversely quantized compressed data is performed in units of the transformation block size included in the predicted differential coding parameter, thereby outputting the compressed data after the inverse transformation processing as a locally decoded predicted differential signal.

The adding unit 9 performs the following processing: adds the local decoded prediction difference signal output from the inverse quantization and inverse transformation unit 8 and the prediction signal representing the predicted image generated by the intra-frame prediction unit 4 or the motion compensation prediction unit 5, thereby generating a local decoded image signal representing the local decoded image.

The intra prediction memory 10 is a recording medium such as a RAM that stores a local decoded image represented by a local decoded image signal generated by the adding unit 9 as an image to be used by the intra prediction unit 4 in the next intra prediction process.

The loop filter unit 11 performs processing to compensate for coding distortion included in the local decoded image signal generated by the adder 9 and outputs the local decoded image represented by the local decoded image signal after coding distortion compensation to the motion compensation prediction frame memory 12 as a reference image.

The motion compensation prediction frame memory 12 is a recording medium such as RAM that stores a local decoded image filtered by the loop filter unit 11 as a reference image used by the motion compensation prediction unit 5 in the next motion compensation prediction process.

The variable-length coding unit 13 performs variable-length coding on the compressed data output from the transform/quantization unit 7, the coding mode and prediction difference coding parameters output from the coding control unit 1, and the intra-frame prediction parameters output from the intra-frame prediction unit 4 or the inter-frame prediction parameters output from the motion-compensated prediction unit 5, thereby generating a bitstream in which the compressed data, coding mode, prediction difference coding parameters, and coded data of the intra-frame prediction parameters/inter-frame prediction parameters are multiplexed. The variable-length coding unit 13 constitutes a variable-length coding unit.

FIG2 is a diagram showing the structure of the motion compensation prediction unit 5 of the moving picture encoding device according to the first embodiment of the present invention.

In Figure 2, the switching switch 21 implements the following processing: when the coding mode selected by the coding control unit 1 is an inter-frame mode other than the direct mode, the coding block divided by the block division unit 2 is output to the motion vector search unit 22; on the other hand, when the coding mode is an inter-frame mode other than the direct mode, the coding block divided by the block division unit 2 is output to the direct vector generation unit 23.

Furthermore, the direct vector generator 23 does not use the coded blocks divided by the block divider 2 when generating the direct vectors. Therefore, the coded blocks do not need to be output to the direct vector generator 23 .

The motion vector search unit 22 searches for an optimal motion vector in inter-frame mode while referring to the coding block output from the switch 21 and the reference image stored in the motion compensation prediction frame memory 12 , and outputs the motion vector to the motion compensation processing unit 24 .

The direct vector generation unit 23 performs the following processing: generates a spatial direct vector of the spatial direct mode based on the motion vector of the coded blocks located around the coding block, and generates a temporal direct vector of the temporal direct mode based on the motion vector of the coded picture that can be referenced by the coding block, and selects the direct vector with a higher correlation between the reference images from the spatial direct vector or the temporal direct vector.

The motion compensation processing unit 24 performs the following processing: using the motion vector searched by the motion vector search unit 22 or the direct vector selected by the direct vector generation unit 23 and one or more reference images stored in the motion compensation prediction frame memory 12, it performs motion compensation prediction processing based on the inter-frame prediction parameters output from the encoding control unit 1, thereby generating a predicted image.

Furthermore, the motion compensation processing unit 24 outputs the inter-frame prediction parameters used when performing the motion compensation prediction process to the variable length coding unit 13. When the coding mode selected by the coding control unit 1 is an inter-frame mode other than the direct mode, the motion vector searched by the motion vector search unit 22 is included in the inter-frame prediction parameters and output to the variable length coding unit 13.

FIG. 3 is a diagram showing the configuration of the direct vector generation unit 23 constituting the motion compensation prediction unit 5 .

In Figure 3, the spatial direct vector generation unit 31 implements the following processing: from the motion vector of the encoded block (the motion vector of the encoded block is stored in a motion vector memory not shown in the figure, or the internal memory of the motion compensation prediction unit 5), the motion vector of the encoded block located around the encoded block is read out, and the spatial direct vector of the spatial direct mode is generated based on the motion vector.

The temporal direct vector generation unit 32 performs the following processing: from the motion vector of the encoded block, it reads the motion vector of the block that is spatially located at the same position as the encoded block, which is the motion vector of the encoded picture that the encoded block can refer to, and generates the temporal direct vector of the temporal direct mode based on the motion vector.

The direct vector determination unit 33 performs the following processing: using the spatial direct vector generated by the spatial direct vector generation unit 31, it calculates the evaluation value of the spatial direct mode, and using the temporal direct vector generated by the temporal direct vector generation unit 32, it calculates the evaluation value of the temporal direct mode, compares the evaluation value of the spatial direct mode with the evaluation value of the temporal direct mode, and selects either the spatial direct vector or the temporal direct vector.

FIG. 4 is a diagram showing the configuration of the direct vector determination unit 33 constituting the direct vector generation unit 23 .

In Figure 4, the motion compensation unit 41 implements the following processing: using the spatial direct vector generated by the spatial direct vector generation unit 31, a list 0 prediction image of the spatial direct mode (for example, the forward prediction image of the spatial direct mode) and a list 1 prediction image of the spatial direct mode (for example, the backward prediction image of the spatial direct mode) are generated, and using the temporal direct vector generated by the temporal direct vector generation unit 32, a list 0 prediction image of the temporal direct mode (for example, the forward prediction image of the temporal direct mode) and a list 1 prediction image of the temporal direct mode (for example, the backward prediction image of the temporal direct mode) are generated.

The similarity calculation unit 42 performs the following processing: as the evaluation value of the spatial direct mode, the similarity of the list 0 prediction image (forward prediction image) and the list 1 prediction image (backward prediction image) of the spatial direct mode is calculated, and as the evaluation value of the temporal direct mode, the similarity of the list 0 prediction image (forward prediction image) and the list 1 prediction image (backward prediction image) of the temporal direct mode is calculated.

The direct vector selection unit 43 performs the following processing: compares the similarity between the list 0 prediction image (forward prediction image) and the list 1 prediction image (backward prediction image) in the spatial direct mode calculated by the similarity calculation unit 42, and the similarity between the list 0 prediction image (forward prediction image) and the list 1 prediction image (backward prediction image) in the temporal direct mode, and selects the direct vector of the direct mode with the higher similarity between the list 0 prediction image (forward prediction image) and the list 1 prediction image (backward prediction image) in the spatial direct vector or the temporal direct vector.

FIG5 is a diagram showing the structure of the moving picture decoding apparatus according to Embodiment 1 of the present invention.

In FIG5 , the variable length decoding unit 51 performs the following processing: variable length decoding is performed on the coded data multiplexed into the bit stream to obtain compressed data, coding mode, prediction difference coding parameters, and intra-frame prediction parameters/inter-frame prediction parameters related to each hierarchically divided coding block, and the compressed data and prediction difference coding parameters are output to the inverse quantization and inverse transformation unit 55. The coding mode and intra-frame prediction parameters/inter-frame prediction parameters are also output to the switch 52. The variable length decoding unit 51 constitutes a variable length decoding unit.

The switching switch 52 implements the following processing: when the coding mode related to the coding block output from the variable length decoding unit 51 is the intra-frame coding mode, the intra-frame prediction parameters output from the variable length decoding unit 51 are output to the intra-frame prediction unit 53; when the coding mode is the inter-frame coding mode, the inter-frame prediction parameters output from the variable length decoding unit 51 are output to the motion compensation prediction unit 54.

The intra prediction unit 53 performs a process of performing an intra prediction process on the coding block using the intra prediction parameters output from the switch 52 to generate a predicted image.

The switch 52 and the intra prediction unit 53 constitute an intra prediction unit.

The motion compensation prediction unit 54 performs the following processing: when the coding mode related to the coding block output from the variable length decoding unit 51 is the inter-frame coding mode of the direct mode, a spatial direct vector of the spatial direct mode is generated based on the motion vectors of the decoded blocks located around the coding block, and a temporal direct vector of the temporal direct mode is generated based on the motion vectors of the decoded pictures that the coding block can refer to. From the spatial direct vector or the temporal direct vector, the direct vector with a higher correlation between the reference images is selected, and the motion compensation prediction processing is performed on the coding block using the direct vector to generate a predicted image.

In addition, the motion compensation prediction unit 54 performs the following processing: when the coding mode related to the coding block output from the variable length decoding unit 51 is an inter-frame coding mode other than the direct mode, the motion vector included in the inter-frame prediction parameters output from the variable length decoding unit 51 is used to perform motion compensation prediction processing on the coding block to generate a predicted image.

The switch 52 and the motion compensation prediction unit 54 constitute a motion compensation prediction unit.

The inverse quantization/inverse transform unit 55 performs the following processing: using the quantization parameter included in the prediction difference coding parameter output from the variable length decoding unit 51, it inversely quantizes the compressed data related to the coding block output from the variable length decoding unit 51, and performs inverse transform processing (for example, inverse DCT (inverse discrete cosine transform) or inverse KL transform) on the inversely quantized compressed data in units of the transform block size included in the prediction difference coding parameter. The inversely transformed compressed data is then output as a decoded prediction difference signal (a signal representing the difference image before compression). Furthermore, the inverse quantization/inverse transform unit 55 constitutes a difference image generation unit.

The addition unit 56 performs processing to generate a decoded image signal representing a decoded image by adding the decoded prediction difference signal output from the inverse quantization and inverse transform unit 55 and a prediction signal representing a predicted image generated by the intra prediction unit 53 or the motion compensation prediction unit 54. The addition unit 56 constitutes a decoded image generation unit.

The intra prediction memory 57 is a recording medium such as a RAM that stores a decoded image represented by a decoded image signal generated by the adding unit 56 as an image to be used by the intra prediction unit 53 in the next intra prediction process.

The loop filter unit 58 performs processing to compensate for coding distortion included in the decoded image signal generated by the adder 56 , and outputs a decoded image represented by the decoded image signal after coding distortion compensation to the motion compensation prediction frame memory 59 as a reference image.

The motion compensation prediction frame memory 59 is a recording medium such as RAM that stores a decoded image filtered by the loop filter unit 58 as a reference image used by the motion compensation prediction unit 54 in the next motion compensation prediction process.

FIG6 is a diagram showing the structure of the motion compensation prediction unit 54 of the moving picture decoding device according to Embodiment 1 of the present invention.

In Figure 6, the switching switch 61 implements the following processing: when the coding mode related to the coding block output from the variable length decoding unit 51 is an inter-frame mode other than the direct mode, the inter-frame prediction parameters (including motion vectors) output from the variable length decoding unit 51 are output to the motion compensation processing unit 63; when the coding mode is an inter-frame mode other than the direct mode, the inter-frame prediction parameters output from the variable length decoding unit 51 are output to the direct vector generation unit 62.

The direct vector generation unit 62 performs the following processing: generates a spatial direct vector of the spatial direct mode based on the motion vector of the decoded block located around the coding block, and generates a temporal direct vector of the temporal direct mode based on the motion vector of the decoded picture that can be referenced by the coding block, and selects the direct vector with a higher correlation between the reference images from the spatial direct vector or the temporal direct vector.

Furthermore, the direct vector generation unit 62 performs a process of outputting the inter prediction parameter output from the switch 61 to the motion compensation processing unit 63 .

The internal structure of the direct vector generating unit 62 is the same as that of the direct vector generating unit 23 in FIG. 2 .

The motion compensation processing unit 63 performs the following processing: using the motion vector included in the inter-frame prediction parameters output from the switching switch 61 or the direct vector selected by the direct vector generation unit 62 and the reference image of one frame stored in the motion compensation prediction frame memory 59, a motion compensation prediction process is performed based on the inter-frame prediction parameters output from the direct vector generation unit 62, thereby generating a predicted image.

In FIG1 , an example is shown in which the components of the moving picture coding apparatus, namely, the coding control unit 1, the block division unit 2, the switch 3, the intra-frame prediction unit 4, the motion-compensated prediction unit 5, the subtraction unit 6, the transform/quantization unit 7, the inverse quantization/inverse transform unit 8, the addition unit 9, the loop filter unit 11, and the variable-length coding unit 13, are each configured by dedicated hardware (e.g., a semiconductor integrated circuit or a one-chip microcomputer having a CPU mounted thereon). However, if the moving picture coding apparatus is configured by a computer, a program describing the processing contents of the coding control unit 1, the block division unit 2, the switch 3, the intra-frame prediction unit 4, the motion-compensated prediction unit 5, the subtraction unit 6, the transform/quantization unit 7, the inverse quantization/inverse transform unit 8, the addition unit 9, the loop filter unit 11, and the variable-length coding unit 13 may be stored in a memory of the computer, and the CPU of the computer may execute the program stored in the memory.

FIG7 is a flowchart showing the processing contents of the moving picture encoding apparatus according to Embodiment 1 of the present invention.

In FIG5 , an example is shown in which the variable length decoding unit 51, the switching switch 52, the intra-frame prediction unit 53, the motion compensation prediction unit 54, the inverse quantization/inverse transform unit 55, the adding unit 56, and the loop filter unit 58, which are components of the motion picture decoding apparatus, are each constituted by dedicated hardware (for example, a semiconductor integrated circuit in which a CPU is mounted, or a single-chip microcomputer, etc.). However, in a case where the motion picture decoding apparatus is constituted by a computer, a program describing the processing contents of the variable length decoding unit 51, the switching switch 52, the intra-frame prediction unit 53, the motion compensation prediction unit 54, the inverse quantization/inverse transform unit 55, the adding unit 56, and the loop filter unit 58 may be stored in a memory of the computer, and the program stored in the memory may be executed by the CPU of the computer.

FIG8 is a flowchart showing the processing contents of the moving picture decoding apparatus according to Embodiment 1 of the present invention.

Next, the operation will be explained.

First, the processing contents of the moving picture encoding apparatus in FIG. 1 will be described.

First, the encoding control unit 1 determines the maximum size of the coding block that serves as the processing unit when performing motion compensation prediction processing (inter-frame prediction processing) or intra-frame prediction processing (intra-frame prediction processing), and determines the upper limit number of layers when the maximum-sized coding block is hierarchically divided (step ST1 of Figure 7).

As a method for determining the maximum size of a coding block, for example, a method of determining a size corresponding to the resolution of an input image for all pictures may be considered.

Another possible method is to quantify the difference in complexity of local motion in the input image as a parameter and determine the maximum size to be a small value in pictures with intense motion and a large value in pictures with less motion.

Regarding the upper limit of the number of layers, for example, it is conceivable to increase the number of layers as the motion of the input image becomes more intense so as to detect finer motion, and to reduce the number of layers as the motion of the input image becomes less.

Furthermore, the encoding control unit 1 selects an encoding mode suitable for each hierarchically divided encoding block from one or more available encoding modes (M intra-frame encoding modes, N inter-frame encoding modes (including the direct mode)) (step ST2).

The method for selecting the coding mode performed by the coding control unit 1 is a well-known technology, so detailed description is omitted. For example, there is a method of using any available coding mode to implement coding processing on the coding block to verify the coding efficiency, and selecting the coding mode with the best coding efficiency from multiple available coding modes.

When a video signal representing an input image is input, the block division unit 2 divides the input image represented by the video signal into coding blocks of the maximum size determined by the coding control unit 1, and divides the coding blocks hierarchically until the upper limit of the number of layers determined by the coding control unit 1 is reached.

Here, FIG9 is an explanatory diagram showing a case where a coding block of a maximum size is hierarchically divided into a plurality of coding blocks.

In the example of FIG. 9 , the coding block of the largest size is the coding block B ⁰ of the 0th layer, which has a size of (L ⁰ , M ⁰ ) in the luminance component.

In the example of FIG. 9 , the maximum-sized coding block B ⁰ is used as a starting point, and hierarchical division is performed to a predetermined depth determined separately using a quad-tree structure to obtain coding blocks B ⁿ .

At depth n, a coding block ^Bn is an image region of size ( ^Ln , ^Mn ).

However, ^Ln and ^Mn may be the same or different, but the example in FIG9 shows a case where ^Ln = ^Mn .

Hereinafter, the size of the coding block ^Bn is defined as the size ( ^Ln , ^Mn ) in the luma component of the coding block ^Bn .

Since the block division unit 2 performs quadtree division, (Ln ⁺¹ , Mn ⁺¹ ) = ( ^Ln /2, ^Mn /2) always holds.

In a color video signal (4:4:4 format) such as an RGB signal where all color components have the same number of samples, the size of all color components is ( ^Ln , ^Mn ). However, when processing the 4:2:0 format, the size of the coding block of the corresponding color difference component is ( ^Ln /2, Mn ^/ 2).

Hereinafter, the coding mode selectable in the coding block ^Bn of the nth layer will be expressed as m( ^Bn ).

In the case of a color video signal composed of multiple color components, the coding mode m( ^Bn ) may be configured to use an independent mode for each color component. However, unless otherwise specified, the following description will refer to the coding mode for the luminance component of a coding block in a YUV signal and 4:2:0 format.

In coding mode m( ^Bn ), there are one or more intra-frame coding modes (collectively referred to as "INTRA") and one or more inter-frame coding modes (collectively referred to as "INTER"). As described above, the coding control unit 1 selects the coding mode with the best coding efficiency for the coding block ^Bn from all coding modes or a subset thereof that can be used in the picture.

As shown in FIG9 , the coding block ^Bn is further divided into one or more prediction processing units (partitions).

Hereinafter, the partitions belonging to the coding block ^Bn are referred to as _Pin ⁽ i: partition number in the nth layer).

How to divide the ^partition _P in the coding block B ⁿ is included as information in the coding mode m(B ⁿ ).

All partitions P _i ⁿ are predicted according to the coding mode m(B ⁿ ), but independent prediction parameters can be selected for each partition P _i ⁿ .

The encoding control unit 1 generates a block division state as shown in FIG. 10 , for example, for the maximum-sized encoding block, and determines the encoding block B ⁿ .

The shaded areas in FIG. 10( a ) represent the distribution of partitions after division, and FIG. 10( b ) shows, in a quad-tree diagram, the allocation of coding modes m(B ⁿ ) to partitions after hierarchical division.

In FIG10(b) , the nodes surrounded by □ represent nodes (coding blocks ^Bn ) to which coding mode m( ^Bn ) is assigned.

When the encoding control unit 1 selects the optimal encoding mode m( ^Bn ) for each partition _P ⁱⁿ the encoding block ^Bn , if the encoding mode m( ^Bn ) is an intra-frame encoding mode (step ST3), the switching switch 3 outputs the ^partition _P in the encoding block ^Bn divided by the block division unit 2 to the intra-frame prediction unit 4.

On the other hand, if the coding mode m(B ⁿ ) is the inter-frame coding mode (step ST3 ), the partitions _P ⁱⁿ of the coding block B ⁿ divided by the block dividing unit 2 are output to the motion compensation prediction unit 5 .

Upon receiving the ^partition _Pin of the coding block ^Bn from the switch 3, the intra prediction unit 4 performs intra prediction processing on the partition ^Pin of the coding block ^Bn using the intra prediction parameters corresponding to the coding mode m( ^Bn ) selected by the coding control unit ₁ , and generates ^an intra prediction image _Pin (step ST4).

After generating the intra-prediction image _Pin ^, the intra-prediction unit 4 outputs the ^intra -prediction image _Pin to the subtraction unit 6 and the addition unit 9. In order to enable the video decoding apparatus in FIG5 to also generate the same intra-prediction image _Pin ^, the intra-prediction parameters are output to the variable-length coding unit 13. The intra-prediction processing in this first embodiment is not limited to the algorithm defined in the AVC/H.264 standard (ISO/IEC 14496-10), but the intra-prediction parameters must include information required to generate completely identical intra-prediction images on both the video encoding apparatus and the video decoding apparatus side.

When the motion compensation prediction unit 5 receives the partition P _i ⁿ of the coding block B ⁿ from the switching switch 3, if the coding mode m(B ⁿ ) selected by the coding control unit 1 is the inter-frame coding mode of the direct mode, the motion compensation prediction unit 5 generates a spatial direct vector of the spatial direct mode based on the motion vectors of the coded blocks located around the partition P _i ⁿ of the coding block B ⁿ , and generates a temporal direct vector of the temporal direct mode based on the motion vectors of the coded pictures that can be referenced by the coding block B ⁿ .

Then, the motion-compensated prediction unit 5 selects the direct vector with the highest correlation between the reference images from the spatial direct vector or the temporal direct vector, and performs motion-compensated prediction processing on the partition Pin of the coding block ^Bn using the inter-frame prediction parameters corresponding to the direct ^vector and the coding mode _m ( ^Bn ), thereby generating an inter- ^frame predicted image _Pin (step ST5).

On the other hand, if the coding mode m( ^Bn ) selected by the coding control unit 1 is an inter-frame coding mode other than the direct mode, the motion-compensated prediction unit 5 searches for a motion ^vector from the partition _Pin of the coding block ^Bn and the reference image stored in the motion-compensated prediction frame memory 12, and performs motion-compensated prediction processing on the partition _Pin of the coding ^{block Bn} using the motion vector and the inter-frame prediction parameters corresponding to the coding mode m( ^Bn ), thereby generating ^an inter-frame prediction image _Pin (step ST5).

When the motion compensation prediction unit 5 generates the inter-frame prediction ^image _Pin , it outputs the inter-frame prediction image _Pin to the subtraction unit ⁶ and the addition unit 9. In order to generate the same inter-frame prediction image ^Pin in the moving picture decoding apparatus of FIG. ₅ , the inter-frame prediction parameters are output to the variable length coding unit 13. The inter-frame prediction parameters used to generate the inter-frame prediction image include:

Describes the partition splitting mode information within the coding block ^Bn

Motion vectors for each partition

When the motion compensation prediction frame memory 12 includes a plurality of reference images, reference image indication index information indicating which reference image is used for prediction

When there are multiple motion vector predictor candidates, index information indicating which motion vector predictor to select for use

Index information indicating which filter to use when there are multiple motion compensation interpolation filters

When the motion vector of the partition can express multiple pixel precisions (half pixel, 1/4 pixel, 1/8 pixel, etc.), selection information indicating which pixel precision to use

Such information is multiplexed into the bit stream by the variable length coding unit 13 in order to generate a completely identical inter-frame predicted image on the moving picture decoding device side.

The outline of the processing contents of the motion compensation prediction unit 5 is as described above, and the detailed processing contents will be described later.

When the intra prediction unit 4 or the motion compensation prediction unit 5 generates a prediction image ( ^intra prediction image _Pin , inter prediction image _Pin ), the subtraction unit 6 subtracts the prediction image (intra prediction image Pin, inter prediction image ^{Pin )} generated by the intra prediction unit 4 or the motion compensation prediction unit 5 from the partition _Pin of the coding _block ^Bn divided by the ^block division _unit ² , thereby generating a difference image, and outputs a prediction difference signal _ein ^representing the difference image to the transformation and quantization unit 7 (step ST6).

When the transform/quantization unit 7 receives the predicted differential signal _e ^indicative of the differential image from the subtraction unit 6, it performs transform processing (e.g., DCT (discrete cosine transform), orthogonal transform processing such as KL transform with a basis designed in advance for a specific learning series) on the differential image in units of the transform block size included in the predicted differential coding parameters output from the coding control unit 1, and quantizes the transform coefficients of the differential image using the quantization parameters included in the predicted differential coding parameters, thereby outputting the quantized transform coefficients as compressed data of the differential image to the inverse quantization/inverse transform unit 8 and the variable-length coding unit 13 (step ST7).

If the inverse quantization and inverse transformation unit 8 receives the compressed data of the differential image from the transformation and quantization unit 7, it uses the quantization parameter included in the predicted differential coding parameter output from the encoding control unit 1 to inverse quantize the compressed data of the differential image, and performs inverse transformation processing (for example, inverse DCT (inverse discrete cosine transform), inverse KL transform and other inverse transformation processing) on the inverse quantized compressed data in units of the transformation block size included in the predicted differential coding parameter, thereby outputting the compressed data after the inverse transformation processing as a local decoding prediction differential signal e _i ⁿ hat (due to the relationship of electronic application, the "^" attached to the alphabetical characters is recorded as a hat) to the addition unit 9 (step ST8).

Upon receiving the locally decoded prediction differential signal _e ^int hat from the inverse quantization/inverse transform unit 8, the adder 9 adds the locally decoded prediction differential signal _e ^int hat to a prediction signal representing a prediction image (intra-frame prediction image P _in , inter-frame prediction image _P ^{in )} generated by the intra-frame prediction unit 4 or the motion compensation prediction unit 5, thereby generating a locally decoded partition image _P ^int hat or a locally decoded coded block image as a collection thereof, i.e., a locally decoded image (step ST9).

When the adder 9 generates the local decoded image, it stores the local decoded image signal representing the local decoded image in the intra prediction memory 10 and outputs the local decoded image signal to the loop filter 11 .

The processes of steps ST3 to ST9 are repeatedly performed until the processes of all the hierarchically divided coding blocks ^Bn are completed. When the processes of all the coding blocks ^Bn are completed, the process moves to step ST12 (steps ST10 and ST11).

The variable length coding unit 13 performs entropy coding on the compressed data output from the transform/quantization unit 7, the coding mode (including information indicating the segmentation status of the coding block) and the predicted differential coding parameters output from the coding control unit 1, and the intra-frame prediction parameters output from the intra-frame prediction unit 4 or the inter-frame prediction parameters output from the motion compensation prediction unit 5.

The variable-length coding unit 13 multiplexes the compressed data, which is the encoding result of the entropy coding, the encoding mode, the prediction difference encoding parameter, and the encoded data of the intra-frame prediction parameter/inter-frame prediction parameter to generate a bit stream (step ST12).

If the loop filter unit 11 receives a local decoded image signal from the adder 9, it compensates for the coding distortion contained in the local decoded image signal and saves the local decoded image represented by the local decoded image signal after coding distortion compensation as a reference image in the motion compensation prediction frame memory 12 (step ST13).

The filtering process by the loop filter unit 11 can be performed in units of the largest coding block or each coding block of the local decoded image signal output from the adder 9, or can be performed after the local decoded image signals equivalent to one picture of macroblocks are output and then the local decoded image signals are aggregated for one picture.

Next, the processing contents of the motion compensation prediction unit 5 will be described in detail.

The switch 21 of the motion compensation prediction unit 5 outputs the partitions _Pin of the coding block ^Bn divided by the block division unit 2 to the motion vector search unit 22 when the coding mode m( ^{Bn )} selected by the coding control unit 1 is an inter mode other than the direct mode.

On the other hand, when the coding mode m(B ⁿ ) is the inter mode of the direct mode, the partitions _P ⁱⁿ of the coding block B ⁿ divided by the block dividing unit 2 are output to the direct vector generating unit 23 .

However, the direct vector generator 23 does not use partition _Pin of coding block ^Bn for direct vector generation. ^Therefore , even if the coding mode m( ^Bn ) is an inter mode of the direct mode, partition _Pin of coding block ^Bn may ^not be output to the direct vector generator 23.

Upon receiving the partition _Pin of the coding block ^Bn from the switch 21, the motion vector search unit 22 of the motion compensation prediction unit 5 searches for an optimal motion vector in the inter-frame mode while referring ^to _the partition ^Pin and the reference image stored in the motion compensation prediction frame memory 12, and outputs the motion vector to the motion compensation processing unit 24.

The process of searching for the optimal motion vector in the inter-frame mode is a well-known technique, and therefore detailed description thereof will be omitted.

When the coding mode m( ^Bn ) is the direct mode, the direct vector generation unit 23 of the motion compensation prediction unit 5 generates a spatial direct mode spatial direct vector and a temporal direct mode temporal direct vector for each partition _P in the coding ^{block Bn} , and outputs either the spatial direct vector or the temporal direct vector as a motion vector to the motion compensation processing unit 24.

Furthermore, as described above, information indicating the partition state of ^the partition _Pin belonging to the coding block ^Bn is included in the coding mode m( ^Bn ), so the direct vector generator 23 can determine the partition _Pin of the coding block ^Bn by referring to the coding mode m( ^{Bn )} .

That is, the spatial direct vector generation unit 31 of the direct vector generation unit 23 reads the motion vectors of the coded blocks located around the partition _P in the coding block ^B from the motion vectors of the coded blocks stored in the motion vector memory (not shown ⁾ or the internal memory, and generates the spatial direct vector of the spatial direct mode based on the motion vectors.

In addition, the time direct vector generation unit 32 of the direct vector generation unit 23 reads out the motion vector of the block that is spatially located at the same position as the partition _P in the coding block B ⁿ , which is the motion vector of the coded picture that the coding block B ⁿ can refer to ^, from the motion vector of the coded block, and generates the time direct vector of the time direct mode based on the motion vector.

Here, FIG. 11 is a schematic diagram illustrating a method of generating a motion vector (temporal direct vector) in the temporal direct mode.

For example, it is assumed that block MB1 in picture B2 is ^a partition _Pin to be coded, and block MB1 is coded in the temporal direct mode.

In this case, the motion vector MV of the block MB2 spatially co-located with the block MB1 is used as the motion vector of the picture P3, which is the picture closest to the picture B2 among the encoded pictures that follow the picture B2 on the time axis.

This motion vector MV refers to the picture P0, and the motion vectors MVL0 and MVL1 used when encoding the block MB1 are obtained by the following equation (3).

After calculating the motion vectors MVL0 and MVL1 , the temporal direct vector generation unit 32 outputs the motion vectors MVL0 and MVL1 to the direct vector determination unit 33 as temporal direct vectors in the temporal direct mode.

The method for generating the time direct vector in the time direct vector generating unit 32 may use the H.264 method as shown in FIG. 11 , but the method is not limited thereto and other methods may be used.

FIG. 12 is a schematic diagram illustrating a method of generating a motion vector (spatial direct vector) in the spatial direct mode.

In FIG12 , current MB (currentMB) indicates a ^partition _Pin of a block to be encoded.

At this time, if the motion vector of the encoded block A on the left of the block of the encoding object is set to MVa, the motion vector of the encoded block B above the block of the encoding object is set to MVb, and the motion vector of the encoded block C above the right of the block of the encoding object is set to MVc, then the motion vector MV of the block of the encoding object can be calculated by finding the median (central value) of these motion vectors MVa, MVb, and MVc as shown in the following formula (4).

MV=median(MVa,MVb,MVc) (4)

In the spatial direct mode, motion vectors are calculated separately for list 0 and list 1, but both can be calculated using the above-mentioned method.

After calculating the motion vectors MV of List 0 and List 1 as described above, the spatial direct vector generator 31 outputs the motion vectors MV of List 0 and List 1 to the direct vector determiner 33 as spatial direct vectors of the spatial direct mode.

The spatial direct vector generation method in the spatial direct vector generation unit 31 may use the H.264 method as shown in FIG. 12 , but is not limited thereto and other methods may be used.

For example, as shown in FIG13 , a space direct vector may be generated using three candidates, one each selected from A1 -An, B1 -Bn, and one selected from C, D, and E, as candidates for the median prediction.

Furthermore, when ref_Idx is different for MV candidates used in generating a space direct vector, scaling may be performed according to the distance in the time direction as shown in FIG. 14 .

Here, scaled_MV represents the vector after scaling, MV represents the motion vector before scaling, and d(x) represents the temporal distance up to x.

In addition, Xr represents the reference image represented by the block to be encoded, and Yr represents the reference image represented by the block position A-D to be scaled.

When the space direct vector generation unit 31 generates the space direct vector, the direct vector determination unit 33 of the direct vector generation unit 23 calculates the evaluation value of the space direct mode using the space direct vector.

Furthermore, when the time direct vector generation unit 32 generates the time direct vector, the direct vector determination unit 33 calculates the evaluation value of the time direct mode using the time direct vector.

The direct vector determination unit 33 compares the evaluation value of the spatial direct mode and the evaluation value of the temporal direct mode, and selects the direct vector of the direct mode from the spatial direct vector or the temporal direct vector through a determination unit as described below and outputs it to the motion compensation processing unit 24.

Hereinafter, the processing contents of the direct vector determination unit 33 will be described in detail.

If the spatial direct vector generation unit 31 generates the spatial direct vectors MVL0 and MVL1, the motion compensation unit 41 of the direct vector determination unit 33 uses the spatial direct vector MVL0 to generate a list 0 prediction image of the spatial direct mode, and uses the spatial direct vector MVL1 to generate a list 1 prediction image of the spatial direct mode.

Here, Figure 15 is an explanatory diagram showing an example of calculating an evaluation value based on the similarity between a forward predicted image and a backward predicted image. In the example of Figure 15, a forward predicted image f _spatial is generated as a list 0 predicted image in spatial direct mode, and a backward predicted image g _spatial is generated as a list 1 predicted image in spatial direct mode.

In addition, if the time direct vector generation unit 32 generates time direct vectors as motion vectors MV of list 0 and list 1, the motion compensation unit 41 uses the time direct vector as the front motion vector MV to generate a predicted image of list 0 of the time direct mode, and uses the time direct vector as the rear motion vector MV to generate a predicted image of list 1 of the time direct mode.

In the example of FIG. 15 , a forward prediction image f _temporal of the temporal direct mode is generated as a list 0 prediction image of the temporal direct mode, and a backward prediction image g _temporal is generated as a list 1 prediction image of the temporal direct mode.

Here, an example is shown in which a forward prediction image is generated as a list 0 prediction image and a backward prediction image is generated as a list 1 prediction image using reference image list 0 representing a reference image in the forward direction and reference image list 1 representing a reference image in the backward direction. However, it is also possible to use reference image list 0 representing a reference image in the backward direction and reference image list 1 representing a reference image in the forward direction to generate a backward prediction image as a list 0 prediction image and a forward prediction image as a list 1 prediction image.

Alternatively, a forward prediction image may be generated using reference image list 0 indicating a forward reference image and reference image list 1 indicating a further forward reference image as list 0 prediction images and list 1 prediction images (described in detail later).

When the list 0 prediction image and list 1 prediction image of the spatial direct mode are generated, the similarity calculation unit 42 of the direct vector determination unit 33 calculates the evaluation value SAD _spatial of the spatial direct mode as shown in the following equation (6).

In equation (6), for convenience of explanation, the list 0 prediction image of the spatial direct mode is set as the forward prediction image f _spatial , and the list 1 prediction image of the spatial direct mode is set as the backward prediction image g _spatial .

SAD _spatial =|f _spatial -g _spatial | (6)

Furthermore, when the list 0 prediction image and the list 1 prediction image of the temporal direct mode are generated, the similarity calculation unit 42 calculates the evaluation value SAD _temporal of the temporal direct mode as shown in the following equation (7).

In equation (7), for convenience of explanation, the list 0 prediction image of the temporal direct mode is set as the forward prediction image f _temporal , and the list 1 prediction image of the temporal direct mode is set as the backward prediction image g _temporal .

SAD _temporal =|f _temporal -g _temporal | (7)

Furthermore, the greater the difference between the forward and backward predicted images, the lower the similarity between the two images (the larger the SAD value, representing the sum of the absolute differences between the two images), and the lower the temporal correlation. Conversely, the smaller the difference between the forward and backward predicted images, the higher the similarity between the two images (the smaller the SAD value, representing the sum of the absolute differences between the two images), and the higher the temporal correlation.

Furthermore, when predicting an image using a direct vector, the prediction should be for an image similar to the block being coded. When generating a predicted image using two vectors, the image predicted by each vector should be similar to the block being coded, indicating a high correlation between the two reference images.

Therefore, by selecting a direct vector with a smaller evaluation value SAD, a mode with a high correlation between reference images can be selected, and the accuracy of the direct mode can be improved.

If the similarity calculation unit 42 calculates the evaluation value SAD _spatial of the spatial direct mode and the evaluation value SAD _{temporal of} the temporal direct mode, the direct vector selection unit 43 of the direct vector determination unit 33 compares the similarity between the forward prediction image f _spatial and the backward prediction image g _spatial in the spatial direct mode, and the similarity between the forward prediction image f _{temporal and the backward prediction image g temporal in} the temporal direct mode by comparing the evaluation value SAD spatial and _the evaluation value SAD _temporal .

When the similarity between the forward prediction image f _spatial and the backward prediction image g _spatial in the spatial direct mode is higher than the similarity between the forward prediction image f _temporal and the backward prediction image g _temporal in the temporal direct mode (SAD _spatial ≤ SAD _temporal ), the direct vector selection unit 43 selects the spatial direct vector generated by the spatial direct vector generation unit 31 and outputs the spatial direct vector as a motion vector to the motion compensation processing unit 24.

On the other hand, when the similarity between the forward prediction image f _temporal and the backward prediction image g _temporal in the temporal direct mode is higher than the similarity between the forward prediction image f _spatial and the backward prediction image g _spatial in the spatial direct mode (SAD _spatial >SAD _temporal ), the temporal direct vector generated by the temporal direct vector generation unit 32 is selected and output as a motion vector to the motion compensation processing unit 24.

When the coding mode m( ^Bn ) is not the direct mode, the motion compensation processing unit 24, upon receiving a motion vector from the motion vector search unit 22, performs motion compensation prediction processing based on the inter-frame prediction parameters output from the coding control unit 1 using the motion vector and a reference image of one frame stored in the motion compensation prediction frame memory 12, thereby generating a predicted image.

On the other hand, when the coding mode m( ^Bn ) is the direct mode, if a motion vector (a direct vector selected by the direct vector selection unit 43) is received from the direct vector generation unit 23, motion compensation prediction processing is performed using the motion vector and a reference image of one frame stored in the motion compensation prediction frame memory 12, based on the inter-frame prediction parameters output from the coding control unit 1, thereby generating a predicted image.

Note that the motion compensation prediction process performed by the motion compensation processing unit 24 is a well-known technique, and therefore detailed description thereof will be omitted.

Here, an example is shown in which the similarity calculation unit 42 calculates the evaluation value SAD as the sum of absolute differences, and the direct vector selection unit 43 compares the evaluation value SAD. However, the similarity calculation unit 42 may calculate the sum of squared differences SSE between the forward prediction image and the backward prediction image as the evaluation value, and the direct vector selection unit 43 may compare the sum of squared differences SSE. Using SSE increases the amount of processing, but allows for more accurate similarity calculation.

Next, the processing contents of the image decoding apparatus in FIG5 will be described.

When the variable length decoding unit 51 receives the bit stream output from the image encoding device of FIG. 1 , it performs variable length decoding processing on the bit stream, decoding the frame size in units of a sequence of pictures of one or more frames or in units of pictures (step ST21 of FIG. 8 ).

The variable length decoding unit 51 determines the maximum size of the coding block that is the processing unit when performing motion compensation prediction processing (inter-frame prediction processing) or intra-frame prediction processing (intra-frame prediction processing) through the same steps as the encoding control unit 1 in Figure 1, and determines the upper limit number of layers when the maximum-sized coding block is hierarchically divided (step ST22).

For example, in an image encoding device, when the maximum size of a coding block is determined according to the resolution of an input image, the maximum size of the coding block is determined according to the size of a previously decoded frame.

Furthermore, when information indicating the maximum size of a coding block and the upper limit of the number of layers is multiplexed in a bit stream, the information decoded from the bit stream is referenced.

The coding mode m(B ⁰ ) of the maximum-sized coding block B ⁰ multiplexed into the bitstream includes information indicating the division state of the maximum-sized coding block B ^0. Therefore, the variable-length decoding unit 51 decodes the coding mode m(B ⁰ ) of the maximum-sized coding block B ⁰ multiplexed into the bitstream to determine the hierarchically divided coding blocks B ⁿ (step ST23).

After determining each coding block B ⁿ , the variable length decoding unit 51 decodes the coding mode m(B ⁿ ) of the coding block B ⁿ and determines the partition _P in belonging to the coding block B ⁿ based on the information of the partition ^P in belonging to the coding mode m( _B ^{n )} .

When the variable length decoding unit 51 identifies the partition _Pin belonging to the ^coding block ^Bn , it decodes the compressed data ^, coding mode, prediction difference coding parameters, and intra prediction parameters/inter prediction parameters for each partition _Pin (step ST24).

When the coding mode m( ^Bn ) of the partition _Pin of the coding ^{block Bn} determined by the variable length decoding unit 51 is the intra coding mode (step ST25), the switch 52 outputs the intra prediction parameters output from the variable length decoding unit 51 to the intra prediction unit 53.

On the other hand, when the coding mode m(B ⁿ ) of ^the partition _Pin is the inter coding mode (step ST25 ), the inter prediction parameters output from the variable length decoding unit 51 are output to the motion compensation prediction unit 54 .

Upon receiving the intra prediction parameters from the switch 52, the intra prediction unit 53 performs intra prediction processing on the partition _Pin of the coding ^{block Bn} using the intra prediction parameters, and generates an intra ^prediction image _Pin (step ST26).

Upon receiving the inter-frame prediction parameters from the switch 52, the motion compensation prediction unit 54 generates a spatial direct vector of the spatial direct mode and a temporal direct vector of the temporal direct mode, similarly to the motion compensation prediction unit 5 in FIG1 , if the coding mode m(B ⁿ ) output from the variable length decoding unit 51 is the inter-frame coding mode of the direct mode.

After the motion compensation prediction unit 54 generates the spatial direct vector of the spatial direct mode and the temporal direct vector of the temporal direct mode, the motion compensation prediction unit 54 selects the direct vector with the highest correlation between the reference images from the spatial direct vector or the temporal direct vector, and performs motion compensation prediction processing on the partition _Pin of the coding block ^Bn using the direct vector and the inter- ^frame prediction parameters to generate the inter-frame prediction ^image _Pin (step ST27).

On the other hand, if the coding mode m( ^Bn ) output from the variable-length decoding unit 51 is an inter-frame coding mode other than the direct mode, the motion compensation processing unit 63 of the motion compensation prediction unit 54 performs motion compensation prediction processing on the partition _Pin of the coding block ^Bn using the motion vector included in the inter-frame prediction parameters output from the switch ⁵² , and generates ^an inter-frame prediction image _Pin (step ST27).

The inverse quantization and inverse transformation unit 55 uses the quantization parameter included in the predicted differential coding parameter output from the variable length decoding unit 51 to inverse quantize the compressed data related to the coding block output from the variable length decoding unit 51, and performs inverse transformation processing (for example, inverse DCT (inverse discrete cosine transform), inverse KL transform, and other inverse transformation processing) on the inverse quantized compressed data in units of the transformation block size included in the predicted differential coding parameter, thereby outputting the compressed data after the inverse transformation processing as a decoded predicted differential signal (a signal representing the differential image before compression) to the addition unit 56 (step ST28).

If the adding unit 56 receives the decoded prediction differential signal from the inverse quantization/inverse transform unit 55, it adds the decoded prediction differential signal and the prediction signal representing the predicted image generated by the intra-frame prediction unit 53 or the motion compensation prediction unit 54 to generate a decoded image, saves the decoded image signal representing the decoded image to the intra-frame prediction memory 57, and outputs the decoded image signal to the loop filter unit 58 (step ST29).

The processes of steps ST23 to ST29 are repeatedly performed until the processes for all the hierarchically divided coding blocks ^Bn are completed (step ST30).

Upon receiving the decoded image signal from the adder 56 , the loop filter unit 58 compensates for coding distortion included in the decoded image signal and stores the decoded image represented by the decoded image signal after coding distortion compensation as a reference image in the motion compensation prediction frame memory 59 (step ST31 ).

The filtering process by the loop filter unit 58 may be performed in units of the largest coding block or each coding block of the decoded image signal output from the adder 56, or may be performed after outputting decoded image signals corresponding to macroblocks of one screen, and then performing the filtering process for one screen.

As described above, according to the first embodiment, the configuration includes: an encoding control unit 1 that determines the maximum size of a coding block as a processing unit when performing prediction processing, and determines the upper limit number of levels when the coding block of the maximum size is hierarchically divided, and selects a coding mode suitable for each hierarchically divided coding block from one or more available coding modes; and a block division unit 2 that divides an input image into coding blocks of the maximum size determined by the encoding control unit 1, and divides the coding blocks hierarchically until the upper limit number of levels determined by the encoding control unit 1 is reached, and selects a coding mode suitable for each coding block divided by the block division unit 2. In the formula, when the inter-frame coding mode of the direct mode is selected by the coding control unit 1, the motion compensation prediction unit 5 generates a spatial direct vector of the spatial direct mode based on the motion vectors of the coded blocks located around the coding block, and generates a temporal direct vector of the temporal direct mode based on the motion vectors of the coded pictures that the coding block can refer to. From the spatial direct vector or the temporal direct vector, the direct vector with a higher correlation between the reference images is selected, and the motion compensation prediction processing is performed on the coding block using the direct vector to generate a predicted image. Therefore, it is possible to select the optimal direct mode in units of specified blocks and reduce the amount of code.

In addition, according to the first embodiment, a variable length decoding unit 51 is provided for variable-length decoding the coded data multiplexed into a bit stream to obtain compressed data and a coding mode associated with each hierarchically divided coding block. When the coding mode associated with the coding block variable-length-decoded by the variable length decoding unit 51 is an inter-frame coding mode of a direct mode, the motion compensation prediction unit 54 generates a spatial direct vector of the spatial direct mode based on the motion vectors of decoded blocks located around the coding block, and generates a temporal direct vector of the temporal direct mode based on the motion vectors of a decoded picture to which the coding block can refer. From the spatial direct vector or the temporal direct vector, a direct vector having a higher correlation between reference images is selected, and motion compensation prediction processing is performed on the coding block using the direct vector to generate a predicted image. This provides an effect of obtaining a motion image decoding device capable of decoding coded data, such that the optimal direct mode can be selected in units of predetermined blocks.

Implementation method 2.

In the above-mentioned first embodiment, an example is shown in which the motion compensated prediction units 5 and 54 (specifically, the similarity calculation unit 42) calculate the similarity between the forward predicted image f _spatial and the backward predicted image g _spatial in the spatial direct mode as the evaluation value SAD _spatial of the spatial direct mode, and calculate the similarity between the forward predicted image f _temporal and the backward predicted image g _temporal in the temporal direct mode as the evaluation value SAD _temporal of the temporal direct mode. However, it is also possible to calculate the variance value σ(spatial) of the motion vectors of the coded blocks (decoded blocks) located around the coding block B ⁿ as the evaluation value of the spatial direct mode, and calculate the variance value σ(temporal) of the motion vectors of the coded blocks (decoded blocks) located around the blocks located at the same spatial position as the coding block B ⁿ in the coded picture (decoded picture) that can be referenced by the coding block B ⁿ as the evaluation value of the temporal direct mode, thereby achieving the same effect as the above-mentioned first embodiment.

That is, in the similarity calculation unit 42, instead of calculating the similarity of the forward predicted image f _spatial and the backward predicted image g _spatial of the spatial direct mode as the evaluation value SAD _spatial of the spatial direct mode, the variance value σ(spatial) of the motion vector of the coded block (decoded block) located around the coding block ^Bn is calculated as shown in Figure 16(a) (refer to the following formula (8)).

In addition, in the similarity calculation unit 42, instead of calculating the similarity of the forward predicted image f _temporal and the backward predicted image g _temporal in the temporal direct mode as the evaluation value SAD _temporal of the ^temporal direct mode, as shown in Figure 16(b), the variance value σ(temporal) of the motion vector of the coded blocks (decoded blocks) around the blocks located at the same spatial position as the coded block B ⁿ in the coded picture (decoded picture) that can be referenced by the coded block B n is calculated (see the following formula (8)).

Here, MV _m,i represents the surrounding motion vectors and represents the average of the surrounding motion vectors.

In addition, m is a symbol indicating spatial or temporal.

The direct vector selection unit 43 compares the variance value σ(spatial) of the motion vector and the variance value σ(temporal) of the motion vector. When the variance value σ(spatial) of the motion vector is greater than the variance value σ(temporal) of the motion vector, it is determined that the reliability of the motion vector of the spatial direct mode (spatial direct vector) is low, and the motion vector of the temporal direct mode (temporal direct vector) is selected.

On the other hand, when the variance value σ(temporal) of the motion vector is greater than the variance value σ(spatial) of the motion vector, it is judged that the reliability of the motion vector of the temporal direct mode (temporal direct vector) is low, and the motion vector of the spatial direct mode (spatial direct vector) is selected.

In the above-mentioned first embodiment, an example is shown in which a time direct vector and a space direct vector are generated and one of the direct vectors is selected. However, in addition to the time direct vector and the space direct vector, other vectors can also be added as candidate vectors, and a direct vector can be selected from these candidate vectors.

For example, space vectors MV_A, MV_B, MV_C and time vectors MV_1 to MV_8 as shown in FIG. 17 may be added to candidate vectors, and a direct vector may be selected from these space vectors and time vectors.

Alternatively, as shown in FIG18 , one vector may be generated from a plurality of encoded vectors and added to the candidate vectors.

In this way, the amount of processing increases by increasing the number of candidate vectors, but the accuracy of the direct vector improves, thereby improving the encoding efficiency.

Although not specifically mentioned in the above-mentioned first embodiment, candidates for direct vectors may be determined in slice units.

Information indicating which vector is a candidate is multiplexed into the slice header.

For example, regarding the time vector, in an image such as a panorama, the effect of the time vector is low, so it is removed from the selection candidate. On the other hand, in an image with a fixed camera, the effect of the space vector is large, so it is added to the candidate.

When there are many candidate vectors, a predicted image closer to the original image can be generated, but the processing load of the encoder increases. Therefore, ineffective vectors are removed from the candidates in advance, and the decision is made based on the locality of the image, so that a balance between processing load and coding efficiency can be achieved.

Regarding the switching of the candidate vectors, for example, a method of setting an ON/OFF flag for each vector and selecting only vectors with the flag turned on as candidates may be considered.

The motion vectors that are candidates for selection may be switched via a slice header or a higher layer such as a sequence header or a picture header. Alternatively, one or more groups of candidate candidates may be prepared and the index of the candidate group may be encoded.

Alternatively, the switching may be performed for each macroblock or coding block. By switching for each macroblock or coding block, localization can be achieved, which has the effect of improving coding efficiency.

Alternatively, selection candidates can be uniquely determined for each partition block size. It is generally believed that as the block size decreases, spatial correlation weakens, and the prediction accuracy of vectors determined by median prediction deteriorates. Therefore, for example, by removing motion vectors determined by median prediction from the candidate list, the processing load can be reduced without reducing coding efficiency.

While the first embodiment described above assumes the presence of both temporal and spatial direct vectors, there may be no motion vectors, such as when intra-frame coding is performed on the coding block ^Bn . In such cases, methods such as setting the vector to zero or removing it from the candidate list are possible.

When the vector is set to zero, the number of candidates increases, so the coding efficiency can be improved, but the processing amount increases. When it is removed from the direct vector candidates, the processing amount can be reduced.

In the above-mentioned first embodiment, an example of generating a direct vector is shown, but the direct vector may be used as a prediction vector used in encoding a normal motion vector.

By using a direct vector as a prediction vector, the amount of processing increases, but the accuracy of prediction improves, so the coding efficiency can be improved.

In the first embodiment described above, an example was shown in which the evaluation value SAD was calculated by combining images that temporally precede and images that temporally follow the coding block ^Bn (see FIG15 ). However, as shown in FIG19 , the evaluation value SAD may be calculated by combining only images that temporally precede. Alternatively, the evaluation value SAD may be calculated by combining only images that temporally follow.

In this case, the time vector is expressed as follows: (9) (10).

Where, is the vector of list 0 and is the vector of list 1.

In addition, d represents a temporal distance, _d0 represents a temporal distance of the reference image in list 0, and _d1 represents a temporal distance of the reference image in list 0.

Furthermore, v _col and d _col represent vectors of blocks at the same spatial position in the reference image, and the temporal distances in the reference image represented by the vectors.

Even when two reference image lists indicate the same reference image, the same method as that shown in FIG. 19 can be applied as long as there are two or more reference images in the list.

In the first embodiment described above, it is assumed that there are two or more reference images in the two reference image lists. However, it is also conceivable that there is actually only one reference image.

In this case, if the same reference image is set in two reference image lists, the temporal vector is not used and only the spatial vector is used for judgment. If different reference images are set, the above method can be used to cope with it.

While the first embodiment above assumes prediction processing from two directions, prediction processing from only one direction is also possible. When predicting from a vector in one direction, information indicating which vector is used is encoded and transmitted. This can address issues such as occlusion and contribute to improved prediction accuracy.

In the direct mode of the above-described first embodiment, prediction using two vectors is assumed, but the number of vectors may be three or more.

In this case, for example, a method is conceivable to generate a predicted image using all candidates having evaluation values SAD equal to or less than a threshold value Th among a plurality of vector candidates.

Alternatively, instead of using all candidates below the threshold Th, the maximum number of vectors to be used may be predetermined in the slice header, etc., and a predicted image may be generated using a number corresponding to the maximum number of vectors starting from candidates with smaller evaluation values.

Generally, it is known that the more reference images used in the predicted image, the better the performance. Therefore, although the amount of processing increases, it contributes to improved coding efficiency.

In the first embodiment described above, the vector is determined by evaluation between reference images, but it may also be evaluated by comparison between a spatially adjacent coded image and a reference image.

In this case, it is considered to use an L-shaped image as shown in FIG. 20 .

Furthermore, when spatially adjacent images are used, the already encoded images may be too late in pipeline processing. In this case, it is considered to use a predicted image instead.

In the above-mentioned first embodiment, an example is shown in which the size of the coding block ^Bn is ^Ln = ^Mn as shown in FIG9 , but the size of the coding block ^Bn may also be ^Ln ≠ ^Mn .

For example, consider the case where the size of the coding block ^Bn is ^Ln = ^kMn as shown in FIG21.

In the next division, (Ln ⁺¹ , Mn ⁺¹ ) = ( ^Ln , ^Mn ). Subsequent divisions may be performed similarly to FIG9 or (Ln ⁺¹ , ^Mn+1 ) = ( ^Ln /2, ^Mn /2) (see FIG22).

Alternatively, as shown in FIG23 , it is possible to select either the partitioning shown in FIG21 or FIG22 . If this is possible, a flag indicating which partitioning is selected is encoded. In this case, this can be achieved simply by horizontally concatenating portions of 16×16 blocks, as in H.264 of Non-Patent Document 1, maintaining compatibility with existing methods.

In the above description, the case where the size of the coding block ^Bn is ^Ln = ^kMn is shown. However, even if the coding block Bn is vertically concatenated as ^kLn = ^Mn , it can of course be divided based on the same considerations.

In the first embodiment described above, an example was shown in which the transform/quantization unit 7 and the inverse quantization/inverse transform unit 8 and 55 perform transform processing (inverse transform processing) using the transform block size included in the prediction difference coding parameter as a unit. However, the transform block size unit may be uniquely determined by the transform processing unit, or a hierarchical structure may be provided as shown in FIG24 . In this case, a flag indicating whether or not to perform division in each hierarchy is encoded.

The above segmentation can be performed in units of partitions or coding blocks.

The above transformation assumes transformation under a square, but it can also be transformation under other rectangles such as a rectangle.

Implementation method 3.

In the above-mentioned embodiment 1, an example is shown in which the direct vector generation units 23 and 62 of the motion compensation prediction units 5 and 54 generate spatial direct vectors and temporal direct vectors. However, when generating the spatial direct vectors and temporal direct vectors, the direct vectors can also be determined by determining an initial search point and searching around the initial search point.

FIG25 is a block diagram showing the motion compensation prediction unit 5 of the moving picture encoding device according to Embodiment 3 of the present invention. In the figure, the same reference numerals as those in FIG2 denote the same or corresponding parts, and therefore their description is omitted.

The direct vector generating unit 25 performs processing to generate a spatial direct vector and a temporal direct vector.

FIG26 is a diagram showing the structure of the direct vector generation unit 25 constituting the motion compensation prediction unit 5 .

In FIG. 26 , the initial vector generation unit 34 performs processing for generating an initial vector based on the motion vector of the coded block.

The motion vector search unit 35 performs a process of determining a direct vector by searching around an initial search point indicated by the initial vector generated by the initial vector generation unit 34 .

FIG. 27 is a diagram showing the configuration of the initial vector generating section 34 constituting the direct vector generating section 25 .

In FIG27 , a spatial vector generator 71 performs processing for generating a spatial vector from a motion vector of an encoded block by the same method as, for example, the spatial direct vector generator 31 in FIG3 .

The temporal vector generator 72 performs a process of generating a temporal vector from the motion vector of the coded block by the same method as the temporal direct vector generator 32 in FIG. 3 , for example.

The initial vector determination unit 73 performs a process of selecting one of the space vector generated by the space vector generation unit 71 and the time vector generated by the time vector generation unit 72 as the initial vector.

FIG. 28 is a diagram showing the configuration of the initial vector determination unit 73 constituting the initial vector generation unit 34 .

In Figure 28, the motion compensation unit 81 implements: using the same method as the motion compensation unit 41 in Figure 4, the processing of generating the list 0 prediction image of the spatial direct mode, the list 1 prediction image of the spatial direct mode, the list 0 prediction image of the temporal direct mode, and the list 1 prediction image of the temporal direct mode is implemented.

The similarity calculation unit 82 performs the following processing: using the same method as the similarity calculation unit 42 in Figure 4, the similarity between the list 0 prediction image and the list 1 prediction image of the spatial direct mode is calculated as a spatial evaluation value, and the similarity between the list 0 prediction image and the list 1 prediction image of the temporal direct mode is calculated as a temporal evaluation value.

The initial vector determination unit 83 performs processing for comparing the spatial evaluation value and the temporal evaluation value calculated by the similarity calculation unit 82 and selecting a spatial vector or a temporal vector according to the comparison result.

FIG29 is a block diagram showing the motion compensation prediction unit 54 of the moving picture decoding device according to Embodiment 3 of the present invention. In the figure, the same reference numerals as those in FIG6 denote the same or corresponding parts, and therefore their description is omitted.

The direct vector generating unit 64 performs processing to generate a spatial direct vector and a temporal direct vector.

Note that the internal structure of the direct vector generating unit 64 is the same as that of the direct vector generating unit 25 in FIG. 25 .

Next, the operation will be explained.

Compared to the first embodiment, the present invention is the same except that the direct vector generators 25 and 64 are used instead of the direct vector generators 23 and 62 of the motion compensation prediction units 5 and 54 . Therefore, only the processing contents of the direct vector generators 25 and 64 will be described.

Since the processing contents of the direct vector generating units 25 and 64 are the same, the processing contents of the direct vector generating unit 25 will be described.

The initial vector generation unit 34 of the direct vector generation unit 25 generates an initial vector MV_first based on the motion vector of the coded block.

That is, the spatial vector generator 71 of the initial vector generator 34 generates a spatial vector from the motion vector of the coded block using the same method as the spatial direct vector generator 31 in Figure 3. However, the spatial vector may be generated using other methods.

The temporal vector generator 72 of the initial vector generator 34 generates a temporal vector from the motion vector of the coded block using the same method as the temporal direct vector generator 32 in Fig. 3. However, the temporal vector may be generated using other methods.

When the space vector generator 71 generates a space vector and the time vector generator 72 generates a time vector, the initial vector determination unit 73 of the initial vector generator 34 selects either the space vector or the time vector as the initial vector MV_first.

That is, the motion compensation unit 81 of the initial vector determination unit 73 generates the list 0 prediction image of the spatial direct mode, the list 1 prediction image of the spatial direct mode, the list 0 prediction image of the temporal direct mode, and the list 1 prediction image of the temporal direct mode by the same method as the motion compensation unit 41 of Figure 4.

The similarity calculation unit 82 of the initial vector determination unit 73 calculates the similarity between the list 0 prediction image and the list 1 prediction image of the spatial direct mode as a spatial evaluation value, and calculates the similarity between the list 0 prediction image and the list 1 prediction image of the temporal direct mode as a temporal evaluation value, using the same method as the similarity calculation unit 42 of Figure 4.

The initial vector determination unit 83 of the initial vector determination unit 73 refers to the comparison result of the spatial evaluation value and the temporal evaluation value calculated by the similarity calculation unit 82 , and selects a vector having a higher similarity between the predicted images, either the spatial vector or the temporal vector.

When the initial vector generating unit 34 generates the initial vector MV_first, the motion vector search unit 35 of the direct vector generating unit 25 searches the range of ±n around the initial search point (block) represented by the initial vector MV_first as shown in Figure 30, thereby determining the direct vector.

The evaluation during the search may be performed by, for example, the same processing as that of the similarity calculation unit 82 in Fig. 28. In this case, assuming that the position indicated by the initial vector is v, the search evaluation value SAD is calculated as shown in the following equation (11).

SAD＝|f(v ₁ -x)-g(v ₂ +x)| (11)

The search range n can be fixed or determined by a higher-level header such as a slice header. In addition, the distribution range of the search points (search range) is assumed to be a square, but it can also be a rectangle, a diamond, etc.

After calculating the search evaluation value SAD, the motion vector search unit 35 outputs the motion vector with the minimum evaluation value SAD within the search range as a direct vector to the motion compensation processing unit 24 .

In this embodiment 3, an example is shown in which a time vector and a space vector are generated and one of the vectors is selected as the initial vector. However, in addition to the time vector and the space vector, other vectors can also be added as candidate vectors and the initial vector can be selected from these candidate vectors.

For example, space vectors MV_A, MV_B, MV_C and time vectors MV_1 to MV_8 as shown in FIG. 17 may be added to candidate vectors, and an initial vector may be selected from these space vectors and time vectors.

Alternatively, as shown in FIG18 , one vector may be generated from a plurality of encoded vectors and added to the candidate vectors.

In this way, the amount of processing increases by increasing the number of candidate vectors, but the accuracy of the initial vector improves, and the encoding efficiency can be improved.

In this third embodiment, direct vector candidates may be determined in slice units.

Information indicating which vector is a candidate is multiplexed into the slice header.

For example, regarding the time vector, in an image such as a panoramic shot, the effect of the time vector is low, so it is removed from the selection candidate. On the other hand, in an image with a fixed camera, the effect of the time vector is large, so it is added to the candidate.

When there are many candidate vectors, a predicted image closer to the original image can be generated, but the processing load of the encoder increases. Therefore, ineffective vectors are removed from the candidates in advance, and the decision is made based on the locality of the image, so that a balance between processing load and coding efficiency can be achieved.

Regarding the switching of the vectors to be candidates, for example, a method of setting an ON/OFF flag for each vector and using only the vectors with the flag turned on as candidates may be considered.

The motion vectors that are candidates for selection may be switched via a slice header or a higher layer such as a sequence header or a picture header. Alternatively, one or more groups of candidate candidates may be prepared and the index of the candidate group may be encoded.

Alternatively, the switching may be performed for each macroblock or coding block. By performing the switching for each macroblock or coding block, localization can be achieved, which has the effect of improving coding efficiency.

Alternatively, selection candidates can be uniquely determined for each partition block size. It is generally believed that as the block size decreases, spatial correlation weakens, and the prediction accuracy of vectors determined by median prediction deteriorates. Therefore, for example, by removing motion vectors determined by median prediction from the candidate list, the processing load can be reduced without reducing coding efficiency.

In this third embodiment, the description assumes the presence of both a temporal vector and a spatial vector. However, there may be cases where no motion vector exists, such as when intra-frame coding is performed on coding block ^Bn . In such cases, methods such as setting the vector to zero or removing it from the candidate list are possible.

When the vector is set to zero, the number of candidates increases, so the coding efficiency can be improved, but the processing amount increases. When it is removed from the direct vector candidates, the processing amount can be reduced.

In this third embodiment, an example of generating a direct vector is shown, but the direct vector can also be used as a prediction vector used in encoding a normal motion vector.

By using a direct vector as a prediction vector, the amount of processing increases, but the accuracy of prediction improves, so the coding efficiency can be improved.

In this third embodiment, an example was shown in which the evaluation value SAD was calculated by combining images that temporally precede and images that temporally follow the coding block ^Bn (see FIG15 ). However, as shown in FIG19 , the evaluation value SAD may be calculated by combining only images that temporally precede. Alternatively, the evaluation value SAD may be calculated by combining only images that temporally follow.

In this case, the time vector is expressed as follows: (12) (13).

Where, is the vector of list 0 and is the vector of list 1.

In addition, d represents the temporal distance, _d0 represents the temporal distance of the reference image in list 0, and _d1 represents the temporal distance of the reference image in list 0.

Furthermore, v _col and d _col represent vectors of blocks at the same spatial position in the reference image, and the temporal distances in the reference image represented by the vectors.

Even when two reference image lists indicate the same reference image, the same method as that shown in FIG. 19 can be applied.

In this third embodiment, it is assumed that there are two or more reference images in two reference image lists, but it is also possible to consider the case where there is actually only one reference image.

In this case, if the same reference image is set in two reference image lists, the temporal vector is not used and only the spatial vector is used for judgment. If different reference images are set, the above method can be used to cope with it.

While this third embodiment assumes prediction processing from two directions, prediction processing from only one direction is also possible. When predicting from a vector in one direction, information indicating which vector is used is encoded and transmitted. This can address issues such as occlusion and contribute to improved prediction accuracy.

In this third embodiment, prediction using two vectors is assumed, but the number of vectors may be three or more.

In this case, for example, a method of generating a predicted image using all candidates having evaluation values SAD equal to or less than a threshold value Th among a plurality of vector candidates is conceivable.

Alternatively, instead of using all candidates below the threshold Th, the maximum number of vectors to be used may be predetermined in the slice header, etc., and a predicted image may be generated using a number corresponding to the maximum number of vectors starting from candidates with smaller evaluation values.

In this third embodiment, the vector is determined by evaluation between reference images, but it may also be evaluated by comparison between a spatially adjacent coded image and a reference image.

In this case, it is considered to use an L-shaped image as shown in FIG. 20 .

Furthermore, when spatially adjacent images are used, the already encoded images may be too late in pipeline processing. In this case, it is considered to use a predicted image instead.

In this third embodiment, after the initial vector is determined, the motion vector is searched. However, whether or not to search the motion vector may be determined using a flag in units of slices.

In this case, although the coding efficiency is reduced, it has the effect of significantly reducing the amount of processing.

The flag may be determined in slice units or in a higher layer such as a sequence or a picture. When the flag is OFF and motion search is not performed, the same operation as in the first embodiment is performed.

In this third embodiment, it is assumed that the direct vector generating units 25 and 64 perform processing independently of the block size, but this processing may be limited to the case where the block size is equal to or smaller than a predetermined size.

A flag indicating whether to limit the size to a predetermined size or less, or information indicating which block size is set to be smaller than or equal to a predetermined size, may be multiplexed in an upper header such as a slice, or may vary according to the maximum CU size.

As the block size decreases, the correlation between reference images decreases, and the error tends to increase. Therefore, regardless of which vector is selected, performance is often not significantly affected. By turning off processing for large block sizes, the amount of processing can be reduced without degrading encoding performance.

Implementation method 4.

In the above-mentioned embodiment 1, an example is shown in which the motion compensation prediction units 5 and 54 generate spatial direct vectors of the spatial direct mode based on the motion vectors of the coded blocks (decoded blocks) located around the coding block, and generate temporal direct vectors of the temporal direct mode based on the motion vectors of the coded pictures (decoded pictures) to which the coding block can refer, and select the direct vector with the higher correlation between the reference pictures from the spatial direct vectors or the temporal direct vectors. However, it is also possible to select a motion vector suitable for generating a predicted image from one or more selectable motion vectors in the motion compensation prediction unit 5 of the motion picture coding device, use the motion vector to perform motion compensation prediction processing on the coding block to generate a predicted image, and output index information representing the motion vector to the variable length coding unit 13.

On the other hand, the motion compensation prediction unit 54 of the moving picture decoding device may generate a predicted image by performing a motion compensation prediction process on the coding block using the motion vector indicated by the index information multiplexed into the bitstream.

FIG31 is a diagram showing the structure of the motion compensation prediction unit 5 of the moving picture encoding device according to the fourth embodiment of the present invention. In the figure, the same reference numerals as those in FIG2 denote the same or corresponding parts, and therefore their description is omitted.

The direct vector generation unit 26 performs the following processing: referring to the direct vector candidate index that records the selectable motion vector and the index information representing the motion vector, selects a motion vector suitable for generating a predicted image from one or more selectable motion vectors, outputs the motion vector as a direct vector to the motion compensation processing unit 24, and outputs the index information representing the motion vector to the variable length coding unit 13.

Furthermore, when performing variable-length coding on compressed data, coding modes, and the like, the variable-length coding unit 13 includes index information thereof in inter-frame prediction parameters and performs variable-length coding.

FIG32 is a block diagram showing the motion compensation prediction unit 54 of the moving picture decoding apparatus according to Embodiment 4 of the present invention. In the figure, the same reference numerals as those in FIG6 denote the same or corresponding parts, and therefore their description is omitted.

The direct vector generation unit 65 performs the following processing: inputs a direct vector candidate index that records a selectable motion vector and index information representing the motion vector, reads out the motion vector represented by the index information included in the inter-frame prediction parameter from the direct vector candidate index, and outputs the motion vector as a direct vector to the motion compensation processing unit 63.

Next, the operation will be explained.

Compared to the first embodiment, the present invention is the same except that the direct vector generators 23 and 62 of the motion compensation prediction units 5 and 54 are replaced by direct vector generators 26 and 65 , so only the processing contents of the direct vector generators 26 and 65 will be described.

The direct vector generation unit 26 of the motion compensation prediction unit 5 generates a direct ^vector for each partition _Pin of the coding block ^Bn when the coding mode m( ^Bn ) is the direct mode.

That is, the direct vector generation unit 26 refers to the direct vector candidate index shown in FIG. 33 and selects a motion vector suitable for generating a predicted image from one or more selectable motion vectors.

In the example of FIG. 33 , five motion vectors are listed as selectable motion vectors. However, in spatial prediction, “median” is most often selected, and therefore, index 0 is assigned to “median”.

When selecting a motion vector suitable for generating a predicted image, the direct vector generation unit 26 calculates the cost R based on the distortion between the predicted image and the original image obtained from the selectable motion vector, and the index code amount of the selectable motion vector, as shown in the following formula (14), and selects the motion vector with the smallest cost R from among multiple motion vectors.

Here, D represents the residual signal between the predicted image and the original image, i represents the index, λ represents the Lagrange multiplier, and () represents the amount of code for the portion shown in ().

When the direct vector generation unit 26 selects the motion vector with the minimum cost R, it outputs the motion vector as a direct vector to the motion compensation processing unit 24 and outputs index information indicating the motion vector to the variable-length coding unit 13 .

For example, if “median” is selected as the motion vector with the minimum cost R, index 0 is output to the variable length coding unit 13 , and if “MV_A” is selected, index 1 is output to the variable length coding unit 13 .

Upon receiving the index information from the direct vector generation unit 26 , the variable length coding unit 13 includes the index information in the inter-frame prediction parameters when performing variable length coding on compressed data, coding mode, and the like.

When the coding mode m(B ⁿ ) is the direct mode, the direct vector generation unit 65 of the motion compensation prediction unit 54 generates a direct vector for each partition _Pin of ^the coding block B ⁿ .

That is, if the direct vector generation unit 65 inputs the same direct vector candidate index as the direct vector generation unit 26 in Figure 31 (for example, the direct vector candidate index in Figure 33) and receives the inter-frame prediction parameter including index information from the switching switch 61, it reads the motion vector represented by the index information from the direct vector candidate index and outputs the motion vector as a direct vector to the motion compensation processing unit 63.

For example, if the index information is index 0, "median" is output as a direct vector, and if the index information is index 1, "MV_A" is output as a direct vector.

As described above, according to this embodiment 4, a motion vector suitable for generating a predicted image is selected from one or more selectable motion vectors, and motion compensation prediction processing is performed on the coding block using the motion vector to generate a predicted image, and index information representing the motion vector is output to the variable length coding unit 13. Therefore, similar to the above-mentioned embodiment 1, it has the effect of being able to select the best direct mode in units of specified blocks to reduce the amount of code.

In this fourth embodiment, the description assumes that there is a motion vector at a selectable position. However, there may be no motion vector, such as when intra-frame coding is performed on coding block ^Bn . In such cases, methods such as setting the vector to zero or removing it from the candidate list are possible.

When the vector is set to zero, the number of candidates increases, so the coding efficiency can be improved, but the processing amount increases. When it is removed from the direct vector candidates, the processing amount can be reduced.

In this fourth embodiment, an example of generating a direct vector is shown, but this vector can also be used as a prediction vector used in encoding a normal motion vector.

By using it as a prediction vector, the amount of processing increases, but the prediction accuracy improves, so the coding efficiency can be improved.

In this fourth embodiment, the candidates for selectable motion vectors are fixed, but the candidates for selectable motion vectors may be determined in units of slices.

Information indicating which vector is a candidate is multiplexed into the slice header.

For example, regarding the time vector, in an image such as a panoramic shot, the effect of the time vector is low, so it is removed from the selection candidate. On the other hand, in an image with a fixed camera, the effect of the time vector is large, so it is added to the candidate.

When there are many candidate vectors, a predicted image closer to the original image can be generated, but the processing load of the encoder increases. Therefore, ineffective vectors are removed from the candidates in advance, and the decision is made based on the locality of the image, so that a balance between processing load and coding efficiency can be achieved.

Regarding the switching of the vectors to be candidates, for example, a method of setting an ON/OFF flag for each vector and using only the vectors with the flag turned on as candidates may be considered.

The motion vectors that are candidates for selection may be switched via a slice header or a higher layer such as a sequence header or a picture header. Alternatively, one or more groups of candidate candidates may be prepared and the index of the candidate group may be encoded.

Alternatively, the switching may be performed for each macroblock or coding block. By switching for each macroblock or coding block, localization can be achieved, thereby improving coding efficiency.

In this fourth embodiment, the order of the indices is fixed, but the order of the indices can be changed per slice. If there is a discrepancy among the vectors selected by the slice, the coding efficiency can be improved by switching the index table so that shorter codes are assigned to vectors that are selected more frequently.

Regarding the coding of the switching information, the order may be coded for each vector, or a plurality of index groups may be prepared and information indicating which index group is to be used may be coded.

Another method is to determine only the default setting, prepare a flag indicating whether to use a setting different from the default setting, and update the index group to switch the setting only when the flag is on.

Here, an example of switching the order of indexes in units of slices is shown, but of course the order of indexes can also be determined by a higher layer such as a sequence or a picture.

Alternatively, the switching can be performed for each partition block or coding block. By switching for each macroblock or coding block, localization can be achieved, thereby improving coding efficiency.

Alternatively, selection candidates can be uniquely determined for each partition block size. It is generally believed that as the block size decreases, spatial correlation weakens, leading to a decrease in the prediction accuracy of vectors determined by median prediction. Therefore, by changing the order of indices determined by median prediction, coding efficiency can be improved.

In this fourth embodiment, five direct vector candidate indices of selectable motion vectors are prepared. However, six or more motion vectors may be prepared as candidate vectors, or fewer than five motion vectors may be prepared as candidate vectors.

For example, a vector near the time vector as shown in FIG. 17 or a vector obtained by weighted addition of surrounding vectors as shown in FIG. 18 may be added as a candidate vector.

While this fourth embodiment assumes prediction processing from two directions, prediction processing from only one direction is also possible. When predicting from a vector in one direction, information indicating which vector is used is encoded and transmitted. This can address issues such as occlusion and contribute to improved prediction accuracy.

In this fourth embodiment, two-directional prediction using two vectors is assumed, but the number of vectors may be 3 or more. In this case, for example, index information indicating all selected vectors may be encoded, or conversely, index information indicating non-selected vectors may be encoded.

Alternatively, as shown in FIG. 34 , a method may be considered in which only index information of one vector is encoded and an image close to the reference image represented by the vector is used.

In this fourth embodiment, an example is shown in which a motion vector having the smallest cost R is selected from a plurality of motion vectors. However, an evaluation value SAD _k may be calculated as shown in the following equation (15), and a motion vector having an evaluation value SAD _k equal to or less than a threshold value Th may be selected.

Here, f _index represents a reference image represented by a vector encoding index information, and g _k represents a reference image represented by a vector MV_k.

Here, an example using the evaluation value SAD _k is shown, but it is of course possible to perform evaluation using other methods such as SSE.

Information indicating the number of vectors used may be multiplexed into an upper header in slice units, etc. Increasing the number of vectors improves coding efficiency, but increases the amount of processing, so there is a trade-off.

Alternatively, the unit may be set in a finer unit such as a coding block or partition instead of a slice unit. In this case, a balance between the amount of processing and the coding efficiency can be achieved according to the locality of the image.

While this fourth embodiment shows an example of selecting a motion vector suitable for generating a predicted image from a plurality of selectable motion vectors, it is also possible, as in the third embodiment, to select a motion vector to be used as an initial vector from a plurality of selectable motion vectors and then search around the initial vector to determine a final motion vector. The structure of the direct vector generator 26 in this case is shown in FIG35 .

The initial vector generation unit 36 in FIG35 corresponds to the initial vector generation unit 34 in FIG26 .

Implementation method 5.

The motion compensation prediction units 5 and 54 of this embodiment 5 have the functions of the above-mentioned embodiment 1 (or, embodiments 2 and 3) and the functions of the above-mentioned embodiment 4, can switch the functions of the above-mentioned embodiment 1 (or, embodiments 2 and 3) and the functions of the above-mentioned embodiment 4 in units of slices, and can use any one of the functions to generate a predicted image.

FIG36 is a diagram showing the structure of the motion compensation prediction unit 5 of the moving picture encoding device according to the fifth embodiment of the present invention. In the figure, the same reference numerals as those in FIG31 denote the same or corresponding parts, and therefore description thereof will be omitted.

The direct vector generation unit 27 performs the following processing: when the direct mode switching flag indicates that index information is not to be transmitted, a direct vector is generated by the same method as the direct vector generation unit 23 in Figure 2 (or the direct vector generation unit 25 in Figure 25). On the other hand, when the direct mode switching flag indicates that index information is to be transmitted, a direct vector is generated by the same method as the direct vector generation unit 26 in Figure 31, and index information representing the direct vector is output to the variable length coding unit 13.

Furthermore, the direct vector generation unit 27 performs a process of outputting a direct mode switching flag to the variable length coding unit 13 .

FIG37 is a diagram showing the structure of the direct vector generation unit 27 constituting the motion compensation prediction unit 5. As shown in FIG37 , the direct vector generation unit 27 includes ...

In FIG37 , the switching switch 91 performs the following processing: when the direct mode switching flag indicates that index information is not to be sent, ^the partition _Pin of the coding block ^Bn is output to a portion corresponding to the direct vector generation unit 23 in FIG2 (or the direct vector generation unit 25 in FIG25 ); and when the flag indicates that index information is to be sent, ^the partition _Pin of the coding block ^Bn is output to a portion corresponding to the direct vector generation unit 26 in FIG31 .

FIG38 is a diagram showing the structure of the motion compensation prediction unit 54 of the moving picture decoding device according to Embodiment 5 of the present invention. In the figure, the same reference numerals as those in FIG32 denote the same or corresponding parts, and therefore description thereof will be omitted.

The direct vector generation unit 66 performs the following processing: when the direct mode switching flag included in the inter-frame prediction parameter indicates that index information is not to be transmitted, a direct vector is generated by the same method as the direct vector generation unit 62 in Figure 6 (or the direct vector generation unit 64 in Figure 29). On the other hand, when the direct mode switching flag indicates that index information is to be transmitted, a direct vector is generated by the same method as the direct vector generation unit 65 in Figure 32.

Next, the operation will be explained.

The direct vector generation unit 27 of the motion compensation prediction unit 5 has the functions of the direct vector generation unit 23 in Figure 2 (or the direct vector generation unit 25 in Figure 25) and the functions of the direct vector generation unit 26 in Figure 31. When the direct mode switching flag input from the outside indicates that the index information is not to be sent, the direct vector is generated by the same method as the direct vector generation unit 23 in Figure 2 (or the direct vector generation unit 25 in Figure 25) and outputs the direct vector to the motion compensation processing unit 24.

Furthermore, the direct vector generation unit 27 outputs the direct mode switching flag to the variable length coding unit 13 .

When the direct mode switching flag indicates that index information is to be transmitted, the direct vector generation unit 27 generates a direct vector using the same method as the direct vector generation unit 65 in FIG. 32 , and outputs the direct vector to the motion compensation processing unit 24 .

Furthermore, the direct vector generation unit 27 outputs the direct mode switching flag and index information to the variable length coding unit 13 .

Upon receiving the direct mode switching flag from the direct vector generator 27 , the variable length coding unit 13 includes the direct mode switching flag in the inter-frame prediction parameters when variable length coding is performed on compressed data, coding mode, etc.

In addition, if the variable length coding unit 13 receives the direct mode switching flag and index information from the direct vector generation unit 27, it includes the direct mode switching flag and index information in the inter-frame prediction parameters when performing variable length coding on compressed data, coding mode, etc.

When the direct vector generation unit 66 of the motion compensation prediction unit 54 receives the inter-frame prediction parameters decoded by the variable length decoding unit 51, if the direct mode switching flag included in the inter-frame prediction parameters indicates that index information is not to be sent, the direct vector is generated by the same method as the direct vector generation unit 62 in Figure 6 (or the direct vector generation unit 64 in Figure 29).

On the other hand, when the direct mode switching flag indicates that the index information is to be transmitted, a direct vector is generated by the same method as the direct vector generating unit 65 in FIG. 32 .

Generally, the mode that transmits index information has more additional information than the mode that does not transmit index information. Therefore, in the case of low bit rates, where the proportion of additional information in the total code amount is large, the mode that does not transmit index information has better performance.

On the other hand, in cases where the ratio of additional information to the total code amount is small, such as in the case of high rate, improvement in coding efficiency can be expected by adding index information and using the optimal direct vector.

In this fifth embodiment, an example is shown in which the direct mode switching flag is included in the inter-frame prediction parameters, but the direct mode switching flag may be multiplexed in a slice header or in a picture or sequence header.

In addition, it is also considered to determine the switching by the partition size.

Generally, as the partition size increases, the proportion of additional information such as motion vectors decreases. Therefore, a configuration is considered in which a mode of transmitting index information is selected for sizes above a certain value, and a mode of not transmitting index information is selected for sizes below that value.

As described above, when the size is determined by partitioning, a flag indicating which size to use may be multiplexed into a higher-level header such as a slice header for each coding block size.

In this embodiment 4, an example of switching the function of the above-mentioned embodiment 1 and the function of the above-mentioned embodiment 4 through a direct mode switching flag is shown, but the function of the above-mentioned embodiment 2 and the function of the above-mentioned embodiment 4 can also be switched, and the function of the above-mentioned embodiment 3 and the function of the above-mentioned embodiment 4 can also be switched.

In addition, the function of the first embodiment and the function of the second embodiment may be switched, the function of the first embodiment and the function of the third embodiment may be switched, or the function of the second embodiment and the function of the third embodiment may be switched.

In addition, any function may be selected from the functions of the above-mentioned first to fourth embodiments.

While this fifth embodiment shows an example of switching between the functions of the first and fourth embodiments described above using a direct mode switching flag, it is also possible to use the flag as an on/off switch, rather than a switching flag. For example, consider whether the on/off flag of the first embodiment is used. If the flag is on, both the first and fourth embodiments are performed, and the mode with the higher coding efficiency is selected, and this information is encoded. This allows switching between direct modes based on the locality of the image, contributing to improved coding efficiency.

In the above description, the first embodiment is turned on/off, but the fourth embodiment may be turned on/off. In addition, a combination of the second and fourth embodiments or the third and fourth embodiments may be used.

While this fifth embodiment shows an example of selecting a motion vector suitable for generating a predicted image from a plurality of selectable motion vectors, it is also possible to select a motion vector to be used as an initial vector from a plurality of selectable motion vectors, and then search around the initial vector to determine a final motion vector, as in the third embodiment described above. The structure of the direct vector generator 27 in this case is shown in FIG.

The initial vector generation unit 37 in FIG39 corresponds to the initial vector generation unit 34 in FIG26 .

Furthermore, the present invention can freely combine the various embodiments, modify any components of the various embodiments, or omit any components of the various embodiments within the scope of the invention.

For example, it is recorded as determining the maximum size and the upper limit of the number of layers when the coding block of the maximum size is hierarchically divided, and selecting a coding mode suitable for each hierarchically divided coding block from one or more available coding modes, but the maximum size or the number of layers, or one or all of the coding modes can also be predetermined in advance.

Implementation method 6.

In the above-mentioned embodiment 4, an example is shown in which the direct vector generation unit 26 of the motion compensation prediction unit 5 in the motion image encoding device refers to the direct vector candidate index as shown in Figure 33 to grasp one or more selectable motion vectors, but the encoding control unit 1 can also generate a list of one or more selectable motion vectors based on the block size of the encoding block, and refer to the direct vector candidate list and the direct vector candidate index representing one or more selectable motion vectors to determine the direct mode vector.

Specifically, it is as follows.

As described above, one or more selectable motion vectors can be uniquely determined for each block size of a partition, for example. However, as shown in FIG40 , when the block size of the partition serving as the coding block is large, the correlation with the surrounding blocks is high, whereas when the block size of the partition is small, the correlation with the surrounding blocks is low.

Therefore, the smaller the block size of the partition, the smaller the number of selectable motion vector candidates can be.

Therefore, as shown in FIG. 41 , the encoding control unit 1 lists one or more selectable motion vectors in advance for each block size of a partition serving as a coding block.

As can be seen from Figure 41, the smaller the block size of the partition, the smaller the number of selectable motion vector candidates. For example, in a partition with a block size of "64", the number of selectable motion vectors is "4", but in a partition with a block size of "8", the number of selectable motion vectors is "2".

“median”, “MV_A”, “MV_B”, “MV_C”, and “temporal” in FIG42 correspond to “median”, “MV_A”, “MV_B”, “MV_C”, and “temporal” in FIG33 .

When determining one or more selectable motion vectors, the encoding control unit 1, for example, refers to the list of Figure 41 to determine one or more motion vectors corresponding to the block size of the partition serving as the encoding object, and outputs the direct vector candidate list representing the one or more motion vectors to the motion compensation prediction unit 5.

For example, when the block size of the partition is "64", "MV_A", "MV_B", "MV_C", and "temporal" are determined as one or more selectable motion vectors.

In addition, when the block size of the partition is "8", "median" and "temporal" are determined as one or more selectable motion vectors.

If the direct vector generation unit 26 of the motion compensation prediction unit 5 receives the direct vector candidate list from the encoding control unit 1, it selects a motion vector suitable for generating the predicted image from one or more motion vectors represented by the direct vector candidate list, similar to the above-mentioned embodiment 4. However, when the block size of the partition is small, the number of selectable motion vector candidates is small, so, for example, the number of calculations of the evaluation value SAD _k shown in the above-mentioned formula (15) is reduced, and the processing load of the motion compensation prediction unit 5 is reduced.

In this way, when the encoding control unit 1 of the moving picture encoding device determines one or more selectable motion vectors, the moving picture decoding device also needs to have the same one or more selectable direct vector candidate lists as the moving picture encoding device.

When the coding mode m( ^Bn ) is the direct mode, the variable length decoding unit 51 of the moving picture decoding apparatus ^outputs , for each partition _P in the coding block ^Bn , the block size of the partition to the motion compensation prediction unit 54, and outputs index information (information indicating a motion vector used in the motion compensation prediction unit 5 of the moving picture coding apparatus) obtained by variable length decoding from the bit stream to the motion compensation prediction unit 54.

Upon receiving the partition block size from the variable length decoding unit 51, the direct vector generation unit 65 of the motion compensation prediction unit 54 inputs a direct vector index as in the fourth embodiment described above, and outputs a motion vector used in the direct mode from one or more predetermined motion vector candidate lists according to the block size.

That is, the direct vector generation unit 65 lists one or more selectable motion vectors for each block size of the partition in advance (refer to Figure 41). When determining one or more selectable motion vectors, it refers to the list of Figure 41 and the direct vector index to output one or more motion vectors corresponding to the block size of the partition to be decoded this time.

For example, when the block size of the partition is "8", if the index information is index 0, "median" is output as a direct vector, and if the index information is index 1, "temporal" is output as a direct vector.

As described above, according to this embodiment 6, one or more selectable motion vectors are determined based on the block size of the partition serving as the coding block. Therefore, in a partition having low correlation with surrounding blocks, motion vectors other than motion vectors suitable for generating a predicted image can be removed from the candidates, thereby achieving the effect of reducing the amount of processing.

In addition, according to the sixth embodiment, when determining one or more selectable motion vectors, the smaller the block size of the partition, the smaller the number of selectable motion vector candidates, so that motion vectors other than motion vectors suitable for generating a predicted image can be removed from the candidates, thereby having the effect of reducing the amount of processing.

In addition, in this embodiment 6, an example is shown in which the maximum block size of the partitions as the coding blocks is "64", but the maximum block size may be larger than 64 or smaller than 64.

FIG. 42 shows an example of a list in which the maximum block size is "128".

In the example of Figure 42, the maximum block size in the list maintained by the encoding control unit 1 and the motion compensation prediction unit 54 is "128", but when the maximum value of the block size of the actual partition is, for example, "32", it is sufficient to refer to the part below "32" in the above list.

In addition, in this embodiment 6, an example is shown in which one or more selectable motion vectors are determined based on the block size of the partition serving as the coding block, but the same effect can also be achieved by determining one or more selectable motion vectors based on the partition pattern of the coding block.

FIG. 43 is an explanatory diagram showing a list of one or more selectable motion vectors for each division pattern of a coding block.

For example, when the partition serving as the coding block is 2partH1, "MV_A", "MV_B", "MV_C", and "temporal" are determined as one or more selectable motion vectors, but when the partition serving as the coding block is 2partH2, there is a high possibility that its motion is different from 2partH1, which is the block on the left.

Therefore, from one or more motion vectors selectable by 2partH2, "MV_A" which is the motion vector of the left block is deleted, and "MV_B", "MV_C", and "temporal" are determined.

In addition, in this sixth embodiment, time-direction vectors are used. However, to reduce the amount of memory used, the size of the data stored in the memory can also be compressed. For example, when the minimum block size is 4×4, time-direction vectors are usually stored in 4×4 units, but it is conceivable to store them in larger block sizes.

When compressing and storing time-direction vectors as described above, processing with a block size smaller than the stored unit can lead to the problem that the referenced position may not represent the correct position. Therefore, processing can be performed without using time-direction vectors in blocks smaller than the stored unit. By removing low-precision vectors from the candidate list, both the amount of processing and the amount of index code can be reduced.

In addition, while direct mode vectors are described in Embodiment 6, the same method can also be used to determine prediction vectors used in normal motion vector encoding. This method is effective in both reducing the amount of processing and improving encoding efficiency.

Furthermore, in this sixth embodiment, when the ref_Idx of a direct vector or a vector to be predicted differs from the ref_Idx of the candidate vectors used in generating a direct vector or determining a predicted vector (the reference destination pictures differ), scaling processing may be performed on the candidate vectors based on the distance in the time direction, as shown in FIG14. If the ref_Idx values are the same, scaling processing based on the distance in the time direction is not performed.

Here, scaled_MV represents the vector after scaling, MV represents the motion vector before scaling, and d(x) represents the temporal distance up to x.

In addition, Xr represents the reference image represented by the block to be encoded, and Yr represents the reference image represented by the block position A-D to be scaled.

Alternatively, as shown in Figure 49, spatial vector candidates can be constructed by searching inter-coded blocks from the target block and using all vectors contained in those blocks as candidates. As mentioned above, while the reference picture to which the direct vector or desired prediction vector should point may or may not be the same as the reference picture to which the candidate vectors point, a configuration can also be employed to use only candidate vectors pointing to the same reference picture, or to correct the vectors by scaling them to point to the same reference picture. The former eliminates low-precision vectors from the candidate list without increasing the processing load. The latter, while processing load increases according to the amount of searching, increases the number of candidates to choose from, thus reducing the amount of code.

Furthermore, when performing scaling as in equation (16), when an inter-coded block is discovered, scaling may be performed on a candidate vector having a different ref_Idx from that of the direct vector or the desired prediction vector (scaling may not be performed if the ref_Idx is the same). Alternatively, scaling may be performed only when no candidate vector having the same ref_Idx exists after a complete search. Although this increases the amount of processing, vectors with improved accuracy can be added to the candidate vectors, thereby reducing the amount of code.

Implementation method 7.

In the above-mentioned embodiment 6, an example is shown in which the encoding control unit 1 of the motion image encoding device and the motion compensation prediction unit 54 of the motion image decoding device pre-maintain a list representing selectable motion vectors, but the variable length encoding unit 13 of the motion image encoding device may also perform variable length encoding on the list information representing the list, multiplex the encoded data of the list information, for example, into a slice header, and transmit it to the motion image decoding device side.

In this case, the variable length decoding unit 51 of the moving picture decoding device variable length decodes the list information from the encoded data multiplexed in the slice header, and outputs the list represented by the list information to the direct vector generation unit 65 of the motion compensation prediction unit 54.

In this way, although the list information representing the list can also be transmitted to the motion image decoding device side in units of slices (or, in units of sequences, pictures, etc.), the list information representing the changed list can also be transmitted to the motion image decoding device side only when the list maintained by the encoding control unit 1 is changed.

The following describes the details of the processing.

FIG44 is a flowchart showing a process of transmitting list information in the moving picture encoding device, and FIG45 is a flowchart showing a process of receiving list information in the moving picture decoding device.

The encoding control unit 1 of the motion image encoding device determines one or more selectable motion vectors based on the block size of the partition serving as the encoding block, similarly to the sixth embodiment, and confirms whether the list referenced when determining the motion vector has been changed. If the list is the same as the last time (step ST41 of FIG. 44 ), the change flag is set to “OFF” (step ST42) to inform the motion image decoding device that the list is the same as the last time.

When the encoding control unit 1 sets the change flag to "OFF", the variable-length encoding unit 13 encodes the "OFF" change flag and transmits the encoded data to the moving picture decoding apparatus (step ST43).

If the list is different from the previous one (step ST41 ), the encoding control unit 1 sets a change flag to “ON” (step ST44 ) to notify the video decoding apparatus that the list is different from the previous one.

When the encoding control unit 1 sets the change flag to "ON", the variable length encoding unit 13 encodes the "ON" change flag and list information indicating the changed list, and transmits the encoded data to the moving picture decoding apparatus (step ST45).

FIG46 shows an example of encoding a change flag “ON” and list information indicating the changed list due to changing “temporal” in the list from selectable to non-selectable.

The variable length decoding unit 51 of the motion image decoding device decodes the change flag from the encoded data (step ST51 of Figure 45), and if the change flag is "OFF" (step ST52), the change flag of "OFF" is output to the motion compensation prediction unit 54.

Upon receiving the “OFF” change flag from the variable length decoding unit 51 , the motion compensation prediction unit 54 recognizes that the list is the same as the previous one, and sets the currently held list as a reference (step ST53 ).

Therefore, the motion-compensated prediction unit 54 refers to the currently held list and determines one or more motion vectors corresponding to the block size of the partition to be decoded this time.

If the change flag is “ON” (step ST52 ), the variable length decoding unit 51 of the moving picture decoding apparatus decodes the list information from the coded data, and outputs the “ON” change flag and the list information to the motion compensation prediction unit 54 (step ST54 ).

When the motion compensation prediction unit 54 receives the "ON" change flag and list information from the variable length decoding unit 51, it recognizes that the list is different from the previous one, changes the currently held list according to the list information, and sets the changed list as the reference object (step ST55).

Therefore, the motion-compensated prediction unit 54 refers to the changed list and determines one or more motion vectors corresponding to the block size of the partition to be decoded this time.

FIG. 47 shows an example in which the currently held list is changed because the change flag is “ON”.

As described above, according to this embodiment 7, the structure is such that only when the list representing one or more selectable motion vectors is changed, the list information representing the changed list is encoded to generate encoded data, so that the function of accepting changes in the list can be installed without causing a significant increase in the amount of code.

In this seventh embodiment, an example is shown in which the list information representing the entire list after the change is encoded even when a part of the selectable motion vector represented by the list is changed. However, as shown in FIG. 48 , a change flag may be prepared for each block size, the change flag of the block size in which the selectable motion vector is changed may be set to “ON”, and only the list information related to the block size may be encoded.

In the example of FIG. 48 , the motion vectors in the block sizes of “64” and “8” are not changed, so the change flag is “OFF” and the list information related to the block size is not encoded.

On the other hand, since the motion vectors are changed when the block sizes are "32" and "16", the change flag is "ON" and list information related to the block sizes is encoded.

Alternatively, the block size unit change flag may be encoded only when the change flag for a certain block size is "ON," and when the change flags for all block sizes are "OFF," only the list unit change flag ("OFF" change flag) may be encoded.

Alternatively, instead of using a list-based change flag, only a block-size-based change flag may be encoded.

Here, an example is shown in which the selectable motion vector can be changed for each block size, but the selectable motion vector may also be changed for each division pattern of the coding block.

Industrial Applicability

The motion image encoding device, motion image decoding device, motion image encoding method and motion image decoding method of the present invention can select the optimal direct mode according to the specified block unit to reduce the amount of code, so they are suitable for motion image encoding devices, motion image decoding devices, motion image encoding methods and motion image decoding methods used in image compression encoding technology, compressed image data transmission technology, etc.

Claims (5)

1. A motion image decoding device, characterized in that it comprises: The variable-length decoding unit performs variable-length decoding processing on the encoded data multiplexed to the bitstream to obtain compressed data, encoding mode, index information for selecting motion vector candidates used in motion compensation prediction processing, and control information for changing the number of motion vector candidates selected based on the index information. A motion compensation prediction unit selects a motion vector from one or more selectable motion vector candidates, representing a motion vector with index information related to a partition as the unit of prediction processing; and uses the selected motion vector to perform motion compensation prediction processing for the partition to generate a prediction image; and The decoding image generation unit adds the difference image decoded from the compressed data and the predicted image to generate a decoded image. The motion compensation prediction unit prepares motion vector candidates based on the control information. These candidates include spatial motion vectors or temporal motion vectors. The spatial motion vectors are selected from motion vectors of multiple decoded partitions surrounding the partition, while the temporal motion vectors are obtained from motion vectors of decoded images that the partition can reference. The number of motion vector candidates can be changed in units of slices. 2. A method for decoding moving images, characterized by comprising: The steps include performing variable-length decoding on encoded data multiplexed to a bitstream to obtain compressed data, encoding mode, index information for selecting motion vector candidates used in motion compensation prediction processing, and control information for changing the number of motion vector candidates selected based on the index information. The steps include: selecting a motion vector represented by the index information related to a partition as the unit of prediction processing from one or more selectable motion vector candidates; and using the selected motion vector to perform motion compensation prediction processing for the partition to generate a prediction image; and The step of generating a decoded image by adding the differential image decoded from the compressed data and the predicted image. The motion vector candidates, based on the control information, include spatial motion vectors or temporal motion vectors. The spatial motion vector is selected from the motion vectors of multiple decoded partitions surrounding the partition, and the temporal motion vector is obtained from the motion vectors of decoded images that the partition can reference. The number of motion vector candidates can be changed in units of slices. 3. A motion picture encoding device, characterized in that it comprises: A motion compensation prediction unit prepares motion vector candidates from spatial and temporal motion vectors, and uses the motion vectors selected from the motion vector candidates to perform motion compensation prediction processing on a partition to generate a predicted image. The spatial motion vectors are obtained from the motion vectors of multiple encoded partitions surrounding the partition, and the temporal motion vectors are obtained from the motion vectors of encoded images that the partition can reference. Variable-length coding units generate index information representing the selected motion vector candidates. The number of motion vector candidates can be changed for each slice, and the variable-length encoding unit generates control information for changing the number of motion vector candidates. 4. A motion picture coding method, characterized in that it comprises: The step of preparing motion vector candidates from spatial motion vectors and temporal motion vectors, wherein the spatial motion vectors are obtained from the motion vectors of multiple encoded partitions located around the partition, and the temporal motion vectors are obtained from the motion vectors of encoded images that the partition can reference; The steps of generating a predicted image by performing motion compensation prediction processing on a partition using motion vectors selected from the motion vector candidates; and The step of generating index information representing the selected motion vector candidates. The number of motion vector candidates can be changed for each slice. When the number of motion vector candidates changes, control information is generated to change the number of motion vector candidates. 5. A recording medium storing a computer-readable program and recording a bit stream containing encoded data, the encoded data determining: an encoding pattern for the encoded blocks being hierarchically segmented from the largest coded block, with the number of levels determined and the segmentation occurring up to an upper limit of said number of levels. The encoded data includes: compressed data, generated based on the difference between the predicted image and the original image produced by motion compensation prediction processing on the partition; index information related to the motion vectors used in the motion compensation prediction processing; and control information for changing the number of motion vector candidates selected according to the index information. When the program is executed by the processing device, the following steps can be achieved: The encoded data is subjected to variable-length decoding to obtain the compressed data, the encoding mode, the index information, and the control information. Motion vectors representing the index information associated with the partition being the unit of prediction processing are selected from the motion vector candidates. Using the selected motion vectors, motion compensation prediction processing is performed for the partition to generate a prediction image. The decoded image is generated by adding the differential image decoded from the compressed data and the predicted image. Based on the control information, a list of motion vector candidates, including spatial motion vectors or temporal motion vectors, is prepared. The spatial motion vectors are selected from the motion vectors of multiple decoded partitions surrounding the partition, and the temporal motion vectors are obtained from the motion vectors of decoded images that the partition can reference. The index information indicates the selected vector candidate when the motion vector is selected from the motion vector candidates. The motion vector candidate includes at least one of a spatial motion vector and a temporal motion vector. The spatial motion vector is obtained from the motion vectors of multiple encoded partitions surrounding the partition, and the temporal motion vector is obtained from the motion vectors of encoded images that the partition can reference. The number of motion vector candidates can be changed in units of slices.