JP2000092489A - Image encoding device, image encoding method, and medium recording program - Google Patents

Abstract

(57) [Summary] In a method of encoding, multiplexing and transmitting a plurality of object moving images to a communication line, when an object having high activity occurs in an encoding process, encoding is performed based on a bit amount previously allocated to the object. Since the volume increases and the transmission capacity of the communication line increases, the transmission capacity is reduced by dropping frames, but it is desired to suppress dropping frames. A priority is assigned to each object (obj), and when a high activity obj occurs, bits are moved between the objs so that a high priority obj does not reduce or improve the image quality. It is controlled by the rate control unit 117. This suppresses an increase in the number of dropped frames without increasing the transmission capacity of the communication line. Further, at the time of encoding each obj, a balance between temporal distortion due to frame drop according to the encoding rate and spatial distortion due to quantization accuracy can be achieved, and encoding processing can be performed at an operating point that is optimal in visual characteristics.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for controlling a code generation amount to control a transmission amount corresponding to a transmission capacity of a communication line when transmitting moving image data obtained by compressing a moving image to a communication line. More particularly, the present invention relates to an image encoding apparatus and an image encoding method that can maintain a desired image quality by giving priority to image quality among a plurality of objects.

[0002]

2. Description of the Related Art Conventionally, H.264 has been used as one of videophone systems. In the H.261 system, for a moving image, data is divided into 8 × 8 pixel blocks for each frame, and this block is subjected to two-dimensional DCT (discrete cosine transform) intra-frame encoding. When the correlation between the two is strong in the image block, the difference between the frames is calculated.
Inter-frame coding for dimensional DCT transformation. When a moving image block similar to the current frame moves relatively adjacently from the previous frame, this is detected and only information of the moving amount and moving direction of the moving image block is sent. Motion compensation that reduces the amount of generated data by not sending the moving image data itself. The coefficient value for each frequency after DCT conversion has a value in the low frequency region, but does not easily occur in the high frequency region. Zero run-length encoding using the value that follows, quantization that adjusts the amount of data generation by changing the quantization step width of data according to the amount of data generation,
By assigning a short code value to a frequently occurring data pattern and a long code amount to a less frequently occurring data pattern, the data pattern is converted into a smaller data amount than the total generated data amount. Variable length coding, etc. are used.

[0003] The compressed data that has been subjected to the variable-length coding is temporarily stored in a buffer memory and then sequentially read out and sent out to a communication line. Based on the accumulated amount, the quantization step width is adaptively changed according to changes in the input moving image and scene changes, frame skipping is performed, and the bit rate output from the buffer memory is kept constant before the communication line Sent to

[0004] Such H. In addition to the H.261 system, MPEG1 (Moving P
icture imagecoding Experts Group 1) Moving picture coding scheme or MPEG2 (Moving Picture imagecoding E)
xperts Group 2) There are moving picture coding methods and the like, and these are all coding methods for rectangular screens.

However, recently, MPEG4 (Moving Picture)
imagecoding Experts Group 4) A method of encoding an image contained in a screen in units of objects, such as a moving image encoding method, and transmitting and displaying compressed data of the plurality of objects to a receiving side is becoming widespread.

For example, as shown in FIG. 19, a moving image captured by a television camera or the like is input (100), and an original image 101 is input.
Are extracted by the object generation unit 102, and objects such as a person and a background are cut out from the original image 101 for each scene. Then, each object is encoded (103a, 103b), multiplexed by the multiplexing unit 104, and transmitted to the communication line 106. That is,
The human image is separately encoded as “object 1” (103a) and the background moving image is separately encoded as “object 2” (103b) and sent to the multiplexing unit 104. Then, the coded data of “Object 1” and “Object 2” are multiplexed by the multiplexing unit 104, and the bit stream 105
Is transmitted to the demultiplexing unit 107 on the receiving side via the communication line 106 in the data format described above. On the receiving side, the multiplexed object is sent to the demultiplexing unit 107, where the coded data of "Object 1" and "Object 2" are separated, and further the "Object 1" ( 108a) and "Object 2" (108
b) is decoded.

[0007] Each object is compounded (108a, 108b), and displayed on the display unit as a reproduced image 110 in consideration of the arrangement position of each object on the screen by the convoy unit 109. That is, the composited “object 1” (108a) and “object 2” (108b) are arranged at the original designated position by the compositor unit 109 and displayed as the reproduced image 110.

In the case of this method, when a moving image is cut out from the original image 101 in units of objects, only necessary objects can be sent to the receiving side. For example, only an important person image such as “Object 1” is transmitted, transmission of an unimportant background moving image such as “Object 2” is stopped, and “Object 1” is superimposed on another background moving image on the receiving side. They can be displayed together. By not sending the background moving image, there is an advantage that the total transmission capacity can be reduced.

Further, the transmission capacity can be reduced by transmitting the background moving image at a lower resolution than the person moving image.

FIG. 20 is a configuration diagram showing a device configuration for realizing the method described in FIG. 19 as hardware. In FIG. 20, reference numeral 112 denotes a shape extraction unit;
a, 114b, to 114n are encoding units, 117a, to 1
17n is a rate control unit, 119a to 119n are buffer memories, and 121 is a multiplexing unit.

In this configuration, an input moving image 111 captured by an imaging device such as a television camera is
And a plurality of encoding units 114a, 114b, to 114n. Then, when the input moving image 111 is input to the shape extracting unit 112, the necessary shape information 113a, 113b,.
13n is generated by the shape extraction unit 112. Then, these pieces of shape information 113a, 113b, to 113
n is one of the encoding units 114a, 114b, to 114n.
Each of the predetermined coding units 114a, 114b, to 114
n and encoded. The encoding units are 114a and 11
Although there are a plurality of 4b and -114n, this is to enable simultaneous and parallel encoding processing, and to encode exclusively for each object.

Each object is encoded by the encoding unit 114.
a to 114n in parallel, using the shape information 113, extracting a region of a moving image portion corresponding to each shape information from the input moving image 111, and then encoding (encoding information 123)
I do.

In this manner, the encoding units 114a and 114
The encoded information 123 obtained by b, .about.114n is provided to buffer memories 119a, 119b,... 119n corresponding to the encoding units 114a, 114b,.

The buffer memories 119a, 119b,.
19n is a temporary holding memory for smoothing the output 120 to a constant transmission rate, as described above.

In this configuration, the buffer memory 11
9a, 119b,... 119n are stored in the respective buffer memories 119a, 119b,.
n corresponding to the rate control units 117a, 117b,.
7n.

Then, each of the rate control units 117a, 117
b, .about.117n are based on the information of the data storage amount received by the corresponding buffer memories 119a, 119
In order to make the outputs 120a and 120n of 9b and 119n constant, they are fed back to the encoding units 114a to 114n (115a and 115n), and the encoding unit 1
The amount of frame data input to the buffer memories 119a, 119b, and 119n is controlled by changing the quantization steps of 14a to 114n.

In some cases, the buffer memory 1
19a, 119b,..., And 119n, the frame data temporarily stored is subjected to frame skipping (dropping of frames), so that the buffer memory 119 as a whole is skipped.
a, 119b, to 119n read output 120a to 1
The transmission rate of 20n is fixed.

Each of the buffer memories 119a, 119b,.
The read outputs 120a to 120n of 119n are multiplexed by the multiplexing unit 121 to become multiplexed outputs 123, which are sent out to the communication line. If the transmission rate of the encoded data corresponding to each object is constant, the multiplex output 123 is also limited to a constant transmission rate. And the multiplexed output 123
Corresponds to the transmission rate of the communication line.

Each of the buffer memories 119a, 119
The assignment of the transmission rates of the read outputs 120a to 120n of b, to 119n depends on how much the corresponding object is compressed or how much frame dropping is allowed.

That is, it depends on how the user allocates importance or priority of each object.

For example, if it is desired to faithfully reproduce the motion of a certain object, it is necessary to slightly increase the compression ratio for the entire object and perform rate control by the rate control units 117a, 117b, to 117n so as to avoid dropping frames as much as possible. There is.

Conversely, if importance is placed on the definition of the object, the rate control section 119 reduces the overall compression ratio of the object and allows a certain amount of frame dropping.
Need to perform rate control.

[0023]

Generally, in order to transmit high-quality moving image data, if the amount of coded data increases when compressing image data, the amount of data is also reduced when quantizing. When the number is increased, a more natural moving image can be reproduced.

However, if the encoded data increases, the transmission rate must be increased in order to transmit the data to the receiving side without delay, and a high-speed variable rate having a sufficient communication capacity such as ATM communication is required. Communication becomes necessary.

However, H.264 for low-speed lines 261
In such a moving image encoding method, it is general that encoded data is transmitted via a fixed-rate communication line. Since video data is transmitted over a communication line with a low transmission rate, the amount of encoded data increases when the video is in a high activity state including fine patterns, so quantization is sacrificed at the expense of spatial resolution. In this case, the increase of the encoded data is suppressed, or the temporal distortion is allowed by dropping frames, so that the increase of the encoded data is also suppressed, whereby the data is transmitted through a communication line having a constant rate.

Here, even in the moving picture coding method for each object by MPEG4 as shown in FIG. 19, when the communication line has a fixed rate, the transmission rate assigned to each object, that is, the bit rate is also fixed and distributed. In general, the transmission rate of each object is added to keep the overall transmission rate constant. In addition, by giving a priority to the ratio of the distribution rate to the area for each object, a high-priority object can be allocated a larger amount of code, and is displayed more precisely than a low-priority object. It becomes possible.

For example, in FIG. 19, for "object 1" as a person image, the bit allocation to the area is larger than that for "object 2" in the background, so that the person image is
It is possible to display finely.

The rate control of each object shown in FIG. 19 can be performed by the method shown in the block diagram of FIG. A shape extraction unit 112 extracts a region of a necessary object such as a human image and a background from an input moving image 111 input from a television camera or the like, and calculates shape information 113a to 113n. The encoding units 114a to 114n corresponding to each object generate the shape information 11
Using 3a to 113n, a moving image region of the corresponding object is cut out from the original image 111, and the texture, motion compensation frame prediction, and shape information in the region are encoded.

Each of the encoded data 116a to 116n is applied to buffer memories 119a to 119n. The rate control units 117a to 117n are encoding units 114a to 114n.
Of the buffer memory 119 by controlling the quantization precision of
By changing the bit amount of the encoded data 116a to 116n input to “a”, the output data 120a to 120n is controlled so as to have a rate previously assigned to each object.

The rate control units 117a to 117n perform frame dropping on a frame basis if necessary, so that the outputs 120a to 120n of the buffer memories 119a to 119n are output.
The rate of n is controlled to be constant. The outputs 120a-120 of the buffer memories 119a-119n
n is multiplexed by the multiplexing unit 121, converted into an output 123 as a bit stream, and transmitted to the communication line.

In FIG. 19, the rate control units 117a to 117n
Are the outputs 120a-12 of the buffer memories 119a-119n for the rate assigned to each object.
By suppressing the rate of 0n to a predetermined rate, the overall rate, that is, the output 123 of the multiplexing unit 121 is adapted to the transmission rate of the communication line.

Here, the conventional H.264 standard is used. In the case of a display on a rectangular screen as in the H.261 system, it can be said that the bit rate is moved from a low activity portion to a high activity portion on the entire screen, so that the overall rate is suppressed to a constant rate.

That is, by automatically extracting a high activity portion such as a portion having a large motion and a low activity portion having a small motion and allocating bits in a well-balanced manner, a high-quality display at a low rate as a whole is realized. It can be said that.

However, when a moving image is divided as an object as shown in FIG. 19, the bit distribution must be performed within the shape of the object. Even if the bit allocation is increased for an object with a high priority, if the object enters a high activity state, it cannot be processed in its own object, and the quantization accuracy is reduced or frames are dropped. Must be performed, leading to a decrease in image quality.

In this case, in particular, in the case of an object having a small area, the margin for bit allocation is reduced, and the image quality is immediately reduced when the high activity state is reached.

Therefore, an object with a high priority has to allocate a large number of bits with a sufficient margin from the beginning. However, in this case, in the low activity state, there is too much room for the number of allocated bits, and a moving image with a higher image quality than necessary is generated.

On the other hand, since objects such as backgrounds have a large area, the number of allocated bits as a whole increases even if quantization accuracy is reduced. However, the movement of the number of allocated bits in the object as a whole is small due to the low activity throughout, and it can be said that there is too much room for the fluctuation of the number of allocated bits.

Therefore, it is urgently necessary to develop a rational technical method for how to optimally allocate the objects.

Accordingly, an object of the present invention is to provide a system for multiplexing image data obtained by encoding moving images of a plurality of objects and transmitting the multiplexed image data to a communication line. An image encoding apparatus and an image encoding method capable of performing an assurance and performing an encoding process at an optimal operating point in terms of visual characteristics, thereby enabling a balanced display for each object. To provide.

[0040]

To achieve the above object, the present invention is as follows. That is, a moving image in which a screen such as MPEG4 is composed of a plurality of objects,
In an encoding device that encodes each object and multiplexes each of the encoded data of the plurality of objects and sends the multiplexed data to a communication line, the encoding device encodes the objects. A bit rate distribution value for performing the multiplexing corresponding to the priority is set, and coding means for coding the target object with a generated code amount corresponding to the allocated bit rate; and coding the object. When an object having high activity occurs in the process of performing, the bit rate of the encoding means is set so as to redistribute the bit rate among the objects so that the image quality of the high-priority object is not reduced or improved. Control means for setting and controlling the rate distribution.

Further, when encoding the objects for each frame, the control means balances the time distortion due to frame drop according to the encoding rate and the spatial distortion due to quantization accuracy. It further has a function of setting and controlling the bit rate distribution so as to secure an optimum operating point in terms of visual characteristics.

Further, in the control means, in the process of moving the bit rate between the objects, the distribution of the bit rate is reduced so that temporal distortion is reduced in the entire coded data transmitted to the communication line. A function for setting and controlling is further provided.

In this system having such a configuration, a moving image whose screen is composed of a plurality of objects is encoded in units of objects, and each encoded data of the plurality of objects is multiplexed and transmitted to a communication line. In encoding, the amount of generated code is equivalent to the given bit rate distribution value for performing the multiplexing for encoding the object and performing the multiplexing with respect to the priority of the image quality of the object. Encodes the target object. However, if an object having high activity occurs in the process of encoding the object, the bit rate is redistributed among the objects so that the image quality of the high priority object is not reduced or improved. For this purpose, the bit rate distribution of the encoding step is set and controlled.

That is, upon transmission, data obtained by encoding an object is sent to a buffer memory, and data is sequentially read out from the buffer memory according to an allowable transmission rate of the transmission line and sent to the transmission line. By performing rate control on the object as a whole with a single buffer memory, control is performed so as to conform to a constant transmission rate for a communication line as a transmission path. At that time, the priority and allocation bits are determined for each object as an initial state, and each object is set to the optimal operating point in terms of visual characteristics that balances spatial distortion due to quantization accuracy and temporal distortion due to frame dropping. The encoding process is performed so that Then, when an object with a high priority enters a high activity state in the process of moving image encoding, rate control is performed so as to move the allocation bits from the object with a low priority while observing the transmission rate of the communication line as a whole. Do. Also, when an object with a low priority enters the high activity state, the rate of the object is reduced, so that when frames are multiplexed, dropped frames are reduced as a whole.

When encoding a plurality of objects, multiplexing processing is performed by allocating bits while taking into account image quality for each object. As a result, the higher the priority of the object in the display of the entire object, the higher the quality of the moving image can be displayed, and the number of dropped frames of the entire object during the multiplexing process can be reduced. A balanced encoding process can be performed.

[0046]

Embodiments of the present invention will be described below with reference to the drawings. The present invention provides a system for multiplexing image data obtained by encoding moving images of a plurality of objects and transmitting the multiplexed image data to a communication line, wherein priorities are assigned when encoding objects, and a large number of dropped frames occur when encoding. When it is estimated that the object will be allocated, the allocation bits are moved between the objects to maintain and improve the image quality of the high-priority object, and in the course of the encoding process of each object, according to the encoding rate. The balance between temporal distortion due to dropped frames and spatial distortion due to quantization accuracy is achieved, and the control is performed so that the encoding process is performed at the optimal operating point in terms of visual characteristics. This makes it possible to display the reproduced image on the reproducing side, and the details will be described below.

<First Embodiment> Here, an example of an encoding apparatus for encoding control on a basic rectangular screen will be described. FIG. 1 is a block diagram illustrating a configuration example of an encoding device according to the present invention, and FIG. 2 is a diagram illustrating a temporal change of a staying amount in a buffer memory in the encoding device.

The encoding apparatus shown in FIG. 261 and the like, and includes an encoding unit 128 and an encoding control unit 124 that controls the encoding unit 128. Among these, the encoding unit 128 includes a motion compensation inter-frame prediction unit 129, a DCT transform unit 130, a quantization unit 133, a first variable length encoding unit 135, and a second
And a multiplexing unit 137.

The motion-compensated inter-frame prediction unit 129, which is a component of the encoding unit 128, can perform motion compensation within a range of 16 × 16 pixels (macroblock) and calculates an error between frames. , A DCT transform unit 130, which is a component of the encoding unit 128,
DCT transform (discrete cosine transform;
(Orthogonal transformation)) to convert the spatial coordinate data into frequency coordinate data.

The quantization unit 133, which is a component of the encoding unit 128, performs linear quantization (linear quantization) on the transform coefficients DCT-transformed by the DCT transform unit 130. The first variable length coding unit 135, which is a component, performs Huffman coding on the transform coefficient quantized by the quantization unit 133.

The second variable length coding unit 131, which is a component of the coding unit 128, performs Huffman coding on the motion vector used for motion compensation by the motion compensation inter-frame prediction unit 129. A multiplexing unit 137 that is a component of the encoding unit 128 multiplexes the main information encoded by the first variable length encoding unit 135 and the side information encoded by the second variable length encoding unit 131. To form a transmission frame. A buffer memory 139, which is a component of the encoding unit 128, temporarily stores the compressed data multiplexed by the multiplexing unit 137, and then sequentially reads out the compressed data and sends it out to a communication line. The information on the amount of data stored in the buffer memory 139 is stored in the encoding control unit 1
24.

The local decoding unit 134, which is a component of the encoding unit 128, decodes the output of the DCT transform unit 130 by performing an inverse DCT transform to decode the output.
4

The encoding control unit 124 is for controlling the encoding unit 128 having such a configuration, and includes information on the amount of data stored in the buffer memory 139 and the DCT output from the local decoding unit 134. An output obtained by decoding the output of the conversion unit 130 is used as an input, and based on information on the amount of data stored in the buffer memory 139 so that the buffer memory 139 does not overflow, adaptively in response to a change in an input moving image or a scene change. The data stored in the buffer memory 139 is transmitted to the communication line while changing the quantization step width in the quantization unit 133, dropping frames, and controlling the bit rate output from the buffer memory 139 to be constant. It is controlled so as to be transmitted.

In such a configuration, when input moving image data 127 captured by an imaging device such as a television camera is input to the present system, the input moving image data 127
Motion compensation inter-frame prediction unit 12 in encoding unit 128
9 given.

The motion compensation inter-frame prediction section 129 calculates an error between frames between the input moving image data 127 and the previous input moving image data to obtain a prediction error signal. The DCT transform unit 130 performs DCT transform on the prediction error signal from the motion compensation inter-frame predictive unit 129 in units of 8 × 8 blocks to transform spatial coordinate data into frequency coordinate data. The converted transform coefficients are linearly quantized and provided to the first variable length coding unit 135.

Then, the first variable-length coding unit 135 performs Huffman coding on the quantized transform coefficients, and
Give 37.

On the other hand, the prediction error signal from the motion compensation inter-frame prediction unit 129 is also supplied to the second variable length coding unit 131, and the second variable length coding unit 131 converts the prediction error signal into motion compensation. A motion vector to be used is obtained, Huffman-encoded, and provided to the multiplexing unit 137.

The multiplexing section 137 includes the main information coded by the first variable length coding section 135 and the second variable length coding section 1.
31 is multiplexed with the side information coded at 31 and supplied to the buffer memory 139. Thus, the multiplexing unit 13
7 is transmitted to the transmission line 14 via the buffer memory 139.
Sent to 0.

On the other hand, the output of DCT transform section 130 is also supplied to local decoding section 134, where it is decoded by inverse DCT transform. Then, this is given to the encoding control unit 124. Then, the encoding control unit 124 adjusts the quantization width in the quantization based on the decoded information from the local decoding unit 134 and the amount of data stored in the buffer memory 139, and performs image frame skipping. By performing such control as to the encoding unit 128, the encoded data is transmitted to the transmission path 140 at a predetermined rate while controlling the code amount so that the buffer memory 139 does not overflow. .

Here, in the coding apparatus having the configuration shown in FIG. 1, one frame is input as input moving image data 127 per unit time, and the input moving image data 127 is subjected to motion compensation frame prediction and further subjected to DCT transformation. Encoding is performed. When image data is encoded, a spatial distortion (S / N ratio) remains in a decoded image obtained by decoding the image data.

The spatial distortion (SN) of the decoded image 126
Assuming that the ratio is D and the code amount multiplexed by the multiplexing unit 137 is R bits, in this case, for example, as a result of encoding the input image data at the time t, decoding in a case where this is decoded The spatial distortion (SN ratio) in the image 126 is D
(T), and the output 138 of the multiplexing unit 137, that is,
The code amount of the data generated by multiplexing in the multiplexing unit 137 is R (t) bits.

The data generated by the multiplexing unit 137 is stored in the buffer memory 139 and then transmitted to the transmission line 140. However, the transmission path 14
The code amount transmitted to 0 is L bits per unit time. Therefore, the amount of stay 125 in the buffer memory 139
Is B (t), the amount of information B (t + 1) stored in the buffer memory 139 when the input moving image at the time t is encoded is: B (t + 1) = B (t) + R (t) −L, and the amount of information B (t + 1) stored in the buffer memory 139 when the input moving image at the time t is dropped, is B (t + 1) = B (t) −L. .

Here, in the present system, it is assumed that frame drop is performed by the following method. That is, when B (t)> L, “frame drop mode” When L ≧ B (t)> 0, “encoding mode” is set. When “frame drop mode”, frame drop is performed, and In the "encoding mode", encoding is performed without performing frame dropping.

FIG. 2 shows an example of a temporal change of the staying amount B in the buffer memory 139 in the system of the present invention. In this example, the time point t (the position indicated by reference numeral 142)
Then, the multiplexing unit 137 generates a code amount of R (t), which is stored in the buffer memory 139. Therefore, the buffer memory 139 stores the new R (t) data, so that there is no room for the code amount that can be stored. Therefore, at time t + 1, the encoding control unit 124 sets the “frame-drop mode”, and thereafter controls the encoding unit 128 in the “frame-drop mode”, and at the time t + 1 (indicated by reference numeral 143). At the time point t + 2 (position indicated by reference numeral 144) and at the time point t + 3 (position indicated by reference numeral 145), a frame drop is performed. Therefore, no new data is stored in the buffer memory 139.

However, since the data corresponding to the code amount of L is transmitted from the buffer memory 139 at each frame timing, the staying amount B gradually decreases with time. At time t + 4, the code amount corresponding to 3 × L is lost, so that the buffer memory 139 has a sufficient margin. Therefore, at the time t + 4, the encoding control unit 1
Numeral 24 denotes an “encoding mode”, and the encoding unit 128 is controlled in the “encoding mode”, so that the next new data is encoded and stored in the buffer memory 139.

As described above, depending on whether the storage capacity of the buffer memory 139 has a margin or not, the encoding mode or the frame drop mode is set to change the buffer memory 13.
Control is continued so that encoded data is transmitted at a constant rate so that 9 does not overflow.

Here, in the present system, the encoding control unit 124 controls the encoding unit 128 so that the image quality of the decoded image can be maintained in a balanced manner so that there is no excessive quality or large deterioration. This is realized as follows by using an index indicating the temporal distortion of the decoded moving image.

That is, the probability that an input frame at a certain time t is encoded is used as an index indicating the temporal distortion of a decoded moving image. The bit rate that can be transmitted on the transmission line 140 is L bits per unit time.
When there is information having the code amount of (t), it takes R (t) / L unit time to transmit this information.

Therefore, the average coding probability of the input frame before and after time t (hereinafter referred to as the coding rate) is represented by S (t)
In other words, S (t) can be expressed as S (t) = L / R (t) (where time t is an encoding mode).

Here, when encoding the input image, the image data of the input image is quantized, but the decoded moving image changes depending on the quantization precision at that time. That is, if the quantization precision is increased, a high SN ratio can be secured and the image quality of the decoded moving image can be maintained. However, the coding rate decreases, and the code amount increases. Conversely, if the coding rate is increased in order to reduce the code amount, the quantization accuracy will be reduced and the SN ratio will be reduced, and the image quality will be reduced. This relationship is called an image quality trade-off function Ss = G [Ds].

Image quality trade-off function Ss = G [Ds]
FIG. 3 shows an example. In FIG. 3, the horizontal axis is SN
The vertical axis indicates the coding rate. As shown by reference numeral 148 in FIG. 3, the image quality trade-off function Ss = G [Ds] has a poor SN when the coding rate is large,
A characteristic curve is drawn in which the SN increases as the coding rate decreases. Therefore, when the input frame includes a large motion, a pattern, or the like, it moves to the lower right, and when there is little motion, it moves to the upper left.

On the other hand, the point at which the combination of the above-mentioned coding rate and quantization accuracy is visually optimal (point indicated by reference numeral 149) is the image quality trade-off function Ss = G [ Ds]. The line connecting these points is the objective function So = O [Do], and this is the curve indicated by reference numeral 147 in FIG.

In order to accurately determine the picture quality trade-off function, it is necessary to actually encode the frame many times while changing the quantization precision, and to measure the coding rate each time. However, such processing is impractical in terms of processing time.

Therefore, as a practical method, the document “Kato et al.,“ Spatio-temporal distortion optimal allocation type coding control algorithm in video coding ”, IEICE Transactions on Communications, BV
ol. J71-B No. 8P. 945-954 1
August 988 ", which proposes two methods for determining a picture quality trade-off function from the current coded frame.

First, the first method described in the above document will be described. After the motion compensation frame prediction and the DCT are completed, a prediction error signal DCT coefficient and a motion vector of the current frame are obtained. At this time, DC for one frame
From the T coefficient histogram and the motion vector code amount, the generated code amount R (q) and the SN ratio D (q) are obtained as functions of the quantization accuracy q.

Therefore, the coding rate S (q) = L / R (q)
And the image quality trade-off function Ss = G [Ds], which is the relationship between the image quality and the SN ratio D (q), is determined with high accuracy.

FIG. 4 shows an example of the image quality trade-off function Ss = G [Ds] described in the above document.
Each of them has a downward slope but has different curves for different input frames as can be seen from the figure.

Next, the second method described in the above document will be described. In the second method, several coding rate S (q) and SN ratio D (q) candidates are determined in advance, and a characteristic matching the coding result of the immediately preceding coding frame is selected from the candidates. Then, based on the selection result, a picture quality trade-off function of the current coded frame is calculated.

FIG. 5 shows a method for estimating the coding rate S (q) by the present method. In FIG. 5, the numbers [No. 1] to [No. 6] represents the number of the coding rate vs. quantization step size characteristic candidate, and point C represents the coding rate and the quantization step size of the previous encoded frame. In the example of FIG. 3] is the most point C
, And the number [No. 3] is changed to the coding rate S at time t.
(Q). The SN ratio Ds (q) is estimated in a similar manner.

As described above, the encoding apparatus according to the first embodiment assigns priorities when encoding objects in a system for multiplexing image data obtained by encoding moving images of a plurality of objects and transmitting the multiplexed image data to a communication line. In coding, when it is estimated that the number of dropped frames is increased as a whole, the allocation bits are moved between the objects to maintain and improve the image quality of the high-priority object. During the object encoding process, the temporal distortion due to frame dropping according to the encoding rate and the spatial distortion due to quantization accuracy are balanced, and the encoding process is performed at the optimal operating point for visual characteristics. By doing so, a balanced display can be achieved for each object.

The example of the coding apparatus for controlling the coding on the rectangular screen has been described above. Next, the second object of the rate control in a plurality of objects, which is the main object of the present invention, is described.
An embodiment will be described.

<Second Embodiment> [Rate Control of the Present Invention for a Plurality of Objects] FIG. 6
FIG. 1 is a configuration diagram of an apparatus of a system that performs rate control on a plurality of objects with a single buffer memory. FIG.
, 112 is a shape extraction unit, 114a, 114b,
To 114n are an encoding unit, 117 is a rate control unit, 153
Is a multiplexer, and 119 is a buffer memory.

The shape extraction unit 112 extracts a region based on the outline of each object constituting the screen from the input moving image 111 obtained by photographing with an imaging device such as a television camera, and extracts the region of each object. Shape information 113
a, 113b,..., and 113n, respectively, for extracting a plurality of object shapes from a screen of one frame. The shape extraction unit 112 also has a function of classifying objects whose outlines are in contact with each other as a set based on the shape information of the extracted objects and outputting the information (classification table) to the rate control unit 117. Have.

The coding units 114a to 114n are coding units for coding the respective assigned objects corresponding to the respective objects, and use the shape information 113a, 113b, to 113n corresponding to each object. The image in the range of the shape information is encoded so as to have the assigned code amount (the code amount corresponding to the allocated bit amount) given under the control of the rate control unit 117.

The encoding units 114a to 114n include:
It has a function of decoding the coded data obtained by the above-mentioned coding process and returning it to the original data, and a function of outputting the restored data to the rate control unit 117 as local decoded data 152a to 152n. have.

The multiplexer 153 is for multiplexing data. The output data 1 is obtained by encoding the objects output from the encoding units 114a to 114n.
16a, .about.116n, and multiplexes them, and sends them to the buffer memory 119. Buffer memory 1
The output 120 of 19 is the transmission output to the transmission line.

The rate control unit 117 receives the local decoded data 152a to 152n and appropriately allocates the code amount to be assigned to each object in consideration of the balance between the individual image quality according to the priority of each object and the image quality of the entire frame. By doing so, the data storage amount information 154 is received from the buffer memory 119, the empty state of the buffer memory 119 is checked from the storage amount information 154, and the output 120 of the buffer memory 119 is set to the assigned rate. , Encoding units 114a, to 114n
The output data 116a,..., 116n are subjected to adjustment control such as frame skipping. The rate control unit 117 also has an adjustment control function of performing rate control and frame dropping for objects belonging to each group from the classification table from the shape extraction unit 112.

In such a configuration, when an input moving image 111 captured by an imaging device such as a television camera is input to the present system, the input moving image 111
12 and the encoding units 114a, 114b, to 114n,
Each given.

The shape extracting unit 11 which has received the input moving image 111
2 extracts a region from each of the objects constituting the screen from the input moving image 111 based on the outline, and obtains shape information 113a, 113 to 113 for each object.
Generate n. Then, the encoding units 114a, 114b,
To 114n, the shape information of the corresponding object among the generated shape information 113a, 113n for each object is given.

The shape extracting unit 112 classifies the objects whose outlines are in contact with each other as a set based on the extracted object shape information. Then, classification table information indicating which object belongs to each of the sets,
Send to rate control section 117.

The encoding units 114a to 114n receiving the input moving image 111 use the input moving image 1 by using the shape information 113a to 113n respectively given to themselves.
Region extraction of a moving image portion corresponding to shape information from 11 is performed,
The motion compensation information is obtained using the extracted moving image portion,
The motion compensation information and the extracted texture information of the moving image portion are encoded and output (output data 116
a, ~ 116n). At this time, the generated code amount in the encoding process is adjusted and controlled by changing the quantization width so as to be a bit amount corresponding to the allocated code amount given by the rate control unit 117 (rate). control).

The encoding units 114a to 114n have a function of decoding the coded data obtained by the above-mentioned encoding process and returning the data to the original data. The data is output to the rate control unit 117 as data 152a to 152n.

The output data 116a and 116n from the encoders 114a to 114n are sent to the multiplexer 153 where they are multiplexed. The output 155 multiplexed by the multiplexer 153 is supplied to the buffer memory 119. The buffer memory 119 takes in this and reads it out at a constant rate.

As a result, the buffer memory 119 smoothes the encoded data of all objects, and the output 120 read from the buffer memory 119 is output to the line at a constant rate.

Information on the amount of data stored in the buffer memory 119 is sent to the rate control unit 117. Then, the rate control unit 117 controls the frame dropping of each of the encoding units 114a to 114n as necessary based on the information of the staying amount and the like, adjusts the generated code amount, and controls the encoded data input to the buffer memory 119. Is controlled not to overflow. Further, the rate control unit 117 controls each encoding unit 1
14a-114n local decrypted data 152a
To 152n, the code amount allocated to each object is appropriately distributed in consideration of the balance between the individual image quality according to the priority of each object and the image quality of the entire frame, and the individual information is encoded by each encoding unit 114a. １１４114n to the corresponding encoding unit.

On the other hand, the shape extracting unit 112 classifies the objects whose outlines are in contact with each other as a set based on the extracted shape information of the object, and determines which object belongs to each set. Is created and given to the rate control unit 117. And
When the need for rate control and frame dropping arises, the rate control unit 117 collectively drops objects belonging to the group for each group from the information in this classification table.

In other words, when dropping frames, the rate control unit 117 performs dropping of frames on a group basis based on the information in the classification table.

The outline of the operation of the encoding apparatus having the configuration shown in FIG. 6 is as described above. This encoding apparatus implements rate control for a plurality of objects with one buffer memory, and performs encoding for a plurality of objects. At the time of multiplexing, multiplexing processing is performed by allocating bits while considering image quality for each object, and the higher the priority of the entire object is displayed, the higher the quality of the moving image can be displayed. It is characterized in that the number of dropped frames in the entire object can be reduced during the encoding process, thereby enabling a balanced encoding process as a whole.

That is, by controlling the rate of a plurality of objects with one buffer memory as a whole, the communication line is controlled to conform to a certain transmission rate. The priority and allocation bits are determined for each object as an initial state, and each object is set to the optimal operating point in terms of visual characteristics that balances spatial distortion due to quantization accuracy and temporal distortion due to frame dropping. Perform encoding processing. Then, when an object with a high priority enters a high activity state in the process of moving image encoding, rate control is performed so as to move the allocation bits from the object with a low priority while observing the transmission rate of the communication line as a whole. Do. Also, when an object with a low priority enters the high activity state, the rate of the object is reduced, so that when frames are multiplexed, dropped frames are reduced as a whole.

When encoding a plurality of objects, multiplexing is performed by allocating bits while taking into account image quality for each object. A high-quality moving image display can be realized, the number of dropped frames of the entire object can be reduced during the multiplexing process, and a balanced encoding process can be performed as a whole.

The details of the control will be described below.

In the encoding apparatus according to the second embodiment, the input original image is subjected to shape extraction processing by the shape extraction unit 131, thereby extracting a region of a necessary object. The shape of the object is extracted.

Here, in order to extract a corresponding area from a general moving image based on the contour of an object, for example, a method described in JP-A-6-251148 can be used.

Then, in the encoding units 114a to 114n,
The texture, motion-compensated frame prediction, and shape information in the area are encoded. And the encoding unit 1
The texture, motion-compensated frame prediction, and shape information encoded by 14a to 114n are sent to the multiplexer 153, where they are multiplexed and sent to the buffer memory 119. Here, different objects are handled for each of the encoding units 114a to 114n, and accordingly,
Assuming that channels 14a to 114n are independent channels, the output 155 multiplexed by the multiplexer 153 is data obtained by multiplexing the data of each channel.

As in the first embodiment, information on the amount of data stored in the buffer memory is sent to the rate control unit 117 for rate control. However, in the present system according to the second embodiment, the entire buffer memory is used. Instead of the amount of stagnation in, the amount of stagnation for each channel. That is, the buffer memory 119
The current data amount information for each channel in the above is sent to the rate control unit 117.

On the other hand, the rate control section 117
4a to 114n, local decoded data 1 given from
The amount of memory allocated to the buffer memory 119 for each channel is calculated from the data amount information for each channel and the quantization accuracy control information 151 from the calculation result.
a to 151n, and gives the quantization accuracy control information 151a to 151n to the encoding units 114a to 114n. The encoding units 114a to 114n perform encoding by setting the quantization accuracy specified by the quantization accuracy control information 151a to 151n.

Here, an example of the configuration of the encoding unit 114 will be described. <Example of Configuration of Encoding Unit 114> FIG.
It is a block diagram showing the internal configuration of 114n. Encoding unit 1
14 comprises an encoding control unit 156, an encoded controlled unit 158, a shape encoding unit 157, and a multiplexing unit 165.

Among these, the encoding control unit 156 adjusts the quantization width in the quantization processing stage in the encoded controlled unit 158 based on the quantization accuracy control information 151 from the rate control unit 117. The frame encoding control section 157 controls the shape encoding section 157. The shape encoding section 157 controls the shape encoding section 157.
The encoding control unit 158 encodes the input moving image data 111 under the control of the encoding control unit 156.
From among them, motion-compensated frame prediction is performed for a shape area corresponding to the shape information 113 provided by the shape extracting unit 112, and this is variably encoded.
It has a function of obtaining a DCT transform coefficient by performing a CT process, quantizing it, variably encoding it, decoding the DCT transform coefficient, and outputting it as local decoded data 152.

Here, the coded controlled unit 158 includes a motion compensation inter-frame predicting unit 159, a DCT transforming unit 160, a second
, A quantization unit 162, a local decoding unit 163, and a first variable length coding unit 164.

The motion-compensated inter-frame predicting section 159 can perform motion compensation within a range of 16 × 16 pixels for a shape area corresponding to the shape information 113 provided by the shape extracting section 112, calculate an inter-frame error, and perform motion-compensated frame prediction. The DCT transform unit 160 performs DCT processing on the motion-compensated frame prediction information. Specifically, the DCT transform unit 160 performs DCT transform on the prediction error information in units of 8 × 8 blocks to obtain spatial coordinate data. Is converted to frequency coordinate data, and the second variable-length encoding unit 161 variably encodes the motion-compensated frame prediction information and outputs the information.

The quantizing section 162 quantizes the transform output of the DCT transform section 160 with a given quantization width and outputs the result. The first variable length coding section 164 outputs the quantized output. Is variably encoded and output. Specifically, the quantized transform coefficient is subjected to Huffman encoding, and the local decoding unit 163 decodes the DCT transform coefficient obtained by the quantization unit 162. And outputs it as local decrypted data 152.

The multiplexing section 165 multiplexes the outputs of the shape coding section 157, the first variable length coding section 164, and the second variable length coding section 161 and outputs the multiplexed data as output data 116. is there.

The encoding unit 114i (i is one of a,..., N) having such a configuration encodes the outline of the object itself based on the shape information 113 given from the shape extraction unit 112. That is, a plurality of encoding units 114a,
To 114n process a specific object among a plurality of objects as a target of the encoding process.
Therefore, the encoding units 114a to 114n determine the object based on the shape information 113i (i is one of a and n) of the different specific object from among the plurality of shape information 113a and 113n. Encode the contour itself.

Further, the input moving image 1 is
11 is input, and the input moving image 111 is input to the coded controlled unit 158.

In the coded controlled unit 158, the input moving image 1
When 11 is input, the motion compensation inter-frame prediction unit 159
Uses this and the shape information 113 from the shape extraction unit 112 to calculate an error between frames that can be motion compensated in a range of 16 × 16 pixels, and obtain a prediction error signal. And DC
The T conversion unit 160 receives the shape information 11 from the shape extraction unit 112.
3, the spatial error data is converted into frequency coordinate data by subjecting the prediction error signal to DCT transform in units of 8 × 8 blocks, and the quantization unit 162 converts the DCT transform by the DCT transform unit 160. The coefficients are linearly quantized.

The first variable length coding section 164 uses the quantized transform coefficient as the shape information 1 from the shape extraction section 112.
Huffman coding (texture coding) with reference to FIG.
Then, it is given to the multiplexing unit 165.

On the other hand, the motion compensation inter-frame prediction unit 15
9, the second variable length coding unit 161 uses the motion vector used for motion compensation in the shape information 1 from the shape extraction unit 112.
Huffman coding (motion compensation coding) with reference to FIG.
This is provided to the multiplexing unit 165. Also, the shape encoding unit 157 encodes the shape itself by encoding the shape information 113 provided from the shape extraction unit 112, and supplies the encoded information to the multiplexing unit 165.

Then, in multiplexing section 165, main information coded in first variable length coding section 164, side information coded in second variable length coding section 161 and shape coding section 157 Are multiplexed to form a transmission frame, and output to the multiplexer 153.

The encoding control section 156 is provided for the rate control section 11
7 based on the quantization accuracy control information 151 from the encoding control unit 158, adjusts the quantization width at the quantization processing stage in the encoding controlled unit 158, controls frame dropping, and controls the shape encoding unit 157. I do.

As described above, in the coded control unit 158 to which the input moving image 111 is input, the shape extraction unit 112 is controlled from the input moving image 111 under the control of the coding control unit 156.
Of the shape area corresponding to the shape information 113 given by the motion compensation frame prediction, variably encodes the result, and supplies the result to the multiplexing unit 165.
After performing T processing and further quantizing this, it is variably coded and provided to a multiplexing unit 165. The multiplexing unit 165 comprises a shape coding unit 157, a first variable length coding unit 164, and a second The output from the multiplexing unit 161 is multiplexed, and the multiplexed output is sent to the multiplexer 153.

Further, the local decoding section 163 performs inverse DCT transform of the DCT transform coefficient with reference to the shape information 113 given from the shape extracting section 112, thereby decoding and outputting as local decoded data 152, This is given to the control unit 117.

The operation summary is as described above. The motion compensation inter-frame prediction unit 159 and the second variable length coding unit 16
1. The DCT transform unit 160, the local decoding unit 163, and the first variable length coding unit 164 perform processing only on the original image (input moving image 111) with respect to the area specified by the shape information 113.

Methods for encoding shape information, motion compensation and texture encoding are described in, for example, ISO / IEC JTC1 / SC29 / WG11.
N1796, "MPEG-4 Video Verfication Model Version 8.
0. "July 1997.

The characteristic graph 166 of FIG.
The characteristic graph 167 of (b) shows an example of the staying amount in the single buffer memory 139 of FIG. 1 in the configuration according to the first embodiment. In the characteristic graph 166, B (t + 3) = B (t) + R1 (t) −3 · L1, and frames are dropped at times t + 1 and t + 2. Here, L1 is the amount of data transmitted to the transmission path per unit time.

Similarly, the characteristic graph 167 becomes B (t + 3) = B (t) + R1 (t) −3 · L2.

The control modes of the characteristic graphs 166 and 167 will be temporarily applied to the parallel encoding system shown in FIG. 6 in the configuration of the second embodiment. Object is 2
In this case, the encoding units 114a and 11
4b, and applying the conventional buffer memory control mode (FIG. 20), the encoding unit 114a and the encoding unit 114b need to be smoothed by the two buffer memories. Therefore, in the case of the system configuration in FIG. 6, it is necessary to smooth the two buffer memories in the buffer memory 119. In this case, if the characteristic graphs 166 and 167 are linearly added as they are and smoothed by the buffer memory 119, a characteristic graph 168 shown in FIG. 8C is obtained. B (t + 3) = B (t) + R1 (t) + R2 (t) -3
・ It becomes L3. Here, L3 is L3 = L1 + 12.
This L3 is the amount of data transmitted to the transmission line per unit time at the output 120 of the buffer memory 119 in FIG. Then, the frames are dropped at times t + 1 and t
+2.

FIG. 9A shows an objective function (“So1 = O1 [Do]”) 169 and an image quality trade-off function (“Ss1 = G1 [Ds]”) 170 for the characteristic graph 166. Objective function 169 and trade-off function 170
Is an operating point at which a visually balanced decoded image is obtained at time t. The operating point can be obtained by the above method.

Similarly, FIG. 9B shows an objective function (“So2 = O2 [Do]”) 172 and an image quality trade-off function (“Ss2 = G2 [Ds])) 1 for the characteristic graph 167.
73 is shown. An intersection 174 of the objective function 172 and the trade-off function 173 is an operating point at time t at which a visually balanced decoded image is obtained. Then, the encoding control unit 15 controls the buffer memory 119 so that the control according to the characteristic graphs 166 and 167 is performed together.
6 to execute such control.
17, the operating points 171 and 174 in FIG.
Is output, and the output 120 of the buffer memory 119 is video data output at the optimal operating point for the two objects, and the video data is transmitted to the communication line.

By the way, the characteristic graph 17 in FIG.
6 is (“object 2”) encoded data R at time t
2 (t) is transmitted to the communication line by performing frame dropping twice at times t + L and t + 2.

However, the characteristic graph 175 shown in FIG.
(“Object 2”), the value of R1 (t) is larger than usual (for example, time t−3), and the time t + 1,
Unless the frame is dropped three times at t + 2 and t + 3, the encoded data at the time t cannot be transmitted to the communication line, and the delay in displaying a moving image on the receiving side becomes longer than usual. And
In one buffer memory 119 of FIG.
When the control is performed by 7, the frame drop occurs at times t + 1, t + 2, and t + 4 as shown in a characteristic graph 177 of FIG.

That is, in the case of encoded data according to the characteristics of the characteristic graph 176, the number of dropped frames is only two, whereas
If the encoded data according to the characteristics of the characteristic graph 175 are subjected to the rate control together, the number of dropped frames becomes three.

Therefore, when the encoded data according to the characteristics of the characteristic graph 175 and the characteristic graph 176 are collectively controlled by the rate control unit 117 in FIG. 6, the following is performed.

First, “object 1” having a high priority
For the encoded data of the (characteristic graph 175), the quantization precision is temporarily increased at time t to improve the image quality, the encoded data R1 (t) is increased to R1 '(t), and the buffer memory 119 is used. From the data transmission amount to the line
From 1 to L1 '(characteristic graph 178), the number of dropped frames is reduced to two times t + 1 and t + 2.

On the other hand, “object 2” having a low priority
Regarding the encoded data of the (characteristic graph 176), at time t, the encoded data R2 (t) is temporarily reduced to R2 ′ (t), and the quantization precision is reduced to degrade the image quality. At the same time, the amount of data transmitted from the buffer memory 119 to the line is reduced from L2 to L2 '(characteristic graph 179).

The coded data of the above “Object 1” and “Object 2” is stored in the buffer memory 11 of FIG.
9 and are collectively controlled by the rate control unit 117 in the same manner as the above-described rate control individually performed. FIG. 11 (c)
Represents the temporal change in the stagnant state of the transmission amount of the buffer memory 119 when the encoded data of the characteristic graph 178 and the encoded data of the characteristic graph 179 are added.

That is, the amount of encoded data at the frame time t is calculated from R1 (t) + R2 (t) to R1 '.
(T) + R2 '(t), and the data transmission amount per unit time of the communication line is set to L3 so as not to change.

As described above, when there is a possibility that many frames are dropped in an object having a high priority, the amount of coded data is changed for the object by changing the quantization accuracy, and at the same time, the unit time of the communication line is changed. Increase the amount of data transmission per unit.

On the other hand, for objects with low priorities, the quantization accuracy is reduced at the same time, the amount of coded data is reduced, and at the same time, the data transmission amount per unit time of the communication line is reduced.

By performing such control, when a plurality of objects are collectively managed in one buffer memory, the number of dropped frames can be reduced as a whole.

For the encoding process of “object 1”, the characteristic graph 175 changes to the characteristic graph 178.
When changing the characteristic of the “object 2” from the characteristic graph 176 to the characteristic of the characteristic 179, the characteristic is changed so as to satisfy the following visually optimum operating point.

"Object 1" in FIG.
In, when the objective function is 181, the trade-off function is 182 in the case of the characteristic graph 175, and the operating point is 184, a visually optimum operating point is obtained. However, when moving to the characteristic graph 178, the trade-off function becomes 183, and the visually optimum operating point moves to 185.

That is, the operating point 185 has a higher SN ratio and a higher coding rate than the operating point 184, indicating that the image quality has been temporarily improved.

On the other hand, looking at “object 2” in FIG. 12B, the objective function is 188, the trade-off function is 187 in the characteristic graph 176, and the operating point is 190. It was an operating point. However, when moving to the characteristic graph 179, the trade-off function becomes 186, and the visually optimal operating point is 189.
Go to In other words, the operating point 189 indicates that the image quality has temporarily deteriorated because the SN ratio has deteriorated and the coding rate has decreased as compared with the operating point 190.

As described above, the "object 1" and the "object 2" are controlled by the encoding control unit 156 in the rate control unit 117 so as to distribute the encoding so as to be within a predetermined rate and perform the encoding process. In this way, the rate control that has been performed individually for each object can also be collectively performed in the buffer memory 119 in the configuration of FIG. And the rate control of each object is
By performing the operation at the intersection of the optimal objective function and the trade-off function for each object, the operation point becomes visually optimal.

However, “object 1” having a high priority
If it is estimated that the amount of coding temporarily increases and the number of dropped frames increases, the coded data amount R ′ of the object at that time and the data transmission amount L ′ per unit time of the object are calculated. By changing, the image quality can be improved and the number of dropped frames can be reduced.

On the other hand, for an object with a low priority, the picture quality is lowered by changing the coded data amount R 'of the object and the data transmission amount L' of the object per unit time at the time. In this case, when "object 1" and "object 2" are managed together in one buffer memory, the feature is that the number of dropped frames does not increase.

This will be described.

In FIG. 13A, the encoding amount R 1 of “object 1” is shown by a characteristic graph 191.
The frame drop has occurred twice in total at the times t + 1 and t + 2. The encoding amount R2 of “object 2” is as shown by a characteristic graph 192 in FIG. 14B, and as shown in the characteristic graph 192 of FIG. , Three times at time t + 2 and at time t + 3.

Therefore, if the encoding processing of “Object 1” and “Object 2” is managed by one buffer memory, the frame drop occurs at time t + 1 as shown in the characteristic graph 193.
, The time t + 2, and the time t + 3, a total of three times.

Therefore, in the present invention, FIGS. 14 (a) and 14 (b)
, The encoding amount at time t is reduced from R2 (t) to R2 ′ (t) for “object 2” having a low priority (characteristic graph 194). When the “object 1” and the “object 2” are collectively managed in one buffer memory, if the encoding control unit 156 controls the rate control unit 117 to execute such control, the buffer memory can be stored. A characteristic graph showing the amount is shown in FIG.
5 and the frame drop occurs at the time t + 1 and the time t
It is reduced to a total of two times at the time of +2.

FIG. 15 shows the relationship between the objective function and the trade-off function in this case. That is,
As shown in FIG. 15A, the relation between the objective function 196 and the trade-off function 197 for “Object 1” does not change, and the visually optimum operating point is 198. However, as for the “object 2”, the coding amount was reduced at the time t, so that the trade-off function moved from 200 to 201 as shown in FIG. Has moved to 203, which means that the SN and the coding rate have deteriorated.

FIG. 16 shows the amount of data staying in the buffer memory 119 in the system of the present invention. FIG.
A characteristic graph 204 of the buffer memory retention amount of “Object 1” in (a) and a characteristic graph 2 of the buffer memory retention amount of “Object 2” in FIG.
In the case of 05, the frame drop occurs three times in total at the time t + 1, the time t + 2, and the time t + 3.
Accordingly, when “object 1” and “object 2” are managed by one buffer memory, the change in the buffer memory staying amount is changed by the characteristic graph 20 shown in FIG.
6, and a frame drop occurs three times at each of the times t + 1, t + 2, and t + 3.

Therefore, as shown in FIG. 17A, regarding the characteristic graph 204, the encoded data amount R1 '(t) and the data transmission amount L1' per unit time are increased to improve the image quality. As shown in a graph 207, the number of dropped frames is reduced to a total of two times at times t + 1 and t + 2.

Regarding the characteristic graph 205 of FIG. 17B, the encoded data amount R2 '(t) and the data transmission amount L2' per unit time are reduced to degrade image quality.
As shown in the characteristic graph 208, the number of dropped frames is reduced to a total of two times at times t + 1 and t + 2.

Therefore, the characteristic graph 207 of FIG.
In order to manage the operation of the characteristic graph 208 of FIG.
If the control is performed by the rate control unit 117 to execute such control by 56, the characteristic graph of the data retention amount in the buffer memory 119 is as shown in 209 in FIG. To a total of two times.

FIG. 18 shows the relationship between the objective function and the trade-off function in this case. That is, in this case, as shown in FIG. 18A, the trade-off function of “Object 1” has shifted from 210 to 211, the operating point has shifted from 213 to 214, and the S / N ratio and coding rate have been changed. Has risen.

On the other hand, regarding “Object 2”, FIG.
As shown in FIG. 8B, the trade-off function has moved from 216 to 217, and the operating point has changed from 218 to 21.
9, the SN ratio and the coding rate are reduced.

As described above, in the second embodiment,
When there are a plurality of objects, each object is controlled by the encoding control unit 156 by the rate control unit 117 in order to perform the encoding process so as to be within a predetermined rate, so that only one buffer memory is used. Even if there is no object, the rate control that had been performed individually for each object can now be performed collectively,
By controlling the rate of each object at the operating point at the intersection of the optimal objective function and trade-off function for each object, it becomes possible to control the visually optimal operating point. is there.

In the case of a scene in which an object with a high priority is temporarily estimated to have a large amount of coding and a large number of dropped frames, the coded data amount R 'of the object at that time and its coded data amount R' By changing the data transmission amount L 'per unit time for the object, the image quality can be improved and the number of dropped frames can be reduced, while the low priority object is set at the time. By controlling the encoded data amount R 'of the object and the data transmission amount L' of the object per unit time to reduce the image quality, it is possible to control the number of dropped frames so as not to increase. .

In the above description, the communication line has a fixed rate, bits are initially allocated to each object in advance, and basically each object has a constant rate. And, before the encoding,
If an object is activated and it is estimated that many drop frames occur for the bitstream transmitted to the communication line due to the effect of the object, the image quality of the high-priority object is not reduced, or By moving bits between each object so as to improve, the drop of frames for the bit stream transmitted to the communication line is also reduced. In the process of encoding each object, the temporal distortion due to frame drop according to the encoding rate and the spatial distortion due to quantization accuracy are balanced, and the encoding process is performed at the optimal operating point in terms of visual characteristics. It is intended to do.

However, the application of the present invention is not limited to a communication line having a constant rate. For example, the present invention can be applied to a communication line in which a transmission error frequently occurs and a transmission rate substantially changes like a wireless line, or a variable rate communication line such as an ATM communication.

In the case of a variable rate communication line, the bit allocation of each object is changed each time the line rate changes. As an allocation method, there is a method of determining an object having a high priority and an object having a low priority by an allocation ratio. Also,
There is a method of distributing the remaining bits to low-priority objects by keeping the distribution bits of the high-priority objects constant to ensure a constant image quality at all times.

From the bits allocated to each object by the above method, the method of the present invention further reduces the drop of frames in the bit stream transmitted to the communication line,
In addition, bits are allocated to each object and coding is performed so as to maintain or improve the image quality of objects with high priority. In the process of encoding each object, the temporal distortion due to frame drop according to the encoding rate and the spatial distortion due to quantization accuracy are balanced, and the encoding process is performed at the optimal operating point in terms of visual characteristics. By doing so, a well-balanced display of the moving image of each object is possible even with a variable rate communication line.

Note that the present invention is not limited to the above-described embodiments, and can be implemented with various modifications. Also,
The method described in the embodiment of the present invention is stored in a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), a semiconductor memory, or the like as a program that can be executed by a computer. Can also be distributed.

[0165]

As described above in detail, according to the present invention, in a system for multiplexing image data obtained by encoding moving images of a plurality of objects and transmitting the multiplexed image data to a communication line, priorities are set at the time of object encoding. When it is estimated that the number of dropped frames is increased as a whole during encoding, the allocation bits are moved between the objects to maintain and improve the image quality of the high-priority object, and In the course of the encoding process, the temporal distortion due to frame drop according to the encoding rate and the spatial distortion due to the quantization accuracy are balanced, and the encoding process can be performed at the optimal operating point in terms of visual characteristics. Thus, a balanced reproduced image can be displayed on the reproducing side for each object.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining the present invention, and is a configuration diagram for explaining an encoding method for displaying on a rectangular screen according to a first embodiment of the present invention.

FIG. 2 is a diagram for explaining the present invention, and is an example of a graph showing a data retention amount of a buffer memory in an encoding method for displaying on a rectangular screen according to the first embodiment of the present invention; .

FIG. 3 is a diagram for explaining the present invention, and illustrates an objective function and a trade-off function that define image quality in an encoding method for displaying on a rectangular screen according to the first embodiment of the present invention. An example of the graph shown.

FIG. 4 is a diagram for explaining the present invention, and is a graph showing a trade-off function that defines image quality in an encoding method for displaying on a rectangular screen according to the first embodiment of the present invention. An example.

FIG. 5 is a diagram for explaining the present invention, and shows a coding rate and a quantization accuracy that define image quality in a coding method for displaying on a rectangular screen according to the first embodiment of the present invention. 6 is an example of a graph showing the relationship.

FIG. 6 is a diagram for explaining the present invention, and is a configuration diagram of an encoding method for a plurality of objects according to a second embodiment of the present invention.

FIG. 7 is a diagram for explaining the present invention, and is a configuration diagram of an encoding unit in a multi-object encoding method according to a second embodiment of the present invention.

FIG. 8 is a diagram for explaining the present invention, and is an example of a graph showing a data retention amount in a buffer memory in a multi-object encoding method according to the second embodiment of the present invention.

FIG. 9 is a diagram for explaining the present invention, and is an example of a graph showing an objective function and a trade-off function of each object in a multi-object encoding method according to the second embodiment of the present invention.

FIG. 10 is a diagram for explaining the present invention, and is an example of a graph showing a data retention amount in a buffer memory in a multi-object encoding method according to the second embodiment of the present invention.

FIG. 11 is a diagram for explaining the present invention, and is an example of a graph showing a data retention amount in a buffer memory in a multi-object encoding method according to the second embodiment of the present invention.

FIG. 12 is a diagram for explaining the present invention, and is an example of a graph showing an objective function and a trade-off function of each object in a multi-object encoding method according to the second embodiment of the present invention.

FIG. 13 is a diagram for explaining the present invention, and is an example of a graph showing a data retention amount in a buffer memory in a multi-object encoding method according to the second embodiment of the present invention.

FIG. 14 is a diagram for explaining the present invention, and is an example of a graph showing a data retention amount in a buffer memory in a multi-object encoding method according to the second embodiment of the present invention.

FIG. 15 is a diagram for explaining the present invention, and is an example of a graph showing an objective function and a trade-off function of each object in the encoding method of a plurality of objects according to the second embodiment of the present invention.

FIG. 16 is a diagram for explaining the present invention, and is an example of a graph showing a data retention amount in a buffer memory in a multi-object encoding method according to the second embodiment of the present invention.

FIG. 17 is a diagram for explaining the present invention, and is an example of a graph showing a data retention amount in a buffer memory in a multi-object encoding method according to the second embodiment of the present invention.

FIG. 18 is a diagram for explaining the present invention, and is an example of a graph showing an objective function and a trade-off function of each object in a multi-object encoding method according to the second embodiment of the present invention.

FIG. 19 is a configuration diagram of a conventional method of displaying a moving image divided into objects.

FIG. 20 is a detailed configuration diagram of a conventional method of displaying a moving image divided into objects.

[Explanation of symbols]

 Reference numeral 111: input moving image 112: shape extraction unit 114, 114a, 144b, to 144n: encoding unit 117: rate control unit 119: buffer memory 121: multiplexing unit 124, 156: encoding control unit 128 , 159... Motion-compensated inter-frame predictors 130 and 160... DCT converters 131 and 161... Second variable-length encoders 133 and 162. Variable length coding units 137, 165: Multiplexing unit 139: Buffer memory 153: Multiplexer 157: Shape coding unit 158: Coding controlled unit

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C052 AA17 AB04 DD04 GA01 GA07 GA08 GB06 GC05 GC07 GF06 5C059 KK01 LB07 MA00 MA05 MA23 MB01 MB14 MB16 MC11 ME02 NN01 NN21 NN28 NN43 PP04 RB01 RB18 RC16 RC19 SS07 SS20 TC07 TB46 TC14 TC15 TC37 TC47 TD11 TD16 UA02 UA32 UA38 UA39

Claims (7)

[Claims] An image encoding apparatus for encoding a moving image whose screen is composed of a plurality of objects in units of objects, multiplexing each encoded data of the plurality of objects, and transmitting the multiplexed data to a communication line. Means for encoding, wherein a distribution value of a bit rate for performing the multiplexing is set in accordance with a priority of a given image quality for a target object, and a generated code amount corresponding to the allocated bit rate. Encoding means for encoding the target object, and when an object with high activity occurs in the process of encoding the object, the image quality of the high-priority object is not reduced or improved. Set the bit rate distribution of the encoding means to redistribute the bit rate among the objects Image encoding device characterized by comprising a Gosuru control means. 2. The encoding device according to claim 1, wherein, when encoding each of the objects on a frame-by-frame basis, a balance between temporal distortion due to frame drop according to a coding rate and spatial distortion due to quantization accuracy is achieved. 2. The image encoding apparatus according to claim 1, further comprising a function of setting and controlling a bit rate distribution so as to secure an optimum operating point in terms of visual characteristics. 3. The control means according to claim 1, wherein, in the step of moving the bit rate between the objects, the bit rate distribution is controlled so that temporal distortion is reduced in the entire coded data transmitted to the communication line. 2. The image encoding apparatus according to claim 1, further comprising a function of performing setting control. 4. An image encoding method for encoding a moving image whose screen is composed of a plurality of objects on an object basis, multiplexing each encoded data of the plurality of objects, and transmitting the multiplexed data to a communication line. In addition to encoding, for the target object, the target object is encoded with a generated code amount corresponding to a given bit rate distribution value for performing the multiplexing in accordance with the priority of the image quality of the object. Encoding step, when a high activity object occurs in the process of encoding the object, the bit rate between the objects so as not to reduce or improve the image quality of the high priority object To set and control the bit rate distribution of the encoding step to redistribute Picture coding method characterized by comprising the steps. 5. In the control step, when each of the objects is encoded for each frame, a balance between time distortion due to frame drop according to a coding rate and spatial distortion due to quantization accuracy is achieved. 5. The image encoding method according to claim 4, further comprising a function of setting and controlling bit rate distribution so as to secure an optimal operating point in terms of visual characteristics. 6. The method according to claim 1, wherein, in the step of moving the bit rate between the objects, the bit rate is distributed so that temporal distortion is reduced in the entire coded data transmitted to the communication line. The image encoding method according to claim 4, further comprising a function of performing setting control. 7. An image encoding program for encoding a moving image whose screen is composed of a plurality of objects on an object basis, multiplexing each encoded data of the plurality of objects, and transmitting the multiplexed data to a communication line, Along with encoding the object, for the target object, the target object is generated with a generated code amount equivalent to a given bit rate distribution value for performing the multiplexing in accordance with the priority of the image quality of the object. An encoding step of encoding, and when a high activity object occurs in the process of encoding the object, the object quality is reduced between the objects so as not to reduce or improve the image quality of the high priority object. Set and control the bit rate distribution of the encoding step to redistribute the bit rate. Control and step, readable by a computer, characterized by comprising, and, a medium recording an executable program.