JPH07193818A - Image processing method and image processing apparatus - Google Patents

Abstract

(57) [Summary] [Purpose] To enable accurate detection of discontinuous scenes such as scene changes and camera flashes. [Structure] A motion vector detection circuit 102 predicts an input image and supplies a sum of absolute values of prediction errors to a selection circuit 104. The selection circuit 104 averages the sum of the absolute values of the prediction error for each frame over a plurality of frames, and compares the average value with the sum of the absolute values of the prediction errors that are individually input. When the state where the sum of absolute values of the prediction errors is larger than the average value independently occurs only once, it is determined as a scene change, and when it occurs twice in succession, it is determined as a camera flash.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method and an image processing apparatus, and more particularly to a method and apparatus suitable for use in compressing and transmitting moving image data.

[0002]

2. Description of the Related Art When moving image data is digitally recorded on a magneto-optical disk, a magnetic tape or the like, or transmitted through a predetermined transmission medium, the data is encoded and compressed to reduce the amount of data. I am trying.

For example, the MPEG image compression method is as follows.
The image of each frame is predicted in any one of the I-picture, P-picture, and B-picture prediction modes, and the prediction error is encoded and transmitted. Since only the prediction error is basically transmitted, the data amount can be reduced as compared with the case where the image data of each frame is transmitted as it is.

[0004]

In the conventional method, which of the I, P and B pictures is to be executed is determined by the GOP.
It is predetermined for each (Group of Picture), and the pattern of the prediction mode of this GOP is fixed regardless of the content of the image.

However, for example, if there is a scene change in a series of pictures to be transmitted, even if the picture of the frame after the scene change is predicted from the frame before the scene change, the prediction error becomes extremely large. , The image quality of the predicted frame deteriorates. Further, if the frame whose image quality is deteriorated is used as a predicted image and the image data of another frame is compressed, the image quality of the image data of the other frame also deteriorates.

[0006] Similarly, even when the scene change does not occur and the related pictures are continuous,
It also occurs when a picture of a predetermined frame of them is an image flashed by a camera.

The present invention has been made in view of such a situation, and suppresses deterioration of image quality due to a scene change, a camera flash, or the like.

[0008]

An image processing method according to claim 1 is an image processing method for processing image data of a picture, wherein an absolute value of a difference between image data in a predetermined range of a plurality of pictures is calculated. , Calculating the average of the absolute values of the differences between a plurality of pictures, comparing the absolute value of the differences between the individual pictures and the average value, and determining the continuity of the pictures according to the comparison result. Characterize.

According to a second aspect of the present invention, in the image processing method for processing image data of a picture, an average value of luminance of image data in a predetermined range of the picture is calculated as a first value, and a plurality of values are calculated. Calculating the average value of the first values of the pictures to obtain the second value, comparing the first value and the second value, and determining the continuity of the pictures according to the comparison result. Is characterized by.

A picture is predicted from another picture according to the continuity obtained by comparing the absolute value and the average value or the continuity obtained by comparing the first value and the second value. It is possible to select the prediction mode for the case.

The absolute value of the difference between the image data can be calculated between consecutive pictures or between pictures separated by two or more.

When an absolute value larger than the average value independently occurs only once, it is judged as a scene change of a picture, and 2
When it occurs consecutively, it can be determined as a camera flash of a picture.

Further, when the first value becomes larger than the second value, it can be determined that the picture is a camera flash.

Further, when the prediction mode for predicting a picture from another picture is selected corresponding to the continuity obtained by comparing the absolute value and the average value, the absolute value larger than the average value is selected. Occurs independently once,
It is determined that it is a scene change of a picture, and when it occurs twice in succession, it is determined to be a camera flash of the picture, and when it is a scene change, it is a picture of the scene change or a picture after it, which is a picture in the vicinity thereof. Process at least one of these as an I picture,
In the case of a camera flash, the picture of the camera flash can be processed as a B picture, or the picture immediately after it can be processed as an I picture.

Further, when the prediction mode for predicting a picture from another picture is selected corresponding to the continuity obtained by comparing the first value and the second value, the first value is When the value becomes larger than the second value, it can be determined as the camera flash of the picture, and the picture of the camera flash can be processed as the B picture or the picture immediately thereafter can be processed as the I picture.

An image processing apparatus can be realized by applying these methods.

[0017]

According to the image processing method of the present invention, the absolute value of the difference between the image data in the predetermined range of the plurality of pictures is calculated, and the average value of the absolute values of the differences is further calculated.
The continuity of the picture is determined according to the result of comparison between the absolute value of the difference and the average value. Therefore, it is possible to detect a scene change of a picture or a camera flash.

Further, in the image processing method according to the second aspect, the average value of the luminance of the image data in the predetermined range of the picture is calculated to obtain the first value, and the first value of the first value is calculated. The average value of a plurality of pictures is calculated as a second value, the first value and the second value are compared, and the continuity of the pictures is determined according to the comparison result. Therefore,
It is possible to detect the camera flash of a picture.

[0019]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In a system for transmitting a moving image signal to a remote place such as a video conference system or a video telephone system, for example, in order to efficiently use a transmission path, line correlation and inter-frame correlation of a video signal are By utilizing this, the image signal is compressed and encoded.

By using the line correlation, the image signal can be compressed by, for example, DCT (discrete cosine transform) processing.

Further, by utilizing the inter-frame correlation, it becomes possible to further compress and encode the image signal. For example, as shown in FIG. 1, when frame images PC1, PC2, and PC3 are generated at times t1, t2, and t3, respectively, the difference between the image signals of the frame images PC1 and PC2 is calculated to generate PC12. Also, the difference between the frame images PC2 and PC3 is calculated to generate PC23. Normally, the images of frames that are temporally adjacent do not have such a large change, so when the difference between the two is calculated, the difference signal has a small value. Therefore, if this difference signal is encoded, the code amount can be compressed.

However, if only the differential signal is transmitted, the original image cannot be restored. Therefore, the image of each frame is set to any one of three types of pictures of I picture, P picture, and B picture,
The image signal is compressed and encoded.

That is, for example, as shown in FIG. 2, 17 frames of image signals from frames F1 to F17 are set as a group of pictures (GOP), which is one unit of processing.
Then, the image signal of the leading frame F1 is encoded as an I picture, the second frame F2 is processed as a B picture, and the third frame F3 is processed as a P picture. Hereinafter, the fourth and subsequent frames F4 to F17 are alternately processed as a B picture or a P picture.

As the image signal of the I picture, the image signal for one frame is transmitted as it is. On the other hand, as the image signal of the P picture, basically, as shown in FIG.
, The difference from the image signal of the I picture or P picture that precedes it in time is transmitted. Further, as the image signal of the B picture, basically, as shown in FIG. 3, a difference from the average value of both the temporally preceding frame and the temporally following frame is obtained, and the difference is encoded.

FIG. 4 shows the principle of the method of encoding a moving image signal in this way. As shown in the figure,
Since the first frame F1 is processed as an I picture, it is directly transmitted as transmission data F1X to the transmission path (intra-picture coding). On the other hand, the second frame F
2 is processed as a B picture, a difference (prediction error) between the temporally preceding frame F1 and the temporally subsequent frame F3 average value (prediction error) is calculated, and the difference is transmitted as transmission data F2X. To be done.

However, there are four types of processing for the B picture, which will be described in more detail. The first process is to transmit the data of the original frame F2 as it is as the transmission data F2X (SP1) (intra (intra-picture prediction) coding), and is the same process as in the I picture. The second processing is to calculate the difference from the frame F3 that is temporally later and transmit the difference (SP2) (backward predictive coding). The third process is to transmit a difference (SP3) from the temporally preceding frame F1 (forward predictive coding). Further, the fourth process is to generate a difference (SP4) between the average value of the frame F1 preceding in time and the average value of the frame F3 following, and transmit this as the transmission data F2X (bidirectional predictive coding). .

Of these four methods, the method that minimizes the amount of transmitted data is adopted.

When the difference data is transmitted, a motion vector x1 (motion vector between the frames F1 and F2) between the difference image and the image of the frame (prediction image) for which the difference is calculated (in the case of forward prediction). , Or x2 (frame F
The motion vector between 3 and F2) (for backward prediction) or both x1 and x2 (for bidirectional prediction) are transmitted with the difference data.

In the frame F3 of the P picture, the frame F1 temporally preceding is used as a predicted image, the differential signal (SP3) from this frame and the motion vector x3 are calculated, and this is transmitted as the transmission data F3X. (Forward predictive coding). Alternatively, the data of the original frame F3 is directly transmitted as the data F3X (SP
1) (Intra coding). As in the case of the B picture, whichever method is used for transmission is selected so that less transmission data is transmitted.

FIG. 5 shows an example of the configuration of an apparatus which encodes and transmits a moving image signal based on the above-mentioned principle, and decodes this. The encoding device 1 encodes the input video signal and transmits it to the recording medium 3 as a transmission path. Then, the decryption device 2 uses the recording medium 3
It reproduces the signal recorded in, decodes it, and outputs it.

In the encoding device 1, the input video signal is input to the pre-processing circuit 11, where the luminance signal and the chrominance signal (color difference signal in this embodiment) are separated, and each is an A / D converter. A / D conversion is performed at 12 and 13. A /
The video signal converted into a digital signal by A / D conversion by the D converters 12 and 13 is supplied to and stored in the frame memory 14. The frame memory 14 stores the luminance signal in the luminance signal frame memory 15 and the color difference signal in the color difference signal frame memory 16, respectively.

The format conversion circuit 17 converts the frame format signal stored in the frame memory 14 into
Convert to block format signal. That is, as shown in FIG. 6, the video signal stored in the frame memory 14 is data in a frame format in which V dots of H dots are collected per line. The format conversion circuit 17 converts this 1-frame signal into 16
The line is used as a unit and divided into N slices.

Then, each slice is divided into M macroblocks. Each macroblock is composed of luminance signals corresponding to 16 × 16 pixels (dots),
The luminance signal is further divided into blocks Y [1] to Y [4] each having a unit of 8 × 8 dots. Then, the luminance signal of 16 × 16 dots includes Cb of 8 × 8 dots.
The signals correspond to the 8 × 8 dot Cr signals.

The data converted into the block format in this way is supplied from the format conversion circuit 17 to the encoder 18, where it is encoded. The details will be described later with reference to FIG. 7.

The signal encoded by the encoder 18 is output to the transmission line as a bit stream. For example, it is supplied to the recording circuit 19 and recorded on the recording medium 3 as a digital signal.

The data reproduced from the recording medium 3 by the reproducing circuit 30 is supplied to the decoder 31 of the decoding device 2 and decoded. For details of the decoder 31,
It will be described later with reference to FIG.

The data decoded by the decoder 31 is input to the format conversion circuit 32 and converted from the block format to the frame format. Then, the luminance signal of the frame format is supplied to and stored in the luminance signal frame memory 34 of the frame memory 33, and the color difference signal is supplied to and stored in the color difference signal frame memory 35. The luminance signal and the color difference signal read from the luminance signal frame memory 34 and the color difference signal frame memory 35 are D / A converters 36 and 37, respectively, to D / A
A-converted, supplied to the post-processing circuit 38, and combined.
Then, for example, it is output and displayed on a display such as a CRT (not shown).

Next, a configuration example of the encoder 18 will be described with reference to FIG. The image data to be encoded is input to the motion vector detection circuit 50 in macroblock units via the picture determination circuit 49. Details of the picture determination circuit 49 will be described later with reference to FIG.
The picture determination circuit 49 selects and determines the prediction mode.

The motion vector detection circuit 50 processes the image data of each frame as an I picture, P picture, or B picture according to the predetermined sequence set by the picture determination circuit 49. Which of I, P, and B pictures to process the images of each frame that is sequentially input is basically determined in advance (for example, as shown in FIG. 2 and FIG. The group of pictures composed of F1 to F17 is processed as I, B, P, B, P, ... B, P), but as will be described later, the picture determination circuit 49
Thus, when a scene change or a camera flash is detected, the prediction mode is adaptively changed.

The image data of a frame (for example, frame F1) processed as an I picture is transmitted from the motion vector detection circuit 50 to the front original image portion 51a of the frame memory 51.
The image data of the frame (for example, frame F2) that is transferred and stored in the reference original image section 51b is the image data of the frame (for example, frame F3) that is transferred and stored in the reference original image portion 51b. , And is transferred to and stored in the rear original image portion 51c.

Further, at the next timing, B
When an image of a frame to be processed as a picture (frame F4) or P picture (frame F5) is input, the image data of the first P picture (frame F3) stored in the backward original image portion 51c until then is It is transferred to the front original image portion 51a, and the next B picture (frame F
The image data of 4) is stored (overwritten) in the reference original image portion 51b, and the image data of the next P picture (frame F5) is stored (overwritten) in the backward original image portion 51c. Such an operation is sequentially repeated.

The signal of each picture stored in the frame memory 51 is read therefrom, and the prediction mode switching circuit 52 performs frame prediction mode processing or field prediction mode processing.

Furthermore, under the control of the prediction determination circuit 54, the arithmetic unit 53 performs intra-picture prediction, forward prediction, backward prediction, or bidirectional prediction. Which of these processes is to be performed is determined in accordance with the prediction error signal (difference between the reference image to be processed and the predicted image corresponding thereto). Therefore, the motion vector detection circuit 50 generates the sum of absolute values (or the sum of squares) of the prediction error signal used for this determination.

Here, the frame prediction mode and field prediction mode in the prediction mode switching circuit 52 will be described.

When the frame prediction mode is set, the prediction mode switching circuit 52 supplies the four luminance blocks Y supplied from the motion vector detection circuit 50.
[1] to Y [4] are directly output to the arithmetic unit 53 in the subsequent stage. That is, in this case, as shown in FIG. 8, the data of the odd field lines and the data of the even field lines are mixed in each luminance block. In this frame prediction mode, 4
Prediction is performed in units of luminance blocks (macroblocks), and one motion vector is associated with four luminance blocks.

On the other hand, the prediction mode switching circuit 5
In the field prediction mode, 2 is a signal input from the motion vector detection circuit 50 in the configuration shown in FIG.
As shown in FIG. 9, of the four luminance blocks, the luminance blocks Y [1] and Y [2] are constituted by, for example, only dots of lines in odd fields, and the other two luminance blocks Y [3]. ] And Y [4] are composed of the data of the even field lines and output to the arithmetic unit 53. In this case, two luminance blocks Y [1] and Y [1]
[2] corresponds to one motion vector, and the other two luminance blocks Y [3] and Y [4] correspond to another motion vector.

The motion vector detection circuit 50 outputs the sum of absolute values of prediction errors in the frame prediction mode and the sum of absolute values of prediction errors in the field prediction mode to the prediction mode switching circuit 52. Prediction mode switching circuit 5
2 compares the absolute value sums of the prediction errors in the frame prediction mode and the field prediction mode, performs a process corresponding to the prediction mode having the smaller value, and outputs the data to the arithmetic unit 53.

However, such processing is actually performed by the motion vector detection circuit 50. That is, the motion vector detection circuit 50 outputs a signal having a configuration corresponding to the determined mode to the prediction mode switching circuit 52, and the prediction mode switching circuit 52 outputs the signal as it is to the arithmetic unit 53 in the subsequent stage.

In the frame prediction mode, the color difference signal is supplied to the arithmetic unit 53 in a state where the data of the odd field lines and the data of the even field lines are mixed, as shown in FIG. In the field prediction mode, as shown in FIG. 9, each color difference block Cb,
The upper half (4 lines) of Cr is a luminance block Y [1],
The color difference signal of the odd field corresponding to Y [2],
The lower half (4 lines) is the luminance block Y [3], Y
The color difference signal of the even field corresponding to [4] is set.

Further, the motion vector detection circuit 50 uses the intra-picture prediction in the prediction judgment circuit 54 as follows.
A sum of absolute values of prediction errors for determining whether to perform forward prediction, backward prediction, or bidirectional prediction is generated.

That is, the sum ΣAij of the signals Aij of the macroblocks of the reference image is used as the sum of the absolute values of the prediction errors of the intra-picture prediction.
Of the absolute value | ΣAij | of the macroblock signal and the sum Σ | Aij | of the absolute value | Aij | of the macroblock signal Aij. Also, as the sum of absolute values of prediction errors in forward prediction, the sum Σ | A of the absolute value | Aij-Bij | of the difference Aij-Bij between the signal Aij of the macroblock of the reference image and the signal Bij of the macroblock of the predicted image.
ij-Bij | is calculated. Further, the sum of absolute values of the prediction errors of the backward prediction and the bidirectional prediction is also obtained in the same manner as in the case of forward prediction (the predicted image is changed to a predicted image different from that in forward prediction).

The sum of these absolute values is supplied to the prediction judgment circuit 54. The prediction determination circuit 54 selects the smallest sum of absolute values of prediction errors of forward prediction, backward prediction, and bidirectional prediction as the sum of absolute values of prediction errors of inter prediction. Furthermore, the sum of the absolute values of the prediction error of the inter prediction and the sum of the absolute values of the prediction errors of the intra-picture prediction are compared, the smaller one is selected, and the mode corresponding to the selected sum of the absolute values is set as the prediction mode. select. That is, if the sum of absolute values of prediction errors in intra-picture prediction is smaller, the intra-picture prediction mode is set. If the sum of absolute values of prediction errors in inter prediction is smaller, the mode in which the corresponding sum of absolute values is the smallest is set among the forward prediction, backward prediction, and bidirectional prediction modes.

In this way, the motion vector detection circuit 50
Supplies the signal of the macroblock of the reference image to the arithmetic unit 53 via the prediction mode switching circuit 52 in a configuration corresponding to the mode selected by the prediction mode switching circuit 52 among the frame or field prediction modes. Of the four prediction modes, the prediction determination circuit 54
The motion vector between the prediction image and the reference image corresponding to the prediction mode selected by is detected, and the variable length coding circuit 5
8 and the motion compensation circuit 65. As described above, the motion vector that minimizes the sum of absolute values of the corresponding prediction errors is selected.

The prediction determination circuit 54, when the motion vector detection circuit 50 is reading the image data of the I picture from the forward original image portion 51a, uses the intra-frame or field (image) prediction mode (motion compensation is performed as the prediction mode. Mode), and the switch 53d of the calculation unit 53 is set.
To the contact a side. As a result, the I-picture image data is input to the DCT mode switching circuit 55.

As shown in FIG. 10 or 11, the DCT mode switching circuit 55 is a state in which the data of four luminance blocks are mixed with the lines of odd fields and the lines of even fields (frame DCT mode), or , The separated state (field DCT mode), and outputs to the DCT circuit 56.

That is, the DCT mode switching circuit 55 is
Mixed data of odd field and even field D
Comparing the coding efficiency in the case of CT processing and the coding efficiency in the case of DCT processing in the separated state,
Select a mode with good coding efficiency.

For example, as shown in FIG. 10, the input signal has a structure in which odd field lines and even field lines are mixed, and the difference between the signal of the odd field line and the signal of the even field line that are vertically adjacent to each other. Is calculated, and the sum (or sum of squares) of the absolute values is calculated.

As shown in FIG. 11, the input signal has a structure in which the odd field lines and the even field lines are separated, and the difference between the signals of the vertically adjacent odd field lines and the even field lines. Calculates the difference between signals and calculates the sum of absolute values (or sum of squares)
Ask for.

Further, both (sum of absolute values) are compared and a DCT mode corresponding to a smaller value is set. That is, if the former is smaller, the frame DCT mode is set, and if the latter is smaller, the field DCT mode is set.

Then, the data having the structure corresponding to the selected DCT mode is output to the DCT circuit 56, and the DCT flag indicating the selected DCT mode is output to the variable length coding circuit 58, the DCT block rearranging circuit 62, and the motion. Output to the compensation circuit 65.

As is clear from a comparison between the prediction mode in the prediction mode switching circuit 52 (FIGS. 8 and 9) and the DCT mode in the DCT mode switching circuit 55 (FIGS. 10 and 11), regarding the luminance block, The data structures in both modes are substantially the same.

When the frame prediction mode (a mode in which odd lines and even lines are mixed) is selected in the prediction mode switching circuit 52, the DCT mode switching circuit 55 is also in the frame DCT mode (odd lines and even lines are mixed). If the field prediction mode (a mode in which the data in the odd field and the data in the even field are separated) is selected in the prediction mode switching circuit 52, the field in the DCT mode switching circuit 55 is high. There is a high possibility that the DCT mode (mode in which the data of the odd field and the data of the even field are separated) is selected.

However, this is not always the case, and the prediction mode switching circuit 52 determines the mode so that the sum of the absolute values of the prediction errors becomes smaller, and the DCT mode switching circuit 55 encodes the mode. The mode is determined so that the efficiency is good.

The I-picture image data output from the DCT mode switching circuit 55 is input to the DCT circuit 56, subjected to DCT (discrete cosine transform) processing, and converted into DCT coefficients. This DCT coefficient is input to the quantization circuit 57, quantized in a quantization step corresponding to the data storage amount (buffer storage amount) of the transmission buffer 59, and then,
It is input to the variable length coding circuit 58.

The variable length coding circuit 58 is a quantization circuit 57.
In accordance with the supplied quantization step (scale), the image data (in this case, I picture data) supplied from the quantization circuit 57 is converted into a variable length code such as a Huffman code and transmitted. Output to the buffer 59.

In the variable length coding circuit 58, a quantization step (scale) is set by the quantization circuit 57, and a prediction mode (intra-picture prediction, forward prediction, backward prediction, or bidirectional prediction is set by the prediction determination circuit 54. Mode indicating whether or not the motion vector is detected by the motion vector detection circuit 50,
A prediction flag (a flag indicating whether the frame prediction mode or the field prediction mode has been set) is set by the prediction mode switching circuit 52, and a DCT flag (whether the frame DCT mode or the field DCT mode is set by the DCT mode switching circuit 55 is set. Has been input, and these are also variable length coded.

The transmission buffer 59 temporarily stores the input data and stores the data corresponding to the storage amount in the quantizing circuit 57.
Output to. When the remaining amount of data increases to the allowable upper limit value, the transmission buffer 59 increases the quantization scale of the quantization circuit 57 by the quantization control signal,
The amount of quantized data is reduced. On the contrary, when the data remaining amount decreases to the allowable lower limit value, the transmission buffer 59 uses the quantization control signal to quantize circuit 57.
The data amount of the quantized data is increased by reducing the quantization scale of. In this way, overflow or underflow of the transmission buffer 59 is prevented.

Then, the data accumulated in the transmission buffer 59 is read out at a predetermined timing, outputted to the transmission path, and recorded on the recording medium 3 via the recording circuit 19, for example.

On the other hand, the I picture data output from the quantization circuit 57 is input to the inverse quantization circuit 60 and inversely quantized in accordance with the quantization step supplied from the quantization circuit 57. The output of the inverse quantization circuit 60 is IDCT
It is input to the (inverse DCT) circuit 61, subjected to inverse DCT processing, and then input to the DCT block rearranging circuit 62. D
The CT block rearrangement circuit 62 rearranges the input data according to the prediction flag supplied from the prediction mode switching circuit 52 and the DCT flag supplied from the DCT mode switching circuit 55.

That is, when the frame prediction mode is set in the prediction mode switching circuit 52, the data read from the motion compensation circuit 65 and supplied to the arithmetic unit 53 is the odd field data and the even field data. It is said that the and are mixed. This data is also supplied to the calculator 63. Therefore, the DCT block rearrangement circuit 62 converts the data supplied from the IDCT circuit 61 when the frame DCT mode is set.
When the field DCT mode is set, the data in the odd-numbered field and the data in the even-numbered field are separated, so that the data is rearranged in a mixed state. , To the computing unit 63.

On the other hand, when the field prediction mode is set in the prediction mode switching circuit 52, the data supplied from the motion compensation circuit 65 to the arithmetic unit 53 is the data of the odd field and the data of the even field separated. It is in a state. Therefore, when the field DCT mode is set by the DCT mode switching circuit 55, the DCT block rearranging circuit 62 I
The data output from the DCT circuit 61 is supplied to the arithmetic unit 63 as it is. However, when the frame DCT mode is set, the data of the odd field and the data of the even field are mixed. ,
Each is sorted into a separated state and output to the computing unit 63.

That is, the DCT block rearrangement circuit 62.
Is IDCT so that it has the same arrangement state as the arrangement state of the data supplied from the motion compensation circuit 65 to the arithmetic unit 53.
The data output from the circuit 61 is rearranged.

In this case, since the data output from the IDCT circuit 61 is I picture data, it is assumed to be intra-picture prediction. Therefore, when the DCT mode switching circuit 55 is outputting the frame DCT flag, I
The data output from the DCT circuit 61 is directly transmitted via the calculator 63 to the forward prediction image section 6 of the frame memory 64.
4a and stored. Also, the field DCT
When the flag is output, the data is sorted and then stored.

When the motion vector detection circuit 50 processes the sequentially input image data of each frame as a picture of I, B, P, B, P, B ... After processing the image data of another frame as an I picture, before processing the image of the next input frame as a B picture, the image data of the next input frame is processed as a P picture. This is because a B picture is accompanied by backward prediction and cannot be decoded unless a P picture as a backward predicted image is prepared in advance.

Then, the motion vector detecting circuit 50 starts the processing of the image data of the P picture stored in the backward original image portion 51c after the processing of the I picture. Then, as in the case described above, the sum of absolute values of the inter-frame difference (prediction error) in macroblock units is supplied from the motion vector detection circuit 50 to the prediction mode switching circuit 52 and the prediction determination circuit 54. Prediction mode switching circuit 52
And the prediction determination circuit 54 sets the frame / field prediction mode, or the prediction mode of intra-picture prediction, forward prediction, backward prediction, or bidirectional prediction in accordance with the sum of the absolute values of the prediction errors of the macroblock of the P picture. To do.

When the intra-picture prediction mode is set, the arithmetic unit 53 switches the switch 53d to the contact a side as described above. Therefore, this data is transmitted to the transmission line via the DCT mode switching circuit 55, the DCT circuit 56, the quantization circuit 57, the variable length coding circuit 58, and the transmission buffer 59, like the I picture data. Further, this data is the inverse quantization circuit 60, the IDCT circuit 61, the DC
It is supplied to and stored in the backward predicted image portion 64b of the frame memory 64 via the T block rearrangement circuit 62 and the computing unit 63.

In the forward prediction mode, the switch 53d is switched to the contact b, and the image data (in this case, the I picture image) stored in the forward prediction image portion 64a of the frame memory 64 is read out. The motion compensation circuit 65 performs motion compensation corresponding to the motion vector output from the motion vector detection circuit 50. That is, when the prediction determination circuit 54 instructs the motion compensation circuit 65 to set the forward prediction mode, the motion compensation circuit 65 sets the read address of the forward predicted image portion 64a to the position of the macroblock currently output by the motion vector detection circuit 50. Data is read out by shifting from the corresponding position by an amount corresponding to the motion vector, and predicted image data is generated.

The predicted image data output from the motion compensation circuit 65 is supplied to the calculator 53a. Calculator 53a
Subtracts the predicted image data corresponding to this macroblock supplied from the motion compensation circuit 65 from the data of the macroblock of the reference image supplied from the prediction mode switching circuit 52, and outputs the difference (prediction error). To do. This difference data is the DCT mode switching circuit 55, DC
T circuit 56, quantization circuit 57, variable length coding circuit 58,
It is transmitted to the transmission line via the transmission buffer 59. Also,
This difference data is stored in the inverse quantization circuit 60 and the IDCT circuit 6
1, locally decoded by the DCT block rearrangement circuit 62 and input to the arithmetic unit 63.

The same data as the predicted image data supplied to the calculator 53a is also supplied to the calculator 63. The calculator 63 adds the predicted image data output by the motion compensation circuit 65 to the difference data output by the DCT block rearrangement circuit 62. As a result, the image data of the original (decoded) P picture is obtained. The image data of the P picture is supplied to and stored in the backward predicted image portion 64b of the frame memory 64.

In this way, the motion vector detection circuit 50 executes the processing of the B picture after the data of the I picture and the P picture are stored in the forward prediction image section 64a and the backward prediction image section 64b, respectively. The prediction mode switching circuit 52 and the prediction determination circuit 54 correspond to the magnitude of the sum of absolute values of inter-frame differences in macroblock units,
The frame / field mode is set, and the prediction mode is set to any of the intra-picture prediction mode, the forward prediction mode, the backward prediction mode, and the bidirectional prediction mode.

As described above, in the intra-picture prediction mode or the forward prediction mode, the switch 53d has the contact a or b.
Can be switched to. At this time, the same processing as in the P picture is performed and the data is transmitted.

On the other hand, when the backward prediction mode or the bidirectional prediction mode is set, the switch 53d is switched to the contact c or d, respectively.

In the backward prediction mode in which the switch 53d is switched to the contact c, the image (in this case, P picture image) data stored in the backward predicted image section 64b is read out and the motion compensation circuit 65 is read. Thus, motion compensation is performed corresponding to the motion vector output by the motion vector detection circuit 50. That is, the motion compensation circuit 65 sets the read address of the backward predicted image portion 64b to the position of the macroblock currently output by the motion vector detection circuit 50 when the backward determination mode setting is instructed by the prediction determination circuit 54. Data is read out by shifting from the corresponding position by an amount corresponding to the motion vector, and predicted image data is generated.

The predicted image data output from the motion compensation circuit 65 is supplied to the calculator 53b. Calculator 53b
Subtracts the prediction image data supplied from the motion compensation circuit 65 from the macroblock data of the reference image supplied from the prediction mode switching circuit 52, and outputs the difference. This difference data is stored in the DCT mode switching circuit 5
5, the DCT circuit 56, the quantization circuit 57, the variable length coding circuit 58, and the transmission buffer 59.

In the bidirectional prediction mode in which the switch 53d is switched to the contact point d, the image (in this case, I picture image) data stored in the forward prediction image section 64a and the backward prediction image section 64b are stored. Image data (in this case, an image of P picture) is read out, and the motion compensation circuit 65 causes the motion vector detection circuit 5 to read.
Motion compensation is performed according to the motion vector output by 0.

That is, in the motion compensation circuit 65, when the bidirectional prediction mode setting is instructed by the prediction determination circuit 54, the motion vector detection circuit 50 sets the read addresses of the forward predicted image portion 64a and the backward predicted image portion 64b to the read addresses. The data is read out by shifting the data from the position corresponding to the position of the output macroblock by the amount corresponding to the motion vector (the motion vector in this case is two for the forward prediction image and the backward prediction image) Generate data.

The predicted image data output from the motion compensation circuit 65 is supplied to the calculator 53c. Calculator 53c
Subtracts the average value of the predicted image data supplied from the motion compensation circuit 65 from the data of the macroblock of the reference image supplied from the motion vector detection circuit 50, and outputs the difference. This difference data is transmitted to the transmission line via the DCT mode switching circuit 55, the DCT circuit 56, the quantization circuit 57, the variable length coding circuit 58, and the transmission buffer 59.

The B picture image is not stored in the frame memory 64 because it is not used as a predicted image of another image.

In the frame memory 64, the forward predictive image portion 64a and the backward predictive image portion 64b are bank-switched as necessary, and a predetermined reference image
The one stored in one or the other can be switched and output as the forward prediction image or the backward prediction image.

In the above description, the luminance block is mainly described, but the same applies to the color difference block in FIG.
Through the macroblocks shown in FIG. 11 are processed and transmitted. The motion vector used for processing the color difference block is obtained by halving the motion vector of the corresponding luminance block in each of the vertical direction and the horizontal direction.

Next, the picture determination circuit 49 of FIG. 7 will be described. The picture determination circuit 49 is, for example, as shown in FIG.
2 and, in this embodiment, a prediction error of a picture stored in the frame memory 101 and a frame memory 101 having a front original image portion 101a, a reference original image portion 101b, and a rear original image portion 101c. And a motion vector detection circuit 102 that outputs the sum of the absolute values to the selection circuit 104. In addition, a brightness DC value calculator 103 is provided as necessary, calculates a DC value (average value) of brightness data in a predetermined range of the input image, and selects the brightness DC value obtained by the calculation. Circuit 104
Is output to.

The selection circuit 104 detects at least one of a scene change and a camera flash from the sum of absolute values of prediction errors input from the motion vector detection circuit 102 or the brightness DC value input from the brightness DC value calculator 103. Then, the picture type is selected and determined according to the detection result. The picture type signal defining the decided picture type is supplied to the motion vector detection circuit 50 of FIG.

The image data input to the picture determination circuit 49 is delayed by the delay circuit 105 for a time necessary for the selection circuit 104 to determine the picture type, and then supplied to the motion vector detection circuit 50. It is designed to be done.

The input image data is supplied to and stored in the frame memory 101 via the motion vector detection circuit 102. Then, the prediction process is performed as in the case of the motion vector detection circuit 50 of FIG. 7, and the sum of absolute values of prediction errors is calculated. The selection circuit 104 calculates the average value of the sum of absolute values input for each frame for a predetermined number of frames. For example, the average value of the absolute value sums for the latest 5 frames is calculated. As a result, this average value is updated every time the sum of absolute values of prediction errors for a new frame is input.

Then, the selection circuit 104 executes the processing shown in FIG. 13 every time the sum of absolute values of prediction errors of a new frame is input.

That is, first, in step S1, a value obtained by multiplying the average value MAE of the absolute value sums of the prediction errors for the immediately preceding five frames by N times (N is a positive real number, for example, 3 times),
It is compared with the sum AE _i of the absolute values of the prediction error of the newly input i-th frame. Regarding the meaning of this comparison, FIG.
Will be described with reference to.

Now, as shown in FIG. 14A, a scene 1 in which pictures from the 0th frame to the 5th frame are continuous and a picture from the 6th frame to the 11th frame is different from the scene 2 Is a picture of. That is, in this case, a scene change has occurred between the fifth frame and the sixth frame.

In the case of this embodiment, the prediction distance (distance of P picture) is set to 1. Therefore, the predicted image is the picture one frame before. Then, basically, the first picture is set as an I picture, and all subsequent pictures are processed as P pictures.
It is predetermined.

FIG. 14B shows a change in sum of absolute values of prediction errors of the picture shown in FIG. 14
In (B), the sum of the absolute values of the prediction errors of the frame number 0 represents the sum of the absolute values of the prediction errors between the pictures of the 0th frame and the 1st frame. Further, the sum of absolute values of prediction errors of frame number 1 represents the sum of absolute values of prediction errors between the first frame and the second frame. Similarly, the sum of the absolute values of the prediction errors of the frame number 4 represents the sum of the absolute values of the prediction errors between the pictures of the fourth frame and the fifth frame. Since the pictures from the 0th frame to the 5th frame are pictures of the same scene 1 as shown in FIG. 14A, the sum of absolute values of the prediction errors is close to the average value MAE. It is a value.

Similarly, from the sum of the absolute values of the prediction errors of the sixth frame and the seventh frame at the position of frame number 6, the 10th frame and the 11th frame of frame number 10 are calculated.
The values up to the sum of the absolute values of the prediction errors of the frame are the sum of the absolute values of the prediction errors of the same scene 2, so that the average value MA
It is close to E.

On the other hand, the sum of absolute values of prediction errors between the fifth frame and the sixth frame in frame number 5 is the sum of absolute values of prediction errors between pictures in which a scene change occurs. , Its value is the average value M
The value is sufficiently larger than AE. Therefore, the average value M
Comparing the N times MAE value N × MAE with the absolute value sum AE _i of the individual prediction errors, if no scene change occurs, the absolute value sum AE of the prediction errors of the i-th frame
_i is a value smaller than N × MAE. On the contrary,
When a scene change occurs, AE _i is N × M
The value is AE or more. Therefore, the presence or absence of a scene change can be determined by comparing AE _i and N × MAE.

When it is determined in step S1 in FIG. 13 that no scene change has occurred (AE _i is
If it is determined that it is smaller than N × MAE), step S
In step 5, the preset picture type is selected as it is.

On the other hand, in step S1, AE _i is
When it is determined that the value is equal to or larger than N × MAE, the process proceeds to step S2, and the absolute value sum AE _i of the prediction errors of the i + 1 th frame next to the i th frame detected in step S1. The value of ₊₁ is the reference value N × MAE
Compared to. The meaning of this comparison will be described with reference to FIG.

Not only the scene change shown in FIG. 14 but also the sum of absolute values of the prediction errors suddenly increases.
As shown in FIG. 15, it also occurs in the case of a camera flash. That is, in the example of FIG. 15A, if the picture of the sixth frame of the pictures from the 0th frame to the 11th frame is a camera flash picture, this camera flash picture is The brightness is higher than that of the picture.

As a result, as shown in FIG. 15B, the sum of the absolute values of the prediction errors of the fifth and sixth frames and the sum of the absolute values of the prediction errors of the sixth and seventh frames are: The value is larger than the reference value N × MAE. On the other hand, for example, the sum of absolute values of prediction errors between the fourth frame and the fifth frame or the sum of absolute values of prediction errors between the seventh frame and the eighth frame is Since each of them is not a picture of the camera flash, the value is close to the average value MAE.

As is apparent from comparison between FIG. 15B and FIG. 14B, in the case of a scene change, the case where the sum of absolute values of prediction errors exceeds the reference value is 1 It occurs only once. On the other hand, in the case of the camera flash, the case where the sum of absolute values of prediction errors is equal to or larger than the reference value occurs twice in succession.

Therefore, in step S1, the absolute value sum AE _i of the prediction errors of the i-th frame is calculated as the reference value N × MA.
When it is determined that it is larger than E, in step S2, it is further determined whether or not the sum AE _{i + 1} of absolute values of the prediction errors of the frame i + 1 frame immediately thereafter is larger than the reference value N × MAE. It is possible to determine whether it is a scene change or a camera flash.

In the case of a scene change, i
The sum of the absolute values of the prediction errors of the + 1st frame is smaller than the reference value. On the other hand, in the case of the camera flash, the sum of absolute values of the prediction errors of the (i + 1) th frame also becomes a value equal to or larger than the reference value. Therefore, step S
2, when it is determined that the sum AE _{i + 1} of the absolute values of the prediction error of the i + 1-th frame is smaller than the reference value N × MAE (when it is determined that it is a scene change), the process proceeds to step S3, and the scene The optimum picture type for change is selected. On the other hand, in step S2, the absolute value sum A of the prediction error of the (i + 1) th frame A
When it is determined that E _{i + 1} is larger than the reference value N × MAE (when it is determined that it is a camera flash), the process proceeds to step S4, and the optimization process of the picture type in the camera flash is executed.

FIG. 16 shows an example of picture type optimization processing in the scene change of step S3. In this example, the processing of the picture of the sixth frame in which the scene has changed is changed from the processing of the P picture to the processing of the I picture. Therefore, the sixth frame picture, the seventh frame picture in which the sixth frame picture is the predicted image, and the eighth frame picture in which the seventh frame picture is the predicted image,
It is possible to prevent the prediction error of each picture after the sixth frame from increasing.

Alternatively, the sixth frame remains P
It is also possible to process it as a picture and process the next seventh frame as an I picture. By doing this, the picture of the sixth frame of scene 2 will be
Since the fifth frame of scene 1 is predicted as the predicted image, the prediction error increases and the image quality deteriorates. However, the picture of the subsequent seventh frame is processed as an I picture, and thus the image quality deteriorates. , Is prevented from being propagated to the picture of the subsequent frame.

FIG. 17 shows an example of the optimization process of the picture type of the camera flash in step S4. In this example, the 6th camera flash
The prediction process of the seventh frame following the frame is changed from the P picture to the I picture. For this reason, the picture of the sixth frame in which the camera flash was present is predicted from the picture of the fifth frame immediately before it, so its image quality deteriorates, but the deterioration of the image quality propagates to the processing of the seventh frame and thereafter. Is prevented.

As described above, the selection circuit 10 of FIG.
4 selects and determines a predetermined picture type and outputs the picture type signal to the motion vector detection circuit 50. The motion vector detection circuit 50 processes the input image data supplied from the delay circuit 105 as described above according to this picture type.

In the above, the scene change and the camera flash are detected from the sum of the absolute values of the prediction errors, but it is also possible to detect the camera flash using the luminance DC value. FIG. 18 shows the principle. That is, the brightness DC value calculator 103 calculates a DC value (average value) of brightness values from the input image data.

As shown in FIG. 18A, if the camera flash occurs in the sixth frame and the camera flash does not occur in other frames, as shown in FIG. The luminance Y _i of the picture of each frame other than the frame is a value close to the average value MY of the luminance of several frames. On the other hand, the luminance Y ₆ of the picture of the sixth frame generated by the camera flash is a value M times (M is a positive real number, for example, 3 times) the average value MY or more. Therefore, by comparing the sum Y _i of the average luminance values of the i-th frame with the reference value M × MY, it is possible to determine whether or not a camera flash has occurred.

Therefore, when the brightness DC value calculator 103 outputs the sum of the average values of the brightness of each frame, the selection circuit 104 of FIG. 12 executes the processing shown in FIG. That is, in step S11, the brightness DC value calculator 103
The sum Y _i of the average values of the brightness of each frame input from
Compare with the reference value M × MY. As shown in FIG. 18, if the camera flash has not occurred, this Y _i becomes a value smaller than the reference value M × MY. In this case, the process proceeds to step S14 and the preset picture type is selected as it is.

On the other hand, in step S11,
If it is determined that the value of Y _i is greater than or equal to the reference value M × MY, the process proceeds to step S12, and the sum Y _{i + 1} of the average values of the brightness of the next frame is larger than the reference value M × MY. It is determined whether or not. Since the camera flash is generated instantaneously, the brightness of the picture of one frame has a large value, and the brightness of the pictures of the frames before and after that often returns to the normal brightness level.
On the contrary, when the brightness level greatly increases over two or more frames, it is considered that the brightness of the original picture, not the camera flash, has increased as a whole.

Therefore, in step S12, i + 1
The sum Y _{i + 1} of the average luminance values of the th frame is the reference value M
When it is determined that the value is larger than × MY, the process proceeds to step S14, the camera flash is not determined, and the preset picture type is selected as it is.

On the other hand, when it is determined that the sum Y _{i + 1} of the average luminance values of the i + 1-th frame is smaller than the reference value M × MY, that is, the luminance level of one frame is
If it is determined that the brightness is higher than the brightness of the frames before and after that, it is determined to be the camera flash, the process proceeds to step S13, and the optimization process of the picture type corresponding to the camera flash is executed.

When detecting the brightness of each frame, as shown in FIG. 20, it is possible to calculate the average value of the brightness of all pixels (X × Y pixels) of the frame, Alternatively, it is also possible to calculate the average value of the luminance of m × n pixels in a predetermined range among the X × Y pixel data. In this case, m × n ranges are
It may be the center of the X × Y range, or may be the upper left, upper right, lower left, lower right, or the like.

FIG. 21 is a block diagram showing the structure of an embodiment of the decoder 31 shown in FIG. Transmission line (recording medium 3)
The encoded image data transmitted via the receiving unit is received by a receiving circuit (not shown) or reproduced by the reproducing circuit 30,
After being temporarily stored in the reception buffer 81 of the decoder 31,
It is supplied to the variable length decoding circuit 82 of the decoding circuit 90. The variable length decoding circuit 82 performs variable length decoding on the data supplied from the reception buffer 81, and calculates the motion vector, the prediction mode, the prediction flag and the DCT flag by the motion compensation circuit 88.
In addition, the quantization step (scale) is output to the inverse quantization circuit 83, and the decoded image data is output to the inverse quantization circuit 83. Furthermore, the DCT flag and the prediction flag are set to the DCT block rearrangement circuit 85.
Output to.

The inverse quantization circuit 83 is used in the variable length decoding circuit 8
The image data supplied from No. 2 is inversely quantized according to the quantization step supplied from the variable length decoding circuit 82, and output to the IDCT circuit 84. The data (DCT coefficient) output from the inverse quantization circuit 83 is the IDCT circuit 8
In step 4, inverse DCT processing is performed to restore the original image data.

This image data is further input to the DCT block rearrangement circuit 85. The DCT block rearrangement circuit 85 corresponds to the DCT flag and the prediction flag,
This data is rearranged so that it has the same arrangement state as the data output by the motion compensation circuit 88 to the arithmetic unit 86, and is output to the arithmetic unit 86.

When the image data supplied from the DCT block rearrangement circuit 85 is I picture data,
The data is output from the computing unit 86, and is supplied to the forward prediction image unit 87a of the frame memory 87 to generate predicted image data of image data (P picture or B picture data) that is input to the computing unit 86 later. Will be remembered. In addition, this data is the format conversion circuit 32.
(Fig. 5).

The image data supplied from the DCT block rearrangement circuit 85 is one frame before (two frames before when the original image order is I, B, P).
If the data is a P-picture in which the image data of P is the predicted image data, and the data is in the forward prediction mode, the image data of one frame before (I-picture) stored in the forward prediction image unit 87a of the frame memory 87 is stored. data from)
Is read out, and the motion compensation circuit 88 uses the variable length decoding circuit 8
Motion compensation corresponding to the motion vector output from 2 is performed. Then, in the arithmetic unit 86, the image data (difference data) supplied from the DCT block rearrangement circuit 85 is added and output. The added data, that is, the decoded P picture data is calculated by the arithmetic unit 8
6 is supplied to and stored in the backward predicted image section 87b of the frame memory 87 for generation of predicted image data of image data (B picture or P picture data) input later.

Even in the case of P-picture data, the intra-picture prediction mode data is stored in the backward-prediction image section 87b as it is without any special processing by the arithmetic unit 86, as in the I-picture data.

In this P picture, the original order of images is
When it is I, P, P, P, it is output as it is,
When it is I, B, P, B, or P, it is an image to be displayed next to the next B picture, and therefore it is not yet output to the format conversion circuit 32 at this point (as described above, the B picture The P picture input later is processed and transmitted before the B picture).

When the image data supplied from the DCT block rearrangement circuit 85 is B picture data,
Corresponding to the prediction mode supplied from the variable length decoding circuit 82, the image data of the I picture stored in the forward prediction image section 87a of the frame memory 87 (in the case of the forward prediction mode) and the backward prediction image section 87b. Stored P-picture image data (in backward prediction mode) or both image data (in bidirectional prediction mode)
Is read out, and the motion compensation circuit 88 performs motion compensation corresponding to the motion vector output from the variable length decoding circuit 82 to generate a predicted image. However, when motion compensation is not required (in the case of the intra-picture prediction mode), the predicted picture is not generated.

The data thus motion-compensated by the motion compensation circuit 88 is processed by the arithmetic unit 86 by the DCT.
It is added to the output of the block rearrangement circuit 85. This addition output is output to the format conversion circuit 32.

However, since this addition output is B picture data and is not used for generating a predicted image of another image, it is not stored in the frame memory 87.

After the B-picture image is output, the P-picture image data stored in the backward-predicted image section 87b is read out and sent to the format conversion circuit 32 via the motion compensation circuit 88 and the arithmetic unit 86. Supplied. However, at this time, motion compensation is not performed.

The decoder 31 includes a prediction mode switching circuit 52 and a DCT in the encoder 18 of FIG.
Although the circuits corresponding to the mode switching circuit 55 are not shown, the processing corresponding to these circuits, that is, the configuration in which the signals of the lines in the odd field and the even field are separated is necessary for the original mixed configuration. The motion compensation circuit 88 executes the processing for returning.

Although the prediction distance (distance of P picture) is set to 1 in the above embodiments, it may be set to 2. For example, as shown in FIG. 22, when an even frame is set as an I picture or a P picture and the sum of absolute values of prediction errors is sequentially detected, the fourth frame of the scene 1 is detected.
The sum of the absolute values of the prediction errors of the picture of the sixth frame of the scene 2 is larger than the reference value M × MAE, and the sum of the absolute values of the other prediction errors is close to the average value MAE.

Therefore, in this case, for example, as shown in FIG.
As shown in FIG. 3, the picture of the sixth frame, which was originally supposed to be processed as a P picture, can be changed to be processed as an I picture. In this way, the picture of the sixth frame at the beginning of scene 2 can be made a high-quality image.

Further, as shown in FIG. 24, when the camera flash is present in the sixth frame, the sum of the absolute values of the prediction errors of the fourth frame and the sixth frame and the prediction of the sixth frame and the eighth frame are performed. The sum of the absolute values of the errors becomes larger than the reference value N × MAE. Therefore, even in this case, in the case of a scene change (FIG. 22), the state in which the absolute value of the prediction error is larger than the reference value occurs once, and in the case of the camera flash (FIG. 24), it occurs twice continuously. Occurs. From this, it is possible to detect a scene change and a camera flash, as in the case described above.

When the camera flash is detected, the picture type optimizing process as shown in FIGS. 25 to 28 can be executed.

In the embodiment of FIG. 25, the processing of the sixth frame having a scene change is changed from the P picture to the B picture, and the processing of the fifth frame immediately before that is changed from the B picture to the P picture. It is changed to processing.

On the contrary, in the embodiment of FIG. 26, the processing of the sixth frame, which had the camera flash, is changed from the P picture to the B picture, and immediately after that, the seventh picture is processed.
The processing of frames has been changed from B pictures to P pictures.

Further, in the embodiment shown in FIG. 27, the processing of the sixth frame, which had a camera flash, is changed from the P picture to the B picture, and the fifth frame one frame before and the one frame after that are changed. The processing of the seventh frame has been changed from the B picture to the P picture.

In this case, it is possible to change the processing of the fifth frame or the seventh frame from B picture to I picture.

The embodiment of FIG. 28 not only changes the processing of the sixth frame generated by the camera flash from the P picture to the B picture, but also changes the fifth frame one frame before and the fifth frame one frame after. 7 frame processing,
The B picture is changed to the P picture, and the processing of the fourth and eighth frames before and after one frame is changed from the P picture to the B picture.

As shown in FIG. 24C, it is possible to detect the camera flash by detecting the sum of the average luminance values of the pictures processed as I pictures or P pictures. Is similar to the case described above.

FIG. 29 shows an embodiment in which the prediction distance is 2 and a scene change occurs in a B picture. In this case, the sum of the absolute values of the prediction errors of the fifth frame and the seventh frame becomes larger than the reference value N × MAE. Therefore, from this, it is possible to detect the scene change in the same manner as in the case described above.

FIG. 30 shows an example of picture type optimization processing in this case. In this embodiment, the processing of the seventh frame, which is the second frame of scene 2, is changed from P picture to I picture.

FIG. 31 shows an embodiment in which the prediction distance is 2 and the camera flash occurs in the B picture. In this case, as shown in FIG. 7B, since the camera flash has occurred in the B picture, the sum of the absolute values of the prediction errors is not sampled, and the sum of the absolute values of the prediction errors is the reference value N × It is never larger than MAE. That is, the camera flash cannot be detected.

Similarly, as shown in FIG. 31C, the average value of the luminance obtained in the odd frames is almost the same as the average value among the plurality of frames. Therefore, even in this case, the camera flash cannot be detected.

Therefore, in this case, as shown in FIG. 32, the picture type optimizing process is not substantially performed. However, in this case, since the frame in which the camera flash is generated is originally processed as the B picture, the adverse effect of the processing does not affect other frames. In other words, FIG.
As shown in FIG. 28, when the frame generated by the camera flash is a P picture, the optimization is performed by using this as a B picture.
In the case of 1, since it is originally planned to be processed as a B picture, it can be considered that optimization has already been performed.

In the above, the prediction distance is set to 2, but
Further, the predicted distance can be set to 3. Also in this case, the scene change or the camera flash can be detected from the sum of the absolute values of the prediction errors or the average value of the brightness in the same manner as described above.

FIG. 33 shows the principle of scene change detection when the predicted distance is 3. As shown in the figure, in the case of this embodiment, the sum of the absolute values of the prediction errors between the fourth frame and the seventh frame is larger than the reference value N × MAE. Therefore, a scene change can be detected.

FIG. 34 shows a picture type optimization process when a scene change is detected when the prediction distance is 3 as shown in FIG. In this embodiment, the processing of the picture of the seventh frame, which is the second picture of scene 2, is changed from P picture to I picture.

The seventh frame is not the first frame of scene 2, but the sixth frame, which is the first frame of scene 2, is a B picture, and therefore, from the picture of the seventh frame, which is an I picture. Accurate prediction is possible by predicting. That is, when there is a scene change, if the first P picture after the scene change is changed to an I picture, the B pictures before that are backward predicted from the I picture.
Accurate processing is possible.

FIG. 35 shows the principle of camera flash detection when the predicted distance is 3. In this embodiment, the sum of the absolute values of the prediction errors of the third frame and the sixth frame and the sum of the absolute values of the prediction errors of the sixth frame and the ninth frame are larger than the reference value N × MAE. Therefore, it is possible to detect the camera flash.

As shown in FIG. 35C, the sum of the average luminance values of the sixth frame is larger than the reference value M × MY. Therefore, the camera flash is also detected from this average luminance value. be able to.

FIG. 36 shows that the predicted distance is 3
, The picture type optimization process is performed when the camera flash is detected. In this embodiment, the processing of the sixth frame in which the camera flash has occurred is changed from the P picture to the B picture, and the processing of the fifth and seventh frames before and after that is changed from the B picture to the P picture. Has been done.

FIG. 37 shows the case where the camera flash occurs in the B picture when the prediction distance is 3. In this case, the camera flash cannot be detected, as shown in FIGS. However, as in the case where the prediction distance is 2, as in the case described above, if the picture in the camera flash is a B picture, the optimization processing is substantially already performed, so special processing is performed. No need.

[0155] The above-mentioned scene change picture type optimization processing is summarized and shown in Table 1.
Further, the picture type optimization processing in the case of the camera flash is collectively shown in Table 2.

[0156]

[Table 1]

[0157]

[Table 2]

Incidentally, in FIG. 12, when both the scene change and the camera flash are detected from the sum of the absolute values of the prediction errors, the luminance DC value calculator 103 is not necessary. If only the camera flash is detected, only the motion vector detection circuit 102 and the frame memory 101 are provided, or the brightness DC value calculator 1 is used.
Only 03 should be provided.

Further, as described above, when the prediction distance in the picture determination circuit 49 is set to a predetermined value such as 1, 2 or 3, the prediction distance in the motion vector detection circuit 50 at the subsequent stage is also set to this. Must be set to the corresponding value.

Furthermore, in the embodiments shown in FIGS. 24, 31, 35 and 37, the P picture distance is set in correspondence with the picture type processing even when the luminance data is processed. It is also possible to set different distances (for example, the processing of the luminance data is performed for each frame).

FIG. 38 shows another embodiment of the encoder. In this embodiment, the motion vector detection circuit 50 and the frame memory 51 are used as the motion vector detection circuit 102 and the frame memory 101 in the picture determination circuit 49 of FIG. That is, the motion vector detection circuit 50 and the frame memory 51 are used to calculate the sum of absolute values of prediction errors, and the calculation result is output to the selection circuit 122, and the selection circuit 122 selects and determines the picture type. Has been done. The picture type selected and determined by the selection circuit 122 is supplied to and stored in the storage device 123 such as a hard disk or a solid-state memory.

In this way, the input image is once supplied to the motion vector detection circuit 50 and the frame memory 51,
After being processed and the picture type is stored in the storage device 123, the same data is input to the motion vector detection circuit 50 again from the beginning. When the data is input again, the motion vector detection circuit 50 processes the input image data according to the picture type supplied from the storage device 123.

Of course, also in this case, the input image data is supplied to the brightness DC value calculator 121, the brightness DC value is detected, and the detection result is supplied to the selection circuit 122. The camera flash can be detected from the value.

[0164]

As described above, according to the image processing method of the first aspect, the absolute value and the average value of the difference between the image data are compared with each other, and the continuity of the picture is determined according to the comparison result. Since the determination is made, it becomes possible to detect the scene change of the picture and the discontinuity of the picture represented by the camera flash, and to execute more appropriate image processing.

According to the image processing method of the second aspect, the first value and the second value are compared with each other, and the continuity of the picture is determined according to the comparison result. Therefore, it is possible to reliably detect the camera flash of the picture. Then, appropriate image processing can be executed according to the detection result.

[Brief description of drawings]

FIG. 1 is a diagram illustrating the principle of high efficiency encoding.

[Fig. 2] Fig. 2 is a diagram illustrating a type of a picture when image data is compressed.

[Fig. 3] Fig. 3 is a diagram for describing a type of a picture when image data is compressed.

FIG. 4 is a diagram illustrating a principle of encoding a moving image signal.

[Fig. 5] Fig. 5 is a block diagram illustrating a configuration example of an image signal encoding device and a decoding device.

FIG. 6 is a diagram illustrating a format conversion operation of a format conversion circuit 17 in FIG.

7 is a block diagram showing a configuration example of an encoder 18 in FIG.

8 is a diagram illustrating the operation of the prediction mode switching circuit 52 of FIG.

9 is a diagram for explaining the operation of the prediction mode switching circuit 52 in FIG.

10 is a diagram illustrating an operation of the DCT mode switching circuit 55 in FIG.

11 is a diagram illustrating an operation of the DCT mode switching circuit 55 in FIG.

12 is a block diagram showing a configuration example of a picture determination circuit 49 in FIG.

13 is a flowchart illustrating the operation of the selection circuit 104 of FIG.

FIG. 14 is a diagram illustrating the principle of detecting a scene change.

FIG. 15 is a diagram illustrating the principle of detecting a camera flash.

FIG. 16 is a diagram illustrating a picture type optimizing process at the time of a scene change.

FIG. 17 is a diagram illustrating a picture type optimizing process at the time of camera flash.

FIG. 18 is a diagram illustrating another principle of camera flash detection.

FIG. 19 is a flowchart showing another operation example of the selection circuit 104 of FIG.

FIG. 20 is a diagram illustrating a luminance detection range of a picture.

21 is a block diagram showing a configuration example of a decoder 31 of FIG.

FIG. 22 is a diagram illustrating the principle of scene change detection when the predicted distance is 2.

[Fig. 23] Fig. 23 is a diagram for describing a picture type optimization process at the time of a scene change when the prediction distance is 2.

FIG. 24 is a diagram illustrating the principle of camera flash detection when the predicted distance is 2.

[Fig. 25] Fig. 25 is a diagram for describing a picture type optimization process at the time of camera flash when the prediction distance is 2.

[Fig. 26] Fig. 26 is a diagram for describing picture type optimization processing at the time of camera flash when the prediction distance is 2.

[Fig. 27] Fig. 27 is a diagram for describing a picture type optimization process at the time of a camera flash when the prediction distance is 2.

[Fig. 28] Fig. 28 is a diagram for describing a picture type optimization process at the time of camera flash when the prediction distance is 2.

FIG. 29 is a diagram illustrating another principle of scene change detection when the predicted distance is 2.

[Fig. 30] Fig. 30 is a diagram for describing a picture type optimization process at the time of a scene change in the case where the prediction distance is 2.

FIG. 31 is a diagram illustrating another principle of camera flash detection when the predicted distance is 2.

[Fig. 32] Fig. 32 is a diagram for describing a picture type optimization process at the time of camera flash when the prediction distance is 2.

FIG. 33 is a diagram illustrating the principle of scene change detection when the predicted distance is 3.

[Fig. 34] Fig. 34 is a diagram for describing a picture type optimization process at the time of a scene change in the case where the prediction distance is 3.

FIG. 35 is a diagram illustrating the principle of camera flash detection when the predicted distance is 3.

[Fig. 36] Fig. 36 is a diagram for describing a picture type optimization process at the time of camera flash when the prediction distance is 3.

FIG. 37 is a diagram illustrating another principle of camera flash detection when the predicted distance is 3.

FIG. 38 is a block diagram illustrating another configuration example of the encoder in FIG.

[Explanation of symbols]

 1 Encoding Device 2 Decoding Device 3 Recording Medium 12, 13 A / D Converter 14 Frame Memory 15 Luminance Signal Frame Memory 16 Color Difference Signal Frame Memory 17 Format Conversion Circuit 18 Encoder 31 Decoder 32 Format Conversion Circuit 33 Frame Memory 34 Luminance Signal Frame memory 35 Color difference signal frame memory 36, 37 D / A converter 49 Picture determination circuit 50 Motion vector detection circuit 51 Frame memory 52 Prediction mode switching circuit 53 Calculator 54 Prediction determination circuit 55 DCT mode switching circuit 56 DCT circuit 57 Quantization Circuit 58 Variable length coding circuit 59 Transmission buffer 60 Inverse quantization circuit 61 IDCT circuit 62 DCT block rearrangement circuit 63 Arithmetic unit 64 Frame memory 65 Motion compensation circuit 81 Reception buffer 8 Variable-length decoding circuit 83 inverse quantization circuit 84 IDCT circuit 85 DCT block rearrangement circuit 86 calculator 87 frame memory 88 the motion compensation circuit 101 frame memory 102 motion vector detection circuit 103 the luminance DC value calculator 104 selecting circuit 105 delay circuits

Claims (10)

[Claims] 1. An image processing method for processing image data of a picture, wherein an absolute value of a difference between image data in a predetermined range of the plurality of pictures is calculated, and an average value of absolute values of the difference of the plurality of pictures is calculated. Is calculated, the absolute value of the difference between the individual pictures is compared with the average value, and the continuity of the pictures is determined according to the comparison result. 2. An image processing method for processing image data of a picture, wherein an average value of luminance of image data in a predetermined range of the picture is calculated to obtain a first value, and the first value of a plurality of the pictures is calculated. An average value of values is calculated to be a second value, the first value and the second value are compared, and the continuity of the picture is determined according to the comparison result. Image processing method. 3. The picture corresponding to continuity obtained by comparing the absolute value and the average value or continuity obtained by comparing the first value and the second value. 3. The image processing method according to claim 1 or 2, wherein a prediction mode is selected when predicting a picture from another picture. 4. The image processing method according to claim 1, wherein the absolute value of the difference between the image data is calculated between successive pictures. 5. The image processing method according to claim 1, wherein the absolute value of the difference between the image data is calculated between the pictures separated by two or more. 6. When the absolute value, which is larger than the average value, independently occurs only once, it is determined as a scene change of the picture, and when it occurs twice in succession, it is determined as a camera flash of the picture. The image processing method according to claim 4 or 5, wherein: 7. The image processing method according to claim 2, wherein when the first value becomes larger than the second value, the camera flash of the picture is determined. 8. A prediction mode for predicting the picture from another picture is selected in accordance with continuity obtained by comparing the absolute value and the average value, When the large absolute value occurs independently only once, it is determined as a scene change of the picture, and 2
When it occurs consecutively, it is determined to be the camera flash of the picture. At the time of a scene change, at least one of the pictures of the scene change and subsequent pictures of the pictures is an I picture. 2. The image processing method according to claim 1, wherein, in the case of a camera flash, the picture in the camera flash is processed as a B picture, or the picture immediately after that is processed as an I picture. 9. A prediction mode for predicting the picture from other pictures is selected according to the continuity obtained by comparing the first value and the second value, When the first value becomes larger than the second value, it is determined as the camera flash of the picture, and the picture of the camera flash is processed as a B picture, or the picture immediately after that is regarded as an I picture. The image processing method according to claim 2, wherein the image processing is performed. 10. An image processing apparatus to which the image processing method according to claim 1 is applied.