JP2006310920A - Moving picture conversion apparatus, moving picture restoration apparatus and method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an apparatus and a method for preventing deterioration of image quality in compression and decompression of moving image data.
An object movement amount in a block constituting moving image data is detected, and a spatial direction thinning process, a time direction thinning process, or a composite thinning process is executed based on the detected movement amount. In particular, in the case of image data in which the amount of movement varies greatly between frames, the maximum amount of movement included in each block of the number N of thinning-out processing unit frames is detected and the spatial direction thinning is preferentially performed. With this configuration, it is possible to minimize image quality degradation due to thinning. Furthermore, it is possible to reduce image quality deterioration due to preferential thinning in the spatial direction by pixel value correction and restore high-quality data.
[Selection] FIG.

Description

  The present invention relates to a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program. In particular, in data conversion executed as data compression processing of moving image data, high-quality data conversion is possible while suppressing image quality deterioration, and high-quality image data restoration from compressed converted data is possible. The present invention relates to a conversion device, a moving image restoration device and method, and a computer program.

  The moving image data is subjected to data conversion for reducing the amount of data, that is, compression processing, when it is stored in a storage medium such as a hard disk or DVD or distributed via a network. In particular, in recent years, the quality of moving image data has been improved, for example, HD (High Definition) data has been improved, and the amount of data has increased rapidly with the improvement of the quality of this data. Under such circumstances, many studies and studies have been made on techniques for improving the compression efficiency in the compression and decompression processing of moving image data and preventing the quality degradation of the decompressed data.

  As a moving image compression processing method, for example, thinning processing of pixels constituting an image frame constituting moving image data, that is, thinning processing in a spatial direction, thinning processing of a frame rate, that is, thinning processing in a time direction, and the like are known. Yes.

  By reducing the amount of data by such data conversion, there is an advantage that saving to a storage medium and data transfer via a network are efficiently performed. However, when the compressed data is restored and reproduced, there is a problem that the image quality deteriorates. In particular, when the original data is a high-definition image, the quality deterioration becomes more remarkable.

Various studies have been made on how to reduce such image quality deterioration. For example, Patent Document 1 discloses an image compression processing configuration in which a parameter based on image brightness information is set and the compression mode is changed according to the image brightness. Patent Document 2 discloses a compression processing configuration in which a screen is divided into a plurality of areas and the compression mode is changed for each area.
JP 2003-169284 A Japanese Patent Laid-Open No. 2002-27466

  As described above, in some conventional techniques, a configuration is disclosed in which the compression mode is changed based on various characteristics obtained from the processing target image and the data quality is improved. In the method, the compressed image data is restored and reproduced, and image quality deterioration is not sufficiently suppressed.

  In particular, moving image data is data including various types of motion data such as a portion where the subject's motion is large, a small portion, a portion where the subject is stationary, etc., and it is appropriate to correspond to these different motions without losing data quality. There is a problem that it is difficult to prevent the deterioration of quality when compression processing is performed and data restoration and reproduction are performed.

  The present invention has been made in view of such problems, and determines the optimum compression processing mode corresponding to the characteristics of each region of the image, particularly the movement of the subject, and optimizes each region according to the determined mode. An object of the present invention is to provide a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program that enable compression and restoration with extremely little quality deterioration by adopting a configuration for performing data conversion processing in various modes. .

The first aspect of the present invention is:
A moving image conversion apparatus that executes data conversion processing of moving image data,
A block division unit that executes block division processing for each frame constituting the moving image data;
A movement amount detection unit that detects a movement amount of a subject in each block divided by the block division unit, and the maximum value of the movement amount of the subject from a plurality of blocks corresponding to pixel positions in a plurality of consecutive frames that are thinning processing execution units. A movement amount detection unit for detecting
The maximum movement amount information detected by the movement amount detection unit is input, and based on the maximum movement amount information, a spatial direction thinning process, a time direction thinning process, or a space for a plurality of consecutive frames serving as a thinning process execution unit A block distribution unit that determines whether to perform the direction thinning process and the time direction thinning process, and executes a process of supplying the block processing target block data to the block processing unit that performs the thinning process;
Block data and the maximum movement amount information are input from the block distribution unit, and based on the maximum movement amount information, one of spatial direction thinning processing, time direction thinning processing, and spatial direction and time direction thinning processing is executed. A block processing unit to
The moving image conversion apparatus is characterized by comprising:

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the movement amount detection unit performs block matching with a past frame for each of a plurality of blocks corresponding to pixel positions in a plurality of consecutive frames serving as a thinning process execution unit. The present invention is characterized in that the processing is executed to execute processing for detecting the maximum value of the subject movement amount from the plurality of blocks.

  Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the movement amount detection unit uses N consecutive frames as a thinning processing execution unit, where N ≧ 2 as a processing unit, and in the N consecutive frames. The present invention is characterized in that a process for detecting the maximum value of the subject movement amount is executed from a plurality of blocks corresponding to pixel positions.

Furthermore, in one embodiment of the moving image conversion apparatus of the present invention, the block distribution unit is configured to determine a thinning processing mode based on maximum movement amount information detected by the movement amount detection unit, and the maximum movement In a setting where the amount is Vt, two predetermined thresholds Va and Vb, where Va> Vb,
Maximum travel: When Vt ≧ Va,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a spatial direction thinning process,
Maximum travel: When Vt <Vb
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a time direction thinning process,
When the maximum movement amount Vt is Vb ≦ Vt <Va,
The present invention is characterized in that execution processing for a plurality of consecutive frames serving as a thinning processing execution unit is determined as thinning processing in the spatial direction and in the time direction.

  Furthermore, in one embodiment of the moving image conversion apparatus according to the present invention, the moving image conversion apparatus further performs pixel expansion after block expansion corresponding to the spatial direction decimation for the block that has been subjected to the spatial direction decimation processing. The present invention is characterized in that a process for calculating pixel position information requiring correction processing as attribute information is executed.

Furthermore, the second aspect of the present invention provides
A moving image restoration apparatus that performs a restoration process of moving image conversion data,
A block extension unit that inputs block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executes an extension process of the block correspondence conversion data based on the conversion mode information;
A synthesis unit that synthesizes the blocks restored by the block extension process in the block extension unit and generates frame data;
The block extension is
Block restoration is performed by block expansion processing according to a thinning mode of at least one of spatial direction thinning processing, time direction thinning processing, or spatial direction and time direction thinning processing executed in the generation of the moving image conversion data. ,
The synthesis unit is
It is a configuration for generating frame data by synthesizing the blocks restored by the block extension process, and further, for the processing block on which the block extension process corresponding to the spatial direction thinning process is executed,
(1) The past frame is thinned out in the time direction, and the current frame is thinned out in the spatial direction.
(2) The current frame is thinned in the spatial direction, and the future frame is thinned in the time direction.
The moving image restoration apparatus is configured to execute a correction process of a pixel value of a block corresponding to either (1) or (2).

Furthermore, in one embodiment of the moving image restoration apparatus of the present invention, the synthesizing unit includes a correction target pixel detection unit that detects a target pixel position for performing pixel value correction processing, and detection information of the correction target pixel detection unit. And a spatial direction thinning region correction unit that performs pixel value correction based on the correction target pixel detection unit, the correction target pixel detection unit for pixels included in a processing block that has been subjected to block expansion processing corresponding to spatial direction thinning processing ,
(A) Pixels in which the current frame is thinning in the spatial direction and the past frame is thinning in the time direction;
(B) a pixel if the current frame is spatial direction thinning and the future frame is time direction thinning;
(C) Other pixels,
The present invention is characterized in that a process for generating an initial setting matrix corresponding to a pixel position where the pixel can be identified is executed.

Furthermore, in one embodiment of the moving image restoration apparatus of the present invention, the correction target pixel detection unit further estimates the movement trajectory of the subject from the thinning processing mode of the current frame and the past frame, or the current frame and the future frame, For the initialization matrix,
(A) a pixel to be replaced with a pixel value of a pixel of a current frame with a pixel value of a pixel of a past frame;
(B) a pixel to be replaced with a pixel value of a pixel of a current frame with a pixel value of a pixel of a future frame;
(C) pixels that do not require correction;
The present invention is characterized in that the processing for generating the correction processing application matrix Ct capable of discriminating the pixels is executed.

  Furthermore, in one embodiment of the moving image restoration apparatus of the present invention, the spatial direction thinning region correction unit executes spatial direction thinning in the current frame based on each element value set in the correction processing application matrix Ct. The pixel value correction of the pixels included in the block area is performed according to any one of the above-described aspects (a) to (c).

Furthermore, the third aspect of the present invention provides
A moving image conversion method for executing data conversion processing of moving image data,
A block division step for executing a block division process for each frame constituting the moving image data;
This is a movement amount detection step for detecting a subject movement amount in each block divided in the block division step, and the maximum value of the subject movement amount from a plurality of blocks corresponding to pixel positions in a plurality of continuous frames as a thinning process execution unit. A movement amount detecting step for detecting
The maximum movement amount information detected in the movement amount detection step is input, and based on the maximum movement amount information, a spatial direction thinning process, a time direction thinning process, or a space for a plurality of consecutive frames serving as a thinning process execution unit A block distribution step for determining which of the direction and time direction thinning processing is to be executed, and executing processing for supplying the block processing target block data to the block processing unit that performs the thinning processing;
A block processing step of inputting block data and the maximum movement amount information, and executing any one of spatial direction thinning processing, time direction thinning processing, and spatial direction and time direction thinning processing based on the maximum movement amount information; ,
A moving image conversion method characterized by comprising:

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the movement amount detecting step includes block matching with a past frame for each of a plurality of blocks corresponding to pixel positions in a plurality of continuous frames serving as a thinning process execution unit. It is characterized in that processing is executed to detect the maximum value of the subject movement amount from the plurality of blocks.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the movement amount detection step includes N consecutive frames as a thinning process execution unit, where N ≧ 2 as a processing unit, and in the N consecutive frames. A process for detecting the maximum value of the subject movement amount from a plurality of blocks corresponding to pixel positions is executed.

Furthermore, in one embodiment of the moving image conversion method of the present invention, the block distribution step is a step of determining a thinning processing mode based on the maximum movement amount information detected in the movement amount detection step, and the maximum movement In a setting where the amount is Vt, two predetermined thresholds Va and Vb, where Va> Vb,
Maximum travel: When Vt ≧ Va,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a spatial direction thinning process,
Maximum travel: When Vt <Vb
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a time direction thinning process,
When the maximum movement amount Vt is Vb ≦ Vt <Va,
It is characterized in that execution processing for a plurality of consecutive frames which are thinning processing execution units is determined as thinning processing in the spatial direction and in the time direction.

  Furthermore, in one embodiment of the moving image conversion method of the present invention, the moving image conversion method further includes, after the block expansion corresponding to the spatial direction decimation, the pixel value for the block that has been subjected to the spatial direction decimation processing. It has a step which performs the process which calculates the pixel position information which needs a correction process as attribute information.

Furthermore, the fourth aspect of the present invention provides
It is a moving image restoration method for executing moving image conversion data restoration processing,
A block expansion step for inputting block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executing a block correspondence conversion data expansion process based on the conversion mode information;
Combining a block restored by the block expansion process in the block expansion step to generate frame data,
The block expansion step includes
Block restoration is performed by block expansion processing according to a thinning mode of at least one of spatial direction thinning processing, time direction thinning processing, or spatial direction and time direction thinning processing executed in the generation of the moving image conversion data. ,
The synthesis step includes
It is a configuration for generating frame data by synthesizing the blocks restored by the block extension process, and further, for the processing block on which the block extension process corresponding to the spatial direction thinning process is executed,
(1) The past frame is thinned out in the time direction, and the current frame is thinned out in the spatial direction.
(2) The current frame is thinned in the spatial direction, and the future frame is thinned in the time direction.
The moving image restoration method includes a step of executing a correction process of pixel values of a block corresponding to either (1) or (2).

Furthermore, in one embodiment of the moving image restoration method of the present invention, the synthesis step includes a correction target pixel detection step for detecting a target pixel position for executing a pixel value correction process, and detection information of the correction target pixel detection step. Based on the spatial direction thinning region correction step for performing the pixel value correction, the correction target pixel detection step for the pixels included in the processing block that has been subjected to the block expansion processing corresponding to the spatial direction thinning processing ,
(A) Pixels in which the current frame is thinning in the spatial direction and the past frame is thinning in the time direction;
(B) a pixel if the current frame is spatial direction thinning and the future frame is time direction thinning;
(C) Other pixels,
A process of generating an initial setting matrix corresponding to a pixel position where the pixel can be discriminated is executed.

Furthermore, in one embodiment of the moving image restoration method of the present invention, the correction target pixel detection step further estimates the movement trajectory of the subject from the thinning processing mode of the current frame and the past frame, or the current frame and the future frame, For the initialization matrix,
(A) a pixel to be replaced with a pixel value of a pixel of a current frame with a pixel value of a pixel of a past frame;
(B) a pixel to be replaced with a pixel value of a pixel of a current frame with a pixel value of a pixel of a future frame;
(C) pixels that do not require correction;
A process of generating a correction processing application matrix Ct capable of discriminating the pixels is executed.

  Furthermore, in one embodiment of the moving image restoration method of the present invention, the spatial direction thinning region correction step is executed in the spatial direction thinning in the current frame based on each element value set in the correction processing application matrix Ct. The pixel value correction of the pixels included in the block area is performed according to any one of the above-described aspects (a) to (c).

Furthermore, the fifth aspect of the present invention provides
A computer program for executing data conversion processing of moving image data on a computer,
A block division step for executing a block division process for each frame constituting the moving image data;
This is a movement amount detection step for detecting a subject movement amount in each block divided in the block division step, and the maximum value of the subject movement amount from a plurality of blocks corresponding to pixel positions in a plurality of continuous frames as a thinning process execution unit. A movement amount detecting step for detecting
The maximum movement amount information detected in the movement amount detection step is input, and based on the maximum movement amount information, a spatial direction thinning process, a time direction thinning process, or a space for a plurality of consecutive frames serving as a thinning process execution unit A block distribution step for determining which of the direction and time direction thinning processing is to be executed, and executing processing for supplying the block processing target block data to the block processing unit that performs the thinning processing;
A block processing step of inputting block data and the maximum movement amount information, and executing any one of spatial direction thinning processing, time direction thinning processing, and spatial direction and time direction thinning processing based on the maximum movement amount information; ,
There is a computer program characterized by comprising:

Furthermore, the sixth aspect of the present invention provides
A computer program for executing a moving image conversion data restoration process on a computer,
A block expansion step for inputting block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executing a block correspondence conversion data expansion process based on the conversion mode information;
Combining a block restored by the block expansion process in the block expansion step to generate frame data,
The block expansion step includes
Block restoration is performed by block expansion processing according to a thinning mode of at least one of spatial direction thinning processing, time direction thinning processing, or spatial direction and time direction thinning processing executed in the generation of the moving image conversion data. ,
The synthesis step includes
It is a configuration for generating frame data by synthesizing the blocks restored by the block extension process, and further, for the processing block on which the block extension process corresponding to the spatial direction thinning process is executed,
(1) The past frame is thinned out in the time direction, and the current frame is thinned out in the spatial direction.
(2) The current frame is thinned in the spatial direction, and the future frame is thinned in the time direction.
The computer program has a step of executing a correction process of a pixel value of a block corresponding to either (1) or (2) above.

  The computer program of the present invention is, for example, a storage medium or communication medium provided in a computer-readable format to a general-purpose computer system capable of executing various program codes, such as a CD, FD, MO, etc. Or a computer program that can be provided by a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the present invention, the optimum compression processing mode corresponding to the characteristics of each region of the image, particularly the movement of the subject is determined, and the data conversion processing is performed in the optimal mode for each region according to the determined mode. By doing so, it becomes possible to perform compression and decompression with very little quality deterioration. In the configuration of the present invention, the maximum movement amount of the subject in the block constituting the moving image data is detected, and based on the detected maximum movement amount, the spatial direction thinning process, the time direction thinning process, or the spatial direction and time is performed. The direction thinning process is selectively executed. With this configuration, data conversion that suppresses deterioration in image quality that causes a super-resolution effect is realized.

  In particular, even in the case of image data in which the amount of movement varies greatly between frames, the moving image conversion apparatus of the present invention detects the maximum amount of movement included in each block of the thinning-out processing unit frame number N to determine the spatial direction. Since the thinning is preferentially performed, image quality deterioration due to thinning can be minimized. Furthermore, according to the moving image restoration apparatus of the present invention, it is possible to reduce image quality degradation due to preferential thinning in the spatial direction by pixel value correction and restore high-quality data.

Hereinafter, the configuration of a moving image conversion apparatus, a moving image restoration apparatus and method, and a computer program according to the present invention will be described with reference to the drawings. The description will be made according to the following items.
(1) Basic configuration of moving image conversion device using super-resolution effect (2) Configuration of moving image conversion device that executes improved thinning process (3) Moving image recovery device that executes improved restoration processing

[(1) Basic configuration of moving image conversion apparatus using super-resolution effect]
First, a basic configuration of a moving image conversion apparatus that performs moving image compression using the super-resolution effect as a base of the present invention will be described. This basic configuration is described in detail in Japanese Patent Application No. 2003-412501 filed earlier by the present applicant. The image is divided into small areas, and the number of pixels is determined according to the moving speed of each area. This is a configuration in which compression of the data amount is realized by adaptively performing decimation and frame rate decimation.

  FIG. 1 shows a configuration example of the moving image conversion apparatus 10 described in Japanese Patent Application No. 2003-412501. The moving image conversion apparatus 10 can reduce the amount of data so that an observer does not perceive image quality deterioration due to the reduction of the data amount by performing a moving image conversion process using the super-resolution effect. It is a thing.

  Note that the super-resolution effect is a visual effect realized based on a visual characteristic that an observer perceives an image obtained by adding a plurality of images within a certain period of time. When human vision perceives a stimulus, it has a function of storing the stimulus for a certain period of time after the presentation of the stimulus ends (referred to as a sensory memory function). There are many reports that this time is 10 ms to 200 ms. This function is also called iconic memory or visual persistence, and is described in, for example, “Visual Information Handbook, Japanese Visual Society, pp.229-230”. The super-resolution effect is considered to be caused by a complicated relationship between the temporal integration function and the sensory memory function in the human visual function.

  The moving image conversion apparatus 10 shown in FIG. 1 performs compression for reducing data so that the observer does not perceive image quality degradation by performing moving image conversion processing using the super-resolution effect caused by the temporal integration function. It is set as the structure to perform. A configuration of the moving image conversion apparatus 10 in FIG. 1 will be described.

  The block dividing unit 11 divides each frame of the input moving image into blocks as divided areas of predetermined pixels and supplies them to the movement amount detecting unit 12. The movement amount detection unit 12 detects the movement amount for each block supplied from the block division unit 11, and transmits the block and the movement amount to the block processing unit 13. The block processing unit 13 performs a moving image conversion process corresponding to the movement amount, that is, a compression process, on the block supplied from the movement amount detection unit 12 to reduce the data amount. The block processing unit 13 supplies the output unit 14 with data about the block with the reduced data amount obtained as a result of the processing. The output unit 14 collectively outputs, as stream data, the data regarding the blocks with the reduced data amount supplied from the block processing unit 13. The restoration unit 15 receives the compressed data output from the output unit 14, executes restoration processing by decompression processing, and generates reproducible moving image data.

  Next, with reference to FIG. 2, details of each unit excluding the restoration unit 15 will be described. The frame of the moving image supplied to the moving image conversion apparatus 10 is input to the image storage unit 21 of the block dividing unit 11. The image storage unit 21 stores the input frame. Each time the number of accumulated frames reaches N (N is a positive integer), the image accumulating unit 21 supplies the N frames to the block dividing unit 22, and M in the N frames The second stored frame (hereinafter referred to as the Mth frame) is supplied to the movement amount detection unit 12 (movement amount detection unit 31). For example, N = 4.

  The block dividing unit 22 divides each of the N frames (consecutive N frames) supplied from the image storage unit 21 into blocks of a certain size (for example, 8 × 8, 16 × 16) and moves them. It outputs to the quantity detection part 12 (block distribution part 32). The block dividing unit 22 also converts each block of the Pth frame stored in the image storage unit 21 (hereinafter referred to as the Pth frame) among the N frames to the movement amount detection unit 12 (movement amount detection). Part 31). The Pth frame is a different frame from the Mth frame.

  Next, the movement amount detection unit 12 will be described. The movement amount detection unit 31 of the movement amount detection unit 12 uses the motion vector of each block of the Pth frame supplied from the block division unit 22 of the block division unit 11 as the Mth frame supplied from the image storage unit 21. For example, the block matching process between the frames is detected and detected, and the detected motion vector is supplied to the block distributor 32. The motion vector represents the amount of movement between the frames in the horizontal direction (X-axis direction) and in the vertical direction (Y-axis direction). In addition, the movement amount detection unit 31 may be configured to enlarge the image in order to improve the detection accuracy of the movement amount and detect the movement amount using the enlarged image.

  The block distribution unit 32 of the movement amount detection unit 12 is supplied with blocks (N blocks in total at the same position in each of N frames) from the block division unit 22 in units of N, and the movement amount detection unit From 31, the amount of movement of the block of the P-th frame among the N blocks is supplied. The block distribution unit 32 supplies the supplied N blocks and the movement amount to any of the block processing units 51 to 53 that perform processing corresponding to the movement amount of the block processing unit 13.

  Specifically, the block distribution unit 32 is supplied from the movement amount detection unit 31 and has a movement amount in the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) between one frame of 2 pixels (pixels) or more. In some cases, the N blocks supplied from the block dividing unit 22 and the movement amount supplied from the movement amount detection unit 31 are output to the block processing unit 51. If the amount of movement in the horizontal direction and the vertical direction between one frame is less than 2 pixels and more than 1 pixel, the block distribution unit 32 outputs N blocks and the amount of movement to the block processing unit 53. . When the movement amount is other than that, the block distribution unit 32 supplies the N blocks and the movement amount to the block processing unit 52.

  That is, the block distribution unit 32 determines an optimal frame rate and spatial resolution based on the movement amount supplied from the movement amount detection unit 21, and performs block processing for performing processing for converting image data according to the frame rate and spatial resolution. The block images are distributed to the units 51 to 53.

  This condition for determining the distribution destination is merely an example, and the distribution destination may be determined under other conditions.

  Next, details of the block processing unit 13 will be described. The block processing unit 13 is composed of the three block processing units 51 to 53 as described above. The block processing unit 51 supplies a total of N blocks (horizontal or vertical) at the same position in each of consecutive N (for example, N = 4) frames supplied from the block distribution unit 32 of the movement amount detection unit 12. Similarly, a process of thinning out the number of pixels according to the amount of movement supplied from the block distributor 32 (spatial direction thinning process) is performed on N blocks in the case where the amount of movement in the direction is 2 pixels or more. .

  Specifically, when the amount of movement in the horizontal direction between one frame is 2 pixels (pixels) or more, the block processing unit 51, when the block is configured by 8 × 8 pixels, as shown in FIG. The pixels in the block are divided into sets of 1 × 4 pixel units. Further, as shown in FIG. 4, the block processing unit 51 sets the pixel values p1 to p4 of each set of 1 × 4 pixels to one pixel value (p1 in this example) of the pixel values. Thinning (thinning of the number of pixels between four pixels) (thinning of the thinning amount 4) is performed.

  When the vertical movement amount between one frame is 2 pixels or more, the block processing unit 51 divides the pixels in the block into a set of 4 × 1 pixel units as shown in FIG. As shown, the pixel values p1 to p4 of each set are thinned out to be one pixel value (p1 in this example).

  When the vertical and horizontal movement amounts between one frame are both 2 pixels or more, the block processing unit 51 divides the pixels in the block into sets of 2 × 2 pixel units as shown in FIG. As shown in FIG. 8, the pixel number p1 to p4 of each set is thinned out to be one pixel value (p1 in this example).

  Since the block processing unit 51 performs such spatial thinning processing on each of the four supplied blocks, the data amount of each block is reduced to one pixel data amount for every four adjacent pixels. The data amount of the entire four blocks is reduced to ¼. The block processing unit 51 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  In the example of FIG. 4, the leftmost pixel value p1 in the 1 × 4 pixel in the example of FIG. 4, the uppermost pixel value p1 in the 4 × 1 pixel in the example of FIG. In the example, the pixel value p1 in the upper left corner of 2 × 2 pixels is set, but the pixel value may be set to any pixel value from p1 to p4. The pixel value may be set to a calculation using the pixel values p1 to p4, for example, an average value.

  Next, processing executed by the block processing unit 52 shown in FIG. 2 will be described. The block processing unit 52 shown in FIG. 2 has a total of N blocks (moving in the horizontal and vertical directions) at the same position in each of the consecutive N frames supplied from the block distribution unit 32 of the movement amount detection unit 12. A process of thinning out the number of frames (a temporal direction thinning process) is performed on N blocks in which both amounts are less than one pixel.

  Specifically, as shown in FIG. 9, the block processing unit 52 converts four blocks Bi at the same position in four consecutive frames F1 to F4 into one block (in this example, Then, the number of frames to be converted into the block Bi) of the frame F1 is thinned out (thinning of the number of frames between four frames). The block processing unit 52 supplies the output unit 14 with data (one block) for four blocks whose data amount has been reduced to ¼ by such time direction thinning processing.

  The block processing unit 53 is supplied from the block distribution unit 32 of the movement amount detection unit 12 and has a total of N blocks (the movement amount in the horizontal direction and the vertical direction is 1 in each of the consecutive N frames). The number of pixels is thinned out (spatial direction thinning process) and the number of frames is thinned out (time direction thinning process) for each of N blocks in the case of more than pixels and less than 2 pixels.

  Unlike the thinning-out process in the block processing unit 51, the block processing unit 53 converts each set of pixel values p1 to p4 into any two pixel values (in this example, as shown in FIGS. 10 and 11). In this case, the number of pixels to be reduced to p1, p3) is thinned out (thinning amount 2 is thinned out).

  That is, the block processing unit 51 thins out three types of pixels of 1 × 4, 4 × 1, or 2 × 2, while the block processing unit 53 performs two types of pixels of 1 × 2 or 2 × 1. Decimation of numbers is performed.

  In the frame number thinning process, the block processing unit 53 is different from the thinning process in the block processing unit 52, as shown in FIG. 12, at the same position in each of the four consecutive frames F1 to F4. Decreasing the number of frames to make a total of four blocks Bi any one of them (two blocks of frames F1 and F3 in the example in the figure) (decimating the number of frames between two frames) Do.

  Since the block processing unit 53 performs the spatial direction thinning process for reducing the data amount by 1/2 and the time direction thinning process for reducing the data amount by 1/2 to the four supplied blocks. As a result, the data amount of the four blocks is reduced to (1/2) × (1/2) = 1/4. The block processing unit 53 supplies data about four blocks whose data amount is reduced to ¼ to the output unit 14.

  The output unit 14 configures and outputs, as stream data, data regarding the blocks obtained from the block processing units 51 to 53 and information indicating what processing has been performed on each block.

  Next, the configuration and processing of the restoration unit 15 illustrated in FIG. 1 will be described with reference to FIG. As shown in FIG. 13, the restoration unit 15 includes a block distribution unit 61, block expansion units 62 to 64, and a synthesis unit 65.

  The stream data obtained from the output unit 14 is input to the block distribution unit 61 of the restoration unit 15. This stream data includes converted (compressed) data generated by the processing up to immediately before and attribute data necessary for restoration. The attribute data includes information related to each block and information related to the specific contents of processing performed on each block. The specific contents of the processing applied to each block include the spatial direction thinning process, the time direction thinning process, the process including both the spatial direction thinning process and the time direction thinning process, and the spatial thinning process. Information about whether the thinning process executed is in the horizontal direction or in the vertical direction.

  Based on the processing contents of each block included in the input attribute information, the block distribution unit 61 sends the conversion data to be restored and the processing applied to each block to any of the block expansion units 62 to 64. Attribute information indicating specific contents is sent.

  Specifically, when the block is data processed by the block processing unit 51 in the moving image conversion apparatus 10 shown in FIGS. 1 and 2, the distribution unit 61 uses the block data thinned out to the block expansion unit 62. And send information such as the direction of spatial thinning.

  If the data is processed by the block processing unit 52 of the moving image conversion apparatus 10, similar information is sent to the block extension unit 63. If processed by the block processing unit 53, the same information is sent to the block extension unit 64. The information related to the processing applied to each block and the information related to each block may be any information as long as it is sufficient information for restoring a moving image.

  The block extension unit 62 performs an extension process on the data that has been subjected to the thinning process in the spatial direction in the block processing unit 51 of the moving image conversion apparatus 10. Based on the information regarding the direction of spatial thinning input from the distribution unit 61, the data expansion processing in the mode shown in FIGS. 14 to 16 is executed to reconfigure the block. Further, the reconstructed block is output to the synthesis unit 65.

  The processing shown in FIGS. 14 to 16 will be described. FIG. 14 shows processing performed by the block extension unit 62 of the restoration unit 15 when the block processing unit 51 of the moving image conversion apparatus 10 has been subjected to horizontal thinning processing (see FIGS. 3 and 4). For example, when the size of the original block is 8 × 8, data corresponding to ¼ of the 16 pixels is sent to the block extension unit 62. With respect to the data of each pixel, as shown in FIG. 14A, a process of expanding one pixel data into four pixels in the horizontal direction is performed to expand one pixel to 1 × 4 pixels. Note that the processing example shown in FIG. 14A is an example in which expansion is performed by arranging the pixel values of the input pixels as they are as pixel values for four pixels, but the values of other input pixels are included. You may arrange | position the result by the calculation based on a some pixel value.

  Subsequently, when the thinning is performed in the horizontal direction, the block extension unit 62 performs the process shown in FIG. That is, the process of expanding one pixel to 1 × 4 pixels is executed for 16 pixels input to the block expansion unit 62, and a set of 16 1 × 4 pixels executed for each of the 16 pixels is shown in FIG. ). By this processing, the block extension unit 62 restores an 8 × 8 block from the input 16 pixels, and outputs the result to the synthesis unit 65.

  FIG. 15 shows processing performed by the block extension unit 62 of the restoration unit 15 when the block processing unit 51 of the moving image conversion apparatus 10 has been subjected to vertical thinning processing (see FIGS. 5 and 6). For example, when the size of the original block is 8 × 8, data corresponding to ¼ of the 16 pixels is sent to the block extension unit 62. With respect to the data of each pixel, 1 pixel is expanded to 4 × 1 pixels by performing the processing of FIG. That is, as shown in FIG. 15A, one pixel is expanded to 1 × 4 pixels by performing processing for expanding one pixel data into four pixels in the vertical direction.

  Subsequently, when the thinning is performed in the vertical direction, the block extension unit 62 performs the process shown in FIG. That is, the process of expanding one pixel to 1 × 4 pixels is performed for 16 pixels input to the block expansion unit 62, and a set of 16 1 × 4 pixels executed for each of the 16 pixels is shown in FIG. ). By this processing, the block extension unit 62 restores an 8 × 8 block from the input 16 pixels, and outputs the result to the synthesis unit 65.

  FIG. 16 shows processing performed by the block extension unit 62 of the restoration unit 15 when the block processing unit 51 of the moving image conversion apparatus 10 has been subjected to thinning processing in the horizontal and vertical directions (see FIGS. 7 and 8). For example, when the size of the original block is 8 × 8, data corresponding to ¼ of the 16 pixels is sent to the block extension unit 62. With respect to the data of each pixel, 1 pixel is expanded to 2 × 2 pixels by performing the processing of FIG. That is, as shown in FIG. 16A, one pixel is expanded to 2 × 2 pixels by performing processing for expanding one pixel data into 2 × 2 pixels in the horizontal and vertical directions.

  Subsequently, when thinning is performed in the horizontal and vertical directions, the block extension unit 62 performs the processing shown in FIG. That is, the process of expanding one pixel to 2 × 2 pixels is executed for 16 pixels input to the block expansion unit 62, and the 16 sets of 2 × 2 pixels executed for each of the 16 pixels are shown in FIG. ). By this processing, the block extension unit 62 restores an 8 × 8 block from the input 16 pixels, and outputs the result to the synthesis unit 65.

  Next, processing of the block extension unit 63 will be described. The block expansion unit 63 performs an expansion process on the data that has been subjected to the thinning process in the time direction in the block processing unit 52 of the moving image conversion apparatus 10.

  As shown in FIG. 17, the block extension unit 63 generates block data of a plurality of frames continuous in the time axis direction based on the block data of one frame. Specifically, the block extension unit 63 has a counter with an initial value of 0, and this counter is incremented by 1 every time one frame is restored, and the value is 4 (= N (however, when the thinning amount m = 4) )) Is reset to 0 when it is reached.

  When data to be expanded is input from the data distribution unit 61 to the block expansion unit 63, the block expansion unit 63 stores the block in the memory of the data expansion unit 63, and the counter upper limit set according to the thinning amount Until reaching the value, block data corresponding to a plurality of frame data is generated as a copy of the block data stored in the memory, and output to the combining unit 65. In the example illustrated in FIG. 17, the thinning-out amount m = 4, and block data corresponding to four frames is generated from one block data and output to the combining unit 65.

  Next, processing of the block extension unit 64 will be described. The block extension unit 64 performs an extension process on the data that has been subjected to the thinning process in the spatial direction and the time direction in the block processing unit 53 of the moving image conversion apparatus 10. Either the spatial direction or temporal direction expansion processing may be executed first.

  The time direction expansion process executed in the block expansion unit 64 performs the process shown in FIG. Specifically, the block extension unit 64 has a counter with an initial value of 0, and this counter is incremented by 1 every time restoration of one frame is completed, and the value is 2 (= N / 2 (however, the amount of thinning in the time direction) When m = 2)), it is reset to 0.

  When the data to be expanded is input from the data distribution unit 61 to the block expansion unit 64, the block expansion unit 64 stores the block in the memory of the data expansion unit 64, and the counter upper limit set according to the thinning amount Until the value is reached, block data corresponding to a plurality of frame data is generated as a copy of the block data stored in the memory and output to the combining unit 65. In the example shown in FIG. 18, the time direction thinning-out amount m = 2, and block data corresponding to two frames is generated from one block data.

  Next, the spatial direction expansion processing executed in the block expansion unit 64 will be described. Based on information such as the direction of spatial thinning input from the distribution unit 61, the block expansion unit 64 performs data expansion processing in the manner shown in FIGS. 19 and 20, and reconfigures the block.

  The processing of FIGS. 19 and 20 will be described. FIG. 19 shows processing by the block extension unit 64 of the restoration unit 15 when the horizontal thinning processing is performed in the block processing unit 53 of the moving image conversion apparatus 10. For example, when the size of the original block is 8 × 8, data corresponding to ½ of 32 pixels is sent to the block extension unit 64. Corresponding to the data of each of the two pixels, the process of FIG. 19A is performed to expand the two pixels to 1 × 4 pixels. That is, as shown in FIG. 19 (a), two pixels are expanded to 1 × 4 pixels by performing processing for expanding one pixel data into two pixels in the horizontal direction.

  Subsequently, when the thinning is performed in the horizontal direction, the block extension unit 64 performs the process shown in FIG. That is, the process of extending 2 pixels to 1 × 4 pixels is executed for 32 pixels input to the block extension unit 64, and a set of 16 1 × 4 pixels executed for each of the 32 pixels is shown in FIG. ). By this processing, the block extension unit 64 restores an 8 × 8 block from the input 32 pixels, and outputs the result to the synthesis unit 65.

  FIG. 20 shows a process performed by the block extension unit 64 of the restoration unit 15 when the block processing unit 53 of the moving image conversion apparatus 10 performs the thinning process in the vertical direction. For example, when the size of the original block is 8 × 8, data corresponding to ½ of 32 pixels is sent to the block extension unit 64. Corresponding to the data of each of the two pixels, the process of FIG. 20A is performed to expand the two pixels to 4 × 1 pixels.

  Subsequently, when the thinning is performed in the vertical direction, the block extension unit 64 performs the process shown in FIG. That is, the process of expanding 2 pixels to 4 × 1 pixels is executed for 32 pixels input to the block expanding unit 64, and a set of 16 4 × 1 pixels executed for each 32 pixels is shown in FIG. ). By this processing, the block extension unit 64 restores an 8 × 8 block from the input 32 pixels, and outputs the result to the synthesis unit 65.

  When the block input from the block expansion units 62 to 64 reaches an amount that can represent one entire frame, the combining unit 65 restores one frame by appropriately arranging the blocks, and outputs the restored one frame. .

  As described above, the moving image conversion apparatus 10 shown in FIG. 1 performs processing for converting an input moving image into a moving image (compressed data) with a reduced data amount. This is a device that prevents the observer from perceiving image quality degradation due to a reduction in the amount of data by performing moving image conversion processing using the super-resolution effect realized based on the visual characteristics of the image.

  Specifically, the block distribution unit 32 determines an optimal frame rate and spatial resolution based on the movement amount supplied from the movement amount detection unit 21, and performs image data conversion according to the optimal frame rate and spatial resolution. It supplies to the block processing parts 51-53 to perform, and it is set as the structure which performs the data conversion process of a different aspect in each block processing part 51-53, and makes an observer perceive image quality degradation by this structure. It realizes a moving image conversion process without any problem. Note that the super-resolution effect is a visual effect realized based on the visual characteristic that an observer perceives a sum of a plurality of images within a certain time as described above. It is considered that the temporal integration function and the sensory memory function in the function are caused in a complicated manner, and the moving image conversion apparatus 10 shown in FIG. 1 has a super-resolution effect caused by the temporal integration function. The moving image conversion process is used.

  The principles and explanations regarding human visual characteristics and super-resolution effect are described in detail in Japanese Patent Application No. 2003-412501. The conditions for generating the super-resolution effect described in Japanese Patent Application No. 2003-412501 will be described below.

In order for the super-resolution effect to occur when the number of pixels of the thinning amount m (pixel) is thinned, it is necessary to cancel all the primary to m−1 order folding components due to the thinning. k (= 1, 2,..., m−1) The condition that the next folding component is canceled is to satisfy the following expressions (Expression 1) and (Expression 2).

In the above equation, φ _t is a sampling position shift amount in thinning out the number of pixels, time t (= 0, 1T, 2T,...), Signal moving speed v, time interval (reciprocal of frame rate) T , Is a value defined by the following formula (formula 3).

  In the above equation, the case where the sampling position is shifted to the right is positive (this condition is different from the setting disclosed in Japanese Patent Application No. 2003-412501).

  If the above equations (Equation 1) and (Equation 2) are satisfied under the conditions of the thinning amount m and the small region movement amount v, the super-resolution effect occurs, and the deterioration of the image quality is hardly perceived by the observer.

[(2) Configuration of Moving Image Conversion Device that Performs Improved Thinning Process]
Similar to the moving image conversion apparatus described in Japanese Patent Application No. 2003-412501, the moving image conversion apparatus of the present invention described above is connected to the block distribution unit 32 of the movement amount detection unit 12 shown in FIG. Blocks (a total of N blocks at the same position in each of N frames) are supplied in units, and the movement amount of the P-th frame in the N blocks is transferred from the movement amount detection unit 31. Is supplied. The block distribution unit 32 supplies the supplied N blocks and the movement amount to any of the block processing units 51 to 53 that perform processing corresponding to the movement amount of the block processing unit 13.

  In the moving image conversion apparatus described in Japanese Patent Application No. 2003-412501, the block distribution unit 32 refers to only the movement amount of a block of a certain frame P (0 ≦ P ≦ N−1) among N blocks. For example, in the case where the data conversion (compression) processing is performed with 4 frames as a processing unit, for example, the first of the 4 frames. Set the fixed movement amount calculation frame, such as setting only the frame as the movement amount calculation frame, or setting only the second frame as the movement amount calculation frame, and calculate the movement amount. Over the unit (N), a fixed thinning mode based on the movement amount determined by the fixed movement amount calculation frame (space direction thinning, time direction) Thinning, or was running the process in the time direction and the spatial decimation).

  However, especially at the boundary between the moving area and the still area in the image, the amount of movement of a certain block may change greatly between N frames set as a processing unit, and the same processing is applied over a plurality of frames. As a result, image quality may sometimes deteriorate.

  FIG. 21 shows how the movement amount fluctuates at the boundary between the moving region and the stationary region when the number of thinning-out processing unit frames is N = 4. However, the same discussion is possible even if N is a value other than that).

FIG. 21 shows a change in image data between four frames (from frame 0 to frame 3) at a certain position in space. Each of the four diagrams included in FIG. 21 represents the state of the frame indicated by the number. A small square in the figure is one block (for example, 4 pixels × 4 pixels), a white part is a background (static area), and a hatched part is a moving subject (moving area). It is assumed that the hatched portion moves to the left at 3 pixels / frame. In the above situation, the calculation result of the movement amount in each of the frames 0 to 3 of the block 71 shown in the drawing is approximately as follows.
Frame 0) Movement amount 0
Frame 1) Movement amount 0
Frame 2) Move left 3 pixels / frame Frame 3) Move left 3 pixels / frame

  In frame 0, there is no moving subject in block 71, and the amount of movement is calculated as zero. In frame 1, there are a few moving subjects, but most of the area is the background, and the amount of movement obtained from the block 71 of frame 1 is greater than 0 but nearly zero. In frames 2 and 3, all of the blocks 71 are occupied by moving subjects moving leftward at 3 pixels / frame, and the amount of movement of the blocks 71 in these frames is “moving leftward at 3 pixels / frame”. Is calculated as

  In the moving image conversion apparatus described in Japanese Patent Application No. 2003-412501, the block distribution unit 32 refers only to the movement amount of a block in a frame P (0 ≦ P ≦ N−1) among N blocks. . Therefore, in the example shown in FIG. 21, in the setting where the number of thinning-out processing unit frames N = 4 and the movement amount detection frame P = 0, it is determined that the movement amount = 0, and the block 71 over these four frames The block 71 is determined to be a still area, and the block 71 is subjected to thinning processing in the time direction in the block processing unit 53 over four frames.

  If the movement amount detection frame P is set to 3, the movement is determined to be 3 pixels / frame in the left direction, and the block processing unit 51 performs spatial thinning on the block 71 over 4 frames. Will be.

  Thus, when the movement amount detection frame P = 0, frames 2 and 3 are thinned out in the time direction even though they are regions having a high movement amount. On the other hand, when P = 3, frame 0 and frame 1 are subjected to spatial thinning even though they are stop areas.

  As described above, in the configuration in which the movement amount detection frame is fixed, if the corresponding block in the thinning-out processing unit frame number N includes a block having a large movement amount, the movement amount detection frame and the movement amount frame are greatly different. In this block, an inappropriate process is performed, and in such a block subjected to such an inappropriate process, the image quality is deteriorated. The present invention provides a configuration that improves this problem. Hereinafter, the structure which improved this problem is demonstrated.

  Data reduction (compression) processing that suppresses image quality degradation that occurs at the boundary between regions moving at different speeds is realized by solving the above-described problems of the moving image conversion apparatus 10 and executing improved thinning processing A moving image conversion apparatus 100 according to the present invention will be described.

  The configuration of the moving image conversion apparatus 100 of the present invention that executes the improved thinning process will be described with reference to FIG. Similar to the moving image conversion apparatus 10 described above with reference to FIG. 1, the moving image conversion apparatus 100 shown in FIG. 22 uses the super-resolution effect due to human visual characteristics, thereby reducing the image quality by reducing the data amount. The amount of data can be reduced so that the observer does not perceive the deterioration, and further, the data can be reduced while suppressing the image quality deterioration that occurs at the boundary between regions moving at different speeds. It is.

  The basic configuration of the moving image conversion apparatus 100 shown in FIG. 22 is substantially the same as the configuration described above with reference to FIG. The block dividing unit 110 divides each frame of the input moving image into blocks and supplies the blocks to the movement amount detecting unit 120.

  The movement amount detection unit 120 detects the movement amount for each block supplied from the block division unit 110 and transmits the block and the movement amount to the block processing unit 130. However, unlike the configuration described with reference to FIG. 1, the movement amount detection unit 120 has a fixed frame P (0 ≦ P ≦ N−1) of N blocks over the thinning-out processing unit frame N. Instead of detecting the movement amount of the block, the block having the maximum movement amount is selected from the N blocks over the thinning-out processing unit frame N, and the movement amount is calculated. Details of these processes will be described later.

  The block processing unit 130 performs a moving image conversion process corresponding to the movement amount on the block supplied from the movement amount detection unit 120 to reduce the data amount. The block processing unit 130 supplies the output unit 140 with data regarding the block whose data amount is reduced, which is obtained as a result of the processing. The output unit 140 collectively outputs the data regarding the blocks with the reduced data amount supplied from the block processing unit 130, for example, as stream data.

  Next, the details of each unit will be described with reference to FIG. First, the block dividing unit 110 will be described. The frame of the moving image supplied to the moving image conversion apparatus 100 is input to the image storage unit 111 of the block dividing unit 110. The image storage unit 111 stores the input frame. The image accumulating unit 111 moves the N frames to the block dividing unit 112 and moves each time the accumulated number of frames reaches N (N is a positive integer) which is a predetermined number of thinning-out processing units. This is supplied to the quantity detection unit 121.

  The block division unit 112 divides each of N frames (consecutive N frames) supplied from the image storage unit 111 into blocks of a certain size (for example, 4 × 4, 8 × 8), and the movement amount detection unit 120. To the block distribution unit 122 and the movement amount detection unit 121.

  Next, the configuration and processing of the movement amount detection unit 120 will be described. The maximum movement amount detection unit 121 of the movement amount detection unit 120 receives the motion vector of each block for N frames supplied from the block division unit 112 of the block division unit 110 as M ( (M is an integer) Referring to the previous frame, for example, the block matching process between the frames is executed and detected, and the largest of the detected N motion vectors is detected by the block distributor 122 and the block processor 131. And 132. The motion vector represents the amount of movement between the frames in the horizontal direction (X-axis direction) and the vertical direction (Y-axis direction). The means for detecting the movement amount may be other than block matching.

  The apparatus described above with reference to FIGS. 1 and 2 selects a fixed Pth frame from N frames corresponding to the number N of thinning-out processing unit frames, and selects the selected Pth frame. The motion vector of each block included in the frame was obtained. In this configuration, for each block of the thinning-out processing unit frame number N, a block having the maximum movement amount is selected, and this thinning-out processing mode is determined for each block included in the thinning-out processing unit frame number N. The amount of movement to do.

  Next, the details of the processing of the maximum movement amount detection unit 121 in the movement amount detection unit 120 in the case where the number of thinning-out processing unit frames N = 4 will be described using FIG. As shown in FIG. 24, the maximum movement amount detection unit 121 of the movement amount detection unit 120 includes four block matching units 210 to 213, a memory 220, and a selector 230. The memory 220 stores frames necessary for block matching among frames input from the image storage unit 111 in the past. For example, when the currently processed frame is frame 0, frame 1, frame 2, and frame 3, four frames of frame 0-M, frame 1-M, frame 2-M, and frame 3-M are stored in the memory 220. Is preserved.

  The block matching units 210 to 213 detect the motion vector of the block of each frame input from the block dividing unit 112 by block matching using the reference frame input from the memory 220. The block matching unit 210 detects the motion vector of the block of frame 0 by block matching using the frame 0-M input from the memory 220 as a reference frame. The block matching unit 211 detects the motion vector of the block of the frame 1 by block matching using the frame 1-M input from the memory 220 as a reference frame. The block matching unit 212 detects the motion vector of the block of the frame 2 by block matching using the frame 2-M input from the memory 220 as a reference frame. The block matching unit 213 detects the motion vector of the block of the frame 3 by block matching using the frame 3-M input from the memory 220 as a reference frame. The output of each block matching unit 210 to 213 is sent to the selector 230.

  The selector 230 outputs the maximum of the four motion vectors input from the block matching units 210 to 213 to the block distribution unit 122. In order to select the largest one from the four motion vectors, for example, the maximum value of the absolute values of a total of eight values consisting of the X and Y components of the four motion vectors is calculated, and the motion vector having that value is calculated. There are methods such as selecting or selecting a motion vector that maximizes the sum of squares of the X and Y components of each motion vector.

The block distribution unit 122 supplies the supplied N blocks and the maximum movement amount to any of the block processing units 131 to 133 that perform processing corresponding to the movement amount of the block processing unit 130. Based on the maximum amount of movement in the horizontal direction (X-axis direction) or the vertical direction (Y-axis direction) between one frame supplied from the maximum movement amount detection unit 121, the block distribution unit 122 performs a thinning process unit frame, The thinning processing mode for N frames is determined. That is,
Perform spatial direction decimation or
Execute time direction decimation or
Perform temporal and spatial thinning, or
It is determined which of these aspects of the thinning process is executed, and according to the determination, the N blocks and the maximum movement amount of the block processing units 131 to 133 that perform processing corresponding to the movement amount of the block processing unit 130 are determined. Supply to any of them.

  Specifically, the block distribution unit 122 supplies the maximum movement amount in the horizontal direction (X-axis direction) or vertical direction (Y-axis direction) between one frame supplied from the maximum movement amount detection unit 121 to 2 pixels (pixels). In the case described above, the N blocks supplied from the block dividing unit 112 and the maximum movement amount supplied from the maximum movement amount detection unit 121 are output to the block processing unit 131. When the amount of movement in the horizontal direction and the vertical direction between one frame is less than 2 pixels and more than 1 pixel, the block distribution unit 122 outputs N blocks and the movement amount to the block processing unit 133. . When the movement amount is other than that, the block distribution unit 122 supplies the N blocks and the movement amount to the block processing unit 132.

  That is, the block distribution unit 122 distributes the block image to the block processing units 131 to 133 that perform optimal thinning processing based on the movement amount supplied from the maximum movement amount detection unit 121. This condition for determining the distribution destination is merely an example, and the distribution destination may be determined under other conditions.

The generalization of the thinning mode determination process performed by the block distribution unit 122 is as follows. In the setting where the maximum movement amount detected by the maximum movement amount detection unit 121 is Vt, two predetermined thresholds Va and Vb, where Va> Vb,
Maximum travel: When Vt ≧ Va,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a spatial direction thinning process,
Maximum travel: When Vt <Vb
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a time direction thinning process,
When the maximum movement amount Vt is Vb ≦ Vt <Va,
Execution processes for a plurality of consecutive frames serving as a thinning process execution unit are determined as thinning processes in the spatial direction and the time direction.

  Next, details of the block processing unit 130 will be described. As shown in FIG. 23, the block processing unit 130 includes three block processing units 131 to 133. The block processing unit 131 supplies a total of N blocks (horizontal or vertical) at the same position in each of consecutive N (for example, N = 4) frames supplied from the block distribution unit 122 of the movement amount detection unit 120. Similarly, a process of thinning out the number of pixels in accordance with the amount of movement supplied from the block distribution unit 122 (spatial direction thinning process) is performed on N blocks when the amount of movement in the direction is 2 pixels or more) .

  Specifically, when the amount of movement in the horizontal direction between one frame is larger than the amount of movement in the vertical direction and is 2 pixels (pixels) or more, the block processing unit 131 includes a block composed of, for example, 8 × 8 pixels. As shown in FIG. 3, the pixels in the block are divided into sets of 1 × 4 pixel units. Further, as shown in FIG. 4, the block processing unit 131 sets the pixel values p1 to p4 of each set of 1 × 4 pixels to one of the pixel values (in this example, p1) of the number of pixels set. Thinning (thinning of the number of pixels between four pixels) (thinning of the thinning amount 4) is performed.

  When the amount of movement in the vertical direction between one frame is larger than the amount of movement in the horizontal direction and is 2 pixels or more, the block processing unit 131 converts the pixels in the block into units of 4 × 1 pixels as shown in FIG. As shown in FIG. 6, the pixel values p1 to p4 of each set are thinned out to one pixel value (p1 in this example).

  Since the block processing unit 131 performs such spatial thinning processing on each of the four supplied blocks, the data amount of each block is reduced to one pixel data amount for every four adjacent pixels. The data amount of the entire four blocks is reduced to ¼. The block processing unit 131 supplies data about four blocks whose data amount has been reduced to ¼ to the output unit 140.

  Since the block distribution unit 122, the block processing unit 132, the block processing unit 133, and the output unit 140 are the same as those in the moving image conversion apparatus 10, description of the operation is omitted. The restoration unit 150 will be described later.

  As described above, in the moving image conversion apparatus 100 of the present invention, the movement amount is calculated for all the N frames of blocks to be processed simultaneously, and the maximum movement amount among the obtained movement amounts is thinned out. The amount of movement is used for determination. That is, in the moving image conversion apparatus 100 of the present invention, for each block of the thinning-out processing unit frame number N, a block having the maximum movement amount is selected, and this maximum movement amount is included in each thinning-out processing unit frame number N. It is set as the movement amount for determining the thinning processing mode for the block. Hereinafter, the effect of using such a process determination method will be described.

  The reason for determining the processing content to be performed on the block using the maximum amount of movement obtained as described above will be described.

First, since a frame included in the thinning-out processing unit frame number N is at the boundary between the moving region and the stationary region, for a block whose movement amount varies greatly between N frames,
(1) Applying time decimation to all N frames;
(2) It is confirmed with reference to FIGS. 25 to 28 in which situation when the spatial thinning is applied to all N frames, in which the image quality deterioration is more remarkable.

  FIG. 25 shows a state between four frames (from frame 0 to frame 3) at a certain position in the space as in FIG. A small square in the figure is one block of 4 × 4 pixels (pixels), a point display area portion is a background (static area), and a line display area portion is a moving subject (moving area). The moving area moves to the right at 4 pixels / frame.

In the above situation, paying attention to the blocks 80 to 83 shown in FIG. 25, in the case of the above (1), that is,
(1) When applying time reduction to all N frames,
FIG. 26 and FIG.

  FIG. 26 is obtained as a result of applying thinning-out processing and restoration (extension) processing when the movement amount detection frame P is set to P = 3 by the conventional moving image conversion apparatus 10 to the four frames in FIG. Represents an image. FIG. 26 shows a thinning process mode of only the blocks 80 to 83 of interest.

  When P = 3, as understood with reference to the frame 3 shown in FIG. 25, since the block 83 area in the frame 3 is the background (still area), a small movement amount of almost 0 is calculated. The blocks 80 to 83 in the 25 4 frames are subjected to a time direction thinning process. The time direction thinning process is a process of applying block pixels in one frame from four frames to all four frames when N = 4. In the time direction thinning process, the number of frame pixels to be left in the thinning process is set in advance, and the frame pixels are copied to the corresponding blocks of all frames according to the setting. FIG. 26 shows a case where a frame to be left in time thinning, that is, a case where the pixel value to be a copy source is frame 0, and FIG.

  In FIG. 26, the pixel value of the block 80 in the lower right part of the frame 0 in the moving area indicated by the line area part is copied over the blocks 81 to 83 of each frame of the frames 1 to 3. During the display time of four frames 0 to 3, a part of the moving subject corresponding to the block 80 existing in the frame 0 continues to be displayed at the same position until reaching the frames 0 to 3.

  FIG. 27 shows a case where the frame to be left in the time thinning is frame 3. In this case, the pixel value of the block 83 in the lower right part of the frame 3 is copied over the blocks 80 to 82 of each frame of the frames 0 to 2, and the display time of the four frames of the frames 0 to 3 is A part of the background corresponding to the block 83 existing in the frame 3 continues to be displayed at the same position until reaching the frames 0 to 3. In this case, a part of the moving area is replaced with the background in the first two frames (frames 0 and 1).

  In other words, for a region where the amount of movement changes between 4 frames, (1) when time thinning is applied to all N frames, an object that does not exist in the original moving image is displayed for several frames (FIG. 26). The shape of the moving subject changes (FIG. 27), which leads to a significant deterioration in image quality.

In contrast,
(2) When spatial thinning is applied to all N frames,
The state of image degradation will be described with reference to FIG. Spatial thinning is a process of copying and generating pixel values of pixels in a block of the same frame from representative pixels in the same block of the same frame.

  As shown in FIG. 28, in the blocks 80 to 83 of the frames 0 to 3, the space thinning process is executed as the process in the same block. As for the blocks 80 and 81 of the frame 0 and the frame 1, there is no problem since the thinning process is performed on the moving subject area and the thinning process is performed in accordance with the movement amount of the block, but the blocks 82 and 83 of the frame 2 and the frame 3 are not problematic. However, even though the amount of movement is a background region of almost zero, spatial thinning is performed. This is not a thinning process suitable for the amount of block movement. However, even if the spatial thinning process is applied, only the spatial resolution of the image is lowered, and the original configuration of the image (positional relationship of the subject, hue when observed globally, etc.) does not change significantly.

That is,
(2) When spatial thinning is applied to all N frames,
Was described with reference to FIGS.
(1) When applying time reduction to all N frames,
Such a remarkable deterioration that the configuration of the image is changed does not occur.

  As described above, blocks where the amount of movement greatly changes between N frames, that is, the boundary between the moving region and the stationary region, that is, blocks of the amount of movement to which time decimation should be applied between N frames and spatial decimation. If a block with a moving amount of blocks that should be applied is uniformly processed by either time thinning or space thinning, significant deterioration may occur if time thinning is performed. It can be seen that when only the spatial direction thinning is applied, the deterioration of the image quality can be suppressed.

  Since the moving image conversion apparatus 100 according to the configuration of the present invention is configured to determine the type of the thinning process based on the maximum value of the moving amount of each frame, the spatial thinning process is performed in a block in which different moving amounts are mixed as described above. As a result, image quality degradation can be minimized.

  The block distribution unit 122 of the movement amount detection unit 120 shown in FIG. 23 receives blocks from the block division unit 112 in units of thinning processing unit frames: N (a total of N blocks at the same position in each of N frames). Further, the maximum movement amount detector 121 supplies the block movement amount of the frame having the maximum movement amount among the N blocks. As described with reference to FIG. 24, the maximum movement amount detection unit 121 selects the block movement amount of the frame having the maximum movement amount among the N blocks, and uses the selector 230 to determine the maximum movement amount. The movement amount information (motion vector information) is output to the block distribution unit 122.

  Based on the maximum movement amount information input from the maximum movement amount detection unit 121, the block distribution unit 122 supplies the block processing units 131 to 133 with N blocks corresponding to the number of thinning-out processing unit frames: N frames. The units 131 to 133 execute space direction thinning, time direction thinning, space direction, and time direction thinning for the input blocks, respectively.

Specifically, according to the maximum movement amount information input from the maximum movement amount detection unit 121, for example, the process is executed in any one of the following modes (1) to (3).
(1) When the maximum movement amount detected by the maximum movement amount detection unit 121 is 2 pixels or more as the movement amount in the horizontal direction or the vertical direction, the block processing unit 131 corresponds to the number of thinning-out processing unit frames: N frames. N blocks to be supplied are supplied, and spatial direction thinning processing according to the maximum movement amount supplied from the block distribution unit 122 is performed.
(2) When the maximum movement amount detected by the maximum movement amount detection unit 121 is less than one pixel in both the horizontal direction and the vertical direction, the block processing unit 132 notifies the block processing unit 132 of N blocks corresponding to the number of thinning process unit frames: N frames. To perform time direction thinning processing.
(3) When the maximum movement amount detected by the maximum movement amount detection unit 121 is 1 pixel or more in both the horizontal direction and the vertical direction and less than 2 pixels, the block processing unit 133 notifies the block processing unit 133: N frames N blocks corresponding to are supplied to perform spatial direction thinning and time direction thinning processing.

  By adopting such a processing configuration, when at least one block having a maximum movement amount of 2 pixels or more is included in the N blocks of the processing target blocks included in the N frame: the number of thinning-out processing unit frames In the block processing unit 131, spatial direction thinning processing is performed, and there is no block having a maximum movement amount of 2 pixels or more in the N blocks, and one block having a maximum movement amount of 1 pixel or more. However, if included, the block processing unit 133 performs thinning in the spatial direction and thinning in the time direction. Only when there is no block having a maximum movement amount of 1 pixel or more in N blocks, the block processing unit 132 performs the time direction thinning process.

  As described above, in the moving image conversion apparatus 100 according to the configuration of the present invention, the type of the thinning process is determined based on the maximum value of the moving amount of each frame. In many blocks where quantities are mixed, spatial thinning processing is performed in many cases, and image quality degradation can be minimized.

[(3) Moving Image Restoration Apparatus that Performs Improved Restoration Process]
Next, a detailed configuration and processing of the restoration unit 150 illustrated in FIG. 22 will be described. The restoration unit 150 includes a block distribution unit 610, block expansion units 620 to 640, and a synthesis unit 650, as shown in FIG. Since the block distribution unit 610 and the block expansion units 620 to 640 are configured to execute the same processing as the restoration unit 15 described above with reference to FIG. 13 (the restoration unit of Japanese Patent Application No. 2003-412501), A detailed description is omitted, and only the synthesis unit 650 will be described.

  It is assumed that the thinning process is performed in units of N frames, and the block size is W horizontal pixels and H vertical pixels. Further, the current processing target frame of the synthesis unit 650 is assumed to be frame n to frame n + N−1 (n = kN (k ≧ 0)).

  A detailed configuration of the combining unit 650 is shown in FIG. FIG. 30 includes a composition unit 651, image data memories 652 and 653, a thinning information memory 654, a thinning information interpolation unit 655, and a spatial thinning region correction unit 656. The combining unit 651 of the combining unit 650 performs the same operation as the combining unit 65 of FIG. That is, the output is for N frames of an image having the same resolution as the original moving image obtained by extending the thinned data.

  The image data memory 652 holds and outputs frames n−1 to n + 2N−1 obtained from the combining unit 651. In other words, the spatial thinning region correction unit 656 can refer to at least the current frame to be processed and the frames before and after it at the time of processing from the frame n to the frame n + N−1.

  The decimation information memory 654 holds the type of decimation processing applied to a total of 3N frames from frame n−N to frame n + 2N−1, and outputs this. In other words, the correction target pixel detection unit 655 can refer to at least the processing content applied to the current processing target frame in addition to the processing content applied to the frames before and after the frame n to frame n + N−1.

  The correction target pixel detection unit 655 determines whether correction processing is necessary for each pixel in the spatial thinning region from frame n + 1 to frame n + N−1, and if correction is necessary, the past frame to the future frame. Determine if the data should be used. Further, the result is output to the spatial thinning region correction unit 656. Details of the correction target pixel detection unit 655 will be described later.

  The spatial thinning region correction unit 656 corrects each pixel of N frames from frame n to frame n + N−1 with reference to the correction target pixel information detected by the correction target pixel detection unit 656. For example, when the result of the correction target pixel detection unit 656 for a certain pixel indicates that correction processing is necessary and a past frame is used, the spatial thinning region correction unit 656 transmits the data of the pixel to the frame n. Replace with the pixel value at the same position of -1.

  FIG. 31 shows an example of pixel value replacement processing executed by the spatial thinning region correction unit 656. In the figure, frames n−1 and 90 and frames n + t and 92 are shown. 0 ≦ t ≦ N−1, and frame n + t indicates any one of N frames from frame n to frame n + N−1. Frames n−1 and 90 are frames that temporally precede frames n + t and 92. The frame includes a plurality of pixels (pixels), and one pixel to be processed is shown in the figure. That is, the pixel 91 in the frame n−1, 90 and the pixel 93 in the frame n + t, 92.

  FIG. 31A shows a frame before correction, and FIG. 31B shows a frame after correction. When the result of the correction target pixel detection unit 656 with respect to the pixel in the drawing of the frame n + t indicates that correction processing is necessary and the past frame is used, as shown in FIG. 31B, the frame n− A pixel value correction process is performed in which the pixel value of the pixel 91 at 1, 90 is set as the value of the pixel 93 of the frame n + t, 92 corresponding to the same position as the pixel 91.

  When the result of the correction target pixel detection unit 656 for a certain pixel indicates that correction processing is necessary and a future frame is used, as shown in FIG. 32, the spatial thinning region correction unit 656 Then, a pixel value correction process for setting the pixel value of the pixel 97 in the frame n + N, 96 as the value of the pixel 95 in the frame n + t, 94 corresponding to the same position as the pixel 97 is executed.

  Further, if the result of the correction target pixel detection unit 656 for a certain pixel indicates that correction processing is not necessary, nothing is performed. Similarly, the correction process is not performed in the case of a time thinning region or a time space thinning region.

The correction target pixel detection unit 655 holds three types of thinning processing content information, and detects correction target pixels based on these pieces of information.
The correction target pixel detection unit 655 holds the following three types of thinning process content information.
* Contents of processing applied to past N frames from frame n-N to frame n-1.
* Contents of processing applied to the current N frames from frame n to frame n + N−1,
* Contents of processing applied to future N frames from frame n + N to frame n + 2N-1.
It is.
The correction target pixel detection unit 655 uses these pieces of information to detect an area that is not originally a moving area but is subjected to spatial thinning processing. Before describing the details of the operation of the correction target pixel detection unit 655, the principle of the correction target pixel detection unit 655 and the meaning of processing will be described using a specific example.

  FIG. 33 shows the type of decimation performed in a part of a row of each frame from frame n-4 to n + 3 with N = 4. The vertical axis is the time axis, and time elapses downward. The horizontal axis indicates the pixel position (horizontal direction). Four frames from n−4 to n−1 are one thinning process unit frame number: N = 4 frames, and n to n + 3 are four subsequent thinning process unit frames.

  In the four frames from n-4 to n-1, in the row corresponding to each frame, the data area 701 painted with diagonal lines is a portion subjected to thinning in the time direction, and the other data area 702 in each row is It is a portion that has been subjected to horizontal spatial thinning and includes a moving area. Also, in the next four frames from n to n + 3 that are continuous in time, the hatched data area 711 is a part subjected to thinning in the time direction, and the other data areas 712 in each row are in the horizontal direction. This is a portion subjected to thinning in the spatial direction and includes a moving area.

  From the transition between frames of the data areas 702 and 712 that have been thinned in the spatial direction, it can be predicted that the area has moved in the left direction. That is, the data area 712 of the subsequent frames n to n + 3 is shifted to the left from the data area 702 of the preceding frames n−4 to n−1 shown in the drawing.

  Further, it is considered that the left end portion of the data area 702 which is a moving area in the frame n−1 is generally present in a data portion 721 indicated by a solid line arrow in the drawing. This is because the maximum value of the movement amount of each block between N frames is used as a reference when determining the processing to be performed on a certain block by the moving image conversion apparatus 100 of the present invention. That is, if it is assumed that the left end of the moving area is on the left side of the data portion 721 in FIG. 33 in the frame n−1, that portion is also subjected to spatial thinning. If it is assumed that the left end of the moving region is on the right side of the data portion 721, the data portion 721 in the figure is not a spatial thinning region.

  The left end portion of the moving region in the frame n + 3 can be estimated in the same manner, and it is determined that the left end portion of the moving region moves according to the locus indicated by the dotted arrow 722 from the frame n−1 to the frame n + 3. be able to.

  From the transition prediction between frames in the moving area, a part of the data areas 723 in the frames n to n + 2 surrounded by a thick line in the drawing is included in the data area 712 that has been subjected to spatial thinning processing as a moving moving area. However, it can be assumed that the data area 723 is not originally a moving area but a stationary area such as a background. However, as a result of executing the processing mode determination process based on the maximum movement amount described above, the data region 723 is thinned in the spatial direction.

  This data area 723 originally belongs to a static area, and it is appropriate to perform correction processing in order to improve image quality. The still region has little motion, and for example, interpolation processing using pixel values in temporally adjacent frames is possible. That is, using the pixel values in the data area 724 in the frame n−1 shown in the drawing, the pixel values in the data area 723 in the frames n to n + 2 are corrected, that is, interpolation processing is performed.

  FIG. 34 shows a specific example of the interpolation process. The pixel value of the constituent pixel of the data portion 731 of the data area 724 of the frame n−1 shown in the figure is set as the pixel value of the corresponding pixel of the frame n + 2, and the constituent pixel of the data portion 732 of the data area 724 of the frame n−1 is set. The pixel value is set as the pixel value of the corresponding pixel in frame n + 1, and the pixel value of the constituent pixel in the data portion 733 of the data area 724 in frame n−1 is set as the pixel value of the corresponding pixel in frame n.

  Thus, if the pixels in these regions are replaced with the values of the pixels at the same position in the past, an improvement in image quality can be expected. The range in which the pixel value is changed can be easily calculated from the estimated moving amount L of the moving area shown in FIG. First, let x be the X coordinate of the data position 725 that is estimated to be the left end of the moving region in the frame n + 3. At this time, the pixel corresponding to 3 × L / 4 pixels to the right in frame n, 2 × L / 4 pixels in frame n + 1, and 1 × L / 4 pixels in frame n + 2 are subject to change processing. Become. In frame n + 3, there is no pixel to be changed.

  Therefore, as shown in FIG. 34, in frame n, pixels are 3 × L / 4 pixels right from x, 2 × L / 4 pixels in frame n + 1, and 1 × L / 4 pixels in frame n + 2. Is used as a correction target pixel, and correction processing for copying the pixel value of the corresponding pixel in frame n−1 is performed.

  FIG. 35 is a diagram for explaining a processing example in the case of performing pixel value correction of the pixel of the current frame using the pixel of the future frame. N = 4 represents the type of decimation performed in a part of a certain row of each frame from frame n to n + 7. The vertical axis is the time axis, and time elapses downward. The horizontal axis indicates the pixel position (horizontal direction). Four frames from n to n + 3 are one thinning processing unit frame number: N = 4 frames, and n + 4 to n + 7 are four subsequent thinning processing unit frames.

  In the four frames from n to n + 3 in the row corresponding to each frame, the hatched data region 751 is a portion subjected to thinning in the time direction, and the other data region 752 in each row is a horizontal space. It is a portion that has been subjected to direction thinning and includes a moving area. Also, in the next four frames from n + 4 to n + 7 that are continuous in time, the hatched data area 761 is a portion subjected to thinning in the time direction, and the other data areas 762 in each row are in the horizontal direction. This is a portion subjected to thinning in the spatial direction and includes a moving area.

  From the transition between frames of the data areas 752 and 762 subjected to the thinning in the spatial direction, it can be predicted that the area has moved in the left direction. That is, the data area 762 of the subsequent frames n + 4 to n + 7 is shifted to the left from the data area 752 of the preceding frames n to n + 3 shown in the figure.

  Further, it is considered that the right end portion of the data area 752 which is a moving area in the frame n + 4 is generally present in a data portion 771 indicated by a solid line arrow in the drawing. This is because the maximum value of the movement amount of each block between N frames is used as a reference when determining the processing to be performed on a certain block by the moving image conversion apparatus 100 of the present invention. That is, assuming that the right end of the moving area is on the right side of the data portion 771 in FIG. 35 in the frame n + 4, that portion is also subjected to spatial thinning. If it is assumed that the right end of the moving region is on the left side of the data portion 771, the data portion 771 in the figure does not become a spatial thinning region.

  The right end portion of the moving area in the frame n can be similarly estimated, and it can be determined that the right end portion of the moving area moves according to the locus indicated by the dotted arrow 772 from the frame n to the frame n + 7. it can.

  From the transition prediction between frames of the moving area, a part of the data area 773 of frames n + 1 to n + 3 surrounded by a thick line in the drawing is included in the data area 751 subjected to spatial thinning processing as a moving area. However, it can be assumed that the data area 773 is not a moving area originally but a static area such as a background. However, as a result of executing the processing mode determination process based on the maximum movement amount described above, the data region 773 is thinned in the spatial direction.

  This data area 773 originally belongs to a still area, and it is appropriate to perform correction processing in order to improve the image quality. The still region has little motion, and for example, interpolation processing using pixel values in temporally adjacent frames is possible. That is, using the pixel values of the data area 774 of the frame n + 4 shown in the drawing, the pixel values of the data area 773 of the frames n + 1 to n + 3 are corrected, that is, interpolation processing is performed.

  FIG. 36 shows a specific example of the interpolation process. The pixel value of the constituent pixel of the data area 781 of the data area 774 of the frame n + 4 shown in the figure is set as the pixel value of the corresponding pixel of the frame n + 1, and the pixel value of the constituent pixel of the data section 782 of the data area 774 of the frame n + 4 is set to the frame. The pixel value of the corresponding pixel of n + 2 is set, and the pixel value of the constituent pixel of the data portion 783 of the data area 774 of frame n + 4 is set as the pixel value of the corresponding pixel of frame n + 3.

  Thus, if the pixels in these areas are replaced with the values of the pixels at the same position in the future, an improvement in image quality can be expected. The range in which the pixel value is changed can be easily calculated from the estimated moving amount L of the moving region shown in FIG. First, let x be the X coordinate of the data position 771 estimated to be the right end of the moving region in the frame n + 4. At this time, in frame n + 3, 3 × L / 4 pixels from the right side of the L / 4 pixel, starting from x, in frame n + 2, 2 × L / 4 pixel from the right side of 2 × L / 4 pixel, and in the frame n + 1, 3 × L A pixel corresponding to 1 × L / 4 pixels from the right side of the / 4 pixel is the target of the change process. In frame n, there is no pixel to be changed.

  Therefore, as shown in FIG. 36, the frame n + 3 corresponds to 3 × L / 4 pixels, the frame n + 2 corresponds to 2 × L / 4 pixels, and the frame n + 1 corresponds to 1 × L / 4 pixels. Correction processing for copying the pixel value of the corresponding pixel of n + 4 is performed.

  In FIGS. 33 to 36, the example of the correction process at the time of the movement in the horizontal direction has been described. However, the same correction process can be performed even in the area where the spatial direction thinning is performed in the vertical direction.

  Further, in the above-described processing example, the number of thinning-out processing unit frames has been described as N = 4. However, details of the operation of the correction target pixel detection unit 655 extended for the general N case are shown in the flowcharts of FIGS. I will explain.

  The flowchart shown in FIG. 37 is a description of the overall processing of the correction target pixel detection unit 655 performed from frame n to frame n + N−1. First, the output and input of the correction target pixel detection unit 655 will be described.

Output of Correction Target Pixel Detection Unit 655 The final output of the correction target pixel detection unit 655 in step S106 in the flow of FIG. 37 is an N matrix Ct (0 ≦ t ≦ N−1) of U rows and V columns. is there. The matrix Ct is a correction processing application matrix Ct.
U is the number of pixels in the vertical direction of the moving image, and V is the number of pixels in the horizontal direction of the moving image.
If the (i, j) element of the matrix Ct is 0, nothing is performed on the pixel (i, j) of the frame n + t corresponding to the matrix.
If the (i, j) element of the matrix Ct is -1, it means that the pixel (i, j) of the frame n + t corresponding to the matrix is replaced with the value of the pixel (i, j) of the past frame n-1. To do.
If the (i, j) element of the matrix Ct is +1, it means that the pixel (i, j) of the frame n + t corresponding to the matrix is replaced with the value of the pixel (i, j) of the future frame n + N.

  30 receives the correction processing application matrix Ct generated by the correction target pixel detection unit 655, and based on each element value set in the correction processing application matrix Ct, the spatial thinning region correction unit 656 shown in FIG. The pixel value correction of the pixels included in the block area where the spatial direction thinning is performed is executed according to any one of the above-described aspects.

Input of Correction Target Pixel Detection Unit 655 The input to the correction target pixel detection unit 655 in step S101 in the flow of FIG. 37 is the input of the above-described output, that is, data necessary to obtain the correction processing application matrix Ct. This is the type of thinning processing performed on a total of 3N frames from frame n−N to frame n + 2N−1. That is,
* Spatial thinning processing,
* Thinning process in the time direction,
* Processing type information indicating the processing mode of either spatial direction or temporal fragrance thinning processing.

For 3N frames n−N to n + 2N−1,
(1) Past N frames of frames n−N to n−1,
(2) Current N frames of frames n to n + N-1 (3) Future N frames of frames n + N to n + 2N-1
(1) The processing performed from frame n-N to frame n-1 (past) is represented by P, and from P (i, j) to the pixel (i, j) of frame n-1 to frame n-1. Represents processing performed.
(2) Processing applied from frame n to frame n + N−1 (current) is represented by D, and processing applied to pixel (i, j) from frame n to frame n + N−1 by D (i, j) Represents.
(3) Processing applied from frame n + N to frame n + 2N−1 (future) is represented by F, and processing applied to pixel (i, j) from frame n + N to frame n + 2N−1 by F (i, j). Represents.

The process of each step of the flowchart shown in FIG. 37 will be described. First, in step S101, the above inputs (D, P, F) corresponding to the horizontal direction spatial thinning processing region are input.
That is,
Processing for current frames n to n + N−1: D
Processing for past frames n-N to n-1: P
Processing for future frames n + N to n + 2N−1: F
Enter.

  In step S102, the variable t indicating the frame is varied from 0 to N−1 (S102, S104). By this processing, the processing of the past, present, and future frames is verified for each frame, and the pixel (pixel) of the frame to be corrected, that is, the correction target pixel is detected (S103). When the processing for all N frames is completed (S105), the processing of the correction target pixel detection unit 655 ends.

As a result, in step S106, N correction processing application matrices Ct (0 ≦ t ≦ N−1) of U rows and V columns are output as final outputs. U is the number of pixels in the vertical direction of the moving image, V is the number of pixels in the horizontal direction of the moving image,
a. (I, j) element of matrix Ct = 0: correction of pixel (i, j) of frame n + t is unnecessary.
b. (I, j) element of matrix Ct = −1: Replace pixel (i, j) of frame n + t with pixel value of pixel (i, j) of past frame n−1.
c. (I, j) element of matrix Ct = + 1: Replace pixel (i, j) of frame n + t with pixel value of pixel (i, j) of future frame n + N.
A matrix indicating any of the above processing modes is output.

  Details of S103 will be described with reference to the flowchart shown in FIG. The flowchart shown in FIG. 38 is a flowchart showing a procedure of correction target pixel detection processing for a single frame.

  The contents P, D of the thinning process executed in the past N frames (n−N to n−1), the current N frames (n to n + N−1), and the future N frames (n + N to n + 2N−1) F and a variable (t) for calculating the currently processed frame (n + t) are input (S201), and a correction processing application matrix Ct as a correction target pixel detection result for the frame n + t is output (S205).

  In the process shown in FIG. 38, detection of the correction target area of the horizontal spatial thinning area and the correction target area of the vertical spatial thinning area is performed independently.

In S202, the correction target pixel is detected only for the region to which the horizontal spatial thinning process is applied, and the detection result is output as the U × V matrix X. X is a matrix composed of -1, 0, 1 as in the correction processing application matrix Ct. That is,
a. (I, j) element of matrix X = 0: pixel (i, j) correction of frame n + t is unnecessary.
b. (I, j) element of matrix X = −1: Replace pixel (i, j) of frame n + t with pixel value of pixel (i, j) of past frame n−1.
c. (I, j) element of matrix X = + 1: Replace pixel (i, j) of frame n + t with pixel value of pixel (i, j) of future frame n + N.
It is a matrix which shows each correction | amendment aspect.

In S203, the correction target pixel is detected only for the region to which the vertical spatial thinning process is applied, and the detection result is output as the U × V matrix Y. Y is a matrix composed of −1, 0, 1 similarly to the correction processing application matrix Ct. That is,
a. (I, j) element of matrix Y = 0: Pixel (i, j) of frame n + t need not be corrected.
b. (I, j) element of matrix Y = −1: Replace pixel (i, j) of frame n + t with pixel value of pixel (i, j) of past frame n−1.
c. (I, j) element of matrix Y = + 1: Replace pixel (i, j) of frame n + t with pixel value of pixel (i, j) of future frame n + N.
It is a matrix which shows each correction | amendment aspect.

The result of adding the matrix X and the matrix Y in S204 is defined as a matrix Ct. That is,
Matrix Ct = X + Y
It is.
Note that an element of the matrix Y corresponding to an element that is [−1] or [+1] in the matrix X is always [0], and a matrix that corresponds to an element that is [−1] or [+1] in the matrix Y. Since the element of X is always [0], the correction processing application matrix Ct is a matrix composed only of [−1], [0], and [+1].

  Next, with reference to the flow shown in FIG. 39, the correction target pixel is detected only in the region of S202 in FIG. 38, that is, the region to which the horizontal spatial thinning process is applied, and the detection result is obtained. Details of the process of outputting as a U × V matrix X composed of −1, 0, 1 will be described.

Step S301
Past N frames (n−N to n−1),
Current N frames (n to n + N-1),
Future N frames (n + N to n + 2N-1),
The contents P, D, F of the thinning process executed in these 3N frames, and
The variable (t) for calculating the currently processed frame (n + t) is input.

  Steps S302 and S303 are matrix X initial setting processing. In this process, each element X (i, j) of the matrix X is set to any one of [−1], [0], and [+1]. The position of an element for which a value other than 0 is set in the matrix X is a candidate for a correction target pixel.

  In step S302, the pixel (i, j) of the current N frame (n to n + N-1) is the spatial thinning in the horizontal direction, and the corresponding pixel (i, j) in the past N frame (n-N to n-1). The element X (i, j) of the matrix X corresponding to the pixel (i, j) where j) is a time thinning is set to [−1]. The other elements X (i, j) of the matrix X are set to [0].

  Subsequently, in step S303, the pixel (i, j) of the current N frame (n to n + N-1) is a spatial thinning in the horizontal direction, and the corresponding pixel (i in the future N frame (n + N to n + 2N-1)). , J) is set to [+1] for the element X (i, j) of the matrix X with respect to all the pixels (i, j) for which time decimation is performed.

With these processes, regarding the type of thinning process for a three-dimensional block including a two-dimensional frame including a pixel (i, j) and a past, present, and future time axis direction,
If the current is spatial thinning and the past is time thinning [-1],
[+1] if the current is spatial decimation and the future is time decimation,
[0] in other cases,
Is set.

The current N frames (n to n + N−1) are horizontal spatial thinning by the processing in steps S302 and S303, and the processing before and after that, that is, in the past or future time, is time thinning. A matrix X from which a region can be extracted is generated.
In particular,
(1) Past N frames = time direction thinning and current N frames = space direction thinning (2) Current N frames = space direction thinning and future N frames = time direction thinning An area corresponding to one of these two patterns An extractable matrix X is generated.

  Next, on the basis of the initial setting matrix X set in this way, processing for changing pixels that actually need correction to the selectable matrix X is executed in steps S304 to S307. Starting from the top row of the initial setting matrix X (S304), correction target pixels are detected in each row of X (S305). When S305 is finished for all rows, the flowchart of FIG. 39 is finished (S307).

  A specific processing example of the processing in the flow shown in FIG. 39 will be described with reference to FIG. FIG. 40 shows a thinning-out processing mode of the past frame (n−4 to n−1) and the current frame (n to n + 3) when N = 4. The temporal direction thinned data 801 and the spatial direction thinned data 802 exist adjacent to each other in the past frame (n−4 to n−1), and the temporal direction thinned data 811 and the spatial direction thinned out also in the current frame (n to n + 3). Data 812 exists adjacently.

In steps S302 and S303 in the flow shown in FIG. 39, for example, a distinguishable initial setting matrix X of the data area 821 shown in FIG. 40 is generated. This region X is the above-mentioned region, that is,
(1) Past N frames = thinning in time direction and current N frames = thinning in space direction The element X (i, j) corresponding to this region and corresponding to the pixel (i, j) included in this region in the matrix X The value is set as [−1].

  However, it is the data area 823 shown in FIG. 40 that actually executes the correction process. The element X (i, j) of the matrix X corresponding to only the pixel (i, j) included in the data area 823 is set as [−1], and the other pixel (i, j not included in the data area 823) It is necessary to reset the element X (i, j) of the matrix X corresponding to) as [0]. This process is executed in steps S304 to S307. Specifically, in step S305, final determination processing of the value of the element X (i, j) of the matrix X is executed.

  Note that the processing shown in FIG. 39 is for only the spatial thinning area in the horizontal direction, but the processing for the spatial thinning area in the vertical direction, that is, the processing in step S203 in FIG. Therefore, the details of S203 will be omitted.

  A detailed sequence of the horizontal line scan process of step S305, that is, the final determination process of the value of the element X (i, j) of the matrix X will be described with reference to the flowchart of FIG.

  In the process shown in the flowchart of FIG. 41, the thinning process content D for the current N frame (n to n + N−1) is referred to determine whether or not correction processing is necessary for each pixel in the jth row of the frame n + t. The matrix X having the element values {-1, 0, +1} indicating the necessity and mode of correction is output.

The input is
If the current is spatial thinning and the past is time thinning [-1],
[+1] if the current is spatial decimation and the future is time decimation,
[0] in other cases,
Is an initial setting matrix X.

  The initial setting matrix X is composed of elements (i, j), and the scan line is executed as a process in a range of i = 0 to U with j being fixed. Steps S402 and S403 are processes for setting i to i = 0. That is, the processing is started from the leftmost side of j row, i = 0 to U, that is, elements X (0, j) ˜ of the initial setting matrix X corresponding to the pixels (0, j) ˜ (U, j) ˜ X (U, j) is set.

  In step S403, it is determined whether or not i <U. If i <U is not satisfied, i.e., i = U, the process is terminated and the process proceeds to the next scan line. If i <U, the process proceeds to step S405. Step S405 is processing for determining whether or not the element X (i, j) = 0 of the initial setting matrix X.

If X (i, j) = 0,
(1) Past N frames = time direction thinning and current N frames = space direction thinning (2) Current N frames = space direction thinning and future N frames = time direction thinning Pixels that do not belong to any of these two patterns Yes, the pixel is clearly excluded from the correction target, and the process returns to step S403 to update the value of i and move the processing target to the next pixel.

In the determination process in step S405,
When it is determined that X (i, j) = 0 is not satisfied, X (i, j) = − 1or + 1,
(1) Past N frames = time direction thinning and current N frames = space direction thinning (2) Current N frames = space direction thinning and future N frames = time direction thinning.

In this case, the process proceeds to step S406.
(1) Past N frames = thinning in time direction and current N frames = thinning in space direction
(2) A value corresponding to the data length (number of pixels) of current N frame = sampling in the spatial direction and future N frame = thinning in the time direction: L is set to 0.

Next, in step S407,
It is determined whether or not the element X (i, j) = 1 of the matrix X.
The element X (i, j) = 1 of the matrix X means that
It means that the present is space thinning and the future is time thinning.
As mentioned above, the initialization matrix X is
[+1] if the current is spatial decimation and the future is time decimation
If the current is spatial thinning and the past is time thinning [-1],
[0] in other cases,
Value is set.

  A specific image data processing configuration in the case of Yes in step S407 and in the case of No will be described with reference to FIGS.

If YES in step S407, that is, as described above,
When the element X (i, j) = 1 of the matrix X, it means that the current is space thinning and the future is time thinning. Examples of this data are shown in FIGS. FIGS. 42 and 43 show the thinning-out processing mode of the current frame (n to n + 3) and the future frame (n + 4 to n + 7) when N = 4. That is, it shows changes with time in one scan line in the current frame (n to n + 3) and the future frame (n + 4 to n + 7) when N = 4.

  The shaded area is the time direction thinning process data area, and the white area is the spatial direction thinning process data area. If Yes in step S407, this is an area where the current is space thinning and the future is time thinning, and corresponds to the section L shown in FIGS.

  The case of No in step S407 means that the element X (i, j) of the matrix X is −1, and that the present is a spatial thinning and the past is a time thinning. Examples of this data are shown in FIGS. 44 and 45 show a thinning-out processing mode of the past frame (n−4 to n−1) and the current frame (n to n + 3) when N = 4. That is, it shows a change with time in one scan line in the current frame (n to n + 3) and the past frame (n−4 to n−1) when N = 4.

  The shaded area is the time direction thinning process data area, and the white area is the spatial direction thinning process data area. In the case of No in step S407, this is an area where the current is spatial thinning and the past is time thinning, and corresponds to the area of the section L shown in FIGS.

When a determination of Yes is made in step S407, the process proceeds to step S408,
L ← L + 1
i ← i + 1
Update processing of the values of L and i in step S410,
Element X of matrix X (i, j) = 1
It is determined whether or not.

X (i, j) ≠ 1
Steps S408 and S410 are repeated until In step S410, when X (i, j) ≠ 1, i in step S412,
i <U
In step S414,
It is determined whether or not the current frame processing mode D = thinning in the time direction.

In step S414,
If it is determined that the processing mode D of the current frame is decimation in the time direction, the process proceeds to step S416.
S ← i-L
E ← i-tL / N-1
Set up.
S is the start pixel position of the region where the value [+1] of the element X (i, J) of the initialization matrix X should be [0], and E is the end pixel position.

  The correspondence between S and E will be described with reference to FIG. In the current frame that is the processing target data, the pixel position where the value X of the element X (i, j) of the initial setting matrix X is set to 1 is an area corresponding to the section L shown in the drawing. From this point, it is necessary to leave the pixel position to be actually corrected and set the value of the element X (i, j) of the matrix X corresponding to the other part = 0.

  As shown in FIG. 42, this section differs depending on N frames n to n + 3 in the current frame. Each frame is indicated by n + t. In each frame n + t (t = 0 to 3 in the case of N = 4), the portion that does not require correction is a section from S to E as shown in the figure.

In the example of FIG.
For frame n + 0, S to E (n)
For frame n + 1, S to E (n + 1)
For frame n + 2, S to E (n + 2)
For frame n + 3, S to E (n + 3)
It becomes.
The process of calculating and setting these S to E sections is the process of step S416.

Step S416 is the determination of step S414, that is,
This is processing when it is determined that the current frame processing mode D = thinning in the time direction. This is a case where the right side adjacent portion of the section L is thinning in the time direction, and determines the processing mode of the data portion 851 shown in FIG. 42. In the case of the data example shown in FIG. That is,
The current frame processing mode D = decimation in the time direction is determined as Yes,
The processing in step S416 is executed to determine the above-described S to E interval, that is, the interval in which the value of the element X (i, j) of the matrix X is to be changed from [+1] to [0].

Next, it progresses to step S420 and during X (i, j),
Assuming that the value of i corresponding to the value of the interval S to E is k,
The set value of the element X (k, j) of the initial setting matrix X is changed to [0].
By this process, the value of the element X (i, j) of the matrix X corresponding to the sections S to E shown in FIG. 42 is changed to [0], set as an area that does not require correction, and an area 852 shown in FIG. Only the element X (i, j) of the matrix corresponding to is set as [+1].

The determination in step S414 is No, that is,
The data configuration when it is determined that the processing mode D of the current frame is not decimation in the time direction corresponds to the configuration shown in FIG. This is a case where the processing mode of the right adjacent portion of the section L, that is, the data portion 861 shown in FIG. in this case,
Step S417, that is,
S ← i−L + tL / N
E ← i-1
Set up.
That is, the same S to E interval as described above, that is, the interval in which the value of the element X (i, j) of the matrix X is to be changed from [+1] to [0] is determined.

As shown in FIG.
For frame n + 0, S (n) to E
For frame n + 1, S (n + 1) to E
For frame n + 2, S (n + 2) to E
For frame n + 3, S (n + 3) to E
It becomes.
The process of calculating and setting the sections S to E is the process of step S417.

Next, it progresses to step S420 and during X (i, j),
Assuming that the value of i corresponding to the value of the interval S to E is k,
The set value of the element X (k, j) of the initial setting matrix X is changed to [0].
By this processing, the value of the element X (i, j) of the matrix X corresponding to the sections S to E shown in FIG. 43 is changed to [0], set as an area that does not require correction, and an area 862 shown in FIG. Only the element X (i, j) of the matrix corresponding to is set as [+1].

  In Step S407, the case of No is a case where the current is space thinning and the past is time thinning, and corresponds to the region of the section L shown in FIGS.

If the determination of No is made in step S407, the process proceeds to step S409,
L ← L + 1
i ← i + 1
Update processing of the values of L and i, and in step S411,
Element X (i, j) of matrix X = −1
It is determined whether or not.

X (i, j) ≠ -1
Steps S409 and S411 are repeated until In step S411, when X (i, j) ≠ −1, i in step S413 is
i <U
In step S415,
It is determined whether or not the current frame processing mode D = thinning in the time direction.

In step S415,
If it is determined that the processing mode D of the current frame is decimation in the time direction, the process proceeds to step S418,
S ← i-L
E ← i−L + L (t + 1) / N−1
Set up.
S is the start pixel position of the region where the value [−1] of the element X (i, J) of the initialization matrix X should be [0], and E is the end pixel position.

  The correspondence between S and E will be described with reference to FIG. In the current frame, which is the processing target data, the pixel position where the value of the element X (i, j) of the initial setting matrix X is set to −1 is an area corresponding to the section L shown in the drawing. From this point, it is necessary to leave the pixel position to be actually corrected and set the value of the element X (i, j) of the matrix X corresponding to the other part = 0.

  As shown in FIG. 44, this section differs depending on N frames n to n + 3 in the current frame. Each frame is indicated by n + t. In each frame n + t (t = 0 to 3 in the case of N = 4), the portion that does not require correction is a section from S to E as shown in the figure.

In the example of FIG.
For frame n + 0, S to E (n)
For frame n + 1, S to E (n + 1)
For frame n + 2, S to E (n + 2)
For frame n + 3, S to E (n + 3)
It becomes.
The process of calculating and setting these S to E sections is the process of step S418.

Step S418 is the determination of step S415, that is,
This is processing when it is determined that the current frame processing mode D = thinning in the time direction. This is a case where the right side adjacent portion of the section L is thinning in the time direction, and determines the processing mode of the data portion 871 shown in FIG. 44. In the case of the data example shown in FIG. That is,
The current frame processing mode D = decimation in the time direction is determined as Yes,
The processing in step S418 is executed to determine the above-described S to E interval, that is, the interval in which the value of the element X (i, j) of the matrix X is to be changed from [−1] to [0].

Next, it progresses to step S420 and during X (i, j),
Assuming that the value of i corresponding to the value of the interval S to E is k,
The set value of the element X (k, j) of the initial setting matrix X is changed to [0].
By this process, the value of the element X (i, j) of the matrix X corresponding to the sections S to E shown in FIG. 44 is changed to [0], set as an area that does not require correction, and an area 872 shown in FIG. Only the element X (i, j) of the matrix corresponding to is set as [−1].

The determination in step S415 is No, ie,
The data configuration when it is determined that the processing mode D of the current frame is not decimation in the time direction corresponds to the configuration shown in FIG. This is a case where the processing mode of the right adjacent portion of the section L, that is, the data portion 881 shown in FIG. 45 is thinning in the spatial direction. in this case,
Step S419, that is,
S ← i−L (t + 1) / N
E ← i-1
Set up.
That is, the same S to E interval as described above, that is, the interval in which the value of the element X (i, j) of the matrix X is to be changed from [−1] to [0] is determined.

As shown in FIG.
For frame n + 0, S (n) to E
For frame n + 1, S (n + 1) to E
For frame n + 2, S (n + 2) to E
For frame n + 3, S (n + 3) to E
It becomes.
The process of calculating and setting the sections S to E is the process of step S419.

Next, it progresses to step S420 and during X (i, j),
Assuming that the value of i corresponding to the value of the interval S to E is k,
The set value of the element X (k, j) of the initial setting matrix X is changed to [0].
By this processing, the value of the element X (i, j) of the matrix X corresponding to the sections S to E shown in FIG. 45 is changed to [0], set as an area that does not require correction, and an area 882 shown in FIG. Only the element X (i, j) of the matrix corresponding to is set as [−1].

By the processing of the flowchart shown in FIG.
The initial setting matrix X set in the processing steps S302 and S303 in FIG.
[-1] if the current frame is spatial thinning and the past frame is time thinning,
[+1] if the current frame is spatial decimation and the future frame is time decimation,
[0] in other cases,
For the initialization matrix X in which each value of is set,
The sections S to E that do not require correction are determined, and a matrix X that is changed to a value [0] indicating that correction is not necessary is generated for the sections S to E.

  The process described with reference to FIGS. 39 and 41 is a process for the correction target area of the horizontal spatial thinning area, and corresponds to the process of step S202 in the process flow shown in FIG. The same process is executed for the correction target area of the vertical direction spatial direction thinning area, and the matrix Y is generated.

In this way, the matrix X indicating the correction target area and correction mode of the horizontal direction thinning area in the horizontal direction and the matrix Y indicating the correction target area and correction mode of the vertical direction thinning area are generated, and are shown in FIG. In step S204, the correction processing application matrix Ct is
Ct ← X + Y
Generate as

This correction processing application matrix Ct is set as a matrix indicating the correction area and correction mode of the pixels (pixels) corresponding to the frame n + t. The correction processing application matrix Ct has an element value of 0, -1, or +1, and each element value indicates a pixel that executes the following processing.
a. (I, j) element of matrix Ct = 0: correction of pixel (i, j) of frame n + t is unnecessary.
b. (I, j) element of matrix Ct = −1: Replace pixel (i, j) of frame n + t with pixel value of pixel (i, j) of past frame n−1.
c. (I, j) element of matrix Ct = + 1: Replace pixel (i, j) of frame n + t with pixel value of pixel (i, j) of future frame n + N.

  The correction target pixel detection unit 655 of the synthesis unit 650 illustrated in FIG. 30 generates a matrix Ct corresponding to each frame n + t by executing the processing described in the processing flow of FIG. 37 and the subsequent steps, and outputs the matrix Ct to the spatial thinning region correction unit 656. To do.

The spatial thinning region correction unit 656 acquires the correction target frame (n + t) from the image data memory, refers to the correction processing application matrix Ct, and determines the pixel (pixel) position that needs to be corrected and the correction mode, that is, the past frame. Whether to refer to the future frame is determined, and pixel value correction is executed according to the determined mode. The correction target pixel is
(1) Past N frames = thinning in time direction and current N frames = thinning in space direction
(2) The data area of the current N frame = spatial direction thinning and the future N frame = time direction thinning, and the pixels in the section excluding the above-described sections S to E.

  With the above pixel value correction processing, it is possible to reduce image quality degradation that may occur when spatial direction thinning is preferentially performed in a region where the amount of movement of the thinning processing unit frame: N frames greatly changes. Become.

  In the above-described embodiment, the configuration example in which the restoration unit that performs the restoration process of the image performs the process of detecting the correction target region pixels has been described. However, the image that generates the compressed image data by performing the thinning process On the conversion device side, the same processing as described above may be executed to obtain the pixel position information of the region to be corrected. If this pixel position information is set as attribute information of the compressed image data that has been subjected to the thinning process and provided to the apparatus that executes the restoration process, the apparatus that executes the restoration process can detect the correction target pixel area described above. It is possible to obtain the pixel position information of the region to be corrected from the attribute information without performing the correction process and perform the correction process.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims should be taken into consideration.

  The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and executed.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  The various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the present invention, the optimal compression processing mode corresponding to the characteristics of each region of the image, particularly the movement of the subject is determined, and the optimal mode is determined for each region according to the determined mode. By adopting a configuration for performing data conversion processing, compression and decompression with very little quality degradation can be performed. In the configuration of the present invention, the maximum movement amount of the subject in the block constituting the moving image data is detected, and based on the detected maximum movement amount, the spatial direction thinning process, the time direction thinning process, or the spatial direction and time is performed. The direction thinning process is selectively executed. With this configuration, data conversion that suppresses deterioration in image quality that causes a super-resolution effect is realized.

  In particular, even in the case of image data in which the amount of movement varies greatly between frames, the moving image conversion apparatus of the present invention detects the maximum amount of movement included in each block of the thinning-out processing unit frame number N to determine the spatial direction. Since the thinning is preferentially performed, image quality deterioration due to thinning can be minimized. Furthermore, according to the moving image restoration apparatus of the present invention, it is possible to reduce image quality degradation due to preferential thinning in the spatial direction by pixel value correction and restore high-quality data.

It is a figure which shows the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure which shows the detail of the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a figure explaining the process of the block process part in a moving image converter. It is a block diagram which shows the structure of the decompression | restoration part which performs decompression | restoration of the compressed data produced | generated by the thinning process. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure explaining the process of the block expansion part in a decompression | restoration part. It is a figure which shows the mode of the fluctuation | variation of the movement amount in the boundary of a movement area | region and a stationary area | region when the number N of thinning-out process unit frames is set to 4. FIG. It is a figure explaining the structural example of the image conversion apparatus which performs the data conversion using the super-resolution effect of this invention. It is a figure which shows the detail of the basic composition of the moving image converter which performs the data conversion using a super-resolution effect. It is a figure explaining the structure and process of the maximum movement amount detection part in a moving image converter. FIG. 6 is a diagram for explaining whether image quality degradation is more conspicuous in a situation where time thinning is applied to all N frames and spatial thinning is applied to all N frames when the number of thinning-out processing unit frames is N It is. FIG. 6 is a diagram for explaining whether image quality degradation is more conspicuous in a situation where time thinning is applied to all N frames and spatial thinning is applied to all N frames when the number of thinning-out processing unit frames is N It is. FIG. 6 is a diagram for explaining whether image quality degradation is more conspicuous in a situation where time thinning is applied to all N frames and spatial thinning is applied to all N frames when the number of thinning-out processing unit frames is N It is. FIG. 6 is a diagram for explaining whether image quality degradation is more conspicuous in a situation where time thinning is applied to all N frames and spatial thinning is applied to all N frames when the number of thinning-out processing unit frames is N It is. It is a block diagram explaining the structural example of a decompression | restoration part. It is a block diagram shown about the detailed structure of the synthetic | combination part which comprises a decompression | restoration part. It is a figure explaining the aspect of the pixel value correction process performed in a synthetic | combination part. It is a figure explaining the aspect of the pixel value correction process performed in a synthetic | combination part. It is a figure showing the classification of the thinning | decimation performed in a part of a certain row | line | column of each frame from the frame n-4 to n + 3 as N = 4. It is a figure explaining the specific example of the pixel value correction process performed in a synthetic | combination part. It is a figure explaining the process example in the case of performing pixel value correction of the pixel of the present frame using the pixel of the future frame. It is a figure explaining the process example in the case of performing pixel value correction of the pixel of the present frame using the pixel of the future frame. It is a figure which shows the flowchart explaining the process sequence which the correction object pixel detection part of a synthetic | combination part performs. It is a figure which shows the flowchart explaining the process sequence which the correction object pixel detection part of a synthetic | combination part performs. It is a process sequence which the correction object pixel detection part of a synthetic | combination part performs, and is a figure which shows the correction object pixel detection process of a horizontal direction space thinning process area | region. It is a figure explaining the specific example of the correction object pixel detection process of a horizontal direction space thinning-out process area. It is a processing sequence executed by the correction target pixel detection unit of the synthesis unit, and is a diagram illustrating a flowchart for explaining processing for detecting non-correction pixel regions S to E in the correction target pixel detection processing of the horizontal direction spatial thinning processing region. . It is a figure explaining the specific example of the process which detects non-correction object pixel area | regions S-E in the correction object pixel detection process of a horizontal direction space thinning-out process area. It is a figure explaining the specific example of the process which detects non-correction object pixel area | regions S-E in the correction object pixel detection process of a horizontal direction space thinning-out process area. It is a figure explaining the specific example of the process which detects non-correction object pixel area | regions S-E in the correction object pixel detection process of a horizontal direction space thinning-out process area. It is a figure explaining the specific example of the process which detects non-correction object pixel area | regions S-E in the correction object pixel detection process of a horizontal direction space thinning-out process area.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image converter 11 Block division part 12 Movement amount detection part 13 Block processing part 14 Output part 15 Restoration part 21 Image storage part 22 Block division part 31 Movement amount detection part 32 Block distribution part 51-53 Block processing part 61 Block distribution part 62 to 64 block extension unit 65 composition unit 71 block 80 to 83 block 90, 92 frame 91, 93 pixel 94, 96 frame 95, 97 pixel 100 image conversion device 110 block division unit 120 movement amount detection unit 130 block processing unit 140 output Unit 150 restoration unit 111 image storage unit 112 block division unit 121 maximum movement amount detection unit 122 block distribution unit 131-133 block processing unit 140 output unit 210-213 block matching unit 220 memory 230 selector 610 minutes Parts 620-640 block extension 650 combining unit 651 combining unit 652 the image data memory 654 thinning information memory 655 correction target pixel detecting section 656 spatial thinning area correction unit

Claims (20)

A moving image conversion apparatus that executes data conversion processing of moving image data,
A block division unit that executes block division processing for each frame constituting the moving image data;
A movement amount detection unit that detects a movement amount of a subject in each block divided by the block division unit, and the maximum value of the movement amount of the subject from a plurality of blocks corresponding to pixel positions in a plurality of consecutive frames that are thinning processing execution units. A movement amount detection unit for detecting
The maximum movement amount information detected by the movement amount detection unit is input, and based on the maximum movement amount information, a spatial direction thinning process, a time direction thinning process, or a space for a plurality of consecutive frames serving as a thinning process execution unit A block distribution unit that determines whether to perform the direction thinning process and the time direction thinning process, and executes a process of supplying the block processing target block data to the block processing unit that performs the thinning process;
Block data and the maximum movement amount information are input from the block distribution unit, and based on the maximum movement amount information, one of spatial direction thinning processing, time direction thinning processing, and spatial direction and time direction thinning processing is executed. A block processing unit to
A moving image conversion apparatus comprising: The movement amount detector
For each of a plurality of blocks corresponding to pixel positions in a plurality of consecutive frames as a thinning process execution unit, a block matching process with a past frame is executed, and a process for detecting the maximum value of the subject movement amount from the plurality of blocks is executed. The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus is configured to perform the above operation. The movement amount detector
The processing for detecting the maximum value of the subject movement amount is executed from a plurality of blocks corresponding to pixel positions in N consecutive frames, where N consecutive frames as a thinning process execution unit, where N ≧ 2, are processing units. The moving image conversion apparatus according to claim 1, wherein the moving image conversion apparatus has a configuration. The block distributor is
Based on the maximum movement amount information detected by the movement amount detection unit, it is a configuration for determining a thinning processing mode,
In the setting where the maximum movement amount is Vt, two predetermined thresholds Va and Vb, where Va> Vb,
Maximum travel: When Vt ≧ Va,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a spatial direction thinning process,
Maximum travel: When Vt <Vb
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a time direction thinning process,
When the maximum movement amount Vt is Vb ≦ Vt <Va,
2. The moving image conversion apparatus according to claim 1, wherein execution processing for a plurality of continuous frames serving as a thinning processing execution unit is determined as thinning processing in a spatial direction and a time direction. The moving image conversion device further includes:
A block that has been subjected to spatial direction thinning processing is configured to perform processing for calculating pixel position information necessary for pixel value correction processing as attribute information after image expansion, after block expansion corresponding to spatial direction thinning. The moving image conversion apparatus according to claim 1. A moving image restoration apparatus that performs a restoration process of moving image conversion data,
A block extension unit that inputs block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executes an extension process of the block correspondence conversion data based on the conversion mode information;
A synthesis unit that synthesizes the blocks restored by the block extension process in the block extension unit and generates frame data;
The block extension is
Block restoration is performed by block expansion processing according to a thinning mode of at least one of spatial direction thinning processing, time direction thinning processing, or spatial direction and time direction thinning processing executed in the generation of the moving image conversion data. ,
The synthesis unit is
It is a configuration for generating frame data by synthesizing the blocks restored by the block extension process, and further, for the processing block on which the block extension process corresponding to the spatial direction thinning process is executed,
(1) The past frame is thinned out in the time direction, and the current frame is thinned out in the spatial direction.
(2) The current frame is thinned in the spatial direction, and the future frame is thinned in the time direction.
A moving image restoration apparatus characterized by being configured to execute a correction process of a pixel value of a block corresponding to either (1) or (2) above. The synthesis unit is
A correction target pixel detection unit that detects a target pixel position for executing correction processing of the pixel value;
A spatial direction thinning area correction unit that performs pixel value correction based on detection information of the correction target pixel detection unit;
The correction target pixel detection unit includes:
For pixels included in a processing block that has been subjected to block expansion processing corresponding to spatial direction thinning processing,
(A) Pixels in which the current frame is thinning in the spatial direction and the past frame is thinning in the time direction;
(B) a pixel if the current frame is spatial direction thinning and the future frame is time direction thinning;
(C) Other pixels,
The moving image restoration apparatus according to claim 6, wherein the moving image restoration apparatus is configured to execute processing for generating an initial setting matrix corresponding to a pixel position where the pixel can be identified. The correction target pixel detection unit includes:
Furthermore, the movement trajectory of the subject is estimated from the thinning processing mode of the current frame and the past frame, or the current frame and the future frame, and the initial setting matrix is
(A) a pixel to be replaced with a pixel value of a pixel of a current frame with a pixel value of a pixel of a past frame;
(B) a pixel to be replaced with a pixel value of a pixel of a current frame with a pixel value of a pixel of a future frame;
(C) pixels that do not require correction;
8. The moving image restoration apparatus according to claim 7, wherein the moving image restoration apparatus is configured to execute a process of generating a correction processing application matrix Ct that can discriminate the pixels. The spatial direction thinning region correction unit is
Based on each element value set in the correction processing application matrix Ct, the pixel value correction of the pixel included in the block area where the spatial direction decimation in the current frame is performed is any one of the above (a) to (c) The moving image restoration apparatus according to claim 8, wherein the moving image restoration apparatus is executed according to an aspect. A moving image conversion method for executing data conversion processing of moving image data,
A block division step for executing a block division process for each frame constituting the moving image data;
This is a movement amount detection step for detecting a subject movement amount in each block divided in the block division step, and the maximum value of the subject movement amount from a plurality of blocks corresponding to pixel positions in a plurality of continuous frames as a thinning process execution unit. A movement amount detecting step for detecting
The maximum movement amount information detected in the movement amount detection step is input, and based on the maximum movement amount information, a spatial direction thinning process, a time direction thinning process, or a space for a plurality of consecutive frames serving as a thinning process execution unit A block distribution step for determining which of the direction and time direction thinning processing is to be executed, and executing processing for supplying the block processing target block data to the block processing unit that performs the thinning processing;
A block processing step of inputting block data and the maximum movement amount information, and executing any one of spatial direction thinning processing, time direction thinning processing, and spatial direction and time direction thinning processing based on the maximum movement amount information; ,
A moving image conversion method characterized by comprising: The movement amount detection step includes:
For each of a plurality of blocks corresponding to pixel positions in a plurality of consecutive frames as a thinning process execution unit, a block matching process with a past frame is executed, and a process for detecting the maximum value of the subject movement amount from the plurality of blocks is executed. The moving image conversion method according to claim 10, wherein: The movement amount detection step includes:
The processing for detecting the maximum value of the subject movement amount is executed from a plurality of blocks corresponding to pixel positions in N consecutive frames, where N consecutive frames as a thinning process execution unit, where N ≧ 2, are processing units. The moving image conversion method according to claim 10. The block distribution step includes:
A step of determining a thinning-out processing mode based on the maximum movement amount information detected in the movement amount detection step;
In the setting where the maximum movement amount is Vt, two predetermined thresholds Va and Vb, where Va> Vb,
Maximum travel: When Vt ≧ Va,
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a spatial direction thinning process,
Maximum travel: When Vt <Vb
The execution process for a plurality of consecutive frames as a thinning process execution unit is determined as a time direction thinning process,
When the maximum movement amount Vt is Vb ≦ Vt <Va,
The moving image conversion method according to claim 10, wherein execution processing for a plurality of continuous frames serving as a thinning processing execution unit is determined as thinning processing in a spatial direction and a time direction. The moving image conversion method further includes:
The block having undergone spatial direction thinning processing has a step of executing processing for calculating pixel position information necessary for pixel value correction processing as attribute information after block expansion corresponding to spatial direction thinning at the time of image restoration. The moving image conversion method according to claim 10. It is a moving image restoration method for executing moving image conversion data restoration processing,
A block expansion step for inputting block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executing a block correspondence conversion data expansion process based on the conversion mode information;
Combining a block restored by the block expansion process in the block expansion step to generate frame data,
The block expansion step includes
Block restoration is performed by block expansion processing according to a thinning mode of at least one of spatial direction thinning processing, time direction thinning processing, or spatial direction and time direction thinning processing executed in the generation of the moving image conversion data. ,
The synthesis step includes
It is a configuration for generating frame data by synthesizing the blocks restored by the block extension process, and further, for the processing block on which the block extension process corresponding to the spatial direction thinning process is executed,
(1) The past frame is thinned out in the time direction, and the current frame is thinned out in the spatial direction.
(2) The current frame is thinned in the spatial direction, and the future frame is thinned in the time direction.
A moving image restoration method comprising a step of executing a correction process of a pixel value of a block corresponding to either (1) or (2) above. The synthesis step includes
A correction target pixel detection step of detecting a target pixel position for executing the correction processing of the pixel value;
A spatial direction thinning area correction step for performing pixel value correction based on detection information of the correction target pixel detection step;
The correction target pixel detection step includes:
For pixels included in a processing block that has been subjected to block expansion processing corresponding to spatial direction thinning processing,
(A) Pixels in which the current frame is thinning in the spatial direction and the past frame is thinning in the time direction;
(B) a pixel if the current frame is spatial direction thinning and the future frame is time direction thinning;
(C) Other pixels,
The moving image restoration method according to claim 15, wherein a process of generating an initial setting matrix corresponding to a pixel position where the pixel can be identified is executed. The correction target pixel detection step includes:
Furthermore, the movement trajectory of the subject is estimated from the thinning processing mode of the current frame and the past frame, or the current frame and the future frame, and the initial setting matrix is
(A) a pixel to be replaced with a pixel value of a pixel of a current frame with a pixel value of a pixel of a past frame;
(B) a pixel to be replaced with a pixel value of a pixel of a current frame with a pixel value of a pixel of a future frame;
(C) pixels that do not require correction;
17. The moving image restoration method according to claim 16, wherein a process of generating a correction processing application matrix Ct capable of discriminating the pixels is executed. The spatial direction thinning area correction step includes:
Based on each element value set in the correction processing application matrix Ct, the pixel value correction of the pixel included in the block area where the spatial direction decimation in the current frame is performed is any one of the above (a) to (c) The moving image restoration method according to claim 17, wherein the moving image restoration method is executed according to an aspect. A computer program for executing data conversion processing of moving image data on a computer,
A block division step for executing a block division process for each frame constituting the moving image data;
This is a movement amount detection step for detecting a subject movement amount in each block divided in the block division step, and the maximum value of the subject movement amount from a plurality of blocks corresponding to pixel positions in a plurality of continuous frames as a thinning process execution unit. A movement amount detecting step for detecting
The maximum movement amount information detected in the movement amount detection step is input, and based on the maximum movement amount information, a spatial direction thinning process, a time direction thinning process, or a space for a plurality of consecutive frames serving as a thinning process execution unit A block distribution step for determining which of the direction and time direction thinning processing is to be executed, and executing processing for supplying the block processing target block data to the block processing unit that performs the thinning processing;
A block processing step of inputting block data and the maximum movement amount information, and executing any one of spatial direction thinning processing, time direction thinning processing, and spatial direction and time direction thinning processing based on the maximum movement amount information; ,
A computer program characterized by comprising: A computer program for executing a moving image conversion data restoration process on a computer,
A block expansion step for inputting block correspondence conversion data constituting the moving image conversion data and conversion mode information of the block correspondence conversion data, and executing a block correspondence conversion data expansion process based on the conversion mode information;
Combining a block restored by the block expansion process in the block expansion step to generate frame data,
The block expansion step includes
Block restoration is performed by block expansion processing according to a thinning mode of at least one of spatial direction thinning processing, time direction thinning processing, or spatial direction and time direction thinning processing executed in the generation of the moving image conversion data. ,
The synthesis step includes
It is a configuration for generating frame data by synthesizing the blocks restored by the block extension process, and further, for the processing block on which the block extension process corresponding to the spatial direction thinning process is executed,
(1) The past frame is thinned out in the time direction, and the current frame is thinned out in the spatial direction.
(2) The current frame is thinned in the spatial direction, and the future frame is thinned in the time direction.
A computer program comprising a step of executing correction processing of pixel values of a block corresponding to either (1) or (2) above.

Images