JP2004247972A - Image processing apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus correcting the unstable frames included in a moving picture highly precisely. <P>SOLUTION: The image processing apparatus for correcting the unstable frames included in a moving picture is provided with: an inter-frame motion vector calculation section for calculating the inter-frame motion vector denoting the motion of an object over a plurality of the frames on the basis of a plurality of the frames included in the moving picture; an in-frame motion vector calculation section for calculating the in-frame motion vector representing a motion of the object in a frame imaging time being a time for imaging a particular frame among a plurality of the frames on the basis of the inter-frame motion vector; and a shake correction section for correcting the shake of the object on the basis of the motion of the object in the particular frame on the basis of the in-frame motion vector. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and an image processing method. In particular, the present invention relates to an image processing apparatus and an image processing method for correcting blurring of a frame included in a moving image.
[0002]
[Prior art]
Conventionally, a video printer that prints an arbitrary image of moving image information captured by a video camera as a still image has been developed. In this video printer, a moving image shot by a video camera is reproduced on a monitor screen, and when a desired image in the moving image is selected by the user, a still image is printed based on the video signal of the frame ( For example, see Patent Document 1.)
[0003]
[Patent Document 1]
Japanese Patent Application Laid-Open No. 6-311466
[Problems to be solved by the invention]
However, since the frames included in the video shot by the video camera have a low resolution and may include disturbance due to camera shake or the like, a beautiful image like a photograph cannot be obtained only from the video signal of the frame. .
[0005]
SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus and an image processing method capable of solving the above-described problems. This object is achieved by a combination of features described in the independent claims. The dependent claims define further advantageous specific examples of the present invention.
[0006]
[Means for Solving the Problems]
That is, according to the first aspect of the present invention, an image processing apparatus that corrects blurring of frames included in a moving image, and shows the movement of a subject across a plurality of frames based on the plurality of frames included in the moving image. An inter-frame motion vector calculation unit that calculates an inter-frame motion vector, and based on the inter-frame motion vector, indicates a motion of a subject within a frame imaging time that is a time during which a specific frame is imaged. An intra-frame motion vector calculation unit that calculates an intra-frame motion vector, and a blur correction unit that corrects subject blur based on the motion of the subject in a specific frame based on the intra-frame motion vector.
[0007]
Based on the inter-frame motion vector, it further includes a super-resolution unit that moves the subject imaged in multiple frames to the same position and overlays multiple frames to increase the resolution of a specific frame The blur correction unit may correct blur of the subject in a specific frame whose resolution is increased by the super resolution unit.
[0008]
When an interframe motion vector calculation unit calculates an interframe motion vector based on an output size acquisition unit that acquires an output size on which a specific frame is to be printed or displayed, and an output size acquired by the output size acquisition unit A frame number determining unit that determines the number of frames to be used, and the inter-frame motion vector calculating unit may calculate the inter-frame motion vector based on the number of frames determined by the frame number determining unit.
[0009]
The intra-frame motion vector calculation unit spans any pixel by moving the subject point, which is a point on the subject to be imaged by one pixel when the subject is stationary, within the frame imaging time. An intra-frame motion vector may be calculated by calculating whether the image is captured.
[0010]
The blur correction unit includes a pixel imaging time calculation unit that calculates a pixel imaging time that is a time during which a subject point is captured by a plurality of pixels in a specific frame based on an intra-frame motion vector, and a pixel imaging time Based on the plurality of pixel imaging times calculated by the calculation unit and the actual pixel value of each of the plurality of pixels, the ideal pixel value that is to be captured by each of the plurality of pixels when the subject is stationary And a pixel value calculation unit.
[0011]
According to the second aspect of the present invention, there is provided an image processing method for correcting blurring of frames included in a moving image, wherein frames between frames indicating movement of a subject across a plurality of frames are based on the plurality of frames included in the moving image. Based on the motion vector calculation step and the inter-frame motion vector, an intra-frame motion vector indicating the motion of the subject within a frame imaging time, which is a time during which a specific frame of a plurality of frames is imaged, is calculated. And correcting a blur of the subject based on the motion of the subject in a specific frame based on the intra-frame motion vector.
[0012]
The above summary of the invention does not enumerate all the necessary features of the present invention, and sub-combinations of these feature groups can also be the invention.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims, and all combinations of features described in the embodiments are included. It is not necessarily essential for the solution of the invention.
[0014]
FIG. 1 shows an example of the configuration of an image processing apparatus 100 according to an embodiment of the present invention. The image processing apparatus 100 is, for example, a video printer, calculates a subject's motion using a plurality of frames included in a moving image, performs super resolution processing and blur correction processing based on the subject's motion, Create and print a still image.
[0015]
The image processing apparatus 100 includes a moving image input unit 102 that inputs a moving image including a plurality of frames, a super resolution unit 104 that performs super resolution processing on a frame input by the moving image input unit 102, and a moving image input unit 102. A shake correction unit 106 that performs a shake correction process on the input frame and a still image output unit 108 that outputs a still image that has been subjected to the super resolution process and the shake correction process are provided.
[0016]
Furthermore, the image processing apparatus 100 is an inter-frame motion vector calculation unit 110 that calculates an inter-frame motion vector indicating the motion of a subject across a plurality of frames, and a time during which a specific frame is imaged. An intra-frame motion vector calculation unit 112 that calculates an intra-frame motion vector indicating the motion of the subject within the frame imaging time, an output size acquisition unit 114 that acquires an output size at which a specific frame is to be output, and an inter-frame motion vector And a frame number determination unit 116 that determines the number of frames used by the calculation unit 110.
[0017]
The output size acquisition unit 114 acquires an output size for printing or displaying a specific frame from the still image output unit 108. Further, the output size acquisition unit 114 may acquire an output size that is input by an operation by the user. Based on the output size acquired by the output size acquisition unit 114, the frame number determination unit 116 determines the number of frames used when the inter-frame motion vector calculation unit 110 calculates the inter-frame motion vector. For example, when the output size is larger than a predetermined size, the frame number determination unit 116 determines the number of frames larger than the predetermined number of frames. Thereby, even if the output size is very large, the inter-frame motion vector calculation unit 110 calculates the inter-frame motion vector using many frames, and the super resolution unit 104 uses many frames. Therefore, a super resolution process can be performed, and a high-quality still image can be obtained.
[0018]
The inter-frame motion vector calculation unit 110 acquires a plurality of frames including a specific frame to be output from the still image output unit 108 from the moving image input unit 102. Then, the inter-frame motion vector calculation unit 110 extracts a plurality of consecutive frames before and after the specific frame based on the number of frames determined by the frame number determination unit 116, and based on the extracted plurality of frames. To calculate an inter-frame motion vector.
[0019]
The intra-frame motion vector calculation unit 112 acquires a plurality of frames extracted by the inter-frame motion vector calculation unit 110. The intra-frame motion vector calculation unit 112 calculates an intra-frame motion vector based on the plurality of frames acquired from the inter-frame motion vector calculation unit 110 and the inter-frame motion vector calculated by the inter-frame motion vector calculation unit 110. . Specifically, the intra-frame motion vector calculation unit 112 is configured such that when a subject is stationary, a subject point that is a point on the subject to be imaged by one pixel moves within the frame imaging time. An intra-frame motion vector is calculated by calculating which pixel the image is captured across.
[0020]
Based on the inter-frame motion vector calculated by the inter-frame motion vector calculation unit 110, the super resolution unit 104 moves the subjects imaged in the plurality of frames to the same position and superimposes the plurality of frames. Thus, processing for enhancing the resolution of a specific frame is performed. Next, based on the intra-frame motion vector calculated by the intra-frame motion vector calculation unit 112, the blur correction unit 106 detects the subject based on the motion of the subject in the specific frame whose resolution is increased by the super resolution unit 104. Correct blur.
[0021]
The still image output unit 108 outputs, as a still image, a specific frame in which the resolution is increased by the super resolution unit 104 and the touch of the subject is corrected by the shake correction unit 106. The still image output unit 108 may be a printing unit that prints a still image or a display unit that displays a still image. Still image output unit 108 may be an interface that outputs a still image to an external image printing apparatus or image display apparatus.
[0022]
Although the super resolution unit 104 has increased the resolution of a specific frame using a plurality of frames, the shake correction unit 106 has corrected the blur of the subject of the specific frame that has been increased in resolution. After the blur correction unit 106 corrects the blur of the subject in a plurality of frames, the super resolution unit 104 may increase the resolution of a specific frame using the plurality of frames in which the blur is corrected.
[0023]
FIG. 2 shows an example of the configuration of the shake correction unit 106 according to the present embodiment. The shake correction unit 106 includes a pixel imaging time calculation unit 200 that calculates a pixel imaging time that is a time during which a subject point is captured by a plurality of pixels in a specific frame, and an ideal image that each of the plurality of pixels should capture. An ideal pixel value calculation unit 202 that calculates the pixel value of the output pixel, and an output image generation unit 204 that generates a still image for output based on the ideal pixel value calculated by the ideal pixel value calculation unit 202.
[0024]
The pixel imaging time calculation unit 200 calculates the pixel imaging time based on the intra-frame motion vector 112 calculated by the intra-frame motion vector calculation unit. Then, the ideal pixel value calculation unit 202 calculates an actual pixel value of each of a plurality of pixels in a specific frame whose resolution has been increased by the super resolution unit 104. Then, the ideal pixel value calculation unit 202 performs a specific operation when the subject is stationary based on the plurality of pixel imaging times captured by the pixel imaging time calculation unit 200 and the actual pixel values of each of the plurality of pixels. An ideal pixel value of each of a plurality of pixels in the frame is calculated. The output image generation unit 204 generates a still image based on the calculated ideal pixel value and outputs the still image to the still image output unit 108.
[0025]
According to the image processing apparatus 100 of the present embodiment, the resolution of a still image to be output based on the motion of the subject across a plurality of frames is increased, and the still image to be output based on the motion of the subject in a specific frame To compensate for camera shake. Therefore, since high resolution and blur correction can be performed with high accuracy, a still image with very high image quality can be printed or displayed.
[0026]
FIG. 3 shows an example of a region on the subject to be captured in each pixel of a specific frame. Specifically, the region A is a region to be captured in the pixel a, the region B is a region to be captured in the pixel b, and the region C is a region to be captured in the pixel c. An arrow 300 indicates an intra-frame motion vector of a specific frame. That is, the movement of the pixel a with respect to the subject is shown. That is, it is shown that the areas A, B, and C are reflected in the pixel a of a specific frame due to camera shake or the like. The shake correction unit 106 of the present embodiment can correct such a subject shake using an intra-frame motion vector.
[0027]
FIG. 4 shows an example of an operation flow of the shake correction unit 106 according to the present embodiment. Hereinafter, the operation of the shake correction unit 106 will be described with reference to FIGS. 3 and 4.
[0028]
The pixel imaging time calculation unit 200 obtains a subject point 300 that appears in each pixel of the specific frame (S400). Then, the pixel imaging time calculation unit 200 calculates the pixel imaging time t that appears in each pixel within the frame imaging time of a specific frame, based on the intra-frame motion vector calculated by the intra-frame motion vector calculation unit 112. Then, the time during which each region is captured in each pixel is calculated (S402). For example, a time t1 when the region A is captured in the pixel a, a time t2 when the region B is captured in the pixel a, and a time t3 when the region C is captured in the pixel a are calculated. Further, the ideal pixel value calculation unit 202 obtains an actual pixel value R of each pixel of a specific frame (S404). For example, the actual pixel value Ra of the pixel a, the actual pixel value Rb of the pixel b, and the actual pixel value Rc of the pixel c are obtained.
[0029]
Next, the ideal pixel value calculation unit 202 uses the actual pixel value R, the pixel imaging time t, and the ideal pixel value I of each pixel to determine the relationship between the actual pixel value R and the ideal pixel value I. A determinant to be expressed is determined (S406). The ideal pixel value I is a pixel value to be captured in each pixel when the subject is stationary with respect to each pixel. The ideal pixel value Ia of the pixel a, the ideal pixel value Ib of the pixel b, and Let it be an ideal pixel value Ib of the pixel c.
[0030]
For example, an expression representing the relationship between the actual pixel value Rb of the pixel a and the ideal pixel value Ia is as follows.
Rb = α1 * Ia + α2 * Ib + α3 * Ic
here,
α1 = t1 / (t1 + t2 + t3)
α2 = t2 / (t1 + t2 + t3)
α3 = t3 / (t1 + t2 + t3)
And
[0031]
Similarly, for the other pixels, the expression representing the relationship between the actual pixel value R and the ideal pixel value I is as follows.
Rc = β1 * Ib + β2 * Ic + β3 * Id
Rd = γ1 * Ic + γ2 * Id + γ3 * Ie
...
[0032]
The ideal pixel value calculation unit 202 calculates the ideal pixel value I of each pixel by substituting the actual pixel value R and pixel imaging time t of each pixel and solving the determinant (S408). Then, the output image generation unit 204 corrects the pixel value of each pixel of a specific frame based on the ideal pixel value I and generates a still image to be output (S410). The operation flow of the blur correction unit 106 is thus completed.
[0033]
3 and 4, the case where each pixel moves only in the main scanning direction with respect to the subject has been described. However, each pixel moves in the main scanning direction and multiple scanning directions with respect to the subject. Even in this case, by calculating the motion vector within the frame of the subject, the ideal pixel value of each pixel can be similarly determined using the actual pixel value of each pixel.
[0034]
According to the blur correction unit 106 according to the present embodiment, the blur of the subject is corrected by calculating an ideal pixel value to be captured in each pixel based on the actual pixel value actually reflected in each pixel. It is possible to generate a still image very close to an actual subject image. Therefore, it is possible to print or display a still image according to the user's intention without performing image processing contrary to the user's intention.
[0035]
As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. Various modifications or improvements can be added to the above embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.
[0036]
【The invention's effect】
As is apparent from the above description, according to the image processing apparatus of the present invention, a high-resolution still image can be obtained from a moving image.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating an example of a configuration of an image processing apparatus.
2 is a diagram illustrating an example of a configuration of a shake correction unit 106. FIG.
FIG. 3 is a diagram illustrating an example of a region on a subject to be captured in each pixel of a specific frame.
4 is a diagram illustrating an example of an operation flow of a shake correction unit 106. FIG.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 100 Image processing apparatus 102 Movie input part 104 Super-resolution part 106 Blur correction part 108 Still image output part 110 Inter-frame motion vector calculation part 112 In-frame motion vector calculation part 114 Frame number determination part 116 Output size acquisition part 200 Pixel imaging Time calculation unit 202 ideal pixel value calculation unit 204 output image generation unit 300 subject point 302 intra-frame motion vector

Claims (6)

An image processing device that corrects blurring of frames included in a video,
An inter-frame motion vector calculating unit that calculates an inter-frame motion vector indicating the motion of the subject across the plurality of frames based on the plurality of frames included in the video;
Based on the inter-frame motion vector, an intra-frame motion vector calculation that calculates an intra-frame motion vector indicating a motion of the subject within a frame imaging time that is a time during which a specific frame of the plurality of frames is captured. And
An image processing apparatus comprising: a shake correction unit configured to correct shake of the subject based on motion of the subject in the specific frame based on the intra-frame motion vector. Based on the inter-frame motion vector, the subject imaged in each of the plurality of frames is moved to the same position, and the plurality of frames are overlapped to enhance the resolution of the specific frame. Further equipped with
The image processing apparatus according to claim 1, wherein the blur correction unit corrects blur of the subject in the specific frame whose resolution is increased by the super resolution unit. An output size acquisition unit for acquiring an output size on which the specific frame is to be printed or displayed;
A frame number determination unit that determines the number of frames used when the inter-frame motion vector calculation unit calculates the inter-frame motion vector based on the output size acquired by the output size acquisition unit;
The image processing apparatus according to claim 1, wherein the inter-frame motion vector calculation unit calculates the inter-frame motion vector based on the number of frames determined by the frame number determination unit. The intra-frame motion vector calculation unit is configured such that when a subject is stationary, a subject point that is a point on the subject to be imaged by one pixel moves within the frame imaging time. The image processing apparatus according to claim 1, wherein the intra-frame motion vector is calculated by calculating whether the image is captured across pixels. The blur correction unit is
A pixel imaging time calculation unit that calculates a pixel imaging time that is a time during which the subject point is captured by a plurality of pixels in the specific frame based on the intra-frame motion vector;
Based on the plurality of pixel imaging times calculated by the pixel imaging time calculation unit and the actual pixel values of each of the plurality of pixels, each of the plurality of pixels should be imaged when the subject is stationary The image processing apparatus according to claim 4, further comprising: an ideal pixel value calculation unit that calculates a pixel value of An image processing method for correcting blur of a frame included in a movie,
Calculating an inter-frame motion vector indicating the motion of the subject across the plurality of frames based on the plurality of frames included in the video;
Calculating an intra-frame motion vector indicating a motion of the subject within a frame imaging time, which is a time during which a specific frame of the plurality of frames is imaged, based on the inter-frame motion vector;
Correcting the shake of the subject based on the motion of the subject in the specific frame based on the intra-frame motion vector.

Images