JP2008053875A - Image processing apparatus and method, program, and program storage medium - Google Patents

Abstract

A motion vector is obtained stably at high speed.
A multi-resolution converting unit converts an input image into an image having a plurality of resolutions, and a multi-resolution image buffer stores a first image at a first timing converted to the plurality of resolutions. Then, the multi-resolution image buffer 32-2 accumulates the second image at the second timing immediately before the first timing converted into a plurality of resolutions, and the corresponding point search unit 33 A corresponding point between each pixel of the second image having the first resolution and each pixel of the first image having the first resolution is searched for, stored in the corresponding point motion vector buffer 34, and stored in the first resolution. Based on the corresponding point information, a corresponding point between each pixel of the second image having the second resolution higher than the first resolution and each pixel of the first image having the second resolution is searched, and sequentially repeat. The present invention can be applied to an imaging apparatus.
[Selection] Figure 4

Description

  The present invention relates to an image processing apparatus and method, a program, and a program storage medium, and in particular, an image processing apparatus and method, a program, and a program storage capable of stably and stably extracting a motion vector in units of pixels. It relates to the medium.

  There is block matching as means for obtaining the motion of an object in a moving image and means for obtaining a correspondence between images.

  For example, as shown in FIG. 1, when two images PA and PB having continuous reproduction timings (image PB is reproduced after image PA is reproduced) are given on image PA. A pixel p1 (or an image area) having a pixel that corresponds to (or to which image area) on the image PB is searched for in a block consisting of pixels near the pixel p1, and the pixel p2 at the corresponding position is searched. Is the process of identifying In FIG. 1, Sb indicates the number of pixels on one side of the matching block size Sb × Sb, and Ss indicates a range in which the matching block is moved, that is, a search range Ss × indicating the search range. The number of pixels on one side of Ss is shown.

  Further, as shown in FIG. 2, in particular, in the case where the frame images B2, B1, CF, F1, and F2 that are temporally continuous in the moving image are used, the above “correspondence” means the pixel p1 and the frame on the frame image CF. If v1 = p2-p1 is obtained based on the pixel p2 on the image F1, it can be expressed in the form of a two-dimensional motion vector v1 starting from the pixel p1. In FIG. 2, the frames are arranged in the time direction in the order of frames B2, B1, CF, F1, and F2, surrounded by solid lines, and squares with dotted lines in the horizontal and vertical directions indicate pixels. In particular, the intersection of dotted lines of each square indicates the pixel center.

  Block matching is used as an important basic technology in a wide range of image processing, and is applied to, for example, high dynamic range, noise reduction, camera shake prevention, frame insertion, and the like (see Patent Document 1).

JP 2002-044669 A

  By the way, recent image processing techniques tend to aim at further improvement of the compression ratio by reducing the block size and taking into consideration the structure and dynamic characteristics of the global object. Further, since the resolution of the image pickup device has increased, the amount of data has increased and it has become more difficult to reduce the processing time per frame. Here, consider the amount of calculation required for block matching in pixel units.

  That is, as shown in FIG. 1, an optical flow is obtained from two images, a reference image PA and a target image PB.

  Here, the image resolution is W × H = W × (Ar × W) (Ar is the aspect ratio) for both images PA and PB, and the matching block size is Sb × Sb = (Kb × W) × (Kb × W), the block movable range (search range) is Ss × Ss = (Ks × W) × (Ks × W), and Sb << MIN (W, H) and SS << MIN (W, H ), And the evaluation of matching between blocks in block matching is e (p1, p2) = Σ (dy is −Sb / 2 to Sb / 2) Σ (dx is −Sb / 2 to Sb / 2) Diff ( C1 (x1 + dx, y1 + dy), C (x2 + dx, y2 + dy)). MIN (W, H) represents a function that takes a smaller value of W or H.

  Further, C (x, y) represents a luminance value at the coordinates p = {x, y}. Diff (C, C ′) is a function representing the dissimilarity between the luminance values C and C ′, and for example, a square error is often used. Each subscript corresponds to a frame number.

  Further, Tc is the time (including data transfer) required for the matching calculation f (C, C ′) between the pixels.

  When performing a full search under these conditions, the calculation time T is T = W × H × Sb × Sb × Ss × Ss × Tc = Ar × Ks ^ 2 × Kb ^ 2 × W ^ 6 × Tc The amount of calculation is proportional to the sixth power of W.

  For example, even if the matching block size is fixed, the amount of calculation is proportional to the fourth power of W. In this case, when processing an HD (High Definition) image quality with 6 times the number of pixels of SD (Standard Definition) image quality, (√6) ^ 6 = 216 times ("^" indicates a power) ). Furthermore, recently, as the pixel size of the image sensor of the camera is reduced, the sensitivity is relatively low, and dark noise tends to increase due to the influence. Therefore, in an image taken in a dark environment, there is a high possibility that block matching breaks down and the robustness is lowered. In addition, when trying to obtain an image with higher frame rate and higher dynamic characteristics, the amount of light per frame decreases, inevitably dark noise increases, further reducing robustness, and the amount of data further increases. This further increases the processing time. As a result, reducing processing time is becoming more difficult.

  Therefore, in order to solve these problems, in the past, dedicated calculation hardware was used, calculation with a limited search range (Lucas-Kanade method or use of epipolar), or some of the above methods. After calculating only the representative points, the necessary part is interpolated, the input image is calculated with multiple resolutions, the matching is calculated after the input image has been made sufficiently high in quality, and the image sensor Attempts have been made to try to solve the problem by using any one of the methods such as improving or by combining a plurality of methods.

  As one of these trials, there has been proposed a method using an image group in which resolution is gradually lowered from a given image, so-called image pyramid. The image pyramid is an image group having a plurality of resolutions generated by repeating the process of reducing the resolution of an input image. When reducing the resolution of an input image, a low-pass filter such as an averaging filter that averages multiple pixels is used, and processing starts from an image at a low resolution level, and gradually switches to an image at a high resolution level. We will improve the calculation of correspondence between the two. By using this method, the matching block size and the search range can be significantly reduced, and as a result, the processing speed can be increased. For example, consider an image pyramid composed of seven types of images by reducing the resolution of a 1280 × 960 image by a multiple of 1/4.

  That is, at the resolution level (0), the resolution is 1280 × 960, the search range Ss × Ss corresponding to the resolution level (0) is 196 × 196, and the matching block size Sb × Sb corresponding to the resolution level (0) is 196 × 196. Then, the number of inspections is 1280 × 960 × 196 ^ 4.

  On the other hand, at the resolution level (0), the resolution is 640 × 480, the search range Ss × Ss corresponding to the resolution level (0) is 96 × 96, and the matching block size Sb × Sb corresponding to the resolution level (0) is Since 96 × 96, the number of inspections is 1/2 ^ 4 times the resolution level (0).

  The resolution level (2) is 320 × 240, the search range Ss × Ss corresponding to the resolution level (0) is 48 × 48, and the matching block size Sb × Sb corresponding to the resolution level (0) is 48 × 48. Therefore, the number of inspections is 1/2 ^ 8 times the resolution level (0).

  Furthermore, at the resolution level (3), the resolution is 160 × 120, the search range Ss × Ss corresponding to the resolution level (0) is 24 × 24, and the matching block size Sb × Sb corresponding to the resolution level (0) is 24 × 24. Therefore, the number of inspections is 1/2 ^ 12 times the resolution level (0).

  Similarly, at the resolution level (4), the resolution is 80 × 60, the search range Ss × Ss corresponding to the resolution level (0) is 12 × 12, and the matching block size Sb × Sb corresponding to the resolution level (0) is Since it is 12 × 12, the number of inspections is 1/2 ^ 16 times the resolution level (0).

  Similarly, at the resolution level (0), the resolution is 40 × 30, the search range Ss × Ss corresponding to the resolution level (0) is 6 × 6, and the matching block size Sb × Sb corresponding to the resolution level (0) is Since 6 × 6, the number of inspections is 1/2 ^ 20 times the resolution level (0).

  Similarly, at the resolution level (6), the resolution is 20 × 15, the search range Ss × Ss corresponding to the resolution level (0) is 3 × 3, and the matching block size Sb × Sb corresponding to the resolution level (0) is Since 3 × 3, the number of inspections is 1/2 ^ 24 times (= 20 × 15 × 3 ^ 4) at the resolution level (0).

  At this time, even if a large Ss and Sb are set at the resolution level (0), relatively small W, H, Ss, and Sb are sufficient at the resolution level (6). It becomes possible. More specifically, for example, even when Ss = Sb = 196, processing using a matching block of at most 3 × 3 pixels and at most 3 × 3 pixels at resolution level (6) is at most 20 × 15 points. You can check with.

  In addition, since it starts with the outline of the object structure at low resolution and gradually refines, the relationship structure between the motion vectors is not destroyed, and the two-dimensional object parts in each image are structured dynamically. It is possible to determine the correspondence in the sense that they are the same part above.

  However, if the response obtained at the lower level cannot be appropriately reflected on the upper level, the accuracy may not be stably improved.

  In any application, it is indispensable to execute block matching after sufficiently improving the image quality in order to cope with captured images under bad conditions such as low illuminance. However, block matching is performed in order to improve the image quality itself, and if the image quality improvement is performed in advance in order to perform block matching, it may be overturned. In addition, improvement of a simple image pickup device requires enormous costs and may not be able to deal with an already existing image.

  As a result, there is no robust method established by any method, and there is a risk that the cost is increased.

  The present invention has been made in view of such a situation, and in particular, enables motion vectors to be detected at high speed and with high accuracy while maintaining noise resistance.

  An image processing apparatus according to an aspect of the present invention includes a multi-resolution unit that converts an input image into an image having a plurality of resolutions, and a first image that is converted into a plurality of resolutions by the multi-resolution unit. And a second storage unit for storing a second image at a second timing immediately before the first timing converted to a plurality of resolutions by the multi-resolution unit. Search by means of storage means, corresponding point search means for searching for corresponding points between the pixels of the second image at the first resolution and the pixels of the first image at the first resolution, and the corresponding point searching means Storage means for storing information of corresponding points between the pixels of the second image at the first resolution and the pixels of the first image at the first resolution, and the corresponding point search means The storage hand Based on the corresponding point information stored in the first resolution, the pixels of the second image at the second resolution higher than the first resolution and the first image at the second resolution. Search for corresponding points with the pixels of the image and repeat sequentially.

  The corresponding point search means includes an evaluation value calculation means for obtaining an evaluation value of a corresponding point between the pixel of the second image at the first resolution and the pixel of the first image at the first resolution by block matching. Can be further included, and a pixel having the smallest evaluation value calculated by the evaluation value calculating means can be searched as a corresponding point.

  The evaluation value calculation means uses each of the pixels of the second image of the first resolution and the first resolution of the first resolution using weighted evaluation according to the distance from the target pixel in the block matching block. An evaluation value of a corresponding point with each pixel of one image can be obtained.

  The evaluation value calculation means uses each pixel of the second image of the first resolution and the first image by using a weighted evaluation according to a difference in pixel value between corresponding pixels between the blocks of the block matching. The evaluation value of the corresponding point with each pixel of the first image of resolution can be obtained.

  Fitting calculation means for calculating five evaluation values for five pixels in the vicinity of corresponding points in an image of a predetermined resolution and obtaining the corresponding points by using an approximate function coefficient based on the five evaluation values. Further, it can be included.

  Fitting calculation means for calculating nine evaluation values for nine pixels in the vicinity of corresponding points in an image of a predetermined resolution and obtaining the corresponding points by using coefficients of an approximation function based on the nine evaluation values. Further, it can be included.

  The multi-resolution processing unit may convert the input image into a plurality of resolution images having a resolution lower than that of the input image or a plurality of resolution images having a resolution higher than that of the input image. it can.

  An image processing method according to an aspect of the present invention includes a multi-resolution step of converting an input image into an image having a plurality of resolutions, and a first timing converted to a plurality of resolutions by the processing of the multi-resolution step. A first accumulation step for accumulating the first image, and a second image at a second timing immediately before the first timing converted to a plurality of resolutions by the processing of the multi-resolution step. A second accumulation step, a corresponding point search step of searching for a corresponding point between the pixel of the second image at the first resolution and the pixel of the first image at the first resolution, and the corresponding point A storage for storing information of corresponding points between the pixel of the second image at the first resolution and the pixel of the first image at the first resolution searched by the processing of the search step The corresponding point search step includes a second resolution higher than the first resolution based on the corresponding point information stored in the storage step. Corresponding points between the pixels of the second image at the resolution and the pixels of the first image at the second resolution are searched and repeated sequentially.

  A program according to one aspect of the present invention includes a multiresolution step of converting an input image into an image of a plurality of resolutions, and a first image at a first timing converted into a plurality of resolutions by the processing of the multiresolution step. And a second image at a second timing immediately before the first timing converted to a plurality of resolutions by the processing of the multi-resolution step. 2, a corresponding point search step for searching for a corresponding point between the pixel of the second image at the first resolution and the pixel of the first image at the first resolution, and the corresponding point searching step A storage space for storing information on corresponding points of the pixel of the second image at the first resolution and the pixel of the first image at the first resolution searched by the processing of The corresponding point search step processing includes a second resolution higher than the first resolution based on the corresponding point information at the first resolution stored in the storage step processing. The corresponding points of the pixels of the second image at the resolution of the first image and the pixels of the first image at the second resolution are searched and repeated sequentially.

  A program storage medium according to one aspect of the present invention stores the program according to claim 9.

  The image processing apparatus of the present invention may be an independent apparatus or a block that performs image processing.

  According to one aspect of the present invention, a motion vector can be extracted in a pixel unit at high speed and stably.

  Embodiments of the present invention will be described below. The correspondence relationship between the invention described in this specification and the embodiments of the invention is exemplified as follows. This description is intended to confirm that the embodiments supporting the invention described in this specification are described in this specification. Therefore, although there is an embodiment which is described in the embodiment of the invention but is not described here as corresponding to the invention, it means that the embodiment is not It does not mean that it does not correspond to the invention. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in this specification. In other words, this description is for the invention described in the present specification, which is not claimed in this application, that is, for the invention that will be applied for in the future or that will appear and be added by amendment. It does not deny existence.

  That is, an image processing apparatus according to an aspect of the present invention includes a multi-resolution converting unit (for example, the multi-resolution converting unit 31 in FIG. 4) that converts an input image into a plurality of resolution images, and a plurality of resolutions by the multi-resolution converting unit. The first storage means (for example, the multi-resolution image buffer 32-1 in FIG. 4) for storing the first image of the first timing converted into the resolution is converted into a plurality of resolutions by the multi-resolution converting means. A second accumulator (for example, the multi-resolution image buffer 32-2 in FIG. 4) that accumulates the second image at the second timing immediately before the first timing, and the first resolution. Corresponding point search means (for example, corresponding point search unit 33 in FIG. 4) for searching for corresponding points between the pixels of the second image in the first image and the pixels of the first image at the first resolution, and the corresponding points Search by search means Storage means for storing information on corresponding points between the pixels of the second image at the first resolution and the pixels of the first image at the first resolution (for example, corresponding point motion vectors in FIG. 4) The corresponding point search means based on the corresponding point information at the first resolution stored in the storage means at a second resolution higher than the first resolution. Corresponding points between the pixels of the second image and the pixels of the first image at the second resolution are searched and repeated sequentially.

  The corresponding point search means includes an evaluation value calculation means for obtaining an evaluation value of a corresponding point between the pixel of the second image at the first resolution and the pixel of the first image at the first resolution by block matching. (For example, the evaluation calculation unit 33c in FIG. 4) can be further included, and a pixel having the smallest evaluation value calculated by the evaluation value calculation means can be searched as a corresponding point. .

  The evaluation value calculation means (for example, the evaluation calculation unit 33c in FIG. 4) uses the weighted evaluation according to the distance from the target pixel in the block of the block matching to use the second image having the first resolution. The evaluation value of the corresponding point between each pixel and each pixel of the first image of the first resolution can be obtained.

  The evaluation value calculation means (for example, the evaluation calculation unit 33c in FIG. 4) uses the weighted evaluation according to the difference in pixel value between corresponding pixels between the blocks of the block matching, and the first resolution of the first resolution. The evaluation value of the corresponding point between each pixel of the second image and each pixel of the first image having the first resolution can be obtained.

  Fitting calculation means for calculating five evaluation values for five pixels in the vicinity of corresponding points in an image of a predetermined resolution, and obtaining the corresponding points using coefficients of an approximation function based on the five evaluation values. For example, the fitting calculation unit 61) of FIG. 14 can be further included.

  Nine evaluation values for nine pixels in the vicinity of corresponding points in an image of a predetermined resolution are calculated, and fitting calculation means for calculating the corresponding points using coefficients of an approximation function based on the nine evaluation values ( For example, the fitting calculation unit 61) of FIG. 14 can be further included.

  In the multi-resolution unit (for example, the multi-resolution unit 31 in FIG. 4), the input image is converted into a plurality of resolution images having a lower resolution than the input image (for example, the multi-resolution unit 31 in FIG. 4). The resolution reduction filter 31a) or an image having a plurality of resolutions higher than the input image can be converted (for example, the resolution enhancement filter 31b of the multi-resolutionization unit 31 in FIG. 4).

  An image processing method and a program according to an aspect of the present invention include a multi-resolution step (for example, steps S22 and S23 in FIG. 6) for converting an input image into an image having a plurality of resolutions, and a plurality of processing steps according to the multi-resolution step. A first accumulation step (for example, steps S22 and S23 in FIG. 6) for accumulating the first image at the first timing converted to the resolution of the first resolution and the multi-resolution conversion process are performed to convert the first image into a plurality of resolutions. A second accumulation step (for example, step S21 in FIG. 6) for accumulating the second image at the second timing immediately before the first timing, and the second resolution at the first resolution. A corresponding point search step of searching for a corresponding point between the pixel of the image and the pixel of the first image at the first resolution; and the corresponding point searching step (for example, FIG. 6). A storing step (for example, storing information of corresponding points between the pixel of the second image at the first resolution and the pixel of the first image at the first resolution searched by the processing of step S25) Step S26) in FIG. 6 is performed, and the processing of the corresponding point search step is higher than the first resolution based on the information of the corresponding point in the first resolution stored in the processing of the storage step. Corresponding points between the pixels of the second image at the second resolution and the pixels of the first image at the second resolution are searched for and repeated sequentially.

  With reference to FIG. 3, the configuration of an embodiment of a camera shake correction processing apparatus to which the present invention is applied will be described.

  The camera shake correction apparatus 1 in FIG. 3 corrects image flicker caused by camera shake according to a motion vector when an image is captured.

  The image buffer 11 temporarily stores the captured image and supplies it to the motion vector detection unit 12 and the camera shake correction unit 14 as appropriate.

  The motion vector detection unit 12 detects a motion vector for each pixel of the image stored in the image buffer 11 and stores it in the motion vector buffer 13.

  The camera shake correction unit 14 reads the image stored in the image buffer 11, corrects the camera shake according to the motion vector stored in the motion vector buffer 13, and outputs it.

  Next, a configuration example of the motion vector detection unit 12 will be described with reference to FIG.

  The multi-resolution unit 31 converts the image stored in the image buffer 11 into an image having a plurality of resolutions by using the low-resolution filter 31a and the high-resolution filter 31b, and supplies the image to the multi-resolution image buffer 32-1. The resolution reduction filter 31a repeats a process for reducing the resolution a predetermined number of times, for example, by averaging pixels adjacent in the horizontal direction and the vertical direction, and sequentially converts the input image into a reduced resolution image. Further, the high resolution filter 31b repeats a process for increasing the resolution a predetermined number of times by interpolating pixels between adjacent pixels in the horizontal direction and the vertical direction, and sequentially converts the input image into a higher resolution image. .

  The multi-resolution image buffer 32-1 temporarily stores an image obtained by converting an input image into an image having a plurality of resolutions including the input image, and stores the plurality of resolutions when the next input image is input. Are transferred to the multi-resolution image buffer 32-2. That is, the multi-resolution image buffers 32-1 and 32-2 temporarily store a plurality of multi-resolution images for two temporally continuous input images. When a multi-resolution image of a new input image is supplied to the multi-resolution image buffers 32-1 and 32-2, the multi-resolution image buffer 32-1, 2-2 erases the image stored so far and multi-resolution of the new input image. The converted image is stored.

  The corresponding point search unit 33 searches for corresponding points for each pixel from the lowest resolution images stored in the multi-resolution image buffers 32-1 and 32-2. That is, the corresponding point search unit 33 uses the image stored in the multi-resolution image buffer 32-1 as the corresponding point that is the end point of the motion vector in the pixel of the previous image stored in the multi-resolution image buffer 32-2. Is determined by searching within the determined search range and stored in the corresponding point motion vector buffer 34.

  The matching block setting unit 33a extracts the pixels constituting the matching block corresponding to the target pixel of the immediately preceding image for the lowest resolution image among the multi-resolution images, and for each pixel in the current image, Pixels constituting a matching block that is a search range are extracted.

  Further, the matching block setting unit 33a, for an image other than the lowest resolution among the multi-resolution images, for the pixel of interest in the image immediately before the predetermined resolution, than the predetermined resolution stored in the initial value motion vector buffer 36. Based on the information on the corresponding point (end point of the motion vector) in the image whose resolution level is one level lower, the matching block is configured with the pixels in the search range on the basis of the initial corresponding point determined by the initial corresponding point determination unit 33e. The pixel, that is, the pixel of interest in the previous image and its neighboring pixels are extracted, and the matching block is moved within the search range on the current image set in correspondence with the pixel of interest in the search range setting unit 33b Thus, pixels constituting the matching block are sequentially extracted.

  The evaluation calculation unit 33c calculates an evaluation value based on the pixels constituting each matching block of the immediately preceding image supplied from the matching block setting unit 33a and the current image.

  The corresponding point determination unit 33d is the pixel of the current image having the smallest evaluation value among the evaluation values calculated by the evaluation calculation unit 33c, that is, the most similar between the immediately preceding image and the current image. Are stored in the corresponding point motion vector buffer 34 as corresponding points of the target pixel (end points of the target pixel's motion vector). Also, the corresponding point determination unit 33d supplies information on corresponding points in the image with the highest resolution (or an image with a predetermined resolution for which a motion vector is to be obtained) to the motion vector generation unit 37.

  When the corresponding point information is newly accumulated in the corresponding point motion vector buffer 34, the motion vector output determination unit 35 transfers the information to the initial value motion vector buffer 36.

  The motion vector generation unit 37 is based on information on corresponding points of the current image corresponding to each pixel in the immediately preceding image in the highest resolution image (or an image of a predetermined resolution for which a motion vector is to be obtained). The motion vector is obtained and sequentially stored in the motion vector buffer 13.

  Next, camera shake correction processing will be described with reference to the flowchart of FIG.

  In step S1, the image buffer 11 buffers the input image.

  In step S <b> 2, the motion vector detection unit 12 reads an image stored in the image buffer 11, executes a motion vector detection process, detects a motion vector for each pixel of the input image, and stores the motion vector in the motion vector buffer 13. Store.

  Here, the motion vector detection process will be described with reference to the flowchart of FIG.

  In step S <b> 21, the multi-resolution converting unit 31 controls the multi-resolution image buffer 32-1 to transfer images having a plurality of resolutions stored so far to the multi-resolution image buffer 32-2. At this time, the multi-resolution image buffer 32-2 stores the newly supplied multi-resolution image as well as deleting the multi-resolution image stored so far.

  In step S22, the multi-resolution converting unit 31 controls the low-resolution filter 31a to sequentially convert the input image into an image with a reduced resolution and store the converted image in the multi-resolution image buffer 32-1. That is, the low resolution filter 31a converts an input image (resolution level (0) image = Level 0 image) in units of 3 × 3 pixels in the horizontal and vertical directions as shown in FIG. The pixel value is converted into one pixel by a filtering process as shown in the following expression (1), thereby converting the pixel value into an image (Level 1 image) having a resolution level (1) lower than the input image. FIG. 7 shows images of resolution levels (0), (1), and (2) (resolutions of Levels 0, 1, and 2) in which the resolution is reduced in order from the bottom, respectively, in the horizontal direction or An arrangement state in a one-dimensional direction in the vertical direction is shown. Moreover, the cross in the figure indicates the pixel center of the pixel p, the vertical bar is the pixel boundary, and each number is a number for identifying the pixel p. Accordingly, in FIG. 7, pixels p0 to p15 are displayed for the resolution level (0) image (Level 0 image), and for the resolution level (1) image (Level 1 image), pixels p0 to p15 are displayed. p8 is displayed, and pixels p0 to p4 are displayed for an image of resolution level (2) (an image of Level 2).

  Equation (1) shows an example of a Gaussian filter.

  Further, the pixel p3 in the image at the resolution level (1) is a pixel value obtained by filtering the pixel values of the pixels p5 to p7 in the image at the resolution level (0), and similarly, in the image at the resolution level (1). The pixel p4 is a pixel value obtained by filtering the pixel values of the pixels p7 to p9 in the resolution level (0) image, and the pixel p5 in the resolution level (1) image is the pixel p9 in the resolution level (0) image. To the pixel value obtained by filtering the pixel value of p11.

  Similarly, the pixel p2 in the resolution level (2) image is a pixel value obtained by filtering the pixel values of the pixels p3 to p5 in the resolution level (1) image.

  In FIG. 7, since a one-dimensional arrangement viewed from the horizontal direction or the vertical direction is expressed, when an image having a resolution that is one level lower is generated, a value obtained by filtering three pixels is obtained. However, in actuality, the value obtained by filtering the 3 × 3 pixel centered on the target pixel is a pixel value having a lower resolution by one level. It will be required. Furthermore, the position of the pixel center of the pixel p0 is geometrically the same in the horizontal and vertical directions, and the pixel center at the right end of each level is one level higher, as shown by the right end in the figure. They are arranged so as to coincide with the pixel boundaries at the edges in the image.

  In step S23, the multi-resolution processing unit 31 controls the high-resolution filter 31b to sequentially convert the input images into high-resolution images and store them in the multi-resolution image buffer 32-2. That is, the high resolution filter 31b converts the input image (resolution level (0) image = level 0 image in the figure) in units of 4 × 4 pixels in the horizontal direction and the vertical direction as shown in FIG. Then, the pixel value is converted into one pixel by bicubic filter processing, thereby converting the pixel value into an image having a resolution level (-1) higher than the input image (Level-1 image). FIG. 8 shows images of resolution levels (0), (−1), and (−2) in which resolution is increased in order from the top (= level 0, −1, and −2 images). An arrangement state of a direction or a one-dimensional direction in the vertical direction is shown. Therefore, in FIG. 8, pixels p0 to p3 are displayed for the image of resolution level (0), and pixels p0 to p7 are displayed for the image of resolution level (−1), and the resolution level (− Pixels p0 to p15 are displayed for the image 2).

  The pixel p2 in the resolution level (−1) image is a pixel value obtained by performing bicubic filter processing on the pixel values of the pixels p0 to p3 in the resolution level (0) image.

  Similarly, the pixel p6 in the image at the resolution level (−2) is a pixel value obtained by performing bicubic filter processing on the pixel values of the pixels p2 to p5 in the image at the resolution level (−1).

  In FIG. 8, since a one-dimensional arrangement viewed from the horizontal direction or the vertical direction is expressed, when an image having a resolution higher by one level (Level) is generated, the filter processing result of four pixels is Although expressed as one pixel having a high resolution, the value obtained by bicubic filter processing for 4 × 4 pixels existing at a predetermined position with respect to the target pixel is actually one level. It is obtained as a pixel value with a high resolution. When the resolution is increased, the pixel boundaries at the upper, lower, left, and right ends are converted so as to coincide.

  In step S24, the corresponding point search unit 33 executes a minimum resolution corresponding point extraction process to extract a corresponding point (end point of the motion vector of each pixel) for each pixel of the image immediately before the minimum resolution.

  Here, the minimum resolution corresponding point extraction processing will be described with reference to the flowchart of FIG.

  In step S41, the matching block setting unit 33a sets an unprocessed pixel as the target pixel pb among the lowest resolution images stored in the multi-resolution image buffer 32-2, and the target pixel pb. A pixel group constituting a matching block corresponding to the pixel pb is extracted and supplied to the evaluation calculation unit 33c. The size of the matching block set by the matching block setting unit 33a is, for example, Sb [k] × Sb [k]. Here, Sb [k] is expressed by the following equation (2).

  In Expression (2), MAX (A, B) indicates that A or B takes a larger value. In addition, INT (A) indicates that the value A is rounded down. Further, k represents a resolution level (k) (Level), and k is an integer satisfying le ≦ k ≦ L. That is, the resolution level (L) image is the lowest resolution image, and the resolution level (le) image is the highest resolution image. Therefore, Sb [0] is the matching block size for the input image (image of resolution level (0)).

  In step S42, the search range setting unit 33b stores in the multiresolution image buffer 32-1 corresponding to the pixel of interest pb on the lowest resolution image (previous image) stored in the multiresolution image buffer 32-2. An unprocessed search range on the lowest resolution image (current image) being set is set. Here, the search range size is, for example, Ss [k] × Ss [k]. Here, Ss [k] is expressed by the following equation (3).

  Therefore, Ss [0] is the search range size for the input image (image of resolution level (0)).

  In step S43, the search range setting unit 33b sets an unprocessed corresponding point pixel pf in the search range on the current image.

  In step S44, the matching block setting unit 33a extracts a pixel group constituting the lowest resolution image (current image) matching block stored in the multiresolution image buffer 32-1 corresponding to the corresponding point pixel pf. To the evaluation calculation unit 33c.

  In step S45, the evaluation calculation unit 33c supplies a pixel group constituting the matching block corresponding to the target pixel pb on the immediately preceding lowest resolution image supplied from the matching block setting unit 33a, and the current lowest resolution image. The following equation (4) is calculated using the pixel group that constitutes the matching block corresponding to the upper corresponding point pixel pf, thereby obtaining an evaluation value e (Pb, Pf) between the target pixel pb and the corresponding point pixel pf. ).

  Here, ROI indicates the range of the matching block. Further, d represents the distance from each pixel constituting the matching block from the target pixel pb or the corresponding point pixel pf. Further, Cb () and Cf () respectively indicate the luminances of the pixels constituting the matching block centered on the target pixel pb or the corresponding point pixel pf. Further, the function g (x, σ) in the equation (4) is a function represented by the equation (5), and as shown in FIG. 10, the intensity is large near 0 with respect to the variable x and deviates from 0. It is a so-called Gaussian function that decreases rapidly according to the above. Therefore, g (x, σd) is a Gaussian function related to the distance between pixels, and g (x, σI) is a Gaussian function related to the luminance difference between pixels.

  That is, equation (4) is based on a method of measuring dissimilarity by square error, and by using equation (5), the distance from the center pixel of the matching block is far and the luminance difference is large. It works to lower the evaluation contribution. That is, the maximum difference is emphasized between the pixels (= the target pixel pb or the corresponding point pixel pf) existing at the center position of the matching block, and the difference between colors that is too large is ignored. These conditions can be adjusted by σd and σI.

  As a result, the evaluation value e calculated by Equation (4) is 0, which is the smallest evaluation value between completely matching patterns, and some image deformation (enlargement, reduction, rotation, distortion) between matching blocks to be compared. ) Is easy to match between pixels, and noise, edges, or different background patterns can be almost ignored.

  In step S46, the search range setting unit 33b determines whether or not the evaluation value e with the target pixel pb has been calculated for all corresponding point pixels pf in the search range. If it is determined in step S46 that the evaluation value e with the target pixel pb has not been calculated for all corresponding point pixels pf, the process returns to step S43. That is, for all the corresponding point pixels pf in the search range, the processes of steps S43 to S46 are repeated until the evaluation value e with the target pixel pb is calculated.

  If it is determined in step S46 that the evaluation value e with the target pixel pb has been calculated for all corresponding point pixels pf within the search range, the process proceeds to step S47.

  In step S47, the search range setting unit 33b determines whether or not there is an unprocessed search range for the pixel of interest pb. If there is an unprocessed search range, the process returns to step S42, and the unprocessed search range is processed. Until there is no more search range, steps S42 to S47 are repeated.

  If it is determined in step S47 that there is no unprocessed search range, in step S48, the corresponding point determination unit 33d searches for a combination of the target pixel pb and the corresponding point pixel pf that minimizes the evaluation value e. The corresponding point pixel pf is stored in the corresponding point motion vector buffer 34 as the corresponding point of the target pixel pb, that is, the motion vector VL = P [L] (pb).

  In step S49, the matching block setting unit 33a determines whether or not there is an unprocessed target pixel. If there is an unprocessed target pixel, the process returns to step S41. That is, the processing in steps S41 to S49 is repeated until the corresponding points are obtained for all the pixels with the lowest resolution. When the corresponding points are obtained for all the pixels of the immediately preceding lowest resolution image in step S49, That is, when motion vectors are obtained for all pixels of the immediately preceding lowest resolution image, the process ends. When the processing is completed, the motion vector output determination unit 35 transfers the information of the corresponding point motion vector buffer 34 to the initial value motion vector buffer 36 when the resolution level is not the highest resolution.

  With the above processing, each pixel in the immediately preceding image is sequentially set as a target pixel, a search range is set so as to cover the entire image, and block matching is repeated while sequentially moving the matching blocks, so that the lowest resolution is achieved. For each pixel of the image, the corresponding point, that is, the end point of the motion vector when the start point is each pixel, is obtained and stored in the corresponding point motion vector buffer 34. At this time, even if the search range is sequentially set for the entire image, the resolution is sufficiently low, and the corresponding points are obtained at high speed.

  Now, the description returns to the flowchart of FIG.

  In step S24, when the corresponding point of each pixel in the image with the lowest resolution is obtained by the lowest resolution corresponding point extraction process, in step S25, the corresponding point search unit 33 sequentially executes the corresponding point extraction process to obtain the image with the lowest resolution. Using the corresponding points, the corresponding points in the high-resolution image are searched sequentially.

  Here, the corresponding point extraction processing will be described sequentially with reference to the flowchart of FIG.

  In step S61, the matching block setting unit 33a sets a counter k (not shown) indicating the resolution level (k) to (L-1). Note that k is an integer of le ≦ k ≦ L as described above.

  In step S62, the matching block setting unit 33a sets an unprocessed pixel as the target pixel pb among the images of the resolution level (k) in the previous image stored in the multi-resolution image buffer 32-2. Then, a pixel group constituting a matching block corresponding to the target pixel pb is extracted and supplied to the evaluation calculation unit 33c. The size of the matching block set by the matching block setting unit 33a is, for example, Sb [k] × Sb [k]. Here, Sb [k] is expressed by the following equation (6).

  In step S63, the initial corresponding point determination unit 33e uses the information on the corresponding points in the image at the resolution level (k + 1), and the initial corresponding point Q [k] corresponding to the target pixel pb at the resolution level (k). (pb) is set, and information about the set initial corresponding point Q [k] (pb) is supplied to the search range setting unit 33b.

  More specifically, as shown in FIG. 12, the initial corresponding point determination unit 33e determines the coordinates (x [k], y [k]) of the target pixel pb at the resolution level (k) as the resolution level (k +). By converting to the coordinates (x [k + 1], y [k + 1]) in the image space of 1), the coordinates of the pixel of interest pb ′ on the image of the corresponding resolution level (k + 1) (x [ k + 1], y [k + 1]). In FIG. 12, the pixel p1 on the image at the resolution level (k) is set as the target pixel pb, and the pixel on the image at the resolution level (k + 1) corresponding to the target pixel pb is the pixel pb ′. It is expressed by coordinates on the image of the resolution level (k + 1).

  Next, in the initial value motion vector buffer 36, since the corresponding point is obtained for each pixel on the image of the resolution level (k + 1), the initial corresponding point determination unit 33e exists in the vicinity of the pixel pb ′. Then, the approximate corresponding point P [k + 1] (pb ′) is obtained by the 4-neighbor bilinear interpolation process using the corresponding points of the four pixels on the image of the resolution level (k + 1). That is, in FIG. 12, only the one-dimensional arrangement in the horizontal direction or the vertical direction is shown, so that only the pixels p0 and p1 are shown as the pixels in the vicinity of the pixel pb ′. Two pixels exist in the vicinity. Here, assuming that there are pixels pA and pB (not shown) in addition to the pixels p0 and p1 as pixels in the vicinity of the pixel pb ′, the initial corresponding point determination unit 33e selects the corresponding points P [k +1] (p0), P [k + 1] (p1), P [k + 1] (pA), P [k + 1] (pB) and approximate corresponding points by 4-neighbor bilinear interpolation P [k + 1] (pb ′) (= P [k + 1] (p0) · {1−α} {1−β} + P [k + 1] (p1) · α {1−β} + P [k + 1] (pA) · {1−α} β + P [k + 1] (pB) · αβ (α (0 ≦ α ≦ 1) is an approximate corresponding point P [ The spatial ratio β + 1 (0 ≦ β ≦ 1) where k + 1] (pb ′) is located is the spatial ratio where the approximate corresponding point P [k + 1] (pb ′) between the pixels pA and pB is located. The right ratio)). Further, the initial corresponding point determination unit 33e changes the coordinates of the approximate corresponding point P [k + 1] (pb ′) from the coordinates in the image at the resolution level (k + 1) to the coordinates in the image at the resolution level (k). Is converted into the initial corresponding point Q [k] (pb).

  The coordinate conversion between the resolution levels (k) and (k + 1) is converted as shown in the following equation (7) when the counter k is larger than 0, that is, when the resolution is lower than the input image. When the counter k is smaller than 0, it is converted as shown in the following equation (8).

  In step S64, the search range setting unit 33b multi-resolution image buffer 32-1 corresponding to the pixel of interest pb on the resolution level (k) image (previous image) stored in the multi-resolution image buffer 32-2. The search range (for example, the initial corresponding point Q [k] (pb) corresponding to the initial corresponding point Q [k] (pb) on the image (current image) of the resolution level (k) stored in Set the search range as the center. Here, the search range size is, for example, Ss [k] × Ss [k]. Here, Ss [k] is expressed by the following equation (9).

  In step S65, the search range setting unit 33b sets the unprocessed corresponding point pixel pf in the search range on the current image.

  In step S <b> 66, the matching block setting unit 33 a constitutes a matching block in the image (current image) of the resolution level (k) stored in the multi-resolution image buffer 32-1 corresponding to the corresponding point pixel pf. A group is extracted and supplied to the evaluation calculator 33c.

  In step S67, the evaluation calculation unit 33c supplies the pixel group constituting the matching block corresponding to the pixel of interest pb on the image at the previous resolution level (k) supplied from the matching block setting unit 33a, and the current resolution. Evaluation of the pixel of interest pb and the corresponding point pixel pf is performed by calculating the above equation (4) using the pixel group constituting the matching block corresponding to the corresponding point pixel pf on the level (k) image. Calculate the value e (Pb, Pf).

  In step S68, the search range setting unit 33b determines whether or not the evaluation value e with the target pixel pb has been calculated for all the corresponding point pixels pf in the search range. If it is determined in step S68 that the evaluation value e with the target pixel pb has not been calculated for all corresponding point pixels pf, the process returns to step S66. That is, the processing of steps S66 to S68 is repeated until the evaluation value e with the target pixel pb is calculated for all corresponding point pixels pf within the search range.

  If it is determined in step S68 that the evaluation value e with the target pixel pb has been calculated for all corresponding point pixels pf within the search range, the process proceeds to step S69.

  In step S69, the corresponding point determination unit 33d searches for a combination of the target pixel pb and the corresponding point pixel pf that minimizes the evaluation value e, and uses the corresponding point pixel pf as a corresponding point of the target pixel pb, that is, a motion vector. The corresponding point motion vector buffer 34 is stored as Vk = P [k] (pb).

  In step S70, the matching block setting unit 33a determines whether or not there is an unprocessed target pixel. If there is an unprocessed target pixel, the process returns to step S62. That is, the processes in steps S62 to S70 are repeated for all the pixels of the image at the resolution level (k) until the corresponding points are obtained. In step S70, all the pixels of the image at the immediately previous resolution level (k) are determined. When the corresponding point is obtained, that is, when the motion vector is obtained for all the pixels of the image having the immediately previous resolution level (k), the process proceeds to step S71.

  In step S71, the matching block setting unit 33a determines whether or not a counter k (not shown) indicating the resolution level is the minimum value, and is determined not to be the minimum value, that is, there is an image with a higher resolution. In step S72, the counter k is decremented by 1, and the process returns to step S62, and the subsequent processes are repeated. That is, by using the corresponding point information obtained at the resolution levels (k + 1) from the resolution levels (L-1) to (le), each of the images in the resolution level k that is one level higher is displayed. The process for obtaining the corresponding points of the pixels is sequentially repeated. At this time, the motion vector output determination unit 35 transfers the information in the corresponding point motion vector buffer 34 to the initial value motion vector buffer 36.

  On the other hand, when it is determined in step S71 that the counter k is the minimum value, that is, when the corresponding point of each pixel in the image that is one level lower than the highest resolution level is obtained, the process ends.

  Through the above processing, each pixel in the immediately preceding image is sequentially set as a pixel of interest from a low resolution image with a high resolution level, and only the initial corresponding point set based on the corresponding point in the immediately preceding resolution level image. The search range is set, and the block matching is repeated while sequentially moving the matching blocks, so that for each pixel of the image of each resolution level, the corresponding point, that is, the end point of the motion vector when the starting point is each pixel is obtained. And stored in the corresponding point motion vector buffer 34. Then, this process is repeated up to the image of the resolution level (le) of the highest resolution 1 level and lower resolution.

  As a result, since it is possible to make the search range only near the initial corresponding point, it is only necessary to set the search range once for each pixel of the image of each resolution level. It can be reduced. As a result, it is possible to reduce the number of evaluation value calculations necessary for searching for corresponding points. Furthermore, although the same processing is repeated at different resolution levels, it is only necessary to set the search range once regardless of the resolution level. Therefore, it is the end point of the motion vector in the image of the desired resolution level. It becomes possible to search for corresponding points at high speed and stably.

  Now, the description returns to the flowchart of FIG.

  In step S25, the corresponding point P [le] (p) of each pixel in the image with the highest resolution level (le) is obtained by sequential corresponding point extraction processing.

  In step S26, the motion vector generation unit 37 finally generates a motion vector by the motion vector generation process using the corresponding point in each pixel of the resolution level (le), and stores it in the motion vector buffer 13, The process ends.

  Here, the motion vector generation processing will be described with reference to the flowchart of FIG.

  In step S91, the motion vector generation unit 37 sets the target pixel p from the unprocessed pixel on the image of the resolution level (le) that is the highest resolution.

  In step S92, the motion vector generation unit 37 reads the corresponding point P [le] (p) of the pixel of interest p stored in the corresponding point motion vector buffer 34. In step S93, the corresponding motion vector is the end point of the motion vector. The motion vector V (p) is generated by calculating the following equation (10) using the pixel of interest that is the starting point from the point.

  In step S94, the motion vector generation unit 37 stores the generated motion vector V (p) at the target pixel p in the motion vector buffer 13.

  In step S95, the motion vector generation unit 37 determines whether or not there is an unprocessed pixel. If it is determined that there is an unprocessed pixel, the process returns to step S91 until there is no unprocessed pixel. The processes of steps S91 to S95 are repeated. If it is determined in step S95 that there are no unprocessed pixels, that is, motion vectors have been generated for all pixels, the process ends.

  As a result of the above processing, it is possible to obtain an initial corresponding point sequentially from the image of the low-resolution resolution level based on the information of the previous corresponding point and set only the vicinity of the initial corresponding point as the search range of the matching block. Therefore, it becomes possible to detect a motion vector stably at high speed. When a motion vector is necessary for a higher resolution image, the multiresolution section 31 may generate a higher resolution image and change the resolution level (le) to be lower. .

  Now, the description returns to the flowchart of FIG.

  In step S <b> 3, the camera shake correction unit 14 reads the motion vector information of each pixel stored in the motion vector buffer 13, and for each pixel of the image read from the image buffer 11, The camera shake generated in the image is corrected and output.

  In step S4, the camera shake correction unit 14 determines whether or not the next image has been supplied to the image buffer 11. If the next image has been supplied, the process returns to step S1 to correct camera shake. The processing to be repeated sequentially. If it is determined in step S4 that the next image has not been supplied, the process ends.

  With the above processing, even for an image with camera shake, the motion vector is obtained at high speed for each pixel by the above-described method. Therefore, the camera shake is corrected based on the motion vector and output. Is possible.

  In the above, the accuracy of the corresponding points is improved by repeated calculation. However, for example, the corresponding points can be accurately obtained by one calculation by parameter fitting.

  FIG. 14 shows a configuration of the motion vector detection unit 12 in which the corresponding points are accurately obtained by one calculation by parameter fitting. In FIG. 14, the same components as those in FIG. 4 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

  14 is different from FIG. 4 in that a fitting calculation unit 61 and a motion vector generation unit 62 are provided. The fitting calculation unit 61 analytically calculates the position of the corresponding point using a plurality of pixels whose resolution level is one level lower by a quadratic function. The motion vector generation unit 62 generates a motion vector based on the corresponding points supplied from the fitting calculation unit 61.

  The camera shake correction process using the motion vector detection unit 12 in FIG. 14 will be described, but is the same as that using the motion vector detection unit 12 in FIG. 4 except for the motion vector generation process in step S26 in the flowchart in FIG. Therefore, only the motion vector generation process in the motion vector detection unit 12 in FIG. 14 will be described, and the other processes are the same, and the description thereof will be omitted.

  First, an example of performing fitting calculation using pixel values in the vicinity of five points will be described with reference to the flowchart of FIG.

  In step S111, the fitting calculation unit 61 sets a pixel of interest p from unprocessed pixels.

  In step S112, the fitting calculation unit 61 obtains an initial corresponding point Q [k] (p) of the target pixel p [k]. That is, for example, with respect to the resolution level (le), it is assumed that le = 1, and the resolution of the image for which the motion vector is finally obtained is the resolution level (0), that is, the motion vector at each pixel of the input image is obtained. For example, the fitting calculation unit 61 obtains a pixel p ′ whose resolution level is one level higher from the target pixel p by the same method as the method shown in FIG. 12, and the pixel P [1] corresponding to the pixel p ′. (p ′) is obtained, and the corresponding point pixel P [1] (p ′) is subjected to coordinate transformation to obtain the initial corresponding point Q [0] of the corresponding point pixel P [0] (p) at the resolution level (0). Find (p).

  In step S113, the fitting calculation unit 61, as shown in FIG. 16, at the resolution level (0), the pixel center q [0] of the pixel closest to Q [0] (p) and the pixels around it. Evaluated value e [i] of equation (4) at four points in the center, that is, q [i]: {i = 0,1,3,5,7} in total, {i = 0,1,3,5 , 7}.

  In step S114, the fitting calculation unit 61 calculates an evaluation value e [i]: {i = 0,1,3,5,7} of five pixels in the vicinity of the initial corresponding point Q [0] (p) as follows: Substituting into the relational expression shown in equation (11), the addition vector (s, t) is obtained, and the vector addition is performed on the pixel q [0] closest to the initial corresponding point Q [0] (p). P [0] (p) is calculated and supplied to the motion vector generation unit 62.

  Here, as shown in FIG. 16, s and t are axes of a space centered on the pixel q [0] closest to the initial corresponding point Q [0] (p). Further, the relationship between the evaluation values according to the positions in the st space can be expressed by, for example, an approximate function F (q) = f (s, t) in the following equation (12).

  Here, a, b, c, d, α, and β are parameters.

  Therefore, an evaluation value e [i]: {i = 0,1,3,5,7} is assigned to the approximate function F (q) = f (s, t) represented by the equation (12) in each st space. Substituting with the above coordinates, the following five relational expressions are obtained.

  Further, if it is assumed that α is 0 among the parameters, partial differentiation of the parameters s and t results in simultaneous equations as shown in the following equations (14) and (15), which are solved. Thus, (s, t) that minimizes the approximate function f is expressed by Equation (11).

  In step S115, the motion vector generation unit 62 generates the motion vector V (p) by calculating Expression (10) described above using the target pixel that is the start point from the corresponding point that is the end point of the motion vector. .

  In step S116, the motion vector generation unit 62 stores the generated motion vector V (p) at the target pixel p in the motion vector buffer 13.

  In step S117, the motion vector generation unit 62 determines whether there is an unprocessed pixel. If it is determined that there is an unprocessed pixel, the process returns to step S111 until there is no unprocessed pixel. The processes of steps S111 to S117 are repeated. If it is determined in step S117 that there are no unprocessed pixels, that is, motion vectors have been generated for all pixels, the process ends.

  With the above processing, it is possible to obtain a highly accurate motion vector by a single calculation rather than iterative calculation only by obtaining at most five evaluation values. Further, since an image having an original resolution or higher is not required in the pyramid, the capacity of the multi-resolution image buffers 32-1 and 32-2 can be saved.

  Further, although the above is an example of fitting using 5 points, evaluation values corresponding to more pixels may be used, for example, fitting may be performed using 9 points.

  FIG. 17 is a flowchart for explaining motion vector generation processing using fitting with nine pixels. Note that the processing in steps S131, S132, and S135 through S137 in FIG. 17 is the same as the processing in steps S111, S112, and S115 through S117 in FIG.

  That is, in step S133, the fitting calculation unit 61, as shown in FIG. 16, has the pixel center q [0] of the pixel closest to Q [0] (p) and the surrounding pixels at the resolution level (0). The evaluation value e [i]: {i = 0 to 8} of the equation (4) is calculated at 8 points in the pixel center, that is, q [i]: {i = 0 to 8} for a total of 9 points.

  In step S134, the fitting calculation unit 61 represents the evaluation values e [i]: {i = 0 to 8} of nine pixels in the vicinity of the initial corresponding point Q [0] (p) by the following equation (16). The addition vector (s, t) is obtained, and the vector addition is performed on the pixel q [0] closest to the initial corresponding point Q [0] (p), and the corresponding point P [0] (p ) Is calculated and supplied to the motion vector generator 62.

  In other words, as described above, if the expression (16) is expressed by using a method using five points and a, b, c, d, and β are expressed using α as a parameter, a new pixel q [ i]: For the four points {i = 2, 4, 6, 8}, the difference between the approximate function F (q) = f (s, t) in equation (12) and the evaluation value e is the smallest. As described above, the parameter α of f (s, t) is determined by the least square method so that the coordinates taking the minimum value of f can be determined by a single calculation. Note that for equation (16), the parameters a, b, c, d, and α are (e [1] + e [5] −2e [0]) / 2, (e [1] −e [5], respectively. ) / 2, (e [3] + e [7]) / 2, (e [3] + e [7] -2e [0]) / 2, (e [2] -e [4] + e [6]- e [8]) / 4.

  New pixel q [i]: The evaluation value e for four points of {i = 2, 4, 6, 8} is expressed by the following equation (17).

  Furthermore, when the expression (17) is expressed by a coordinate system in the st space shown in FIG. 16, it is expressed by the following expression (18).

  In such a case, as shown in the following equation (19), F (q [i]): {i = 2, 4, 6, 8} that is a true value and an evaluation value e [i that is a sample value ]: If an error square sum E with {i = 2, 4, 6, 8} is defined, the parameter α is a value when the error square sum E is minimized.

  The error sum of squares E shown in Expression (19) is minimized when the condition that the function partially differentiated by α is 0 is satisfied, as shown in Expression (20) below.

  When equation (19) is solved according to this condition, α is obtained as shown in equation (21) below.

  By determining all the parameters by the above processing, the function F (q) = f (s, t) is expressed as s and t, respectively, as shown in the following equations (22) and (23). If the value obtained by partial differentiation is 0, equation (16) can be obtained.

  With the above processing, it is possible to obtain a motion vector with higher accuracy by only calculating nine evaluation values and performing one calculation instead of repeated calculation. The processing of the motion vector detection unit 12 of FIG. 14 described with reference to the flowcharts of FIGS. 15 and 17 has been described when the resolution level (le) is le = 1 as an example of description. If the motion vector holding method changes, le may be a value other than 1.

  In addition, the method described so far considered the motion vector image as a change from the past to the future only in one direction, but if the two input images are replaced, a motion vector from the future to the past can be generated by the same processing. .

  With the processing as described above, it is possible to realize robust and highly accurate motion vector detection at high speed.

  According to the present invention, various image processing using motion vectors and corresponding point search, such as image compression, image special effects, still image / frame image quality improvement, moving image improvement, object tracking, etc., using particularly unfavorable image groups. The accuracy and processing speed in image processing can be improved.

  Incidentally, the series of image processing described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

  FIG. 18 shows a configuration example of a general-purpose personal computer. This personal computer incorporates a CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via the bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

  An input / output interface 1005 includes an input unit 1006 including an input device such as a keyboard and a mouse for a user to input an operation command, an output unit 1007 for outputting a processing operation screen and an image of a processing result to a display device, a program and various data. A storage unit 1008 including a hard disk drive to be stored, a LAN (Local Area Network) adapter, and the like, and a communication unit 1009 for performing communication processing via a network represented by the Internet are connected. Also, a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor A drive 1010 for reading / writing data from / to a removable medium 1011 such as a memory is connected.

  The CPU 1001 is read from a program stored in the ROM 1002 or a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage unit 1008, and loaded from the storage unit 1008 to the RAM 1003. Various processes are executed according to the program. The RAM 1003 also appropriately stores data necessary for the CPU 1001 to execute various processes.

  In this specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series in the order described, but of course, it is not necessarily performed in time series. Or the process performed separately is included.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

It is a figure explaining how to obtain a conventional motion vector. It is a figure explaining how to obtain a conventional motion vector. It is a block diagram explaining the structural example of the camera-shake correction processing apparatus to which this invention is applied. It is a block diagram explaining the structural example of the motion vector detection part of FIG. 4 is a flowchart for explaining camera shake correction processing by the camera shake correction processing apparatus of FIG. 3; It is a flowchart explaining the motion vector detection process by the motion vector detection part of FIG. It is a figure explaining the multi-resolution process of the low resolution by a multi-resolution part. It is a figure explaining the multi-resolution conversion process of the high resolution by a multi-resolution part. It is a flowchart explaining the minimum resolution corresponding | compatible point extraction process by the motion vector detection part of FIG. It is a figure explaining a Gaussian function. It is a flowchart explaining the corresponding point extraction process by the motion vector detection part of FIG. It is a figure explaining the setting method of an initial corresponding point. It is a flowchart explaining the motion vector production | generation process by the motion vector detection part of FIG. It is a block diagram explaining the other structural example of the motion vector detection part of FIG. It is a flowchart explaining the motion vector generation process by the motion vector detection part of FIG. It is a figure explaining the motion vector production | generation process by the motion vector detection part of FIG. It is a flowchart explaining the other motion vector production | generation process by the motion vector detection part of FIG. And FIG. 11 is a diagram illustrating a configuration example of a personal computer.

Explanation of symbols

  31 multi-resolution conversion unit, 31a imaging unit, 31b imaging unit, 32, 32-1, 32-2 multi-resolution image buffer, 33 corresponding point search unit, 33a matching block setting unit, 33b search range setting unit, 33c evaluation calculation unit, 33d corresponding point determination unit, 33e initial corresponding point determination unit 33e, 34 corresponding point motion vector buffer, 35 motion vector output determination unit, 36 initial value motion vector buffer, 37 motion vector generation unit

Claims (10)

Multi-resolution means for converting an input image into a plurality of resolution images;
First accumulation means for accumulating a first image at a first timing converted into a plurality of resolutions by the multiresolution means;
Second storage means for storing a second image at a second timing immediately before the first timing converted into a plurality of resolutions by the multi-resolution means;
Corresponding point search means for searching for corresponding points between the pixels of the second image at the first resolution and the pixels of the first image at the first resolution;
Storage means for storing information on corresponding points of the pixels of the second image at the first resolution searched by the corresponding point search means and the pixels of the first image at the first resolution. ,
The corresponding point search means is based on the information of the corresponding points in the first resolution stored in the storage means, and the pixels of the second image at a second resolution higher than the first resolution. An image processing apparatus that searches for a corresponding point with a pixel of the first image at the second resolution and repeats sequentially. The corresponding point search means includes
Evaluation value calculating means for obtaining an evaluation value of a corresponding point between a pixel of the second image at the first resolution and a pixel of the first image at the first resolution by block matching;
The image processing apparatus according to claim 1, wherein a pixel having the smallest evaluation value calculated by the evaluation value calculating unit is searched as a corresponding point. The evaluation value calculation means uses each of the pixels of the second image of the first resolution and the first of the first resolution using weighted evaluation according to the distance from the target pixel in the block matching block. The image processing apparatus according to claim 2, wherein an evaluation value of a corresponding point with each pixel of the image is obtained. The evaluation value calculation means uses each of the pixels of the second image of the first resolution and the first resolution using weighted evaluation according to a difference in pixel value between corresponding pixels between the blocks of the block matching. The image processing apparatus according to claim 2, wherein an evaluation value of a corresponding point with each pixel of the first image is obtained. Fitting calculation means for calculating five evaluation values for five pixels in the vicinity of corresponding points in an image of a predetermined resolution and obtaining the corresponding points by using an approximate function coefficient based on the five evaluation values. The image processing apparatus according to claim 2, further comprising: Fitting calculation means for calculating nine evaluation values for nine pixels in the vicinity of corresponding points in an image of a predetermined resolution and obtaining the corresponding points by using coefficients of an approximation function based on the nine evaluation values. The image processing apparatus according to claim 2, further comprising: The multi-resolution unit converts the input image into a plurality of resolution images having a resolution lower than that of the input image or a plurality of resolution images having a resolution higher than that of the input image. Image processing device. A multi-resolution step for converting an input image into a plurality of resolution images;
A first accumulation step of accumulating a first image at a first timing converted into a plurality of resolutions by the processing of the multi-resolution step;
A second accumulating step of accumulating a second image at a second timing immediately before the first timing converted into a plurality of resolutions by the multi-resolution process;
A corresponding point search step of searching for corresponding points between the pixels of the second image at the first resolution and the pixels of the first image at the first resolution;
A storing step for storing information on corresponding points between the pixel of the second image at the first resolution and the pixel of the first image at the first resolution searched by the processing of the corresponding point searching step; Including
The processing of the corresponding point search step includes a second resolution at a second resolution higher than the first resolution based on the information on the corresponding points at the first resolution stored in the processing at the storage step. An image processing method for searching for a corresponding point between a pixel of an image and a pixel of the first image at the second resolution, and sequentially repeating. A multi-resolution step for converting an input image into a plurality of resolution images;
A first accumulation step of accumulating a first image at a first timing converted into a plurality of resolutions by the processing of the multi-resolution step;
A second accumulating step of accumulating a second image at a second timing immediately before the first timing converted into a plurality of resolutions by the multi-resolution process;
A corresponding point search step of searching for corresponding points between the pixels of the second image at the first resolution and the pixels of the first image at the first resolution;
A storing step for storing information on corresponding points between the pixel of the second image at the first resolution and the pixel of the first image at the first resolution searched by the processing of the corresponding point searching step; Including
The processing of the corresponding point search step includes a second resolution at a second resolution higher than the first resolution based on the information on the corresponding points at the first resolution stored in the processing at the storage step. A program for searching for corresponding points between pixels of an image and pixels of the first image at the second resolution, and sequentially repeating the program.   A program storage medium in which the program according to claim 9 is stored.

Images