JP2019021990A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To suppress deterioration of a vector detection accuracy when a feature point of a tracking destination is distributed in an image.SOLUTION: A region division part (103) divides an image of a frame into a plurality of regions. A feature point extraction part (104) extracts a feature point of the image in each region obtained by dividing the plurality of regions. A vector detection part (105) detects a motion vector on the basis of the feature point. A region separation part (106) separates the plurality of regions divided into at least two region separations. A tracking destination feature point calculation part (107) calculates a tracking destination feature point used for the motion vector detection of the next frame on the basis of a value of the motion vector detected by the frame and the feature point. A reconstruction part (108) evenly reconstructs a position of the tracking destination feature point in a region separation including distribution at the position of the tracking destination feature point, when there is the distribution at the position of a tracking destination feature point A in the region separation.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for detecting motion in an image.

  In order to perform camera shake correction on video shot using an imaging device such as a digital still camera or a digital video camera, the amount of motion between frame images is detected and alignment is performed on a plurality of images. There is a need. As a method for detecting the amount of motion between frame images, there are a method of using information of an external device such as a gyro sensor, a method of estimating the amount of motion from a captured frame image, and the like.

  Various methods for estimating the amount of motion using a frame image have been proposed in the past. A typical example is motion vector detection by template matching. In template matching, first, one of two consecutive frame images in a video is used as an original image, and the other is used as a reference image. Then, a rectangular area having a predetermined size arranged on the original image is used as a template block, and a correlation with a distribution of pixel values in the template block is obtained at each position of the reference image. At this time, the position where the correlation is highest in the reference image is the destination of the template block, and the direction and amount of movement to the destination when the position of the template block on the original image is used as a reference is the motion vector. .

  In order to improve the detection rate of motion vectors, there is a technique for extracting feature points, placing template blocks at the extracted feature points, and performing template matching between frame images. Here, if feature points are extracted from the entire image, the distribution of feature points is often non-uniform. When a motion vector obtained for feature points having a non-uniform distribution is used for the purpose of camera shake correction, a region where the feature points are concentrated is the main camera shake correction.

Therefore, in order to distribute the feature points uniformly, the image is divided into grids, feature values representing the feature size are calculated for each pixel, and the pixel with the largest feature value in each grid is extracted as the feature point. There is a technique (for example, Patent Document 1).
Furthermore, there is a technique for tracking feature points in order to improve the detection rate of motion vectors. Feature point tracking can be realized by sequentially detecting motion vectors of feature points extracted from an image over a plurality of continuous frame images (for example, Patent Document 2).

JP 2008-192060 A JP 2012-94050 A

  By the way, when the feature point tracking process is performed as described above, the feature point of the tracking destination may be unevenly distributed in a part of the region in the image. Thus, if the feature points of the tracking destination are unevenly distributed in a part of the region, the motion vector detection performance is deteriorated.

  Accordingly, an object of the present invention is to suppress a decrease in vector detection accuracy when feature points of tracking destinations are unevenly distributed in an image.

  The present invention relates to a dividing unit that divides a frame image into a plurality of regions, an extracting unit that extracts image feature points in each of the divided regions, and a motion vector is detected based on the feature points. Based on the detection means, the dividing means for dividing the divided plurality of areas into at least two area sections, and the motion vector values and feature points detected from the previous frame, the motion vector of the next frame A calculation means for calculating a tracking destination feature point used for detection; and if the position of the tracking destination feature point is biased in the area section, the position of the tracking destination feature point is biased in the area section And a resetting means for resetting the positions of the tracking destination feature points evenly.

  According to the present invention, it is possible to suppress a decrease in vector detection accuracy when the feature points of the tracking destination are unevenly distributed in the image.

1 is a diagram illustrating a schematic configuration example of an image processing apparatus according to an embodiment. It is a figure used for description of a feature point, a tracking destination feature point, and a motion vector. It is explanatory drawing of a template matching process timing. It is a figure explaining the bias of a motion vector (tracking destination feature point). It is explanatory drawing of a subject area | region division and a background area | region division. FIG. 6 is a diagram in which a subject area section is divided into a plurality of counting areas. It is explanatory drawing of the example which reset the biased tracking destination feature point equally. It is explanatory drawing of the deletion method of a tracking destination feature point. It is a figure which shows the example of schematic structure of the image processing apparatus which divided the process by area | region division. It is a figure which shows an example of the area division of the structural example which divided the process by area division. It is explanatory drawing of the example of a processing timing delay when a tracking destination feature point is biased. It is a figure used for description of an area division and a designated area.

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.
<First Embodiment>
FIG. 1 is a diagram illustrating a schematic configuration example of a digital camera as an application example of the image processing apparatus according to the first embodiment. In addition to the digital camera, the image processing apparatus of the present embodiment includes, for example, a digital video camera, various portable terminals such as a smartphone and a tablet terminal having a camera function, an industrial camera, an in-vehicle camera, a medical camera, and the like. The present invention can also be applied to various imaging devices.
The image processing apparatus according to the present embodiment illustrated in FIG. 1 includes an image sensor unit 101, a memory 102, a region dividing unit 103, a feature point extracting unit 104, a vector detecting unit 105, a region segmenting unit 106, and a tracking destination feature point calculating unit 107. The resetting unit 108 is configured.

The image sensor unit 101 photoelectrically converts a subject image obtained through an optical system (not shown) such as a lens and a diaphragm, converts the image into digital image data, and outputs the digital image data in a so-called raster scan order. In the case of this embodiment, the image sensor unit 101 outputs image data for each frame of a moving image by repeatedly performing photoelectric conversion, conversion into digital image data, and output of the image data at a constant time period.
The memory 102 stores and holds image data of each frame of the moving image input from the image sensor unit 101. The memory 102 stores at least the image data of the current frame and the previous frame image data.

The area dividing unit 103 divides the image for each frame read from the memory 102 into a plurality of areas. The area dividing unit 103 divides the frame image into a plurality of areas at the timing when the frame image is input. Details of the area dividing process in the area dividing unit 103 will be described later.
The feature point extraction unit 104 obtains one feature point for each divided region divided by the region dividing unit 103. Details of the feature points extracted for each divided region will be described later.
The vector detection unit 105 detects a motion vector based on the feature points extracted for each divided region. Details of motion vector detection based on feature points will be described later.

Hereinafter, the region division by the region division unit 103, the extraction of feature points by the feature point extraction unit 104, and the vector detection by the vector detection unit 105 will be described in detail.
In motion vector detection processing, motion vector detection processing using template matching described above is known. In the motion vector detection process, in order to improve the vector detection rate, as described above, a process of extracting feature points by dividing an image into a grid shape, a process of tracking feature points, and the like are performed.

FIG. 2A is a diagram used for explaining the outline of the tracking process using the grid arrangement, the extracted feature points, the template matching region, and the extracted feature points in the vector detection process.
The area dividing unit 103 in FIG. 1 divides a frame image into a plurality of areas (grids). Each of the grids is arranged in the number set in the horizontal and vertical directions as a feature extraction grid 1010 and a peripheral grid 1011 (a grid with a dot pattern in FIG. 2A) having a preset size. . The peripheral grid 1011 is arranged around a feature extraction grid group including the feature extraction grids 1010. The peripheral grid 1011 is used for template matching processing of the vector detection unit 105 in the subsequent stage, but is not used for feature point extraction in the feature point extraction unit 104. FIG. 2B will be described in a third embodiment to be described later.

  The feature point extraction unit 104 extracts one feature point 1001 for each feature extraction grid 1010. Specifically, the feature point extraction unit 104 extracts the position where the feature amount is determined to be the largest in the feature extraction grid 1010 as the feature point of the feature extraction grid 1010. There are various methods for obtaining the feature amount. As an example, the feature point extraction unit 104 performs an n × operation centered on the point of interest with respect to the result of performing the band pass filter process on the image of the feature extraction grid 1010. An integral value obtained by integrating the values of n pixels is obtained as a feature amount. In the case of this example, it is considered that the larger the integral value, the larger the feature amount.

  The vector detection unit 105 uses one of two frame images adjacent on the time axis of the moving image as an original image and the other as a reference image, and the pixel value and the original in a rectangular area centering on a feature point in the reference image. A motion vector is detected by performing a correlation value calculation process with pixel values in a rectangular area of the image. Specifically, the vector detection unit 105 determines where the point corresponding to the feature point on the first image exists on the second image at the timing when the second image is input. To detect. In the case of vector detection based on a feature point, the vector detection unit 105 calculates a movement amount corresponding to the feature point from the position of the feature point on the first image to the detection position on the second image. Detect as a vector. The vector detection unit 105 detects and outputs motion vectors by the number of feature points (that is, the number of divided regions divided by the region dividing unit 103).

  In the case of FIG. 2A, a rectangular search area 1005 and a template area having a predetermined size centering on the feature point 1001 in the feature extraction grid 1010 in the image of the current frame in the template matching process. 1003 is set. Then, the vector detection unit 105 calculates a vector value 1004 of the motion vector by a template matching process using the search area 1005 and the template area 1003. Note that the search area 1005 and the template area 1003 in the template matching process are sequentially set in accordance with the so-called raster scan order in the sensor output of the image sensor unit 101.

  Further, in the processing of the next frame, the vector detection unit 105 performs template matching processing centering on the tracking destination feature point 1002 calculated by the tracking destination feature point calculation unit 107 described later in the previous frame. . The tracking destination feature point 1002 is calculated by adding the vector value 1004 calculated in the previous frame to the feature point 1001. Accordingly, processing similar to that described above is performed in order over a plurality of frames, and vector values are sequentially added to the tracking destination feature points calculated sequentially in each frame, thereby realizing feature point tracking.

  FIG. 3A is a timing chart from the image output of the image sensor unit 101 (hereinafter referred to as sensor output) to the end of the template matching process. FIG. 3B illustrates a feature extraction grid 1010, a peripheral grid 1011, a search area 1005, a template area 1003, and a tracking destination feature point 1002 similar to those in FIG. FIG. 3B also shows an arrow representing a so-called raster scan 1020 and the first (first) template position 1101 at which template matching processing can be started. In FIG. 3A, image output (sensor output) of the image sensor unit 101 is performed within a period (frame period) of a pulse interval of one frame of the vertical synchronization signal VD, and a template is based on the image data of the sensor output. A matching image is generated. Generation of the template matching image is started in accordance with the sensor output in the raster scan order, and is completed at a timing delayed by the time required for the image generation processing. The template matching process is performed using a template matching image. Each period T1 to T8 in FIG. 3A represents the time required for the template matching process corresponding to the tracking destination feature point. . Note that the time required for the template matching process increases or decreases according to the size (size) of the template area.

  In the example of FIG. 3B, only eight feature extraction grids 1010 are illustrated, and one tracking destination feature point 1002 is obtained for each of the feature extraction grids 1010. Therefore, in FIG. 3A, eight periods T1 to T8 are shown as the time required for the template matching process for each of the eight tracking destination feature points 1002. Further, as shown in FIG. 3B, the template matching process in the period of one frame is started from the template position 1101 set for the tracking destination feature point 1002 that is first (first) in the raster scan order. The The template matching process is completed when all the processes in the periods T1 to T8 corresponding to the eight tracking destination feature points 1002 are completed.

  Here, as in the digital camera according to the present embodiment, when a motion vector is used for camera shake correction at the time of shooting, a frame image is divided into, for example, a subject area section and a background area section, and each area is divided. A motion vector is detected for each segment. As described above, by detecting the motion vectors of the subject region segment and the background region segment and performing the blur correction process using the motion vectors, it is possible to correct the blur of the subject with high accuracy.

  On the other hand, in the feature point tracking process at the time of detecting the motion vector, the tracking destination feature point gradually moves within the image range of each temporally continuous frame. For this reason, for example, depending on the movement of the subject or the movement of the camera itself, the tracking destination feature point may be unevenly distributed in, for example, a partial region within the image range of the frame. As described above, in the case of detecting a motion vector for each area segment, if the tracking point feature point is unevenly distributed, for example, a change in motion in the target area segment such as a subject is accurately captured. It may not be possible.

  FIG. 4A to FIG. 4C are diagrams illustrating an example of a case in which a change in motion cannot be accurately captured due to bias in the tracking destination feature point within the subject region segment. FIG. 4A shows a face area 800 as a subject area in the image of the first frame, and FIG. 4B shows a plurality of movements based on tracking destination feature points in the face area 800 in the next second frame. An example in which a vector 801 is detected is shown. In the example of FIG. 4B, the face region 800 is rotated clockwise from the example of FIG. 4A in the clockwise direction (clockwise direction), so that a plurality of movements according to the clockwise rotation are performed. An example in which the vector 801 is detected is shown.

  By the way, when the face of the subject is photographed, the tracking destination feature points are unevenly distributed in a part of the region due to, for example, panning of the camera. As a result, a plurality of motion vectors 802 are obtained as shown in FIG. It may be detected densely in some areas. When a plurality of motion vectors 802 are densely detected in a part of the area as shown in FIG. 8C, whether the face area 800 has been rotated clockwise or the face area 800 has been panned. It is difficult to distinguish whether it has moved in the left direction. In this way, if the tracking destination feature points are unevenly distributed in some areas and motion vectors are detected densely in some areas, it is impossible to capture the change in the exact motion of the image, and the vector detection accuracy Will fall. If such a low-precision motion vector is used for camera shake correction at the time of shooting, the camera shake correction performance of the camera also deteriorates.

Therefore, the image processing apparatus according to the first embodiment sets at least two sections, that is, a first area section and a second area section, for a frame image, and detects motion vectors detected for each feature extraction grid. Based on the information, it is determined to which region section the tracking destination feature point belongs. Furthermore, the image processing apparatus according to the first embodiment determines the deviation of the position of the tracking destination feature point for each area segment. The tracking destination feature points are reset (rearranged) so as to be evenly distributed.
The image processing apparatus according to the first embodiment is configured to classify an area segment, determine an area segment to which a tracking destination feature point belongs, determine a bias, and reset a tracking destination feature point according to the determination result. An area classification unit 106, a tracking destination feature point calculation unit 107, and a resetting unit 108 are included.

  The region partitioning unit 106 partitions the plurality of feature extraction grids described above into a first region partition and a second region partition. There are various methods for the classification process, and any classification process may be used. In the present embodiment, an example using the classification process based on the face detection result is given as an example. FIG. 5 is an image diagram of segmentation processing using face detection technology. In the case of the example in FIG. 5, the area classification unit 106 includes a rectangular area 201 including a face area detected by the face detection technique, and a rectangular area from the rectangular area 201 including the face area to the lower end of the frame image range. 202 is set as the first area section. In the present embodiment, the first area section is referred to as a subject area section 200. Further, the area segmenting unit 106 sets the remaining area excluding the subject area segment 200 within the frame image range as the second area segment. In the present embodiment, the second region segment that is the remaining region excluding the subject region segment 200 within the frame image range is referred to as a background region segment. Then, the area segmentation unit 106 determines whether each position of each tracking destination feature point at the time of motion vector detection by the motion vector detection unit 105 belongs to either the subject area segment 200 or the background area segment. In this way, each tracking destination feature point is classified. By dividing the tracking destination feature points in this way, the motion vector detected by the vector detecting unit 105 is also classified as to which of the subject area section 200 and the background area section belongs.

  The tracking destination feature point calculation unit 107 tracks the next frame based on the vector value of the previous frame detected by the vector detection unit 105 and the feature point of the previous frame extracted by the feature point extraction unit 104. The target feature point (tracking destination feature point) is calculated. Then, the tracking destination feature point calculation unit 107 uses the calculated location information (coordinate information) of the tracking destination feature points and information indicating the area classification to which the tracking destination feature points belong (referred to as classification information) Store in the memory.

  The resetting unit 108 determines whether or not the position of each tracking destination feature point in the subject area segment 200 is biased. If it is determined that the bias is generated, the resetting unit 108 The tracking destination feature points are reset so as to be substantially uniform.

  Specifically, first, the resetting unit 108 counts the number of tracking destination feature points in the subject region segment 200, divides the subject region segment 200 into a plurality of count regions as shown in FIG. The number of tracking destination feature points is counted for each region. FIG. 6 shows an example in which the subject area section 200 is divided into four parts, upper, lower, left and right, and divided into four counting areas (upper left counting area 301, upper right counting area 302, lower left counting area 303, lower right counting area 304). Cite. FIG. 7A shows each motion vector 400 detected in the upper left counting area 301, the upper right counting area 302, the lower left counting area 303, and the lower right counting area 304 shown in FIG. Since each motion vector 400 as shown in FIG. 7A is obtained based on tracking destination feature points, the number of these motion vectors 400 corresponds to the number of tracking destination feature points. In the example of FIG. 7A, there are four tracking destination feature points in the upper left counting area 301, two in the upper right counting area 302, one in the lower left counting area 303, and one in the lower right counting area 304. In total, there are eight objects in the subject area section 200.

  In the case of the example of FIG. 7A, the eight tracking destination feature points are equally divided into four counting areas, and the number of tracking destination feature points for each counting area is two. It is considered that the feature points are not biased. That is, in the example of FIG. 7A, since there are four tracking destination feature points in the upper left counting area 301, as shown in FIG. 7B, the number of tracking destination feature points is reduced by two. Make two. In the example of FIG. 7A, since two tracking destination feature points already exist in the upper right counting area 302, as shown in FIG. 7B, these two tracking destinations are displayed in the upper right counting area 302. Keep feature points as they are. On the other hand, in the example of FIG. 7A, there is only one tracking destination feature point in each of the lower left counting area 303 and the lower right counting area 304. For this reason, as shown in FIG. 7B, two tracking destination feature points deleted from the upper left counting area 301 are reset (rearranged) one by one in the lower left counting area 303 and the lower right counting area 304, respectively. To do. As a result, as shown in FIG. 7B, the number of tracking destination feature points in each counting area is set to two, and a state in which the positions of the tracking destination feature points are not biased is realized.

  In order to realize such resetting of the tracking destination feature points, the resetting unit 108 calculates, as a target number, a number obtained by equally dividing the number of tracking destination feature points in the subject region segment 200 by the number of counting regions. Then, the number of tracking destination feature points in the counting area is compared with the target number. Then, when the number of tracking destination feature points in the counting area exceeds the target number, the resetting unit 108 reduces the number of tracking destination feature points in the counting area to a number not exceeding the target number (to the target number or less). To do.

  In the case of the present embodiment, as a method for determining tracking destination feature points to be reduced when the number of tracking destination feature points in the counting area exceeds the target number, for example, FIGS. 8A to 8C will be described. Such a technique can be used. Here, an example will be described in which two tracking destination feature points are deleted from the upper left counting area 301 of FIG. FIG. 8A shows the upper left counting area 301 in a state before the tracking destination feature point is deleted. The resetting unit 108 first obtains the centroid positions 501 of the coordinates of the four tracking destination feature points corresponding to the four motion vectors 400 in the upper left counting area 301 in FIG. 8A, and is closest to the centroid position 501. The tracking destination feature point at the position is targeted for reduction. FIG. 8B shows the upper left counting area 301 in a state after one tracking destination feature point at a position closest to the center of gravity position 501 in FIG. 8A is deleted. Next, the resetting unit 108 obtains a new centroid position 502 from the coordinates of the three tracking destination feature points corresponding to the three motion vectors 400 in the upper left counting area 301 in FIG. The feature point to be tracked closest to the track is selected as the reduction target. FIG. 8C shows the upper left counting area 301 in a state after one tracking destination feature point closest to the center of gravity position 502 in FIG. 8B is deleted. The number of tracking destination feature points is two. In this example, two of the four tracking destination feature points are deleted. However, when three or more tracking destination feature points are deleted, the calculation of the centroid position and the centroid position are calculated as described above. The process of deleting the closest tracking destination feature point is repeated. In the above example, every time one tracking destination feature point is deleted, the centroid position is recalculated and updated. However, the centroid position is the first one obtained after that. It may be used as it is when deleting.

Further, when resetting (compensating) the tracking destination feature point for the counting area, the resetting unit 108 obtains the feature quantity of the tracking destination feature point in the counting area as an example, and the feature quantity is the largest. A method of resetting (compensating) the tracking destination feature point at the position can be used.
In the above description, the subject region segment 200 is taken as an example. However, the resetting unit 108 performs the tracking destination feature point resetting process similarly for the background region segment.

As described above, in the image processing apparatus according to the first embodiment, when there is a deviation in the position of each tracking destination feature point in the area division into which the frame image range is divided, the image processing apparatus in the area division is approximately. The tracking destination feature points are reset so that they are evenly arranged. Thereby, according to the image processing apparatus of the first embodiment, it is possible to suppress a decrease in vector detection accuracy even when the tracking destination feature points are unevenly distributed within the frame image range.
<Second Embodiment>
FIG. 9 is a diagram illustrating a schematic configuration example of a digital camera as an application example of the image processing apparatus according to the second embodiment. Note that the image processing apparatus according to the second embodiment can also be applied to various imaging apparatuses as in the case of the first embodiment.

  The image processing apparatus according to the second embodiment illustrated in FIG. 9 includes an image sensor unit 601, a memory 602, a region segmentation unit 603, and optical flow detection units 609 and 619. The optical flow detection unit 609 includes an area division unit 604, a feature point extraction unit 605, a vector detection unit 606, a tracking destination feature point calculation unit 607, and a resetting unit 608. The optical flow detection unit 619 includes a region dividing unit 614, a feature point extraction unit 615, a vector detection unit 616, a tracking destination feature point calculation unit 617, and a resetting unit 618. In the present embodiment, the optical flow detection unit 609 and the optical flow detection unit 619 have the same configuration. However, the optical flow detection unit 609 and the optical flow detection unit 619 may have different configurations as long as the optical flow can be detected in different regions. Good.

  The image sensor unit 601 has the same configuration as that of the image sensor unit 101 of FIG. 1 described above, and outputs image data obtained by imaging. The memory 602 stores and holds image data of each frame of the moving image input from the image sensor unit 601. Similar to the memory 102 in FIG. 1, the memory 602 stores at least the image data of the current frame and the previous frame image data. In the case of the second embodiment, the image data for each frame read from the memory 602 is respectively sent to the region dividing unit 604 of the optical flow detecting unit 609, the region dividing unit 614 of the optical flow detecting unit 619, and the region dividing unit 603. Sent.

  The area division unit 603 divides the image range of the frame into a first area division and a second area division similar to the area division unit 106 in FIG. 1 described above. FIG. 10 is a diagram illustrating an example of area division in the area division unit 603 in the case of the second embodiment. FIG. 10 includes two face areas 900 detected by the face detection process, and a rectangular area including the face area 900 to the lower end of the image range of the frame is set as the subject area section 901. An example in which the area division 902 is set is given.

  In the case of the second embodiment, the region segmentation unit 603 sends information indicating the subject region segment, which is an example of the first region segment, to the optical flow detection unit 609, and the background region segment, which is an example of the second region segment, Is sent to the optical flow detector 619. Therefore, in the present embodiment, the optical flow detection unit 609 performs processing for the subject region segment, and the optical flow detection unit 619 performs processing for the background region segment.

Hereinafter, the optical flow detection unit 609 will be described as an example.
The area division unit 604 of the optical flow detection unit 609 divides the image of the subject area section in the image data of the frame for each predetermined area (the grid described above). The feature point extraction unit 605 obtains a feature amount for each grid divided by the region division unit 604, and arranges the feature point at a position where the feature is determined to be the largest. As a result, the position of one feature point is determined for each grid. The feature amount is the same as that in the first embodiment described above.

  The vector detection unit 606 detects a motion vector by performing correlation value calculation processing in the same manner as the vector detection unit 105 in FIG. 1, with one of the subject region sections of the two frames as an original image and the other as a reference image. . The vector detection unit 606 detects and outputs motion vectors as many as the number of feature points (that is, the number of feature extraction grids in the subject region segment).

  Similar to the tracking destination feature point calculation unit 107 in FIG. 1, the tracking destination feature point calculation unit 607 and the vector value of the previous frame detected by the vector detection unit 606 and the previous extracted by the feature point extraction unit 605 Based on the feature point of the frame, the tracking destination feature point of the next frame is calculated. Then, the tracking destination feature point calculation unit 607 stores the information on the position of the calculated tracking destination feature point (coordinate information) and information indicating the area classification to which the tracking destination feature point belongs in an internal memory.

Similar to the resetting unit 108 in FIG. 1, the resetting unit 608 determines whether or not there is a bias in the position of each tracking destination feature point within the subject area segment, and determines that there is a bias. In such a case, the tracking destination feature points are reset so as to be substantially uniform within the subject area section.
The optical flow detection unit 619 performs the same processing as the optical flow detection unit 609 except that the processing target is the background region classification.

  As described above, the image processing apparatus according to the second embodiment divides the first area section (subject area section) and the second area section (background area section), and performs optical flow detection processing. Do. In the second embodiment, as in the first embodiment, when there is a bias in the position of each tracking destination feature point in the area section, the arrangement is made to be substantially evenly arranged in the area section. The tracking destination feature point is reset. As a result, the image processing apparatus according to the second embodiment can suppress a decrease in vector detection accuracy even when the tracking destination feature points are unevenly distributed within the frame image range.

<Third Embodiment>
Hereinafter, a third embodiment will be described.
In the third embodiment, an example in which the frame image range is divided according to the pixel input order of the frame image (for example, the pixel reading order from the image sensor unit 101) will be described.
Here, as an example in which a plurality of tracking destination feature points are unevenly distributed in a partial region within the image range of the frame, in addition to the example given in the above-described embodiment, for example, as shown in FIG. In some cases, the tracking destination feature points are unevenly distributed so as to be concentrated in the lower region within the image range of the frame. As shown in FIG. 2B, the state in which the tracking target feature points are concentrated so as to be concentrated in the lower region within the image range of the frame occurs, for example, when the camera is tilted during photographing with a digital camera. There are many. 2B also shows a feature extraction grid 1010, a peripheral grid 1011, a search area 1005, a template area 1003, a tracking destination feature point 1002, and the like similar to those in FIG.

  FIG. 11A is a timing chart from the sensor output of the image sensor unit 101 to the end of the template matching process, as in FIG. 3A, and a large number of tracking destinations are displayed in the lower area as shown in FIG. The processing timing when the feature points are concentrated is shown. FIG. 11B shows a feature extraction grid 1010, a peripheral grid 1011, a search area 1005, a template area 1003, and tracking similar to those in FIG. 3B described above when a large number of tracking destination feature points are concentrated in the lower area. FIG. 10 is a diagram showing a previous feature point 1002. FIG. 11B also shows an arrow representing the raster scan 1020 and the first (first) template position 1201 at which template matching processing can be started, as in FIG. 3B. In the example of FIG. 11B, the eight tracking destination feature points 1002 are unevenly distributed so as to be concentrated in the lower region within the image range of the frame. Then, the eight periods T1 to T8 in FIG. 11A represent the time required for the template matching process for each of the eight tracking destination feature points 1002 in FIG.

  Since the template matching process is performed in accordance with the raster scan order that is the pixel reading order from the image sensor unit 101 (sensor output), the lower area in the image range of the frame is an area that is processed at the end in the raster scan order. It becomes. When many tracking destination feature points are concentrated in the lower region as shown in FIG. 2B and FIG. 11B, as shown in FIG. 11A, within the period of the pulse interval of the vertical synchronization signal VD, It may become impossible to complete all template matching processes in the periods T1 to T8. That is, when the tracking destination feature points are concentrated in the lower region, as shown in FIG. 11B, the first template position 1201 at which the template matching process is started is a position corresponding to the last stage in the raster scan order. Then, when the template matching process starts from the template position 1201 corresponding to the last stage of the raster scan order, the period 1200 until the start of the processes of the periods T1 to T8 becomes longer as shown in FIG. For this reason, before the processing in the periods T1 to T8 is completed, the next pulse of the vertical synchronization signal VD may come and the processing may not be completed.

Also, when a motion vector is used for camera shake correction at the time of shooting as in the digital camera of this embodiment, the template matching process is completed within one frame period to detect a motion vector, and the motion vector is detected for the next frame. Need to provide feedback to the process.
In the example of FIGS. 3A and 3B described above, all the processes in the periods T1 to T8 are completed within the period of one frame, and the template matching process is completed. For this reason, the motion vector can be fed back to the processing of the next frame, and as a result, camera shake correction at the time of photographing with the digital camera can be performed correctly.
On the other hand, if the template matching process cannot be completed within the period of one frame as in the examples of FIGS. 11A and 11B, the correct motion vector can be fed back to the process of the next frame. Disappear. In this case, the vector detection performance is lowered, and as a result, the camera shake correction performance at the time of shooting of a digital camera or the like is also lowered.

  For this reason, in the image processing apparatus according to the third embodiment, the plurality of grids described above are divided into at least two area sections, the first area section and the second area section described above, and a first described later. The area is also divided into at least two designated areas, the designated area and the second designated area. The image processing apparatus according to the third embodiment also counts the number of tracking destination feature points in the first designated area, and determines whether the number of tracking destination feature points in the first designated area exceeds a predetermined number. When the number of tracking destination feature points in the first designated area exceeds a predetermined number, the image processing apparatus according to the third embodiment sets the predetermined number of tracking destination feature points in the first designated area. Delete up to the maximum number. Furthermore, the image processing apparatus according to the third embodiment resets (compensates) the number of tracking destination feature points deleted in the first designated area to a position in the same area section in the second designated area. To do. As the method of deleting the tracking destination feature point, the deletion method based on the center of gravity position described in the first embodiment, the method of deleting in order from the smallest feature amount, or the like can be used. In addition, a method similar to that described in the first embodiment can also be used as a method of resetting (compensating) the tracking destination feature points.

Since the configuration of the image processing apparatus according to the third embodiment is the same as that shown in FIG. 1, the illustration thereof is omitted. In the image processing apparatus according to the third embodiment, processing as described below is performed.
The area segmentation unit 106 of the image processing apparatus according to the third embodiment converts each of the above-described divided areas (a plurality of grids) into a first area segment and a second area segment similar to those in the first embodiment. Sort. In the case of the third embodiment, similarly to the first embodiment described above, based on the result of face detection, the subject area section is set as the first area section, and the background area section is set as the second area section. Suppose that

  Furthermore, in the case of the third embodiment, the area dividing unit 106 divides each divided area (a plurality of grids) into a first designated area and a second designated area described later. In the third embodiment, the first designated area and the second designated area are set in accordance with the pixel input order of the frame image (the pixel output order of the sensor output). For example, when the pixel input order of the input frame image is the raster scan order in which scanning in the horizontal direction is performed from the upper left pixel of the frame image range, the first designated area is the lower area of the frame image range, The second designated area is the upper area of the frame image range. That is, the first designated area is an area that is processed in the last stage in the raster scan order, while the second designated area is an area that is initially processed in the raster scan order.

  FIG. 12 shows the arrangement of the first area section (subject area section 200) and the second area section (background area section), the first designated area 701, and the second designated area 702 in the third embodiment. It is the figure which showed the image. In FIG. 12, the first area section (subject area section 200) and the second area section (background area section) are the same area sections as described above with reference to FIG. On the other hand, in the third embodiment, the first designated area 701 is an area that is processed at the end in the raster scan order, and the second designated area 702 is an area that is initially processed in the raster scan order. The

  The tracking destination feature point calculation unit 107 according to the third embodiment calculates the tracking destination feature points in the same manner as described above, information on the positions of the calculated tracking destination feature points, the area classification to which the tracking destination feature points belong, and the designated area Is stored in an internal memory.

  The resetting unit 108 of the third embodiment counts the number of tracking destination feature points calculated in the first designated area 701, and the number of tracking destination feature points in the first designated area 701 exceeds a predetermined number. Determine whether or not. If the number of tracking destination feature points in the first designated area 701 exceeds a predetermined number, the resetting unit 108 does not exceed the predetermined number of tracking destination feature points in the first designated area 701. Delete up to a number. Furthermore, the resetting unit 108 according to the third embodiment resets (compensates) the tracking destination feature point deleted from the first specified area 701 with respect to the second specified area 702. When resetting the deleted tracking destination feature point in the second designated area 702, the tracking destination feature point deleted from the first designated area 701 belonged to the second designated area 702. Fill in the same area section as the area section. That is, the resetting unit 108 resets the tracking destination feature points deleted from the subject area segment 200 in the first designated area 701 so as to compensate in the subject area section 200 of the second designated area 702, for example. (Relocation) is performed. In addition, the resetting unit 108 performs resetting such that, for example, the tracking destination feature point deleted from the background area section in the first designated area 701 is compensated for in the background area section of the second designated area 702. .

  As described above, the image processing apparatus according to the third embodiment divides the frame image range into the first area section and the second area section, and the process is performed at the end in the raster scan order. The area is divided into one designated area and a second designated area that is initially processed. Then, when the number of tracking destination feature points in the first designated area exceeds a predetermined number, the image processing apparatus of the third embodiment deletes the tracking destination feature points up to a number not exceeding the predetermined number, and performs the second designation. Reset the tracking destination feature point within the same area segment of the area. As a result, according to the image processing apparatus of the third embodiment, even when the tracking destination feature points are unevenly distributed within the frame image range, a reduction in vector detection accuracy is suppressed, and a predetermined period such as one frame period is determined. Processing can be completed within a period of time.

<Fourth Embodiment>
A fourth embodiment will be described below. Since the configuration of the image processing apparatus according to the fourth embodiment is the same as the configuration of FIG. 9 described above, its illustration is omitted.
As in the second embodiment described above, the image processing apparatus according to the fourth embodiment performs optical flow for each of the subject area segment as the first area segment and the background area segment as the second area segment. Perform detection processing.
Furthermore, the image processing apparatus according to the fourth embodiment also performs the area division process of the first designated area and the second designated area as in the third embodiment.

  In the image processing apparatus of the fourth embodiment, as in FIG. 9 described above, the image sensor unit 601 outputs captured image data, and the image data read from the memory 602 is optical flow detection units 609 and 619. The data are sent to the area sorting unit 603, respectively.

  The area segmentation unit 603 of the fourth embodiment divides the image range of the frame into the first area segment and the second area segment similar to those described above, and the first designation similar to that in the third embodiment. Divide the area into an area and a second designated area. Then, the region segmentation unit 603 sends information indicating the first region segment (subject region segment) to the optical flow detection unit 609, and information indicating the background region segment that is an example of the second region segment. Send to 619. Therefore, in the present embodiment, the optical flow detection unit 609 performs processing for the subject region segment, and the optical flow detection unit 619 performs processing for the background region segment. In the case of the third embodiment, the region segmentation unit 603 sends information indicating the first designated area and the second designated area to both the optical flow detection units 609 and 619.

Hereinafter, the optical flow detection unit 609 will be described as an example.
The area division unit 604 of the optical flow detection unit 609 divides the image of the subject area section in the image data of the frame for each predetermined area (the grid described above). The feature point extraction unit 605 obtains a feature amount for each grid divided by the region division unit 604, and arranges the feature point at a position where the feature is determined to be the largest. As a result, the position of one feature point is determined for each grid. The feature amount is the same as that in the first embodiment described above.

  Similar to the example of FIG. 9 described above, the vector detection unit 606 detects and outputs motion vectors by the number of feature points (that is, the number of feature extraction grids in the subject region segment). In addition, the tracking destination feature point calculation unit 607 calculates tracking destination feature points as in the example of FIG. 9 described above, information on the positions of the calculated tracking destination feature points, and an area to which each tracking destination feature point belongs. Information indicating the classification is stored in an internal memory.

The resetting unit 608 of the fourth embodiment counts the number of tracking destination feature points calculated in the subject area segment 200 of the first designated area 701, and in the subject area section 200 of the first designated area 701. It is determined whether or not the number of tracking destination feature points exceeds a predetermined number. When the number of tracking destination feature points in the subject area segment 200 in the first designated area 701 exceeds a predetermined number, the resetting unit 108 tracks in the subject area section 200 in the first designated area 701. The previous feature points are deleted up to a number not exceeding a predetermined number. Furthermore, the resetting unit 608 of the fourth embodiment sets the tracking destination feature points deleted from the subject area segment 200 of the first designated area 701 to the subject area section 200 of the second designated area 702. Reset (complement).
In the optical flow detection unit 619 in the case of the fourth embodiment, the same processing as that of the optical flow detection unit 609 is performed except that the processing target is the background area section.

  As described above, the image processing apparatus according to the fourth embodiment performs optical flow detection processing for each of the first region segment (subject region segment) and the second region segment (background region segment). In the fourth embodiment, as in the third embodiment, when the number of tracking destination feature points in the first designated area exceeds a predetermined number, the tracking destination feature points are deleted up to a number not exceeding the predetermined number, The tracking destination feature point is reset within the same area section of the second designated area. As a result, according to the image processing apparatus of the fourth embodiment, even if the tracking destination feature points are unevenly distributed within the frame image range, the reduction in vector detection accuracy is suppressed, and a predetermined period such as one frame period is determined. Processing can be completed within a period of time.

  The first and third embodiments are examples in which the configuration for vector detection is one set, and the second and fourth embodiments are examples in which the configuration for vector detection is a plurality of sets (two sets). ing. Basically, the configuration of the first or third embodiment may be used to achieve the purpose of suppressing the decrease in vector detection accuracy. However, for example, when the region divided as the region division is divided not only into the subject region such as the face but also into various other region divisions, or when applied to other purposes, the second and fourth embodiments. It is preferable to use a configuration in which these configurations are appropriately combined.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  101: Image sensor unit, 102: Memory, 103: Region division unit, 104: Feature point extraction unit, 105: Vector detection unit, 106: Region segmentation unit, 107: Tracking destination feature point calculation unit, 108: Reset unit

Claims (18)

A dividing means for dividing the frame image into a plurality of regions;
Extraction means for extracting feature points of an image in each of the divided areas;
Detecting means for detecting a motion vector based on the feature points;
A dividing means for dividing the plurality of divided areas into at least two area sections;
Calculation means for calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame;
If the position of the tracking destination feature point is biased in the area section, the position of the tracking destination feature point is reset evenly in the area section where the position of the tracking destination feature point is biased. Setting means;
An image processing apparatus comprising: A segmentation means for segmenting the image of the frame into at least two region segments;
Dividing means for dividing the image into a plurality of areas for each of the area sections;
Extracting means for extracting feature points of an image in each of the divided areas for each of the area sections;
Detecting means for detecting a motion vector based on the feature points for each of the region sections;
Calculation means for calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame for each region section;
If the position of the tracking destination feature point is biased in the area section, the position of the tracking destination feature point is reset evenly in the area section where the position of the tracking destination feature point is biased. Setting means;
An image processing apparatus comprising: The resetting means includes
Dividing the area section into a plurality of counting areas, counting the tracking destination feature points for each counting area,
The tracking destination feature points that exceed the target number are deleted from the counting area in which the number of the counted tracking destination feature points exceeds the target number, and the deleted tracking destination feature points are counted. The image processing apparatus according to claim 1, wherein the number of the previous feature points is reset in a count area smaller than the target number.   The said resetting means sets the number obtained by equally dividing the number of the tracking destination feature points in the area section based on the number of the counting areas as the target number. The image processing apparatus described.   The resetting means includes the tracking destination feature point at a position closest to the center of gravity position of each position of the tracking destination feature point in the counting area in which the number of tracking destination feature points exceeds the target number. The image processing apparatus according to claim 3, wherein the image processing apparatus is set as a tracking destination feature point to be deleted.   The image processing apparatus according to claim 3, wherein the resetting unit resets the tracking destination feature point at a position having the largest feature amount in the counting area. A dividing means for dividing the frame image into a plurality of regions;
Extraction means for extracting feature points of an image in each of the divided areas;
Detecting means for detecting a motion vector based on the feature points;
A dividing means for dividing the plurality of divided areas into at least two area sections and at least two designated areas of a first designated area and a second designated area;
Calculation means for calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame;
Counting means for counting the number of the tracking destination feature points in the first designated area for each of the area sections;
The tracking destination feature points are deleted from the region classification of the first designated area where the number of the tracking destination feature points exceeds a predetermined number to a number not exceeding the predetermined number, and the second designated area Resetting means for resetting the deleted number of tracking destination feature points in the area section in
An image processing apparatus comprising: Partitioning means for partitioning the image of the frame into at least two region sections and at least two designated areas of the first designated area and the second designated area;
Dividing means for dividing the image into a plurality of areas for each of the area sections;
Extracting means for extracting feature points of an image in each of the divided areas for each of the area sections;
Detecting means for detecting a motion vector based on the feature points for each of the region sections;
Calculation means for calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame for each region section;
Counting means for counting the number of the tracking destination feature points in the first designated area for each of the area sections;
For each region segment, the tracking destination feature points are deleted from the first designated area where the number of tracking destination feature points exceeds a predetermined number to a number not exceeding the predetermined number, Resetting means for resetting the deleted number of tracking destination feature points in the designated area;
An image processing apparatus comprising: The sorting means includes
An area where input is performed at the end of the frame image input order is set as the first designated area,
9. The image processing apparatus according to claim 7, wherein a section in which input is initially performed in the input order is set as the second designated area.   The resetting means, when the same area section as the area section to which the deleted tracking destination feature point belongs is included in the second designated area, the same area section in the second designated area. The image processing apparatus according to claim 7, wherein the tracking destination feature point is reset.   The said reset means performs the said deletion in order from the tracking destination feature point with few feature-values in each tracking destination feature point of the said 1st designated area, The any one of Claim 7 to 10 characterized by the above-mentioned. The image processing apparatus according to item.   The resetting means includes a tracking destination feature point at a position closest to a centroid position of each position of the tracking destination feature point in the first designated area where the number of the tracking destination feature points exceeds the predetermined number. The image processing apparatus according to any one of claims 7 to 11, wherein the tracking target feature point to be deleted is set.   The image according to any one of claims 7 to 12, wherein the resetting unit resets the tracking destination feature point at a position having the largest feature amount in the second designated area. Processing equipment. A division step of dividing the frame image into a plurality of regions;
An extraction step of extracting feature points of an image in each of the divided areas;
A detection step of detecting a motion vector based on the feature points;
A dividing step of dividing the plurality of divided areas into at least two area sections;
A calculation step of calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame;
If the position of the tracking destination feature point is biased in the area section, the position of the tracking destination feature point is reset evenly in the area section where the position of the tracking destination feature point is biased. A setting process;
An image processing method for an image processing apparatus, comprising: A segmentation step of segmenting the image of the frame into at least two region segments;
A division step of dividing the image into a plurality of regions for each of the region divisions;
An extraction step of extracting feature points of an image in each of the divided areas for each of the area sections;
A detection step of detecting a motion vector based on the feature points for each of the region sections;
A calculation step of calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame for each region section;
If the position of the tracking destination feature point is biased in the area section, the position of the tracking destination feature point is reset evenly in the area section where the position of the tracking destination feature point is biased. A setting process;
An image processing method for an image processing apparatus, comprising: A division step of dividing the frame image into a plurality of regions;
An extraction step of extracting feature points of an image in each of the divided areas;
A detection step of detecting a motion vector based on the feature points;
A dividing step of dividing the plurality of divided areas into at least two area sections and at least two designated areas of a first designated area and a second designated area;
A calculation step of calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame;
A counting step for counting the number of the tracking destination feature points in the first designated area for each region section;
The tracking destination feature points are deleted from the region classification of the first designated area where the number of the tracking destination feature points exceeds a predetermined number to a number not exceeding the predetermined number, and the second designated area A resetting step for resetting the deleted number of tracking destination feature points in the area section in
An image processing method for an image processing apparatus, comprising: A partitioning step of partitioning the image of the frame into at least two region partitions and at least two specified regions of the first specified region and the second specified region;
A division step of dividing the image into a plurality of regions for each of the region divisions;
An extraction step of extracting feature points of an image in each of the divided areas for each of the area sections;
A detection step of detecting a motion vector based on the feature points for each of the region sections;
A calculation step of calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame for each region section;
A counting step for counting the number of the tracking destination feature points in the first designated area for each region section;
For each region segment, the tracking destination feature points are deleted from the first designated area where the number of tracking destination feature points exceeds a predetermined number to a number not exceeding the predetermined number, A resetting step of resetting the deleted number of tracking destination feature points in the designated area;
An image processing method for an image processing apparatus, comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 13.

Images