JP2017229061A - Image processing apparatus, control method therefor, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of accurate region tracking and a control method for the same.SOLUTION: On the basis of a specified position, if distance information satisfying a reliability condition is obtained for a region including the specified position, an image region for extracting a feature amount used for region tracking is identified using the distance information, and, if it is not obtained, it is identified without using the distance information.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image processing apparatus, a control method therefor, and an imaging apparatus, and more particularly to a technique for tracking a specific area between images.

  By searching a region similar to the region in the image captured at a certain time t in one or more images captured after the time t, it is possible to detect the temporal movement of the region. For example, by detecting the movement of the area (face area) of a specific subject in movie shooting, the exposure condition is dynamically changed so that the specific subject is kept in focus or the exposure of the specific subject is appropriate. (Patent Document 1).

JP 2005-318554 A

  When searching for a region similar to a specific image region, a technique called matching is generally used. For example, in template matching, the pixel pattern of a certain image area is set as a feature amount (template), and the degree of similarity (for example, the correlation amount) is calculated for each position while relatively changing the position of the template in the search area of another image. Then, the position with the highest similarity is detected. If it is determined that the similarity at the detected position is sufficiently high, it is estimated that an image region having the same pattern as the template exists at that position.

  The search accuracy by matching largely depends on how to set feature quantities used for matching. For example, when tracking a face area of a specific person, if a pixel pattern of an area that includes only a part of the face area is set as a feature amount, erroneous detection is likely to occur because the face feature amount is small. Conversely, if a pixel pattern that includes the entire face area but has a large proportion of the peripheral area of the face area (for example, the background area) is set as the feature amount, the contribution of the background similarity increases, and erroneous detection is likely to occur. .

  The present invention has been made in view of the above-described problems of the prior art, and an object of the present invention is to provide an image processing apparatus capable of accurately tracking an area and a control method thereof.

  The above-described object is achieved by using a specifying unit that specifies an image region for extracting a feature amount in an image based on a specified position, an extraction unit that extracts a feature amount from the image region, and a feature amount. Search means for searching for a region similar to the image region in a plurality of time-sequential images, and the specifying unit obtains distance information that satisfies the reliability condition for the region including the designated position. This is achieved by an image processing apparatus characterized by specifying an image region using distance information without using distance information if it is not obtained.

  According to the present invention, it is possible to provide an image processing apparatus capable of accurately tracking an area and a control method thereof.

1 is a block diagram showing a functional configuration example of a digital camera according to an embodiment. The figure which shows the pixel array example of the image pick-up element of FIG. The block diagram which shows the function structural example of the tracking part of FIG. The figure regarding the template matching in 1st Embodiment The figure regarding the histogram matching in 1st Embodiment The figure regarding the acquisition method of the object distance in a 1st embodiment. The figure which shows typically the identification method of the to-be-photographed object area | region in 1st Embodiment. Flowchart of imaging processing in the first embodiment Flowchart of subject tracking processing in the first embodiment Flowchart of imaging processing in the second embodiment Flowchart of subject tracking processing in the second embodiment The figure which shows typically the update determination method of the feature-value in 2nd Embodiment.

  Hereinafter, a digital camera as an example of an image processing apparatus according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention can also be implemented in an electronic device that does not have a photographing function. Examples of electronic devices that can implement the present invention include, but are not limited to, digital cameras, mobile phones, tablet terminals, game machines, personal computers, navigation systems, home appliances, and robots.

● <First embodiment>
(Configuration of imaging device)
FIG. 1 is a block diagram illustrating a functional configuration example of a digital camera 100 according to the first embodiment of the present invention. The digital camera 100 can shoot and record moving images and still images. Each functional block in the digital camera 100 is connected to be communicable with each other via a bus 160. The operation of the digital camera 100 is realized by the main control unit 151 (central processing unit) executing a program to control each functional block.

  The digital camera 100 according to the present embodiment can acquire distance information of a photographed subject. The distance information may be, for example, a distance image representing the distance of the subject corresponding to the pixel value. The distance information may be acquired by any method, but in the present embodiment, the distance information is acquired based on the parallax image. Although the parallax image acquisition method is not limited, in the present embodiment, the parallax image is acquired using the imaging element 141 including a plurality of photoelectric conversion elements sharing one microlens. Note that a parallax image may be acquired by using the digital camera 100 as a multi-view camera such as a stereo camera, and data of a parallax image captured by an arbitrary method may be acquired from a storage medium or an external device.

  In addition, the digital camera 100 includes a tracking unit 161 that realizes a subject tracking function by continuously searching for a region similar to the designated subject region. The tracking unit 161 generates distance information from the parallax image and uses it for searching for a subject area. Details of the configuration and operation of the tracking unit 161 will be described later.

  The photographic lens 101 (lens unit) includes a fixed first group lens 102, a zoom lens 111, an aperture 103, a fixed third group lens 121, a focus lens 131, a zoom motor 112, an aperture motor 104, and a focus motor 132. The fixed first group lens 102, the zoom lens 111, the diaphragm 103, the fixed third group lens 121, and the focus lens 131 constitute a photographing optical system. In addition, although the lenses 102, 111, 121, and 131 are illustrated as one lens for convenience, each may be configured by a plurality of lenses. Further, the photographing lens 101 may be configured as a detachable interchangeable lens.

The diaphragm control unit 105 controls the operation of the diaphragm motor 104 that drives the diaphragm 103 to change the aperture diameter of the diaphragm 103.
The zoom control unit 113 controls the operation of the zoom motor 112 that drives the zoom lens 111 to change the focal length (view angle) of the photographing lens 101.

  The focus control unit 133 calculates the defocus amount and the defocus direction of the photographing lens 101 based on the phase difference between a pair of focus detection signals (A image and B image) obtained from the image sensor 141. The focus control unit 133 converts the defocus amount and the defocus direction into the drive amount and drive direction of the focus motor 132. The focus control unit 133 controls the operation of the focus motor 132 based on the drive amount and the drive direction, and controls the focus state of the photographic lens 101 by driving the focus lens 131. In this way, the focus control unit 133 performs automatic focus detection (AF) using a phase difference detection method. The focus control unit 133 may execute contrast detection AF based on the contrast evaluation value obtained from the image signal obtained from the image sensor 141.

  A subject image formed on the imaging surface of the image sensor 141 by the photographing lens 101 is converted into an electrical signal (image signal) by a photoelectric conversion element included in each of a plurality of pixels arranged in the image sensor 141. In the present embodiment, m pixels in the horizontal direction and n pixels (n and m are plural) are arranged in a matrix in the image sensor 141, and each pixel has two photoelectric conversion elements (photoelectric conversion regions). ) Is provided. Signal reading from the image sensor 141 is controlled by the sensor control unit 143 in accordance with an instruction from the main control unit 151.

(Pixel arrangement of the image sensor 141)
FIG. 2 is a diagram schematically showing an example of pixel arrangement in the image sensor 141, and representatively shows an area composed of 16 pixels of 4 pixels in the horizontal direction and 4 pixels in the vertical direction. Each pixel of the image sensor 141 is provided with one microlens 210 and two photoelectric conversion elements 201 and 202 that receive light through the microlens 210. In the example of FIG. 2, since the two photoelectric conversion elements 201 and 202 are arranged in the horizontal direction, each pixel has a function of dividing the pupil region of the photographing lens 101 in the horizontal direction.

  In addition, the image sensor 141 is provided with a primary color Bayer array color filter having a repetition unit of four pixels of 2 pixels in the horizontal direction and 2 pixels in the vertical direction. The color filter has a configuration in which rows in which R (red) and G (green) are repeatedly arranged in the horizontal direction and rows in which G and B (blue) are repeatedly arranged in the horizontal direction are alternately arranged. The pixel 200R provided with the red filter is called a red pixel, the pixel 200G provided with the G (green) filter is called a green pixel, and the pixel 200B provided with a B (blue) filter is called a blue pixel.

  In the following description, the first photoelectric conversion element 201 is called an A pixel, the second photoelectric conversion element 202 is called a B pixel, a signal read from the A pixel is called an A signal, and a signal read from the B pixel is called a B signal. is there. An image composed of A signals obtained from a plurality of pixels included in a certain area and an image composed of B signals constitute a set of parallax images. Therefore, the digital camera 100 can generate two parallax images by one shooting. Further, by adding the A signal and the B signal for each pixel, a signal similar to that of a general pixel having no pupil division function can be obtained. Hereinafter, this added signal may be referred to as an A + B signal, and an image composed of the A + B signal may be referred to as a captured image.

  Thus, from one pixel, the output (A signal) of the first photoelectric conversion element 201, the output (B signal) of the second photoelectric conversion element 202, and the first photoelectric conversion element 201 and the second photoelectric conversion element. Three types of signals, that is, the addition output (A + B signal) of the conversion element 202 can be read out. The A signal (B signal) may be obtained by subtracting the B signal (A signal) from the A + B signal instead of reading.

  Note that the photoelectric conversion elements may be divided and arranged in the vertical direction, or pixels having different division directions of the photoelectric conversion elements may be mixed. Further, the photoelectric conversion element may be divided in both vertical and horizontal directions. Moreover, you may divide | segment into three or more in the same direction.

  Returning to FIG. 1, the image signal read from the image sensor 141 is supplied to the signal processing unit 142. The signal processing unit 142 applies signal processing such as noise reduction processing, A / D conversion processing, and automatic gain control processing to the image signal and outputs the image signal to the sensor control unit 143. The sensor control unit 143 stores the image signal received from the signal processing unit 142 in a RAM (Random Access Memory) 154.

  The image processing unit 152 applies predetermined image processing to the image data stored in the RAM 154. Image processing applied by the image processing unit 152 includes so-called development processing such as white balance adjustment processing, color interpolation (demosaic) processing, and gamma correction processing, signal format conversion processing, scaling processing, subject detection processing, subject recognition processing, and the like. However, it is not limited to these. The image processing unit 152 can also generate information related to subject brightness and the like for use in automatic exposure control (AE). The results of subject detection processing and subject recognition processing may be used for other image processing (for example, white balance adjustment processing). Note that when performing contrast detection AF, the image processing unit 152 may generate an AF evaluation value. The image processing unit 152 stores the processed image data in the RAM 154.

  When recording the image data stored in the RAM 154, the main control unit 151 generates a data file corresponding to the recording format, for example, by adding a predetermined header to the image processing data. At this time, the main control unit 151 compresses the amount of information by encoding image data in the compression / decompression unit 153 as necessary. The main control unit 151 records the generated data file on a recording medium 157 such as a memory card.

When displaying image data stored in the RAM 154, the main control unit 151 uses the image data in the RAM 154 as a video memory after the image processing unit 152 scales the image data so as to fit the display size on the display unit 150. Write to area (VRAM area).
The display unit 150 reads display image data from the VRAM area of the RAM 154 and displays it on a display device such as an LCD or an organic EL display.

The digital camera 100 according to the present embodiment functions as an electronic viewfinder (EVF) by immediately displaying the captured moving image on the display unit 150 during moving image shooting (in shooting standby state or during moving image recording). Let A moving image and its frame image displayed when the display unit 150 functions as an EVF are referred to as a live view image or a through image.
In addition, when taking a still image, the digital camera 100 displays the still image taken immediately before on the display unit 150 for a certain period of time so that the user can check the shooting result. These display operations are also realized by the control of the main control unit 151.

  The operation unit 156 is a switch, button, key, touch panel, or the like for the user to input an instruction to the digital camera 100. An input through the operation unit 156 is detected by the main control unit 151 through the bus 160, and the main control unit 151 controls each unit in order to realize an operation according to the input.

  The main control unit 151 has one or more programmable processors such as a CPU and an MPU, for example, and controls the respective units by reading the program stored in the storage unit 155 into the RAM 154 and executing it, thereby realizing the functions of the digital camera 100. To do. The main control unit 151 also executes an AE process that automatically determines an exposure condition (shutter speed or accumulation time, aperture value, sensitivity) based on the subject luminance information. Information on the subject brightness can be acquired from the image processing unit 152, for example. The main control unit 151 can also determine the exposure condition based on the area of a specific subject such as a person's face.

  The main control unit 151 fixes the aperture during moving image shooting, and controls the exposure based on the electronic shutter speed (accumulation time) and the magnitude of the gain. The main control unit 151 notifies the sensor control unit 143 of the determined sense of accumulation and the magnitude of the gain. The sensor control unit 143 controls the operation of the image sensor 141 so that shooting is performed according to the notified exposure condition.

  In the present embodiment, a total of three images, that is, a set of parallax images and a captured image can be acquired by one shooting, and the image processing unit 152 processes each image and writes it to the RAM 154. The tracking unit 161 obtains subject distance information from a set of parallax images and uses it for subject tracking processing for a captured image. When the subject tracking is successful, the tracking unit 161 outputs information on the position of the subject region in the captured image and information on the reliability.

  The result of subject tracking can be used for automatic setting of a focus detection area, for example. As a result, a tracking AF function for a specific subject area can be realized. Further, AE processing can be performed based on the luminance information of the focus detection area, or image processing (for example, gamma correction processing or white balance adjustment processing) can be performed based on the pixel value of the focus detection area. Note that the main control unit 151 may superimpose and display an index (for example, a rectangular frame surrounding the area) indicating the current position of the subject area on the display image.

The battery 159 is managed by the power management unit 158 and supplies power to the entire digital camera 100.
The storage unit 155 stores a program executed by the main control unit 151, setting values necessary for executing the program, GUI data, user setting values, and the like. For example, when a transition from the power OFF state to the power ON state is instructed by operating the operation unit 156, the program stored in the storage unit 155 is read into a part of the RAM 154, and the main control unit 151 executes the program.

(Configuration and operation of the tracking unit)
FIG. 3 is a block diagram illustrating a functional configuration example of the tracking unit 161. The tracking unit 161 includes a matching unit 1610, a feature extraction unit 1620, and a distance map generation unit 1630. The tracking unit 161 identifies an image region (subject region) to be tracked from a designated position, and extracts a feature amount from the subject region. Then, an area having a high similarity to the subject area of the previous frame is searched as a subject area using the extracted feature amount in each supplied captured image. In addition, the tracking unit 161 acquires distance information from a pair of parallax images and uses it for specifying a subject area.

  The matching unit 1610 searches for the subject area in the supplied image using the feature amount of the subject area supplied from the feature extraction unit 1620. Although there is no particular limitation on a method for searching for a region based on the feature amount of the image, the matching unit 1610 uses at least one of template matching and histogram matching.

Hereinafter, template matching and histogram matching will be described.
Template matching is a technique in which a pixel pattern is set as a template and an area having the highest similarity with the template is searched for in the image. As the similarity between the template and the image region, a correlation amount such as a sum of absolute differences between corresponding pixels can be used.

  FIG. 4A schematically shows a template 301 and a configuration example 302 thereof. When performing template matching, the feature extraction unit 1620 supplies color (hue) information used for the template as a feature amount to the matching unit 1610. Here, the template 301 has a horizontal pixel number W and a vertical pixel number H, and binarization is performed in which pixels that match the feature amount and pixels that do not match the feature amount are respectively replaced with different fixed values. . The matching unit 1610 performs pattern matching using the binarized template 301.

Therefore, the feature quantity T (i, j) of the template 301 used for pattern matching can be expressed by the following equation (1) when the coordinates in the template 301 are expressed in a coordinate system as shown in FIG.
T (i, j) = {T (0, 0), T (1, 0), ..., T (W-1, H-1)} (1)

  FIG. 4B shows an example of a subject area search area 303 and its configuration 305. The search area 303 is a range where pattern matching is performed in the image, and may be the whole or a part of the image. The coordinates in the search area 303 are represented by (x, y). Also in the search area 303, binarization is performed in which pixels that match the feature amount and pixels that do not match the feature value are respectively replaced with different fixed values. The area 304 has the same size as the template 301 (the number of horizontal pixels W and the number of vertical pixels H), and is an object for calculating the similarity with the template 301.

The feature quantity S (i, j) of the region 304 used for pattern matching can be expressed by the following equation (2) when the coordinates in the template 301 are expressed in a coordinate system as shown in FIG.
S (i, j) = {S (0, 0), S (1, 0), ..., S (W-1, H-1)} (2)

ここで、V(x, y)は、領域３０４の左上頂点の座標(x, y)における評価値を表す。 The collation unit 1610 calculates a sum of absolute difference (SAD) value represented by the following equation (3) as an evaluation value V (x, y) representing the similarity between the template 301 and the region 304.

Here, V (x, y) represents an evaluation value at the coordinates (x, y) of the upper left vertex of the region 304.

  The collation unit 1610 sets the area 304 by one pixel from the upper left of the search area 303 to the right, and when x = (X-1)-(W-1) is reached, next sets x = 0 one pixel at the bottom. The evaluation value V (x, y) is calculated at each position while shifting each. The coordinates (x, y) where the calculated evaluation value V (x, y) indicates the minimum value indicates the position of the region 304 having the pixel pattern most similar to the template 301. The collation unit 1610 detects an area 304 where the evaluation value V (x, y) is the minimum value as a subject area existing in the search area. When the reliability of the search result is low (for example, when the minimum value of the evaluation values V (x, y) exceeds the threshold value), it may be determined that the subject area has not been found.

  Here, an example is shown in which a template that is binarized according to whether or not a color corresponding to a feature amount is used for pattern matching. However, there are many patterns that match each of a plurality of colors included in the feature amount. A valuated template may be used. Also, feature values based on lightness or saturation may be used instead of color feature values. Moreover, although the example which uses SAD as an evaluation value of similarity was shown, you may use other evaluation values, for example, normalized cross-correlation (NCC: Normalized Cross-Correlation), ZNCC, etc.

Next, details of histogram matching will be described.
FIG. 5A shows an example of the subject area 401 and its histogram 402. When histogram matching is performed, information on the color (hue) used for the color histogram is supplied from the feature extraction unit 1620 to the matching unit 1610 as a feature amount. If the number of bins in the color histogram is M (M is an integer equal to or greater than 2), the color histogram p (m) 402 generated by the matching unit 1620 can be expressed by the following equation (4).
p (m) = {p (0), p (1), ..., p (M-1)} (4)
Note that p (m) is a normalized histogram. This color histogram p (m) has only bins corresponding to colors included in the feature amount. That is, if the number of bins is M, the number of colors supplied as the feature amount is also M.

FIG. 5B shows an example of a subject area search area 403 and a color histogram 405. The color histogram q (m) 405 of the region 404 is expressed by the following equation (5), where the number of bins is M.
q (m) = {q (0), q (1), ..., q (M-1)} (5)
Note that q (m) is a normalized histogram. The color histogram q (m) is also a histogram having only bins corresponding to the colors included in the feature amount.

ここで、D(x, y)は、領域４０４の左上頂点の座標(x, y)における評価値を表す。 The tracking unit 161 uses a Bhattacharyya coefficient expressed by the following equation (6) as an evaluation value D (x, y) of similarity between the color histogram p (m) of the subject region 401 and the color histogram q (m) of the region 404. Can be calculated.

Here, D (x, y) represents an evaluation value at the coordinates (x, y) of the upper left vertex of the region 404.

  Similar to the template matching, the matching unit 1610 calculates the evaluation value D (x, y) while shifting the region 404 within the search region 403. The coordinates (x, y) at which the calculated evaluation value D (x, y) is the maximum value indicate the position of the region 404 that is most similar to the subject region 401. The collation unit 1610 detects an area 404 where the evaluation value D (x, y) is the maximum value as a subject area existing in the search area.

  Here, an example is shown in which the feature amount of color is used for histogram matching, but the feature amount of hue or saturation may be used. Further, although an example in which the Bhattacharyya coefficient is used as the evaluation value of the similarity is shown, other evaluation values such as a histogram intersection may be used.

  The distance map generation unit 1630 calculates a subject distance from a set of parallax images and generates a distance map. The distance map is one piece of distance information in which each pixel represents the subject distance, and is sometimes called a depth map, a depth image, or a distance image. The distance map may be generated without using a parallax image. For example, the subject distance for each pixel may be obtained by obtaining the position of the focus lens 131 at which the contrast evaluation value is maximized for each pixel, and a distance image may be generated.

  A method for calculating the subject distance will be described with reference to FIG. In FIG. 6, assuming that an A image 1151a and a B image 1151b are obtained, the light beam is refracted as shown by a solid line from the focal length of the photographing lens 101 and the distance information between the focus lens 131 and the imaging element 141. I understand. Therefore, it can be seen that the subject in focus is at the position 1152a. Similarly, it can be seen that there is a subject in focus at position 1152b when the B image 1151c is obtained for the A image 1151a and at position 1152c when the B image 1151d is obtained. As described above, in each pixel, the distance information of the subject at the pixel position can be calculated from the relative position between the A image including the pixel and the corresponding B image.

  For example, assume that an A image 1151a and a B image 1151d are obtained in FIG. In this case, the distance 1153 from the intermediate pixel 1154 corresponding to half of the image shift amount to the subject position 1152c or the defocus amount corresponding to the distance 1153 is stored as the pixel value of the pixel 1154. In this manner, the distance information of the subject can be calculated for each pixel, and a distance map can be generated.

  Note that the distance map may be generated by dividing the image into minute regions and calculating the defocus amount for each minute region. What is necessary is just to produce | generate A image and B image from the pixel contained in a micro area | region, detect the phase difference (image shift | offset | difference amount) by correlation calculation, and to convert into defocus amount. Even in this case, in the distance map generated, each pixel indicates the subject distance, but the pixels included in the minute region indicate the same subject distance. The distance map generation unit 1630 supplies the generated distance map to the feature extraction unit 1620.

  Although the distance map may be generated for the entire image, it may be generated only for the partial area designated for extracting the feature amount.

The feature extraction unit 1620 extracts a feature amount used for tracking (searching) the subject area from the subject area.
When subject tracking is performed, generally, a user specifies a position in an image to be tracked before starting the tracking. For example, in the shooting standby state, the position in the image displayed on the display unit 150 can be designated by the user through the operation unit 156. For example, if the display unit 150 is a touch display, the main control unit 151 acquires the coordinates of the tap operation or the coordinates of the position specified by the cursor that can move on the image through the operation of the operation unit 156. Information on the designated position is input from the main control unit 151 to the feature extraction unit 1620.

  A method by which the feature extraction unit 1620 specifies a subject area from which a feature amount is extracted will be described with reference to FIG. FIG. 7A shows the captured image, and the designated position 503 indicates the coordinates in the face 501 of the person. Further, it is assumed that the house 502 as the background has color information similar to the face 501 of the person.

The feature extraction unit 1620 generates a color histogram Hin _{in the} subject area using a predetermined area including the designated position 503, for example, a predetermined rectangular area centered on the designated position 503 as a temporary subject area. In addition, the feature extraction unit 1620 uses all regions other than the temporary subject region as reference regions, and generates a color histogram H _Out regarding this reference region. The color histogram represents the frequency of colors included in the image. Here, as an example, the pixel value is converted from the RGB color space to the HSV color space, and a color histogram for the hue (H) is generated. However, other types of color histograms may be generated.

Then, the feature extraction unit 1620 calculates an information amount I (a) represented by the following equation (7).
I (a) = -log ₂ (H _in (a) / H _out (a)) (7)
Here, a is an integer indicating the bin number. The absolute value of the information amount I (a) decreases as the ratio of the number of pixels of the color corresponding to the bin included in the temporary subject region to the number of pixels of the color corresponding to the bin included in the reference region increases. . That is, as the value of the information amount I (a) is smaller, the color corresponding to the information amount I (a) has a larger proportion of the provisional subject area than the proportion of the reference region. It is likely that the color is a characteristic color of the subject area. The feature extraction unit 1620 calculates the information amount I (a) for all bins.

  The feature extraction unit 1620 replaces each calculated information amount I (a) with any value within a specific range (for example, a range of 8-bit values (0 to 255)). At this time, the feature extraction unit 1620 substitutes a larger value as the value of the information amount I (a) is smaller. Then, the feature extraction unit 1620 replaces the value of each pixel included in the captured image with a value in which the information amount I (a) corresponding to the color of the pixel is replaced.

  Through such processing, the feature extraction unit 1620 generates a subject map based on the color information. FIG. 7B shows an example of a subject map, in which a pixel close to white has a high probability of being a subject pixel, and a pixel close to black has a low probability of being a subject pixel. For convenience, the subject map is shown as a binary image in FIG. 7B, but it is actually a multi-tone image. Since a part of the house 502 as a background of the captured image has a color similar to that of the person's face 501, the subject map based on the color information does not sufficiently identify the person's face 501. A rectangular area 504 shown in FIG. 7C shows an example of an object area that is finally set (updated) based on an area in which the pixel value is equal to or greater than a predetermined threshold in the object map.

  When the feature amount extracted from such a subject area is used, the possibility that the human face 501 can be accurately tracked is reduced. Therefore, in the present embodiment, the distance map generated by the distance map generation unit 1630 is used to improve the accuracy of the subject area set based on the color information. FIG. 7D shows a distance map generated for the captured image shown in FIG. 7A, with the subject distance corresponding to the designated position 503 as a reference, white as the subject distance difference is small and black as the subject distance is large. An example of conversion is shown below. For convenience, the distance map is shown as a binary image in FIG. 7D, but it is actually a multi-tone image.

  The feature extraction unit 1620 generates a subject map that takes distance information into consideration, for example, by multiplying the value of the corresponding pixel of the subject map based on the distance map and color information. FIG. 7E shows an example of a subject map that takes distance information into account (that is, based on both color information and distance information). In the subject map shown in FIG. 7E, the human face 501 and the house 502 as the background can be distinguished with high accuracy. A rectangular region 505 illustrated in FIG. 7F illustrates an example of a subject region set based on a region having a pixel value equal to or greater than a predetermined threshold in the subject map illustrated in FIG. The rectangular area 505 is a rectangular area circumscribing the person's face 501, and the number of background pixels included in the area is very small. When the feature amount extracted from such a subject area is used, the possibility that the human face 501 can be accurately tracked increases.

  In this way, by referring to the distance information in addition to the color information related to the predetermined range including the designated position, it is possible to set a more accurate subject area and extract a feature quantity suitable for accurate tracking. become.

  In addition, at the time when the position of the tracking target is specified, there is a case where valid distance information (having sufficient reliability to refer to) is not obtained for the specified position and its neighboring area. For example, distance map generation is performed only for a specific area (for example, a focus detection area), and the specified position is out of the specified area, or the specified position is not in focus, and the reliability of the distance information is low. Cases can be considered.

  Therefore, the feature extraction unit 1620 sets the subject region by referring to the distance information in addition to the color information if distance information having sufficient reliability to refer to the vicinity of the designated position (temporary subject region) is obtained. To do. On the other hand, when the distance information having sufficient reliability to refer to the vicinity of the designated position (temporary subject area) is not obtained, the feature extraction unit 1620 selects the subject area based on the color information without referring to the distance information. Set. The distance information having sufficient reliability to be referred to is, for example, the distance obtained when the temporary subject area is in focus or close to focus (that is, the defocus amount is equal to or less than a predetermined threshold). It may be information, but is not limited to this.

(Processing flow of the imaging device)
With reference to the flowcharts of FIGS. 8 and 9, the moving image shooting operation with subject tracking processing by the digital camera 100 of the present embodiment will be described. The moving image shooting operation is executed during shooting standby or when recording a moving image. Note that although details such as resolution of images (frames) to be handled differ between shooting standby and moving image recording, the contents of processing related to subject tracking are basically the same, and therefore will be described without particular distinction below. .

In step S801, the main control unit 151 determines whether the power of the digital camera 100 is ON. If it is not determined to be ON, the main control unit 151 ends the process. If it is determined to be ON, the process proceeds to step S802.
In step S802, the main control unit 151 controls each unit, executes an imaging process for one frame, and advances the process to step S803. Here, a set of parallax images and a captured image for one screen are generated and stored in the RAM 154.

  In step S803, the main control unit 151 causes the tracking unit 161 to execute subject tracking processing. Details of the processing will be described later. Note that the position and size of the subject area are notified from the tracking unit 161 to the main control unit 151 by subject tracking processing. The main control unit 151 sets a focus detection area based on the notified subject area.

  In step S804, the main control unit 151 causes the focus control unit 133 to execute focus detection processing. The focus control unit 133 connects a plurality of A signals obtained from a plurality of pixels arranged in the same row among a plurality of pixels included in the focus detection region of the pair of parallax images, and generates a plurality of A images. The B signals are connected to generate a B image. Then, the focus control unit 133 calculates the correlation amount between the A image and the B image while shifting the relative position between the A image and the B image, and determines the relative position where the similarity between the A image and the B image is the highest. Obtained as the phase difference (deviation amount) between the A and B images. Further, the focus control unit 133 converts the phase difference into a defocus amount and a defocus direction.

  In step S805, the focus control unit 133 drives the focus motor 132 according to the lens drive amount and drive direction corresponding to the defocus amount and defocus direction obtained in step S804, moves the focus lens 131, and returns the process to step S801.

  Thereafter, the processes of S802 to S805 are repeatedly executed until it is not determined in S801 that the power switch is ON. Thus, the subject area is searched for a plurality of time-series images, and the subject tracking function is realized. In FIG. 8, the subject tracking process is performed every frame, but may be performed every several frames for the purpose of reducing the processing load and power consumption.

(Subject tracking process)
Next, details of the subject tracking process in S803 will be described using the flowchart of FIG.
In step S901, the tracking unit 161 determines whether a subject tracking start instruction has been detected. If it is determined that a start instruction has been received, the process proceeds to step S902. If not, the process proceeds to step S906. The start instruction may be a tracking position designation input from the operation unit 156, for example. Information on the designated position is notified from the main control unit 151. At this time, there is a high possibility that the distance information of the designated position is not obtained or the reliability of the distance information is low because the designated position is out of focus. Therefore, the processing content is different from that after the focus detection processing is performed for the designated position.

  In S902, the tracking unit 161 (feature extraction unit 1620) determines whether effective (highly reliable) distance information is obtained for the designated position and its vicinity, and if it is determined that the information is obtained, the process proceeds to S904. If it is not determined that it has been obtained, the process proceeds to S903.

  In step S903, the tracking unit 161 (feature extraction unit 1620) specifies the subject area from the specified position using only the color information as described above, extracts the feature amount of the subject area, and advances the process to step S905.

  In step S904, the tracking unit 161 (feature extraction unit 1620) identifies the subject region from the specified position using both the color information and the distance information as described above, and extracts the feature amount (pixel pattern or histogram) of the subject region. The process proceeds to S905.

  In S905, the tracking unit 161 (collation unit 1610) performs matching processing on the search region of the captured image using the feature amount extracted in S903 or S904, and searches for a region having the highest feature amount similarity. . The tracking unit 161 notifies the main control unit 151 of information related to the position and size of the searched area as a tracking result, and ends the tracking process.

  On the other hand, in S906, the tracking unit 161 (feature extraction unit 1620) determines whether or not the most recently extracted feature amount is extracted from the subject area specified using both the color information and the distance information. If the tracking unit 161 (feature extraction unit 1620) determines that the most recently extracted feature value is extracted from the subject area specified using both the color information and the distance information, the process proceeds to S905. If not, the process proceeds to S907.

  In S907, the tracking unit 161 (feature extraction unit 1620) determines whether or not effective distance information has been obtained for the subject area detected by the previous collation, and if it is determined, the process proceeds to S908. If it is not determined that it has been obtained, the process proceeds to S905.

  In S908, the tracking unit 161 (feature extraction unit 1620) respecifies (updates) the subject region from the designated position using both the color information and the distance information, and extracts the feature amount of the updated subject region, as in S904. The process proceeds to S905. Note that the feature amount extracted in the past (for example, extracted in the process of S903 immediately before) may be added to the feature amount extracted in S908.

  In the matching process executed in S905 during the continuation process, the updated feature quantity is used if the feature quantity has been updated in S908, and the most recently extracted feature quantity is continued if the feature quantity has not been updated in S908. Use.

For example, even if the focus detection process for the subject area detected by the previous collation is started, it cannot be said that the reliability of the distance information is high unless the defocus amount is equal to or less than a predetermined threshold. In such a case, it is processed in the procedure of S901, S906, S907, and S905.
If the defocus amount of the tracked subject area is equal to or less than a predetermined threshold, highly reliable distance information can be acquired for the subject area. In such a case, it is processed in the procedure of S901, S906, S907, S908, and S905.
When the subject region is specified using not only color information but also highly reliable distance information, the subject region and the feature amount are updated, and the updated feature amount is used in the subsequent tracking processing. In this case, processing is performed in the steps S901, S906, and S905.

  As described above, according to the present embodiment, when specifying an image region (subject region) to be tracked based on a designated position in the image, by using distance information in addition to the color information of the image, the subject region Accuracy can be improved. Therefore, it is possible to improve the accuracy of the tracking process that uses the feature amount extracted from the subject area.

  If the distance information is not highly reliable, the subject area is identified based on the color information until the reliability becomes high, and the distance information is obtained when highly reliable distance information can be obtained. Furthermore, the subject area is specified again (updated). Therefore, even when a position where distance information is not obtained or a position where the reliability of distance information is low is designated as a tracking target, the accuracy of the tracking process can be improved as time passes.

● <Second Embodiment>
In the first embodiment, when a feature amount can be extracted from a subject region specified based on distance information and color information with high reliability, the feature amount is not updated. As a result, accumulation of drift can be avoided and subject tracking resistant to occlusion can be realized. On the other hand, for example, when the environment in which the subject exists changes, the subject tracking accuracy may decrease when the luminance or hue of the subject changes from when the feature amount is extracted.

  Therefore, the present embodiment is characterized in that when the difference in distance information between the subject area and the surrounding area satisfies a predetermined condition, the feature amount extracted using distance information with high reliability is also updated. . Since the present embodiment can be implemented by the digital camera 100 having the configuration of FIG. 1 as in the first embodiment, the following mainly describes differences in operation from the first embodiment.

With reference to the flowchart of FIG. 10, the moving image shooting operation with subject tracking processing by the digital camera 100 of the present embodiment will be described.
S1001 to S1003 and S1005 to S1006 in FIG. 10 are the same as S801 to S805 in FIG. The present embodiment is different from the first embodiment in that after subject tracking processing is performed in S1003, feature amount update processing is performed in S1004.

Next, details of the feature amount update processing performed in S1004 of FIG. 10 will be described using the flowchart of FIG.
In S1101, the tracking unit 161 (feature extraction unit 1620) determines whether there is a large difference in distance information between the subject region and the surrounding region from the subject region searched in the matching process (S905) and the obtained distance information. Determine whether or not.

  12 (a) and 12 (c) generate different captured images, and FIGS. 12 (b) and 12 (d) generate the captured images of FIGS. 12 (a) and 12 (c), respectively. The distance map is schematically shown. In FIG. 12A, a house 1202 exists as a background with a distance behind the person 1201, and in FIG. 12C, another person 1206 exists in front of the person 1205.

  The distance map of FIG. 12B shows the distance information of each pixel as white as the difference based on the distance information corresponding to the person 1201 that is the target of the tracking process is smaller and black as it is larger. Similarly, the distance map of FIG. 12B shows the distance information of each pixel as white as the difference based on the distance information corresponding to the person 1205 that is the target of the tracking process is smaller and as black as it is larger. In the drawing, FIGS. 12B and 12D show the distance map as a binary image, but it is actually a multi-value grayscale image. The reference distance information may be the average value of distance information or the most frequently used distance information.

  An area 1203 in FIG. 12B and an area 1207 in FIG. 12D are subject areas specified by the subject tracking process in S1003, and an area 1204 and an area 1208 are peripheral areas of the areas 1203 and 1207, respectively. . Here, the center of the subject area is vacant by excluding the subject area from the area that is three times larger than the subject area in the horizontal and vertical sizes by expanding the subject area by the same amount in the vertical and horizontal directions. It is defined as a hollow area. However, this is an example, and other methods may be used.

  The tracking unit 161 (feature extraction unit 1620) extracts a region having distance information similar to the distance information in the main subject region (the difference is within a predetermined range) from the peripheral region, and the extracted region is a peripheral region. It is determined whether the proportion of the area is equal to or greater than a predetermined threshold. The tracking unit 161 (feature extraction unit 1620) ends the feature amount update processing if the ratio is determined to be equal to or greater than the threshold value, and proceeds to S1102 if the ratio is not determined to be equal to or greater than the threshold value.

  The determination in S1101 will be described. If the ratio of the portion having distance information similar to the distance information in the main subject region in the peripheral region is small (for example, less than the threshold value), the subject region that is the tracking target and the background region can be clearly distinguished. It is believed that there is. For this reason, even if the feature amount is updated based on a captured image that satisfies this condition, it is considered that the influence of the background on the updated feature amount is small.

  On the other hand, if the ratio of the portion having distance information similar to the distance information in the main subject region in the peripheral region is large (for example, if it is greater than or equal to a threshold), it is difficult to distinguish the subject region that is the tracking target from the background region The situation is considered.

  In the examples of FIGS. 12B and 12D, the white area is an area having distance information similar to the distance information corresponding to the main subject area. The threshold used in S1101 can be determined experimentally, for example. Here, the ratio of the area having distance information similar to the distance information in the main subject area (the difference is within a predetermined range) in the peripheral area is less than the predetermined threshold in the example shown in FIG. In the example shown in 12 (d), it is determined that the value is equal to or greater than a predetermined threshold.

  In S1102, the tracking unit 161 (feature extraction unit 1620) uses the new feature amount extracted from the subject area searched by the matching process based on the evaluation value (Equation (3)) calculated by the matching process, and the matching process. It is determined whether or not the similarity with the feature amount used for searching the subject area is low. Specifically, the feature extraction unit 1620 determines whether or not the new evaluation value calculated by the collation unit 1610 is higher than the update threshold, or the evaluation value based on the Bhattacharyya coefficient (expression (6)) is another update. It is determined whether it is lower than the threshold value.

  If a feature quantity with a low similarity to the feature quantity used for the search is extracted from the searched subject area, the subject area has been searched, but the appearance of the subject area has changed, and the feature quantity is The need to update is considered high. On the other hand, when a feature amount having a high similarity to the feature amount used for the search is extracted from the searched subject region, the change in the appearance of the subject region is small, and the necessity to update the feature amount is low. it is conceivable that.

  Therefore, the tracking unit 161 (feature extraction unit 1620) proceeds to the processing of S1103 if it is determined in S1102 that the similarity is low, and ends the feature amount update processing if it is not determined that the similarity is low.

In S1103, the tracking unit 161 (feature extraction unit 1620) updates the feature amount used for the collation process with the new feature amount extracted from the searched subject area, similarly to S908. There is no particular restriction on the updating method. For example, the feature value that has been used for the matching process may be completely replaced with a new feature value, or the feature value that has been updated by using the feature value that has been used for the matching process and the new feature value. May be calculated. For example, if the evaluation value is based on the absolute difference sum (equation (3)), the updated feature amount can be obtained according to equation (8).
T (i, j) = Tpre (i, j) x α + Tnow (i, j) x (1-α), 0 ≤ α ≤ 1 (8)
Here, Tpre (i, j) is a feature amount used in the matching process, Tonow (i, j) is a new feature amount, and T (i, j) is an updated feature amount.

If the evaluation value is based on the Bhattacharyya coefficient (equation (6)), the updated feature amount can be obtained according to equation (9).
p (m) = ppre (m) × α + pnow (m) × (1-α), 0 ≦ α ≦ 1 (9)
Here, ppre (m) represents the feature amount used for the matching process, pnow (m) represents the new feature amount, and p (m) represents the updated feature amount.

  In both formulas (8) and (9), α = 0 indicates an update that completely replaces with the newly extracted feature value, and α = 1 indicates that the feature value is not updated. The degree of update α can be adaptively determined according to at least one of the magnitude of the difference in distance information determined in S1101 and the similarity determined in S1102, for example.

  For example, after satisfying the determination conditions in S1101 and S1102, the greater the difference in distance information and the lower the similarity, the smaller the value of the update degree α (the greater the contribution of the new feature value). The updated feature value can be calculated. Further, after satisfying the determination conditions in S1101 and S1102, as the difference in distance information is smaller and the similarity is higher, the value of the update degree α is increased (the contribution of the new feature amount is reduced). The updated feature value can be calculated.

  Furthermore, when an operation for determining the in-focus distance or exposure (for example, an operation corresponding to a shooting preparation instruction or a shooting start instruction and an operation of a shutter button included in the operation unit 156) is detected, the object is detected at that time. The tracking process is likely to be successful. Therefore, when an operation for determining the in-focus distance or the exposure is detected, the determination in S1101 and S1102 is performed so that the feature is easily updated with a new feature amount extracted from the subject area detected at that time. The threshold value used may be changed.

  As described above, according to the present embodiment, the feature amount can be updated when the feature information can be accurately extracted from the subject region using the distance information. Therefore, even when the appearance of the subject area to be tracked changes, the feature amount can be updated without reducing the tracking accuracy, and the subject tracking performance can be further improved.

(Other embodiments)
In the above-described embodiment, the case where subject tracking is performed at the time of shooting has been described. However, if distance information can be acquired, similar subject tracking can be performed even when a moving image is reproduced. In this case, the distance information recorded in the frame of the moving image may be acquired, or if each frame is recorded in the form of a set of parallax images, the distance information is generated from the parallax image, and the parallax image May be combined to generate a moving image frame for playback. Of course, the distance information may be acquired by other methods.

  When subject tracking is performed during reproduction, the tracking result can be used, for example, for controlling a moving image display method. For example, it can be controlled so that the tracked subject area is displayed at the center of the screen, or can be controlled to be scaled and displayed so that the size of the tracked subject area is constant. . In addition, an index (for example, a circumscribed rectangular frame of the subject area) for specifying the subject area being tracked may be superimposed and displayed. Note that these are merely examples, and the tracking results may be used for other purposes.

  The superimposed display of the index for specifying the subject area being tracked may be different depending on whether the subject area is specified with reference to the distance information or when only the color information is specified. For example, when the subject area is specified using only the color information, since the accuracy of the subject area may be low, an indicator of a fixed position and size is displayed. Further, when the subject area is specified with reference to the distance information, the position and size of the index are dynamically changed according to the position and size of the subject area.

  Further, the present invention is applicable not only to moving images but also to shooting and reproduction of a plurality of time-series images such as continuous shooting and interval shooting.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Further, the above-described embodiment is merely a specific example for the purpose of helping understanding of the present invention, and the present invention is not intended to be limited to the above-described embodiment in any sense. All embodiments that fall within the scope of the claims are encompassed by the present invention.

DESCRIPTION OF SYMBOLS 100 ... Digital camera, 101 ... Shooting lens, 131 ... Focus lens, 132 ... Focus motor, 133 ... Focus control part, 141 ... Image sensor, 151 ... Main control part, 152 ... Image processing part, 161 ... Tracking part

Claims (21)

A specifying means for specifying an image region in the image for extracting the feature amount based on the specified position;
Extracting means for extracting a feature amount from the image region;
Search means for searching an area corresponding to the image area in a plurality of time-series images using the feature amount;
The specifying means uses the distance information if the distance information satisfying the reliability condition is obtained for the region including the designated position, and does not use the distance information otherwise. Identify areas,
An image processing apparatus.   The image processing apparatus according to claim 1, wherein the reliability condition is that a defocus amount of an area including the designated position is equal to or less than a threshold value. A specifying means for specifying an image region in the image for extracting the feature amount based on the specified position;
Extracting means for extracting a feature amount from the image region;
Search means for searching an area corresponding to the image area in a plurality of time-series images using the feature amount;
The specifying unit specifies the image region without using the distance information before the distance information satisfying the reliability condition is obtained for the region including the designated position, and the distance information is obtained after the information is obtained. To identify the image area,
An image processing apparatus. The reliability condition is that a defocus amount with respect to an area including the designated position is equal to or less than a threshold value.
The image processing apparatus according to claim 3. When the image region is in a state specified using the distance information from a state specified without using the distance information, the extraction unit updates the feature amount.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The extraction unit, when updating the feature quantity, generates an updated feature quantity from the feature quantity extracted from the image region specified using the distance information and the feature quantity extracted in the past. The image processing apparatus according to claim 5, wherein:   7. The image processing according to claim 1, wherein when the distance information is not used, the specifying unit specifies the image region using color information in the image. apparatus.   The said specifying means specifies the said image area | region based on the map which shows the probability which is a pixel of the to-be-photographed object of the designated position produced | generated based on color information. Image processing device.   The said specifying means specifies the said image area | region using the color information in the said image, and the said distance information, when using the said distance information, The any one of Claims 1-7 characterized by the above-mentioned. An image processing apparatus according to 1.   The specifying means generates a map indicating the likelihood of being a pixel of the subject at the designated position, which is generated based on color information, and the subject at the designated position, which is generated based on the distance information. The image processing apparatus according to claim 9, wherein the image region is specified from a map indicating the probability of being a pixel.   The image processing apparatus according to claim 1, wherein the feature amount is a pixel pattern or a histogram of the image area. Extraction means for extracting feature values from the image area;
Search means for searching for a region similar to the image region in a plurality of time-series images using the feature amount;
The extraction unit updates the feature amount used for the search by the search unit based on distance information in a region similar to the image region and distance information in a peripheral region similar to the image region. An image processing apparatus.   The extraction means uses the search means for the search based on a ratio of areas having distance information in which the difference between distance information in an area similar to the image area in the peripheral area is within a predetermined range. The image processing apparatus according to claim 12, wherein it is determined whether to update the amount.   The extraction means performs the search when the ratio of areas having distance information whose difference from distance information in an area similar to the image area in the peripheral area is within a predetermined range is greater than or equal to a threshold value. The image processing apparatus according to claim 13, wherein the feature amount used for the update is updated.   The extraction means uses the search means for the search according to the ratio of the area having distance information in which the difference from the distance information in the area similar to the image area in the peripheral area is within a predetermined range. The image processing apparatus according to claim 14, wherein the degree of updating the amount is changed. The image processing apparatus according to any one of claims 1 to 15,
A focus detection means for performing focus detection on an area including an area similar to the image area searched by the search means;
An imaging device comprising: An image sensor having a function of dividing the pupil region of the taking lens;
Generating means for generating the distance information from a parallax image obtained from the imaging element;
The imaging apparatus according to claim 16, further comprising: A specifying step in which the specifying unit specifies an image region in the image for extracting the feature amount based on the designated position;
An extracting step in which an extracting means extracts a feature amount from the image region;
A search step for searching for a region similar to the image region in a plurality of time-series images using the feature amount; and
In the specifying step, the specifying means uses the distance information if the distance information satisfying the reliability condition is obtained for the region including the designated position, and does not use the distance information otherwise. Identifying the image area;
And a control method for the image processing apparatus. A specifying step in which the specifying unit specifies an image region in the image for extracting the feature amount based on the designated position;
An extracting step in which an extracting means extracts a feature amount from the image region;
A search step for searching for a region similar to the image region in a plurality of time-series images using the feature amount; and
In the specifying step, the specifying unit specifies the image region without using the distance information before the distance information satisfying the reliability condition is obtained for the region including the designated position, and after the information is obtained. Identifying the image region using the distance information;
And a control method for the image processing apparatus. An extraction step in which the extraction means extracts a feature amount from the image region;
A search step for searching for a region similar to the image region in a plurality of time-series images using the feature amount; and
Updating the feature amount used for the search in the search step based on distance information in a region similar to the image region and distance information in a peripheral region similar to the image region in the extraction step; A control method for an image processing apparatus.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-15.

Images