JP2006174022A - Image processing apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a subject pattern accurately even when a focus state is bad.
An image processing apparatus comprising means for detecting a predetermined subject pattern from an image output from an image pickup means, further comprising a focus detection means for detecting a focusing distance of the image pickup means, and the subject pattern detection means. Performs the collation of the subject pattern by selectively switching the reference pattern for collating the subject pattern according to the focus state estimated based on the in-focus distance.
[Selection] Figure 2

Description

  The present invention relates to an image processing apparatus and method for automatically detecting a subject pattern from an image.

  In recent years, proposals have been made to improve accuracy by applying a technique for detecting a specific subject pattern from an image to focus control of a camera. For example, the present applicant has proposed a camera equipped with a subject recognition function. (Patent Document 1) In this conventional example, there is a proposal for improving the accuracy of camera exposure and focusing by detecting a face from an image.

  On the other hand, as a technique for detecting a face from an image, for example, Non-Patent Document 1 lists various methods.

  However, the current face detection technology is accurate when a face with proper focus and brightness indicating whether the image is in focus or not, or when facing the front, is accurate. Although the detection can be performed well, the detection performance is remarkably deteriorated when the photographing condition is bad. For example, although Patent Document 1 proposes to improve the accuracy of camera exposure and focusing using the face detection result, a method of accurately detecting a face in a situation where the shooting state is poor is explained. Not.

A method is also known in which focus control is performed from the shortest distance based on multi-point distance measurement data, and a face is detected only in a focused area. However, in this conventional example, it is necessary to prepare multipoint distance measurement data. Although the problem is avoided by detecting only the in-focus face when the face detection result is used for camera control, the problem is not essentially solved.
JP 2001-330882 A Yang, “Detecting Faces in Images: A Survey”, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.24, NO.1, JANUARY 2002

  The present invention has been made in view of the above problems, and an object of the present invention is to accurately detect a subject pattern even when the focus state is poor.

  In order to achieve the following object of the present invention, in an image processing apparatus comprising means for detecting a predetermined subject pattern from an image output from an image pickup means, a focus detection means for detecting a focus distance of the image pickup means And the subject pattern detection means selectively matches a reference pattern for matching the subject pattern according to a focus state estimated based on the in-focus distance, and performs the subject pattern matching. To do.

  With the configuration of the present invention, it is possible to detect a subject pattern with high accuracy even when the focus state is poor. With the configuration of the present invention, the subject pattern can be detected with high accuracy even when the focus state is poor.

  FIG. 1 shows a schematic configuration of this embodiment.

  Reference numeral 10 denotes an imaging unit, an imaging lens, an imaging device such as a CCD, a gain adjustment circuit that adjusts the gain of the output signal of the imaging device, and an A / D conversion circuit that converts the output signal of the imaging device after gain adjustment into a digital signal The signal from the image sensor is output as digital image data.

  An image memory 20 temporarily stores the image data output from the imaging unit 10.

  A focus control unit 30 controls the position of the focusing lens of the photographing lens so that the contrast of the image data is maximized based on the contrast of a predetermined area of the image data.

  Reference numeral 40 denotes exposure control means, which determines the shutter speed and aperture value according to a predetermined program diagram so that the image data has a predetermined brightness based on the brightness of a predetermined area of the image data, and the image pickup device of the image pickup means 10 Control the exposure time and aperture of the taking lens.

  Reference numeral 50 denotes a face detection unit, which detects a face pattern from image data stored in the image memory 20 and outputs a position in the image.

  Reference numeral 60 denotes an area setting unit that sets an image area to be controlled by the focus control unit 30 and the exposure control unit 40 according to the output of the face detection unit 50.

  A focus detection unit 70 detects the position of the focusing lens of the photographing lens controlled by the focus control unit 30.

  The operation of each of the above blocks is controlled by a control unit that controls the entire camera (not shown).

  Next, the operation of this embodiment will be described with reference to FIG.

  When a user turns on a power source (not shown) to photograph a subject and enters a preparation state for photographing, the imaging means 10 outputs image data, and the output image data is stored in the image memory 20. (S101) That is, the aperture value of the imaging lens that determines the exposure state, the shutter speed, the gain of the imaging device, and the focusing lens of the imaging lens that determines the focus state are set to the initial state, and the signal output from the imaging device is digital It is converted into a signal and output as image data from the A / D conversion circuit.

  Next, the focus control means 30 controls the position of the focusing lens of the photographic lens based on the contrast of a predetermined area of the image data stored in the image memory 20 so that the contrast of the image data is maximized. (S102) That is, the luminance contrast in the case where the focusing lens is moved by repeatedly evaluating the luminance contrast in a predetermined area near the center of the image data output from the imaging means 10 while moving the focusing lens of the photographing lens. Based on the amount of change, the position of the focusing lens is controlled so that the focusing lens comes to the position with the highest luminance contrast.

Next, the exposure control means 40 determines a shutter speed and an aperture value according to a predetermined program diagram so that the image data has a predetermined brightness based on the luminance of a predetermined area of the image data, and the image pickup device of the image pickup means 10. Control the exposure time and aperture of the taking lens. (S103)
Next, the focus detection means 70 detects the position of the focusing lens of the taking lens. Then, the distance from the focusing lens position to the subject (hereinafter referred to as the focusing distance) from the position of the focusing lens is output as focus information. At the same time, the focal length and aperture value of the photographing lens are output as focus information. (S104) The relationship between the position of the focusing lens and the object distance is stored in advance as a table. For example, when the photographing lens is a zoom lens with a variable focal length, a table for each focal length is prepared.

  Next, the face detection means 50 detects the face pattern from the image data stored in the image memory 20 using the focus information, and outputs the position in the image. (S105) FIG. 3 shows a configuration diagram of the face detecting means 50, and FIG. 4 shows a processing flow. Hereinafter, the processing details of the face detecting means will be described with reference to FIGS.

First, focus information is input. (S501)
Next, luminance image data is generated in the luminance distribution generation means 51 from the image data. (S502) For example, when the input image data is RGB image data, YUV image data is generated by predetermined matrix conversion, and luminance image data is generated with the Y component as a luminance value. The conversion from RGB to YUV is performed by a method as disclosed in Japanese Patent Application Laid-Open No. 2003-102025, for example.

  Next, a skin color mask is generated by the skin color mask generation means 52 from the image data. (S503) For example, when the input image data is RGB image data, YUV image data is obtained by a predetermined matrix conversion, and when the UV component satisfies a predetermined range, the skin color pixel is set, the skin color pixel is 1, and the other pixels are set. The binary image data set to 0 is output as a skin color mask. For the skin color pixel determination range, the YUV coordinate value distribution is examined in advance for the skin regions of a large number of human image data, and the range is stored as a skin color pixel determination parameter.

  Next, the detection size setting means 53 sets the size of the face to be detected from the image. (S504) Since there is a possibility that faces of various sizes are shot in the image data depending on the shooting magnification of the subject, the size is set here to detect a face of a predetermined size. Do. As will be described later, in this embodiment, in order to detect faces of a plurality of sizes, the setting of the size of the face to be detected and the detection of the face of the set size from the image are repeated.

  Next, an appropriate reference pattern is selected from a plurality of reference patterns stored in the reference pattern storage unit 57 based on the focus information and the set face detection size in the reference pattern selection unit 59. (S505) The focus state of the face image photographed by the camera is uniquely determined by the distance from the camera to the subject (in this case, the face), the focusing distance, the focal length of the photographing lens, and the aperture value. More specifically, the amount of blur is determined in pixel units depending on the pixel size of one pixel of the image sensor. In addition, since the size range of the face to be detected is almost fixed, the amount of blur corresponding to the size of the face to be detected in the image is determined by using the focus distance, the focal length of the photographing lens, and the aperture value as focus information. If used, it can be estimated roughly. In this embodiment, using this principle, an appropriate reference pattern is selected from the focus information and the size of the face to be detected. Here, in this embodiment, the luminance distribution and the skin color distribution are used as the reference pattern, and in the luminance distribution pattern, the distribution is investigated in advance for a large number of faces of human image data having different focus states. The distribution is stored in the reference pattern storage means 57 as a reference pattern for each focus state. The size of the reference pattern is stored as a large size pattern because the high frequency component is larger for a face with a better focus state (ie, a focused face).

  Next, the search range setting means 54 sets a range for detecting a face in the image using luminance image data and a skin color mask according to the size of the face. (S506) That is, when detecting a face of a predetermined size, detection may be performed from all regions of the image, but the number of detections is enormous, so the face is previously obtained using luminance image data and a skin color mask. Is stored as a mask in order to delete a region that does not exist or is not likely to be detected. Here, it is determined whether or not to search based on whether the average value of the luminance image data in the face area of the size to be detected is within the predetermined area and the ratio of the skin color pixels is within the predetermined range.

  Next, the position in the image where the pattern to be matched with the face is cut out is set. (S507) That is, by repeating the setting here and the subsequent detection process, the position where the face pattern to be detected is collated is sequentially scanned from the entire image. At this time, a setting is made so that only the position to be searched is searched according to the result of the search range stored in the search range setting means 54.

Next, the collation pattern extraction means 55 cuts out the collation pattern from the luminance image data and the skin color mask based on the size set in S504 and the cutout position set in S507. (S508)
Next, the collation pattern extraction unit 55 normalizes the luminance image data and skin color mask collation pattern cut out in S508 to the same size as the reference pattern selected by the reference pattern selection unit 59. (S509) In the luminance image data, the luminance distribution is normalized so as to have the same average luminance and luminance dispersion value as the reference pattern.

  Next, the pattern matching unit 56 collates the matching pattern output by the matching pattern extraction unit 55 with the reference pattern selected by the reference pattern selection unit 59 to obtain the similarity. (S510) For example, the reciprocal of the sum of absolute differences between the luminance distribution of the matching pattern and the luminance distribution of the reference pattern is the first similarity, and the sum of the absolute differences of the difference between the skin color distribution of the matching pattern and the skin color distribution of the reference pattern The reciprocal is used as the second similarity, and the similarity is obtained as a predetermined weighted sum of the first and second similarities. In S505, the weight is set so that the higher the degree of similarity is output as the reference pattern for the face in good focus is used as the reference pattern.

Next, it is determined whether or not the similarity of the pattern to be collated is equal to or greater than a predetermined threshold, and the position coordinates and the similarity are stored for a pattern equal to or greater than the predetermined threshold. (S511)
Next, the processing from S507 to S511 is repeated so as to scan the entire area of the image according to the search range stored in the search range setting means 54. Further, the processing from S504 to S511 is repeated for a predetermined detection size.

  Next, when the search for the face within the predetermined size range is completed for the entire image, the face area determination means 58 detects the face, and among the face candidate areas in which the position coordinates are stored, the areas overlap. The candidate of the face with a low similarity is deleted with respect to the candidate. (S512) Here, since the weight is set so that the similarity is larger as the reference pattern for the face with good focus state is used as the reference pattern in S510, the faces with different focus states are detected in duplicate. If the face pattern is selected, the face candidate is deleted so as to give priority to a face pattern with good focus.

  Further, the face area determination means 58 refers to the reference pattern selected by the reference pattern selection means 59 based on the luminance distribution of the image extracted by the matching pattern extraction means 55 for the area where the face is detected, A final determination of whether or not is made, the fake face pattern is deleted, and the final face detection result is output as position coordinates indicating the area of the image. (S511) Since the face area determination unit 58 needs to perform determination with higher accuracy than the pattern matching unit 56 in order to perform final face determination, the reference pattern used here is more accurate than the reference pattern used by the pattern matching unit 56. It is desirable to use a high pattern. The reference pattern may be something like the luminance pattern of an image, for example, Rowley et al. Entitled “Neural network-based face detection” in IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL.20, NO.1, JANUARY 1998. When using a method, it may be a weighting coefficient applied to a neural network.

  Based on the result of the face detection means 50, when a face is detected, the area setting means 60 changes the target area for exposure and focus control. (S106) That is, when one face is detected in the image, the detected face area is set as a control target area, and is output to the focus control means 30 and the exposure control means 40. Hereinafter, as shown in FIG. 1, focus control (S107) by the focus control means 30, exposure control (S108) by the exposure control means 40, and imaging (S109) are performed, and the control target is narrowed down to the person who is the main subject. Imaging is performed based on accurate focus and exposure control. Needless to say, the processes of S107, S108, and S109 are the same as those of S102, S103, and S101, respectively. When a plurality of faces are detected in the image, the area setting unit 60 sets the face area having the highest similarity as a control target area with reference to the similarity to each reference pattern of the detected face, It outputs to the focus control means 30 and the exposure control means 40. The subsequent processing is the same as when one face is detected. At this time, the region setting unit 60 may output a plurality of regions to the focus control unit 30 and the exposure control unit 40. For example, the focus control means 30 can perform depth priority focus control so that a plurality of face regions fall within the depth of focus. Further, the region setting means 60 outputs the similarity of each detected face area to the exposure control means 40, so that the exposure control means 40 determines the weight of each face area based on the similarity, and the weight Exposure control based on attached evaluation metering can be performed.

If no face is detected as a result of the face detection means 50, it is determined that no person is present in the photographed image, and is matched with a predetermined area of a normal image performed in the initial state in S102 and S103. Returning to the focus control and exposure control states, imaging is performed. (S110)
As described above, in this embodiment, even when the focus state is poor, the face can be accurately detected, and the focus control and the exposure control can be suitably performed based on the detection result. In particular, in this embodiment, since the focus state of the face to be detected is predicted and the face pattern optimized for each focus state is collated, the face detection accuracy is not deteriorated.

  In this embodiment, focus control and exposure control are performed based on the face detection result, but the present invention can also be applied to control such as strobe control. Alternatively, the processing target may be narrowed down to the detected face area, and white balance processing, color correction, and gamma correction processing may be suitably performed.

  In the present embodiment, the human face is detected as the subject pattern, but other subject patterns that are important for each scene may be used.

  In this embodiment, the exposure accuracy of a digital still camera is used to improve the detection accuracy of a specific subject pattern and feed back to the control. For example, an image input device having an exposure control function such as a video camcorder or a surveillance camera. This embodiment is also applied.

  In the present embodiment, the digital still camera has been described. However, the present embodiment may be applied to a case where an image captured by a digital still camera is processed by a computer. For example, in addition to an image captured by a digital still camera, in-focus distance indicating a focus state, a focal length of a photographing lens, an aperture value, and photographing information of a pixel size of one pixel of an image sensor may be acquired.

  In addition, a recording medium (or storage medium) that records software program codes for realizing the functions of the above-described embodiments is supplied to a system or apparatus, and the computer (or CPU or MPU) of the system or apparatus stores the recording medium. You may implement | achieve by reading and executing the stored program code. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. A part or all of the actual processing may be performed, and the functions of the above-described embodiments may be realized by the processing.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. The CPU of the expansion card or the function expansion unit may perform part or all of the actual processing, and the functions of the above-described embodiments may be realized by the processing.

  When the above recording medium is used, program code corresponding to the flowchart described above is stored in the recording medium.

FIG. 2 is a block diagram illustrating a configuration of a camera according to the present embodiment. The figure which shows operation | movement of a present Example. The figure which shows the structure of the face detection means of a present Example. The figure which shows operation | movement of the face detection process of a present Example.

Claims (8)

  An image processing apparatus comprising means for detecting a predetermined subject pattern from an image output from the image pickup means, further comprising a focus detection means for detecting an in-focus distance of the image pickup means, wherein the subject pattern detection means detects the subject pattern. An image processing apparatus, wherein the subject pattern is collated by selectively switching a reference pattern for collation according to a focus state estimated based on the in-focus distance.   The image processing apparatus according to claim 1, wherein the subject pattern is a face.   The imaging unit includes a focus control unit, and the focus control unit determines an image area to be a focus control target of the imaging unit according to a result of the subject pattern detection unit, and performs focus control of the imaging unit. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized.   An image processing method for detecting a predetermined subject pattern from an image, comprising a shooting information acquisition step for acquiring shooting information of the image, and selecting a reference pattern for matching the subject pattern according to the shooting information An image processing method characterized in that the subject pattern is collated by switching between the two.   5. The image processing method according to claim 4, wherein the photographing information is information representing a focus state of the image.   The image processing method according to claim 4, wherein the photographing information includes a focusing distance of an imaging unit when the image is photographed.   The image processing method according to claim 4, wherein the subject pattern is a face.   A recording medium storing a program for executing the image processing method according to claim 4.

Images