JP2009055160A - Imaging apparatus and imaging method - Google Patents

Abstract

Even during a zoom operation, detection is continued without losing sight of a person's face in the composition.
In step S3, the enlargement / reduction ratio of the image signal is calculated, and in step S4, the position and size of the face detection frame in the current image being zoomed are linearly predicted based on the enlargement / reduction ratio. In step S5, the correction amount of the face detection frame is calculated based on face detection information (face position and size) up to 4 V or less, and in step S6, the face detection frame is provisionally corrected using the calculated correction amount. In step S7, it is determined whether the temporarily corrected face detection frame exceeds a threshold value, and the correction amount is corrected in step S8 according to the determination result. In step S9, the face detection frame is corrected with the correction amount obtained by correcting the face detection frame. The present invention can be applied to a digital video camera.
[Selection] Figure 4

Description

  The present invention relates to an imaging apparatus and an imaging method, and more particularly to an imaging apparatus and an imaging method suitable for detecting a subject's face and focusing on the detected face.

  Conventionally, there are digital still cameras and digital video cameras having an autofocus function. Conventionally, techniques for detecting a human face from the composition and focusing on the detected human face, correcting the brightness of the image based on the detected face, and correcting the white balance have been proposed. (For example, refer to Patent Document 1).

  Furthermore, digital still cameras using the above-described technology have also appeared.

JP 2005-31855 A

  According to the above-described technique, the focused state can be maintained as long as the human face can be detected. However, since a certain amount of processing time is required for face detection, when the focal length of the zoom lens is changed (hereinafter referred to as during zoom operation), the face is continuously detected following the zoom operation. It is difficult.

  That is, when there is a face at the center in the composition, the size changes by the zoom operation, and when the face is at a position other than the center in the composition, not only the size of the face by the zoom operation, Since the position also changes, it is difficult to continuously detect the position of the face. Furthermore, camera shake is likely to occur during the zoom operation, so that the position of the detected face becomes unstable.

  The present invention has been made in view of such a situation, and makes it possible to continue detection without losing sight of a human face in a composition even during a zoom operation.

  An imaging apparatus according to one aspect of the present invention is based on a zoom unit that zooms in or out a captured image in response to a zoom operation from a user, and an image signal that belongs to a detection frame provided on the captured image. A focus control unit that performs focus control, and a change unit that provides the detection frame on the captured image, and changes the position and size of the detection frame in response to zoom-in or zoom-out by the zoom unit. .

  The changing unit may change the position and size of the detection frame based on a focal length of the zoom unit before and after the zoom operation.

  The imaging apparatus according to one aspect of the present invention can further include a detecting unit that detects a human face from the captured image, and the changing unit is configured to capture the captured image based on the detected face. The detection frame can be provided on the top.

  The changing means is based on the position and size of the detection frame provided on the captured image, the motion vector of the detected face, the reliability that the detected frame is a face, or the face detection The correction can be made based on at least one of the delay times.

  An imaging method according to one aspect of the present invention is based on a zoom unit that zooms in or out a captured image in response to a zoom operation from a user, and an image signal that belongs to a detection frame provided on the captured image. In the imaging method of the imaging apparatus including the focus control means for performing focus control, the detection frame is provided on the captured image, and the position and the size of the detection frame correspond to zoom in or zoom out by the zoom means. The step of changing is included.

  In one aspect of the present invention, a detection frame is provided on the captured image, and the position and size of the detection frame are changed in response to zoom-in or zoom-out.

  According to one aspect of the present invention, focus can be controlled based on a detection frame.

In addition, according to one aspect of the present invention, detection can be continued without losing sight of a human face in a composition even during a zoom operation.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  An image pickup apparatus (for example, the digital video camera 10 in FIG. 1) according to one aspect of the present invention is a zoom unit (for example, the image pickup lens in FIG. 1) that zooms in or out in response to a zoom operation from a user. 11), focus control means (for example, the motor driver 20 in FIG. 1) that performs focus control based on an image signal belonging to a detection frame provided on the captured image, and the detection frame on the captured image. And changing means (for example, the control unit 23 in FIG. 1) that changes the position and size of the detection frame in response to zoom-in or zoom-out by the zoom means.

  The imaging apparatus according to one aspect of the present invention further includes detection means (for example, the face detection unit 28 in FIG. 1) that detects a human face from the captured image.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.

  FIG. 1 shows a configuration example of a digital video camera according to an embodiment of the present invention. The digital video camera 10 mainly includes an imaging lens 11 that condenses an optical image of a subject on an image sensor 12, an image sensor 12 that converts the collected optical image into an electrical image signal, and predetermined analog image processing. An analog signal processing unit 13 that performs digital image processing, an A / D conversion unit 14 that digitizes an image signal, and a digital signal processing unit 15 that performs predetermined digital image processing.

  The digital video camera 10 includes a liquid crystal panel 16 that displays captured images and various types of information, a viewfinder 17 that is used for determining a composition, and a recording device 18 that records captured image signals. The

  Further, the digital video camera 10 includes a control unit 23 that controls each unit of the digital video camera 10, an operation unit 24 that receives various operations from the user, and face detection that detects a human face from the image signal after the digital image processing. The unit 28 is provided.

  The imaging lens 11 includes a zoom lens capable of continuously changing a focal length, a focus lens that is moved to focus on a subject, an iris that changes the diameter of an aperture, an ND mechanism that inserts an ND filter, and camera shake during imaging. It consists of a shift anti-shake type camera shake correction lens.

  The image sensor 12 is configured by CCD or CMOS, generates an analog image signal by photoelectric conversion of the collected optical image, and outputs the analog image signal to the analog signal processing unit 13. The analog signal processing unit 13 performs analog image processing such as noise removal on the image signal from the image sensor 12 and outputs the image signal to the A / D conversion unit 14.

  The digital signal processing unit 15 performs digital image processing such as gamma correction on the image signal digitized by the A / D conversion unit 14.

  The recording device 18 records an image signal on a removable recording medium such as a semiconductor memory, a disk-shaped recording medium, or a tape-shaped recording medium.

  The timing generator (TG) 19 controls the charge discharge timing (that is, the image signal generation timing) to the image sensor 12. The motor driver 20 controls a drive motor 21 that drives the focus lens and a motor 22 that drives the zoom lens.

  The operation unit 24 includes buttons and switches provided on the casing of the digital video camera 10 or a touch panel provided so as to be superimposed on the liquid crystal panel 16, and accepts a recording start / stop operation, a zoom operation, and the like from the user. The corresponding operation signal is output to the control unit 23.

  The control unit 23 controls various processes performed by the digital video camera 10 by the built-in CPU executing programs stored in the program ROM 26. For example, based on the face detection information from the face detection unit 28, the control unit 23 performs AF processing for focusing on the detected face, AE processing for adjusting brightness, WB processing for white balance, and the like.

  Further, for example, the control unit 23 increases the size when the entire integration frame for calculating the coring value, which is a conventional detection frame, the small frame for extracting the subject at the center, and the small frame cannot be detected. There are four basic detection frames, a medium frame for detection and a large frame including the background. The detection values of the three detection frames of large, medium, and small are always calculated. A face detection frame for detecting a face is provided separately from the three large, medium, and small detection frames, and is used for focus control and the like. Furthermore, in order to acquire a detection value of a body other than the face, a body detection frame including a part or the entire face detection frame may be provided and used for focus control.

The EEPROM 25, which is a non-volatile memory, stores image signals, various auxiliary information, and setting information. The program ROM 26 stores various calculation parameters and the like in addition to a program executed by the CPU built in the control unit 23. The RAM 27 stores a program executed by the CPU built in the control unit 23, parameters that change as appropriate in the execution of the program, and the like.

  The face detection unit 28 analyzes the image signal supplied from the digital signal processing unit 15, detects a human face from the image signal, and outputs face detection information to the control unit 23 as a detection result.

  Specifically, the human face is detected by matching a plurality of reduced images obtained by reducing the image signal with a plurality of different reduction ratios and a plurality of templates indicating face brightness distribution information prepared in advance. This template is inclined with respect to each of the XYZ axes of the three-dimensional orthogonal coordinate system of the face, and the actual inclination of the face can be determined by matching with this template. When an area with a reduced image matches the template, it is determined that a face exists in the area, and the position of the face is detected. The size of the face in the image can be obtained from the reduction ratio of the actual image. Further, the rotation angle, yaw, pitch, and roll angle around three orthogonal axes can be obtained from the matched template. Note that face detection by the face detection unit 28 currently requires a maximum time of about 4 V (4 frames). Therefore, the face detection information output from the face detection unit 28 has a delay of about 4 V at the maximum with respect to the input of the corresponding image signal.

  In addition to detecting the position and size of the face in the face detection unit 28, for example, face ID, face width, face estimation distance, subject importance, appearance time, appearance time, appearance frequency existing in the image Age, sex, reliability indicating the certainty of the face, delay time due to face detection, and the like may also be detected and included in the face detection information and supplied to the control unit 23.

  The control unit 23 determines the position and size of a human face as a subject based on the face detection information, and the position and size of the face detection frame so that the entire detected face or a part of the face is included. To decide. In addition to the face detection information, the position and size of the face detection frame are determined in consideration of the current digital camera 10 state such as the operation mode of the digital video camera 10, the position of the zoom lens (focal length), and the brightness of the image. It may be determined.

  Since the face detection information corresponding to the image signal after the zooming operation is delayed by about 4 V at the maximum during the zoom operation, the control unit 23 waits for the face detection frame immediately before the zooming operation is performed without waiting for this. The position and size of the current face detection frame are estimated from the position and size of and the focal length variation caused by the zoom operation. Further, the position and size of the face detection frame are corrected in consideration of face detection information and motion vectors supplied with a delay of about 4 V at maximum. This makes it possible to set an accurate face detection frame according to the position and size of the human face.

  Next, a process of tracking the face detection frame during the zoom operation by the digital video camera 10 (hereinafter referred to as a face detection frame tracking process) will be described.

  In the face detection frame following process, for example, the zoom-in operation is performed from the state before the zoom operation shown in FIG. 2, that is, the state where the human face exists at a position shifted from the center. As shown in FIG. 2, the face detection frame for detecting the face is also changed from the state shown in FIG. 2 to the state shown in FIG. 3 in accordance with the zoom operation, following the change in the position and size of the face. It is processing of.

  The flow of the face detection frame following process will be described with reference to the flowchart of FIG.

  In step S1, the control unit 23 determines whether or not the zoom operation is currently in progress. The process proceeds to step S2 until it is determined that the zoom operation is in progress. Follow-up processing is performed. If it is determined in step S1 that the zoom operation is being performed, the process proceeds to step S3.

In step S3, the control unit 23 calculates the enlargement / reduction ratio mag of the image signal based on the zoom position (focal length) Zpo immediately before the zoom operation is performed and the current zoom position (focal length) Zp during the zoom operation. To do.
mag = Zp / Zpo

In step S4, the control unit 23 linearly predicts the position and size of the face detection frame in the current image during the zoom operation based on the enlargement / reduction ratio mag of the image signal calculated in step S3. Specifically, the coordinates ffpo of the four vertices of the face detection frame immediately before the zoom operation are multiplied by the enlargement / reduction ratio mag, and the coordinates of the four vertices of the face detection frame in the current image during the zoom operation. Calculate ffp. The center of the image is the origin of the coordinate system.
ffp = mag × ffpo

  In step S5, the control unit 23 performs the zoom operation during the zoom operation predicted in step S4 based on the face detection information (face position and size) corresponding to the image signal of up to 4V or less supplied from the face detection unit 28. The correction amount adj of the position and size of the face detection frame in the current image is calculated. Specifically, the shift of the current face detection frame is obtained from the position and size of the face included in the face detection information, and the face position / size of the face information, the certainty of the face, and how many V are delayed Based on the information and the like, the correction amount adj ′ of the face detection frame is obtained.

Furthermore, since the face that is the subject is not stationary and is expected to move even during the zoom operation, as shown in FIG. 5, the correction amount of the face detection frame is also considered in consideration of the movement of the face (motion vector). Adjust adj '. The adjustment of the face detection frame correction amount adj ′ by the motion vector is performed by adding each vertex correction amount adj ′ of the face detection frame to a value obtained by multiplying the motion vector value mv by the motion vector reflection rate km. . It is assumed that the motion vector reflection rate km varies depending on a preset value, the brightness of the current image, the certainty of face information, or the like.
adj ＝ adj '＋ km × mv

  In step S6, the control unit 23 temporarily corrects the position and size of the face detection frame in the current image during the zoom operation predicted in step S4 with the correction amount adj of the face detection frame calculated in step S5. And calculate the size.

  In step S7, the control unit 23 determines whether or not the position and size of the face detection frame temporarily corrected in step S6 exceeds a threshold value. The correction amount adj of the face detection frame is corrected so that the position and size of the face detection frame do not exceed the threshold value. If it is determined in step S7 that the threshold value is not exceeded, step S8 is skipped.

  For example, when the size of the face detection frame after provisional correction becomes larger than a predetermined maximum value (threshold value) in response to a zoom-in operation or the like, the size of the face detection frame after correction becomes the maximum value. As described above, the correction amount adj of the face detection frame is corrected. Alternatively, the correction amount adj is set to 0 so as not to be corrected.

  In addition, for example, when the size of the face detection frame after provisional correction is smaller than a predetermined minimum value (threshold value) in response to a zoom-out operation or the like, the size of the face detection frame after correction is the minimum value. The correction amount adj of the face detection frame is corrected so that Alternatively, the correction amount adj is set to 0 so as not to be corrected.

  Further, for example, when a part of the face detection frame after provisional correction protrudes from a predetermined region (detectable region) of the image in response to a zoom-in operation or the like, the position and size of the face in the protruding axial direction Correction is not performed (the axial correction amount adj that protrudes is set to 0). Alternatively, it is corrected so that the position and size immediately before the protrusion is restored.

  As described above, when the following face detection frame protrudes from the detectable region, the face detection frame remains on the spot without being moved. Then, while the face is detected by the face detection, it stays on the spot, and when the face returns to the detection possible region, the face detection frame is moved again. However, when the face detection frame exceeds the detectable region and face detection cannot be performed, a new face detected by face detection is followed.

  In step S9, the control unit 23 corrects the position and size of the face detection frame using the correction amount adj of the face detection frame calculated in step S5 or the result obtained by correcting it in step S8.

  In step S10, the control unit 23 corrects the size of the face detection frame in accordance with the direction of the zoom operation (zoom in or zoom out). Specifically, correction is mainly performed to reduce the size of the detection frame when zooming in. Thereafter, the process returns to step S1, and the subsequent processing is repeated. This is the end of the description of the face detection frame following process performed by the digital video camera 10.

  According to the face detection frame following process described above, the face detection frame can be accurately matched to the face even during the zoom operation. Therefore, even if there is a zoom operation, the subject (face) is not lost, so that it is always possible to focus on the face.

  In this embodiment, the face detection frame is made to follow the zoom operation. However, not only the face detection frame but also the body detection frame and the whole person detection frame may be made to follow.

  The present invention can also be applied to an imaging apparatus such as a digital still camera having a zoom function in addition to a digital video camera.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

It is a block diagram which shows the structural example of the digital video camera to which this invention is applied. It is a figure which shows the example of a display of the image before zoom operation | movement. It is a figure which shows the example of a display of the image after a zoom operation | movement is started. It is a flowchart explaining a face detection frame follow-up process. It is a figure which shows the example of a display of an image when a face moves after zoom operation | movement is started.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Digital video camera, 11 Imaging lens, 12 Image sensor, 13 Analog signal processing part, 14 A / D conversion part, 15 Digital signal processing part, 20 Motor drive, 23 Control part, 24 Operation part, 28 Face detection part

Claims (5)

Zoom means for zooming in or out the image to be captured in response to a zoom operation from the user;
Focus control means for performing focus control based on an image signal belonging to a detection frame provided on the captured image;
An imaging apparatus comprising: a detection frame provided on a captured image; and changing means for changing a position and a size of the detection frame in response to zoom in or zoom out by the zoom means. The imaging apparatus according to claim 1, wherein the changing unit changes a position and a size of the detection frame based on a focal length of the zoom unit before and after the zoom operation. Further comprising a detecting means for detecting a human face from the captured image,
The imaging apparatus according to claim 1, wherein the changing unit provides the detection frame on a captured image based on the detected face. The changing means includes the position and size of the detection frame provided on the captured image, the motion vector of the detected face, the reliability that the detected frame is a face, or the face detection The imaging apparatus according to claim 1, wherein the correction is performed based on at least one of the delay times. Zoom means for zooming in or out the image to be captured in response to a zoom operation from the user;
In an imaging method of an imaging apparatus comprising: a focus control unit that performs focus control based on an image signal belonging to a detection frame provided on a captured image;
An imaging method comprising: providing the detection frame on an imaged image, and changing a position and a size of the detection frame in response to zoom-in or zoom-out by the zoom unit.

Images