JP2013132011A - Imaging device, imaging method, and program - Google Patents

Abstract

An image can be obtained when a sleeping face is photographed.
Face detection is performed from a preview image, and it is further determined whether or not the face is a sleeping face. If it is a sleeping face, camera shake prevention shooting is performed by image synthesis. In addition, when the face is a sleeping face and a smiling expression, best shot photography may be performed instead of camera shake prevention photography by image composition. Thus, an appropriate image can be obtained even in a dim environment where a sleeping face is photographed or in a posture that is difficult to photograph.
[Selection] Figure 1

Description

  In particular, the present invention relates to an imaging apparatus, an imaging method, and a program suitable for taking an image of a sleeping face.

  2. Description of the Related Art Conventionally, a technique for detecting a human face from a photographed image and detecting whether the detected face is a sleeping face is known. For example, in Patent Literature 1, a face image is acquired by capturing a face of a person, and whether or not the person is closed is estimated based on the face image, and depending on whether or not the person is smiling A technique for determining a person's arousal state is disclosed.

JP 2010-9093 A

  Conventionally, a human face is detected from a preview image in time series, and it is determined whether or not the detected face is a sleeping face. Some of them have a function of suppressing the sound and taking a picture in consideration of a sleeping person. However, when trying to shoot a sleeping face, there is a problem that it is impossible to shoot due to a sufficient environment and posture because many images are originally taken in a dimly lit room.

  The present invention has been made in view of the above-described problems, and an object thereof is to obtain an appropriate image when a sleeping face is photographed.

  An imaging apparatus according to the present invention is detected by an imaging unit that images a subject to generate a preview image, a face detection unit that detects the face of the subject from the preview image generated by the imaging unit, and the face detection unit. A facial expression detecting means for detecting whether or not the sleeping face is a sleeping face, and the imaging so that a still image is continuously photographed in accordance with a photographing instruction from a user while the sleeping face is detected by the facial expression detecting means. And a control unit that corrects a positional deviation between the plurality of continuously shot still images and generates a recording image by combining the corrected plurality of still images. It is characterized by.

  According to the present invention, an appropriate image can be obtained when a sleeping face is photographed.

1 is a block diagram illustrating a schematic configuration example of an imaging apparatus according to an embodiment. 6 is a flowchart illustrating an example of a shooting processing procedure according to the first embodiment. 10 is a flowchart illustrating an example of a shooting processing procedure according to the second embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a block diagram illustrating a schematic configuration example of an imaging apparatus 100 according to the embodiment.
1, an imaging apparatus 100 according to the embodiment includes an optical system 1 including a lens and a diaphragm, a mechanical shutter (mechanical shutter) 2, an imaging element 3, a CDS circuit 4, an A / D converter 5, a timing signal generation circuit 6, And a drive circuit 7. Further, a signal processing circuit 8, an image memory 9, an image recording medium 10, a recording circuit 11, an image display device 12, a display circuit 13, a face detection circuit 14, a facial expression detection circuit 15, and a system control unit 16 are provided.

  The CDS circuit 4 performs analog signal processing on an image signal obtained via the optical system 1, the mechanical shutter 2, and the image sensor 3. The A / D converter 5 converts an analog signal into a digital signal. The timing signal generation circuit 6 is a circuit that generates a signal for operating the imaging device 3, the CDS circuit 4, and the A / D converter 5. The drive circuit 7 is a circuit that drives the optical system 1, the mechanical shutter 2, and the image sensor 3.

  The signal processing circuit 8 performs signal processing necessary for the image data, and stores the signal processed image data in the image memory 9. The recording circuit 11 records the image-processed image data on the image recording medium 10 that is removable from the imaging apparatus 100. The display circuit 13 displays an image based on the signal processed image data on the image display device 12.

  The face detection circuit 14 is a circuit for detecting a person's face from image data, and the expression detection circuit 15 is a circuit for detecting whether the face detected by the face detection circuit 14 is a smile. . Then, the system control unit 16 controls the entire imaging apparatus 100.

  Hereinafter, a photographing operation using the mechanical shutter 2 in the imaging apparatus 100 configured as described above will be described. First, the drive circuit 7 drives the aperture and the lens of the optical system 1 by a control signal from the system control unit 16, and the optical system 1 forms an object image set to an appropriate brightness on the image sensor 3. Let Next, the mechanical shutter 2 is driven by the drive circuit 7 in accordance with a control signal from the system control unit 16 so as to shield the image sensor 3 in accordance with the operation of the image sensor 3 so as to have a necessary exposure time. At this time, when the image pickup device 3 has an electronic shutter function, it may be used together with the mechanical shutter 2 to ensure a necessary exposure time.

  The image pickup device 3 is driven by a drive circuit 7 by a drive pulse based on an operation pulse generated by a timing signal generation circuit 6 controlled by a system control unit 16, and converts an object image into an electrical signal by photoelectric conversion to generate an analog image signal. Output as. The analog image signal output from the image pickup device 3 is subjected to an operation pulse generated by the timing signal generation circuit 6 controlled by the system control unit 16, and the clock synchronization noise is removed by the CDS circuit 4. 5 is converted into a digital image signal.

  Next, the signal processing circuit 8 performs image processing such as color conversion, white balance, and gamma correction, resolution conversion processing, and image compression processing on the digital image signal. The signal processing circuit 8 may output the digital image signal as it is to the image memory 9 or the recording circuit 11 as it is without performing signal processing by the control signal from the system control unit 16. Further, in response to a request from the system control unit 16, the signal processing circuit 8, for example, information on the image data generated in the process of signal processing such as the spatial frequency of the image, the average value of the designated area, the data amount of the compressed image, or the like The information extracted from the information is output to the system control unit 16. Further, the recording circuit 11 outputs information such as the type and free capacity of the image recording medium 10 to the system control unit 16 when requested by the system control unit 16.

  Next, the reproduction operation when image data is recorded on the image recording medium 10 will be described. First, the recording circuit 11 reads image data from the image recording medium 10 by a control signal from the system control unit 16. Then, when the image data is compressed data, the signal processing circuit 8 performs an image expansion process according to a control signal from the system control unit 16 and stores it in the image memory 9. The image data stored in the image memory 9 is subjected to resolution conversion processing in the signal processing circuit 8 and then converted into a signal suitable for the image display device 12 in the display circuit 13, and the image is displayed on the image display device 12. Is displayed.

  The face detection circuit 14 performs a known face detection process on the image data from the signal processing circuit 8 to detect a face area of a person included in a captured image of the image sensor 3. As a known face detection process, for example, a skin color region is extracted from the gradation color of each pixel represented by image data, and a face is detected with a matching degree with a face contour plate prepared in advance. is there. Also disclosed is a method of performing face detection by extracting facial feature points such as eyes, nose and mouth using a well-known pattern recognition technique. Note that the face detection processing technique is not limited to the technique described above, and various known techniques can be used.

  The facial expression detection circuit 15 determines facial expressions such as a sleeping face and a smile on the face detected by the face detection circuit 14. Specifically, features necessary for calculating facial expressions, such as the contours of facial components such as eyes, nose, and mouth that make up the face, and the positions of facial components such as the positions of the eyes, corners of the eyes, nose, mouth corners, and lips Get the quantity. As a method for acquiring the feature amount, a discriminator for each face component obtained by template matching using a template of each face component or machine learning learning using a large number of sample images of the face component It can be obtained by a technique using. Then, the expression detection circuit 15 calculates an expression level indicating the degree to which the face is a specific expression based on the feature amount, and determines whether or not the face is a specific expression. The method for discriminating the facial expression is not limited to this, and various known methods can be used.

(First embodiment)
Next, with reference to the flowchart shown in FIG. 2, the first embodiment of the present invention will be described with reference to the sleeping face detection performed by the system control unit 16 and the accompanying camera shake prevention photographing control based on the image composition. This sleeping face detection and accompanying image stabilization shooting is executed according to a computer program stored in a memory (not shown) in the system control unit 16. The sleeping face detection control is sequentially performed in time series on the preview images that are sequentially captured at a predetermined frame rate by the image sensor 3. The interval at which sleeping face detection control is performed may be about once every several frames, although it depends on the processing capability of the face detection circuit 14.

FIG. 2 is a flowchart illustrating an example of a shooting processing procedure in the present embodiment.
First, in step S201, a human face detection process is performed by the face detection circuit 14 from the preview image generated by the above-described procedure to detect a face image. Next, in step S202, sleeping face detection processing is performed based on the face image detected in step S201 by the facial expression detection circuit 15.

  In the sleeping face detection process, it is determined whether or not eyes are closed with respect to the detected face, and when it is determined that the same face is closed with eyes over a predetermined number of frame images, the sleeping face is determined. It is determined that As to whether or not the eyes are closed, the facial expression detection circuit 15 receives information indicating the position and size of the detected eye area from the face detection circuit 14 and detects the detection area so as to surround the eyes. . An area where the luminance value is equal to or less than the threshold value is extracted from the detection area as a black eye. Then, a reference value is set for the face size, and if the size of the black eye is equal to or less than the reference value, it is determined that the eyes are closed.

  Next, in step S203, it is determined whether or not the face of the subject is a sleeping face as a result of the sleeping face detection process in step S202. If the result of this determination is that the subject's face is a sleeping face, the process proceeds to step S204, and setting is made to perform camera shake prevention shooting by image composition when a shooting instruction is issued from the user.

  Here, camera shake prevention photographing by image composition will be described. First, a plurality of images are continuously shot with a short exposure time that is not affected by camera shake. Then, a misalignment between the images is corrected from the plurality of photographed images, and a plurality of corrected images are synthesized to generate a recording image.

  For example, when two images are taken continuously, a method of combining images is to set a reference area in a part of one of the two images to be compared, and to calculate the reference from the other image. The region having the highest similarity with the region is detected. At this time, the difference between the center position of the reference area of one image and the center position of the area detected from the other image becomes a motion vector in the reference area. For example, if one image is divided into a plurality of areas and each divided area is set as a reference area, a plurality of motion vectors in the entire screen can be obtained. Then, by weighting each motion vector, a representative motion vector between the two images can be obtained.

  As the weighting method, for example, the weighting of the motion vector in the reference area closer to the center of the screen is increased, or the weighting of the motion vector having a large scalar amount is increased (the motion of the subject is regarded as a moving object). There is a method of increasing the weight of the vector). On the other hand, there is also a method of increasing the weight of a motion vector having a small scalar amount (assuming that the background does not move significantly and increasing the weight of the background motion vector).

  As another method for obtaining a representative motion vector, there is a method of classifying motion vectors into a plurality of groups according to their directions and obtaining a representative motion vector from the motion vectors included in the largest number of groups. If face detection is successful, the position of the face can be used as a reference area, and the motion vector of the face area can be obtained as a representative motion vector.

  In the present embodiment, when performing image synthesis, since a face that is a sleeping face is detected, a motion vector is obtained using this face area as a reference area. Then, using the motion vectors, the captured images are combined so that the positions of the faces coincide with each other. Since continuous shooting and composition processing are performed, the processing time increases, and when the subject moves, it cannot be combined well, but when shooting a sleeping face, the subject has almost no movement. It is an effective method.

  On the other hand, if the result of the determination in step S203 is that the subject's face is not a sleeping face, the process proceeds to step S205, where normal single shooting is set when a shooting instruction is issued from the user.

  As described above, when trying to photograph a sleeping face, there are many cases where the photographing is performed in an unstable posture, for example, the photographing is often performed in a dimly lit room, and the camera is held downward to bend forward. Therefore, if a normal shooting method is used, a blurred image is easily shot. On the other hand, according to the present embodiment, when a sleeping face is detected, the above-described camera shake prevention shooting by image synthesis is performed. As a result, it is possible to capture an image in which camera shake is suppressed. Further, when a sleeping face is not detected, normal single shooting is performed, so that continuous shooting and an increase in processing time due to camera shake prevention shooting can be prevented.

(Second Embodiment)
Conventionally, when a user issues a shooting instruction in a state where the face detected in the preview image is determined to be a smile, the image is continuously shot, and one optimal image is selected from the plurality of shot images. There is what is called the best shot shooting function that can leave. With this function, it is possible to evaluate the smile level, the eye-meditation result, and the blurring degree of a plurality of shot images, and leave an optimal single image based on the evaluation results. In the present embodiment, a process when a smile is detected while the sleeping face is determined will be described with reference to the flowchart shown in FIG.

FIG. 3 is a flowchart illustrating an example of a shooting processing procedure in the present embodiment. Note that the camera shake prevention photographing by detecting the sleeping face and accompanying image synthesis is executed according to a computer program stored in a memory (not shown) in the system control unit 16.
First, in step S301, the face detection circuit 14 performs a human face detection process from the preview image to detect a face image. In step S302, the facial expression detection circuit 15 performs a sleeping face detection process and a smile detection process based on the face image detected in step S301.

  In the smile detection process, an edge is detected from the detected face image, and smile evaluation is performed based on the direction of the edge. Therefore, the detected face image is divided into a plurality of regions, and the value of how much the direction of the edge detected in each region matches the direction of the edge that is easy to detect when a smile is detected. Asking. For example, in the case of a smile, the constriction line is emphasized, the mouth corner is raised, and the eyes are lowered. Evaluate that the more likely the smile is so that the edge direction in each area of the divided face image matches the edge direction that satisfies these conditions, and smile when the smile level exceeds a predetermined value. It is determined that there is. The sleeping face detection process is the same as that in the first embodiment.

  Next, in step S303, it is determined whether or not the face of the subject is a sleeping face as a result of the sleeping face detection process in step S302. If the result of this determination is that the subject's face is a sleeping face, the process proceeds to step S304; otherwise, the process proceeds to step S305. In step S304, it is determined whether the face of the subject is a smile as a result of the smile detection process in step S302. If the result of this determination is that the subject's face is not smiling, the process proceeds to step S306, and setting is made to perform camera shake prevention shooting by image composition when a shooting instruction is issued from the user. Camera shake prevention shooting by image composition is the same as in the first embodiment.

  On the other hand, if the result of the determination in step S304 is that the subject's face is smiling, the process proceeds to step S307, and the above-described best shot shooting is set to be performed when a shooting instruction is issued from the user.

  In the best shot shooting, when the user receives a still image shooting instruction while the subject is determined to be smiling, the predetermined number of still images are continuously shot. Then, the degree of smile in each still image is obtained, an evaluation value is calculated based on the degree of smile, the result of eye-browsing, and the amount of blur of the still image, and the still image with the highest evaluation value is selected as a recording image.

  If the result of determination in step S303 is that the subject's face is not a sleeping face, the process proceeds to step S305. In step S305, it is determined whether or not the subject's face is smiling as in step S304. If the result of this determination is that the subject's face is a smile, the process advances to step S307 to set to perform best shot shooting. On the other hand, if the result of determination in step S305 is that the subject's face is not smiling, the process proceeds to step S308, where normal single shooting is set when a shooting instruction is issued by the user.

  If a smiling face is detected while it is determined to be a sleeping face, taking both image stabilization and best shot shooting by combining images will increase the time required for continuous shooting and post-shooting processing. It is easy to miss a photo opportunity. In addition, facial expressions are likely to change in a laughing scene, and if camera shake prevention shooting is performed by image synthesis, image synthesis may fail or high-precision synthesis may not be possible.

  Therefore, according to the present embodiment, only the best shot is taken when a smile is detected while the sleeping face is being determined. Accordingly, it is possible to prevent an increase in shooting time and processing time due to camera shake shooting by image synthesis, and failure of image synthesis, and it is possible to pick up a highly accurate image by best shot shooting.

(Other embodiments)
In addition, embodiment mentioned above shows an example and is not limited to this. For example, in the second embodiment, when a smile is detected, the best shot shooting is set in step S307 in FIG. Therefore, when a shooting instruction is issued from the user, a plurality of images may be continuously shot at a high shutter speed that is not affected by camera shake, and the exposure of the shot image may be determined. If the exposure is appropriate, normal best shot shooting processing is performed so as to leave an optimal image based on the evaluation result of each image. On the other hand, as a result of determining the exposure of the photographed image, if the exposure is not appropriate, the image may be added to create an image with the appropriate exposure.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

DESCRIPTION OF SYMBOLS 1 Optical system 2 Mechanical shutter 3 Image pick-up element 7 Drive circuit 14 Face detection circuit 15 Facial expression detection circuit 16 System control part

Claims (8)

Imaging means for imaging a subject and generating a preview image;
Face detection means for detecting the face of the subject from the preview image generated by the imaging means;
Facial expression detection means for detecting whether the face detected by the face detection means is a sleeping face;
While the sleeping face is detected by the facial expression detection means, the imaging means is controlled to continuously shoot still images according to a shooting instruction from the user, and the plurality of continuously shot still images An image pickup apparatus comprising: a control unit that corrects misalignment between the two and generates a recording image by combining the plurality of corrected still images. Imaging means for imaging a subject and generating a preview image;
Face detection means for detecting the face of the subject from the preview image generated by the imaging means;
Facial expression detection means for detecting facial expressions detected by the face detection means;
While the sleeping face is detected by the facial expression detection means and no smile is detected, the imaging means is controlled to continuously take still images in accordance with a shooting instruction from the user, and the continuous shooting is performed. Control means for correcting a positional deviation between the plurality of still images, and generating a recording image by combining the corrected plurality of still images,
While the smile is detected by the facial expression detection means, the control means controls the imaging means to continuously shoot still images according to a shooting instruction from the user, and the continuous shooting is performed. An image pickup apparatus that selects an image having the highest evaluation value from among a plurality of still images as a recording image.   The imaging apparatus according to claim 2, wherein the evaluation value is a value based on a facial expression of the subject, a result of a blurred eye, and a still image blur amount.   While the smile is detected by the facial expression detection means, the control means synthesizes the plurality of still images according to the exposure of the plurality of continuously shot still images to form a recording image. The imaging apparatus according to claim 2 or 3, wherein An imaging method of an imaging apparatus including an imaging unit that images a subject and generates a preview image,
A face detection step of detecting the face of the subject from the preview image generated by the imaging means;
A facial expression detection step for detecting whether or not the face detected in the face detection step is a sleeping face;
While the sleeping face is detected in the facial expression detection step, the imaging unit is controlled to continuously shoot still images according to a shooting instruction from the user, and the plurality of continuously shot still images And a control step of generating a recording image by combining the plurality of corrected still images. An imaging method of an imaging apparatus including an imaging unit that images a subject and generates a preview image,
A face detection step of detecting the face of the subject from the preview image generated by the imaging means;
A facial expression detection step for detecting facial expressions detected in the face detection step;
While the sleeping face is detected and the smile is not detected in the facial expression detection step, the imaging unit is controlled to continuously shoot still images in accordance with a shooting instruction from the user, and the continuous shooting is performed. And a control step of correcting a positional deviation between the plurality of still images and generating a recording image by combining the corrected plurality of still images,
While the smiling face is detected in the facial expression detection step, the control step controls the imaging unit to continuously take still images in accordance with a shooting instruction from the user, and the continuous shooting. An image pickup method comprising: selecting an image having the highest evaluation value as a recording image from a plurality of still images. A program for controlling an image pickup apparatus including an image pickup unit that picks up a subject and generates a preview image,
A face detection step of detecting the face of the subject from the preview image generated by the imaging means;
A facial expression detection step for detecting whether or not the face detected in the face detection step is a sleeping face;
While the sleeping face is detected in the facial expression detection step, the imaging unit is controlled to continuously shoot still images according to a shooting instruction from the user, and the plurality of continuously shot still images And a control step of generating a recording image by combining the corrected still images and causing the computer to execute a program. A program for controlling an image pickup apparatus including an image pickup unit that picks up a subject and generates a preview image,
A face detection step of detecting the face of the subject from the preview image generated by the imaging means;
A facial expression detection step for detecting facial expressions detected in the face detection step;
While the sleeping face is detected and the smile is not detected in the facial expression detection step, the imaging unit is controlled to continuously shoot still images in accordance with a shooting instruction from the user, and the continuous shooting is performed. Correcting a positional deviation between the plurality of still images, and causing the computer to execute a control step of generating a recording image by combining the corrected plurality of still images,
While the smiling face is detected in the facial expression detection step, the control step controls the imaging unit to continuously take still images in accordance with a shooting instruction from the user, and the continuous shooting. A program for selecting an image having the highest evaluation value as a recording image from among the plurality of still images.

Images