JP6741237B2 - Image processing device, terminal device, image processing method, and storage medium - Google Patents

Description

The present invention relates to an image processing device, a terminal device, an image processing method, and a storage medium .

2. Description of the Related Art Some image pickup devices such as digital still cameras and video cameras are provided with a facial treatment function capable of beautifully capturing a face, which is called beautiful skin and whitening. If this facial processing function operates in a scene where no human is shown, the areas near the skin color will be blurred or whitish, so the human face will appear in the captured image (live view image) during shooting. It works on condition that it is in the image. For example, conventionally, as an image processing function, an imaging device that detects a face of a subject (for example, a photographer) at the time of photographing a person and performs exposure correction, soft focus correction, and skin softening correction on the captured image is disclosed. Proposed as in Reference 1. In addition, a technique is proposed as in Patent Document 2 that detects whether a person's face is included in the captured image, and if the face is included, further detects the lip area to perform gloss processing. ing.

Japanese Patent No. 5239126 JP, 2009-65269, A

By the way, in self-portraits such as taking a picture of yourself, there are cases in which multiple pictures are taken continuously while changing the direction of the face or the position of the camera each time the release operation is performed. , If the face of a person turns sideways with respect to the camera during continuous shooting and the face does not face the front, the face cannot be detected. Sometimes facial treatment was not applied. As a result, an image that has been subjected to facial treatment and an image that has not been subjected to facial treatment will coexist in a plurality of consecutively captured images.

An object of the present invention is to ensure that when a person is photographed as a subject, predetermined image processing can be performed on the captured image.

In order to solve the above-mentioned problems, the present invention provides
An image acquisition means for acquiring a captured image,
First detection means for detecting a face of a person from the captured image acquired by the image acquisition means;
Second detecting means for detecting a face of a person from the captured image obtained by correcting the captured image based on the information for correcting the aberration of the imaging lens forming the imaging means;
Can not detect the face of a person by said first detection means and, where it was able to detect a person's face by said second detection means, said relative obtained IMAGING image image processing means for performing predetermined image processing corresponding to a human face thereof,
A images processing device Ru comprising a.

According to the present invention, when a person is photographed as a subject, it is possible to reliably perform predetermined image processing on the captured image.

FIG. 1 is a block diagram showing the basic components of the image capturing apparatus, which is an example of the case where the image processing apparatus is applied to an image capturing apparatus. FIG. 1 is a diagram showing an external view of an image pickup device configured to allow a so-called self-portrait to photograph the photographer himself while watching a live view image. 9 is a flowchart for explaining an operation of the image pickup apparatus (a characteristic operation of the first embodiment), which is started when the facial image capturing mode is designated. 4 is a flowchart following the operation of FIG. 3. 9 is a flowchart for explaining the operation of the image pickup apparatus (the characteristic operation of the second embodiment), which is started when the beauty shooting mode is designated. (1) is a diagram for specifically explaining step B6 in FIG. 5, and (2) is a diagram for specifically describing step B10. 9 is a flowchart for explaining the operation of the image pickup apparatus (the characteristic operation of the third embodiment), which is started when the facial photographing mode is designated. (1) is a diagram for specifically explaining step C6 in FIG. 7, and (2) is a diagram for specifically describing step C12. 9 is a flowchart for explaining an operation of the image pickup apparatus (a characteristic operation of the fourth embodiment), which is started when the facial image capturing mode is designated.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
(First embodiment)
First, a first embodiment of the present invention will be described with reference to FIGS.
The first embodiment exemplifies a case where it is applied to an imaging device as an image processing device, and as will be described in detail later, a captured image (still image) captured according to a shooting instruction operation (release operation) is displayed. When a captured image is generated by acquiring and performing a capturing process, when a face of a person can be detected from the captured image when performing the capturing process on the captured image, the captured image If the human face cannot be detected, it is determined whether or not the captured image needs to be subjected to the predetermined image processing, and the predetermined image processing is performed. When it is determined that it is necessary to perform the processing, it is instructed to perform a predetermined image processing on the captured image.

FIG. 1 is a block diagram showing basic components of the image pickup apparatus.
This image pickup device has a basic function such as an image pickup function capable of shooting a subject in high definition, an image playback function for arbitrarily reading and playing back a recorded image (recorded image) recorded and stored, and a captured image. A digital camera having a face detection function of detecting a person's face while analyzing (live view image), and a facial treatment function of performing a process called beautiful skin and whitening on a captured image (live view image). ..

The face detection function described above is a function for recognizing a person's face by comprehensively determining the contour of the person's face and the shape and positional relationship of the baht (eyes, mouth, nose, forehead, etc.) forming the face. If the person's face is oriented substantially in front of the imaging device (camera), it can be determined that the person's face is included in the captured image (still image: live view image). If the person's face is sideways (for example, right beside) with respect to the (camera), it may not be possible to determine that the person's face is included in the captured image (live view image). It should be noted that such a face detection function is a technique generally used in a camera, and in the present embodiment, a well-known technique thereof is used, so a detailed description thereof will be omitted. To do. In addition, the facial processing function described above is a function for performing facial processing on a captured image (live view image), that is, not only targeting only the face of a person in the captured image, but the skin color of the neck or arm. This is a function that also applies processing to so-called skin-whitening and whitening that part.

Further, the present embodiment is, for example, during continuous shooting in which a plurality of images are continuously shot while changing the direction of the face and the position of the camera for each shooting instruction operation (release operation) at the time of self-shooting when shooting oneself. As a result of performing face detection on each captured image, when a person's face can be detected from the captured image, a predetermined image processing (face treatment) is performed on the captured image. However, if it becomes impossible to detect the face of a person due to the change of the direction or position of the face of the camera, the predetermined image processing (face treatment) is performed on the captured image. It is determined whether or not it is necessary to perform the facial treatment, and if it is determined that the processing needs to be performed, the facial treatment is performed on the captured image.

The control unit 1 operates by the power supply from the power supply unit (secondary battery) 2 and controls the overall operation of the image pickup apparatus (digital camera) according to various programs stored in the storage unit 3. The control unit 1 is provided with a CPU (central processing unit), memory, etc., which are not shown. The storage unit 3 has, for example, a ROM, a flash memory, and the like, and stores programs and various applications for realizing the present embodiment according to the operation procedure shown in FIGS. 3 and 4 described later. It has a program memory 3a, a work memory 3b for temporarily storing data such as flags. The storage unit 3 may have a configuration including a removable portable memory (recording medium) such as an SD card or an IC card capable of storing a photographed image, and a communication function (not shown) is used. In the state of being connected to the network through the network, it may include a storage area on the side of a predetermined server device. A portable memory such as an SD card is a memory that stores a plurality of high-quality photographed images (recorded images).

Although not shown, the operation unit 4 is provided with various keys such as a power key for turning on/off the power, a release key for instructing shooting, and a mode designation key for designating various operation modes. Performs power on/off, shooting processing, mode designation processing, and the like in accordance with the operation of this key. Note that the mode designation key is used to switch between a shooting mode (REC mode) and a playback mode of a captured image (recorded image) (PLAY mode) as a basic operation mode. In, it is possible to specify a beauty shooting mode, etc. for operating the beauty processing function. The control unit 1 executes a mode changing process, a photographing process, and the like in response to a key operation on the operation unit 4. The display unit 5 is a high-definition liquid crystal that displays an image, an organic EL (Electro Luminescence), or the like, and serves as a monitor screen (live view screen) that displays a captured image (live view image) in real time, or a captured image. It may be the playback screen for playing back images (recorded images).

The image pickup unit 6 is capable of picking up an image of a subject with high precision by forming an image of the subject from the image pickup lens 6a on an image pickup device (CCD, CMOS, or the like) not shown, and further various sensors, It has an analog processing unit and a digital processing unit. The control unit 1 is configured to acquire a captured image from the image capturing unit 6 and perform a capturing process according to a capturing instruction operation (release operation) to generate a captured image. That is, the control unit 1 acquires the image captured by the image capturing unit 6 as a RAW (Bayer pattern) image, and adjusts, for example, data interpolation (demosaic), color conversion, white balance, and sharpness to the RAW image. In addition to the development processing, the photographed image is generated by performing a facial processing as a predetermined image processing during the development processing. Then, the control unit 1 subjects the generated image (captured image) to image compression processing to convert it into a standard file format, and then records and stores it in the recording medium of the storage unit 3. In addition, when the face of a person is detected in the facial photographing mode for operating the facial processing function, the facial processing is performed not only on the RAW image but also on the live view image as described above. ..

FIG. 2 is a diagram showing an external view of an image pickup apparatus having a configuration capable of so-called self-portrait, in which the photographer himself/herself is photographed while watching the live view image.
The housing 11 constituting the image pickup device includes a thin rectangular display block 11a, a rectangular frame-shaped frame block 11b surrounding the display block 11a, and a lens block 11c connecting the display block 11a and the frame block 11b. With the configuration, the central portion of one end of the display block 11a (the central portion of the upper end in the illustrated example) and the central portion of the one end of the lens block 11c (the central portion of the lower end in the illustrated example) are hinge portions for lateral rotation (not shown). It is mounted so as to be rotatable in the lateral direction via. Further, the insides of both ends of the frame block 11b (inside the left and right upper sides in the example of FIG. 2A) and the ends of the lens block 11c (the left and right sides in the illustrated example) are hinge parts for vertical rotation (shown in the drawing). It is attached rotatably in the vertical direction via (omitted).

The display block 11a is provided with the display unit 5 in the center thereof, and the lens block 11c is provided with the image pickup lens 6a constituting the image pickup unit 6 in the center thereof. By rotating the display block 11a by 180° with respect to the lens block 11c via the display block 11a, it is possible to orient the display unit 5 and the imaging lens 6a in the same direction or in the opposite direction. The display block 11a and the lens block 11c are integrated into a main body block, and the inside of both end portions of the frame block 11b (inside the upper left and right sides in the example of FIG. 2A) and both end portions of the main body block (left and right in the illustrated example) The side portion) may be rotatably attached in the vertical direction via a hinge portion (not shown) for vertical rotation. 2(1) is a state in which the frame block 11b is closed with respect to the display block 11a in a state in which the display unit 5 and the imaging lens 6a are oriented 180° in the opposite direction for normal photographing, and FIG. Shows a state in which the frame block 11b is opened with respect to the display block 11a in a state in which the display unit 5 and the imaging lens 6a are oriented in the same direction so that a self-portrait can be taken.

Next, the operation concept of the image pickup apparatus (digital camera) in the first embodiment will be described with reference to the flowcharts shown in FIGS. 3 and 4. Here, each function described in these flowcharts is stored in the form of a readable program code, and the operation according to this program code is sequentially executed. Further, the operation according to the above-mentioned program code transmitted via a transmission medium such as a network can be sequentially executed. This is the same in other embodiments described later, and the operation unique to the present embodiment can be executed using a program/data externally supplied via a transmission medium in addition to the recording medium. 3 and 4 are flowcharts showing the outline of the operation of the characteristic part of the present embodiment in the overall operation of the image pickup apparatus, and when the flow of FIGS. 3 and 4 is exited, the overall operation is performed. Return to the main flow (not shown).

FIG. 3 and FIG. 4 are flowcharts for explaining the operation of the imaging device (the characteristic operation of the first embodiment) that is started when the beauty portrait shooting mode is designated. Hereinafter, the characteristic operation of the first embodiment will be described by taking as an example a case where continuous photography is performed in which a plurality of self images are continuously captured while changing the face orientation and the camera position for each release operation in the beauty shooting mode. Shall be explained.
First, the control unit 1 checks whether or not a release operation has been performed (step A1 in FIG. 3) or checks whether or not an operation for canceling the facial photographing mode has been performed (step A2). If the operation for canceling the beauty shooting mode is performed (YES in step A2), the process exits the flow of FIGS. 3 and 4, but if the release operation is performed (YES in step A1), A captured image (live view image) is acquired from the image capturing unit 6 (step A3).

Then, the process of detecting the face of the person by analyzing the acquired captured image (live view image) (step A4) is performed, and whether or not the face of the person can be detected from the captured image (whether the face is captured) Is checked) (step A5). That is, the shape, positional relationship, and the like of the person's eyes, mouth, nose, and forehead are comprehensively determined to detect whether or not the person's face is included in the captured image.

Now, when the face of the person in the captured image cannot be detected, for example, when the face of the person is not captured, or even when the face of the person is captured, the face of the person is substantially in front (for example, about If the orientation is not within 45° (NO in step A5), development processing not including facial treatment is performed to generate a photographed image (step A6). That is, a captured image is generated by subjecting the image captured by the image capturing unit 6 to development processing such as data interpolation (demosaic), color conversion, and adjustment of auto white balance and sharpness. To do. Then, the control unit 1 subjects the generated image (captured image) to image compression processing to convert it into a standard file format, and then records and stores it in the recording medium of the storage unit 3 (step A7). Then, the process returns to step A1 described above.

On the other hand, when the face of the person in the captured image is detected (YES in step A5), the process proceeds to the next step A8, and the development process including the facial treatment is performed to generate the captured image. That is, the above-described development processing is performed on the picked-up image as well as the predetermined facial image processing, and the facial processing is not limited to, for example, the face of a person, but skin-colored portions such as necks and arms. As an example, image processing is performed to correct the exposure of the image, whiten the skin by correcting the saturation and brightness, and to make wrinkles, rough skin, and spots less noticeable by performing soft focus. However, the facial treatment may be performed only in the detected face area. Then, the control unit 1 subjects the generated image (captured image) to image compression processing to convert it into a standard file format, and then records and stores it in the recording medium of the storage unit 3 (step A9). In this way, when the face of a person is detected and the facial processing is performed as the predetermined image processing, the facial flag (not shown) is set to “ON” as information indicating whether or not the predetermined image processing (face processing) can be executed. Processing is performed (step A10).

Then, the process proceeds to step A11 of FIG. 4 to check whether the above-mentioned facial beauty flag is turned on. Since the facial beauty flag is now turned on (YES in step A11), the state of the imaging device (camera) is predetermined. It is checked whether the state has changed to (step A12). Here, the fact that the imaging device (camera) has changed to a predetermined state means that the shooting environment has changed significantly, the positional relationship between the imaging device (camera) and the subject (person), and the angle of view with respect to the subject (person). If there is a large change in the image, it is checked whether the state has changed to a state in which it is determined that there is no intention to continue shooting that requires facial treatment.

That is, as an example, there is a case where the white balance light source determination is switched in the state where the AWB (auto white balance) adjustment is set. Further, as another example, when the light value value (LV value) suddenly changes in a state in which the AE (automatic exposure) adjustment is set (for example, the sum of the luminance values for each pixel of the captured image is 3 frames). If there is a change of 30% or more during the period). Further, as another example, when the lens unit is rotated (the positional relationship (direction) between the imaging lens 6a and the display unit 5 is changed), or when AF (autofocus) is set, the focus distance by AF is This is the case where there is a sudden change (change of 100% or more).

If the image pickup apparatus (camera) has not changed to a predetermined state (NO in step A12), it is checked whether a release operation has been performed (step A13), or whether an operation for canceling the facial photographing mode has been performed. (Step A14). When the operation for canceling the facial photographing mode is performed (YES in step A14), the process exits the flow of FIGS. 3 and 4, but when the release operation is performed (YES in step A13), A picked-up image (live view image) is acquired from the image pickup unit 6 (step A15), and the picked-up image is generated by subjecting the picked-up image to development processing including the above-described facial treatment (step A16). Then, the generated image (captured image) is subjected to image compression processing to be converted into a standard file format, and then recorded and stored in the recording medium of the storage unit 3 (step A17). After that, the process returns to step A11 to check whether the facial beauty flag is turned on.

In this way, during continuous shooting during self-portrait, in the state where the information (possible ON) is set as the information (face flag) indicating whether or not predetermined image processing (face processing) can be executed (YES in step A11). On the condition that the image pickup apparatus (camera) has not changed to a predetermined state (NO in step A12), it is determined that the facial processing needs to be performed without performing the processing of detecting the face of the person. The facial treatment is applied to the captured image. In other words, even if the person's face cannot be detected because the person's face is facing sideways, the face flag is not turned on unless the state of the imaging device (camera) has changed to a predetermined state. If the image is held, the facial treatment is applied to each of the picked-up images sequentially picked up while the image is held. On the other hand, if the imaging device (camera) has changed to a predetermined state (YES in step A12), the facial flag is turned off (step A18), and then the process returns to step A1 to perform the above-described process. repeat.

As described above, in the first embodiment, when the face of a person can be detected in the captured image captured by the image capturing unit 6, the control unit 1 performs predetermined image processing on the captured image. If a person's face cannot be detected, it is determined whether or not predetermined image processing needs to be performed on the captured image, and it is determined that predetermined image processing needs to be performed. Since it is instructed that predetermined image processing should be applied to the captured image, even if the orientation or position of the person's face changes when shooting a person as a subject, Predetermined image processing can be reliably performed.

The control unit 1 performs a release operation during continuous shooting in which a captured image is acquired from the image capturing unit 6 and a shooting process is performed each time a shooting instruction operation (release operation) is performed to generate a plurality of shot images. Since the operation of detecting the face of a person is performed for each captured image acquired every time the camera is shot, for example, when taking a self-portrait, the direction of the person's face and the position of the camera can be changed for each shooting instruction operation (release operation). Even when a plurality of images are continuously captured while changing the image, it is possible to reliably perform the predetermined image processing on each captured image.

If information (ON) is set as the information (face flag) indicating whether or not predetermined image processing can be performed on the captured image in which the person's face can be detected, the person's face is detected. Since it is determined that the predetermined image processing needs to be performed on the captured image that cannot be processed, it is sufficient to turn on the facial beauty flag instead of executing the processing for detecting the face of the person. The load can be reduced, and the face detection process only needs to be performed once, so that power can be saved.

For the captured image that has detected that the state of the imaging device (camera) has changed to a predetermined state, information (off) is set as information (face flag) indicating whether or not predetermined image processing can be executed. If it is determined that it is not necessary to perform the predetermined image processing, it is determined whether or not the predetermined image processing can be performed by determining whether the state of the photographing apparatus (camera) has changed to the predetermined state. Can be determined based on.

When the state of the image capturing device (camera) has changed to a predetermined state, it means that the light source determination of white balance is switched, the light value is changed rapidly, the imaging lens unit is rotated, or the focus distance is changed rapidly. , Which is the case in which the shooting environment changes significantly, or the positional relationship between the imaging device (camera) and the subject (person) or the angle of view with respect to the subject changes significantly. It can be detected that the state of the (camera) has changed to a predetermined state.

Since the predetermined image processing to be performed on the captured image is processing to apply a facial treatment to the skin color portion in the captured image, the same facial treatment is applied to not only the face of a person but also the neck, arms and the like. You can

In the above-described first embodiment, when the state of the image capturing apparatus (camera) changes to a predetermined state, the white balance light source determination is switched, the light value value is rapidly changed, and the imaging lens unit is rotated. In this case, any one of the cases where the focus distance is drastically changed is shown, but it is needless to say that the combination thereof may be used.
Further, although the face detection processing is not performed when the beauty flag is on, the face detection processing may be performed even when the beauty flag is on.

(Second embodiment)
The second embodiment of the present invention will be described below with reference to FIGS. 5 and 6.
In the above-described first embodiment, whether or not the predetermined image processing (face treatment) can be executed is determined based on whether or not the state of the photographing apparatus (camera) has changed to the predetermined state. However, in the second embodiment, when the face of the person cannot be detected, another detection process different from this detection process is performed, and a predetermined result is detected based on the detection result of the other detection process. This is to determine whether or not it is necessary to perform image processing. That is, if the face of the person can be detected, the skin color of the face is temporarily stored, and predetermined image processing (face-cleaning process) is executed when the face of the person cannot be detected. It is determined whether or not the possibility is based on whether or not the area of the skin color equivalent to the skin color of the temporary storage is equal to or larger than a predetermined area. Here, in both embodiments, the basic or the same thing is shown with the same reference numeral, and the description thereof will be omitted. Hereinafter, the characteristic part of the second embodiment will be mainly described. ..

FIG. 5 is a flow chart for explaining the operation of the image pickup apparatus (the characteristic operation of the second embodiment) that is started when the facial image capturing mode is designated.
First, the control unit 1 checks whether or not a release operation has been performed in a state where the facial photography mode has been designated (step B1) or whether an operation for canceling the facial photography mode has been performed (step B2). ). If the operation for canceling the facial beauty photographing mode is performed (YES in step B2), the flow in FIG. 5 is ended, but if the release operation is performed (YES in step B1), the imaging unit 6 A captured image (live view image) is acquired from (step B3).

Then, a process of detecting a human face by analyzing the acquired captured image (live view image) is performed (step B4), and it is checked whether or not the human face is included in the captured image (step B5). When it is detected that a person's face is included in the captured image (YES in step B5), a face detection area (a rectangular area surrounded by a broken line in the figure) is identified as shown in FIG. 6(1). The skin color information is acquired from the face detection area and temporarily stored in the work memory 3b (step B6). In this case, in order to remove some spots, moles, etc., the color in the area with the widest area among the areas in which the same color (same or similar color) is continuous in the face detection area is the skin color information. To get as. The skin color information of the temporary storage may be information of one color or may be information indicating a range of colors.

Next, the captured image is subjected to development processing including facial treatment to generate a captured image (step B7). The facial treatment in this case is similar to that of the first embodiment, and the exposure of the image is corrected and the saturation and the brightness are corrected not only for the face of the person but also for the skin-colored portions such as the neck and the arms. Image processing is performed to make the skin whiter by performing the processing described above, and to make wrinkles, rough skin, and spots less noticeable by performing soft focus, etc., but the facial processing is performed only within the detected face area. May be applied. Next, the generated image (captured image) is subjected to image compression processing to be converted into a standard file format, and then recorded and stored in the recording medium of the storage unit 3 (step B8). Then, the process returns to step B1 described above, and the above operation is repeated.

Here, the image captured by the release operation (YES in step B1) is acquired next (step B3), and the face detection processing (step B4) is performed on this captured image, and as a result, the face of the person is detected. If not possible (NO in step B5), the process proceeds to step B9 to check whether the skin color information is temporarily stored. That is, it is checked whether face detection is performed before the release operation this time and the skin color information is temporarily stored in the work memory 3b. When the skin color information is temporarily stored (YES in step B9), the process proceeds to step B10. In this step B10, the skin color information temporarily stored at the time of face detection is read out, and a region having the same color as the skin color information (the same color or an approximate color) and having a predetermined area or more (for example, the entire captured image (5% or more of the area) is detected as a skin area. As a result of the detection processing, it is checked whether the skin area satisfying the above conditions can be detected from the captured image (step B11).

FIG. 6B shows the skin color information (face skin color) temporarily stored at the time of face detection when the face cannot be detected because the face of the person in the captured image is facing sideways. ) Is a region in which the same color is continuous, and is a diagram illustrating a case where a region having a predetermined area or more is detected as a skin region. In the figure, a shaded region (face and neck region) Indicates the detected skin area. If the corresponding skin area cannot be detected (NO in step B11), a development process that does not include facial treatment is performed to generate a captured image (step B12), and the generated image (captured image) is generated. Then, the image is compressed and converted into a standard file format, and then recorded and stored in the recording medium of the storage unit 3 (step B8). Then, the process returns to step B1 described above.
Also, when the skin color information is not temporarily stored (NO in step B9), similarly, development processing that does not include facial treatment is performed to generate a captured image (step B12), and the generated image (captured image) is generated. The image is subjected to image compression processing to be converted into a standard file format, and then recorded and saved in the recording medium of the storage unit 3 (step B8). After that, the process returns to step B1 described above, and the above operation is repeated.

On the other hand, even if a person's face cannot be detected while the skin color information (face skin color) is temporarily stored at the time of face detection as described above, color information equivalent to the skin color information in the temporary storage is obtained. As a result of performing the process of detecting an area that is included and has a predetermined area or more as the human skin area (step B10), if the skin area can be detected (YES in step B11). When it is determined that the facial processing needs to be performed, the process proceeds to step B7, and the captured image is subjected to the development processing including the facial processing to generate a photographed image. In this case, for example, when there are a plurality of skin areas having a predetermined area or more, if the widest area is detected as the face area, it is possible to apply the facial treatment to only the face area. Becomes Next, the generated image (captured image) is subjected to image compression processing to be converted into a standard file format, and then recorded and stored in the recording medium of the storage unit 3 (step B8). Then the above

As described above, in the second embodiment, when the face of a person cannot be detected, the control unit 1 performs another detection process different from this detection process, and the detection result of the other detection process. Since it is determined whether or not the predetermined image processing needs to be performed based on the above, it is possible to perform multifaceted detection regardless of the unique detection of the face detection processing.

In the state in which the skin color information (face skin color) is temporarily stored at the time of face detection, as another detection processing described above, it is determined whether the skin color area equivalent to the skin color in the temporary storage is a predetermined area or more. Since the detection process is performed and whether or not the facial treatment can be executed is determined based on the detection result, even if the face cannot be detected, the skin portion can be accurately specified in the captured image. Then, facial treatment can be performed.

(Third Embodiment)
Hereinafter, a third embodiment of the present invention will be described with reference to FIGS. 7 and 8.
In the third embodiment, similarly to the above-described second embodiment, when the face of a person cannot be detected, another detection process different from this detection process is performed, and the detection of the other detection process is performed. Based on the result, it is determined whether or not it is necessary to perform predetermined image processing. Further, in the third embodiment, as another detection process when the face of the person cannot be detected, the color of the hair is temporarily stored when the face of the person can be detected. A process of detecting whether or not the area of the skin color equivalent to the hair color of the temporary memory is equal to or larger than a predetermined area is performed, and based on the detection result, it is determined whether or not the predetermined image processing (facial treatment) can be performed. It was done like this. Here, in the first and third embodiments, basically or the same in terms of names are denoted by the same reference numerals, the description thereof will be omitted, and the characteristic part of the third embodiment will be mainly described below. I shall explain.

FIG. 7 is a flow chart for explaining the operation of the image pickup apparatus (the characteristic operation of the third embodiment) which is started when the facial photographing mode is designated. Since FIG. 7 is basically the same as FIG. 5 of the second embodiment, basically the same parts will be briefly described.
First, the control unit 1 checks whether or not a release operation has been performed in the state where the beauty shooting mode is designated (step C1), and also checks whether an operation to cancel the beauty shooting mode has been performed (step C2). ). When the operation for canceling the facial beauty shooting mode is performed (YES in step C2), the flow in FIG. 7 is exited, but when the release operation is performed (YES in step C1), the imaging unit 6 A captured image (live view image) is acquired from (step C3). Then, a process of detecting a person's face by analyzing the acquired captured image (live view image) is performed (step C4), and it is checked whether or not the person's face is included in the captured image (step C5).

When it is detected that the person's face is included in the captured image (YES in step C5), a face detection area (a rectangular area surrounded by fine broken lines in the figure) is identified as shown in FIG. 8(1). At the same time, the peripheral area is specified as the hair area based on the face detection area, and the hair color information is acquired from the hair area and temporarily stored in the work memory 3b (step C6). That is, the hair area is a rectangular area (a rectangular area surrounded by a rough broken line in the figure) excluding the face detection area, and its length (width) in the X direction is 1.5 times the width of the face detection area. The length (height) in the Y direction is an area having a size 1.5 times the height of the face detection area, and the lower end of the area is arranged at a position matching the lower end of the face detection area. The area is detected as a hair area.

The color in the area having the widest area among the areas in which the same color (same or similar color) is continuously detected in the hair area thus detected is acquired as the hair color information and temporarily stored. Keep it. The hair color information of the temporary storage may be information of one color or may be information indicating a range of colors. Then, the captured image is subjected to development processing including facial treatment to generate a photographed image, but facial treatment may be performed only within the detected face area (step C7). The generated image (captured image) is subjected to image compression processing to be converted into a standard file format, and then recorded and stored in the recording medium of the storage unit 3 (step C8). Then, the process returns to step C1 described above.

Here, next, an image captured by the release operation (YES in step C1) is acquired (step C3), and the face detection process (step C4) is performed on the captured image, and as a result, the face of the person is detected. If not possible (NO in step C5), the process moves to step C9 to check whether the hair color information is temporarily stored. That is, it is checked whether face detection was performed before the release operation this time and hair color information was temporarily stored in the work memory 3b. When the hair color information is temporarily stored (YES in step C9), the process proceeds to step C10, and the face possibility area in the captured image (a rectangle surrounded by a fine broken line in the figure) is displayed as shown in FIG. 8B. Area) is detected. In this case, an area in which the same color (same color or similar color) as the skin color (general skin color) used during normal face detection is continuous is detected as a face possibility area. Here, if the face possibility area cannot be detected (NO in step C11), a development process that does not include facial treatment is performed to generate a captured image (step C14), and the generated image (captured image) is generated. On the other hand, after image compression processing is performed to convert it into a standard file format, it is recorded and saved in the recording medium of the storage unit 3 (step C8). After that, the process returns to step C1 described above, and the above operation is repeated.

When the face possibility area can be detected (YES in step C11), the hair color information temporarily stored at the time of face detection is read out, and color information equivalent to this hair color information is read. A process is performed to detect an area that is included and has a predetermined area or more as a person's hair area (step A12). That is, the color is the same as the color information of the hair (the same color or an approximate color) and is a rectangular area excluding the face detection area as shown in FIG. 8B (a rectangular area surrounded by a rough broken line in the figure). In the size area, the length (width) in the X direction is 2.0 times the width of the face detection area, and the length (height) in the Y direction is 1.5 times the height of the face detection area. Then, the area arranged at the position where the lower end of the area matches the lower end of the face possibility area is detected as the hair area.

Then, whether or not the hair region corresponding to the above condition can be detected from the captured image, that is, the hair region is temporarily stored at the time of face detection to the extent that it can be regarded as hair (for example, about 30%). It is checked whether or not the area occupying the existing hair color can be detected as the hair area (step C13). Here, if the hair region cannot be detected (NO in step C13), development processing that does not include facial treatment is performed to generate a captured image (step C14), and image compression processing is performed on this generated image (captured image). Then, the data is converted into a standard file format and recorded and stored in the recording medium of the storage unit 3 (step C8). Then, the process returns to step C1 described above.
Similarly, when the hair color information is not temporarily stored (NO in step C9), similarly, a development process that does not include facial treatment is performed to generate a captured image (step C14), and the generated image (captured image) is generated. After the image compression processing is performed on the above to convert it into a standard file format, it is recorded and stored in the recording medium of the storage unit 3 (step C8). After that, the process returns to step C1 described above, and the above operation is repeated.

On the other hand, in the state where the color information of the hair is temporarily stored at the time of face detection, even if the face of the person cannot be detected, the area has the same color information as the color information of the hair, and has a predetermined area or more. As a result of performing the process of detecting the area of No. 1 as the hair area of the person (step C12), if the hair area can be detected (YES in step C13), it is determined that the facial processing needs to be performed. Then, the process proceeds to step C7 to perform a development process including a facial beauty process on the face possibility area in the captured image to generate a captured image. That is, when there is a hair area around the face possibility area, it is determined that the face possibility area is a face area, and the facial treatment is performed on the face possibility area. Then, the generated image (captured image) is subjected to image compression processing to be converted into a standard file format, and then recorded and stored in the recording medium of the storage unit 3 (step C8). After that, the process returns to step C1 described above, and the above operation is repeated.

As described above, in the third embodiment, when the control unit 1 can detect a person's face, it temporarily stores the hair color and cannot detect the person's face. In this case, if the face possibility area is specified in the captured image and there is an area equal to the skin color of the temporary storage and having a predetermined area or more in the periphery, the area is detected as a hair area. Since it is determined whether or not the facial processing can be executed based on the detection result, if there is a hair area around the face possibility area, the face possibility area is not the face area. It can be determined that there is a face, and the face treatment can be performed, and the face treatment can be performed by accurately specifying the face portion of the person in the captured image based on the hair color of the person.

In the above-described third embodiment, a face possibility area is detected when a person's face cannot be detected, and if a hair area exists around this face possibility area, Although the face possibility area is judged to be the face area and the facial treatment is performed, the hair area is detected based on the hair color of the temporary memory, and the peripheral face area is detected based on this hair area. You may make it detect and apply a facial treatment.

(Fourth Embodiment)
Hereinafter, a fourth embodiment of the present invention will be described with reference to FIG.
In the fourth embodiment, similarly to the second and third embodiments described above, when a human face is not detected, another detection process different from this detection process is performed, and the detection result of this other detection process is displayed. Based on this, it is determined whether or not it is necessary to perform predetermined image processing. Further, in the fourth embodiment, as another detection process when the face of the person cannot be detected, the aberration correction process is performed based on the information for correcting the aberration of the image pickup lens 6a forming the image pickup unit 6. At the same time, a process of detecting a person's face from the corrected captured image is performed, and based on the detection result, it is determined whether or not a predetermined image process (face beauty process) can be performed. Note that, here, as the aberration correction processing, a known technique is used to correct the distortion aberration. Here, in the first and fourth embodiments, basically or the same things in terms of names are denoted by the same reference numerals, the description thereof will be omitted, and the characteristic part of the fourth embodiment will be mainly described below. I shall explain.

FIG. 9 is a flow chart for explaining the operation of the image pickup apparatus (the characteristic operation of the fourth embodiment) which is started when the facial image capturing mode is designated. Since FIG. 9 is basically the same as FIG. 5 of the second embodiment, basically the same parts will be briefly described.
First, the control unit 1 checks whether or not a release operation has been performed in the state where the beauty shooting mode is designated (step D1), and checks whether an operation to cancel the beauty shooting mode has been performed (step D2). ). If the operation for canceling the facial photographing mode has been performed (YES in step D2), the flow in FIG. 9 is exited, but if the release operation has been performed (YES in step D1), the imaging unit 6 A captured image (live view image) is acquired from (step D3).

Then, a process of detecting a human face by analyzing the acquired captured image (live view image) is performed (step D4), and it is checked whether or not the human face is included in the captured image (step D5). When it is detected that a person's face is included in the captured image (YES in step D5), the captured image is subjected to development processing including facial beauty processing to generate a captured image. The facial treatment may be applied only to the inside (step D7). The generated image (captured image) is subjected to image compression processing to be converted into a standard file format, and then recorded and stored in the recording medium of the storage unit 3 (step D7). Then, the process returns to step D1 described above.

Here, next, an image captured by the release operation (YES in step D1) is acquired (step D3), and the face detection processing (step D4) is performed on this captured image, and as a result, the face of the person is detected. If it is not possible, for example, if the face of the person moves from the central part of the imaging lens 6a to the peripheral part thereof (NO in step D5), the process moves to step D8 and the captured image is processed. Distortion aberration correction processing is performed (step D9). That is, the distortion aberration correction is performed based on the information for correcting the aberration of the imaging lens 6a, and the process of detecting the person's face again is performed on the corrected captured image (step D9). In this case, if the person's face cannot be detected even if the distortion aberration correction is performed (NO in step D10), development processing that does not include facial treatment is performed to generate a captured image (step D11), and the generated image (captured image is captured. The image is subjected to image compression processing to be converted into a standard file format, and then recorded and stored in the recording medium of the storage unit 3 (step D7).

On the other hand, as a result of performing the distortion aberration correction (step D8), when the face of the person can be detected (YES in step D10), it is determined that the facial treatment needs to be performed, and the process proceeds to step D6. , The captured image is subjected to development processing including facial treatment to generate a captured image. Similar to the first embodiment, the facial treatment may be performed not only on the face of a person but also on skin-colored portions such as the neck and arms, but facial treatment may be performed only on the facial region. Next, the generated image (captured image) is subjected to image compression processing to be converted into a standard file format, and then recorded and stored in the recording medium of the storage unit 3 (step D7). Then, the process returns to step D1 described above, and the above operation is repeated.

As described above, in the fourth embodiment, the control unit 1 performs the aberration correction process based on the information for correcting the aberration of the image pickup lens 6a included in the image pickup unit 6 when the face of the person cannot be detected. In addition to performing the processing for detecting the face of the person from the corrected captured image and determining whether or not to execute the facial treatment based on the detection result, the face of the person is determined to be the aberration of the imaging lens 6a. Even if the position moves to a large position, the face detection process can be performed again after performing the aberration correction process, and the face detection becomes more reliable.

In addition, when the face cannot be detected even after performing the process of detecting the skin area of the person, which is the detection process of the above-described second embodiment, further, as the process of the above-described third embodiment, You may make it perform the process which detects a person's hair area. In this case, the hair color may be acquired when the face is first detected.

Further, if the face of the person cannot be detected even after performing the aberration correction which is the detection processing of the above-described fourth embodiment, the skin area of the person is further processed as the processing of the above-described second embodiment. May be performed, or as the processing of the third embodiment described above, processing of detecting the hair area of a person may be performed. In this case, the skin color or the hair color may be acquired when the face is first detected.

In each of the above-described embodiments, the case where the invention is applied to the image pickup apparatus including the image pickup means as the image processing apparatus is not limited to this, and the image processing apparatus is not provided with the image pickup means but is provided with the separate communication means. The captured image captured by the external device including the unit may be acquired via the communication unit.

In each of the above-described embodiments, as the predetermined image processing, the facial processing is exemplified by skin beautification and whitening. However, the present invention is not limited to this, and the facial processing for processing the shape of the face, eyelashes, eyebrows, mouth, etc. Of course, the target of image processing is not limited to the face, and a process of correcting the background of the face according to the skin color of the face may be performed.

In each of the above-described embodiments, the case where a plurality of images are continuously shot while changing the direction of the face and the position of the camera for each shooting instruction operation (release operation) has been described. However, continuous shooting is performed according to the shooting instruction operation. The same is applicable when operating the function. For example, when a plurality of still images (for example, 3 or 5 at 3 second intervals) are continuously shot at high speed by one release operation, the face of a person is detected for each captured image. Alternatively, the face of a person may be detected during moving image shooting.

Further, although cases have been described with the above-described embodiments where the image processing apparatus is applied to an imaging apparatus, the present invention is not limited to this, and, for example, a personal computer with a camera function, a PDA (personal portable information communication device), and a tablet. It may be a terminal device, a mobile phone such as a smartphone, an electronic game, a music player, or the like.

Further, the “apparatus” and “unit” shown in each of the above-described embodiments may be separated into a plurality of housings according to their functions, and are not limited to a single housing. Further, the steps described in the above-described flowcharts are not limited to time-series processing, and a plurality of steps may be processed in parallel or separately and independently.

Although the embodiment of the present invention has been described above, the present invention is not limited to this, and includes the invention described in the claims and the scope equivalent thereto.
The inventions described in the claims of the present application will be additionally described below.
(Appendix)
(Claim 1)
The invention according to claim 1 is
An acquisition means for acquiring a captured image,
First detection means for detecting a face of a person from the captured image acquired by the acquisition means;
The captured image is subjected to a capturing process to generate a captured image for storage, and a predetermined image process is performed on the captured image when the face of the person can be detected by the first detecting unit. Image processing means to perform,
Determination means for determining whether or not it is necessary to perform the predetermined image processing on the captured image when the face of the person cannot be detected by the first detection means,
If it is determined by the determination means that the predetermined image processing needs to be performed on the captured image for which the face of the person cannot be detected by the first detection means, On the other hand, control means for instructing the image processing means to perform the predetermined image processing,
An image processing apparatus comprising:
(Claim 2)
The invention described in claim 2 is the image processing apparatus according to claim 1,
The image processing means performs the shooting process and the predetermined image process according to a shooting instruction operation,
Each of the first detection means is acquired by the acquisition means each time the shooting instruction operation is performed during continuous shooting in which the shooting instruction operation is continuously performed and a plurality of shot images are generated. Performs the operation to detect the face of a person in the captured image,
It is characterized by
(Claim 3)
The invention described in claim 3 is the image processing apparatus according to claim 2,
Setting means for performing, when the face of the person can be detected by the first detecting means, processing for setting, as information indicating whether or not the predetermined image processing can be executed, to the captured image , Prepare for more,
When the setting information sets the permission information, the determination means sets the predetermined value for the captured image even when the face of the person cannot be detected by the first detection means. It is determined that the image processing of
It is characterized by
(Claim 4)
The invention according to claim 4 is the image processing apparatus according to claim 3,
Further comprising a second detecting means for detecting whether or not the state of the image processing apparatus has changed to a predetermined state,
The setting unit determines whether or not information indicating whether or not the predetermined image processing can be performed is performed on the captured image in which the state of the image processing apparatus is detected to be changed to the predetermined state by the second detecting unit. Perform the process of setting information,
The determining means determines that the predetermined image processing need not be performed when the setting information has been set by the setting means.
It is characterized by
(Claim 5)
The invention described in claim 5 is the image processing apparatus according to claim 4,
The second detection means determines whether the state of the image processing apparatus is predetermined to be at least one of switching of light source determination of white balance, rapid change of light value value, rotation of imaging lens unit, and rapid change of focus distance. Change to the state of
It is characterized by
(Claim 6)
The invention according to claim 6 is the image processing apparatus according to claim 1 or 2,
What is the detection processing by the first detection unit for the captured image when the face of the person cannot be detected by the first detection unit when the image processing unit performs the photographing process? Further comprising a third detecting means for performing another different detecting process,
The discriminating means performs the predetermined image processing on the captured image based on the detection result of the third detecting means when the face of the person cannot be detected by the first detecting means. Determine if it is necessary,
It is characterized by
(Claim 7)
The invention according to claim 7 is the image processing apparatus according to claim 6,
Further comprising: an acquisition unit that acquires the skin color information of the person from the face region in the captured image when the first detection unit cannot detect the face of the person,
The third detection means is an area having color information equivalent to the skin color information acquired by the acquisition means from the captured image acquired by the acquisition means, and an area having a predetermined area or more, Perform processing to detect the skin area of a person,
In the case where the determination means cannot detect the face of the person by the first detection means and the skin area of the person can be detected by the third detection means, It is determined that the predetermined image processing needs to be performed on the captured image,
It is characterized by
(Claim 8)
The invention according to claim 8 is the image processing apparatus according to claim 6,
When the face of the person can be detected by the first detecting means, an acquisition means for acquiring the color information of the hair of the person from the area around the face is further provided.
The third detection means is an area having color information equivalent to the color information of the hair acquired by the acquisition means in the captured image acquired by the acquisition means, and an area having a predetermined area or more, Performs processing to detect as a person's hair area,
When the determination means cannot detect the face of the person by the first detection means and when the hair area of the person can be detected by the third detection means, It is determined that the predetermined image processing needs to be performed on the captured image,
It is characterized by
(Claim 9)
According to a ninth aspect of the invention, in the image processing apparatus according to the sixth aspect,
The acquisition unit acquires a captured image captured by an imaging unit included in the image processing device,
The third detection unit performs a process of detecting a face of a person from a captured image that is corrected based on information that corrects aberration of an imaging lens that forms the imaging unit,
When the determination means cannot detect the face of the person by the first detection means and when the face of the person can be detected by the third detection means, It is determined that the predetermined image processing needs to be performed on the captured image,
It is characterized by
(Claim 10)
According to a tenth aspect of the invention, in the image processing apparatus according to any one of the first to ninth aspects,
The predetermined image processing performed by the image processing unit on the captured image acquired by the acquisition unit is a process of applying a facial processing to a skin color portion in the captured image.
It is characterized by
(Claim 11)
The invention according to claim 11 is
A process of acquiring a captured image,
A process of detecting a person's face from the acquired captured image;
A process of performing a shooting process on the captured image to generate a captured image for storage, and a process of performing a predetermined image process on the captured image when the face of the person can be detected,
A process of determining whether or not the predetermined image processing needs to be performed on the captured image when the face of the person cannot be detected;
If it is determined that the predetermined image processing needs to be performed on the captured image in which the face of the person cannot be detected, the predetermined image processing should be performed on the captured image. Process to instruct that,
And an image processing method including:
(Claim 12)
The invention according to claim 12 is
To the computer
A function to acquire a captured image,
A function of detecting a person's face from the acquired captured image;
A function of performing shooting processing on the captured image to generate a captured image for storage, and performing predetermined image processing on the captured image when the face of the person can be detected,
A function of determining whether or not the predetermined image processing needs to be performed on the captured image when the face of the person cannot be detected when the image capturing unit performs the image capturing process,
If it is determined that the predetermined image processing needs to be performed on the captured image in which the face of the person cannot be detected, the predetermined image processing should be performed on the captured image. And the ability to instruct
It is a program for realizing.

1 Control Section 3 Storage Section 3a Program Memory 4 Operating Section 5 Display Section 6 Imaging Section 6a Imaging Lens

Claims (8)

An image acquisition means for acquiring a captured image by the imaging means,
First detection means for detecting a face of a person from the captured image acquired by the image acquisition means;
Second detecting means for detecting a face of a person from the captured image obtained by correcting the captured image based on the information for correcting the aberration of the imaging lens forming the imaging means;
Can not detect the face of a person by said first detection means and, where it was able to detect a person's face by said second detection means, a person with respect to the obtained captured image Image processing means for performing predetermined image processing according to the face,
Images processing device Ru comprising a. Storage means for storing the skin color information of the detected face when the face of the person can be detected by the first detecting means;
When the face of the person cannot be detected by both the first detecting unit and the second detecting unit, the color information of the captured image in which the face of the person cannot be detected is stored by the storage unit. Based on the skin color information, further comprising a determination unit that determines whether or not to perform the predetermined image processing on the captured image in which the face of the person cannot be detected,
The image processing means performs the predetermined image processing when the determination means determines to perform the predetermined image processing on the captured image in which the face of the person cannot be detected,
The image processing apparatus according to claim 1. Storage means for storing hair color information of the person whose face has been detected when the face of the person has been detected by the first detection means;
When the face of the person cannot be detected by both the first detecting unit and the second detecting unit, the color information of the captured image in which the face of the person cannot be detected and the color information stored by the storing unit are stored. Based on the color information of the hair, further comprises a determination means for determining whether or not to perform the predetermined image processing on the captured image in which the face of the person cannot be detected,
The image processing means performs the predetermined image processing when the determination means determines to perform the predetermined image processing on the captured image in which the face of the person cannot be detected,
The image processing apparatus according to claim 1. During the continuous shooting in which the first detection unit, the second detection unit, and the image processing unit are continuously instructed to shoot, and the captured image is acquired by the image acquisition unit for each shooting command. Perform each function ,
The image processing apparatus according to claim 1 . The image acquisition means acquires a captured image by the imaging device,
The first detecting unit, the second detecting unit, and the image processing unit perform their respective functions during a self-portrait when the user operating the image pickup apparatus takes a picture of his or her face with the image pickup apparatus. ,
The image processing apparatus according to claim 1. A terminal device comprising the image processing device according to claim 5 and the imaging device according to claim 5. An image acquisition process for acquiring a captured image by the imaging means,
A first detection process for detecting a human face from the captured image acquired by the image acquisition process;
A second detection process of detecting a person's face from the captured image in which the acquired captured image is corrected based on the information for correcting the aberration of the imaging lens forming the imaging unit;
When the person's face can not be detected by the first detection processing and the person's face can be detected by the second detection processing, a predetermined value is set for the acquired captured image. Image processing to apply the image processing of
An image processing method comprising: An image acquisition process for acquiring a captured image by the imaging means,
A first detection process for detecting a person's face from the captured image acquired by the image acquisition process;
A second detection process of detecting a person's face from the captured image obtained by correcting the acquired captured image based on information for correcting aberration of the imaging lens forming the imaging unit;
When the person's face cannot be detected by the first detection processing and the person's face can be detected by the second detection processing, a predetermined value is set for the acquired captured image. Image processing to apply the image processing of
A storage medium that non-temporarily stores a program that causes a computer to execute.

Images