JP2003030647A - Image processor, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of detecting only a human body area from images. SOLUTION: The red eye detection part 281 of a digital camera 1 extracts the red eye candidate areas to be the candidates of a red eye area of a non-flash image photographed without flash light emission and a flash image photographed with the flash light emission. Since a red eye phenomenon occurs only in the case of the flash light emission, both images are compared and the red eye candidate area present only in the flash image is detected as the red eye area. Then, a human body detection part 282 extracts a skin color area around the detected red eye area and detects the skin color area as the human body area. Thus, only the human body area is detected at a high speed.

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to a technique for detecting a person in an image.

[0002]

2. Description of the Related Art Conventionally, a digital image obtained by photographing with a digital still camera (hereinafter referred to as "digital camera") or by capturing an output image with a scanner is used. A technique has been proposed in which a person in an image serving as a main subject is detected by performing the above processing.

As a general technique for detecting a person in an image, an area (hereinafter, referred to as "skin color area") that is close to the human skin color in the image is extracted, and the skin color area is extracted as a person area. There is known a process of detecting as (hereinafter, referred to as “person area”). However, if the skin color region is simply extracted from the image in this way, there is a possibility that even an object or a building having a color close to the skin color may be detected as a human region.

In order to deal with this, there is a technique of performing shape determination processing to determine whether the shape of the extracted skin color area is very similar to the shape of a person's face or the like, and detecting the skin color area as a person area only when the shape is very similar. Are known.

Further, assuming that a red-eye effect will occur in the main subject (person) in the case of an image at the time of flash emission, a red area is detected from the extracted skin color area, and if there is a red area, it is detected. A technique for detecting a skin color region as a human region is known.

[0006]

However, when the shape determination is performed as described above, there is a problem that a large amount of processing time is required because of complicated processing. In particular, it is difficult to apply such shape determination processing to a digital camera or the like in which there is a strong demand for shortening the image processing time.

Further, in the case of detecting the red area as described above, for example, when there is a red illumination light source in the area of a building having a color close to the skin color, the area of the building is also a person area. However, the human region cannot be detected with sufficient accuracy.

The present invention has been made in view of the above problems, and an object thereof is to provide a technique capable of detecting only a human region in an image while realizing a processing speed of a relatively short time. To do.

[0009]

In order to solve the above-mentioned problems, an invention according to claim 1 is an image processing apparatus, wherein a first image taken without emitting a flash to the same subject, A red-eye region detecting means for detecting a red-eye region in the second image, and a skin-color region in the vicinity of the red-eye region in the second image by using a second image captured by emitting a flash. And a person area detecting unit that detects the person area.

According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the red-eye area detecting means is a red-eye candidate which is a candidate for the red-eye area of each of the first image and the second image. A candidate area extracting unit that extracts an area and a red eye candidate area of the first image and a red eye candidate area of the second image are compared, and a red eye candidate area only in the second image is set as the red eye area. And a comparison means.

According to a third aspect of the present invention, in the image processing apparatus according to the second aspect, the red-eye area detecting means includes an occurrence rate calculating means for calculating a red-eye occurrence rate at which the red-eye phenomenon of the subject occurs. The method further includes: a criterion changing unit that changes the criterion for extracting the red eye candidate region by the candidate region extracting unit according to the red eye occurrence rate.

According to a fourth aspect of the present invention, in the image processing apparatus according to any one of the first to third aspects, an image pickup means for picking up the subject and the person area detected by the person area detecting means are provided. Image processing condition setting means for setting image processing conditions and / or imaging conditions of the imaging means based on the image processing conditions.

According to a fifth aspect of the present invention, there is provided an image processing method, wherein the same subject is photographed without emitting a flash and a second image photographed with a flash. And a red-eye region detecting step of detecting a red-eye region in the second image, and a human-region detecting step of detecting a skin-color region near the red-eye region in the second image as a human region. ing.

According to a sixth aspect of the present invention, there is provided a program, which causes a computer to take a first image of a same subject photographed without emitting a flash and a second image photographed by emitting a flash. A red-eye area detecting unit that detects a red-eye area in the second image using the image; and a human-area detecting unit that detects a skin-color area near the red-eye area in the second image as a human area. It functions as an image processing apparatus provided.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

<1. First Embodiment><1-1. Structure of Digital Camera> FIG. 1 to FIG.
FIG. 1 is a diagram showing an example of an external configuration of a digital camera 1 which is an image processing device that acquires an image as digital data,
1 is a front view, FIG. 2 is a rear view, and FIG. 3 is a bottom view. As shown in FIG. 1, the digital camera 1 is composed of a box-shaped camera body 2 and a rectangular parallelepiped imaging unit 3.

A zoom lens 301 with a macro function, which is a taking lens, is provided on the front side of the image pickup section 3, and a light control sensor for receiving the reflected light of the flash light from the object, like a silver halide lens shutter camera. 305 is provided.

On the front side of the camera body 2, a grip part 4 is provided at the left end, a built-in flash 5 is provided at the upper center, and a shutter button 8 is provided on the top side. The shutter button 8 is a two-step switch that can detect a half-pressed state (S1) and a full-pressed state (S2) as used in a silver halide camera.

On the other hand, as shown in FIG.
On the rear side of the LCD, a liquid crystal display (LC
D: Liquid Crystal Display) 10 is provided.
Further, below the LCD 10, key switch groups 221 to 226 for operating the digital camera 1 and a power switch 227 are provided.

Further, on the rear side of the camera body 2,
An operation changeover switch 14 for switching the operation mode of the camera between “shooting” and “playback” is provided. When the operation mode is set to "shoot", it is possible to take a picture and generate an image of the subject, and when the operation mode is set to "play", the image recorded on the memory card is read. Is reproduced on the LCD 10. The operation changeover switch 14 is a two-contact slide switch, and when it is slide-set to the lower position, the operation mode becomes “shooting”,
When slide set to the upper position, the operation mode is "Play"
Becomes

Further, a quad switch 230 is provided on the right side of the rear surface of the camera so that various operations can be performed. For example, when the operation mode is “shooting”, button 2
By pressing 31, 32, the zooming magnification is changed.

On the back surface of the image pickup unit 3, as shown in FIG.
Electronic View Finder (EVF)
r) 31 is provided. The EVF 31 can electronically display a subject through an eyepiece lens. The photographer can take a photograph while checking the subject using either the LCD 10 or the EVF 31.

The LCD 10 is provided below the EVF 31.
A person detection mode switching button 322 is provided together with an LCD button 321 for turning on / off. The shooting modes of the digital camera 1 include a normal shooting mode in which normal shooting is performed by standard control, and a person detection mode in which a person in an image is detected. By pressing the person detection mode switching button 322, the photographing mode can be switched between the normal photographing mode and the person detection mode.

As shown in FIG. 3, a card slot 17 is provided on the bottom surface of the camera body 2. The card slot 17 can be loaded with a removable memory card 91 or the like for recording captured images and the like.

<1-2. Internal Configuration of Digital Camera> Next, the internal configuration of the digital camera 1 will be described. FIG. 4 is a diagram showing an outline of the arrangement of each component in the image pickup unit 3. FIG. 5 is a block diagram showing the configuration of the digital camera 1.

As shown in FIG. 4, an image pickup circuit having a CCD 303 is provided at an appropriate position behind the zoom lens 301 in the image pickup section 3. Further, inside the image pickup unit 3, a zoom motor M1 for changing the zoom ratio of the zoom lens 301 and for moving the lens between the storage position and the image pickup position, and a zoom lens 30 for adjusting the focus.
1, a focus motor M2 for moving the focus lens 311 in the first lens unit 3 and a diaphragm 3 provided in the zoom lens 301.
And a diaphragm motor M3 for adjusting the opening diameter of No. 02.

As shown in FIG. 5, the zoom motor M1, the focus motor M2, and the aperture motor M3 are respectively driven by the zoom motor drive circuit 215, the focus motor drive circuit 214, and the aperture motor drive circuit 216 provided in the camera body 2. Driven. In addition, each drive circuit 214-
Reference numeral 216 drives each of the motors M1 to M3 based on a control signal provided from the overall control unit 270 of the camera body unit 2.

The CCD 303 converts the optical image of the subject formed by the zoom lens 301 into R (red), G (green),
An image signal of a B (blue) color component (a signal composed of a signal sequence of pixel signals received by each pixel) is photoelectrically converted and output.

The timing generator 314 generates a drive control signal for the CCD 303 based on the reference clock transmitted from the timing control circuit 202 of the camera body 2. The timing generator 314 generates, for example, a clock signal such as a timing signal for starting / ending integration (starting / ending exposure), a read control signal (a horizontal synchronizing signal, a vertical synchronizing signal, a transfer signal, etc.) for a light receiving signal of each pixel. , To the CCD 303.

The signal processing circuit 313 subjects the image signal (analog signal) output from the CCD 303 to predetermined analog signal processing. The signal processing circuit 313 is C
It has a DS (correlated double sampling) circuit and an AGC (auto gain control) circuit, the noise of the image signal is reduced by the CDS circuit, and the level of the image signal is adjusted by adjusting the gain by the AGC circuit.

The dimming circuit 304 controls the light emission amount of the built-in flash 5 during flash photography to a predetermined light emission amount set by the overall control unit 270. During flash photography, the flash light reflected from the subject is received by the dimming sensor 305 simultaneously with the start of exposure, and when the amount of received light reaches a predetermined amount of light emission, the dimming circuit 304 outputs a light emission stop signal. The light emission stop signal is guided to the flash control circuit 217 via the overall control unit 270 provided in the camera body unit 2, and the flash control circuit 217.
In response to the light emission stop signal, the light emission of the built-in flash 5 is forcibly stopped, whereby the light emission amount of the built-in flash 5 is controlled to a predetermined light emission amount.

In the camera body 2, the A / D converter 205 converts (A / D-converts) the signal of each pixel of the image into, for example, a 10-bit digital signal. A /
The D converter 205 converts each pixel signal (analog signal) into a digital signal based on the A / D conversion reference clock input from the timing control circuit 202.

The timing control circuit 202 includes a reference clock, a timing generator 314, and an A / D converter 20.
It is configured to generate a clock for 5.
The timing control circuit 202 is controlled by the overall control unit 270.

The digital signal converted by the A / D converter 205 is sent to the first image processing section 240 and the overall control section 2.
70 respectively. The digital signal input to the first image processing unit 240 is subjected to various image processing in the first image processing unit 240, stored in the memory card 91 as a captured image, or used as a live view display image. On the other hand, the digital signal input to the overall control unit 270 is used by the overall control unit 270 to calculate the brightness, contrast, color balance, etc. of the subject.

The first image processing section 240 is realized as, for example, a one-chip IC (Integrated Circuit),
A plurality of types of image processing such as black level correction processing, white balance correction processing, gamma correction processing, and pixel interpolation processing are performed on the image converted into a digital signal by the A / D converter 205. The processing contents of these image processings are set for each image by the overall control unit 270. For example, regarding the white balance correction processing, the white balance adjustment value set for each image is input from the overall control unit 270. Each pixel data of the image is stored in the first image processing unit 240.
Pixel interpolation processing is performed to interpolate missing color pixels
It will have three color data.

The image memory 209 includes a first image processing section 24.
This is a memory that stores image data output from 0. The image memory 209 has a capacity capable of storing at least two frames of images.

The VRAM (video RAM) 210 is an LC
It is a buffer memory of an image reproduced and displayed in D10.
Further, the VRAM 211 is a buffer memory for an image displayed on the EVF 31. The VRAM 210 is the LCD 10
Image data corresponding to the number of pixels of
It has a storage capacity capable of storing image data corresponding to the number of pixels of the VF 31.

In the shooting standby state when the operation mode is "shooting", the image pickup unit 3 shoots an image, for example, every 1/30 seconds, and the A / D converter 205 converts the image into a digital signal. Then, after predetermined signal processing is performed by the first image processing unit 240, it is temporarily stored in the image memory 209, and VRA is also transmitted via the overall control unit 270.
M210, transferred to 210, LCD10 and EVF
20 (live view display). This allows the user to visually recognize the subject image.

When the operation mode is "reproduction", the image read from the memory card 91 is the overall control section 270.
After the predetermined signal processing is performed by the LCD VRAM 210
And is reproduced and displayed on the LCD 10. EVF20
However, the same display is performed.

The card interface 212 is an interface for writing and reading images to and from the memory card 91 via the card slot 17.

The flash control circuit 217 is a circuit for controlling the light emission of the built-in flash 5, and the overall control section 270.
While the built-in flash 5 is caused to emit light based on the control signal from, the light emission of the built-in flash 5 is stopped based on the above-mentioned light emission stop signal.

The operation unit 250 includes the above-mentioned various switches and buttons, and information input by a user is input via the operation unit 250 to the overall control unit 2.
70 is transmitted.

The overall control unit 270 is composed of a microcomputer having a CPU, and centrally controls the photographing function and the reproducing function. The overall control unit 270 stores a ROM 261 that stores a control program for controlling the driving of the above-described members of the digital camera 1 and processing the acquired images, and a number of arithmetic operations according to the control program. A RAM 262, which is a work area, is electrically connected. The program data recorded in the memory card 91 which is a recording medium can be read out via the card interface 212 and stored in the ROM 261. That is, the control program is
It can be installed in the digital camera 1 from the memory card 91.

In FIG. 5, an AF control section 271, an exposure control section 272, an image processing control section 273 and an image recording section 2 are shown.
Reference numeral 74 denotes a function realized by being arithmetically processed by the CPU or the like of the overall control unit 270 according to the control program stored in the ROM 261.

The AF control section 271 performs so-called AF (autofocus) control. AF control unit 27
1 evaluates the contrast of the image signal input from the A / D converter 205, and drives the position of the focus lens 311 so that this contrast becomes the highest. As a result, the position of the image formed by the zoom lens 301 is matched with the image pickup surface of the CCD 303.

The exposure control unit 272 sets exposure conditions and controls exposure. Specifically, the brightness of the subject is calculated from the image signal input from the A / D converter 205, and the aperture value and shutter speed (to be exact, CCD
(Charge accumulation time of 303). Further, when the appropriate exposure amount cannot be obtained even by adjusting the aperture value and the shutter speed, the built-in flash 5 is set to emit light. According to the set exposure condition, a control signal is transmitted to the timing control circuit 202, the aperture motor drive circuit 216, and the flash control circuit 217 to perform exposure control.

The image processing control unit 273 has the A / D converter 2
The first image processing unit 240 that performs image processing on the image output from 05 and the second image processing unit 283 that performs image processing on the image stored in the image memory 209 by software (see FIG. 6). ) Related image processing conditions are set. For example, the color balance of the image signal input from the A / D converter 205 is calculated, and the white balance adjustment value is set according to the calculated color balance.

The image recording section 274 is set from the thumbnail image of the image captured in the image memory 209 and the switch included in the operation section 250 when the shutter button 8 instructs the image recording section 274 when the operation mode is "photographing". A compressed image compressed in JPEG format is generated according to the input compression rate, and tag information (frame number, exposure value, shutter speed, compression rate, shooting date, flash on / off data at the time of shooting, shooting Both images are stored in the memory card 91 together with information such as distance and scene information).

<1-3. Operation of Digital Camera> Further, when the shooting mode is the person detection mode, the overall control unit 270 is configured to perform a person detection process for detecting a person area in an image at the time of shooting. FIG. 6 is a block diagram showing the functions of the digital camera 1 relating to the person detection processing. In FIG. 6, the red-eye detection unit 281, the person detection unit 282, the second image processing unit 283, and the distance calculation unit 28.
Reference numeral 4 denotes a function realized by being arithmetically processed by the CPU or the like of the overall control unit 270 according to the control program stored in the ROM 261.

The red-eye detecting section 281 detects the red-eye area in the image stored in the image memory 209. When detecting the red-eye area, a first image (hereinafter, referred to as “non-flash light emission image”) captured without emitting the built-in flash 5 and a second image (hereinafter, referred to as “non-flash emission image”) are emitted. "Flash emission image").

The person detecting section 282 detects a skin color area near the red-eye area detected by the red-eye detecting section 281 as a person area in the image. The second image processing unit 283
Image processing such as correction of the red-eye area in the detected image is realized by software. Further, the distance calculation unit 284 is
The focus position information of the focus lens 311 is acquired from the AF control unit 271 and the distance from the position information to the main subject (person) is calculated as the shooting distance.

FIG. 7 is a block diagram showing the functional arrangement of the red-eye detector 281. The occurrence rate calculation unit 291 calculates the probability of occurrence of the red-eye phenomenon of the subject in the image as the red-eye occurrence rate. The candidate area extraction unit 293 uses the pixel values of the non-flash image and the flash image, respectively, and extracts the red area as an area that is a candidate for a red eye area (hereinafter, referred to as a “red eye candidate area”). Judgment standard changing unit 292
Changes the criterion for the candidate area extraction unit 293 to extract the red-eye candidate area according to the red-eye occurrence rate calculated by the occurrence rate calculation unit 291. Further, the candidate area comparison unit 294 compares the red-eye candidate area between the non-flash image and the flash image, and detects the red-eye candidate area only in the flash image as the red-eye area. Details of these functions will be described later.

FIG. 8 and FIG. 9 are diagrams showing the flow of the operation of the digital camera 1 during shooting in the person detection mode. Hereinafter, the digital camera 1 will be described with reference to FIGS. 6 to 9.
The shooting operation in the person detection mode will be described.

First, when the shutter button 8 is half-pressed (S1 state) (step ST11), preparation for image capturing is made. That is, the focus lens 311 is driven by the AF control unit 271, and the position of the image formed by the zoom lens 301 is matched with the image pickup surface of the CCD 303. (Step ST12).

Next, the exposure control unit 272 sets the exposure conditions based on the brightness of the subject. That is, the aperture value,
The shutter speed and whether or not the built-in flash 5 emits light (emission setting) (non-emission setting) are set (step ST13). Furthermore, the image processing control unit 273
However, the white balance adjustment value is set based on the color balance of the subject (step ST14).

When the preparation for photographing is completed as described above, the release permission is given. That is, the shutter button 8 is unlocked, and the shutter button 8 can be fully pressed (S2 state) (step ST15). If the shutter button 8 is not operated for a predetermined time in this state (step S
If No in T16, the process returns to step ST11. On the other hand, if the shutter button 8 is fully pressed (Yes in step ST16), the process proceeds to step ST21 in FIG.

When the light emission of the built-in flash 5 is set (Yes in step ST21), subsequently, the built-in flash 5 is not emitted and the CCD 303 is exposed for the set exposure time to capture the image of the subject as a non-flash image. Be done. After the exposure, the image signal output from the CCD 303 is subjected to predetermined processing in the signal processing circuit 313, the A / D converter 205 and the first image processing unit 240, and stored in the image memory 209. This non-flash image is
Since the image is captured without the built-in flash 5 emitting light, the image has relatively low brightness (step ST2).
2).

Next, the built-in flash 5 is emitted (step ST23), the CCD 303 is exposed for a set exposure time, and the image of the same subject as the non-flash image is captured as a flash image. After exposure, C
The image signal output from the CD 303 is the signal processing circuit 3
13, predetermined processing is performed in the A / D converter 205 and the first image processing unit 240 and similarly stored in the image memory 209. This flash image shows the built-in flash 5.
Since it was photographed by emitting light, the image has a relatively high brightness (appropriate exposure) (step ST24).

FIG. 10 is a diagram showing an example of a non-flash image, and FIG. 11 is a diagram showing an example of a flash image for the same subject as in FIG. As shown in these figures, the non-flash image P1 becomes a darker image than the flash image P2.

The images P1 and P2 include two persons PS1 and PS2 as subjects and a building PS3 having a color close to the skin color. Also, the person PS1 wears a red accessory R1 and the person PS2's clothes have a red button R2. A red illumination light source R3 exists in the building PS3. The eye area E of the persons PS1 and PS2 in the flash image of FIG.
1 and E2 have a red-eye effect.

Next, the red-eye detecting section 281 performs the red-eye detecting process for detecting the red-eye area in the flash image by using the two images stored in the image memory 209 as described above (step ST25). .

FIG. 12 is a diagram showing the flow of the red-eye detection processing of the red-eye detection section 281. First, the occurrence rate calculation unit 291
Calculates the red-eye occurrence rate P (step ST101).
The red-eye occurrence rate P can be calculated by the following equation 1 using the shooting distance D, the brightness difference Brd between the flash image and the non-flash image, and the predetermined constant A.

[0063]

[Equation 1]

Here, the photographing distance D is calculated by the distance calculation unit 284. The brightness difference Brd is the difference between the brightness Br2 of the flash image and the brightness Br1 of the non-flash image, and the brightness Br1 and Br2 use the average value of the RGB values of the pixel data in the image. Note that the brightness Br1, Br
For 2, the average value of only the G value, or the average value of the Y values converted into YCrCb may be used. Furthermore, the brightness difference Br
The value of d may be calculated based on the light emission amount of the built-in flash 5.

Next, as preprocessing for extracting the red-eye candidate areas of the flash image and the non-flash image, the judgment criterion changing unit 292 changes and sets the judgment criterion for judging the red-eye candidate area as the red-eye candidate area. (Step ST1
02).

In this embodiment, since there is a difference in brightness between the flash image and the non-flash image, the chromaticity u ', which is not affected by the brightness from the RGB value of the pixel data,
v ′ is calculated, and the chromaticity u ′ and v ′ are “CIE 1976 U”.
The red-eye candidate area is determined by whether or not it is within the specific area of the "CS chromaticity diagram". The chromaticity u ′ and v ′ can be calculated by the following equations 2 and 3.

[0067]

[Equation 2]

[0068]

[Equation 3]

FIG. 13 is a simplified diagram of "CIE 1976 UCS Chromaticity Diagram". In FIG. 13, a point indicated by reference symbol RP indicates a red reference point which is set in advance by measurement or the like and which is an ideal red color at the time of the red eye phenomenon. Also, from the red reference point RP in the plus and minus u'directions, respectively, L
A rectangular area having a width of Lv in the u, plus, and minus v ′ directions is shown as a red determination reference area RA. If the chromaticities u ′ and v ′ of the pixel data are within the red determination reference area RA, it is determined that the pixel data is a red eye candidate area. That is, if the size (area) of the red determination reference area RA is changed, the determination reference for extracting the red eye candidate area will be changed.

Here, the area of the red judgment reference area RA is L
It can be calculated by multiplying twice u and twice Lv. And
The widths Lu and Lv of the red determination reference area RA are set as in the following Expressions 4 and 5 based on the preset initial values Lu0 and Lv0 and the red-eye occurrence rate P.

[0071]

[Equation 4]

[0072]

[Equation 5]

As a result, the size of the red judgment reference area RA can be set smaller as the red-eye occurrence rate P is higher, and the size of the red judgment reference area RA can be set larger as the red-eye occurrence rate P is lower. That is, since the size of the red determination reference area RA is changed according to the red-eye occurrence rate P, only an area having a high probability as a red-eye area can be extracted as a red-eye candidate area.

Subsequently, the candidate area extracting unit 293 determines whether or not the chromaticity u ', V'of each pixel data is included in the red determination reference area RA as described above, and the candidate area in the flash image A red eye candidate area is extracted (step ST103).
At this time, if no red-eye candidate area is extracted from the flash image (No in step ST104), it is determined that there is no red-eye area (step ST109).

On the other hand, if at least one red-eye candidate area is extracted from the flash image (step ST104).
If Yes, the red-eye candidate area in the non-flash image is similarly extracted (step ST105).

FIG. 14 is a diagram showing the extracted red-eye candidate regions in the non-flash image P1 in FIG. 10, and FIG. 15 is a diagram showing the extracted red-eye candidate regions in the flash image P2 in FIG. In the non-flash image P1, as shown in FIG. 14, the accessory R1, the button R2, and the illumination light source R3 are extracted as the red eye candidate area.
On the other hand, in the flash image P2, as shown in FIG. 15, the accessory R1, the button R2, and the illumination light source R
In addition to 3, the eye areas E1 and E2 of the persons PS1 and PS2 are extracted as red eye candidate areas. That is, the red-eye area is extracted as the red-eye candidate area only in the flash image P2.

Next, the candidate area comparison unit 294 compares the red-eye candidate areas extracted in each of the non-flash image and the flash image (step ST10).
6). When there is a red-eye candidate area that exists only in the flash image (Yes in step ST107)
Detects the red-eye candidate area in the flash image as a red-eye area (step ST108). This is because the red-eye phenomenon occurs only when the built-in flash 5 emits light, and thus the red-eye candidate area only in the flash image can be determined as the red-eye area. As a result, other red eye candidate areas are not detected as red eye areas,
Only the red-eye area can be surely detected.

FIG. 16 shows the non-flash image P1 (see FIG.
(See FIG. 15) and the flash image P2 (see FIG. 15) are compared with each other to indicate an area detected as a red eye area. As shown in the figure, only the human eye areas E1 and E2 are detected as red eye areas.

If there is no red-eye candidate area that exists only in the flash image (N in step ST107)
o), it is determined that there is no red-eye area (step ST109).

When the red-eye area in the flash image is detected as described above, the person detecting unit 282 next extracts the skin color area in the flash image in order to detect the person area (FIG. 9, step). ST26). The extraction of the skin color area is similar to the extraction of the red eye candidate area, and the chromaticity u ′, v ′ of the pixel data in the flash image is “CIE 1976 UC”.
It is determined whether or not it is included in a preset area for determining the skin color of the person in the “S chromaticity diagram”.

At this time, only the skin color area in the vicinity of the red eye area is extracted by determining the skin color area from the pixel data in the vicinity of the red eye area. Then, the extracted skin color area is detected as a person area. This is because the red-eye area is always included in the person's face area, and thus the skin-colored area near the red-eye area can be determined to be the person area (face area). As a result, a building or the like having a color similar to the skin color is not extracted as the skin color region, and only the skin color region that is the person region can be reliably extracted.
Further, it is not necessary to determine the skin color area for all pixel data in the flash image, and the processing speed can be improved.

FIG. 17 shows a flash image P2 (FIG. 15).
The area detected as the person area in FIG.
As shown in the figure, only the skin color areas near the red-eye areas E1 and E2 are detected as the person area PA, and the building PS3 having a color close to the skin color is not detected as the person area PA.

In the red eye detection process, if it is determined that there is no red eye area (FIG. 12, step ST109), it is considered that there is no person in the image, and the person area is not detected.

Next, the image processing control unit 273 sets the image processing conditions for the second image processing unit 283 based on the information on the detected person area and red eye area. And the second
The image processing unit 283 performs image processing on the flash image based on the set conditions. As a result, image processing based on the person area and the red eye area can be performed on the flash image (step ST27).

For example, since the person area is specified,
There is a difference between the human and non-human areas, such as the process of optimally adjusting the brightness and saturation of the human area (face area) only, or the process of blurring the non-human areas to make the person stand out beautifully. Image processing can be performed with different processing contents. Further, since the red-eye area is specified, it is possible to easily perform the red-eye correction processing for replacing the red-eye area with a color such as gray. Furthermore, it is possible to determine a shooting scene such as “landscape” or “portrait” from the number of person areas and perform image processing according to the shooting scene.

A compressed image and a thumbnail image of the flash image subjected to the image processing are generated by the image recording unit 274 and stored in the memory card 91 together with the tag information (step ST28).

By the way, when the built-in flash 5 is set to emit no light (No in step ST21), the built-in flash 5
Since an image with proper exposure can be obtained without emitting light, the image of the subject is captured without using the built-in flash 5, is subjected to predetermined processing, and is stored in the image memory 209 (step ST31). Since the red-eye phenomenon does not occur in the image captured at this time, the red-eye detection processing is not performed, the skin-color area is extracted regardless of the red-eye area, and the skin-color area is detected as a human area (step ST3).
2). Then, similar to step ST27, image processing based on the detected person area is performed (step ST3).
3). Since the red-eye phenomenon does not occur, the red-eye correction process is not performed. When the image processing is performed, it is stored in the memory card 91 in the same manner as described above (step ST2).
8).

In the present embodiment, the built-in flash 5 is automatically set to emit light or not to emit light based on the brightness of the subject, but the user may be able to set these. .

Although the first embodiment has been described above, in the present embodiment, the red-eye phenomenon occurs only when the flash is emitted by using the non-flash image and the flash image. Therefore, the red-eye candidate area only in the flash image is detected as the red-eye area, so that the red-eye area can be surely detected.

Further, since the skin color area in the vicinity of the detected red-eye area is detected as the human area, only the human area can be surely detected. Further, since it is not necessary to perform complicated processing such as shape determination of the skin color area, the human area can be detected relatively quickly.

Further, since the image processing condition is set based on the person area, the image processing is performed with different processing contents for the person area and the area other than the person area, and the image processing optimal for the person area is performed. It can be performed. Further, since the red-eye area is specified, the red-eye correction process can be easily performed.

<2. Second Embodiment> Next, a second embodiment of the present invention will be described. In the first embodiment, the human area is detected and the image processing condition is set based on the human area. However, in the present embodiment, the imaging condition is further set based on the human area. It has become. The digital camera 1 which is the image processing apparatus according to the present embodiment has the same configuration as that shown in FIGS. 1 to 5.

FIG. 18 is a block diagram showing the functions of the digital camera 1 relating to the person detection processing in this embodiment. The functional configuration of the red-eye detecting unit 281 is similar to that shown in FIG. Components having the same functions as those in the first embodiment are designated by the same reference numerals, and detailed description thereof will be omitted. In FIG. 18, the red eye occurrence determination unit 28
Reference numeral 5 denotes a function realized by being arithmetically processed by the CPU or the like of the overall control unit 270 according to the control program stored in the ROM 261. Red eye occurrence determination unit 28
Reference numeral 5 determines whether or not the red-eye phenomenon occurs based on the brightness of the subject and the shooting distance before the shooting operation. Red eye occurrence determination unit 28
Details of the function of 5 will be described later.

19 to 21 are diagrams showing the flow of the operation of the digital camera 1 at the time of photographing in the person detection mode of the present embodiment. Hereinafter, the photographing operation of the digital camera 1 in the present embodiment in the human detection mode will be described with reference to FIGS. 18 to 21.

First, when the shutter button 8 is half-pressed (S1 state) (step ST51), the red-eye occurrence determination unit 2
85 calculates the subject brightness Br from the image signal input from the A / D converter 205, and emits the built-in flash 5 based on the calculated subject brightness Br and the shooting distance D input from the distance calculation unit 284. If so, a red-eye occurrence determination process is performed to determine whether the red-eye effect occurs (step ST52).

FIG. 22 is a diagram showing a judgment function used in the red eye occurrence judgment processing. In the figure, the horizontal axis represents subject brightness Br and the vertical axis represents shooting distance D. An area indicated by a symbol FA is a red-eye generating area in which the red-eye phenomenon occurs when the built-in flash 5 emits light, and an area other than the red-eye generating area FA is an area in which the red-eye phenomenon does not occur even when the built-in flash 5 emits light. This red-eye generation area FA is preset by measurement or the like. The red-eye occurrence determination unit 285 determines that the red-eye phenomenon will occur if the point determined by the subject brightness Br and the shooting distance D is within the red-eye occurrence area FA.

If it is determined that the red eye phenomenon occurs (Yes in step ST52), step ST53.
Go to. The processing in steps ST53 to ST57 is
This is the same as the processing in steps ST22 to ST26 in FIG. That is, the non-flash image and the flash image are captured (steps ST53 to ST55), the red-eye region is detected by the red-eye detection process using both images (step ST56), and the skin color region near the red-eye region is extracted. The skin color area is detected as a human area (step ST5).
7).

On the other hand, when it is determined that the red-eye effect does not occur (No in step ST52), step S
Proceed to T58. The processes in steps ST58 and ST59 are the same as steps ST31 and ST32 in FIG. That is, the image of the subject is captured without using the built-in flash 5 (step ST31), the skin color region is extracted without performing the red-eye detection process, and the skin color region is detected as a human region (step ST59).

Next, the person detecting section 282 sets a control area surrounding the person area detected as described above (FIG. 20, step ST61). FIG. 23 is a diagram showing an example of the set control area CA. As shown in the figure, the control area CA is set to a rectangular area having a minimum area surrounding the detected person area PA.

The control area CA is used as an area for calculating the brightness, contrast, color balance, etc. of the subject in the subsequent processing. That is, based on the control area CA including the person area, the imaging condition including the exposure condition and the image processing condition such as the white balance adjustment value are set.

When the control area CA is set, the area CA1 corresponding to the control area CA is displayed on the EVF 31 as shown in FIG. 24 (step ST62). The area CA1 is shown in FIG.
As shown in, the subject image displayed in live view is displayed on the same screen. This allows the user to check the area used for setting the imaging condition and the image processing condition, and gives the user a sense of security.
It should be noted that the LCD 10 may also display the same as the EVF 31.

Next, the AF control section 271 performs AF control based on the control area CA to drive the focus lens 311 (step ST63). Specifically, the contrast of only the area corresponding to the control area CA in the image signal input from the A / D converter 205 is evaluated, and the focus lens 311 is set so that this contrast becomes the highest.
Drive the position of. This makes it possible to bring the person area, which is the main subject, into the most focused state.

Next, the exposure control unit 272 sets the exposure conditions based on the control area CA. Specifically, of the image signals input from the A / D converter 205, the control area CA
Based on the brightness of only the area corresponding to, the aperture value, shutter speed, and whether or not the built-in flash 5 emits light (emission setting) (non-emission setting) are set (step ST64).

Next, the image processing controller 273 controls the control area C.
The white balance adjustment value is set based on A. Specifically, the white balance adjustment value is set based on the color balance of only the area corresponding to the control area CA in the image signal input from the A / D converter 205 (step ST65).

When the setting of the image pickup condition and the image processing condition based on the control area CA is completed as described above, the release is permitted (step ST66). If the shutter button 8 is not operated for a predetermined time in this state (No in step ST67), the process returns to step ST11 in FIG.

On the other hand, if the shutter button 8 is fully pressed (Yes in step ST67), the process proceeds to step ST71 in FIG. Then, when the built-in flash 5 is set to emit light (Yes in step ST71), the built-in flash 5 emits light (step ST72) and a captured image is captured (step ST73). On the other hand, built-in flash 5
When the non-light emission is set (No in step ST71), the captured image is captured without the built-in flash 5 emitting light (step ST73). Since the exposure condition at the time of capturing the captured image is set based on the control area CA, it is possible to obtain the proper exposure for the person area which is the main subject.

Even when the built-in flash 5 emits light, the red-eye phenomenon does not occur in the subject in the photographed image. If it is determined that red eye phenomenon will occur,
This is because the built-in flash 5 is emitting light in step ST54 of FIG. 19 and this light emission functions as pre-light emission for preventing red-eye effect.

Next, the captured image taken in is subjected to predetermined processing in the signal processing circuit 313 and the A / D converter 205, and further subjected to image processing including white balance correction processing in the first image processing section 240. (Step ST74). At this time, since the white balance adjustment value is set based on the control area CA, image processing is performed so that the human area as the main subject has an appropriate white balance.

The photographed image subjected to the image processing by the first image processing section 240 is stored in the image memory 209.
At this time, similarly to the first embodiment, the image memory 20
It is also possible for the second image processing unit 283 to perform image processing for the captured image stored in 9 such that the processing content differs between the person area and the area other than the person area.

A compressed image and a thumbnail image of the photographed image subjected to the image processing are generated by the image recording section 274 and stored in the memory card 91 together with the tag information (step ST75).

Although the second embodiment has been described above, in the present embodiment, the imaging condition and the image processing condition are set based on the control area CA including the detected person area. It is possible to obtain an image shot with the optimal control for the person area that is the main subject.

<3. Third Embodiment> Next, a third embodiment of the present invention will be described. In the first embodiment, the detection of the red-eye area and the detection of the human area are performed inside the digital camera 1. However, processing other than such image acquisition may be performed by a computer.

FIG. 25 is a diagram showing the configuration of the image processing system 7 in such a case. The image processing system 7 includes a digital camera 71 that acquires a non-flash image and a flash image, and a computer 72 that processes an image obtained by the digital camera 71.

As the digital camera 71, the same structure as the digital camera 1 of the above-mentioned embodiment can be used. The digital camera 71 includes steps ST11 to ST16 and ST21 to S16 of the processing shown in FIGS. 8 and 9.
The processing of T24 and ST28 is performed. That is, a non-flash image and a flash image are taken, red-eye detection processing,
Both images are directly stored in the memory card without performing the person detection processing and the image processing based on the person area. The tag information generated when the memory card is stored should include at least information indicating whether the image is a non-flash image or a flash image, and information on the shooting distance.

The non-flash image and the flash image acquired by the digital camera 71 are transferred to the computer 72 via the memory card. The data may be transferred to the computer 72 via a transmission cable or the like.

The computer 72 includes a CPU, ROM and R
It is composed of a general-purpose computer including an AM, a storage device, a display, a memory card reader, and the like. A program is installed in the computer 72 in advance via a recording medium 73 such as a magnetic disk, an optical disk, or a magneto-optical disk. As a result, the general-purpose computer 72 can be used as an image processing device that performs the processing according to the present invention.

That is, the CPU or the like of the computer 72 operates as the red-eye detection unit 281, the person detection unit 282, the image processing control unit 273, and the second image processing unit 283 shown in FIG. A red eye area in the flash image is detected, and a person area is detected based on the red eye area. Further, based on the detected person area, the second image processing unit 283 performs image processing such that the processing contents are different between the person area and the area other than the person area.

As described above, the red-eye detection processing, the person detection processing, and the image processing based on the person area can be performed by the computer 72. In this case, the processing load on the digital camera 71 is greatly reduced. It

<4. Modifications> Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments.

For example, some or all of the functions realized by the software processing by the overall controller 270 of the digital camera 1 of the first and second embodiments may be realized by a dedicated circuit. . Conversely, some or all of the functions implemented by the dedicated circuit such as the first image processing unit 240 of the digital camera 1 may be implemented by software processing by the overall control unit 270.

Although the red-eye candidate area is determined by using the chromaticities u ′ and v ′ of the pixel data, the chromaticity such as the chromaticity in the XYZ color system, the hue and the saturation in the HLS color system, and the like. Values other than u ′ and v ′ may be used for the determination of the red eye candidate area. Even in such a case, if the determination standard in the color system to be used is set in advance and the determination standard is changed according to the red-eye occurrence rate P, only the region with a high probability as the red-eye region is red-eye. It can be extracted as a candidate area.

The shape of the red determination reference area RA has been described as a rectangular area, but the shape is not limited to this. For example, the shape of the red reference point RP may be Lu in the plus and minus u'directions, respectively. It may be an elliptical shape or the like having a width of Lv in each of the plus and minus v ′ directions. Further, it is not necessary to have a width in both u ′ and v ′ directions, and a straight line having a width in only one of them may be used.

In the second embodiment, the rectangular control area CA surrounding the person area is set, but the person area may be used as it is as the control area. In this way, by setting the person area as the control area as it is, it is possible to acquire an image captured by optimal control according to the person area.

[0124]

As described above, according to the first to sixth aspects of the present invention, the first image taken without the flash and the second image taken with the flash are used. This improves the detection probability of the red-eye area in the second image. In addition, since the human area is detected after the red-eye area is detected, only the human area can be detected, and the processing speed can be improved without requiring complicated processing such as shape determination.

According to the second aspect of the present invention, the red-eye phenomenon occurs only when the flash is fired. Therefore, the red-eye candidate area in only the second image when the flash is fired is detected as the red-eye area. Thus, the red-eye area can be detected easily and reliably.

Further, according to the third aspect of the present invention, since the determination criterion is changed according to the red eye occurrence rate, only a region having a high probability as a red eye region can be detected as a red eye candidate region.

According to the fourth aspect of the present invention, the image processing condition is set based on the person area, so that the image processing is performed with different processing contents for the person area and the area other than the person area. Alternatively, it is possible to perform optimal image processing for the person area. Further, since the imaging condition can be set according to the person area, it is possible to perform the optimum shooting according to the person area.

[Brief description of drawings]

FIG. 1 is a front view of a digital camera that is an image processing apparatus according to the present invention.

FIG. 2 is a rear view of the digital camera.

FIG. 3 is a bottom view of the digital camera.

FIG. 4 is a diagram showing an outline of arrangement of respective components in an imaging unit.

FIG. 5 is a block diagram showing a configuration of a digital camera.

FIG. 6 is a block diagram showing functions of a digital camera relating to person detection processing according to the first embodiment.

FIG. 7 is a block diagram showing a functional configuration of a red-eye detection unit.

FIG. 8 is a diagram showing a flow of operations at the time of shooting by the digital camera in the human detection mode in the first embodiment.

FIG. 9 is a diagram showing a flow of operations at the time of shooting by the digital camera in the human detection mode according to the first embodiment.

FIG. 10 is a diagram showing an example of a non-flash image.

FIG. 11 is a diagram showing an example of a flash image.

FIG. 12 is a diagram showing a flow of red-eye detection processing of a red-eye detection unit.

FIG. 13 is a simplified diagram of a CIE 1976 UCS chromaticity diagram.

FIG. 14 is a diagram showing extracted red-eye candidate regions in a non-flash image.

FIG. 15 is a diagram showing extracted red-eye candidate regions in a flash image.

FIG. 16 is a diagram showing a region detected as a red-eye region in a flash image.

FIG. 17 is a diagram showing an area detected as a person area in a flash image.

FIG. 18 is a block diagram showing functions of a digital camera relating to person detection processing according to the second embodiment.

FIG. 19 is a diagram showing a flow of operations at the time of shooting by the digital camera in the human detection mode according to the second embodiment.

FIG. 20 is a diagram showing a flow of operations at the time of shooting by the digital camera in the person detection mode according to the second embodiment.

FIG. 21 is a diagram showing a flow of operations at the time of shooting by the digital camera in the human detection mode according to the second embodiment.

FIG. 22 is a diagram showing a determination function used in red-eye occurrence determination processing.

FIG. 23 is a diagram showing an example of a control area.

FIG. 24 is a diagram showing an example of EVF display.

FIG. 25 is a diagram showing a configuration of an image processing system.

[Explanation of symbols]

1 digital camera 5 built-in flash 7 Image processing system 91 memory card 270 Overall control unit 281 Red-eye detector 282 person detection unit 301 zoom lens 302 aperture 304 dimming circuit 322 Person detection mode switching button

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 9/64 H04N 1/46 ZF term (reference) 5B057 AA01 CA01 CA08 CA12 CB01 CB08 CB12 CC03 CE16 CH01 CH11 DA08 DB02 DB06 DB09 DC16 DC25 5C022 AA13 AC42 AC69 5C066 AA01 GA01 GA02 GB03 5C079 HB01 JA17 JA25 LA02 MA01 MA11 NA11 5L096 AA02 AA06 BA20 CA02 DA01 FA15 GA41 HA08 LA05 MA03

Claims (6)

[Claims] 1. An image processing apparatus, wherein the same subject is photographed using a first image photographed without a flash and a second image photographed with a flash. Image processing, comprising: red-eye area detecting means for detecting a red-eye area in the two images; and person area detecting means for detecting a skin color area near the red-eye area in the second image as a human area. apparatus. 2. The image processing apparatus according to claim 1, wherein the red-eye area detecting unit extracts a candidate red-eye area that is a candidate for the red-eye area of each of the first image and the second image. Means for comparing the red-eye candidate area of the first image with a red-eye candidate area of the second image, and a candidate-area comparing means for setting the red-eye candidate area only in the second image as the red-eye area. An image processing device characterized by: 3. The image processing apparatus according to claim 2, wherein the red-eye area detecting means includes an occurrence rate calculating means for calculating a red-eye occurrence rate at which the red-eye phenomenon of the subject occurs, and the candidate area extracting means. An image processing apparatus, further comprising: a determination criterion changing unit that changes a determination criterion for extracting a red eye candidate region according to the red eye occurrence rate. 4. The image processing apparatus according to claim 1, further comprising: an image pickup unit for picking up the subject, an image processing condition based on the person area detected by the person area detecting unit, and And / or an image processing condition setting unit that sets an image capturing condition of the image capturing unit. 5. A red-eye region in the second image is captured by using a first image captured without flash emission and a second image captured with flash emission for the same subject. An image processing method, comprising: a red-eye area detecting step of detecting; and a human area detecting step of detecting a skin color area in the vicinity of the red-eye area in the second image as a human area. 6. A computer using a first image captured without flash emission and a second image captured with flash emission for the same subject, in the second image A program for functioning as an image processing apparatus, comprising: a red-eye area detecting unit that detects a red-eye area; and a human-area detecting unit that detects a skin-colored area near the red-eye area in the second image as a human area.