JP2018010498A - Image processing device and method and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To apply lighting by a virtual light source applied to the face of a first image to the face of a second image on the same condition, with a simple operation.SOLUTION: A CPU 101 detects the face from the first image of an HDD 105 and performs the lighting based on the virtual light source. A user can adjust the position of the virtual light source or the like by using an input device 106. The CPU 101 copies a relative position with respect to the face as the target of the virtual light source, according to the copy operation of the user. The CPU 101 detects the face from the second image, arranges the virtual light source in the same relative position with respect to the face of the second image on the basis of the relative position copied by the copy operation, according to the paste operation of the user, and performs the lighting to the face of the second image.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an image processing apparatus and method for performing brightness correction, and a control program.

  It is known to correct the brightness of a dark part of a subject by applying a lighting effect by a virtual light source to the subject in the image (see, for example, Patent Document 1).

JP 2006-072692 A

  With the technique described in Patent Document 1, it is difficult to give the same lighting effect to the same subject in a plurality of images. Specifically, when the user sets a virtual light source at a desired position on the face of a person included in the first image, the same lighting effect by the virtual light source is applied to the face included in the other image. It is difficult.

  It is an object of the present invention to present an image processing apparatus and method, and a control program that can apply a virtual light source set to a subject of an image to a subject of another image with the same setting.

  An image processing apparatus according to the present invention includes a face detection unit that detects a face from a first image and a second image, a lighting unit that performs lighting with a virtual light source on the face detected by the face detection unit, and the first The virtual light source parameter applied to the first image by the lighting means is applied to the second image face at the same relative position as the relative position of the virtual light source with respect to the face of the first image. And control means applied to the image.

  According to the present invention, lighting under the same conditions as lighting applied to the face included in the first image can be applied to the face included in the second image with a simple operation.

It is a schematic block diagram of one Example of this invention. It is an example of the user interface of a present Example. It is a flowchart which shows the writing process of a present Example. It is a top view which shows the relationship between the normal vector in the case of a point light source, and a virtual light source irradiation direction. It is a top view which shows the relationship between the normal vector in the case of a surface light source, and a virtual light source irradiation direction. It is a flowchart of a virtual light source parameter copy process of the present embodiment. It is a flowchart of the paste process of the virtual light source parameter of a present Example. It is an example of the content of the table which stores a face detection result. It is a flowchart of a writing process when there are a plurality of faces in the image data of the paste destination.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the present specification, image processing that performs virtual lighting by setting a virtual light source is hereinafter referred to as relighting. The angle of the virtual light source in the horizontal direction with respect to the photographing optical axis of the imaging device is denoted as light_h, and the angle with respect to the vertical axis that intersects the photographing optical axis at right angles is denoted as light_v. The horizontal coordinate value of the position of the virtual light source is denoted as light_x, the vertical coordinate value is denoted as light_y, and the depth coordinate value is denoted as light_z. The light on / off information, light source type, light intensity, light source position, angle, and light source color temperature related to the virtual light source are collectively referred to as virtual light source parameters hereinafter.

  FIG. 1 shows a schematic block diagram of an embodiment of an image processing apparatus according to the present invention. The main functions of the image processing apparatus 100 shown in FIG. 1 are realized by a computer program that runs on a computer.

  A CPU 101 controls the entire image processing apparatus 100. Reference numeral 102 denotes a ROM that stores an operation processing procedure of the CPU 101 (for example, a program such as a computer startup process and a basic input / output process). Reference numeral 103 denotes a RAM that functions as a main memory of the CPU 101. Various programs including a control program for realizing processing to be described later are loaded into the RAM 103 from the hard disk drive 105 or the like and executed by the CPU 101. The RAM 103 provides a work area when the CPU 101 executes various processes.

  Reference numeral 104 denotes a display which performs various displays under the control of the CPU 101. A hard disk drive (hereinafter referred to as HDD) 105 is used for storing application programs, data, libraries, and the like. Image data that is a target of the reline tool is stored in the HDD 105.

  An input device 106 includes a pointing device and a keyboard. Reference numeral 107 denotes a storage medium mounting unit (media drive) to which the storage medium can be attached and detached. The storage medium is, for example, a memory card that stores a photographed image that can be attached to and detached from a digital still camera.

  A network interface 108 is connected to the computer network 111 via the communication line 110. The CPU 101 transmits / receives data to / from devices on the computer network 111 via the network interface 108.

  Reference numeral 109 denotes a system bus that connects the above-described units, and includes an address bus, a data bus, and a control bus.

  FIG. 2 is an example of a user interface of the image processing apparatus 100. A method of setting the virtual light source parameter by the user will be described with reference to FIG.

  Reference numeral 201 denotes a folder selection area where the user selects a folder with image data. The CPU 101 reads the folder structure of the HDD 105 and displays a folder tree.

  Reference numeral 202 denotes a folder focus frame indicating a folder selected by the user. When the user clicks a desired folder with the pointing device or the like of the input device 106, the CPU 101 draws the folder focus frame 202 at the position of the folder. Thereafter, the CPU 101 reads image data in the folder from the HDD 105.

  A preview area 203 displays the image processing result. A thumbnail area 204 displays a list of image data in a folder specified by the user in the folder selection area 201 as a reduced image. Reference numeral 205 denotes a thumbnail area scroll bar for changing the display range when image data in the folder selected by the user cannot be displayed as a reduced image in the thumbnail area 204. When the user operates the thumbnail area scroll bar 205, the CPU 101 updates the display of the thumbnail area 204 to change the display area of the thumbnail area 204.

  Reference numeral 206 denotes an image focus frame indicating image data selected as an image processing target by the user. When the user clicks a desired reduced image in the thumbnail area 204 with the pointing device of the input device 106, the CPU 101 determines that the image data to be processed is designated by the user. Then, the CPU 101 draws the image focus frame 206 so as to surround the reduced image at the position where the user clicked, reads image data corresponding to the reduced image, and displays it in the preview area 203.

  Reference numeral 207 denotes a display result example of image data selected by the user as an image processing target. Reference numeral 208 denotes an example in which a virtual light source set by the user is displayed superimposed on the image data.

  Reference numeral 209 denotes a light source on / off designation radio button, which is used to instruct the CPU 101 whether or not the virtual light source is lit in a pseudo manner. When the user presses ON, the CPU 101 determines that the operation of turning on the virtual light source is performed by the user, and performs relighting described later on the assumption that the virtual light source is turned on. When the user presses OFF, the CPU 101 determines that the user has performed an operation for turning off the virtual light source, and does not perform relighting assuming that the virtual light source is turned off. After rewriting, the effect of rewriting is deleted.

  210 is a light source type selection list box. When the user presses the down arrow button of the light source type selection list box 210, point light sources and surface light sources are listed as the light source types. When the user selects any of them, the CPU 101 determines that there is an operation for designating the type of the virtual light source from the user, and performs relighting described later. Reference numeral 211 denotes a light intensity slider. When the user operates the knob of the slider 211, the CPU 101 determines that the user has changed the light intensity of the virtual light source, and performs rewriting described later with the light intensity corresponding to the position of the knob. .

  Reference numeral 212 denotes a horizontal coordinate input edit box for inputting the light source horizontal coordinate value light_x. Specifies the horizontal coordinate value of the virtual light source as a numerical value. The center coordinate of the image is 0, the left coordinate value is a negative value, and the right coordinate value is a positive value. Reference numeral 213 denotes a vertical coordinate input edit box for inputting the light source vertical coordinate value light_y. Specifies the vertical coordinate of the virtual light source as a numerical value. The center coordinate of the image is 0, the upper coordinate value is a negative value, and the lower coordinate value is a positive value. Reference numeral 214 denotes a depth coordinate input edit box for inputting the light source depth coordinate value light_z. Specifies the depth coordinate value of the virtual light source as a numerical value. Assume that the position of the imaging device in the virtual space is 0, the coordinate value in the direction approaching the subject from the imaging device is a positive value, and the coordinate value in the direction away from the imaging device is a negative value. The coordinate values specified in the edit boxes 212, 213, and 214 are collectively referred to as virtual light source absolute coordinate values hereinafter.

  Reference numeral 215 denotes a horizontal angle input edit box for inputting a horizontal angle light_h. The horizontal angle light_h with respect to the imaging optical axis of the imaging device of the virtual light source is designated. It is 0 when parallel to the photographic optical axis of the imaging apparatus, positive value when it is directed rightward from the photographic optical axis, and negative value when it is directed leftward. Reference numeral 216 denotes a vertical angle input edit box for inputting a vertical angle light_v. An angle (vertical angle) light_v with respect to a vertical axis that intersects the imaging optical axis at a right angle is designated. It is 0 when parallel to the imaging optical axis of the imaging device, positive value when facing upward, and negative value when facing downward.

  The initial value of each edit box 212 to 216 is set to zero. When the user designates a value in each of the edit boxes 212 to 216, the CPU 101 changes the position and angle of the virtual light source in the virtual space according to the input value, assuming that the position change operation of the virtual light source has been performed. Perform rewriting.

  Reference numeral 217 denotes a light source color temperature input edit box, which allows input of values from 2700K to 10000K. The initial value of the edit box 217 is set to a color temperature of 4500 K of the white fluorescent lamp. When the user changes the value of the light source color temperature input edit box 217, the CPU 101 determines that the user has changed the color temperature of the virtual light source, and performs rewriting described later at the specified light source color temperature.

  The CPU 101 stores the virtual light source parameter specified by the user using the elements 209 to 217 in the RAM 103 in association with the image data to be processed. When any of the virtual light source parameters is changed, the CPU 101 updates the corresponding stored value in the RAM 103.

  Reference numeral 218 denotes a virtual light source parameter copy button. When the user presses the virtual light source parameter copy button 218, the CPU 101 determines that there is a virtual light source parameter copy instruction from the user. Then, the CPU 101 converts various setting information designated by the elements 209 to 217 regarding the virtual light source into a relative position by converting the virtual light source parameter into a relative position by a method to be described later, and stores it in the RAM 103.

  Reference numeral 219 denotes a virtual light source parameter paste button. When the user presses the virtual light source parameter paste button 219, the CPU 101 determines that there is a virtual parameter paste instruction from the user. When the virtual light source parameter is stored in the RAM 103, the CPU 101 converts the virtual light source parameter into an absolute coordinate by a method described later, and then performs relighting on the image data to be processed.

  Reference numerals 220 to 224 denote image data of an explanation example before or after processing. For example, assume that it is desired to obtain the same lighting effect on the image data 221 after lighting the image data 220 with a virtual light source. In this case, the user presses the virtual light source parameter copy button 218 with the image data 220 selected, and then selects the image data 221 and presses the virtual light source parameter paste button 219. When the virtual light source parameter paste destination image data includes a plurality of faces like the image data 222 and 223, the same relighting as that of the copy source is applied to each face. The virtual light source parameter copy process and the virtual light source parameter paste process will be described later.

  Reference numeral 225 denotes an end button. When the user presses the end button 225, the CPU 101 determines that there is an end instruction from the user, discards the information stored in the RAM 103, and ends the image processing by the image processing apparatus 100.

  FIG. 3 is an operation flowchart of the present embodiment in which similar lighting is performed on other images based on the virtual light source parameters set in certain image data. The CPU 101 executes the control program stored in the RAM 103, thereby realizing the processing shown in FIG.

  In step S300, the CPU 101 reads image data to be processed and related information associated with the image data. The related information includes distance information expressed as a distance map between the imaging device and the subject at the time of imaging. The distance map is image data obtained by quantifying the physical distance from the image sensor to the subject for each pixel of the subject image when the image data selected as the processing target is captured. The distance map is generated by calculating the subject distance from a plurality of images with different blurring in the imaging device, and is recorded on the storage medium together with the image data as related information of the image data. Instead of the distance map, three-dimensional shape data obtained by photogrammetry with a stereo camera or three-dimensional measurement by spot light irradiation or scanning may be used.

  In step S <b> 301, the CPU 101 reads virtual light source parameters from the RAM 103. The virtual light source parameter includes set values of light on / off information, light source type, light intensity, virtual light source position, angle, and light source color temperature. When the user selects unadjusted image data in the thumbnail area 204, the CPU 101 reads an initial value. When the user has changed the virtual light source parameter in the elements 209 to 217, the CPU 101 reads the changed virtual light source parameter. When the user presses the virtual light source parameter paste button 219, the CPU 101 reads the virtual light source parameter calculated by the paste process.

  In step S302, the CPU 101 detects a human face included in the subject image of the image data to be processed. First, the CPU 101 detects the number of faces and assigns a unique face ID in order from 0 to each detected face. The CPU 101 determines the face center coordinates, face size, face angle, and face orientation in association with the face ID. The center coordinate value of the face is output in the same coordinate system as the absolute coordinates of the virtual light source.

  In step S303, the CPU 101 assigns 0 to the counter i. The counter i is a loop variable for repeating the processing from step S305 to S308 for the number of faces.

  In step S304, the CPU 101 determines whether or not the counter i is smaller than the number of faces. If the counter i is smaller than the number of faces, the CPU 101 proceeds to step S305. If the counter i is a value equal to or greater than the number of faces, the CPU 101 determines that the processes in steps S305 to S308 have been repeated for the number of faces, and proceeds to step S309.

  In step S <b> 305, the CPU 101 calculates a physical distance from the imaging device for the face with the face ID i. In order to calculate the physical distance, the CPU 101 acquires the distance from the distance map to the center coordinates of the face.

  In step S306, the CPU 101 detects the facial organ of the face whose face ID is i. The facial organs are the left and right eyes, nose and mouth. The CPU 101 outputs the coordinates corresponding to the left and right ends and the upper and lower ends of the two-dimensional image data of each facial organ as the facial organ detection result.

  In step S307, the CPU 101 generates facial normal information from the detection result of the facial organs. The CPU 101 determines the direction of the face, that is, the normal of the face, from the result of simulating the unevenness of the face in the virtual space of the subject image. The CPU 101 selects the mask closest to the predetermined mask from the face orientation, and deforms the mask to the front according to the size of the face and the position of the facial organ.

  In step S308, the CPU 101 stores the face normal information calculated in step S307 in the RAM 103 as face normal information having a face ID i.

  In step S309, the CPU 101 reads the face normal information with face IDs 0 to i from the RAM 103, and synthesizes the face normal information in the distance map according to the distance of each face, so that the distance accuracy of the face portion is high. Generate a distance map.

In step S310, the CPU 101 generates output image data based on the virtual light source parameters. Output RGB values (Rout, Gout, Bout) of the processing target irradiated by the virtual light source are obtained by the following formula (1). That is,
Rout = [Rt + A × cos (θ) × (1 / D ^ 2) × Rv] / M
Gout = [Gt + A × cos (θ) × (1 / D ^ 2) × Gv] / M (1)
Bout = [Bt + A × cos (θ) × (1 / D ^ 2) × Bv] / M
Here, (Rt, Gt, Bt) is the input pixel value to be processed, A is the intensity of the virtual light source, D is the distance between the virtual light source and the subject to be relighted, and (Rv, Gv, Bv) is the light source reflection color. Indicates. M represents a preset constant for normalizing the output RGB value after relighting. The angle θ represents an angle formed by the irradiation direction of the virtual light source and the normal vector of the subject of the processing target pixel.

  The input pixel values (Rt, Gt, Bt) to be processed are pixel values constituting the image data read in step S300. The intensity A of the virtual light source is a value designated by the user with the light intensity slider 211. The strength at the time of pasting will be described later.

The CPU 101 calculates the distance D between the virtual light source and the subject to be relighted from the position of each pixel calculated in step S309 and the absolute coordinates of the virtual light source (light_x, light_y, light_z). The distance D is calculated according to the following formula (2). That is,
D = {(tP_x-light_x) ² + (tP_y-light_y) ² + (tP_z-light_z) ² } ^1/2 (2)
Here, (tP_x, tP_y, tP_z) indicates the coordinate position of the target image.

  The light source reflection colors (Rv, Gv, Bv) are determined according to the light source color temperature specified by the user in the light source color temperature input edit box 217. The lower the color temperature value, the more reddish reflection color, and the higher the color temperature, the bluish reflection color. The CPU 101 refers to a known conversion table and converts the light source color temperature to the light source reflected color (Rv, Gv, Bv).

  The CPU 101 identifies the irradiation direction of the virtual light processing target from the absolute position and angle of the virtual light source, and calculates the angle θ from the irradiation direction and the normal vector in the virtual space of each pixel calculated in step S309. Is calculated.

  FIG. 4 is a conceptual diagram showing the relationship between the normal vector and the virtual light source irradiation direction when the virtual light source is a point light source. 401 is a virtual light source. Reference numeral 402 denotes a processing target pixel in the virtual space. Reference numeral 403 denotes a straight line indicating the irradiation direction of light emitted from the virtual light source to the processing target pixel 402. The virtual light source 401, which is a point light source, radiates output light radially. Reference numeral 404 denotes a normal vector in the processing target pixel 402. Reference numeral 405 denotes an angle formed by the normal vector 404 and the irradiation direction 403, which is an incident angle θ on the processing target pixel 402.

  FIG. 5 is a conceptual diagram showing the relationship between the normal vector and the virtual light source irradiation direction when the virtual light source is a surface light source. Reference numeral 501 denotes a virtual light source. The coordinates indicating the position of the virtual light source indicate the central coordinates of the surface light source serving as the virtual light source 501. Reference numeral 502 denotes a processing target pixel in the virtual space. Reference numeral 503 denotes a straight line indicating the irradiation direction of light emitted from the virtual light source to the processing target pixel 502. The virtual light source 501 that is a surface light source irradiates light vertically from its output surface. Reference numeral 504 denotes a normal vector in the processing target pixel 502. Reference numeral 505 denotes an angle formed by the normal vector 504 and the irradiation direction 503, which is an incident angle θ on the processing target pixel 502.

  FIG. 6 is a flowchart of virtual light source parameter copy processing. The CPU 101 implements the processing shown in FIG. 6 by executing a control program stored in the RAM 103. A method of converting the virtual light source position stored in the RAM 103 in absolute coordinates into a relative position with respect to the face in the virtual light source parameter copy process will be described with reference to FIG.

  In step S601, the CPU 101 determines whether or not the user has performed a virtual light source parameter copy operation. When the user presses the virtual light source parameter copy button 218, the CPU 101 determines that a copy operation has been performed, and proceeds to step S602. If the user has not pressed the virtual light source parameter copy button 218, the CPU 101 determines that there is no copy operation, and ends the processing shown in FIG.

  In step S602, the CPU 101 reads the position of the face of the main subject. Hereinafter, the face of the main subject is referred to as the main face. If the subject image has one face, the CPU 101 sets that face as the main face. When there are a plurality of faces in the subject image, the CPU 101 sets the largest face among the faces existing at the in-focus distance as the main face. The position of the main face is stored in advance in the RAM 103 according to the calculation results in step S302 and step S305.

In step S <b> 603, the CPU 101 calculates relative coordinates with respect to the main face from the virtual light source absolute coordinates stored in the RAM 103 and the position of the main face. The relative coordinates (reLight_x, reLight_y, reLight_z) of the virtual light source are calculated by the following formula 3. That is,
reLight_x = sFace_x-sLight_x
reLight_y = sFace_y-sLight_y (3)
reLight_z = sFace_z-sLight_z
However, the virtual light source absolute coordinates in the parameter copy source image are (sLight_x, sLight_y, sLight_z), and the center coordinates of the main face are (sFace_x, sFace_y, sFace_z).

  In step S <b> 604, the CPU 101 stores the copied virtual light source parameter in the RAM 103. At that time, the CPU 101 stores the relative coordinates calculated in step S603 for the position of the virtual light source constituting the virtual light source parameter. For other virtual light source parameters, the CPU 101 stores the values applied to the copy source image data as they are. The copied virtual light source parameter is held in the RAM 103 until the user selects different image data and presses the virtual light source parameter copy button 218 or the image processing apparatus 100 is terminated.

  FIG. 7 is a flowchart showing virtual light source parameter paste processing. The CPU 101 implements the processing shown in FIG. 7 by executing a control program stored in the RAM 103. A method of converting virtual light source relative coordinates to absolute coordinates in the virtual space of the subject image of the paste destination image in the virtual light source parameter paste processing will be described with reference to FIG.

  In step S700, the CPU 101 determines whether or not the user has performed a paste operation of the virtual light source parameter. When the user presses the virtual light source parameter paste button 219, the CPU 101 determines that a paste operation has been performed, and proceeds to step S702. If the user has not pressed the virtual light source parameter paste button 219, the CPU 101 determines that there has been no paste operation, and ends the processing shown in FIG.

  In step S <b> 701, the CPU 101 determines whether there is a virtual light source parameter copied in the RAM 103. If there is, the CPU 101 proceeds to step S702. If not, the CPU 101 proceeds to step S708.

  In step S <b> 702, the CPU 101 reads the virtual light source parameter copied from the RAM 103.

  In step S703, the CPU 101 acquires the position and distance of the main face in the paste destination image. The acquisition method is the same as in steps S302 and S305. The CPU 101 stores the acquired position and distance of the main face in the RAM 103 (or HDD 105) in preparation for writing.

In step S704, the CPU 101 converts the relative coordinates stored as the virtual light source position into absolute coordinates in the paste destination image. The virtual light source absolute coordinates (tLight_x, tLight_y, tLight_z) are calculated by the following equation (4). That is,
tLight_x = tFace_x-reLight_x
tLight_y = tFace_y-reLight_y (4)
tLight_z = tFace_z-reLight_z
However, the center coordinates of the main face are (tFace_x, tFace_y, tFace_z) and the relative coordinates of the virtual light source (reLight_x, reLight_y, reLight_z).

  In step S705, the CPU 101 copies the virtual light source parameter referenced in step S702 in association with the paste destination image. Then, the CPU 101 changes the position of the light source constituting the virtual light source parameter to the absolute coordinates calculated in step S704. Thereby, the virtual light source can be arranged at the same relative position as the face of the copy source image with respect to the face of the paste destination image.

  In step S706, the CPU 101 performs writing on the paste destination image. In step S707, the CPU 101 updates the display of the target image in the preview area 203 and the thumbnail area 204 with the output image data based on the lighting result. After step S707, the CPU 101 ends the process shown in FIG.

  In step S708, the CPU 101 displays a warning notifying the user that there is no virtual light source parameter to be pasted, and ends the process shown in FIG.

  FIG. 8 shows an example of the result of CPU 101 detecting faces in steps S302 and S305. The CPU 101 stores the face detection result in the RAM 103 (or HDD 105).

Reference numeral 801 denotes a face number for uniquely identifying the detected face. The face No. 801 includes a serial number incremented for each face as a result of detection from a plurality of images. Used to manage a plurality of faces detected from a plurality of images.
Reference numeral 802 denotes a file name of image data subjected to face detection.
Reference numeral 803 denotes a determination result as to whether the face is a main face. The method for determining whether the face is a main face is as described above.
Reference numeral 804 denotes a face ID. It consists of a value assigned from 0 for each target image data.
Reference numeral 805 denotes center coordinates of the face in the horizontal direction in the image data. Reference numeral 806 denotes center coordinates of the face in the vertical direction in the image data. Reference numeral 807 denotes a face distance. This is a numerical representation of the physical distance of the subject's face from the image sensor.
Reference numeral 808 denotes a face size in the image data.
Reference numeral 809 denotes a face angle with the optical axis of the imaging apparatus as an axis.
Reference numeral 810 denotes a face direction. It is 0 when the front of the face is facing the imaging device.

  Reference numerals 811 to 817 denote entries or records indicating examples of detection results for each face. An entry 811 shows the result for the image example 220. An entry 812 shows the result for the image example 221. Entries 813 and 814 show results for the image example 222. Entries 815 and 816 indicate the results for the image example 223. An entry 817 indicates the result for the image example 224.

  In the above description, the face is detected in the relighting process and the virtual light source parameter paste process. Instead, face detection and face distance acquisition may be performed when a folder including image data to be processed is selected in the folder selection area 201. In this case, face detection and face distance acquisition in the relighting process and the virtual light source parameter paste process can be replaced by a face detection result reading process.

  In the above description, the main face is determined from the distance measurement frame position used at the time of focusing. However, when the user instruction or the user sets the virtual light source, the closest face may be set as the main face. By doing this, it is possible to apply pseudo lighting processing to the face of the other image, for the face designated as the main subject by the user or the face whose brightness is adjusted with the most effort.

  In the above description, the virtual light source parameter converted into the relative position is stored in the RAM 103 when the virtual light source parameter is copied. Instead, it may be stored in the HDD 105 as a separate file from the image file, or may be stored in the image file header or footer. In this case, it is possible to perform the same relighting process via a file even between devices having different copy source image files and paste destination image files.

  Although the case where there is one virtual light source has been described, this is for ease of understanding. Even when multiple virtual light sources can be used, the same relighting process is performed by converting the individual light source positions so that the relative position of the copy source image to the main face is the same as the relative position of the paste destination main face. Can be done.

  According to this embodiment, not only the position of the subject image on the plane but also the physical distance from the image sensor to the subject's face is calculated. Thereby, even if the position in the depth direction differs between the face included in the virtual light source parameter copy source image (first image) and the face included in the virtual light source parameter paste destination image (second image), A lighting effect similar to that of the first image can be given to the second image.

  Even if the second image includes a plurality of faces, the main face is specified in the second image, and the main face of the second image is set so that the relative position of the main face of the first image and the virtual light source is the same. The virtual light source position at is determined. Accordingly, the same lighting as the main face included in the first image can be applied to the main face of the second image. The virtual light source position with respect to the main face of the second image may be determined in consideration of the orientation of the main face in the first and second images. By doing so, lighting in the same irradiation direction as that for the main face of the first image can be applied to the main face of the second image.

  When the virtual light source parameter for the first image is copied, the position information is converted into a relative value and then copied, and when the second image is pasted, it is converted into an absolute value on the same coordinates as the main face. The same lighting environment can be realized for the second image. By holding the copied virtual light source parameters, the same lighting environment can be realized for a plurality of second images.

  A second embodiment in which the same lighting as the copy source is realized on each of the plurality of faces of the virtual light source parameter paste destination image will be described. FIG. 9 shows a flowchart of the operation. The CPU 101 executes the control program stored in the RAM 103, thereby realizing the processing shown in FIG. For example, the processing described in steps S702 to S706 is applied to each face in the copy destination image of the virtual light source parameter.

  When copying the virtual light source parameter before executing the processing shown in FIG. 9, the CPU 101 stores in advance in the RAM 103 the face brightness before rewriting of the main face and the face brightness after rewriting in the copy source image. . Hereinafter, the face brightness before rewriting is referred to as reference face brightness, and the face brightness after rewriting is referred to as target face brightness. The reference face luminance is the average luminance of the face position in the image data before relighting. The target face luminance is the average luminance of the face position in the rewritten image data. The CPU 101 obtains the reference face luminance and the target face luminance between steps S603 and S604, and stores them in the RAM 103 together with the virtual light source parameters in step S604.

  Since the processing of steps S901, S902, and S903 is the same as that of steps S702, S302, and S303, description thereof will be omitted.

  In step S904, the CPU 101 determines whether or not the counter i is smaller than the face number FaceNum. When the counter i is smaller than the face number FaceNum, the CPU 101 proceeds to step S905. When the counter i is greater than or equal to the face number FaceNum, the CPU 101 determines that the processing from step S905 to S912 has been repeated by the face number FaceNum, and the process proceeds to step S913. Since the processing from step S905 to S907 is the same as the processing from step S305 to S307, detailed description thereof is omitted.

  In step S908, the CPU 101 obtains the brightness of the face regarding the face having the face ID i. First, the CPU 101 calculates the range of image data including the face of interest from the center coordinates of the face and the size of the face, and sets the average luminance of the calculated range as the luminance of the face.

  In step S909, the CPU 101 converts the relative coordinates stored as the virtual light source position into absolute coordinates so that the relative position of the virtual light source is the same with respect to the face whose face ID is i. The virtual light source absolute coordinates (tLight_x, tLight_y, tLight_z) can be calculated by Equation (4) described above. In the equation (4), the center coordinates of the face with the face ID i are (tFace_x, tFace_y, tFace_z), and the relative coordinates of the virtual light source (reLight_x, reLight_y, reLight_z).

In step S <b> 910, the CPU 101 calculates the light intensity of the virtual light source such that the face brightness after lighting is the same as the target face brightness for the face with the face ID i. The light intensity outA that is the same as the target luminance can be calculated by Expression (5). That is,
outA = (tLumi-b2Lumi) x inA / (tLumi-b1Lumi) (5)
However, the reference face luminance is b1Lumi, the target face luminance is tLumi, the light intensity of the virtual light source parameter read in step S901 is inA, and the face luminance acquired in step S908 is b2Lumi. When the face luminance b2Lumi acquired in step S908 is larger than the target face luminance tLumi, the light intensity outA is set to zero.

  In step S911, the CPU 101 generates an output image for the face with the face ID i using the light source absolute coordinates calculated in step S909 and the light intensity calculated in step S910. Since the output image calculation method is the same as that in step S310, detailed description thereof is omitted.

  In step S <b> 912, the CPU 101 associates the face ID with i with the output image generated in step S <b> 912 and stores it in the RAM 103.

  In step S913, the CPU 101 synthesizes the output image associated with the face ID = 0 to the face ID = (FaceNum-1) at the original position of each face.

  In the second embodiment, for each face included in the virtual light source parameter paste destination image (second image), the same relative to the main face of the virtual light source parameter copy source image (first image). The same lighting can be applied by setting a virtual light source in relation.

  When the same face as the main face of the first image is included in the second image, lighting in the same irradiation direction as that of the main face of the first image is applied to the face of the second image. Also good.

  Further, since the light intensity of the virtual light source is changed according to the difference between the luminance of the main face of the first image and the luminance of each face included in the second image, the luminance of each face of the second image You can apply lighting that makes the same. In addition, even when the face is farther away from the imaging device in flash photography, the brightness of the virtual light source becomes stronger as the face becomes darker. Can also be obtained.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (9)

Face detection means for detecting a face from the first image and the second image;
Writing means for performing lighting with a virtual light source on the face detected by the face detection means;
The virtual light source parameter applied to the first image in the lighting means is applied to the face of the second image at the same relative position as the relative position of the virtual light source with respect to the face of the first image. And an image processing apparatus having a control unit applied to the second image.   The image processing apparatus according to claim 1, wherein the virtual light source parameter is a position and an angle of a virtual light source.   The image processing apparatus according to claim 2, wherein the virtual light source parameter further includes one or more of on / off of a virtual light source, light intensity, light source type, and light source color temperature.   Further, distance information indicating a distance from the imaging device is acquired at the time of capturing the first image and the second image, and the face detection is performed from the imaging device using the detection result of the face detection unit and the distance information. 2. The image processing apparatus according to claim 1, further comprising means for calculating a distance to the face detected by the means.   The writing means applies the facial brightness of the second image to the face of the second image so that the facial brightness after writing to the face of the second image is the same as the facial brightness after writing to the face of the first image. The image processing apparatus according to claim 1, wherein the light intensity of a virtual light source to be applied is adjusted.   6. The lighting device according to claim 1, wherein each of the plurality of faces of the second image performs lighting based on a virtual light source applied to the face of the first image. An image processing apparatus according to 1. A first face detection step of detecting a face from the first image;
A first lighting step of performing lighting with a virtual light source on the face detected in the first face detecting step;
A second face detection step of detecting a face from the second image;
An image processing method comprising: a second lighting step of performing lighting on the face detected from the second image at the same relative position as the relative position of the virtual light source with respect to the face of the first image. . A first face detection step of detecting a face from the first image;
A first lighting step of performing lighting with a virtual light source on the face detected in the first face detecting step;
A copy step of copying a relative position of the virtual light source applied to the face detected in the first face detection step in the first lighting step with respect to the face detected in the first face detection step;
A second face detection step of detecting a face from the second image;
A second lighting step of placing a virtual light source at the relative position copied in the copy step with respect to the face detected from the second image, and lighting the face detected from the second image; An image processing method comprising: In a control program executed by the control means in an image processing apparatus having a face detection means for detecting a face from an image, a lighting means for lighting a target based on a virtual light source, an operation means, and a control means There,
A first face detection step for causing the face detection means to detect a face from a first image;
Supplying the face detected in the first face detection step as the target to the writing means, and causing the writing means to perform lighting by a virtual light source;
Receiving a change in the virtual light source parameter of the virtual light source in the first lighting step from the operation means;
Relative to the face detected in the first face detection step of the virtual light source applied to the face detected in the first face detection step in the first writing step according to the copy operation by the operation means A copy step to copy the position;
A second face detection step of causing the face detection means to detect a face from a second image;
Supplying the lighting means with the face detected in the second face detection step;
An arrangement step of arranging a virtual light source at the relative position copied in the copy step with respect to the face detected from the second image in accordance with a paste operation by the operation unit with respect to the writing unit;
A control program for an image processing apparatus, comprising: a second writing step for causing the lighting unit to perform lighting on a face detected from a second image by lighting using the virtual light source arranged in the arranging step. .

Images