JP2018073122A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of acquiring highly accurate normal information from an image of a subject. An image processing device that acquires normal information of a subject by using an image pickup device having an image pickup means and an illumination means, wherein the image processing device is based on the reflection characteristics of the surface of the subject. The irradiation means for irradiating the subject with light from the lighting means at a wavelength at which the intensity of the reflected light from the subject is equal to or higher than a predetermined value, and the reflected light from the subject are recorded in the image pickup means with sensitivity corresponding to the wavelength. The normal information on the surface of the subject is calculated based on the generation means for generating image data, at least one of the reflection characteristics, the wavelength and the sensitivity, and the image data generated by the generation means. It is provided with a calculation means. [Selection diagram] Fig. 2

Description

  The present invention relates to an image processing apparatus and an image processing method.

  An illuminance difference stereo method technique is known as a technique for acquiring normal line information, which is a kind of shape information on the surface of an object. Also, a gradient illumination method technique is known as an illuminance difference stereo method for acquiring normal line information with high accuracy with respect to a human face shape (Patent Document 1). Gradient illumination method technology simultaneously controls (flashes and dimms) a predetermined number of white light illuminations, acquires multiple images of subjects illuminated under different illumination conditions, and based on the acquired multiple subject images The line information is acquired.

U.S. Pat. No. 8,134,555

However, in the technique of Patent Document 1, recording is performed because the spectral radiance of illumination is relatively low and the spectral sensitivity of the camera is relatively low in a wavelength band where the spectral reflectance of the subject is relatively high. The image quality of the subject image was low (S / N ratio was low). For this reason, it is difficult for the technique of Patent Document 1 to acquire high-accuracy normal information.
In view of the above problems, an object of the present invention is to provide an image processing apparatus that can acquire high-precision normal information from an image of a subject.

  In order to achieve the above object, an image processing apparatus according to an aspect of the present invention is an image processing apparatus that acquires normal information of a subject using an imaging device having an imaging unit and an illumination unit, and the subject An irradiation means for irradiating the subject with light from the illumination means at a wavelength at which the intensity of the reflected light from the subject is a predetermined value or more based on the reflection characteristics of the surface of the object, and the reflected light from the subject Based on the generation means for generating the image data to be recorded in the imaging means with the sensitivity according to the above, at least one of the reflection characteristic, the wavelength and the sensitivity, and the image data generated by the generation means, Calculating means for calculating normal line information on the surface of the subject.

  According to the present invention, highly accurate subject normal information can be acquired.

1 is a schematic schematic diagram of an imaging apparatus according to Embodiment 1. FIG. It is a figure which shows the detail of the main components of the imaging device of FIG. It is a figure for demonstrating the basic technical idea of this invention. FIG. 3 is a functional block diagram of an image processing unit in the first embodiment. It is a schematic diagram which shows the property of the reflection in the translucent object from which a composition differs. 3 is a flowchart of processing in an image processing unit according to the first embodiment. 6 is a functional block diagram of an image processing unit according to Embodiment 2. FIG. 10 is a flowchart of processing in an image processing unit according to the second embodiment.

Hereinafter, an embodiment of an imaging device of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and all the combinations of features described in the present embodiment are not necessarily essential to the solution means of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.
Embodiment 1
In the present embodiment, an imaging apparatus capable of acquiring normal line information of a subject with high accuracy when the spectral reflection characteristic of the subject is known will be described. In the present embodiment, the subject is a person, and normal information can be acquired with high accuracy for the skin (human skin) of the person by using the photographing apparatus.

(Outline of imaging device)
FIG. 1 is a schematic diagram of an imaging apparatus 100 according to the first embodiment. As illustrated in FIG. 1, the imaging apparatus 100 includes a central control unit 101, imaging units (cameras) 102 to 105, illumination units 106 to 117, and a frame 118. The central control unit 101 controls the imaging units 102 to 105 and the illumination units 106 to 117 to photograph (capture) the subject 119. The frame 118 fixes the imaging units 102 to 105 and the illumination units 106 to 117. By fixing the imaging units 102 to 105 and the illumination units 106 to 117, the positions and orientations of the imaging units 102 to 105 and the illumination units 106 to 117 can be fixed during imaging. Note that the imaging apparatus 100 includes a data communication control unit (not shown) and a power source (not shown). However, the explanation is omitted because it is not important in explaining the present embodiment.

  The central control unit 101 controls the imaging units 102 to 105 and the illumination units 106 to 117. In addition, the central control unit 101 includes an image processing unit 210 that processes image data received from the imaging units 102 to 105 and calculates normal information of the subject 119. Serial port communication based on the USB 3.0 standard is applied to communication of control signals and image data between the central control unit 101 and the imaging units 102 to 105. Accordingly, a plurality of imaging units 102 to 105 can be connected using a USB hub. TCP / IP communication according to the Ethernet standard is applied to communication of control signals between the central control unit 101 and the lighting units 106 to 117. Accordingly, a plurality of lighting units 106 to 117 can be connected using an Ethernet hub. The inter-device connection standard is not limited to the above standard. For example, IEEE1394, CameraLink, CoaXPress, DMX, and DALI can be applied. TCP / IP is an abbreviation for Transmission Control Protocol / Internet Protocol.

The imaging units 102 to 105 photograph the subject 119 and record image data. The recorded image data is immediately transferred from the imaging units 102 to 105 to the central control unit 101. The imaging units 102 to 105 are synchronously controlled by the central control unit 101, and the exposure timing can be adjusted.
The illumination units 106 to 117 irradiate the subject 119 with light. Each of the illumination units 106 to 117 can control light emission, light adjustment, color adjustment, and polarization (details will be described with reference to FIG. 2). The lighting units 106 to 117 are synchronously controlled by the central control unit 101, and the light emission (irradiation) timing can be adjusted. In the present embodiment, the imaging apparatus 100 includes a plurality of illumination units.

  The frame 118 fixes the imaging units 102 to 105 and the illumination units 106 to 117. The shape of the frame 118 is desirably a shape that allows the imaging units and the lighting units to be arranged so that the distances of the imaging units 102 to 105 and the lighting units 106 to 117 with respect to the subject 119 are approximately equal. Therefore, in this embodiment, a spherical shape is adopted as the shape of the frame 118. In order to give the frame 118 a spherical shape, the following mechanism is used. In the explanation of the Earth, latitude is defined by connecting multiple circular iron pipes to connect the arbitrarily defined North Pole and South Pole, and circular iron pipes are further connected at multiple longitudes. To do. Thus, a spherical frame is realized by piping the iron pipe. In FIG. 1, the frame 118 is shown two-dimensionally, but in this embodiment, the frame 118 has a three-dimensional shape, and the imaging units 102 to 105 and the illumination units 106 to 117 are also arranged three-dimensionally.

The frame 118 may be provided with a flexible arm (not shown). By providing the flexible arm, the angular resolution between the illumination units using the frame 118 can be set to 1 ° or less, for example.
Further, the shape of the frame 118 is not limited to the shape described above. For example, the frame 18 may have a hemispherical shape, or may have a cylindrical shape by standing a plurality of straight pipes. Further, the material of the frame 118 is not limited to the iron pipe. The frame 118 may be made of any material as long as it can form a spherical shape. Examples of such materials include polystyrene foam, wire, wood, and cured resin.
The subject 119 is a person in this embodiment. The subject 119 is not limited to a person, and may be an object (including an animal) other than a person.

(Description of main components of imaging device)
FIG. 2 is a block diagram illustrating details of the central control unit 101, the imaging unit 102, and the illumination unit 106 of the imaging apparatus 100.
(Central control unit)
First, the configuration of the central control unit 101 will be described based on FIG. The central control unit 101 includes a central processing unit (CPU) 201, a RAM 202, a ROM 203, an auxiliary storage I / F 204, an auxiliary storage unit 205, a general-purpose input / output I / F 206, and a general-purpose input / output unit 207. The central control unit 101 further includes a display I / F 208, a display unit 209, an image processing unit 210, a photographing I / F 211, an illumination I / F 212, and a bus 213. CPU is an abbreviation for Central Processing Unit. RAM is an abbreviation for Random Access Memory. ROM is an abbreviation for Read Only Memory. I / F is an abbreviation for Interface.

  The CPU 201 comprehensively controls each unit described below. The RAM 202 functions as a main memory, work area, and the like for the CPU 201. The ROM 203 stores a control program executed by the CPU 201 and the like. The auxiliary storage I / F 204 functions as a digital data transfer path between an auxiliary storage unit 205 (to be described later) and other components in the central control unit 101. In this embodiment, it is assumed that the auxiliary storage I / F 204 is a serial ATA compliant device. Note that a device compliant with another standard may be adopted as the auxiliary storage I / F 204 in accordance with the device standard of the auxiliary storage unit 205 described later. ATA is an abbreviation for Advanced technology attachment.

  The auxiliary storage unit 205 stores a control program, image data, and the like. The auxiliary storage unit 205 is a non-volatile device that retains data even during a power failure, and generally has a larger storage capacity than the RAM 202 and complements the storage capacity of the RAM 202. In the present embodiment, the auxiliary storage unit 205 is an HDD (hard disk drive). The auxiliary storage unit 205 may apply another standard (a storage device other than the HDD may be adopted). Examples of such standards include SSD (solid state drive), CF (compact flash (registered trademark)), SD memory card, miniSD card, and microSD card.

The general-purpose input / output I / F 206 functions as a digital data transfer path between a general-purpose input / output unit 207 (to be described later) and other components in the central control unit 101. In the present embodiment, it is assumed that the general-purpose input / output I / F 206 is a USB-compliant device. Note that a device compliant with another standard may be adopted as the general-purpose input / output I / F 206 in accordance with the device standard of the general-purpose input / output unit 207 described later. An example of such a standard is PS / 2. USB is an abbreviation for Universal serial bus.
The general-purpose input / output unit 207 acquires input information input by the user. In this embodiment, it is assumed that the general-purpose input / output unit 207 is a keyboard. Note that a mouse may be added to the keyboard as the general-purpose input / output unit 207.
The general-purpose input / output unit 207 is not limited to a keyboard or a mouse. For example, the general-purpose input / output unit 207 may be configured by a touch panel, an infrared sensor, an eye tracker, and an electroencephalogram sensor.

The display I / F 208 functions as a digital data transport path between a display unit 209 described later and other components in the central control unit 101. In the present embodiment, it is assumed that the display I / F 208 is a DVI-compliant device. Note that a device compliant with another standard may be adopted as the display I / F 208 in accordance with the device standard of the display unit 209 described later. The display unit 209 is a visual user interface. In the present embodiment, it is assumed that the display unit 209 is a liquid crystal display. DVI is an abbreviation for Digital visual interface.
Note that the display unit 209 is not limited to a liquid crystal display. For example, the display unit 209 may be a cathode ray tube type display, a plasma display, an organic EL display, an LCOS projector, or a DLP projector. LCOS is an abbreviation for Liquid crystal on silicon. DLP is an abbreviation for Digital light processing.

  The image processing unit 210 is a dedicated device for image processing. The image processing unit 210 is mounted on an HW (hardware) as an image processing circuit, and the image processing circuit (the image processing unit 210) can perform processing at high speed. In the present embodiment, the image processing unit 210 is an FPGA board. The imaging I / F 211 functions as a digital data transfer path between the imaging units 102 to 105 and other components in the central control unit 101. In the present embodiment, the imaging I / F 211 is a device based on the USB 3.0 standard. FPGA is an abbreviation for Field-programmable gate array. Note that a device compliant with another standard can be adopted as the imaging I / F 211 in accordance with the apparatus standard of the imaging units 102 to 105 described later. Examples of such standards include IEEE 1394, CameraLink, CoaXPress, and GigE.

The illumination I / F 212 functions as a digital data transport path between the illumination units 106 to 117 and other components in the central control unit 101. In the present embodiment, it is assumed that the illumination I / F 212 is a device according to the Ethernet standard. In addition, according to the apparatus standard of the illumination units 106-117 mentioned later, the device based on another standard can also be employ | adopted as illumination I / F212. Examples of such standards include DMX and DALI. The bus 213 functions as a transfer path for various data.
The above are the main components of the central control unit 101, but the components are not limited to those described above. Moreover, you may take the structure which abbreviate | omitted any of the above-mentioned component. In addition, although the component of the central control part 101 exists other than the above, since it is not essential for implementation of this embodiment, description is abbreviate | omitted.

(Configuration of imaging unit)
Next, the configuration of the imaging unit 102 will be described. Since the imaging units 103 to 105 have the same configuration as the imaging unit 102, only the imaging unit 102 will be described, and description of the imaging units 103 to 105 will be omitted.
The imaging unit 102 includes a linear polarization filter 221, a lens 222, an ND filter 223, a diaphragm 224, a shutter 225, an optical low-pass filter 226, and an IR cut filter 227. The imaging unit 102 further includes a color filter 228, a sensor 229, an A / D conversion unit 230, and a camera control unit 231. ND is an abbreviation for Neural density, IR is an abbreviation for Infrared, and A / D is an abbreviation for Analog / digital. Since the imaging unit 102 is a camera in this embodiment, the control unit of the imaging unit 102 is referred to as a camera control unit 231.
The linear polarization filter 221 has a function of obtaining an optical image of a linearly polarized subject by extracting only an electric field vector component having a predetermined polarization principal axis direction from an incident non-polarized subject optical image.
The lens 222 is a focus lens. In this embodiment, a focus lens having a focal length of 50 mm is used. The ND filter 223 is an optical filter that is applied to the optical image of the subject that has undergone focus imaging, and has a function of reducing the luminance of the optical image of the subject.

The diaphragm 224 controls the opening of the imaging unit 102. In the present embodiment, the diaphragm 224 is configured by diaphragm blades.
The shutter 225 controls the exposure of the imaging unit 102. In the present embodiment, it is assumed that the shutter 225 is a global shutter. A shutter other than the global shutter may be adopted as the shutter 225. For example, a rolling shutter may be employed as the shutter 225.
The optical low-pass filter 226 is an optical low-pass filter that is applied to the optical image of the subject that has undergone focus imaging, and blocks high-frequency components of the spatial frequency of the subject optical image. This reduces moiré that occurs due to a mismatch between the spatial resolution of the sensor 229 described later and the spatial frequency of the subject optical image.

The IR cut filter 227 is an optical filter that is applied to an optical image of a subject that has undergone focus imaging, and blocks near-infrared wavelength components from the spectral wavelength components of the subject optical image. Thereby, the spectral wavelength component to which the sensor 229 described later is sensitive can be limited to the visible light band, and the color tone discomfort due to the near infrared component can be reduced.
The color filter 228 is an optical filter that is applied to the optical image of the subject that has undergone focus imaging, and filters only the wavelength component in a predetermined visible light band among the spectral wavelength components of the subject optical image. Thereby, the spectral wavelength component to which the sensor 229 described later is sensitive can be limited to only a wavelength component in a predetermined visible light band. There are a plurality of spectral wavelengths of the color filter, and a relatively large amount of filtering (filtering) is performed at R = 700 nm, G = 546 nm, and B = 436 nm. As a result, it is possible to express colors using tristimulus values. Note that the spectral wavelength of the color filter is not limited to the above-described value, and a filter having an arbitrary characteristic value in the center wavelength or bandwidth can be applied within the visible light range.

The sensor 229 is, for example, a sensor such as a CMOS or a CCD, and detects the amount of light by converting the optical image of the subject focused by each lens 222 into an electrical signal. At the time of detection, the photodiode that configures the sensor 229 with the Bayer array pattern configured by the color filter 228 has sensitivity to a wavelength component in a predetermined visible light band. The detected light quantity is output from the sensor 229 to the A / D conversion unit 230 as an analog value. The A / D conversion unit 230 performs A / D conversion on the analog value to generate a digital value. The digital value (digital data) is output to the camera control unit 231. The camera control unit 231 realizes functions such as focusing, opening a shutter, closing a shutter, and adjusting a diaphragm by controlling other components of the imaging unit 102. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. CCD is an abbreviation for Charge coupled device.
The above is the configuration of the imaging unit 102, but the constituent elements are not limited to the above, and any of the above constituent elements may be omitted. Further, although there are other components of the imaging unit 102 than the above, description thereof is omitted because it is not essential for the implementation of the present embodiment.

(Configuration of lighting unit)
Next, the configuration of the illumination unit 106 will be described. Since the illumination units 107 to 117 have the same configuration as the illumination unit 106, only the illumination unit 106 will be described, and description of the illumination units 107 to 117 will be omitted.
The illumination unit 106 includes an illumination unit control unit 241, a constant current source / dimming control unit 242, a light emitting unit 243, a color filter 244, and a linear polarization filter 245.
The lighting unit control unit 241 controls other components of the lighting unit 106 such as a function of instructing light emission, instructing dimming, instructing toning, and instructing a polarization main axis direction of linearly polarized light. Realize with. The configuration of the lighting unit control unit 241 conforms to the configuration of the central control unit 101. The CPU 201, the RAM 202, the ROM 203, the auxiliary storage I / F 204, the auxiliary storage unit 205, the general-purpose input / output I / F 206, and the display of the central control unit 101 It has the same function as the I / F 208. The lighting unit control unit 241 has a configuration similar to the central control unit 101, but is smaller and lighter than the central control unit 101. The general-purpose input / output I / F 2411 of the illumination unit control unit 241 is assumed to be a USB-compliant device in this embodiment. The general-purpose input / output I / F 2412 of the lighting unit controller 241 has a function of transmitting a PWM (pulse width modulation) signal. The PWM signal is a control signal for a device used for general purposes. The lighting unit control unit 241 has a function of adjusting the duty ratio of the PWM signal based on a control signal input from the central control unit 101 via the lighting I / F 212. PWM is an abbreviation for Pulse width modulation.

The constant current source / dimming control unit 242 receives power from a constant voltage DC power source (not shown) and also receives a PWM signal from the illumination unit control unit 241. The constant current source / dimming control unit 242 has a function of converting input power into constant current power and switching on / off of the constant current power intermittently based on the input PWM signal. The constant current source / dimming control unit 242 is an electronic circuit composed of electronic components including a resistance element and a transistor.
Constant current power is input to the light emitting unit 243 from the constant current source / dimming control unit 242. The light emitting unit 243 has a function of emitting light by photoelectric conversion from the input constant current power (light emitting function). As a device having a light emitting function, there is a light emitting diode (hereinafter referred to as “LED”). LED is an abbreviation for Light Emitting Diode.

  In the present embodiment, the brightness of the light emitted from the illumination unit 106 (˜117) is better to improve the S / N ratio of the optical image of the subject imaged by the imaging unit 102 (˜105). . For this reason, LED (light emission part 243) employ | adopts what is classified into power LED with a forward maximum allowable current value of several hundred mA scale. In this embodiment, the dimming function of the LED having a high linearity of the duty ratio and the light emission luminance is realized by the PWM control method. In the present embodiment, it is important that the light emitting unit 243 has high toning performance. For this reason, an LED device with high color rendering properties is used. A white LED whose intensity is concentrated at a specific spectral wavelength may be adopted as the light emitting unit 243.

The color filter 244 is an optical filter applied to the light emitted from the light emitting unit 243, and filters (filters) only the wavelength component in a predetermined visible light band. In the present embodiment, a variable bandwidth liquid crystal tunable filter is employed as the color filter 244. The central control unit 101 can control the bandwidth of the color filter 244. Note that the color filter 244 may be a filter with a fixed bandwidth.
The linear polarization filter 245 is an optical filter applied to the light emitted from the light emitting unit 243, and has a function of emitting linearly polarized light by extracting only an electric field vector component having a predetermined polarization main axis direction.

The illumination unit 106 may include a plurality of any or all of the constant current source / dimming control unit 242, the light emitting unit 243, the color filter 244, and the linear polarization filter 245. In that case, it is possible to further have a function of simultaneously emitting (generating a plurality of) lights having different luminance, chromaticity, and polarization main axis direction.
The above is the configuration of the lighting unit 106, but the constituent elements are not limited to those described above, and a configuration in which any of the above constituent elements is omitted may be employed. Moreover, although the illumination unit 106 also has components other than the above-mentioned, since it is not essential for implementation of this embodiment, description is abbreviate | omitted.

(Basic technical idea of this embodiment)
The outline of the basic technical idea of the present invention will be described with reference to FIG. FIG. 3A is composed of four graphs arranged in a horizontal row, and shows the spectral transfer characteristics of a conventional imaging device (for example, International Patent Application Publication WO2007 / 88709). In the process of generating image data (Image) from the optical image of the subject by A / D conversion, the imaging device is affected by three spectral transfer characteristics. The three spectral transfer characteristics are a spectral transfer characteristic of illumination, a spectral transfer characteristic of a subject, and a spectral transfer characteristic of a camera. 3A is a graph showing the spectral transfer characteristic of illumination, Object is a graph showing the spectral transfer characteristic of the subject, Camera is a graph showing the spectral transfer characteristic of the camera, and Image is a graph showing the spectral transfer characteristic of the camera. It is a graph which shows the produced | generated image data. The vertical axis of each graph indicates the signal intensity RI, and the horizontal axis indicates the wavelength λ.

The signal intensity RI of the image data is expressed as an integral value of the product λI of the transmission characteristic of the spectral distribution characteristic λL of the illumination, the spectral reflectance characteristic λO of the subject, and the spectral filter characteristic λC of the color filter of the camera, as shown in Equation 1. Is done.
RI = ∫λIdλ
λI = λL · λO · λC (Formula 1)
The spectral peak (luminance peak) wavelength λl of the illumination, the spectral reflection peak wavelength λo of the subject, and the spectral peak (sensitivity peak) wavelength λc of the camera color filter are generally different (λl ≠ λo, λc ≠ λo). Therefore, as shown in the image of FIG. 3, the signal intensity RI of the image data is relatively low depending on the value of the wavelength λo (value r1 in FIG. 3A).

  FIG. 3B shows the spectral transfer characteristics of the imaging apparatus 100 of the present embodiment. In the present embodiment, the spectral peak wavelength λl of illumination and the spectral peak wavelength λc of the color filter of the camera (imaging unit) are the same as the spectral reflection peak wavelength λo of the subject. The signal intensity RI can be relatively increased by using an illumination and a camera that realize this wavelength relationship (a relationship that the three wavelengths are substantially equal) (value r2 in FIG. 3B). In the imaging apparatus 100 of the present embodiment, since the signal intensity RI can be increased, it is possible to acquire image data with a high S / N ratio.

(Function of image processing unit)
The image processing unit 210 according to the first embodiment will be described with reference to FIG. As shown in FIG. 4, the image processing unit 210 includes a set value acquisition function unit 401, a light emission control function unit 402, an image data generation function unit 403, and a normal line information calculation function unit 404. FIG. 4 is a functional block diagram.
The set value acquisition function unit 401 acquires a predetermined set value. That is, the set value acquisition function unit 401 acquires illumination spectral parameters, subject spectral parameters, and camera spectral parameters.
In the present embodiment, illumination spectral parameters are illumination brightness control parameters a and b. The parameter a is the first transmittance, and is a control parameter that controls the luminance of light having a center wavelength of 420 nm. The parameter b is the second transmittance and is a control parameter for controlling the luminance of light having a center wavelength of 590 nm. The range of values that parameters a and b can take is 0% to 100%, respectively.

The spectral parameters of the subject are the spectral absorption molar coefficients (absorption molar coefficients) Rb and Rg of the molecules that compose (configure) the subject surface. In this embodiment, Rb is the spectral absorption molar coefficient at 420 nm of the hemoglobin molecule, and Rg is the spectral absorption molar coefficient at 590 nm of the hemoglobin molecule.
The spectral parameter of the camera is the sensitivity area of the image sensor (area ratio of the color filter). In this embodiment, it is the same as the relative area of the spectral filter of the image sensor, and the red sensitivity area Sr = 1, the green sensitivity area Sg = 2, and the blue sensitivity area Sb = 1.
The light emission control function unit 402 generates light emission control information (information for controlling light emission) sent from the central control unit 101 to the illumination units 106 to 117. The light emission control function unit 402 transmits light emission control information (light emission control signal) to the illumination unit control unit 241 using TCP / IP by Ethernet of the illumination I / F 212.

  The lighting unit controller 241 converts the received light emission control information into a PWM signal. That is, the illumination unit controller 241 generates a PWM signal from the light emission control signal. More specifically, if the received light emission control signal corresponds to the duty ratio of the PWM signal and the light emission control signal indicates an output of 50% with respect to the maximum value of the light emission luminance, the lighting unit control The unit 241 generates a PWM signal having a duty ratio of 50%. If the light emission control signal indicates an output of 100% with respect to the maximum value of the light emission luminance, the illumination unit control unit 241 generates a PWM signal with a duty ratio of 100%. This PWM signal is transmitted from the illumination unit controller 241 to the constant current source / dimming controller 242.

  The constant current source / dimming control unit 242 intermittently turns on / off the constant current power based on the received PWM signal. The light emitting unit 243 performs photoelectric conversion using the supplied constant current power and emits light. In the first embodiment, the light emission control function unit 402 causes the spectrally variable bandwidth tunable filter of the color filter 244 to emit two spectra having peak wavelengths at the center wavelength of 420 nm and 590 nm. At this time, the light emission control function unit 402 acquires the first transmittance a and the second transmittance b from the set value acquisition function unit 401, and adjusts the spectral sensitivity characteristic.

  The image data generation function unit 403 generates a shooting control signal (information for instructing shooting) to be transmitted from the central control unit 101 to the imaging units 102 to 105. Then, the image data generation function unit 403 transmits a shooting control signal for instructing shooting to the camera control unit 231 using serial communication by USB of the imaging I / F 211. The camera control unit 231 executes the exposure control function of the shutter 225 based on the received shooting control signal. As the exposure control function of the shutter 225 is executed, the camera control unit 231 acquires image data by subjecting the optical image of the subject to AD conversion. The acquired image data is transferred from the camera control unit 231 to the central control unit 101 using USB serial communication of the imaging I / F 211. The transferred image data is stored in the RAM 202. Note that the transferred image data may be stored in the auxiliary storage unit 205.

The F value, focusing surface, aperture control, exposure time control, and imaging sensitivity of the lens 222 are determined based on camera setting information stored in the imaging units 102 to 105 in advance. The central control unit 101 can use camera setting information as information for instructing the imaging units 102 to 105 to perform shooting.
Thereby, subject images by two spectra having peak wavelengths at center wavelengths of 420 nm and 590 nm are individually acquired as image data. In the following description, a subject image obtained by irradiating light from a light source having a peak wavelength at a central wavelength of 420 nm is denoted as Img420, and obtained by irradiating light from a light source having a peak wavelength at a central wavelength of 590 nm. The subject image obtained is denoted as Img590.

The image data generation function unit 403 further performs correction processing on the acquired image data. This correction process will be described in S604 of FIG. The image data generation function unit 403 generates image data by acquiring and correcting the image data. The image data generation function unit 403 generates a plurality of image data.
The normal information calculation function unit 404 calculates the normal information of the subject by using a plurality of image data with different illumination conditions acquired by performing different light control on each of the illumination units 106 to 117. In addition, normal information is acquired from a plurality of image data having predetermined spectral characteristics, and the normal information is adaptively weighted and added to the plurality of normal information having different spectral characteristics, thereby improving the accuracy of the normal information. Plan.

(Reflection of translucent object)
With reference to FIG. 5, the outline of the reflection property of a translucent object having a different composition such as human skin will be described. In order to calculate the normal information of a semi-transparent object, it is important to decompose the reflected light component and record it as image data for each reflected light component. The reflected light (Ref) from the human skin is a regular reflected light component (Spec) having a high reflection intensity at a reflection angle equal to the incident angle of the incident light to the human skin, and a surface diffused light component (Diff) having no declination characteristics. ), An internal scattered light component (Scat) that has a declination characteristic but is difficult to analyze. That is, Ref can be expressed as Equation 2.
Ref = Spec + Diff + Scat (Formula 2)
In order to decompose the specularly reflected light component and other components (surface diffused light component and internal scattered light component) and record them in the image data, for example, a polarization technique disclosed in Patent Document 1 is used. Hereinafter, the principle of a method for decomposing the surface diffuse light component and the internal scattered light component and recording the image data in the image data will be described.

(Light reflection on human skin surface)
FIG. 5A and FIG. 5B are schematic views showing the reflection of light on the human skin surface. More specifically, FIG. 5A shows an example of surface diffused light of human skin, and FIG. 5B shows an example of internal diffused light. In order to obtain high-accuracy normal information from semi-transparent materials with different compositions such as human skin, it is an effective means to separate surface diffused light and internally scattered light by utilizing the different composition of the object. is there.
Human skin is roughly composed of an epidermis layer and a dermis layer. It is known that the thickness of the skin layer is about 0.1 mm, and the thickness of the dermis layer is about 3 mm. Examples of pigments that affect color development include melanin in the epidermis layer and hemoglobin in the dermis layer. Although other pigments are present in the skin, the number of molecules present is relatively small, and therefore the description thereof is omitted in this embodiment. The x-axis in FIGS. 5 (a) and 5 (b) is an axis parallel to the human skin and along the lower surface of the dermis layer. The z axis is an axis perpendicular to human skin.
It can be seen from the ratio (difference) between the thickness of the epidermis layer and the dermis layer that the internal reflection of human skin is mainly caused by irregular reflection in the dermis layer (FIG. 5B).

FIG. 5C shows the wavelength-spectral absorption characteristics of the skin pigment in the visible light band (400 to 720 nm) and the ultraviolet and infrared bands in the vicinity thereof. In FIG. 5C, the left vertical axis represents the absorption molar coefficient (spectral absorption molar coefficient), the horizontal axis represents the wavelength, and the right vertical axis represents the absorption coefficient. Such wavelength-spectral absorption characteristics are shown, for example, in FIG. 3 of International Patent Application Publication No. WO2007 / 088709.
As can be seen from FIG. 5C, the spectral absorption characteristic of hemoglobin (oxygenated hemoglobin, reduced hemoglobin) has a small absorption molar coefficient especially in the vicinity of 700 nm in the visible light band. For this reason, if the human skin is irradiated with white light having a high color rendering property including a spectral component of 700 nm, the proportion of the internally scattered light in the reflected light increases.

In addition, as shown in FIG. 5C, hemoglobin has a maximum extinction molar coefficient at 420 nm and takes a high extinction molar coefficient even near 590 nm. It can be seen that light in these frequency bands is absorbed by hemoglobin in the process of repeated irregular reflection in the dermis and is not emitted outside the skin.
As described above, in the reflection on the translucent bodies having different compositions, it is possible to separate the surface diffused light and the internal scattered light by utilizing the wavelength-spectral absorption characteristics of the translucent body. In particular, in order to efficiently acquire (separate) surface diffused light from human skin, it is preferable to irradiate light in a wavelength band with a large spectral absorption molar coefficient (absorption molar coefficient) of hemoglobin and record it on the camera.

Next, a method for setting a light source that is optimal when the race is known by using light having center wavelengths of 420 nm and 590 nm will be described.
The epidermis layer on the surface of human skin contains melanin, which is a dominant pigment in skin coloration. As shown in FIG. 5C, in the spectral absorption characteristic of melanin (eumelanin), the light absorption molar coefficient decreases as the wavelength of light increases. This indicates that in the epidermis layer of the human skin surface, green light is reflected more strongly than blue light, and red light is reflected more strongly than green light.

Generally, it is known that the content of melanin in the epidermis layer varies depending on race. On the other hand, it is known that the content of hemoglobin in the dermis layer is unchanged regardless of race. White skin pigment is composed of melanin, which is relatively low in the epidermis layer, and hemoglobin having a certain content in the dermis layer. Black skin pigment is composed of melanin, which has a relatively high content in the epidermis layer, and hemoglobin having a certain content in the dermis layer. Asian skin has an intermediate property between whites and blacks, and is composed of melanin having a relatively medium content in the epidermis and hemoglobin having a constant content in the dermis.
By comparing the three races, if only light with a central wavelength of 420 nm is used for illumination, the reflected light intensity at the epidermis layer differs depending on the race, so the S / N ratio when recorded on the camera is reduced. It turns out that there is a possibility.

From the spectral absorption characteristics of melanin (eumelanin) in FIG. 5 (c), the spectral absorption molar coefficient of melanin is about 2 × 10 ³ cm ⁻¹ M ⁻¹ at 420 nm and about 1 × 10 ³ cm ^{− at} 590 nm. ¹ M ⁻¹ . From this, it can be seen that 590 nm light can obtain reflected light twice as much from the skin layer as compared to 420 nm light.
From the spectral absorption characteristics of hemoglobin in FIG. 5 (c), the spectral absorption molar coefficient of hemoglobin is about 5 × ¹⁰ 5 ^{cm -1 M} -1 in 420 nm, about 5 × in 590nm ¹⁰ 4 ^{cm -1 M} -1 It is. From this, it can be seen that 590 nm light can obtain reflected light of five times the amount of light from the dermis layer compared to 420 nm light.
It can also be seen that the spectral absorption molar coefficient ratio between melanin and hemoglobin is 1: 250 at 420 nm and 1:50 at 590 nm. For this reason, when the luminance is the same, 590 nm light has twice as much reflected light from the skin layer and five times more reflected light from the dermis layer than 420 nm light.

Consider a case where the melanin content in the skin layer of one race A is 2.5 times higher than the melanin content in the skin layer of another race B. In this case, when the light of 590 nm is used for illumination rather than the light of 420 nm, the reflected light from the skin layer is relatively increased, so that the S / N ratio of the optical image of the recorded subject is increased.
As described above, translucent bodies having different compositions such as human skin are characterized in that the spectral absorption characteristics change because the pigments differ depending on the composition, and the reflection properties change depending on the spectral wavelength. When this feature is applied to human skin, irradiating a subject (person) using 420 nm light and 590 nm light according to the race is the S / N of the diffused light component recorded in the camera. It can be seen that this is an effective means for improving the ratio. Both 420 nm light and 590 nm light are light in a wavelength band that causes light absorption of a predetermined amount or more.

(Image processing program processing)
A processing flowchart of the image processing program in this embodiment will be described with reference to FIG. In the present embodiment, the image processing program may have two implementation forms as follows. Any one of the mounting forms may be adopted.
(1) The image processing program is implemented as HW in the image processing unit 210, and the image processing unit 210 is executed in accordance with the driving of the imaging apparatus 100 (the image processing unit 210 executes the image processing program). .
(2) The image processing program is always stored in the ROM 203 or the auxiliary storage unit 205, and the image processing program is transferred to the RAM 202 in accordance with the driving of the imaging device 100. After the transfer, an image processing program is executed by the central processing unit (CPU) 201.

Next, the processing flow of the image processing unit 210 will be described with reference to FIG. 6 (that is, the case of (1) will be described). FIG. 7 is a functional block diagram corresponding to FIG.
In step S601, the light emission control function unit 402 controls light emission. Specifically, the band components having the center wavelengths of 420 nm and 590 nm are passed by the bandwidth variable liquid crystal tunable filter of the color filter 244. The spectral sensitivity characteristic of the variable bandwidth liquid crystal tunable filter is expressed by Equation 3 below.
LE = a.L420 + b.L590 (Formula 3)
In Equation 3, LE is the spectral sensitivity characteristic of the variable bandwidth liquid crystal tunable filter. L420 is a passband characteristic at a center wavelength of 420 nm. L590 is a passband characteristic at a center wavelength of 590 nm. a is the first transmittance, and b is the second transmittance. Equation 3 shows that the spectral sensitivity characteristic of the variable bandwidth liquid crystal tunable filter is a weighted linear sum of a bandpass filter with a center wavelength of 420 nm and a bandpass filter with a center wavelength of 590 nm.

The light emission control function unit 402 irradiates the subject with light from the illumination unit at a wavelength at which the intensity of the reflected light from the subject becomes a predetermined value or more based on the reflection characteristics of the surface of the subject.
In order for the light emission control function unit 402 to realize Equation 3, the lighting unit 106 may have the following configuration. That is, the illumination unit 106 may include a plurality of constant current source / dimming control units 242, light emitting units 243, and color filters 244. The illumination unit 106 may be configured to simultaneously emit light having the center wavelengths of 420 nm and 590 nm using two filters with fixed bandwidths having the center wavelength of 420 nm and the center wavelength of 590 nm. Further, in order for the light emission control function unit 402 to realize Equation 3, the illumination unit 106 may include an LED whose intensity is concentrated at a specific spectral wavelength.

In step S <b> 602, the image processing unit 210 (light emission control function unit 402) uses the set value acquisition function unit 401 to acquire the first transmittance a and the second transmittance b in Equation 3 (used in Equation 3). To do. The first transmittance a and the second transmittance b are input from the RAM 202 to the image processing unit 210. That is, the set value acquisition function unit 401 acquires the transmittances a and b from the RAM 202, and the light emission control function unit 402 acquires the transmittances a and b from the set value acquisition function unit 401. Since it can be said that the first transmittance a and the second transmittance b are set values, it is described that the set values are acquired in S602 of FIG. Since there are other setting values (to be described later), “maru 1 (1 in circle)” is added after “acquire” in S602.
The image processing unit 210 may acquire the first transmittance a and the second transmittance b from the ROM 203, the auxiliary storage unit 205, or the general-purpose input / output unit 207. Further, the first transmittance a and the second transmittance b may be set based on age and skin color instead of being set based only on race. This is because age and skin color are also correlated with the melanin content in the human skin epidermis layer.

In step S603, the image processing unit 210 (image data generation function unit 403) acquires image data. In order for the image processing unit 210 to acquire image data, first, the image data (Img420 and Img590) is transferred from the imaging unit 102 to the RAM 202, and then the image processing unit 210 acquires the image data from the RAM 202. The color filter 228 includes two filters having a center wavelength of 420 nm and a center wavelength of 590 nm. The sensor 229 can decompose and record Img420 and Img590 by the spectral function of the color filter 228. In this way, Img 420 and Img 590 recorded in the imaging unit 102 are transferred to the RAM 202.
In step S604, the image processing unit 210 (image data generation function unit 403) performs image data correction processing. In step S604, before actually performing the correction process, the image data generation function unit 403 first determines whether to correct the image data using the illumination brightness control parameter and the sensitivity area of the imaging sensor. Specifically, the following threshold th is calculated using the illumination brightness control parameter and the sensitivity area of the imaging sensor.
th = a · (Rb ⁻¹ ) · Sb / b · (Rg ⁻¹ ) · Sg (Formula 4)

Here, the spectral reflectance from the dermis layer of human skin is approximated by applying the inverse of the spectral absorption molar coefficients Rb and Rg. For this reason, th is in the form of the ratio of the product of the relative intensity of illumination, the spectral reflectance of the subject, and the spectral sensitivity of the camera. If th >> 1, Img420 determines that the S / N ratio of the subject image is relatively higher than Img590. In this case, Img590 is corrected using Img420 as a feature amount. Accordingly, it is possible to perform correction so that the S / N ratio of the subject image is increased by using Img420 having a high S / N ratio with respect to Img590 having a relatively high resolution.
In the present embodiment, when th> 2, the following correction processing is performed. If th < 2, the following correction process is not performed, and the process proceeds to S606. Note that whether or not to perform the correction process is determined based on the magnitude relationship between th and 2, but whether or not the correction process is performed may be determined based on the magnitude relationship between th and a numerical value other than 2.

（式５） Next, the correction process will be described. The following filter processing is applied using image data recorded by decomposing Img420 and Img590. By applying the filter processing, the S / N ratio of the image data in which the subject optical image is recorded by the reflected light having the center wavelength of 590 nm is improved. Specifically, the filtering process is performed (calculated) according to the following Expression 5.

(Formula 5)

  In Expression 5, ImgFilter represents a filter image obtained as an output of filter processing. h represents the weighting coefficient of each pixel in the patch centered on the pixel to be filtered. σs represents a first parameter that determines the distribution of h. σr represents a second parameter that determines the distribution of h. The filter process represented by Expression 5 is a process using a joint bilateral filter. The joint bilateral filter is described in, for example, the document “Petschnig, Georg, et al.” Digital photography with flash and no-flash image pairs. “ACM transactions on graphics (TOG). Vol. 23. No. 3. ACM, 2004.”.

Expression 5 is characterized in that the weighting factor h is a function based on the pixel value of Img420 and that the calculated weighting factor h and Img590 different from Img420 are convolved. The weight coefficient h is a function of the first transmittance a. For this reason, it is possible to change the weighting coefficient h according to the melanin content of the skin by race.
Note that the image data generation function unit 403 may use a filter similar to the bilateral filter that performs a filtering process on image data to be corrected using a plurality of pieces of image data. An example of such a filter is a guided filter.

  In step S <b> 605, the image processing unit 210 (image data generation function unit 403) uses the set value acquisition function unit 401 to set predetermined setting values ((a) spectral parameters of illumination, (b) subject spectral parameters, and (c). ) Camera spectral parameters). The predetermined setting value is acquired when the setting value acquisition function unit 401 (image processing unit 210) reads the predetermined setting value from the RAM 202 in response to a reference (request) from the image data generation function unit 403. . The setting value acquired in S605 is used in the image data correction in S604.

  In step S <b> 606, the image processing unit 210 (normal information calculation function unit 404) outputs the first normal information Normal 420 from the plurality of input Img 420 and the second normal information Normal 590 from the input plurality of Img 590. And get. Since Img420 and Img590 absorb internal scattered light in the dermis layer of human skin, they relatively contain much surface diffused light from the human skin epidermis layer. For this reason, it is possible to acquire normal information with high accuracy. Alternatively, the third normal information NormalFilter may be acquired using the filter image ImgFilter generated by Expression 5.

  When acquiring the first normal line information Normal 420 and the second normal line information Normal 590 in S606, for example, an illuminance difference stereo method is used. The photometric stereo method is described in the document “Woodham, Robert J.” Photometric method for determining surface orientation from multiple images. "Optical engineering 19.1 (1980): 191139-191139."

Here, the income method of the normal information of the subject will be described. When the Shape from Shading method is used, the angle formed by the illumination direction L and the normal line N is calculated according to the following Expression 6.
i = ρ _d N · L = ρ _d cos (θ) (Formula 6)
The shape from shading method is described in the literature “Lee, Kyouung Mu, and CC Kuo.” Shape from shading with a linear element surface model. "Pattern Analysis and Machine Intelligence, IEEE Transactions on 15.8 (1993): 815-822."
In Equation 6, i represents the relative luminance values, [rho _d represents the diffuse reflectance of the object surface (diffuse albedo). When Expression 6 is applied to image data, there is a problem that it is difficult to uniquely specify normal information from single image data (a problem of Shape from Shading method). This is because the image data has two azimuths (vertical and horizontal), and the normal can have two degrees of freedom, and the determination of the absolute value is added to this.

（式７） As a method for solving the problem of the Shape from Shading method, in the illuminance difference stereo method, normal information is acquired by using a plurality of image data having different illumination directions. When shooting a plurality of times in the lighting conditions of n as to obtain image data, the direction vector of the incident light and L ₁ ~L _n, the source light illumination and a ₁ ~a _n. R pixels are acquired from the image, normal vectors of the pixels are set to N _{1 to} N _R, and reflectances are set to ρ _d1 to ρ _dn . Also in the illuminance difference stereo method, the normal is calculated from the subject surface based on ρ _d (diffuse reflectance: diffuse albedo). As the observation data, _nR elements I ₁₁ to InR are obtained by the combination of the illumination condition number n and the pixel number R, and when these are handled as a matrix, the observation matrix I can be expressed as the following Expression 7. .

(Formula 7)

In Equation 7, S indicates the azimuth and illuminance of the light source, and assuming that the azimuth and illuminance of all the light sources are known (that is, S is given), a matrix N including information on the normal and the reflectance. Can be expressed as in Equation 8.
^{N = S + I S + =} (S T S) -1 S T ( Equation 8)
Since N is a matrix of R rows and 3 columns, Equation 8 is solved as a simultaneous equation when n = 3, and a solution is obtained as a least square method when n> 3.
As described above, in the Shape from Shading method and the illuminance difference stereo method, it is understood that the normal of the subject is acquired assuming ρ _d (diffuse reflectance: diffuse albedo). For this reason, in order to obtain normal information with high accuracy using these methods, it is necessary to remove reflected light components (regular reflected light components and internal scattered light components) other than the surface diffused light components in the reflection characteristics of the subject. Is important.

In S606, the normal information calculation function unit 404 corrects the acquired first normal information Normal 420 and the second normal information Normal 590 as follows (normal C that is final normal information is corrected as follows). Calculation to obtain).
NormalC =
a / (a + b) · Normal 420 + b / (a + b) · Normal 590 (Formula 9)
It is also possible to perform correction using the spectral reflectance of the subject as shown in Equation 10 below.
NormalC =
(Rb ⁻¹ ) / {(Rb ⁻¹ ) + (Rg ⁻¹ )} · Normal420
+ (Rg ⁻¹ ) / {(Rb ⁻¹ ) + (Rg ⁻¹ )} · Normal590
(Formula 10)
Moreover, it is also possible to perform correction using the illumination brightness control parameter, the spectral reflectance of the subject, and the sensitivity area of the image sensor, as shown in Equation 11 below.
NormalC =
A / (A + B) ・ Normal420 + B / (A + B) ・ Normal590
A = a · (Rb ⁻¹ ) · Sb B = b · (Rg ⁻¹ ) · Sg
(Formula 11)

As described above, according to the present embodiment, the weight (a, b) according to the spectral characteristic of the illumination unit, the weight (Rb ⁻¹ , Rg ⁻¹ ) according to the spectral reflectance of the subject, and the sensitivity characteristic of the image sensor. The product of the three transfer characteristics of the weights (Sb, Sg) is optimized. This optimization improves the S / N ratio of the normal information to be acquired.
In step S <b> 607, the image processing unit 210 (normal information calculation function unit 404) uses the setting value acquisition function unit 401 to set predetermined setting values ((a) spectral parameters of illumination, (b) spectral parameters of the subject, and ( c) Camera spectral parameters) are acquired. The predetermined set value is acquired in response to a reference (request) from the normal vector information calculation function unit 404. The predetermined set value is used for normal information calculation in S606.

(Effect of Embodiment 1)
According to the first embodiment, by adjusting the spectral distribution characteristics of the illumination unit and the spectral filter characteristics of the camera, the spectral peak wavelength λl of illumination and the spectral peak wavelength λc of the color filter of the camera are changed to the spectral reflection peak wavelength λo of the subject. (FIG. 3B). As a result, the S / N ratio of the captured subject image increases (becomes a predetermined value or more), so that normal line information can be acquired with high accuracy.
Further, the imaging apparatus 100 according to the present embodiment can acquire normal information of a subject with high accuracy when the spectral reflection characteristics of the subject are known. The subject is specifically a person, and the imaging apparatus 100 can acquire normal information on the skin of the person with high accuracy.
In addition, after separating the specularly reflected light component from the human skin and the surface diffused light component / internally scattered light component by the method using the polarization of Patent Document 1, the surface diffused light and the internal scattered light implemented in this embodiment are separated. May be separated. In this case, in the process of acquiring the normal information using the surface diffused light component, the influence of the specularly reflected light component can be eliminated, so that the normal information can be calculated with higher accuracy.

(Modification)
In the first embodiment, the subject 119 is a person, but the subject 119 is not limited to a person. The subject only needs to have dimensions that fit within the frame 118 of the imaging apparatus 100.
In the first embodiment, the imaging units 102 to 105 and the illumination units 106 to 117 are fixed by the frame 118. However, the imaging units 102 to 105 and the illumination units 106 to 117 are fixed by using a structure or mechanism other than the frame. May be.
In the first embodiment, the central control unit 101 includes the general-purpose input / output unit 207 and the display unit 209. However, the general-purpose input / output unit 207 and the display unit 209 may be provided outside the central control unit 101. Further, the number of imaging units and the number of illumination units are not limited to those illustrated.

In the first embodiment, the central control unit 101 is included in the imaging apparatus 100, but the central control unit 101 may not be included in the imaging apparatus 100. In this case, the central control unit 101 is provided separately from the imaging device, for example, as an image processing device.
At least a part of the functional blocks shown in FIG. 4 may be realized by hardware. When realized by hardware, for example, a dedicated circuit may be automatically generated on the FPGA from a program for realizing each step by using a predetermined compiler. Further, a Gate Array circuit may be formed in the same manner as an FPGA and realized as hardware. Further, it may be realized by an ASIC (Application Specific Integrated Circuit).

Embodiment 2
In the first embodiment, the imaging device 100 has been described on the assumption that the melanin factor portion (a mole, a stain, or a freckles) does not exist significantly in the skin of the person who is the subject (the melanin factor portion is not considered). In the second embodiment, an imaging apparatus capable of acquiring normal line information with high accuracy when a melanin factor portion is significantly present in the skin of a person will be described. For this reason, the image pickup apparatus of the second embodiment statistically analyzes the image data of the subject and corrects the first transmittance a and the second transmittance b, thereby increasing the image data by the surface reflected light. Acquired with accuracy. The melanin factor part refers to a mole or the like generated on the skin due to a factor (cause) of melanin.
In the following description, differences from the first embodiment will be mainly described, and description of processing units and processing flows having the same configurations as those of the first embodiment will be omitted. The same components as those in the first embodiment are denoted by the same reference numerals. The configuration of the imaging apparatus 100 according to the second embodiment is the same as that of FIGS. 1 and 2 (first embodiment), but the function of the image processing unit 210 is different from that of the first embodiment. The image processing unit 210 according to the second embodiment will be described with reference to FIGS. 7 and 8.

(Configuration of Image Processing Unit 210)
In FIG. 7, the image processing unit 210 includes a set value acquisition function unit 401, a light emission control function unit 402, an image data generation function unit 403, a normal information calculation function unit 404, and a set value correction function unit 701. When the image processing unit 210 in FIG. 7 is compared with the image processing unit 210 in FIG. 4, a setting value correction function unit 701 is added to the image processing unit 210 in FIG. 7.
The set value correction function unit 701 calculates a first corrected transmittance a + that is a correction value for the first transmittance a and a second corrected transmittance b + that is a correction value for the second transmittance b. .
When the surface area of the melanin factor part (a mole, a stain or a freckles) is significant with respect to the surface area of human skin, light absorption increases in the melanin factor part. As a result, the ratio of Img420 to Img590 is different between the melanin factor region and the other regions (inside and outside of the melanin factor portion). Therefore, the dispersion value (difference) of the ratio in the surface area of the skin increases as the surface area of the melanin factor increases.
When the surface area of the melanin factor portion is large, by correcting the first transmittance a and the second transmittance b so as to emit a large amount of light of 590 nm with a relatively small spectral absorption molar coefficient of the melanin pigment, A diffuse reflected light component can be acquired stably with respect to the surface area of the skin of interest.

The light emission control function unit 402 acquires the first corrected transmittance a + and the second corrected transmittance b + calculated by the set value correction function unit 701 from the set value acquisition function unit 401, and a shown in Equation 3 LE is recalculated by applying a + to b and b + to b. As described above, LE is the spectral sensitivity characteristic of the variable bandwidth liquid crystal tunable filter. A method for calculating a + and b + will be described later with reference to FIG.
The image data generation function unit 403 calculates the threshold th by applying a + to a and b + to b in Equation 4. Based on the calculation result of th, calculation of the weight function (coefficient) h in Expression 5 and joint bilateral filter processing are executed.
The normal line information calculation function unit 404 calculates NormalC by applying a + to a and applying b + to b in Equation 9 or Equation 11.

(Processing flow of the image processing unit 210)
The processing flow of the image processing unit 210 according to the second embodiment will be described with reference to FIG. Steps S601 to S605 and S606 to S607 are the same as those in the first embodiment (FIG. 6).
In S801 of FIG. 8, the image processing unit 210 (setting value correction function unit 701) performs the following processing. More specifically, the feature amount d shown in Expression 12 is calculated in the target subject areas of Img420 and Img590.
d = Var (Img420 / Img590) (Formula 12)
In Expression 12, Var (x) represents a variance value of the input value x (a variance value of Img420 / Img590). The input value x is a ratio between the intensity of the image data (Img420) with a wavelength of 420 nm (first wavelength) and the intensity of the image data (Img590) with a wavelength of 590 nm (second wavelength).

In the case of | d | >> 1, it is shown that the difference (fluctuation) in the absorption of light between the inside and outside of the melanin factor portion, which is the difference in reflected light between the spectral images, is significant. When | d | >> 1, the set value correction function unit 701 performs a process of correcting (adjusting) the first transmittance a and the second transmittance b (Formula 13).
If | d | ≈1 or | d | <1, the setting of the correction values (first transmittance a and second transmittance b) is terminated, and the process proceeds to S606.
The set value correction function unit 701 uses the feature quantity d calculated by Equation 12 to use the first corrected transmittance a + for the first transmittance a and the second transmittance b shown in Equation 13 below. And the second corrected transmittance b + is calculated.
a + = a · (1−d) b + = b · d (Formula 13)
In Expression 13, the feature amount d may be subjected to normalization processing to simplify the calculation.

If the set value is corrected in S801, the set value correction function unit 701 sends the corrected set value to the set value acquisition function unit 401. Then, in the process of “acquiring setting values” in S602, S605, and S607, the corrected setting value is acquired. This is shown by the arrow appearing to the right from S801 in FIG.
(Effect of Embodiment 2)
According to the second embodiment, it is possible to acquire normal information with high accuracy when a feature of a melanin factor part (a mole, a stain, or a freckles) is significantly present in the skin of a person who is a subject.

(Modification)
Although the setting value correction function unit 701 is included in the image processing unit 210 in FIG. 7, the setting value correction function unit 701 may be provided separately from the image processing apparatus 210 as a setting value correction unit, for example.
(Other embodiments)
The present invention supplies a program (computer program) that realizes one or more functions of the image processing unit according to the above-described embodiment to a system or apparatus via a network or a storage medium. The above processor can also be realized by a process of reading and executing a program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  DESCRIPTION OF SYMBOLS 101 ... Central control part, 102-105 ... Imaging part, 106-117 ... Illumination unit, 210 ... Image processing part, 401 ... Setting value acquisition function part, 402 ... Light emission control function part, 403 ... Image data generation function part, 404 ... Normal information calculation function

Claims (18)

An image processing apparatus that obtains normal information of a subject using an imaging apparatus having an imaging means and an illumination means,
Irradiating means for irradiating the subject with light from the illuminating means at a wavelength at which the intensity of the reflected light from the subject is a predetermined value or more based on the reflection characteristics of the surface of the subject;
Generation means for causing the imaging means to record the reflected light from the subject with sensitivity corresponding to the wavelength, and to generate image data;
Calculation means for calculating normal information of the surface of the subject based on at least one of the reflection characteristic, the wavelength and the sensitivity, and image data generated by the generation means;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein a plurality of the illumination units are provided at different positions, and the irradiation unit irradiates the subject with a plurality of lights simultaneously from the plurality of illumination units.   The surface of the subject is a translucent body having a predetermined thickness, and the light has a wavelength band that causes a predetermined amount or more of light absorption inside the translucent body. The image processing apparatus described.   When there are a plurality of wavelengths, the irradiating unit irradiates the light of the plurality of wavelengths from the illuminating unit with a predetermined transmittance, and the generating unit performs the image by the imaging unit according to the reflected light of the plurality of wavelengths. The image processing apparatus according to claim 1, wherein a plurality of data is generated.   5. The irradiating means irradiates the light with a wavelength that prevents an internally scattered light component generated on the surface of the subject from being included in the reflected light. The image processing apparatus according to item.   The generation unit corrects the image data based on the reflection characteristic, the transmittance, and the sensitivity, and the calculation unit calculates normal information using the corrected image data. The image processing apparatus according to claim 4.   When the wavelengths are the first wavelength and the second wavelength, the generation unit filters the image data based on the second wavelength based on a parameter obtained from the image data based on the first wavelength. The image processing apparatus according to claim 6, wherein the correction is performed.   The image processing apparatus according to claim 7, wherein the first wavelength is 420 nm and the second wavelength is 590 nm.   The image processing apparatus according to claim 1, wherein the reflection characteristic is an extinction molar coefficient of a molecule constituting the surface of the subject.   The image processing apparatus according to claim 4, 6, 7, or 8, wherein the irradiation unit adjusts the luminance of the light having the plurality of wavelengths according to the transmittance.   10. The image processing apparatus according to claim 1, wherein when the imaging unit includes a plurality of color filters, the sensitivity is an area ratio of the plurality of color filters.   The calculation means calculates a weighting factor based on the reflection characteristic, the sensitivity, and the transmittance, and calculates the normal line information based on the calculated weighting factor. The image processing apparatus according to 7, 8 or 10.   The imaging apparatus according to claim 4, 6, 7, 8, 10, or 12, further comprising a correcting unit that corrects the transmittance according to a result of statistical analysis using the image data. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 13, wherein the correction unit corrects the transmittance according to a result of statistical analysis regarding light absorption on the surface of the subject.   When the wavelengths are the first wavelength and the second wavelength, the correction unit is based on a change in a ratio between the intensity of the image data based on the first wavelength and the intensity of the image data based on the second wavelength. The image processing apparatus according to claim 13, wherein the transmittance is corrected. An irradiation means for irradiating the subject with light;
Imaging means for recording reflected light from the subject;
The image processing apparatus according to any one of claims 1 to 15,
An imaging apparatus comprising: An image processing method for obtaining normal information of a subject using an imaging device having an imaging means and an illumination means,
Irradiating the subject with light from the illumination means at a wavelength at which the intensity of reflected light from the subject is greater than or equal to a predetermined value based on the reflection characteristics of the surface of the subject;
Recording the reflected light from the subject on the imaging means with sensitivity according to the wavelength, and generating image data;
Calculating normal information of the surface of the subject based on at least one of the reflection characteristics, the wavelength, and the sensitivity, and the generated image data;
An image processing method comprising:   The program for functioning a computer as a means of the image processing apparatus of any one of Claim 1 to 15.

Images