JP2012029029A - Image processing device, image processing method and imaging device - Google Patents

Abstract

An image processing apparatus, an image processing method, an image processing program, and an imaging apparatus suitable for generating output images having different gradation characteristics by converting luminance information of an input image.
An image processing apparatus includes a photometry unit and an HDR main line processing unit. In the photometry unit, pixels obtained by imaging with a short exposure time T2 shorter than a standard exposure time T3 are obtained. A histogram is generated from the data S2, the level of the contrast ratio of the captured image is determined based on the histogram, and the characteristic of the brightness level correction gain used when converting the brightness of the HDR color image data is controlled based on the determination result. A level correction control parameter is generated, and the HDR main line processing unit 26 selects an LUT corresponding to the parameter from a plurality of types of conversion LUTs having a characteristic of further suppressing the gain of the low luminance part as the contrast ratio is higher. The luminance of the HDR color image data is converted using the selected LUT.
[Selection] Figure 8

Description

  The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus suitable for generating an output image having different gradation characteristics by converting luminance of an input image.

  Since the electronic camera has a narrow dynamic range (hereinafter referred to as “D range”) of the image sensor, exposure control is performed. In exposure control, the amount of light input to the image sensor is adjusted by the electronic shutter speed and the aperture so as to fall within the D range of the image sensor (see, for example, Patent Document 1). In the exposure control, the shutter speed and the aperture are controlled so that the average value of the signal level output from the image sensor becomes a specific value (for example, 18% of the D range). Generally, exposure control is performed by closed loop control.

On the other hand, HDR cameras exist as cameras that can obtain captured images with a wide D range. In order to compensate for the narrowness of the D range of the image sensor, an HDR image is generated by capturing a plurality of images with different exposure times and synthesizing them (for example, refer to Patent Document 2). The HDR camera does not require the above-described exposure control.
Temporarily, the exposure amount at the time of exposure for the longest time is set as the long-time exposure amount S3, the exposure amount at the time of exposure at the shortest exposure time is set as the short-time exposure amount S1, and Let the exposure amount be a medium time exposure amount S2. The HDR camera can increase its dynamic range as the ratio of S3 and S1 increases. Further, it is known that the image quality (S / N) of an output image from the HDR camera is improved as the number of images taken with an intermediate exposure time is increased.

In HDR cameras, range compression is required to comply with standards such as JPEG and MPEG. Conventionally, many methods based on the retinex theory considering human visual characteristics have been published as methods for compressing the dynamic range of an input image by converting luminance information of the input image. For example, Non-Patent Document 1 concludes from experiments that “the light image entering the eye is represented by the product of the illumination light component and the reflection component, but it is the reflection component that has a strong visual correlation”. Based on this theory, a method for compressing the D range of the image while preserving the reflection component of the image and compressing only the illumination light component when compressing the dynamic range while preventing image quality degradation has been proposed (for example, (See Patent Document 3 and Patent Document 4). This process of compressing the D range is generally called tone map processing.

JP 2003-244538 A JP 2009-49547 A JP 2006-352825 A JP 2008-511048 A

E.H.Land, J.J.McCann, "Lightness and retinex theory", journal of the Optical Society of America, 61 (1), 1 (1971)

An HDR camera system in which saturation does not occur has a great feature that a stable image is always obtained because exposure control is unnecessary. However, the above tone map processing tends to impair the contrast of the video output image. In other words, the tone map process operates to increase the dark area gain and lift the dark area brightly, so when the contrast ratio of the subject is large, the image becomes sleepy with a hazy appearance. .

For example, consider taking a picture of the inside of a car with a camera installed in a car in an environment under hot weather. In this case, with a normal camera, the scenery from the car window stops, or the image inside the car is crushed black. On the other hand, the HDR camera clearly describes the scenery from the vehicle window, and also describes the scenery in the car that is originally crushed by increasing the gain by the tone map processing described above. In this way, the HDR camera can depict both the inside and outside of the vehicle as images, but by applying tone map processing, the images become sleepy images with a low contrast ratio.
Therefore, the present invention has been made paying attention to such unsolved problems of the conventional technology. According to some aspects of the present invention, it is possible to provide an image processing apparatus, an image processing method, and an imaging apparatus that are suitable for generating output images having different gradation characteristics by converting luminance information of an input image.

[Mode 1] In order to achieve the above object, an image processing apparatus according to mode 1 includes:
Among a plurality of image data obtained by imaging a subject with a plurality of different exposure times, the brightness of the first composite image data generated by combining two or more image data is converted to have a different gradation characteristic. Image conversion means for generating two composite image data;
Luminance distribution information generating means for generating luminance distribution information that is information related to the luminance distribution of the captured image of the subject based on at least one of the plurality of image data;
Conversion gain control means for controlling the characteristics of the gain used for conversion of the luminance of the image conversion means based on the luminance distribution information generated by the luminance distribution information generation means,
The image conversion means converts the luminance of the first composite image data using the gain controlled by the conversion gain control means.

With such a configuration, the first composite image generated by combining two or more pieces of image data among a plurality of image data obtained by imaging the subject with a plurality of different exposure times by the image conversion means. It is possible to generate the second composite image data having different gradation characteristics by converting the luminance of the data.
On the other hand, the luminance distribution information generation means can generate luminance distribution information related to the luminance distribution of the captured image based on at least one of the plurality of image data. Furthermore, it is possible to control the characteristics of the gain used for the luminance conversion of the image conversion means based on the luminance distribution information by the conversion gain control means.

The image conversion means can convert the luminance of the first composite image data using the gain controlled by the conversion gain control means.
This makes it possible to automatically control the characteristics of the gain used for conversion based on the luminance distribution information, and to reduce image quality deterioration in the output image after gradation conversion by appropriately controlling the characteristics of the gain. it can.
For example, by controlling the gain for the low gradation part of the first composite image data, it is possible to reduce image quality degradation due to a decrease in the contrast ratio of the low gradation part in the output image after the luminance conversion. it can.

Further, the upper limit of the gain can be changed according to the characteristics of the input image (first composite image data), the desired output image (second composite image data), and the like.
Thereby, it is possible to obtain an output image having desired image quality characteristics while reducing image quality deterioration due to a decrease in contrast ratio.
Here, the luminance conversion may be performed in units of one pixel or may be performed in units of a plurality of pixels at a specific position. For example, in the case of a plurality of pixels, an average luminance is obtained, an upper limit of gain corresponding to the average luminance is set, and luminance is converted according to the upper limit.

[Mode 2] Furthermore, the image processing apparatus according to mode 2 is the same as the image processing apparatus according to mode 1.
The luminance distribution information is a luminance histogram of the captured image.
With such a configuration, the state of the brightness contrast of the captured image can be known from the brightness histogram, so that accurate gain control can be performed.
[Mode 3] Furthermore, the image processing apparatus of mode 3 is the image processing apparatus of mode 1 or 2,
The image conversion means converts the luminance of the first synthesized image data to generate second synthesized image data in which the dynamic range of the luminance of the first synthesized image data is compressed.

With such a configuration, the image conversion means can output the luminance of the input image data from an output device such as a display device, for example, when the first composite image data is high dynamic range image data. Conversion can be performed in accordance with the gradation range.
For example, when the first composite image data has a 20-bit gradation range, the gradation range is adjusted to the gradation range that can be displayed by the display device, and a gradation range lower than this range. For example, the second composite image data compressed to an 8-bit gradation range can be generated.

[Mode 4] Furthermore, the image processing apparatus according to mode 4 is the image processing apparatus according to any one of modes 1 to 3,
The luminance distribution information generating means generates the luminance distribution information based on at least one image data of image data corresponding to the remaining exposure time excluding the longest exposure time among the plurality of image data.
With such a configuration, when capturing an image in a relatively bright place, the luminance distribution information can be generated from image data other than the overexposed image, so that accurate information can be generated. In particular, by using image data other than the longest and shortest, it is possible to generate luminance distribution information from image data that is blacked and has no whiteout.

[Embodiment 5] Furthermore, the image processing apparatus of Embodiment 5 is the image processing apparatus of any one of Embodiments 1 to 4,
The conversion gain control means determines whether the contrast ratio of the captured image is high or low based on the luminance distribution information. The gain used for the tone conversion is controlled to be smaller as the contrast ratio is higher.
With such a configuration, when it is determined that the contrast ratio is high, the gain can be adjusted so that the luminance after conversion does not become too high for pixel data having a predetermined luminance or less. Therefore, it is possible to generate a sharp output image with no black floating while restricting the low luminance portion of the input image from becoming too bright.

[Mode 6] Further, the image processing apparatus of mode 6 is the same as the image processing apparatus of mode 5,
The conversion gain control means controls the gain so that the gain used for gradation conversion of the pixel data having the predetermined luminance or less is fixed at a constant value.
With such a configuration, an upper limit value (limit) can be set for the gain for pixel data having a predetermined luminance or less. Accordingly, by appropriately setting the upper limit value of the gain, it is possible to reduce image quality deterioration due to a decrease in contrast of a low-luminance portion in the output image after converting the luminance of the input image.

[Mode 7] Furthermore, the image processing apparatus of mode 7 is the image processing apparatus of mode 5 or 6,
When it is determined that the contrast ratio is low, the conversion gain control means controls the gain used for gradation conversion of pixel data having the predetermined luminance or less so that the gain becomes higher as the contrast ratio is lower.
With such a configuration, when it is determined that the contrast ratio is low, the gain can be adjusted so that the luminance after conversion does not become too low for pixel data having a predetermined luminance or less. Therefore, it is possible to generate a sharp output image with no black floating while limiting the low luminance portion of the input image so as not to be too dark.

[Embodiment 8] Furthermore, the image processing apparatus of Embodiment 8 is the image processing apparatus of any one of Embodiments 5 to 7,
The luminance distribution information generation means divides the corresponding luminance gradation range of the image data into three independent ranges of a high luminance side, an intermediate luminance, and a low luminance side for each of the at least one image data Are divided into one or more division ranges, and the number of pixels belonging to each division range is counted to generate the luminance distribution information.
With such a configuration, the number of pixels on the high luminance side and the number of pixels on the low luminance side can be easily grasped, and the contrast ratio of the captured image can be determined easily and at high speed.

[Embodiment 9] Furthermore, the image processing apparatus of Embodiment 9 is the image processing apparatus of Embodiment 8,
The conversion gain control unit is configured such that the total number of pixels in the divided range belonging to the high luminance side range is equal to or greater than a first threshold value and the total number of pixels in the divided range belonging to the low luminance side range is equal to or larger than a second threshold In addition, it is determined that the contrast ratio of the captured image is high.
With such a configuration, a reference for determining that the contrast ratio is high can be easily set by the first threshold value and the second threshold value.

[Mode 10] Furthermore, the image processing apparatus of mode 10 is the image processing apparatus of mode 8 or 9,
The conversion gain control means determines that the contrast ratio of the captured image is low when the total number of pixels in the divided range belonging to the range on the high luminance side is equal to or smaller than a third threshold value.
With such a configuration, a reference for determining that the contrast ratio is low can be easily set by the third threshold.

[Mode 11] Furthermore, an image processing apparatus according to mode 11 is the image processing apparatus according to any one of modes 1 to 10,
Two or more pieces of image data including image data corresponding to the longest exposure time among a plurality of pieces of image data obtained by imaging a subject with a plurality of different exposure times from the image pickup device capable of picking up the subject. Image data acquisition means for acquiring;
A synthesized image data generating unit that linearly synthesizes the two or more image data acquired by the image data acquiring unit based on an exposure time ratio to generate the first synthesized image data;
The luminance distribution information generation unit generates the luminance distribution information based on image data corresponding to at least one exposure time out of the two or more pieces of image data excluding the longest exposure time.

With such a configuration, the image data acquisition unit includes image data corresponding to the longest exposure time among a plurality of image data obtained by imaging the subject with a plurality of different exposure times from the image sensor. The above image data can be acquired.
Furthermore, the composite image data generating means can generate the first composite image data by linearly combining the acquired two or more pieces of image data based on the exposure amount ratio.
The luminance distribution information generation means can generate the luminance distribution information based on image data corresponding to at least one exposure time out of the two or more pieces of image data excluding the longest exposure time. .

[Mode 12] Furthermore, the image processing apparatus of mode 12 is the image processing apparatus of mode 11,
The image sensor can read out a pixel signal by a nondestructive readout method that reads out a pixel signal corresponding to the accumulated charge while maintaining the accumulated charge from each pixel, and in order from the shorter exposure time in each frame period. The pixels are exposed at the different exposure times, pixel signals are read from the exposed pixels by the non-destructive readout method, and pixel signal data constituting the plurality of image data is output in the read order. ,
The image data acquisition means acquires the at least one image data prior to acquiring image data corresponding to the longest exposure time in each frame period;
The luminance distribution information generation means generates the luminance distribution information based on the preceding image data which is the at least one image data acquired in advance.
The conversion gain control means linearly synthesizes the two or more pieces of image data acquired in the same frame period as the frame period from which the preceding image data was acquired based on the luminance distribution information generated based on the preceding image data. Controlling the gain for converting the luminance of the generated first composite image data;
The image conversion means converts the luminance of the first composite image data corresponding to the same frame period using the controlled gain corresponding to the same frame period.

With this configuration, the pixel signal data corresponding to one exposure time shorter than this is obtained from the image sensor prior to the pixel signal data corresponding to the longest exposure time, and the luminance distribution information is obtained. Can be generated.
Further, the conversion gain control means is a first synthesized image generated by linearly synthesizing two or more image data acquired in the same frame as the image data used to generate this information based on the generated luminance distribution information. The gain for converting the brightness of the data can be controlled.

Therefore, the frame of the image data that is the generation target of the luminance distribution information can be made the same as the frame of the image data that is used to generate the first composite image data whose gain is controlled based on the luminance distribution information. As a result, it is possible to suppress the occurrence of an oscillation (flicker) phenomenon corresponding to the luminance fluctuation of the subject when the frame is different. Therefore, a stable output image can always be obtained.
Also, for example, in applications that require real-time performance, such as in-vehicle cameras and machine vision, acquisition of composite images (video) that have been tone-converted according to the tone of the output device at a timing close to real time It is effective to do.

[Mode 13] On the other hand, in order to achieve the above object, an image processing method according to mode 13 includes:
Among a plurality of image data obtained by imaging a subject with a plurality of different exposure times, the brightness of the first composite image data generated by combining two or more image data is converted to have a different gradation characteristic. An image conversion step for generating two composite image data;
A luminance distribution information generation step for generating luminance distribution information, which is information relating to the luminance distribution of the captured image of the subject, based on at least one of the plurality of image data;
A conversion gain control step for controlling a characteristic of a gain used for conversion of the luminance of the image conversion unit based on the luminance distribution information generated in the luminance information generation step,
In the image conversion step, the luminance of the first composite image data is converted using the gain controlled in the conversion gain control step.
Thereby, the same operation and effect as those of the image processing apparatus of aspect 1 can be obtained.

[Mode 14] In order to achieve the above object, an imaging apparatus according to mode 14
An image sensor capable of imaging a subject;
An image processing apparatus according to the eleventh or twelfth aspect.
With such a configuration, operations and effects equivalent to those of the image processing apparatus according to mode 11 or 12 can be obtained.

1 is a block diagram illustrating a configuration of an imaging apparatus 1 according to the present invention. 2 is a block diagram showing an internal configuration of an HDR image sensor 10. FIG. It is a figure which shows the structure of the color filter array 10c. It is a figure which shows an example of the exposure for every line of each pixel in the sensor cell array of HDR sensor 10d, and the read-out operation of a pixel signal. It is a figure which shows an example of the reset timing of stored charge, and the read-out timing of the pixel signal of each exposure time. It is a figure which shows an example of the output format of the pixel data from HDR sensor 10d. 2 is a block diagram illustrating an example of an internal configuration of an image processing device 20. FIG. 3 is a block diagram illustrating an example of an internal configuration of a photometry unit 25 and an HDR main line processing unit 26. FIG. 3 is a block diagram illustrating an example of an internal configuration of an HDR linear processing unit 30 and a level correction processing unit 31. FIG. It is a block diagram which shows an example of an internal structure of the synthetic | combination process part 30a. 3 is a block diagram illustrating an example of an internal configuration of a luminance distribution information generation unit 40. FIG. 3 is a block diagram illustrating an example of an internal configuration of a correction gain calculation unit 51. FIG. It is a figure which shows an example of the output timing of the pixel data in each sub-frame. It is a figure which shows an example of the conversion LUT of 1st Embodiment. It is a block diagram which shows an example of an internal structure of correction | amendment gain calculation part 51 '. It is a figure which shows an example of the conversion LUT and limit value of 2nd Embodiment.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. 1 to 14 are diagrams illustrating a first embodiment of an image processing apparatus, an image processing method, an image processing program, and an imaging apparatus according to the present invention.
(Configuration example of imaging device)
First, the configuration of an imaging apparatus according to the present invention will be described with reference to FIGS. FIG. 1 is a block diagram showing a configuration of an imaging apparatus 1 according to the present invention. FIG. 2 is a block diagram showing an internal configuration of the HDR image sensor 10.
As illustrated in FIG. 1, the imaging device 1 includes an HDR imaging device 10 and an image processing device 20.

As shown in FIG. 2, the image sensor 10 includes a lens 10a, a microlens 10b, a color filter array 10c, an HDR sensor 10d, a drive circuit 10e, and a readout circuit 10f.
The lens 10a collects reflected light from the subject and guides it to the microlens 10b. There are types such as a single focus lens, a zoom lens, and an auto iris lens depending on the imaging conditions.
The microlens 10b collects light transmitted through the lens 10a on each sensor cell (pixel) of the sensor cell array included in the HDR sensor 10d.

The color filter array 10c separates light having a wavelength corresponding to one predetermined type of color element from light transmitted through the microlens 10b and makes the separated light incident on each corresponding pixel (hereinafter referred to as CF). Part) and at least the number of pixels.
The HDR sensor 10d has a sensor cell array in which each sensor cell (pixel) includes a photodiode and a CMOS element, and outputs a plurality of types of image data having different exposure times by controlling the exposure time by an electronic shutter method. It is.
The drive circuit 10e drives the HDR sensor 10d and the readout circuit 10f based on a drive mode set by a register (not shown) and a synchronization signal from the image processing device 20. In the present embodiment, driving is performed so that a subject is imaged at different exposure times, and a plurality of types of captured image data having different exposure times are output.

The reading circuit 10f reads a captured image signal from the HDR sensor 10d, performs various signal processing including A / D conversion on the read captured image signal (analog), and outputs the signal to the image processing device 20 as digital captured image data. To do. At this time, the readout circuit 10f outputs captured image data in synchronization with a synchronization signal from the image processing device 20 supplied via the drive circuit 10e.
The image processing device 20 performs noise removal processing, composition processing, color processing, luminance image data generation processing, luminance value → gain conversion processing based on a plurality of types of captured image data output from the HDR imaging element 10 and having different exposure times. , Level conversion (tone mapping) processing, γ conversion processing, and the like. A detailed configuration will be described later.

(Color filter array configuration example)
Next, the configuration of the color filter array 10c will be described with reference to FIG.
Here, FIG. 3 is a diagram showing a configuration of the color filter array 10c.
In FIG. 3, R, G, and B correspond to CF portions that selectively transmit light in the wavelength region of any one of the three primary colors (red, green, and blue), and are attached to these. The attached numbers indicate row numbers and column numbers. For example, if R _00, a filter portion that transmits light in the wavelength region of R corresponding to the first column of the pixel line number 1. Note that in the present embodiment, the horizontal direction is the row direction, and the vertical direction is the column direction, and the imaging apparatus 2 performs processing in units of lines composed of a plurality of pixels arranged in the horizontal direction. .

  Specifically, the color filter array 10c (hereinafter referred to as the CF array 10c) converts the light incident through the microlens 10b into light in a wavelength region corresponding to red (hereinafter referred to as R light), green. Separating light in a wavelength region corresponding to a predetermined color from light in a corresponding wavelength region (hereinafter referred to as G light) and light in a wavelength region corresponding to blue (hereinafter referred to as B light); It is composed of a plurality of CF sections that respectively input separated lights to corresponding pixels.

  More specifically, as shown in FIG. 3, the CF array 10 c separates R light from incident light, and a plurality of R light transmission filter units (R in FIG. 3) that enters the separated R light into pixels. ) And a plurality of G light transmission filter portions (G in FIG. 3) that separate the G light from the incident light and enter the separated G light into the pixel R → G → R → G → R → G → A plurality of filter lines RGFL having a configuration that is continuous in the horizontal direction in this order are provided.

  Further, a plurality of G light transmission filter sections and a plurality of B light transmission filter sections (B in FIG. 3) for separating the B light from the incident light and entering the separated B light into the pixels are G → B. A plurality of filter lines GBFL having a configuration that is continuous in the horizontal direction in the order of G → B → G → B →. The plurality of filter lines RGFL and filter lines GBFL are continuously arranged in the vertical direction in the order of RGFL → GBFL → RGFL → GBFL → RGFL →... That is, a plurality of G light transmission filter sections, a plurality of B light transmission filter sections, and a plurality of R light transmission filter sections are arranged in a Bayer shape.

(About the drive control method of the HDR sensor 10d)
Next, a method for controlling the exposure time of the HDR sensor 10d of the image sensor 10 and a method for reading a pixel signal from the sensor cell array will be described with reference to FIGS. Here, FIG. 4 is a diagram illustrating an example of exposure and pixel signal readout operations for each pixel line in the sensor cell array of the HDR sensor 10d. FIG. 5 is a diagram showing an example of the reset timing of the accumulated charge and the readout timing of the pixel signal for each exposure time. FIG. 6 is a diagram illustrating an example of an output format of pixel data from the HDR sensor 10d.

Here, the exposure time of the present invention is controlled by a nondestructive readout line L that performs nondestructive readout of the pixel signal of the ultrashort exposure time T_S1 with respect to the exposure region (scan region) of the sensor cell array.
1 and a non-destructive readout line L2 for non-destructive readout of a pixel signal having a short exposure time T_S2
And set. Further, a read & reset line L3 for resetting the accumulated charge of each pixel line and reading the pixel signal of the standard exposure time T_S3 is set. T_S1 ~
As shown in FIG. 5, the relationship of T_S3 is “T_S1 <T_S2 <T_S3”, and after reset, first, the non-destructive read line L1 is set when T_S1 has elapsed. Next, the non-destructive read line L2 is set when T_S2 elapses, and the read & reset line L3 is set when T_S3 elapses.

Specifically, the non-destructive read lines L1 and L2 and the read & reset line L3 are read and reset when charges corresponding to the standard exposure time T_S3 are sequentially accumulated in the pixel lines in the exposure region, as shown in FIG. The line L3 is set so as to sequentially read out pixel signals of the lines of the respective pixels and to sequentially reset the accumulated charges. On the other hand, in the line of each pixel after resetting the exposure area, the pixel signal of the line of each pixel is obtained during the ultrashort exposure time T_S1 and the short exposure time T_S2 during the period in which the charge for the standard exposure time T_S3 is accumulated. Non-destructive read lines L1 and L2 are set to sequentially read non-destructively.

In the present embodiment, as shown in FIG. 4, the ultrashort exposure time T_ immediately after the reset
The pixel signal (analog data) S1 corresponding to S1 is read to the first line memory, and the pixel signal (analog data) S2 corresponding to the short exposure time T_S2 is read to the second line memory. Further, a pixel signal (analog data) S corresponding to the standard exposure time T_S3.
3 is read into the third line memory. Then, as shown in FIG. 3, the read pixel signals S1 to S3 are sequentially output to the ADC in the order of S1 to S3 through the selection circuits, and are converted into digital data (pixel data) there. The converted pixel data is output to the image processing device 20 in line units in the order of conversion (in order of S1 to S3).

In addition, as shown in FIG. 4, the readout timing of the pixel signals on the non-destructive readout lines L1 and L2 and readout & reset line L3 is sequentially scanned for each pixel line (see FIG. 4). Scan direction in FIG. 4). In the read & reset line L3, the accumulated charge is reset, and the pixel signal of the pixel that has been exposed for the standard exposure time T_S3 immediately before the accumulated charge is reset is read. Further, immediately after the reset, the pixel signal of the ultrashort exposure time T_S1 is read out.

For example, it is assumed that the pixel signal S3 of the standard exposure time T_S3 is read and reset on the first line that is the first line of the exposure region. Thereafter, every time the pixel signal S3 is read from the third line memory to the outside, scanning of the read & reset line L3 is sequentially performed one line at a time in the scan direction in FIG. At this time, when the read & reset line L3 reaches the first line again, scanning is performed so that the standard exposure time T_S3 has just passed. In such a procedure, the pixel signal is read out and the stored charge is reset at the time of standard exposure for each pixel line for the pixel line in the exposure region of the sensor cell array.

On the other hand, when the accumulated charge is reset, the pixel signal S1 of the pixel that has been exposed for the ultrashort exposure time T_S1 in the nondestructive readout line L1 with respect to the pixel line after the reset
Of non-destructive readout, and then the short exposure time T_ in the non-destructive readout line L2
Non-destructive readout of the pixel signal S2 of the pixel subjected to the exposure of S2 is performed. According to such a procedure, the pixel signals S1 and S2 at the time of exposure are sequentially read out for each line of each pixel of the sensor cell array with the ultrashort exposure time T_S1 and the short exposure time T_S2.

  The pixel signals S1 to S3 read out in this way are stored in the first line memory to the third line memory for each line and output to the selection circuit in line units. From the selection circuit, analog pixel data S1 to S3 are output to the ADC in the order of S1 to S3. The ADC converts the analog pixel data S1 to S3 into digital pixel data S1 to S3. Then, as shown in FIG. 6, the pixel data is sequentially output from the ADC to the image processing apparatus 20 in the order of S <b> 1 to S <b> 3 in units of lines. That is, as shown in FIG. 6, the pixel data S <b> 1 to S <b> 2 are input to the image processing device 20 in the order of S <b> 1 to S <b> 2 prior to the pixel data S <b> 3.

(Configuration example of image processing apparatus)
Next, a detailed configuration of the image processing apparatus 20 will be described with reference to FIGS.
Here, FIG. 7 is a block diagram illustrating an example of an internal configuration of the image processing apparatus 20. FIG. 8 is a block diagram illustrating an example of the internal configuration of the photometry unit 25 and the HDR main line processing unit 26. FIG. 9 is a block diagram illustrating an example of an internal configuration of the HDR linear processing unit 30 and the level correction processing unit 31. FIG. 10 is a block diagram illustrating an example of an internal configuration of the composition processing unit 30a. FIG. 11 is a block diagram illustrating an example of an internal configuration of the luminance distribution information generation unit 40. FIG. 12 is a block diagram illustrating an example of an internal configuration of the correction gain calculation unit 51.

As shown in FIG. 7, the image processing apparatus 20 includes a preprocessing unit 21, delay units 22 to 24, a photometry unit 25, and an HDR main line processing unit 26.
The pre-process unit 21 performs fixed pattern noise removal processing, clamping processing, and the like using the pixel data S1 of the ultrashort exposure time T_S1 on the pixel signals (pixel data S1 to S3) from the HDR sensor 10d.

Specifically, the fixed pattern noise removal process is a process of subtracting the pixel data of the ultrashort exposure time T_S1 for each corresponding pixel from the pixel data of the short exposure time T_S2 and the standard exposure time T_S3. That is, since the pixel data of the ultra-short exposure time T_S1 is data corresponding to the pixel signal immediately after the reset, the amount of accumulated charge is small and governed by the component of the fixed pattern noise. By subtracting from the exposure time data, only the fixed pattern noise component can be removed.
The clamp process receives the pixel data S1 to S3 from the HDR sensor 10d, detects whether the signal is a signal in the light shielding area, and if the signal is detected as the light shielding area, the signal level is the black (reference) level. Thus, the DC component of all input pixel data is clamped.

The pixel data S <b> 1 to S <b> 3 that have undergone the fixed pattern noise removal process and the clamp process are output to the HDR main line processing unit 26 in synchronism with the output timing via the delay units 22 to 24. Hereinafter, the captured image data (pixel data for one frame) respectively corresponding to the exposure times T_S1 to T_S3 are referred to as captured image data S1 to S3, similarly to the pixel data S1 to S3.
The delay units 22 to 24 have delay elements that delay the pixel data S1 to S3 output in order according to the length of the exposure time by the delay amounts d_S1 to d_S3, respectively. And it has the function to synchronize the output timing of pixel data S1-S3.
Specifically, the delay element of the delay unit 22 delays the pixel data S3 by the delay amount d_S3, the delay element of the delay unit 23 delays the pixel data S2 by the delay amount d_S2, and the delay element of the delay unit 24 is the pixel data S1. Is delayed by a delay amount d_S3 (d_S3 <d_S2 <d_S1).

The output delay time of the pixel data S2 and S3 with respect to the pixel data S1 is simply set to dT2 (T_
S2-T_S1) and dT3 (T_S3-T_S1), and delay amounts for correcting other processing delays are Δt1, Δt2, and Δt3. In this case, “(d_S3 = Δt3 + dT3) =
The relationship (d_S2 = Δt2 + dT2) = (d_S1 = Δt1) ”holds.
The photometry unit 25 performs a histogram generation process and a level correction control parameter generation process based on the captured image data S1 or S2 output from the preprocessing unit 21.

  The HDR main line processing unit 26 performs HDR synthesizing processing and tone map processing based on the captured image data S1 to S3 input in synchronization via the delay units 22 to 24 and the level correction control parameter from the photometry unit 25. , Color processing and gamma conversion processing. Then, the image data signal after the gamma conversion processing is output as a video signal. The video signal is an image signal compressed in an 8-bit range, for example, according to an industry standard.

(Configuration example of photometry unit 25)
As shown in FIG. 8, the photometry unit 25 includes a luminance distribution information generation unit 40 that generates a luminance histogram of a captured image based on the pixel data S1 or S2, and a control parameter that generates a level correction control parameter based on the generated histogram. The generator 41 is included.
As shown in FIG. 8, the HDR main line processing unit 26 generates an HDR linear image processing unit 30 that generates HDR color image data based on the pixel data S1 to S3, and converts the HDR color image data into an industry standard (standard) range (standard). And a level correction processing unit 31 that outputs an HDR video signal.

(Configuration example of HDR linear processing unit 30)
First, based on FIG.9 and FIG.10, the detailed structure of the HDR linear process part 30 shown in FIG. 8 is demonstrated.
As shown in FIG. 9, the HDR linear processing unit 30 includes a synthesis processing unit 30a and a color processing unit 30b.
The combining processing unit 30a combines the signals of the pixel data S1, S2, and S3 corresponding to the three types of exposure times T_S1, T_S2, and T_S3 based on the ratio of the exposure times, and generates HDR_RAW.
Pixel data is generated.
Specifically, as illustrated in FIG. 10, the synthesis processing unit 30a includes normalization gain calculation units 70 and 71 that calculate gains for normalization, and normalization that normalizes pixel data using the normalization gains. Parts 72 and 73.

The normalization gain calculation unit 70 calculates a normalization gain RS3 corresponding to the ratio between the exposure times T_S3 and T_S1. For example, when normalization is performed based on T_S1, “RS3 = T_S1 / T_S3” is calculated.
The normalization gain calculation unit 71 calculates a normalization gain RS2 corresponding to the ratio between the exposure times T_S2 and T_S1. For example, when normalization is performed based on T_S1, “RS2 = T_S1 / T_S2” is calculated.
The normalizing unit 72 multiplies the pixel data S3 by the normalization gain RS3, normalizes the pixel data S3, and outputs normalized pixel data NS3.

The normalizing unit 73 multiplies the pixel data S2 by the normalization gain RS2, normalizes the pixel data S2, and outputs normalized pixel data NS2.
The composition processing unit 30a further weights the normalized pixel data NS1, NS2, NS3 using the composition weights W1, W2, W3 and the composition weights W1, W2, W3. Weighting units 75, 76, 77.

The combination weight calculation unit 74 calculates combination weights W1, W2, and W3 based on the pixel data S1, S2, and S3. Specifically, the composite weight is calculated using a weight function F (x) represented by the following formula (1), where the pixel data S1 to S3 are elements x.

F (x) = (x / MAX) ⁿ (1)

In the above equation (1), MAX is the maximum value of the gradation range of the pixel data S1, S2, S3, for example, “MAX = 255” if the gradation range is 8 bits, and the gradation range of 10 bits. For example, “MAX = 1023”.

Using the above equation (1), W1 to W3 can be calculated by, for example, the following equations (2) to (4).

W1 = F (S2) (2)
W2 = F (S3) -W1 = F (S3) -F (S2) (3)
W3 = 1-W2-W1 = 1-F (S3) (4)

The weighting unit 75 multiplies the normalized pixel data NS3 by the composite weight W3 and outputs the weighted pixel data WS3.
The weighting unit 76 multiplies the normalized pixel data NS2 by the composite weight W2 and outputs the weighted pixel data WS2.
The weighting unit 77 multiplies the pixel data S1 by the composite weight W1 and outputs weighted pixel data WS1.
The combination processing unit 30a further includes a combining unit 78 that combines the weighted pixel data WS1, WS2, and WS3 to generate HDR_RAW pixel data and outputs the generated HDR_RAW pixel data to the color processing unit 30b.

For example, the combining unit 78 combines the weighted pixel data WS1, WS2, and WS3 by combining the weighted pixel data WS1, WS2, and WS3, assuming that the pixel data S1 to S3 are each 10-bit data.
W pixel data is generated. Note that HDR_RAW pixel data for one frame becomes HDR image data (HDR_RAW image data).
The color processing unit 30b has a line memory (not shown), HDR_RAW pixel data (pixel data to be processed), HDR_RAW pixel data of pixels around the pixel to be processed, stored in the line memory, and Is used to perform color interpolation processing. That is, using the HDR_RAW pixel data delayed by the line memory, for each point of the image, RGB
Processing for generating a color signal (data) defined in the color space (color signal processing) is performed.

The color processing unit 30b also performs sharpness processing that corrects the brightness difference and the hue difference of the contour portion of the captured image to clarify the contour portion of the captured image.
Through the color interpolation process, the HDR_RAW pixel data is converted into HDR color pixel data corresponding to each color element of RGB for each pixel.
Thereby, for each pixel, pixel data R_hdr, G (green) corresponding to the R (red) color element
Pixel data G_hdr corresponding to the color element of B and pixel data B_ corresponding to the color element of B (blue)
HDR color pixel data having hdr is generated. In addition, HDR color image data is comprised from HDR color pixel data R_hdr, G_hdr, and B_hdr corresponding to the pixel data S1 to S3 for one frame.

(Configuration example of luminance distribution information generation unit 40)
Next, a detailed configuration of the luminance distribution information generation unit 40 shown in FIG. 8 will be described based on FIG.
As shown in FIG. 11, the luminance distribution information generation unit 40 includes a selector 60 and a histogram generator 61.
The selector 60 has a function of selectively switching the pixel data input to the histogram generator 61 to one of the pixel data S1 and S2 in accordance with the switching signal. This switching is performed according to a preset mode or automatically according to shooting conditions.

As shown in FIG. 11, the histogram generator 61 includes a level divider 62, counters 63a to 63e, and output registers 64a to 64e.
The level divider 62 determines whether the gradation level of the input pixel data S1 or S2 belongs to each of the division ranges obtained by dividing the gradation range (for example, 0 to 1023) into a plurality of ranges. . And a count pulse is input into the counter corresponding to the division range of the determination result among the counters 63a to 63e.

Each of the counters 63a to 63e counts (counts) the count pulse input from the level divider 62, and outputs the count value to the output register having the same letter at the end of the sign among the output registers 64a to 64e. To do. In addition, the counters 63a to 63e reset (for example, initialize to 0) the count value in response to an external RESET signal.
Each of the output registers 64a to 64e stores a count value input from the counter having the same letter at the end of the code among the counters 63a to 63e, and controls the stored count value according to an external HOLD signal. Output to the parameter generator 41.

In the present embodiment, the gradation range of the pixel data S1 or S2 is divided into the following five division ranges, where the signal level of the pixel data signal is DATA and the maximum value of the signal level range is MAX.
(1) “Saturation: MAX = DATA”
(2) “High Range: (MAX + 1) / 2 ≦ DATA <MAX”
(3) “Middle Range: (MAX + 1) / 8 ≦ DATA <(MAX + 1) / 2”
(4) “Low Range: (MAX + 1) / 64 ≦ DATA <(MAX + 1) / 8”
(5) “Bottom Range: 0 ≦ DATA <(MAX + 1) / 64”
The division ranges (1) to (5), the counters 63a to 63e, and the output registers 64a to 64e correspond to the division range (1), with the counter 63a and the output register 64a corresponding to the division range (2). Counter 63b and output register 64b correspond.

Further, the counter 63c and the output register 64c correspond to the segment range (3), the counter 63d and the output register 64d correspond to the segment range (4), and the counter 63e and the output to the segment range (5). Register 64e corresponds.
That is, it is determined whether the input pixel data (DATA) belongs to any one of the above-described division ranges (1) to (5), and the pixel data belonging to each division range is counted by the counters 63a to 63e (one frame). Count the period). And a histogram is produced | generated from the pixel count of each division range of division range (1)-(5) with respect to each pixel data of captured image data S1 or S2.

It is desirable that the ultrashort exposure time T_S1 is 1/100 to 1/1000 of the standard exposure time T_S3, and the short exposure time T_S2 is 1/10 to 1/100 of the standard exposure time T_S3. In other words, it is desirable that the exposure amount (accumulated charge amount) S1 by exposure during the ultrashort exposure time T_S1 is 1/100 to 1/1000 of the exposure amount S3 by exposure during the standard exposure time T_S3. Further, it is desirable that the exposure amount S2 by the exposure with the short exposure time T_S2 is set to 1/10 to 1/100 of the exposure amount S3 by the exposure with the standard exposure time T_S3.

  In the present embodiment, in the generation of the histogram, one of the pixel data S1 and S2 can be selected as input data. However, it is desirable to use S2 rather than S1. The reason is that in S1, a histogram can be generated with high accuracy in the high luminance region (bright portion), but the low luminance region (dark portion) may be crushed. On the other hand, if S3 is used, a histogram can be generated with high accuracy in the low luminance region (dark portion), but the high luminance region (bright portion) may be saturated. For these reasons, it is desirable to generate (photometric) a histogram by S2.

(Regarding the control parameter generator 41)
Next, the control parameter generator 41 shown in FIG. 8 will be described.
The control parameter generation unit 41 controls the level correction processing unit 31 based on the photometry result (histogram) from the luminance distribution information generation unit 40 to control the correction content of the gradation level of the HDR color image data. Is calculated.
In the present embodiment, the level of the contrast ratio is determined from the histogram, and a parameter that specifies which correction content to use from a plurality of types of correction content prepared in advance is calculated based on the determination result ( decide.

Specifically, in the histogram, a value obtained by adding the number of pixels (saturated pixel number) in the segmented range (1) and the number of pixels in the segmented range (2) is equal to or greater than the first threshold Th1 and the segmented range (5) When the number of pixels is equal to or greater than the second threshold Th2, it is determined that the contrast ratio is high (high contrast).
Here, Th1 changes according to the resolution of the captured image, and is set to 20% of the total number of pixels, for example. Th2 also changes according to the resolution of the captured image, and is set to 20% of the total number of pixels, for example.

  If it is determined that the contrast is high, a level correction control parameter that specifies the correction content for high contrast is calculated. In the present embodiment, the first and second threshold values Th1 and Th2 are prepared as stepwise numerical values between 20% to 30% of the total number of pixels, and the degree of the contrast ratio is determined stepwise. Then, as the contrast ratio is higher, a level correction control parameter for designating correction contents for suppressing (relaxing) the level increase on the low luminance side is calculated.

On the other hand, when the numerical value obtained by adding the number of pixels (saturated pixels) in the section range (1) and the number of pixels in the section range (2) is equal to or less than the third threshold Th3, it is determined that the contrast ratio is low. In the present embodiment, for example, the third threshold value Th3 is prepared as a stepwise numerical value within a range of 15 to 20% of the total number of pixels, and the degree of low contrast ratio is determined stepwise. Then, the level correction control parameter for designating the correction content that makes the level increase on the low luminance side steeper as the contrast ratio is lower is calculated.
The level correction control parameter is a parameter of a conversion formula for converting luminance information of HDR color image data into a luminance level correction gain, or a plurality of types of parameters generated in advance for various parameters using this conversion formula. Information specifying one of conversion LUTs (Look Up Table) may be used.

(Configuration example of level correction processing unit 31)
Next, returning to FIG. 9, the level correction processing unit 31 includes a line memory 50, a correction gain calculation unit 51, a multiplier 52, and a γ conversion unit 53.
The line memory 50 corrects the calculation delay in the correction gain calculation unit 51 and matches the output timing (phase) of the HDR color image data and the luminance level correction gain. That is, the line memory 50 serves as a delay element that delays the output timing by the calculation time required by the correction gain calculation unit 51.
The correction gain calculation unit 51 calculates a luminance level correction gain based on the luminance values of the processing target pixel and its surrounding pixels and the level correction control parameter from the photometry unit 25.

The multiplier 52 performs level conversion (tone mapping) by multiplying each pixel data of the HDR color image data by the luminance level correction gain. Specifically, the level conversion is an operation of expanding (broadening) the histogram uniformly by expanding the dark area image.
The γ conversion unit 53 quantizes (range-compresses) the level-converted HDR color image data into 8 bits according to a γ curve (tone curve).
Specifically, the γ converter 53 converts the HDR color after level conversion from the LUT in which the tone curve information corresponding to the displayable gradation range (for example, the industry standard gradation range) of the external display device is stored. A conversion value corresponding to the value of each pixel data of the image data is acquired. That is, γ conversion is performed by replacing the gradation value (luminance value) indicated by each pixel data with the corresponding conversion value in the tone curve.

  For example, when the gradation range of the HDR color image data is 20 bits and the displayable gradation range of the external display device is 8 bits, the luminance value of the 20-bit gradation range stored in the LUT is set to 8 A conversion value corresponding to each input luminance value is acquired from information of a tone curve to be converted into a luminance value in a bit gradation range. At this time, information indicating the displayable gradation range may be acquired from the external display device.

(Configuration Example of Correction Gain Calculation Unit 51)
Next, a detailed configuration of the correction gain calculation unit 51 will be described with reference to FIG.
As shown in FIG. 12, the correction gain calculation unit 51 includes a luminance value calculation unit 51a, a line memory 51b, and a luminance value → gain conversion unit 51c.
The luminance value calculation unit 51a generates luminance image data Y indicating the luminance of the captured image based on the luminance values of the processing target pixel of the HDR color image data and its peripheral pixels, and extracts the illumination light component from the luminance image data Y.

Specifically, the HDR color image data generated by the HDR linear processing unit 30 is converted into luminance image data Y, and the luminance image data Y is subjected to blurring processing using a low-pass filter (hereinafter referred to as LPF). . The purpose of performing this blurring process is to separate the illumination light component from the HDR color image data and obtain a luminance image of the separated illumination light component.
First, the luminance value calculation unit 51a performs HDR color pixel data R_hdr (x, y) and G_hdr (x, y) corresponding to red (R), green (G), and blue (B) constituting the HDR color image data. , B_hdr (x, y) is converted into luminance pixel data P (x, y). In addition,
Luminance image data Y is composed of luminance pixel data P (x, y) for one image. Further, (x, y) is a two-dimensional coordinate indicating the position of the pixel. For example, the upper left corner of the image is the coordinate with the origin (x, y) = (0, 0).

In the present embodiment, the luminance value calculation unit 51a performs the HDR color pixel data R_hdr (x, y), G_hdr (x, y), B_h of the HDR color image data according to the following equation (5).
dr (x, y) is converted into luminance pixel data P (x, y).

P (x, y) = 0.3 × R_hdr (x, y) + 0.6 × G_hdr (x, y) + 0.1 × B_hdr (x, y) (5)

Further, the luminance value calculation unit 51a performs a blurring process using a known Gaussian filter according to the following equation (6), and converts each luminance pixel data P (x, y) of the luminance image data Y to the illumination light component data L. Convert to (x, y).

Here, since the illumination light component is composed of components having a relatively low frequency, the illumination light component can be extracted from the luminance component by applying a low-pass filter such as a Gaussian filter.

L (x, y) = Gauss (P (x, y)) (6)

The line memory 51b is a memory that stores HDR color pixel data as much as necessary for performing the blurring process. For example, if the LPF has a size of 5 × 5, it is composed of a line memory that can store data for five rows of HDR color pixel data.

The luminance value → gain conversion unit 51c is configured to output the illumination light component data L (x, y) based on the illumination light component data L (x, y) from the luminance value calculation unit 51a and the level correction control parameter from the photometry unit 25. ) Is converted into a luminance level correction gain, and the luminance level correction gain is output to the multiplier 52.
In the present embodiment, the luminance value → gain conversion unit 51c has a plurality of types of conversion LUTs having different gain characteristics on the low luminance side corresponding to the level correction control parameters calculated according to the contrast ratio of the captured image. It has. Specifically, the conversion LUT has a characteristic in which the gain on the low luminance side is kept low as the contrast ratio is higher and gradually changes, and the gain is lower than that when the contrast ratio is lower. And has a characteristic of changing sharply.

The conversion LUT is calculated in advance according to the following equation (7), and each L (x, y) corresponding to the luminance pixel data P (x, y) in the gradation range (for example, 20 bits) of the HDR color image data. Is a data table storing a luminance level correction gain K (L (x, y)) for. In the present embodiment, a gain K (L (x, y)) for broadening the luminance histogram of the HDR color image data is stored.

K (L (x, y)) = 1 / L _N (x, y) ⁱ (7)

However, the exponent part i is a positive real number equal to or less than 1, and L _N (x, y) is a normalized illumination light component and has a value in the range of “0 to 1”. A plurality of types of conversion LUTs are generated by changing the value of i in the above equation (7) or correcting the characteristics of the low-brightness portion of the created conversion LUT according to the contrast ratio. To do. For example, when a plurality of types of conversion LUTs are generated by changing the value of i, the level correction control parameter can be set to the value of i.

(Operation example of the imaging apparatus 1)
Next, a specific operation of the imaging apparatus 1 according to the present embodiment will be described based on FIGS.
Here, FIG. 13 is a diagram illustrating an example of output timing of pixel data in each subframe. FIG. 14 is a diagram illustrating an example of the conversion LUT according to the present embodiment.
When imaging of the subject is started, the light reflected from the subject is collected by the lens 10a and is incident on the microlens 10b. Incident light from the lens 10 a
Parallelized at b and incident on each pixel of the sensor cell array via each CF portion of the CF array 10c. Since the CF array 10c has a configuration in which CF portions corresponding to the three primary colors of RGB are arranged in a Bayer arrangement, only light of color elements corresponding to each CF portion among R light, G light, and B light is applied to each pixel. It will be incident.

On the other hand, when the imaging is started, the readout and reset line L3 is set one line at a time from the start line, and the pixel signal S3 is read from each pixel of the scanned line. Reset. Note that the pixel signal S3 that is read out first is processed so as to be ignored in each component in the subsequent stage because the pixel signal S3 that has been read out is not exposed with the standard exposure time T_S3. Subsequently, for each scanning line, after resetting each pixel, the nondestructive readout line L1 is set at the elapse timing of the ultrashort exposure time T_S1, and the pixel signal S1 is read out. Subsequently, for each scanning line, after resetting each pixel, the non-destructive readout line L2 is set at the elapse timing of the short exposure time T_S2, and the pixel signal S2 is read out. Then, the read & reset line L3 is set again in order from the start line, the pixel signal S3 is read from each pixel of the scanned line, and then the accumulated charge of each pixel is reset. The pixel signal S3 read at this time and the pixel signals S1 and S2 read before the pixel signal S3 are to be processed in each component in the subsequent stage.
Thereafter, while imaging is being performed, the above-described setting of L1 to L3, readout of the pixel signals S1 to S3, and reset processing are repeatedly performed.

  The pixel signals S1 to S3 read out in this way are stored in the first line memory to the third line memory of the reading circuit 10f for each line and output to the selection circuit in units of lines. From the selection circuit, analog pixel data S1 to S3 are output to the ADC in the order of S1 to S3. The ADC converts the analog pixel data S1 to S3 into digital pixel data S1 to S3. Then, the pixel data is sequentially output from the ADC to the image processing apparatus 20 in the order of S1 to S3 in units of lines. That is, as shown in FIG. 13, the pixel data S <b> 1 to S <b> 2 are input to the image processing device 20 in the order of S <b> 1 to S <b> 2 prior to the pixel data S <b> 3.

On the other hand, when the image processing apparatus 20 receives the pixel data S1 to S3 from the HDR sensor 10d in order from S1, the preprocessing unit 21 performs a fixed pattern noise removal process and a clamp on the pixel data in the order received. Apply processing. Then, among the pixel data S1 to S3 subjected to these processes, S1 and S2 are output to the photometry unit 25 as they are, and the processed pixel data S1 to S3 are output to the delay units 22 to 24.
The delay units 22 to 24 absorb the time difference of reception of the pixel data S1 to S3 by the preprocessing unit 21, synchronize the output timing of the pixel data S1 to S3, and output them to the HDR main line processing unit 26.

On the other hand, when the pixel data S1 and S2 are input from the preprocessing unit 21, the photometry unit 32 selectively selects one of S1 and S2 as the level divider 62 in the selector 60 of the luminance distribution information generation unit 40. input. Here, the pixel data S2 is input to the level divider 62.
The level divider 62 compares the luminance value (DATA) of the input pixel data S2 with “(MAX + 1) / 2”. For example, if “DATA> (MAX + 1) / 2”, then: Compare MAX with DATA. Accordingly, if “DATA = MAX”, it is determined that the input pixel data S2 belongs to the division range (1), and a count pulse is input to the counter 63a. If “DATA <MAX”, it is determined that the input pixel data S2 belongs to the segment range (2), and a count pulse is input to the counter 63b.

If “DATA> (MAX + 1) / 2”, then DATA and “(MAX + 1)
) / 8 ". As a result, if “DATA> (MAX + 1) / 8”, it is determined that the input pixel data S2 belongs to the divided range (3), and a count pulse is input to the counter 63c.
If “DATA <(MAX + 1) / 8”, then DATA is compared with “(MAX + 1) / 64”. As a result, if “DATA> (MAX + 1) / 64”, it is determined that the input pixel data S2 belongs to the segment range (4), and a count pulse is input to the counter 63d.

If “DATA <(MAX + 1) / 64”, it is determined that the input pixel data S2 belongs to the segment range (5), and a count pulse is input to the counter 63e.
Further, the level divider 62 sets “DATA = (MAX + 1) / 2” to the section range (2), “DATA = (MAX + 1) / 8” to the section range (3), and “DATA = ( In the case of “MAX + 1) / 64”, it is determined that the input pixel data S2 belongs to the segment range (4).
The counters 63a to 63e count the count pulses from the level divider 62, and the count value is overwritten and stored in the output registers 64a to 64e every time it counts up.

When such photometric processing is repeatedly performed on the pixel data S2 for one frame period, the RESET signal and the HOLD signal from the control circuit (not shown) become active, the count values of the counters 63a to 63e are reset, and the output register The count value (number of pixels) before reset stored in 64a to 64e is output to the control parameter generating unit 41. The count values (number of pixels) stored in the output registers 64a to 64e constitute a luminance histogram of the captured image.
When receiving the photometric result (histogram) from the luminance distribution information generation unit 40, the control parameter generation unit 41 first calculates the total pixel number HP of the divided ranges (1) and (2), and calculates the total pixel number HP. The first threshold Th1 is compared. Further, the number of pixels LP in the section range (5) is compared with the second threshold Th2.

Here, Th1 is set by two numerical values, for example, Th1L of 20% of the total number of pixels and Th1H of 26% (if the total number of pixels is 100, for example, Th1L = 20, Th1H = 26 Become). Furthermore, it is assumed that the second threshold Th2 is set with two numerical values, for example, Th2L of 20% of the total number of pixels and Th2H of 26%.
The control parameter generation unit 41 determines that the contrast ratio is high when the comparison results are “HP ≧ Th1H” and “LP ≧ Th2H”. Then, a level correction control parameter for selecting a conversion LUT that tabulates the conversion characteristics of Lr1 that draws the gentlest curve among the plurality of conversion curves Lr1 to Lr5 shown in FIG. 14 is calculated.

If the comparison result includes “HP ≧ Th1L”, it is determined that the contrast ratio is slightly high. Then, a level correction control parameter for selecting a conversion LUT that tabulates the conversion characteristics of Lr2 that draws the second gentlest curve among the conversion curves Lr1 to Lr5 shown in FIG. 14 is calculated.
If the number of pixels LH or LP does not satisfy any of the above conditions, the control parameter generation unit 41 then compares the total number of pixels HP with the third threshold Th3. Here, it is assumed that Th3 is set with two numerical values, Th3H of 20% of the total number of pixels and Th3L of 15% of the total number of pixels.
The control parameter generation unit 41 determines that the contrast ratio is low when the comparison result is “HP ≦ Th3L”. Then, the level correction control parameter for selecting the conversion LUT that tabulates the conversion characteristics of Lr5 that draws the curve with the steepest change among the plurality of conversion curves Lr1 to Lr5 shown in FIG. 14 is calculated.

Further, when the comparison result is “Th3H>HP> Th3L”, it is determined that the contrast ratio is slightly low. Then, the level correction control parameter for selecting the conversion LUT that tabulates the conversion characteristics of Lr4 that draws the second steepest curve among the plurality of conversion curves Lr1 to Lr5 shown in FIG. 14 is calculated.
When none of the above determination results is satisfied, the control parameter generation unit 41 displays the conversion characteristics of Lr3 that draws an intermediate change curve among the plurality of conversion curves Lr1 to Lr5 shown in FIG. A level correction control parameter for selecting the converted conversion LUT is calculated.

Then, the control parameter generation unit 41 outputs the calculated level correction control parameter to the brightness value → gain conversion unit 51c of the correction gain calculation unit 51.
When the HDR linear processing unit 30 receives the pixel data S1 to S3 delayed by the delay units 22 to 24 and synchronized in output timing in parallel with the processing of the photometry unit 25, first, the synthesis processing unit 30a. In the above, HDR_RAW pixel data is generated.
Specifically, the synthesis processing unit 30a calculates the normalized gains RS3 and RS2 in the normalized gain calculation units 70 and 71 from the information of the exposure times T_S1 to T_S3 held in the memory or acquired from the HDR imaging device 10.

In parallel, the composite weight calculator 74 calculates composite weights W1 to W3 from the pixel data S1 to S3 using the above equations (1) to (4).
Next, the normalizing units 72 and 73 multiply the pixel data S3 and S2 by the normalization gains RS3 and RS2, respectively, thereby calculating the normalized pixel data NS3 and NS2.
Next, in the weighting units 75 to 77, the normalized pixel data NS3 is multiplied by the composite weight W3, the normalized pixel data NS2 is multiplied by the composite weight W2, and the pixel data S1 is multiplied by the composite weight W1 to obtain weighted pixels. Data WS1, WS2 and WS3 are generated.

Next, the combining unit 78 combines (adds) the weighted pixel data WS1, WS2, and WS3 to generate HDR_RAW pixel data, and outputs the generated HDR_RAW pixel data to the color processing unit 30b.
When the HDR_RAW pixel data is input from the composition processing unit 30a, the color processing unit 30b accumulates the input HDR_RAW pixel data in the line memory. When a certain amount of HDR_RAW pixel data is accumulated in the line memory, color correction processing and sharpness processing are performed to generate HDR color pixel data. Then, the generated HDR color pixel data is output to the line memory 50 of the level correction processing unit 31 and the correction gain calculation unit 51, respectively.
When the line memory 50 receives the HDR color pixel data from the color processing unit 30b, the line memory 50 accumulates the data and delays the output timing. Specifically, the output timing of the luminance level correction gain from the correction gain calculation unit 51 is delayed so that the output timing of the HDR color pixel data corresponding to the gain is synchronized.

On the other hand, when the correction gain calculation unit 51 receives the HDR color pixel data, the luminance value calculation unit 51a converts the received HDR color pixel data into the luminance pixel data P (x, y) based on the above equation (5). And the converted luminance pixel data P (x, y) is stored in the line memory 51b.
When the luminance pixel data P (x, y) necessary for the blurring process is accumulated in the line memory 51b, the blurring process using the Gaussian filter of the above equation (6) is executed, and the luminance pixel data P (X, y) is converted into illumination light component data L (x, y).

Then, the illumination light component data L (x, y) is output to the luminance value → gain conversion unit 51c.
The luminance value → gain conversion unit 51c is configured to store a plurality of types of conversion LUTs based on the level correction control parameter from the control parameter generation unit 41 and the illumination light component data L (x, y) from the luminance value calculation unit 51a. The LUT used for conversion is selected from among them.
Specifically, the conversion LUT corresponding to the level correction control parameter is selected from a plurality of types of conversion LUTs in which the conversion characteristics of the conversion curves Lr1 to Lr5 shown in FIG. Then, the luminance level correction gain corresponding to the illumination light component data L (x, y) is acquired from the selected LUT, and is output to the multiplier 52.
At this time, the line memory 50 outputs the HDR color pixel data corresponding to the gain to the multiplier 52 in synchronization with the output timing of the luminance level correction gain of the correction gain calculation unit 51.

The multiplier 52 multiplies the HDR color pixel data input from the line memory 50 by the luminance level correction gain input from the correction gain calculation unit 51 to perform level conversion (tone mapping) of the HDR color pixel data.
The HDR color pixel data after level conversion is output to the γ conversion unit 53, and the γ conversion unit 53 uses the color pixel data after level conversion as an input value, for example, an LUT corresponding to a tone curve that is quantized to 8 bits. Is used to obtain a conversion value corresponding to the input value. Then, the acquired conversion value is output as an RGB video signal.

For example, in the above processing, when it is determined that the contrast ratio is high and the conversion LUT corresponding to the conversion curve Lr1 shown in FIG. 14 is selected, the gain on the dark side is lower than that of the conversion curve Lr5 or the like. Therefore, a natural image output can be obtained while maintaining the contrast ratio of the captured image.
As described above, the pixel signals S1 to S3 of each frame are input to the preprocessing unit 21 in the order of pixel data S1 → S2 → S3.

In FIG. 13, S1 frames 0 to 2, S2 frames 0 to 2, and S3 frames 0 to 2 are subframes, and the period from the start of S1 frame 0 to the end of S3 frame 0 is a period of one frame. Become.
In the HDR main line processing unit 26, as described above, processing (main line system processing) for generating HDR color image data using all the pixel data S1 to S3 is performed.
Therefore, in this embodiment, the start timing of the main line system processing is set to the timing at which the first pixel data S3 (line unit) of each subframe is input, as shown in S3 frames 0 to 2 in FIG. In each subframe, every time new pixel data S3 is input, main line processing is sequentially performed as in pipeline processing.

Further, as described above, in the photometry unit 25, a histogram is generated using the pixel data S2, and a level correction control parameter calculation process for controlling a gain for level conversion is performed based on the generated histogram.
In this embodiment, as shown in S1 frames 0 to 2 or S2 frames 0 to 2 in FIG. 13, the photometric processing start timing of the photometry unit 25 is sent from the preprocessing unit 21 to the first pixel data S1 of each subframe or S2 (line unit) is input timing. In each subframe, each time new pixel data S1 or S2 is input, photometric processing is sequentially performed as in pipeline processing.

As a result, for example, when the photometric processing is performed using the pixel data S2, a delay amount shown in FIG. 13 occurs between the end timing of the S2 frame and the start timing of the S3 frame in each subframe. Further, a time difference due to scanning shown in FIG. 13 occurs between the start timing of main line system processing and the start timing of photometric system processing in each subframe. With this time difference, it is desirable that the photometry process and the level correction control parameter calculation process are completed, and further, the luminance level correction gain calculation process in the correction gain calculation unit 51 is also completed. However, since the pixel data S1 or S2 is required for one subframe to generate the histogram, the time difference between the scans is insufficient. In addition to this, it takes time for the level correction control parameter calculation process and the luminance level correction gain calculation process. Thus, in the present embodiment, the line memory 50 delays the main line processing by these shortage times, so that the photometric result in each frame is generated from the pixel data S1 to S3 of the same frame. It can be reflected on color image data. That is, feedforward type control can be performed.

In the conventional feedback-type exposure control, the photometric result is reflected in the next frame, so that a delay of one frame or more occurs. On the other hand, in this embodiment, only the delay in the photometry unit 25 and the correction gain calculation unit 51 is required.
As described above, in the imaging device 1 of the present embodiment, the photometry unit 25 uses the pixel data S1 or S2 of the short exposure time T_S2 acquired prior to the pixel data S3 of the standard exposure time T_S3 to generate a histogram. Can be generated.

Therefore, photometric processing can be performed using an image with few saturated pixels and blackout pixels, and a highly accurate histogram can be generated.
Furthermore, it is possible to determine the level of the contrast ratio of the captured image based on the generated histogram, and to calculate a level correction control parameter for controlling the luminance level correction gain for converting the luminance level based on the determination result.

Accordingly, the contrast ratio of the captured image can be determined using a highly accurate histogram, and gain control of level conversion processing according to the level of the contrast ratio can be appropriately and automatically performed.
Further, the HDR linear processing unit 30 can generate HDR color pixel data by linearly synthesizing the pixel data S1 to S3 whose output timings are synchronized by the delay units 22 to 24 and performing color correction processing.

Further, the correction gain calculation unit 51 can generate luminance pixel data P (x, y) from the HDR color pixel data and generate illumination light component data L (x, y) from P (x, y). . Furthermore, based on the level correction control parameter, a conversion LUT corresponding to the contrast ratio of the captured image is selected from a plurality of types of conversion LUTs having different conversion characteristics, and the illumination light component data L ( A luminance level correction gain corresponding to x, y) can be acquired.
Specifically, the higher the contrast ratio, the more the conversion LUT having the characteristic of suppressing the dark side gain lower is selected.

Thereby, a natural image output can be obtained while maintaining the contrast ratio of the captured image.
Further, the multiplier 52 multiplies the HDR color pixel data output in the line memory 50 with a delay in accordance with the output timing of the brightness level correction gain and the brightness level correction gain, thereby converting the level of the HDR color pixel data. It can be performed.
Accordingly, the photometric result (histogram) in each frame can be reflected on the HDR color image data generated from the pixel data S1 to S3 of the same frame, and feedforward control can be performed. That is, since the image (frame) subject to photometry and the target image (frame) to be subjected to luminance control can be made the same, it is possible to suppress the occurrence of oscillation (flicker) phenomenon according to the luminance variation of the subject. Therefore, a stable output image can always be obtained. Further, it has an advantage that it can be easily incorporated into a system that requires real-time performance.

In addition, since the HDR color image data is subjected to level conversion (tone map) processing based on the illumination light component data L, it is possible to maintain gradation and maintain good color reproducibility in the entire dynamic range. Can be output.
Even if the image (frame) subject to photometry and the image (frame) subject to brightness control are not the same, that is, even if the result of the photometric image (frame) is reflected in the next frame, photometry Since the process is performed prior to the main line system process, a sufficient calculation time for photometry can be secured. Thereby, complicated control such as dynamic control using a histogram is possible.

[Second Embodiment]
Next, 2nd Embodiment of this invention is described based on drawing. 15 to 16 are diagrams showing a second embodiment of the image processing device, the image processing method, the image processing program, and the imaging device according to the present invention.
(Differences from the first embodiment)
In the correction gain calculation unit 51 in the image processing apparatus 20 of the first embodiment, the gain is controlled by selecting an LUT corresponding to the parameter from a plurality of types of conversion LUTs based on the level correction control parameter. It was. On the other hand, in the present embodiment, instead of the correction gain calculation unit 51, the correction gain calculation unit 51 ′ sets the luminance level correction gain used for level conversion on the low luminance side according to the parameter based on the level correction control parameter. The second embodiment is different from the first embodiment in that control is performed with a fixed value (limit value).

Accordingly, the level correction control parameter calculated by the control parameter generation unit 41 is different from the configuration for determining the limit value and the configuration of the correction gain calculation unit 51. Other configurations are the same as those described above. This is the same as in the first embodiment.
Hereinafter, the same parts as those in the first embodiment are denoted by the same reference numerals and omitted as appropriate, and different parts will be described in detail.

(Regarding the control parameter generator 41)
First, similarly to the first embodiment, the control parameter generation unit 41 of the present embodiment first determines the level of the contrast ratio based on the histogram generated by the luminance distribution information generation unit 40. Then, from this determination result, the level correction control parameter for designating the correction content that suppresses (limits) the level increase on the low luminance side is calculated as the contrast ratio is higher.
However, in the present embodiment, the level correction control parameter that specifies a smaller value as the limit value of the luminance level correction gain is calculated as the contrast ratio is higher.

(Configuration of Correction Gain Calculation Unit 51 ′)
Next, the configuration of the correction gain calculation unit 51 ′ of the present embodiment will be described based on FIG.
Here, FIG. 15 is a block diagram illustrating an example of an internal configuration of the correction gain calculation unit 51 ′.
The correction gain calculation unit 51 ′ includes a luminance value calculation unit 51a, a line memory 51b, a luminance value → gain conversion unit 51d, and a limiter 51e.
The luminance value → gain converting unit 51d receives the input value from the normal conversion LUT (here, one type) for the illumination light component data L (x, y) input from the luminance value calculating unit 51a. A luminance level correction gain corresponding to L (x, y) is acquired. Then, the acquired luminance level correction gain is output to the limiter 51e together with the input value L (x, y).

The limiter 51e selects a limit value corresponding to the value (specified content) of the level correction control parameter from a plurality of types of limit values based on the level correction control parameter from the control parameter generation unit 41. The value of the level correction control parameter may be used as the limit value as it is.
Then, based on the luminance level conversion gain and L (x, y) from the luminance value → gain conversion unit 51d, the input gain corresponds to low-luminance illumination light component data L (x, y) having a predetermined luminance or less. If it is, the limit value is output to the multiplier 52 instead of the input gain.

(Operation example of the imaging apparatus 1)
Next, based on FIG. 16, the operation of the imaging apparatus 1 of the present embodiment will be described.
Here, FIG. 16 is a diagram illustrating an example of the conversion LUT and limit values of the present embodiment.
Since the process up to generating the histogram is the same as that in the first embodiment, the subsequent process will be described.
When receiving the photometric result (histogram) from the luminance distribution information generation unit 40, the control parameter generation unit 41 first calculates the total pixel number HP of the divided ranges (1) and (2), and calculates the total pixel number HP. The first threshold Th1 is compared. Further, the number of pixels LP in the section range (5) is compared with the second threshold Th2.

Here, for example, Th1 is set as a numerical value of 20% of the total number of pixels (for example, if the total number of pixels is 100, Th1 = 20). Furthermore, it is assumed that the second threshold Th2 is set to a numerical value of 20% of the total number of pixels, for example.
The control parameter generation unit 41 determines that the contrast ratio is high when the comparison results are “HP ≧ Th1” and “LP ≧ Th2”. And the level correction control parameter which designates the smallest limit value Lm1 among limit values Lm1-Lm3 shown in FIG. 16 is calculated.

If the number of pixels LH or LP does not meet the above condition, the control parameter generation unit 41 then compares the total number of pixels HP with the third threshold Th3. Here, it is assumed that Th3 is set to a numerical value of 15% of the total number of pixels.
The control parameter generation unit 41 determines that the contrast ratio is low when the comparison result is “HP ≦ Th3”. And the level correction control parameter which designates the largest limit value Lm3 among limit values Lm1-Lm3 shown in FIG. 16 is calculated.
If none of the above determination results is applicable, the control parameter generation unit 41 calculates a level correction control parameter for designating an intermediate limit value Lm2 among the limit values Lm1 to Lm3 shown in FIG.
Then, the control parameter generation unit 41 outputs the calculated level correction control parameter to the limiter 51e of the correction gain calculation unit 51 ′.

Here, the generation processing of HDR color image data and the illumination light component data L (x, y) of the luminance value calculation unit 51a are the same as those in the first embodiment, and hence the luminance value → gain conversion unit 51d. The process will be described.
The luminance value → gain conversion unit 51d corrects the luminance level corresponding to the illumination light component data L (x, y) from the conversion LUT based on the illumination light component data L (x, y) from the luminance value calculation unit 51a. The gain is acquired and output to the limiter 51e together with L (x, y).
The limiter 51e selects a limit value specified by the parameter based on the level correction control parameter from the control parameter generator 41. That is, the limit value designated by the parameter among the limit values Lm1 to Lm3 shown in FIG. 16 is selected.

The luminance level correction gain from the luminance value → gain conversion unit 51d corresponds to L (x, y) whose gain is equal to or less than a predetermined luminance from the corresponding illumination light component data L (x, y). In this case, the selected limit value is output to the multiplier 52 instead of the gain.
If the luminance is not less than the predetermined luminance, the input luminance level correction gain is output to the multiplier 52.
Specifically, as shown in FIG. 16, illumination light component data included in a range from the intersection point of the dotted line extending vertically from the intersection point of the line of the limit values Lm1 to Lm3 and the horizontal axis to the origin point. The value (luminance) is in a range not exceeding the predetermined luminance. That is, the range below the predetermined luminance changes according to the selected limit value. Therefore, information on the illumination light component data L (x, y) is also obtained from the brightness value → gain conversion unit 51d together with the brightness level correction gain.

On the other hand, the line memory 50 outputs the HDR color pixel data corresponding to the gain to the multiplier 52 in synchronization with the output timing of the luminance level correction gain of the correction gain calculation unit 51.
The multiplier 52 multiplies the HDR color pixel data input from the line memory 50 by the luminance level correction gain input from the correction gain calculation unit 51 to perform level conversion (tone mapping) of the HDR color pixel data.

The HDR color pixel data after level conversion is output to the γ conversion unit 53, and the γ conversion unit 53 uses the color pixel data after level conversion as an input value, for example, an LUT corresponding to a tone curve that is quantized to 8 bits. Is used to obtain a conversion value corresponding to the input value. Then, the acquired conversion value is output as an RGB video signal.
For example, in the above processing, when it is determined that the contrast ratio is high and the limit value Lm1 shown in FIG. 16 is selected, the gain on the dark side is suppressed to a low value (a constant value) compared to the limit value Lm3 and the like. (Fixed), the contrast ratio of the captured image is maintained and a natural image output is obtained.

As described above, in the imaging apparatus 1 of the present embodiment, in the correction gain calculation unit 51 ′, based on the level correction control parameter from the photometry unit 25, the contrast ratio of the captured image is determined from a plurality of types of limit values having different values. The limit value can be selected according to the height of.
Furthermore, using the selected limit value, a limit value can be output to the multiplier 52 instead of this gain for the luminance level correction gain corresponding to the illumination light component data L (x, y) having a predetermined luminance or less. it can.
Specifically, the higher the contrast ratio is, the limit value that fixes the dark side gain to a lower constant value is selected.
Thereby, a natural image output can be obtained while maintaining the contrast ratio of the captured image.

(Correspondence)
In each of the above embodiments, the HDR imaging device 10 corresponds to an imaging device, the preprocessing unit 21 corresponds to an image data acquisition unit, and the HDR linear processing unit 30 corresponds to a composite image data generation unit.
In each of the above embodiments, the level correction processing unit 31 corresponds to an image conversion unit, the luminance distribution information generation unit 40 corresponds to a luminance distribution information generation unit, and the control parameter generation unit 41 includes a conversion gain control unit. Corresponding to
In each of the above embodiments, the histogram generation process in the luminance distribution information generation unit 40 corresponds to the luminance distribution information generation step, and the level correction control parameter generation process in the control parameter generation unit 41 corresponds to the conversion gain control step. Correspond.
Also, the luminance level conversion processing (tone mapping) of the HDR color image data in the level correction processing unit 31 corresponds to an image conversion step.

(Note)
Note that the imaging device 1 in each of the above embodiments can be combined with other devices (not shown) such as a display device and a memory device to constitute an electronic device such as a digital camera or a digital video camera.
In each of the above embodiments, a histogram is generated as the luminance distribution information. However, the present invention is not limited to this configuration. For example, if it is possible to determine whether the contrast ratio is high or low, for example, the high luminance side and the low luminance side Other luminance distribution information such as information obtained by counting only the number of pixels may be generated.

In each of the above embodiments, the HDR sensor 10d of the image sensor 10 has a sensor cell array configured using CMOS technology, but the configuration is not limited thereto. For example, other configurations such as a configuration having a sensor cell array composed of CCDs may be used.
Further, in each of the above embodiments, the image processing apparatus 20 acquires a plurality of types of image data having different exposure time lengths from the image sensor 10, combines the acquired plurality of types of image data, and generates HDR image data. The configuration to be generated has been described as an example, but is not limited to this configuration.
For example, a configuration in which the image sensor 10 includes means for generating HDR image data, or a configuration in which another external device generates HDR image data using image data acquired from the image sensor 10 may be employed.

Each of the above embodiments is a preferable specific example of the present invention, and various technically preferable limitations are given. However, the scope of the present invention is particularly limited in the above description. As long as there is no description, it is not restricted to these forms. In the drawings used in the above description, for convenience of illustration, the vertical and horizontal scales of members or parts are schematic views different from actual ones.
Further, the present invention is not limited to the above-described embodiments, and modifications, improvements, and the like within the scope that can achieve the object of the present invention are included in the present invention.

DESCRIPTION OF SYMBOLS 1 ... Imaging device, 10 ... HDR imaging device, 10a ... Lens, 10b ... Micro lens, 10c ... Color filter array, 10d ... HDR sensor, 10d ... Drive circuit, 10f ... Reading circuit, 20 ... Image processing device, 21 ... Pre Process unit, 22-24 ... Delay unit, 25 ... Photometry unit, 26 ... HDR main line processing unit, 30 ... HDR linear processing unit, 30a ... Composition processing unit, 30b ... Color processing unit, 31 ... Level correction processing unit, 40 ... Luminance distribution information generation unit 41 ... control parameter generation unit 50 ... line memory 51 ... correction gain calculation unit 51a ... brightness value calculation unit 51b ... line memory 51c, 51d ... brightness value → gain conversion unit 51e ... Limiter, 52... Multiplier, 53... Gamma conversion unit, 60... Selector, 61 ... Histogram generator, 62 ... Level divider, 63a-63 ... counter, 64A～64e ... output register, 70, 71 ... normalizing gain calculation unit, 72, 73 ... normalization unit, 74 ... combining weight calculator, 75-77 ... weighting unit, 78 ... combining portion

Claims (14)

Among a plurality of image data obtained by imaging a subject with a plurality of different exposure times, the brightness of the first composite image data generated by combining two or more image data is converted to have a different gradation characteristic. Image conversion means for generating two composite image data;
Luminance distribution information generating means for generating luminance distribution information that is information related to the luminance distribution of the captured image of the subject based on at least one of the plurality of image data;
Conversion gain control means for controlling the characteristics of the gain used for conversion of the luminance of the image conversion means based on the luminance distribution information generated by the luminance distribution information generation means,
The image processing device, wherein the image conversion means converts the luminance of the first composite image data using the gain controlled by the conversion gain control means.   The image processing apparatus according to claim 1, wherein the luminance distribution information is a histogram of luminance of the captured image.   The image conversion means converts the luminance of the first synthesized image data to generate second synthesized image data in which the dynamic range of the luminance of the first synthesized image data is compressed. The image processing apparatus according to claim 1 or 2.   The luminance distribution information generation means generates the luminance distribution information based on at least one image data of image data corresponding to the remaining exposure time excluding the longest exposure time among the plurality of image data. The image processing apparatus according to claim 1, wherein the image processing apparatus is characterized.   The conversion gain control means determines whether the contrast ratio of the captured image is high or low based on the luminance distribution information. The image processing apparatus according to any one of claims 1 to 4, wherein a gain used for tone conversion is controlled to be a smaller value as the contrast ratio is higher.   The image processing apparatus according to claim 5, wherein the conversion gain control unit controls the gain so that a gain used for gradation conversion of the pixel data having the predetermined luminance or less is fixed at a constant value.   When it is determined that the contrast ratio is low, the conversion gain control means controls the gain used for gradation conversion of pixel data having the predetermined luminance or less so that the gain becomes higher as the contrast ratio is lower. The image processing apparatus according to claim 5, wherein:   The luminance distribution information generation means divides the corresponding luminance gradation range of the image data into three independent ranges of a high luminance side, an intermediate luminance, and a low luminance side for each of the at least one image data 8. The image according to claim 5, wherein the luminance distribution information is generated by dividing the image into one or more divided ranges and counting the number of pixels belonging to each divided range. Processing equipment.   The conversion gain control unit is configured such that the total number of pixels in the divided range belonging to the high luminance side range is equal to or greater than a first threshold value and the total number of pixels in the divided range belonging to the low luminance side range is equal to or larger than a second threshold The image processing apparatus according to claim 8, wherein the contrast ratio of the captured image is determined to be high.   The conversion gain control means determines that the contrast ratio of the captured image is low when the total number of pixels of the divided range belonging to the range on the high luminance side is equal to or smaller than a third threshold value. The image processing apparatus according to claim 9. Two or more pieces of image data including image data corresponding to the longest exposure time among a plurality of pieces of image data obtained by imaging a subject with a plurality of different exposure times from the image pickup device capable of picking up the subject. Image data acquisition means for acquiring;
A synthesized image data generating unit that linearly synthesizes the two or more image data acquired by the image data acquiring unit based on an exposure time ratio to generate the first synthesized image data;
The luminance distribution information generating means generates the luminance distribution information based on image data corresponding to at least one exposure time out of the two or more pieces of image data excluding the longest exposure time. The image processing apparatus according to any one of claims 1 to 10. The image sensor can read out a pixel signal by a nondestructive readout method that reads out a pixel signal corresponding to the accumulated charge while maintaining the accumulated charge from each pixel, and in order from the shorter exposure time in each frame period. The pixels are exposed at the different exposure times, pixel signals are read from the exposed pixels by the non-destructive readout method, and pixel signal data constituting the plurality of image data is output in the read order. ,
The image data acquisition means acquires the at least one image data prior to acquiring image data corresponding to the longest exposure time in each frame period;
The luminance distribution information generation means generates the luminance distribution information based on the preceding image data which is the at least one image data acquired in advance.
The conversion gain control means linearly synthesizes the two or more pieces of image data acquired in the same frame period as the frame period from which the preceding image data was acquired based on the luminance distribution information generated based on the preceding image data. Controlling the gain for converting the luminance of the generated first composite image data;
The image conversion means converts the luminance of the first composite image data corresponding to the same frame period using the controlled gain corresponding to the same frame period. The image processing apparatus described. Among a plurality of image data obtained by imaging a subject with a plurality of different exposure times, the brightness of the first composite image data generated by combining two or more image data is converted to have a different gradation characteristic. An image conversion step for generating two composite image data;
A luminance distribution information generation step for generating luminance distribution information, which is information relating to the luminance distribution of the captured image of the subject, based on at least one of the plurality of image data;
A conversion gain control step for controlling a characteristic of a gain used for conversion of the luminance of the image conversion unit based on the luminance distribution information generated in the luminance information generation step,
In the image conversion step, the luminance of the first composite image data is converted using the gain controlled in the conversion gain control step. An image sensor capable of imaging a subject;
An image processing apparatus comprising: the image processing apparatus according to claim 11.

Images