JPH08154201A - Image synthesizing method and image synthesizing apparatus - Google Patents

Abstract

(57) [Abstract] [Purpose] When synthesizing images of a high-contrast subject captured at different charge storage times, realizes image synthesizing that can output images in which the signal level of the synthetic image is continuous in all ranges. To do. An image pickup device 1 capable of controlling a charge storage time, a gain ratio calculation means 3 for calculating a gain ratio of two or more images having different charge storage periods, and two or more images having different charge storage periods. Level weighting means 5 and 6 for weighting L and H according to the signal level, and speed converting means 8 for converting the addition output of the level weighting means into the speed of a standard television signal.
And a level compression means 9 for compressing the level of the obtained high dynamic signal to a reference level.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing method and an image synthesizing apparatus for subjecting a subject having a high dynamic range of light intensity in the natural world to image synthesizing without subjecting to saturation or blackout.

[0002]

2. Description of the Related Art Conventionally, as a technique in the field of image synthesis, for example, Japanese Unexamined Patent Publication No. Hei 6-141229 (Japanese Patent Application No. 4-2829)
No. 8508), some images are captured under different exposure conditions (charge storage periods), and the gains are combined and combined by a constant gain ratio based on the difference in the charge storage periods. FIG. 16 is a configuration diagram of the
201 is a CCD, 202 is a memory for storing images, 20
3 is multiplication means for multiplying the signal level by a constant, 204, 205
Is a level weighting means for adding a weight in accordance with the level of the image signal, 206 is an adding means for adding the image signals, 207 is a speed converting means for converting the scanning speed of the output signal of the adding means 206 to ½, and 208 is Level compression means for compressing the level of the image signal, and 209 is timing control means for controlling the timing of each block. The operation of this conventional high dynamic range imaging device will be described below.

The CCD 201 alternately picks up image signals with different charge storage times (that is, different apertures) for each field at a speed twice as fast as a normal TV scanning. CCD
Of the signals read from 201, the signal with the short charge storage time is recorded in the memory 202, and is read from the memory 202 at substantially the same timing as the signal with the long charge storage time is read. The signal read from the memory 202 is multiplied by a constant by the multiplier 203, and the signal levels of the same subject having different charge accumulation times are in principle made equal. Next, a weight (L, H) corresponding to the signal level is attached to each signal by the level weighting means 204 and the level weighting means 205. Then, the adder 206 adds them to combine them into a single image. Since the combined image has a speed twice as high as the normal speed at the time of reading the signal from the CCD 201, the scanning speed of the signal is converted to 1/2 so as to correspond to the scanning of the standard television signal.

[0004]

However, in the conventional configuration as described above, the multiplier 20 corresponding to the charge storage time is used.
There is a problem that the gain cannot be accurately matched by the constant multiple gain matching by 3, and when capturing an area in which the brightness continuously changes, the brightness is inverted or discontinuous in the combined image. Was. That is, the gain ratio between images having different charge accumulation times, which should be constant regardless of the input level in principle, is actually set to the input level as shown in FIG. 3 due to element variations and quantization errors. On the other hand, it does not become a constant value. Furthermore, when synthesizing images obtained by an image pickup system with a γ correction means, there are many image pickup systems that perform γ correction in which a γ curve for a low luminance input, that is, a gain for a low luminance input is suppressed. Cannot be done. As a result, the image signal after the weighted addition has poor connection (continuity) at the overlap portion as shown in FIG. 4, and the quality of the combined image deteriorates.

In view of the above point, the present invention accurately adjusts the gains of the images captured with different charge accumulation times for a subject having a high contrast according to the level of the image signal, and the signal level of the image synthesized. It is an object of the present invention to provide a high dynamic range imaging / synthesizing method and an apparatus therefor capable of outputting images that are continuous in all ranges.

[0006]

In order to achieve the above-mentioned object, the present invention provides an image pickup device capable of controlling a charge storage time and a gain ratio calculation means for calculating a gain ratio between two or more images having different charge storage periods. And 2 with different charge accumulation periods
Level weighting means for weighting according to the signal level of one or more images, and the weighted signal or the level of the signal obtained by converting the weighted signal into the speed of a standard television signal is compressed to a reference level. It is composed of level compression means.

[0007]

With the above-described structure, the present invention calculates the gain ratio based on the signal level of two or more images having different charge accumulation times, adjusts the gains by the multiplying means, and then adjusts the respective signal levels. According to the weighting, the weighted images are added together and the synthesized signal or the high dynamic range signal level obtained by converting to the speed of the standard TV signal is compressed to the reference level. Get an image with less saturation and noise.

[0008]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the arrangement of an image synthesizing apparatus for realizing image synthesis with a high dynamic range according to the first embodiment of the present invention.

In FIG. 1, 1 is an image pickup device for photoelectrically converting a subject, 2 is a memory for recording an image signal, and 3 is a gain ratio for an image having a different charge storage time, based on the signal level thereof. Gain ratio calculation means, 4 is multiplication means, 5 and 6 are level weight means for adding weight according to the level of the image signal, 7 is addition means for adding signals, and 8 is an output signal of the addition means 7 for standard television signal. Speed conversion means for converting the speed of the image signal, 9 is a level compression means for compressing the level of the image signal output from the speed conversion means 8, 10 is a timing control means for controlling the timing of each block, 11
Is an image composition unit.

The operation of the first embodiment constructed as above will be described below. First, a subject is imaged on an image pickup section of an image pickup element (hereinafter referred to as CCD) 1 and converted into an electric signal (electric charge). The intensity of light is proportional to the amount of charge, and the structure is such that the charge obtained during the charge accumulation period is converted into a voltage and output as an electric signal. Therefore, if the charge storage time is shortened, signal saturation does not occur even if the light intensity is strong, and if the charge storage time is lengthened, a sufficiently large signal can be obtained even if the light intensity is weak. Since the CCD is a very general element at present, the description of its basic operation is omitted. The timing controller 10 first controls the charge accumulation period (photoelectric conversion period) of the CCD 1.

FIG. 2 shows the control timing. Figure 2
Is the charge accumulation period from the top, the signal charge read timing,
The signal charge high-speed transfer period and the signal reading timing are shown. In the charge accumulation period, CS1 having a short period and CL1 having a long period are sequentially repeated, and the sum of CS1 and CL1 becomes approximately one field period. Of the signals read from the CCD 1, the signal with the short charge storage period of CS1 is recorded in the memory 2, and the signal with the long charge storage time of CL1 is C.
It is read from the memory 2 at almost the same timing as it is read from the CD 1. The gain ratio calculation means 3 calculates the gain ratio from the ratio between the signal of CS1 having a short charge storage period and the signal of CL1 having a long charge storage time within a constant input level range. FIG. 5 shows an example of a block diagram of the gain ratio calculation means 3.

In FIG. 5, 11a is a comparator, 12 is a divider for calculating the gain ratio, 18a is a write control device for controlling writing to the gain ratio table, and 13a is a gain ratio for each level of CL1 of the overlap section. Is a gain ratio table for storing The comparator 11a determines whether the level of the image signal of CL1 has not reached the level of the overlap part, is within the level of the overlap part, or has reached the saturation level, and outputs the result to the write control device 18a. Output to. In the present embodiment, the range of 80% to less than 100% of the input level is the overlap portion. The level of the CL1 image signal is 80
If it is less than%, it is judged that the level of the overlap part has not been reached, if it is 80% or more and less than 100%, it is judged to be within the range of the level of the overlap part, and if it is 100%, it is judged that it is saturated. To do.

When the level of the image signal of CL1 is within the range of the level of the overlap portion, the gain ratio is calculated by the divider 12, and the writing control device 18a is CL.
The gain ratio value is written in the address of the gain ratio table 13a corresponding to the signal level of 1. Further, when the level of the image signal of CL1 has not reached the level of the overlap portion, the writing control device 18a does not write the value of the gain ratio in the gain ratio table. This is because, in the image composition described below, for pixels whose CL1 image signal level does not reach the level of the overlapping portion, only the CL1 image signal is used as the composition result. Then, when the level of the image signal of CL1 is saturated (when the image signal level of CL1 is 100%), the writing control device 18a outputs C
The gain ratio when the image signal of L1 is 99% level (maximum level not saturated) is written in the gain ratio table 13a.

The gain ratio table 13a stores the gain ratio for each level of the overlapping portion of the CL1 image signal, and outputs the gain ratio according to the level of the CL1 image signal. When the level of CL1 does not reach the level of the overlap part, the level weight H described later is applied, so that the value of the gain ratio is arbitrary, and the gain ratio is output according to the level of CL1. It is not necessary to consider the above. FIG. 6 shows the gain ratio calculation means 3,
The input / output characteristics for the image signal of CL1 are shown. Note that the calculation of the gain ratio by the divider 12 does not have to be performed for each field, and may be refreshed at regular intervals. The multiplier 4 multiplies a signal of CS1 having a short charge accumulation period by a gain ratio.

Next, the level weight means 6 attaches a weight H corresponding to the signal level. The level weighting unit 5 weights the signal corresponding to the charge accumulation period of CL1. An example of the characteristics of each weight is shown in FIG. In FIG. 7, the horizontal axis is the level of the input signal, and the vertical axis is the weight for the signal. The level weight L is 8 of the input signal level.
A weight of 1 is set up to 0%, the weight value is linearly reduced from 80% to 100%, and the weight value is set to 0 at an input level of 100%.
And On the other hand, the level weight H is a weight of 0 up to 80% of the input signal level, the weight value is linearly increased from 80% to 100%, and all the weight values are 1 when the input level is 100% or more.

In this way, the image signals having different dynamic ranges are weighted and added by the adder 7 to be combined into one image. By this weighting, a portion with a good image state (a portion with good S / N and no saturation) is extracted, and an image with a high dynamic range is synthesized. Since the speed of the synthesized image is about twice as high as the normal speed at the time of reading the signal from the CCD 1, the scanning speed of the signal is converted to 1/2 so as to correspond to the scanning of the standard television signal.

The level compression means 9 compresses the level of the combined image in the range of 0% to 100%. As a level compression method, for example, as shown in Japanese Patent Application No. 6-3457, level compression that combines different level conversions for each pixel and different level conversions depending on the spatial frequency is performed to obtain a subjective feeling when observing an image. It is possible to perform level compression while maintaining a high contrast.

As described above, according to this embodiment, it is possible to adaptively perform the gain matching required when synthesizing images having different charge accumulation periods, based on the gain ratio at each level of the overlap portion. Further, as compared to the conventional method of fixing the gain ratio by the ratio of the charge accumulation period, the gain ratio is adaptively calculated, so that accurate gain matching is possible, and the level range of the overlapping part can be reduced,
An image can be synthesized by selecting the signal with the better S / N.

As the level compression means of this embodiment,
If the display has sufficient level reproduction capability (so-called blackout or whiteout does not occur),
A linear level compression method or another nonlinear method having an S-shaped characteristic may be used.

In the gain ratio calculating means 3 of the present embodiment, instead of storing the gain ratio for each level of the overlapping part in the gain ratio table, the gain ratio at each level is calculated based on the gain ratio of several levels. May be calculated. FIG. 8 is a diagram showing an example of the configuration of a gain ratio calculation means for calculating the gain ratio at each level based on the gain ratio at several levels.

In FIG. 8, 11b is a comparator, 12 is a divider, 18b is a write control device for controlling writing to the gain ratio table, 13b is a gain ratio table for storing the gain ratio at several levels of the overlapping part, 1
Reference numeral 4 denotes an interpolation extrapolating means for interpolating and extrapolating the gain ratio based on the gain ratio data in the gain ratio table and outputting the gain ratio according to the level of CL1. The operation of the above configuration will be described below.

When the level of the CL1 image signal is 80%, 90% or 99%, the comparator 11b detects this and outputs a different detection signal to the writing control device 18b. The write control device 18b writes the gain ratio calculated by the divider 12 to the address corresponding to each level of the gain ratio table 13b only when the comparator 11b detects each level. The gain ratio table 13b stores gain ratios at several levels of the overlapping part, and outputs the number of gain ratio values required for curve interpolation according to the level of CL1. The interpolation / extrapolation means 14 performs interpolation / extrapolation by curve interpolation on the value of the gain ratio output from the gain ratio table 13b, and outputs the gain ratio according to the level of CL1.
Even if linear interpolation is performed as the interpolation method by the interpolation / extrapolation means 14, substantially the same effect can be obtained.

The gain ratio calculation means 3 of the first embodiment
In, the level range of CL1 for actually calculating the gain ratio does not have to be the same as the range of the overlap part,
If it is within the overlap range, by calculating the gain ratio only at the level such as the median, maximum frequency value, and average value of the level range of the overlap part, the capacity of the gain ratio table is reduced and it is almost the same. The effect of can be obtained.

The median value of the level range of the overlap portion is the median value (9) of the level range of the overlap portion in the gain ratio calculation section of FIG.
It can be detected by setting it to 0%).

FIG. 9 is a diagram showing an example of the configuration of the gain ratio calculation means for outputting the gain ratio at the level with the highest frequency within the range of the overlap section. In FIG. 9, 15
Reference numerals a and 15b are counters for counting the frequency of each level of the overlap portion, 19a and 19b are changeover switches for switching the read counter and the write counter in the field cycle, and 20 is the maximum frequency value of the CL1 image signal in the overlap portion. Maximum frequency luminance detecting means, 11c is a comparator, 12 is a divider, 18c is a write controller, and 13c is a gain ratio table. The change-over switches 19a and 19b switch between the read counter and the write counter in the field cycle. Counter 15
The values a and 15b are reset every time the write counter is switched to, and count the frequency of each level of the overlapping part in the field. Maximum frequency brightness detection means 2
0 detects the level with the highest frequency. The comparator 11c is
This is detected when the image signal of CL1 is equal to the frequency maximum level. The writing control device 18c writes the gain ratio calculated by the divider 12 in the gain ratio table when the image signal of CL1 is equal to the frequency maximum level.

FIG. 10 is a diagram showing an example of the configuration of a gain ratio calculating means for outputting the gain ratio at the average level within the range of the overlap portion. In FIG. 10, 15a, 1
Reference numeral 5b is a counter for counting the frequency of each level of the overlap portion, 19a and 19b are changeover switches for switching the read counter and the write counter in the field cycle, and 21 is an average for detecting the average luminance value of the CL1 image signal in the overlap portion. Luminance detecting means, 11c is a comparator, 12 is a divider, 18c is a write controller, 1
3c is a gain ratio table. Of the above configuration, the operation other than the average luminance detecting means 21 is the same as that shown in FIG. The average brightness detecting means 21 calculates the average level of the overlapping part from the frequency of each level counted by the counter 15a or the counter 15b.

By using the gain ratio at the maximum frequency level or average level of the overlap part, the capacity of the gain ratio table can be reduced and the connection (continuity) at the overlap part at the time of image composition can be improved. .

In this embodiment, an example is shown in which output signals of an image pickup system having a linear relationship between the input level and the output level are combined. However, in a general image pickup apparatus, a signal after γ correction is output. It An example using such an imaging system will be described below.

FIG. 11 is a block diagram showing the arrangement of an image synthesizing apparatus according to the second embodiment of the present invention. In FIG. 11, reference numeral 1 is an image sensor for photoelectrically converting an object, 17 is a γ correction unit, 2 is a memory for recording an image signal, and 3 is a gain ratio based on the signal level of images having different charge accumulation times. Gain ratio calculating means for calculating, 4 is multiplying means, 5 and 6 are level weighting means for adding weight according to the level of the image signal, 7 is adding means for adding signals, and 8 is an output signal of the adding means 7 as standard. Speed conversion means for converting the speed of the television signal, 9 is level compression means for compressing the level of the image signal output from the speed conversion means 8, 10 is timing control means for controlling the timing of each block, 11
Is an image composition unit.

Regarding the second embodiment of the present invention constructed as described above, the operation other than the γ correction means 17 is the same as that of the first embodiment of the present invention. The γ correction means 17 is (Equation 1)
And has the input / output characteristics shown in FIG. (Equation 1) OUT = IN ^ γ (where ^ indicates a power) In the first embodiment of the present invention, when images having different charge accumulation times are combined, the level of each image is theoretically expressed by Although the composition is performed assuming that it is 2), the image shown in (Equation 3) is composited in the second embodiment of the present invention. (Equation 2) Output level = a1 · input level Output level = a2 · input level gain ratio = a1 / a2 (Equation 3) Output level = (a1 · input level) ^ γ Output level = (a2 · input level) ^ γ Gain ratio = (a1 / a2) ^ γ Here, ^ indicates a power. Similar to the first embodiment of the present invention, in order to synthesize 80% to 100% of the input level as an overlap portion with γ = 0.45, as in the first embodiment of the present invention, as shown in FIG. , The range of 90% to 100% in the level of the image signal after the γ correction may be set as the overlap portion. Therefore, the weight distribution of the level weight L and the level weight H is as shown in FIG.

As described above, according to the second embodiment of the present invention, with respect to the image after the γ correction, a high dynamic range image can be combined with the same structure as that of the image without the γ correction. It will be possible. In addition, since the gain ratio shown in (Equation 3) is calculated from the image data level, it is possible to synthesize a high dynamic range image having continuous levels from a dark area to a bright area even if the value of γ is unknown. . Furthermore, even when the input / output characteristics of photoelectric conversion are different from the photoelectric conversion characteristics shown in (Equation 3) (there are many imaging systems that suppress the gain in the low-luminance portion in order to improve S / N), image data. Since the gain ratio is calculated for each level of the image data based on the above, it is possible to synthesize a high dynamic range image with continuous levels from a dark region to a bright region.

In the embodiment of the present invention, the FIT-CCD is used.
Although an example using the above is shown, the IL-CCD can be used if the continuity of the charge accumulation time is sacrificed to some extent, and the control of the CCD at this time is as shown in FIG.

When synthesizing color images, the gain ratio is calculated for any one of the luminance component, the R component, and the G component in all the embodiments of the present invention, and the other components are also calculated. You may use that value, or
You may calculate each component individually.

In the embodiment of the present invention, the images picked up at the double speed of the television signal are composed and then the speed is converted. However, if the resolution in the time direction is sacrificed, the images picked up at the normal speed will be displayed. A configuration in which the data is output as it is after synthesis is also possible and is included in the present invention.

[0035]

As described above, according to the present invention, when synthesizing images with different exposure conditions, it is possible to perform accurate gain adjustment based on the level of the image signal, and to adjust the level from dark areas to bright areas. It is possible to combine continuous high dynamic range images.

Further, according to the present invention, a high dynamic range image can be synthesized with the same configuration regardless of the presence or absence of the γ correction means.

Further, according to the present invention, γ of the γ correction means
Even if the value is unknown or the input / output characteristics of the photoelectric conversion of the image pickup system including the γ correction means are slightly different from the theoretical characteristics, it is possible to synthesize a high dynamic range image with continuous levels from dark areas to bright areas. It will be possible.

As described above, according to the present invention, it is possible to output an image in which the signal level of an image obtained by image pickup and synthesis is continuous in all ranges for a subject having a high contrast, and the effect is obtained. large.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a first embodiment of an image synthesizing apparatus of the present invention.

FIG. 2 is a timing diagram showing an outline of drive timing of an image sensor.

FIG. 3 is a diagram showing characteristics of an input level and a gain ratio at the time of image combination.

FIG. 4 is a diagram showing the relationship between the input level and the output level of the synthesis result when the gain ratio is incorrect.

FIG. 5 is a block diagram showing a configuration example of a gain calculation means in the first embodiment of the present invention.

FIG. 6 is a characteristic diagram of a gain ratio calculated by a gain calculating means in the first embodiment.

FIG. 7 is a weight distribution diagram of the level weight means in the first embodiment.

FIG. 8 is a block diagram showing a second configuration example of the gain calculation means in the first embodiment.

FIG. 9 is a block diagram showing a third configuration example of the gain calculating means in the first embodiment.

FIG. 10 is a fourth gain calculation means in the first embodiment.
Block diagram showing an example configuration

FIG. 11 is a block diagram showing the configuration of a second embodiment of the image synthesizing apparatus of the present invention.

FIG. 12 is an input / output characteristic diagram of the γ correction means in the second embodiment of the present invention.

FIG. 13 is a diagram showing an image signal after γ correction in the second embodiment of the present invention.

FIG. 14 is a weight distribution diagram of the level weight means in the second embodiment of the present invention.

FIG. 15 is a timing chart showing an outline of drive timing of the image sensor.

FIG. 16 is a block diagram of a conventional high dynamic range imaging / synthesizing device.

[Explanation of symbols]

 1 image sensor 2 memory 3 gain ratio calculation means 4 multiplication means 5 level weighting means 6 level weighting means 7 addition means 8 speed conversion means 9 level compression means 10 timing control means

Claims (5)

[Claims] Claims: 1. Regarding images photographed at different apertures, a gain ratio is calculated based on an image signal within a certain range of the input level, and the gain of the image signal is adjusted from the gain ratio. The data of the image taken with the aperture open is used, the data of the image taken with the aperture open is used for the bright areas, and the weighted sum of the image taken with the aperture open and the image taken with the aperture open in the middle of both. An image synthesizing method characterized by synthesizing data according to. 2. The gain ratios of images photographed with different apertures are the gain ratio at a level where the frequency is maximum within a fixed range of the input level, or the average of the gain ratios within the fixed range of the input level. Value, or median,
Alternatively, the image synthesizing method according to claim 1, wherein the gain ratio is calculated as a gain ratio at a median level within a certain range of the input level. 3. The gain ratio of images photographed at different apertures is calculated by interpolating or extrapolating gain ratios at a plurality of levels within a fixed range of the input level. Image composition method. 4. A gain ratio is calculated for any of a luminance component, an R component and a G component of an input image, or a gain ratio is independently calculated for each of the R, G and B components of an input image. The image synthesizing method according to any one of claims 1 to 3, characterized in that: 5. An image pickup device capable of controlling a charge storage time, a gain ratio calculation means for calculating a gain ratio of two or more images having different charge storage periods, and two or more images having different charge storage periods. It has level weighting means for weighting according to the signal level and level compression means for compressing the level of the weighted signal or the signal obtained by converting the weighted signal into the speed of the standard television signal to the reference level. Image synthesizer.