JP7116001B2 - Imaging device and image processing method - Google Patents

Description

The present invention relates to an imaging device and an image processing method, and more particularly to an imaging device and an image processing method for determining the hue of a captured image.

In-vehicle cameras installed in vehicles for driving support and autonomous driving are required to have a wide angle of view in order to improve the accuracy of detecting pedestrians and bicycles coming out from the side of the vehicle. On the other hand, in-vehicle cameras are required to accurately detect the distance to distant objects for ACC functions (adaptive cruise control, constant speed driving, inter-vehicle distance control), etc., requiring high spatial resolution. be done. However, the widening of the angle of view is generally achieved by reducing the focal length of the objective lens of the camera, which in turn reduces the spatial resolution. Therefore, it is difficult to achieve both widening of the angle of view and improvement of spatial resolution only by improving the objective lens.

On the other hand, there is also known a technique of improving spatial resolution by adopting an image sensor of a different type. In-vehicle cameras use RGGB (Red-Green-Green-Blue) pixel array (or Bayer array) color image sensors to recognize traffic lights, signs, and white and orange lines on the road for driving support. are often adopted. In recent years, a technology has been proposed to replace the RGGB array image sensor with a color image sensor such as an RCCC (Red-Clear-Clear-Clear) pixel array. According to this, the spatial resolution of the image sensor can be improved, and both widening of the angle of view and improvement of the spatial resolution can be achieved.

The RCCC pixel array includes, for example, one pixel (R pixel) that detects red light, and three clear pixels (C pixels) that detect both blue, green, and red light, a total of four pixels. The cells are arranged in two rows and two columns (2×2) to form one unit (unit cell), and the unit cells are repeatedly arranged. The interval between C pixels is 1 pixel, which is different from the RGGB arrangement or the like in which the interval between pixels of the same color is 2 pixels. In this RCCC array, the detection values of C pixels that make up 3/4 of the entire pixels can be used as they are for a grayscale image referred to in distance measurement, and interpolation processing from surrounding pixels that reduces the spatial resolution is performed. can be limited to R pixel locations only in a quarter of the total pixels. Therefore, it is expected that a higher spatial resolution can be obtained compared to the RGGB pixel array.

However, in the RCCC pixel array, which consists of two types of pixels, R (Red) and C (Clear), the three primary colors of R (Red), Green (Green), and Blue (Blue) are classified and the amount of light is detected. Compared to the pixel array, the amount of information about color is limited, and it is necessary to improve the processing method for color.
Application of an RCCC pixel array image sensor to an in-vehicle camera is described in Patent Document 1. However, there is no specific description of the processing of the captured image.

Patent Document 2 also discloses application of an image sensor having a pixel array consisting of two types of R and C to an in-vehicle camera. In Patent Document 2, the magnitude of the correlation between the image of the image sensor and the checkerboard pattern according to the pixel type Red/Clear of the image sensor is used to discriminate between the red tail lamps of automobiles and lights such as headlights and street lights. In this method, since correlation, which is a statistical quantity, is used, the hue is determined for an image area having a certain number of pixels. For this reason, when objects having different hues coexist within this area, an accurate determination result cannot be obtained. Further, it is difficult to set a region in which patterns of different hues are not mixed in the region for determining the hue, unless the pattern has a certain size and the shape can be assumed. As described above, it is difficult for the prior art to accurately determine the hue by the RCCC pixel arrangement. A similar problem may occur in an image sensor having a pixel array consisting of two types of pixels other than the RCCC pixel array.

JP 2017-046051 A U.S. Patent Application Publication No. 2007/0221822

The present invention provides an imaging apparatus that can accurately discriminate a predetermined hue when using an image sensor in which two types of pixels are arranged, and that can achieve both a wide angle of view and an improvement in spatial resolution. An object of the present invention is to provide an image processing method.

An imaging device according to a first aspect of the present invention includes a first pixel that detects light in a first wavelength range in the visible region, and the first wavelength range in addition to the light in the first wavelength range. an image sensor configured by repeatedly arranging filter units including second pixels that detect light of different visible light wavelengths; and positions of the second pixels based on the amount of light detected by the first pixels. and a second interpolation image obtained by interpolating the position of the first pixel based on the amount of light detected by the second pixel. an interpolation processing unit; and a color information generation processing unit that determines a hue at the position based on the amount of light detected by a set of pixels at the same position in the first interpolation image and the second interpolation image. characterized by

An imaging device according to a second aspect of the present invention includes a first pixel that detects light in a first wavelength range in the visible region, and the first wavelength range in addition to the light in the first wavelength range. an image sensor configured by repeatedly arranging filter units including second pixels that detect light of different visible light wavelengths; and positions of the second pixels based on the amount of light detected by the first pixels. and a second interpolation image obtained by interpolating the position of the first pixel based on the amount of light detected by the second pixel. An interpolation processing unit and a color image generation processing unit that generates a color image based on the first interpolation image and the second interpolation image. The color image generation processing unit generates a first component image having a first wavelength component in the first wavelength range based on the first interpolation image, and generates the first interpolation image and the second interpolation image. generating a difference image that is a difference between the interpolated images; multiplying the difference image by a first distribution ratio to generate a second component image having components in a second wavelength range different from the first wavelength range; and multiplying the difference image by a second allocation ratio to generate a third component image having components in a third wavelength range different from the first wavelength range and the second wavelength range. .

An image processing method according to the present invention includes: a first pixel for detecting light in a first wavelength range in the visible region; acquiring an image from an image sensor configured by repeatedly arranging filter units including a second pixel that detects light of a wavelength of , and based on the amount of light detected by the first pixel, the second pixel obtaining a first interpolated image by interpolating the positions of; obtaining a second interpolated image by interpolating the positions of the first pixels based on the amount of light detected by the second pixels; and determining a hue at the position based on the detected light amount of a set of pixels at the same position in the first interpolation image and the second interpolation image.

According to the present invention, it is possible to provide an imaging device and an image processing method capable of accurately discriminating a predetermined hue when using an image sensor in which two types of pixels are arranged, and widening the angle of view. and spatial resolution improvement can be compatible.

1 is an overall configuration diagram illustrating an example of a basic configuration of an imaging device according to a first embodiment; FIG. 1 is a schematic diagram showing a pixel arrangement of an image sensor with an RCCC pixel arrangement; FIG. 4 is a graph showing sensitivity characteristics of R pixels and C pixels of an image sensor having an RCCC pixel array; 4 is a schematic diagram showing an example of arithmetic processing in an interpolation processing unit 102; FIG. 4 is a graph showing an example of measured characteristics of C pixel values and R pixel values of an RCCC image in which colors used in road signs are captured; 4 is a graph showing an example of measured characteristics of C pixel values and R pixel values of an RCCC image in which colors used in traffic lights are captured; 4 is a flowchart for explaining a procedure for obtaining hue information from an image captured by the image sensor 101. FIG. FIG. 8 is a flow chart showing the details of the procedure for determining the hue based on the pixel value ratio (R/C ratio) in step 704 of FIG. 7; FIG. 4 is a flowchart showing a procedure for generating a colorized image in a color image generation processing unit 110 from a grayscale image captured by the image sensor 101 in the first embodiment. FIG. 10 is a diagram (graph) for explaining a first example of a distribution ratio α between a G component image and a B component image; FIG. 11 is a diagram (graph) for explaining a second example of the allocation ratio α between the G component image and the B component image; FIG. 10 is an overall configuration diagram illustrating an example of the basic configuration of an imaging device according to a second embodiment; 13 is a schematic diagram illustrating a configuration example of a color information generation table 1202 in FIG. 12; FIG. FIG. 11 is an overall configuration diagram illustrating an example of the basic configuration of an imaging device according to a third embodiment; 11 is a flow chart showing a procedure for generating a colorized image in a color image generation processing unit 110 from an image captured by an image sensor 101 in the third embodiment. FIG. 11 is a schematic diagram illustrating a pixel array (CyCCC array) in an image sensor 101 according to a fourth embodiment; 4 is a graph showing sensitivity characteristics of Cy pixels and C pixels of an image sensor having a CyCCC pixel array;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be labeled with the same numbers. It should be noted that although the attached drawings show embodiments and implementation examples in accordance with the principles of the present disclosure, they are for the purpose of understanding the present disclosure and are in no way used to interpret the present disclosure in a restrictive manner. is not. The description herein is merely exemplary and is not intended to limit the scope or application of this disclosure in any way.

Although the present embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure, other implementations and configurations are possible without departing from the scope and spirit of the present disclosure. It is necessary to understand that it is possible to change the composition/structure and replace various elements. Therefore, the following description should not be construed as being limited to this.

[First embodiment]
An example of the basic configuration of the imaging device according to the first embodiment will be described with reference to the overall configuration diagram of FIG. This imaging apparatus 1 has an image sensor 101 , an interpolation processing unit 102 , a color information generation processing unit 103 , a recognition processing unit 104 , a color image generation processing unit 110 and an image recording unit 112 .

The image sensor 101 is an image sensor such as a CMOS sensor or a CCD sensor for acquiring optical images. These image sensors have photodiodes arranged in an array on a plane, and a plurality of pixels provided by these photodiodes are configured to detect the light quantity distribution on the plane. An image of an imaging target can be obtained by detecting light focused by a condensing member such as a lens or a mirror (not shown) with the image sensor 101 .

The imaging apparatus 1 of the first embodiment uses an image sensor having the RCCC pixel array shown in FIG. 2 as the image sensor 101 . The RCCC pixel array includes an R (Red) pixel 201 that receives light from a color filter that transmits red light in the visible range and is capable of detecting red light, and a transparent color filter that receives light from a transparent color filter and receives visible light. It has two types of pixels, C (clear) pixels 202 that are capable of detecting the total amount of light. Filter units 203 of 2×2 pixels each including one R pixel 201 and three C pixels 202 are repeatedly arranged over multiple rows and multiple columns.

As shown in the graph of FIG. 3, the R pixels 201 are capable of receiving light in the wavelength range of red light, while the C pixels 203 are capable of receiving light in the wavelength range of visible light including blue, green, and red light. Light can be received over the entire area. The image output by the image sensor 101 is a grayscale image in which pixel values based on the amount of red light detected by the R pixels 201 and pixel values based on the amount of all visible light detected by the C pixels 202 are mixed.

Based on the grayscale image output from the image sensor 101 , the interpolation processing unit 102 obtains the red light component light amount at the position of the C pixel 202 by interpolating from the detected light amount of the surrounding R pixels 201 . In addition, based on the grayscale image output from the image sensor 101 , the interpolation processing unit 102 interpolates the light amount of the entire visible light at the position of the R pixel 201 from the detected light amount of the surrounding C pixels 202 to obtain it.

The processing of the interpolation processing unit 102 can be realized, for example, by the calculation shown in FIG. In the arithmetic processing shown in FIG. 4, the unit of processing is 3×3 pixels centering on the pixel to be interpolated. When performing interpolation processing in the image sensor 101 having the RCCC pixel array shown in FIG. 2, the interpolation processing can be performed based on four pixel patterns, pattern 1 to pattern 4. Pattern 1 is a case where the 9 pixels at the bottom right of FIG. Pattern 4 shows a case in which nine pixels at the upper left of FIG. 2 are used as a unit of processing.

Patterns 1 to 4 all target the pixel at the center of 3×3 pixels for interpolation processing. For example, in patterns 2 to 4, the center pixel is the C pixel 202, so the R component is interpolated at the center C pixel 202. FIG. On the other hand, in pattern 1, since the central pixel is the R pixel 201, the interpolation calculation of the C component in the central R pixel 201 is performed. Interpolation calculation of a pixel to be interpolated (center of 3×3 pixels) is performed based on pixel values of pixels surrounding the pixel to be interpolated.

In pattern 1, the center R pixel 201 (R22) of 3×3 pixels is the object of interpolation processing, and its C component (C22) is calculated based on the pixel values of four C pixels 202 (C22 =(C12+C32+C12+C32)/4).
In pattern 2, since the C pixel 202 (C22) is at the center of the 3×3 pixels, its R component (R22) is calculated based on the pixel values of the two R pixels 201 on the left and right (R22=(R212+R23 )/2).
In pattern 3, the C pixel 202 (C22) is at the center of the 3×3 pixels, and its R component (R22) is calculated based on the pixel values of the two upper and lower R pixels 201 (R22=(R12+R32) /2).
In pattern 4, the C pixel 202 (C22) is at the center of the 3×3 pixels, and its R component (R22) is calculated based on the pixel values of the four R pixels 201 (R22=(R11+R13+R31+R33 )/4).

This calculation is suitable for the pixel array of FIG. 2, and it goes without saying that different calculations having the same effect as described above can be adopted for different pixel arrays. In addition, the above interpolation calculation was performed as the average value of the surrounding pixel values, but different calculation methods (for example, weighted averaging) can be adopted according to the required performance of the imaging device and the environmental conditions. Needless to say.

By repeating this process, an image (R image) representing the spatial distribution of the light intensity of red light and an image (C image) representing the spatial distribution of the overall visible light intensity having the same number of pixels as the R image are generated. can be done. Further, following the processing shown in FIG. 4, interpolation processing for correcting image distortion may be further performed on the R image and the C image.

The color information generation processing unit 103 outputs lightness information and hue information of the captured image to the recognition processing unit 104 based on the C image and the R image generated by the interpolation processing unit 102 . The recognition processing unit 104 refers to the lightness component and the hue component to recognize traffic signals, road signs, lights of vehicles ahead, white lines on the road, orange lines, and the like. Hue is defined for traffic lights and the like. For example, traffic lights have three colors of blue-green (cyan), orange, and red, and road signs often use red, blue, white, and orange. As for the lights of the vehicle, the tail lamp and the brake lamp are red, and the winker is orange. Therefore, the recognition processing unit 104 is configured to be able to distinguish these colors from each other.

The color information generation processing unit 103 can calculate the lightness information of the captured image based on the C image generated by the interpolation processing unit 102, for example.

The color information generation processing unit 103 has an R/C value comparison processing unit 105 inside. The R/C value comparison processing unit 105 receives the C image and the R image from the interpolation processing unit 102, and detects the detected light amount of the R component and the detection of the C component with respect to the pixels arranged at the same position in the C image and the R image. A ratio (R/C ratio) to the amount of light is calculated. Then, the R/C value comparison processing unit 105 compares the R/C ratio with the reference value of the color to be discriminated to discriminate the hue, and outputs the discrimination result as hue information.

In the present embodiment, assuming recognition of traffic lights, road signs, vehicles traveling ahead, and lines on the road surface, red determination reference value storage unit 106, orange determination reference value storage unit 107, A value storage unit 108 and a blue/green determination reference value storage unit 109 are provided. The four reference values stored in these storage units are compared with the R/C ratio. Although the hues of blue-green (cyan) of traffic lights and blue of road signs are different, they can be distinguished by their brightness and their positions, and these hues are collectively treated as blue and green.

A color image generation processing unit 110 colorizes an image using the C image and the R image generated by the interpolation processing unit 102, and converts an R (Red) component image, a G (Green) component image, and a B (Blue) component image corresponding to the three primary colors. Generate and output a component image. The image recording unit 112 stores the R (Red) component image, the G (Green) component image, and the B (Blue) component image generated by the color image generation processing unit 110. The image recording unit 112 stores the R (Red) component image, the G (Green) component image, and the B (Blue) component image. consists of The color image generation processing unit 110 of the first embodiment outputs the input R image to the image recording unit 112 as an R component image, and has a CR image generation processing unit 111 inside. As the G component image and the B component image, an image based on the CR image generated by calculating the difference value between each pixel of the C image and the R image in the CR image generation processing unit 111 is output.

FIG. 5 shows an example of measured characteristics of C pixel values and R pixel values of an RCCC image obtained by photographing colors used in road signs. More specifically, the color chart (X-Rite Colorchecker SG (registered trademark)) was photographed under white light with five levels of brightness using a camera equipped with an RCCC pixel array image sensor. This is an example of characteristics. The C image and the R image are obtained as a result of the interpolation processing described above. The graph in FIG. 5 shows parts close to the standard colors of safety colors specified in JIS Z9103 used for road signs, extracted from the above color chart (for example, red [row 3, column L], orange [row 6, column L ]・Yellow [row 4, column H]・white (achromatic) [row 6, column J]・blue [row 3, column F]) Plotting the relationship between the amount of light detected by the C pixel 202 and the amount of light detected by the R pixel 201 is.

In FIG. 5, reference numerals 501 and 502 denote plots based on the pixels in the red portion and C vs. R characteristics for red light determined by the least squares method, respectively. Also, reference numerals 503 and 504 represent the C vs. R characteristics for orange light determined by the plot based on the orange portion and the least squares method, respectively. Reference numerals 505 and 506 represent the C vs. R characteristics for yellow light determined by the plot based on the yellow area and the least squares method, respectively. Reference numerals 507 and 508 represent plots based on white (achromatic) parts and C vs. R characteristics for achromatic light determined by the least-squares method, respectively. 509 and 510 represent the blue site-based plots. From these data, the ratio of the amount of light detected by the R pixel to the amount of light detected by the C pixel (R/C) for each color is constant regardless of the brightness, and recognition of road signs etc. for driving support with the in-vehicle camera Since each color of red, orange, yellow, white (achromatic color), and blue referred to in the object has a different ratio, it is understood that these hues can be classified based on R/C.

FIG. 6 is an example of measured characteristics of C pixel values and R pixel values of an RCCC image obtained by photographing colors used in traffic lights. This FIG. 6 is also an example of actual measurement characteristics as a result of photographing the color chart described above under white light of five brightness levels with a camera equipped with an image sensor having an RCCC pixel array. FIG. 6 extracts parts close to the colors used in traffic lights (red [row 3, column L], orange [row 6, column L], cyan [row 8, column B]), and shows the detected light amount of the C pixel and the R pixel. It plots the relationship between the amounts of detected light. Reference numerals 501 and 502, and reference numerals 503 and 504 are the same as those in FIG. 5, so overlapping descriptions will be omitted. In FIG. 6, reference numerals 601 and 602 represent the C vs. R characteristics for yellow light determined by the plot based on the cyan color portion and the least squares method, respectively. From these data, it is possible to discriminate the color of the signal as well as the hue in the same manner as in FIG.

FIG. 7 is a flowchart for explaining the procedure for obtaining hue information from an image captured by the image sensor 101. The process executed by the image sensor 101, the interpolation processing unit 102, and the color information generation processing unit 103 in FIG. be.

First, in step 701, an image is picked up by the image sensor 101 and acquired. This acquired image has pixel values based on the amount of red light detected by the R pixels 201 of the image sensor 101 having the RCCC pixel array shown in FIG. It is a grayscale image with mixed values.

At subsequent step 702, the above-described interpolation processing is executed for each pixel position of the grayscale image obtained at step 701 to generate an R image and a C image. Then, in step 703, the ratio (R/C ratio) between the pixel value of the R image and the pixel value of the C image at the same pixel position is obtained, and in step 704, the hue is discriminated based on the value of the R/C ratio. .

FIG. 8 is a flow chart showing the details of the procedure for determining the hue based on the pixel value ratio (R/C ratio) in step 704 . In FIG. 8, Tr is the lower limit of the R/C ratio for distinguishing red, To is the lower limit of the R/C ratio for distinguishing orange, and Ty is the R/C ratio for distinguishing yellow. The lower limit value and Tg indicate the lower limit value of the R/C ratio for discriminating achromatic colors (white, gray, black), respectively.

At step 801, the value of the R/C ratio is compared with the lower limit Tr. If R/C>Tr (Yes), the hue of the pixel is determined to be red. If R/C≤Tr (No), the process proceeds to step 802 .

At step 802, the value of the R/C ratio is compared with the lower limit value To. If R/C>To (Yes), the hue of the pixel is determined to be orange. If R/C.ltoreq.To (No), go to step 803;

At step 803, the value of the R/C ratio is compared with the lower limit value Ty. If R/C>Ty (Yes), the hue of the pixel is determined to be yellow. If R/C≤Ty (No), go to step 804 .

At step 804, the value of the R/C ratio and the lower limit Tg are compared. If R/C>Tg (Yes), the hue of the pixel is determined to be achromatic (white, gray, black).

If all of steps S801 to S804 result in "No", the hue of the pixel is determined to be blue or green. Thus, the determination procedure of step S704 is completed.

FIG. 9 is a flowchart showing a procedure for generating a colorized image in the color image generation processing unit 110 from a grayscale image captured by the image sensor 101 .

Similarly to FIG. 7, in steps 701 and 702, after the image sensor 101 captures and acquires a grayscale image, the above-described interpolation processing is executed for each pixel position of the grayscale image, and the R image and A C image is generated.

In the subsequent step 901 , the CR image generation processing unit 111 generates a difference image (CR image) between the pixel values of the generated C image and the pixel value of the R image. In step 902, the CR image generation processing unit 111 generates an image (α(CR)) obtained by multiplying the pixel values of the CR image by the distribution ratio α (0≦α≦1). to be a G component image.

Further, in step 903, an image obtained by multiplying the pixel value of the CR image by (1-α) ((1-α)(CR)) is generated in the CR image generation processing unit 111, and the B component image It is said that The distribution ratio α is a value indicating the proportion of the G component image included in the difference image (CR). Hereinafter, α may be referred to as "first allocation ratio" and (1-α) as "second allocation ratio".

The sum of the pixel value α(CR) given to the G component image in step 902 and the pixel value (1−α)(CR) given to the B component image in step 903 is CR. In other words, the sum of the first distribution ratio α and the second distribution ratio (1−α) is one. In the relationship between the wavelength-sensitivity characteristics of the R pixel 201 and the C pixel 202 shown in FIG. Since the amount of light detected by the pixel 202 and the R pixel 201 is estimated to be greater by the sum of the G component and the B component, the above relationship is established. It is also possible to add processing for increasing/decreasing the brightness of each component (not shown) to the obtained R component image, G component image, and B component image.

FIG. 10 is a diagram (graph) for explaining a first example of the distribution ratio α between the G component image and the B component image. In the graph of FIG. 10, the vertical axis indicates the value of the R/C ratio (0 or more and 1 or less), and the horizontal axis indicates the ratio of the G component image to the sum (G+B) of the G component image and the B component image, that is, the first A distribution ratio α (0 or more and 1 or less) is shown. In this graph, dotted lines indicate the approximate distribution of each hue within the space indicated by the vertical and horizontal axes.

As shown in FIG. 10, if only the magnitude of the R/C ratio is used as a factor, it is possible to determine whether the pixel is red, cyan, or gray. However, the hues arranged in the horizontal direction in the graph of FIG. 10 cannot be discriminated only by the R/C ratio. For example, blue, cyan, and green cannot be distinguished from each other. Also, pink and yellow (or orange) cannot be distinguished only by the magnitude of the R/C ratio.

FIG. 10 illustrates determination of hue when the value of α is fixed at 0.5. That is, in FIG. 10, a locus 1001 indicates changes in the R/C ratio, and α is fixed at 0.5 regardless of the R/C ratio. As the R/C ratio increases from 0 to 1, the hue of the pixel changes sequentially from cyan, achromatic, intermediate between pink and orange, and red. When the image capturing apparatus of the first embodiment is applied to an on-vehicle camera, the color of the sky on a clear sky and the green light (cyan), the color of the road surface and the white line (achromatic color), the red light and the tail lamp of the vehicle (red) are particularly well reproduced. Therefore, based on the image sensor 101 having the RCCC pixel array, it is possible to obtain a colorized image with little visual discomfort.

FIG. 11 is a diagram (graph) for explaining a second example of the distribution ratio α between the G component image and the B component image. The difference from FIG. 10 is that, at least when the R/C ratio is within a predetermined range, the distribution ratio α is varied in accordance with changes in the R/C ratio so that the distribution ratio α changes. This is because, when the imaging apparatus 1 is used as an in-vehicle camera, orange appears more frequently than pink in the captured image.

Trace 1101 in FIG. 11 shows the change in R/C ratio. In FIG. 11, when the R/C ratio is smaller than the first value (for example, R/C<0.25), the distribution ratio α is fixed at a constant value (for example, α=0.5). . On the other hand, when the R/C ratio is greater than or equal to the first value and less than or equal to the second value, the distribution ratio α is increased as the R/C ratio increases. Then, when the R/C ratio is larger than the second value (for example, R/C>0.75), the distribution ratio α is fixed at a constant value (for example, α=1). A function indicating the trajectory 1101 can be stored in a storage unit (not shown) in the color image generation processing unit 110 . The locus 1101 in FIG. 11 is merely an example, and it goes without saying that the shape of the locus 1101 can be changed as appropriate.

When the distribution ratio α is changed in accordance with an increase in the R/C ratio as shown by the locus 1101, the hue of the pixel is cyan or gray when the R/C ratio is small (α=0.5). can judge. On the other hand, when the R/C ratio is greater than or equal to the first value and less than or equal to the second value, the distribution ratio α increases as the R/C ratio increases. As a result, the hue assigned to the pixels with the above R/C ratio changes from cyan to achromatic to orange to red. When the imaging device 1 is used as an in-vehicle camera, the colors of the clear sky and green traffic lights (cyan), road surfaces and white lines (achromatic colors), and red traffic lights and tail lamps (red) are reproduced particularly well. In addition, compared to the example of FIG. 10, it is possible to improve the reproducibility of colors such as orange lines on the road surface, yellow traffic lights, blinkers, and the like.

Note that the locus 1101 in FIG. 11 is preferably set so as to form a continuous curve throughout the R/C ratio changing from 0 to 1. This is because if the locus 1101 becomes a discontinuous curve, the change in hue becomes large in the vicinity of the discontinuous sky, causing a problem of increased noise.

As described above, according to the imaging device 1 and the image processing method according to the first embodiment, even when an image sensor having two types of pixels such as an RCCC pixel array is applied, a predetermined It becomes possible to accurately discriminate hues. In the case of the RCCC pixel array, it is possible to achieve both a wide angle of view and an improvement in spatial resolution compared to the RGGB pixel array, but the information that can be obtained regarding color is usually limited. However, in the first embodiment, it is possible to generate an image that reflects the color of the object, as in the case of the imaging apparatus using the RGGB array image sensor.

In the first embodiment, an example of the RCCC pixel array is shown. It does not have to be 1:3. In addition, in order to emphasize the accuracy of recognizing red objects (traffic lights, tail lights, and signs) as an on-vehicle camera application, an embodiment in which two types of pixels, one having a red color filter and the other having a transparent color filter, is used. Although shown, pixels having non-red color filters and pixels having transparent color filters may be paired depending on the application.

[Second embodiment]
Next, an imaging device and an image processing method according to the second embodiment will be described with reference to FIGS. 12 and 13. FIG. FIG. 12 is a block diagram showing the overall configuration of an imaging device according to the second embodiment. In FIG. 12, the same reference numerals as in FIG. 1 denote the same constituent members as in the apparatus of the first embodiment, and duplicate descriptions will be omitted below.

The second embodiment differs from the first embodiment in the configuration of the color information generation processing unit 103. FIG. The color information generation processing section 103 of the second embodiment includes an address generation section 1201 and a color information generation table 1202 .

The address generator 1201 is configured to receive the C image and the R image and output corresponding address signals. Specifically, the address generation unit 1201 generates an address signal corresponding to a set of pixel values at the same pixel position in the input C image and R image, and outputs the address signal to the color information generation table 1202 .

The color information generation table 1202 stores address signals, and lightness information and hue information corresponding to the address signals as a table. Based on the address information input from the address generation unit 1201, the color information generation table 1202 specifies and outputs the corresponding lightness information and hue information.

For the color information generation table 1202, for example, the data structure shown in FIG. 13 is applied. In the example of FIG. 13, the address signal supplied from the address generation unit 1201 is obtained by concatenating the R pixel value and the C pixel value to the upper bit and the lower bit, such as {R, C}. The generated data is assigned to be applied, and hue information corresponding to the R/C ratio by the R pixel value and C pixel value corresponding to the address is stored as corresponding data. According to such a configuration, it is possible to easily generate hue information without requiring complicated calculations.

In addition, in the second embodiment, it is possible to easily generate color information in which not only the hue division by the R/C ratio but also the brightness division by the level of the C pixel value is added. In addition, it is possible to determine complex color information by changing the threshold value for the division of the R/C ratio value and the number of divisions according to the level of the C pixel value.

[Third embodiment]
Next, an imaging device and an image processing method according to the third embodiment will be described with reference to FIGS. 14 and 15. FIG. FIG. 14 is a block diagram showing the overall configuration of an imaging device according to the third embodiment. In FIG. 13, the same reference numerals as in FIG. 1 are given to the constituent members that are common to the apparatus of the first embodiment, and redundant description will be omitted below.

The imaging apparatus according to the third embodiment differs from that according to the first embodiment in the configuration of the color image generation processing section 110 . The color image generation processing unit 110 of the third embodiment is configured to obtain a visually pleasing color image even when the amount of light detected by the pixels exceeds the upper limit. In the image generation processing in the imaging apparatus of the above-described embodiment, an appropriate color image can be obtained in the case where the amount of light detected by the pixels does not exceed the upper limit and is not saturated. becomes hue. This is because the difference between the C pixel value and the R pixel value becomes smaller than the actual value in the saturated portion, and the values given to the G component and the B component become relatively small. This phenomenon can be effectively suppressed in the third embodiment.

As shown in FIG. 14, the color image generation processing unit 110 of the third embodiment includes a CR image generation processing unit 111 similar to that of the first embodiment, a brightness saturated pixel determination processing unit 1401, A saturated pixel replacement processing unit 1402 , an R component brightness correction unit 1403 , a G component brightness correction unit 1404 and a B component brightness correction unit 1405 are provided. The R component brightness correction unit 1403, the G component brightness correction unit 1404, and the B component brightness correction unit 1405 constitute a brightness correction unit as a whole.

The functions of the CR image generation processing unit 111 are the same as in the first embodiment (FIG. 1). When the detected light amount of a pixel in the C image exceeds the upper limit, the brightness saturated pixel determination processing unit 1401 determines that the pixel is a brightness saturated pixel. It should be noted that it is also possible to adopt a configuration in which a pixel having a brightness near the upper limit, although the amount of detected light does not exceed the upper limit, is determined to be a saturated pixel.

The saturated pixel replacement processing unit 1402 replaces the brightness of the R component, the G component, or the B component with the pixel determined to be saturated in brightness based on the determination result of the saturated brightness pixel determination processing unit 1401. Output by switching to the upper limit value. That is, the saturated pixel replacement processing unit 1402 has a function of replacing each component value with an upper limit value so that a saturated pixel is treated as white.

The saturated pixel replacement processing unit 1402 receives the R image pixel brightness after being corrected by the R component brightness correction unit 1403 . Similarly, the saturated pixel replacement processing unit 1402 receives the pixel brightness of the G image and the B image after being corrected by the G component brightness correcting unit 1404 and the B component brightness correcting unit 1405, respectively.

When 8 bytes (255 bits) are assigned to each of the R component, the G component, and the B component, the total value of RGB representing brightness is 255×3=765 bits for white.
On the other hand, 255 bits are also assigned to the C image input to the CR image generation processing unit 111, and there is a threefold difference in the number of bits.

Therefore, if the output of the CR image generation processing unit 111 is directly input to the saturated pixel replacement processing unit 1402, the maximum brightness of the non-saturated pixels and the brightness of the saturated pixels adjusted to the upper limit value can be used. A step of about three times the brightness is generated, resulting in an image with a large visual sense of incongruity. Therefore, the color image generation processing unit 110 of the third embodiment includes an R component brightness correction unit 1403, a G component brightness correction unit 1404, and a B component brightness correction unit 1405, and increases the brightness of each component. After correcting in the direction of , it is supplied to the saturated pixel replacement processing unit 1402 . As an example, the correction amount β in the R component brightness correction unit 1403, the G component brightness correction unit 1404, and the B component brightness correction unit 1405 is a value of 1 or more, preferably β≈3. If the lightness of the R component image, the G component image, and the B component image adjusted to β times exceeds the respective upper limits, the saturated pixel replacement processing unit 1402 may adjust the lightness so as to reach the upper limits.

FIG. 15 is a flow chart showing the procedure for generating a colorized image in the color image generation processing unit 110 from the image captured by the image sensor 101 . Steps 701, 702, 703, 901, 902, and 903 are the same as in FIG. 9, so duplicate descriptions will be omitted.

In step 1501 after step 903, the brightness saturation pixel determination processing unit 1401 determines the presence and position of saturated pixels (pixels having brightness equal to or greater than the upper limit value) in the C image.

In the subsequent step 1502, the pixel values (brightness) of the R component image, the G component image, and the B component image are multiplied by β (β ≧1). This reduces a step in brightness when a saturated pixel is replaced with white.

Then, in step 1503, the pixel values of the R component image, the G component image, and the B component image at the position of the saturated pixel are replaced with the respective upper limit values. By this step 1503, the portions where the pixel values are saturated and the pseudo red component is strongly displayed are replaced with white. According to the device configuration and the processing method described above, in addition to the effects described in the first embodiment, a color image with little visual discomfort can be obtained even when the amount of light detected by the pixels exceeds the upper limit. effect is obtained.

[Fourth embodiment]
Next, an imaging device and an image processing method according to the fourth embodiment will be described with reference to FIGS. 16 and 17. FIG. As shown in FIG. 16, in this fourth embodiment, the image sensor 101 has a pixel array different from the RCCC pixel array. In this image sensor 101, one filter unit 203 has two types of pixels: a Cy pixel 1601 for detecting cyan, which is a complementary color of red, and a C pixel 202. One Cy pixel 1601 and three C pixels 202 is a pixel arrangement (CyCCC pixel arrangement) in which the 2×2 pixel filter units 203 are repeated over a plurality of rows and a plurality of columns.

As shown in FIG. 17, Cy pixels 1601 are sensitive to blue and green light. A difference image (C-Cy) between the C image and the Cy image is the R image. By performing the same processing as in the first embodiment on the C image and the R image thus obtained, the same effect as in the first embodiment can be obtained. Since the Cy pixels 1601 are sensitive to blue light and green light, the sensitivity of the image sensor 101 to green light and blue light is improved compared to the RCCC pixel arrangement. When white light is incident on the image sensor 101, a large amount of light is detected as compared to the R pixel 201. Therefore, the Cy pixel 1601 tends to have a higher S value than the R pixel 201 as a whole image including subjects of various hues. /N ratio can be given, and as a result, it can be expected that the accuracy of hue determination is improved compared to the first embodiment.

It should be noted that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. In addition, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

DESCRIPTION OF SYMBOLS 101... Image sensor 102... Interpolation process part 103... Color information generation process part 104... Recognition process part 110... Color image generation process part 111... CR image generation process part 112... Image recording part 201 R pixel 202 C pixel 203 filter unit 1201 address generation unit 1202 color information generation table 1401 saturated brightness pixel determination processing unit 1402 saturated pixel replacement processing unit 1403 R component brightness correction 1404...G component brightness correction section 1405...B component brightness correction section 1601...Cy pixel.

Claims (10)

A first pixel that detects light in a first wavelength range in the visible region, and a second pixel that detects light in a visible light wavelength different from the first wavelength range in addition to light in the first wavelength range. an image sensor configured by repeatedly arranging filter units including pixels of
A first interpolated image obtained by interpolating the position of the second pixel based on the amount of light detected by the first pixel, and the position of the first pixel based on the amount of light detected by the second pixel an interpolation processing unit configured to generate a second interpolated image obtained by interpolation;
a color information generation processing unit that determines a hue at the position based on the amount of light detected by a set of pixels at the same position in the first interpolation image and the second interpolation image ,
A color image generation processing unit that generates a color image based on the first interpolation image and the second interpolation image,
The color image generation processing unit
generating a first component image having a first wavelength component in the first wavelength range based on the first interpolated image;
generating a difference image that is the difference between the first interpolated image and the second interpolated image;
multiplying the difference image by a first allocation ratio to generate a second component image having components in a second wavelength range different from the first wavelength range;
configured to multiply the difference image by a second allocation ratio to generate a third component image having components in a third wavelength range different from the first wavelength range and the second wavelength range. An imaging device characterized by: 2. The color information generation processing unit according to claim 1, wherein the color information generation processing unit determines the hue at the position based on the ratio of the amount of light detected by a set of pixels at the same position in the first interpolated image and the second interpolated image. imaging device. The first distribution ratio or the second distribution ratio is set to a constant value regardless of a change in the ratio of the detected light amounts of the sets of pixels at the same position in the first interpolated image and the second interpolated image. 2. The imaging device according to claim 1 , wherein: 2. The method according to claim 1 , wherein said first distribution ratio or said second distribution ratio changes according to a change in a ratio of amounts of light detected by a set of pixels at the same position in said first interpolated image and said second interpolated image. The imaging device described. 3. The imaging device according to claim 1 , wherein the first pixels are pixels capable of detecting red light, and the second pixels are pixels capable of detecting red light, green light, and blue light. the first pixel is a pixel capable of detecting blue light and green light;
3. The imaging device according to claim 1 , wherein said second pixel is a pixel capable of detecting red light, green light and blue light. A first pixel that detects light in a first wavelength range in the visible region, and a second pixel that detects light in a visible light wavelength different from the first wavelength range in addition to light in the first wavelength range. acquiring an image from an image sensor comprising a repetitive array of filter units including pixels of
obtaining a first interpolated image by interpolating the positions of the second pixels based on the amount of light detected by the first pixels;
obtaining a second interpolated image by interpolating the position of the first pixel based on the amount of light detected by the second pixel;
Determining the hue at the position based on the detected light amount of the set of pixels at the same position in the first interpolated image and the second interpolated image ,
further comprising generating a color image based on the first interpolated image and the second interpolated image;
The step of generating the color image includes:
generating a first component image having a first wavelength component in the first wavelength range based on the first interpolated image;
generating a difference image that is the difference between the first interpolated image and the second interpolated image;
multiplying the difference image by a first allocation ratio to generate a second component image having components in a second wavelength range different from the first wavelength range;
An image processing method comprising: multiplying the difference image by a second distribution ratio to generate a third component image having components in a third wavelength range different from the first wavelength range and the second wavelength range . 8. The method according to claim 7 , wherein the step of determining the hue determines the hue at the position based on a ratio of amounts of light detected by a set of pixels at the same position in the first interpolated image and the second interpolated image. Image processing method. The first distribution ratio or the second distribution ratio is set to a constant value regardless of a change in the ratio of the detected light amounts of the sets of pixels at the same position in the first interpolated image and the second interpolated image. 8. The image processing method according to claim 7 , wherein 8. The method according to claim 7 , wherein said first distribution ratio or said second distribution ratio changes according to a change in a ratio of amounts of light detected by a set of pixels at the same position in said first interpolated image and said second interpolated image. The described image processing method.

Images