JP2015032870A - Imaging device, imaging apparatus and imaging processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device capable of reproducing a high-luminance region while disposing low-sensitivity pixels with a low density.SOLUTION: The imaging device comprises: a first pixel which receives a first color component of subject light and outputs a first component value corresponding to the first color component; a second pixel which receives a second color component of the subject light and outputs a second component value corresponding to the second color component; and a third pixel which receives a first color component of the subject light with a sensitivity lower than that of the first pixel and outputs a third component value. The number of third pixels is less than the numbers of first pixels and second pixels. Neighboring pixels surrounding the second pixel or neighboring pixels surrounding at lest another second pixel closest to the second pixel include the third pixel. In neighboring pixels surrounding the third pixel, the first pixel is included and the other third pixel is not included.

Description

  The present invention relates to an imaging device, an imaging device, and an imaging processing program.

  Conventionally, a high-sensitivity pixel and a low-sensitivity pixel are arranged on the image sensor, and a high dynamic range image is generated using the output of the low-sensitivity pixel, and a high-resolution image is generated using the output of all pixels. Have been proposed (see Patent Document 1, for example).

JP 2004-336468 A

  In order to reproduce the image structure of the high luminance region, it is necessary to dispose the low sensitivity pixels to a certain degree of density, and there is a problem that the reproducibility of the low luminance region and S / N must be sacrificed.

The imaging device according to claim 1, wherein the first pixel that receives the first color component of the subject light and outputs the first component value corresponding to the first color component, and the second pixel of the subject light. A second pixel that receives the color component and outputs a second component value corresponding to the second color component, and the first color component of the subject light is received with lower sensitivity than the first pixel. A third pixel that outputs a third component value, the number of pixels of the third pixel is less than either of the first pixel and the second pixel, and adjacent pixels surrounding the second pixel, Alternatively, the adjacent pixel surrounding at least one other second pixel closest to the second pixel includes the third pixel, and the adjacent pixel surrounding the third pixel includes the first pixel. In addition, another third pixel is not included.
The imaging device according to claim 4 is a third pixel obtained by multiplying the imaging element according to any one of claims 1 to 3 and a third component value output from the imaging element by a predetermined number of times. The fourth component value corresponding to the first color component in the third pixel is obtained when the third pixel receives the first color component with the same sensitivity as the first pixel. Sensitivity correction means for correcting the component value, the second component value in the second pixel, the first component value in the first pixel, the fourth component value in the third pixel, and the first component Generating a first component value and a second component value in each pixel of the first pixel and the second pixel based on a saturation level of the value and a magnitude relationship between the fourth component value and the saturation level; And image generating means for generating an image.
The imaging processing program according to claim 9, wherein the first pixel that receives the first color component of the subject light and outputs the first component value corresponding to the first color component; and the second pixel of the subject light A second pixel that receives the first color component and outputs a second component value corresponding to the second color component, and the first color component of the subject light is received with lower sensitivity than the first pixel. A third pixel that outputs a third component value, the number of pixels of the third pixel is smaller than any of the first pixel and the second pixel, and the third pixel By multiplying the third component value output from the image sensor not adjacent to the third pixel by a predetermined number, the third component value in the third pixel is equal to the sensitivity of the third pixel equal to that of the first pixel. The sense of correcting to the fourth component value corresponding to the first color component in the third pixel, obtained when the first color component is received at Correction processing, second component value in the second pixel, first component value in the first pixel, fourth component value in the third pixel, saturation level of the first component value, An image is generated by generating a first component value and a second component value in each pixel of the first pixel and the second pixel based on the magnitude relationship between the fourth component value and the saturation level. The image generation processing is executed by a computer.

  According to the present invention, it is possible to provide an image pickup device, an image pickup apparatus, and an image pickup processing program capable of reproducing a high luminance region while arranging low sensitivity pixels at a low density.

It is a figure which shows the structure of the imaging device in one embodiment of this invention. It is a figure which shows the structure of the image pick-up element in one embodiment of this invention. It is a figure which shows the structural example of a low sensitivity pixel. It is a flowchart which shows the imaging process in one embodiment of this invention. It is a figure for demonstrating low sensitivity pixel smoothing. It is a figure for demonstrating reference | standard color difference production | generation, RB component interpolation, and color difference interpolation. It is a figure which shows the modification of a structure of an image pick-up element. It is a figure which shows the provision form of the computer program product for the control apparatus of the imaging device by this invention for performing an imaging process.

  FIG. 1 is a diagram illustrating a configuration of an imaging apparatus 100 according to an embodiment of the present invention. The imaging device 100 includes a lens 110, an imaging element 120, and a control device 130. The lens 110 transmits subject light 510 coming from the subject 500 and forms an optical image of the subject on the image sensor 120. The image sensor 120 receives a G (green) component of the subject light 510 and outputs a G component value, and receives an R (red) component of the subject light 510 and receives an R component value. And a plurality of B (blue) pixels that receive the B (blue) component of the subject light 510 and output a B component value are two-dimensionally, for example, according to a Bayer array Be placed. As described later with reference to FIG. 2, some of the plurality of G pixels receive a G (green) component of the subject light 510 with low sensitivity and output a low-sensitivity G component value (low pixel). Sensitivity pixel). In accordance with imaging control by an imaging control unit 1310 in the control device 130 to be described later, each pixel arranged in the imaging device 120 converts each color component of the subject light 510 into each color component value by photoelectric conversion, and outputs each color component value. As a result, a RAW image (original image) is output from the image sensor 120.

  The control device 130 is a computer that executes an imaging processing program. The control device 130 executes the imaging processing program to thereby generate an imaging control unit 1310, a white balance correction unit 1320, a sensitivity correction unit 1330, a reference color difference generation unit 1340, a color component interpolation unit 1350, and a color component generation for a high luminance area. Unit 1360, reference color component selection unit 1370, and image generation unit 1380 are functionally included. Imaging control unit 1310, white balance correction unit 1320, sensitivity correction unit 1330, reference color difference generation unit 1340, color component interpolation unit 1350, high luminance area color component generation unit 1360, reference color component selection unit 1370, and image generation unit 1380 Each performs each step of the imaging process mentioned later using FIG.

  FIG. 2 is a diagram illustrating a configuration of the image sensor 120 according to the present embodiment. The image sensor 120 shown in FIGS. 2A and 2B has the same pixel arrangement, that is, represents the same image sensor. For convenience of the following description, a pixel arranged at a pixel position of a horizontal coordinate value x of the image sensor 120 and a vertical coordinate value y of the image sensor 120 is a pixel of interest, and FIG. FIG. 2B is a diagram illustrating the image sensor 120 when the pixel is a pixel, and FIG. 2B is a diagram illustrating the image sensor 120 when the pixel of interest is a B pixel.

  As shown in FIGS. 2A and 2B, for example, a quarter of the plurality of G pixels arranged according to the Bayer array together with the plurality of R pixels and the plurality of B pixels is the object light 510. It is replaced with a D pixel (low sensitivity pixel) that receives the G (green) component with low sensitivity and outputs a low sensitivity G component value. Specifically, in the horizontal row in which the R pixel is arranged, half of the plurality of G pixels arranged according to the Bayer arrangement is replaced with the D pixel, and the G pixel is placed on one of the pixel positions adjacent to the R pixel, and the other is placed on the other side. D pixels are arranged. The number of D pixels is one third of the number of G pixels and one half of the number of each of R and B pixels. D pixels are included in adjacent pixels surrounding the R pixel. D pixels are included in adjacent pixels surrounding the B pixel. The adjacent pixels surrounding the D pixel include the G pixel and do not include other D pixels. The light reception sensitivity of the G component by the D pixel is 1 / K (the predetermined number K is about 2 for example) compared to the G pixel. When the low-sensitivity G component value output by the D pixel is multiplied by a predetermined number K, the value becomes equal to the G component value output by the G pixel.

  FIG. 3 is a diagram illustrating a configuration example of the D pixel (low sensitivity pixel) 210. Similar to the G pixel, the D pixel 210 in this example includes a semiconductor substrate 220, a photoelectric conversion element 230 formed on the semiconductor substrate 220, a microlens 240, and a color filter 250. The microlens 240 transmits the subject light 510 that has passed through the lens 110. The color filter 250 has a spectral sensitivity characteristic corresponding to green, and transmits the G component of the subject light 510 that has passed through the lens 110 and the microlens 240. The photoelectric conversion element 230 receives the G component of the subject light 510 transmitted through the color filter 250 with low sensitivity, converts the received G component into an electrical signal having a low sensitivity G component value, and outputs the electrical signal. To do.

  Unlike the G pixel, the D pixel 210 has a structure for receiving the G component of the subject light 510 with low sensitivity. As some examples of such a structure, the following first to fifth structures will be described with reference to FIG. FIG. 3 illustrates all of the first to fifth structures, but the D pixel 210 may have any one or more of the first to fifth structures.

  The first structure is provided by the photoelectric conversion element 230 of the D pixel 210. The photoelectric conversion element 230 of the D pixel 210 is made smaller than the photoelectric conversion element of the G pixel so that the light reception area of the photoelectric conversion element 230 of the D pixel 210 is narrower than the light reception area of the photoelectric conversion element of the G pixel.

  The second structure is provided by the light shielding mask 260 directly above the photoelectric conversion element 230. The light shielding mask 260 narrows the opening 265 so as to cover the photoelectric conversion element 230 so that the light reception area of the photoelectric conversion element 230 of the D pixel 210 is narrower than the light reception area of the photoelectric conversion element of the G pixel.

  The third structure is provided by the light shielding member 270 disposed in the vicinity of the microlens 240. Part of the subject light 510 transmitted by the light shielding member 270 through the microlens 240 is reduced so that the amount of light received by the photoelectric conversion element 230 of the D pixel 210 is less than the amount of light received by the photoelectric conversion element of the G pixel. Shield from light.

  The fourth structure is provided by the color filter 250. The color filter 250 of the D pixel 210 so that the light amount of the G component of the subject light 510 transmitted through the color filter 250 of the D pixel 210 is smaller than the light amount of the G component of the subject light 510 transmitted through the color filter of the G pixel. Is made larger than the density of the color filter of the G pixel.

  The fifth structure is provided by the light reduction filter 280 that covers the color filter 250. Subject through which the light quantity reduction filter 280 passes through the light quantity reduction filter 280 so that the quantity of subject light 510 incident on the color filter 250 of the D pixel 210 is smaller than the quantity of subject light 510 incident on the color filter of the G pixel. The amount of light 510 is reduced.

The imaging control unit 1310 in the control device 130 performs imaging control, so that a RAW image (original image) is output from the imaging element 120. The white balance correction unit 1320 in the control device 130 preferably performs white balance correction processing on the RAW image. In a normal white balance correction process, the R pixel value (R component value of R pixel) and B pixel value (B component value of B pixel) of the RAW image are respectively multiplied by white balance correction values gainR and gainB, and then the image sensor Clip to be limited to below the saturation level. However, in the white balance correction process of the present embodiment, the clip level is expressed by the clip level expressed by the following equation (1) and output. Here, min () represents the minimum value in ().
Clip level = Image sensor saturation level x min (gainR, gainB, K) (1)

  Under normal shooting conditions, the white balance correction values gainR and gainB have a value of about 2 and the predetermined number K is 2 as described above. Therefore, according to the above white balance correction processing, the R component value and the B component As for the value, the dynamic range close to twice the conventional value can be reproduced.

  However, since the dynamic range of the G pixel value (G component value of the G pixel) is low, a false color is generated if the color difference is generated as it is. That is, in a high luminance region that is equal to or higher than the saturation level of the G component value of the G pixel, the G pixel value becomes smaller than the original G component by the amount by which the exposure amount of the G component exceeds the saturation level. Cr = RG, Cb = BG) is calculated as a larger value than the actual value.

  Therefore, in the present embodiment, the extent to which the exposure amount of the G component exceeds the saturation level is estimated using the low-sensitivity D pixel value (the low-sensitivity G component value of the D pixel), and the reference color difference By subtracting the estimated value from the correction value, correct reference color differences Cr and Cb are obtained even in the high luminance region.

  Furthermore, since both the R and G components and the reference color differences Cr and Cb are obtained in the high luminance region, the G component of the high luminance region is obtained based on them.

  In the low luminance region, the G component generated based on the G pixel value is more accurate. Therefore, it is determined whether the region is a high luminance region or a low luminance region, and an appropriate G component is selected for each.

  Even if the low sensitivity pixel (D pixel) level is designed to be 1 / K times that of the G pixel, an output error of about several percent may actually occur. Further, K times the noise of the G pixel is generated in the low-sensitivity pixel multiplied by K, and this becomes noticeable during high-sensitivity shooting. In such a case, a protruding dot-like false structure occurs in the low-sensitivity pixel, and it is preferable to remove it by MaxMin filter processing.

  FIG. 4 is a flowchart of imaging processing performed by the control device 130 included in the imaging device 100 according to the present embodiment.

  In step S310, the imaging control unit 1310 in the control device 130 exposes the optical element 120 to the optical image of the subject 500 and drives the imaging element 120 to obtain a RAW image. The image sensor 120 clips and outputs each pixel value of the RAW image so as to be limited to a saturation level or less of the image sensor.

  In step S315, the white balance correction unit 1320 in the control device 130 obtains white balance correction values gainR and gainB by a known method, and applies them to the R pixel value and the B pixel value of the RAW image, respectively. In normal white balance correction processing, the white balance correction value is applied to the R and B pixel values and then clipped so as to be limited to the saturation level of the image sensor or less. In this embodiment, the above-described equation (1) is used. Clip at the clip level and output. In this way, R and B pixel values that can be values above the saturation level are output. Assume that the R and B pixel values referred to hereinafter are the pixel values after this processing.

  In step S320, the sensitivity correction unit 1330 in the control device 130 corrects the sensitivity by multiplying the D pixel value by a predetermined number K, and clips and outputs the clip level below the same clip level as in step S315. By multiplying the low-sensitivity G component value output from the D pixel by a predetermined number K, the level becomes equal to the level of the G component value output from the G pixel and can be equal to or higher than the saturation level. It is assumed that the D pixel value referred to hereinafter is a pixel value after this processing.

  In step S325, the sensitivity correction unit 1330 in the control device 130 preferably performs low-sensitivity pixel smoothing on the K-sensitized low-sensitivity pixel value (D pixel value) output in step S320 after the sensitivity correction. For all the low-sensitivity pixel positions (x, y), a known MaxMin filter process is performed on the low-sensitivity pixel values multiplied by K with reference to adjacent G pixel values, and the result is expressed as a G component G (x , Y). As the MaxMin filter processing, for example, the maximum value and the minimum value of the G component values of the eight G pixels adjacent to the low sensitivity pixel (D pixel) at the target pixel position (x, y) indicated by a rectangular frame in FIG. And the G component value at the D pixel position may be clipped within the maximum value within the minimum value. Eight G pixels adjacent to the low-sensitivity pixel (D pixel) at the target pixel position (x, y) are shown in a circular frame in FIG. 5, and the pixel positions of these eight G pixels are (x−2, y). ), (X-1, y-1), (x, y-2), (x + 1, y-1), (x + 1, y), (x + 1, y + 1), (x, y + 2), (x-1 , Y + 1).

  In step S330, the reference color difference generation unit 1340 in the control device 130 performs, for example, the following processing on each R pixel position (x, y) illustrated in FIG. 2A and outputs the reference color difference Cr. Further, similar processing is performed for the B pixel, and the reference color difference Cb is output for the B pixel position (x, y) shown in FIG.

First, G component direction determination processing is performed for all R pixel positions (x, y) to determine whether the image structure at (x, y) is the vertical direction or the horizontal direction. As an example of simple processing, for example, using the following formulas (2) and (3), when CV-CH <-th, the image structure is determined as “vertical direction”, and when CV-CH> th The image structure is determined as “horizontal direction”, and when −th ≦ CV−CH ≦ th, the image structure is determined as “not in any direction”. As shown in FIG. 2A, the vertical direction is the y direction, and the horizontal direction is the x direction. The predetermined threshold th may be set to about several times the fluctuation width of the pixel value due to noise.
CV = | G (x, y-1) −G (x, y + 1) | (2)
CH = | G (x-1, y) −G (x + 1, y) | (3)

Next, based on the direction determination result, for example, by the following process, a reference color difference Cr is generated at all R pixel positions (x, y), and whether or not the G color component is saturated is determined. I do. When the image structure is determined to be “vertical direction”, the reference color difference Cr is generated by calculating the following equation (4). If G (x, y-1) = saturation level and G (x, y + 1) = saturation level, it is determined that the G color component is “saturated”. When G (x, y-1) = saturation level or G (x, y + 1) = saturation level, it may be determined that the G color component is “saturated”.
Cr = R (x, y)-(G (x, y-1) + G (x, y + 1)) / 2 (4)

When the image structure is determined to be “lateral direction”, the reference color difference Cr is generated by calculating the following equation (5). When G (x−1, y) = saturation level and G (x + 1, y) = saturation level, it is determined that the G color component is “saturated”. When G (x−1, y) = saturation level or G (x + 1, y) = saturation level, it may be determined that the G color component is “saturated”.
Cr = R (x, y)-(G (x-1, y) + G (x + 1, y)) / 2 (5)

When it is determined that the image structure is “not in any direction”, the reference color difference Cr is generated by calculating the following equation (6). G (x, y-1) = saturation level and G (x, y + 1) = saturation level and G (x-1, y) = saturation level and G (x + 1, y) = saturation level In this case, it is determined that the G color component is “saturated”. When G (x, y-1) = saturation level, G (x, y + 1) = saturation level, G (x-1, y) = saturation level, or G (x + 1, y) = saturation level The G color component may be determined to be “saturated”.
Cr = R (x, y)-(G (x-1, y) + G (x + 1, y) + G (x, y-1) + G (x, y + 1)) / 4 (6 )

  If the closest D pixel value of the R pixel located at the target pixel position (x, y) is less than the saturation level, or if the G color component is not determined to be “saturated”, the reference color difference is generated. The unit 1340 outputs the reference color difference Cr generated in step S334 without correction in step S335.

When it is determined that the closest D pixel value of the R pixel located at the target pixel position (x, y) is equal to or higher than the saturation level and the G color component is “saturated”, the reference color difference generation unit 1340 The reference color difference Cr generated in step S334 is corrected by the following equation (7), and the corrected reference color difference Cr is output in step S335.
Reference color difference after correction Cr = reference color difference before correction− (D pixel value in the vicinity of (x, y) −saturation level) (7)

  In step S340 and step S345, the color component interpolation unit 1350 in the control device 130 performs the following processing to generate an R component and a reference color difference Cr at all B pixel positions, all G pixel positions, and all D pixel positions. . Further, the same processing is performed for the B component and the reference color difference Cb, and the B component and the reference color difference Cb at all R pixel positions, all G pixel positions, and all D pixel positions are generated by interpolation.

The color component interpolation unit 1350 determines the direction of the R component for all B pixel positions (x, y) shown in FIG. 2B, and determines whether the image structure at all B pixel positions (x, y) is the oblique first direction. It is determined whether the direction is the diagonal second direction or not. The diagonal first direction is a direction in which y in the vertical direction decreases as the value of x in the horizontal direction increases, and y in the vertical direction increases as the value of x in the horizontal direction decreases. The oblique second direction is a direction in which y in the vertical direction increases as the value of x in the horizontal direction increases, and y in the vertical direction decreases as the value of x in the horizontal direction decreases. As simple direction determination here, the determination process using the following formula | equation (8) and (9) is performed, for example. In the case of CD1-CD2 <-th, the image structure is determined as “oblique first direction”, and in the case of CD1-CD2> th, the image structure is determined as “oblique second direction”, and −th ≦ CD1-CD2 ≦ In the case of th, it is determined that the image structure is “not in any direction”. The predetermined threshold th may be set to about several times the fluctuation width of the pixel value due to noise.
CD1 = | R (x-1, y-1) −R (x + 1, y + 1) | (8)
CD2 = | R (x-1, y + 1) -R (x + 1, y-1) | (9)

In step S340, the color component interpolation unit 1350 interpolates and generates an R component at the B pixel position (x, y), for example, by the following process based on the direction determination result. When it is determined that the image structure is “oblique first direction”, the R component at the B pixel position (x, y) is generated by interpolation using the equation (10). When it is determined that the image structure is the “diagonal second direction”, the R component at the B pixel position (x, y) is generated by interpolation using Expression (11). When it is determined that the image structure is “not in any direction”, the R component at the B pixel position (x, y) is generated by interpolation using Expression (12).
R (x, y) = (R (x-1, y-1) + R (x + 1, y + 1)) / 2 (10)
R (x, y) = (R (x-1, y + 1) + R (x + 1, y-1)) / 2 (11)
R (x, y) = (R (x-1, y-1) + R (x + 1, y + 1) + R (x-1, y + 1) + R (x + 1, y-1) ) / 4 (12)

The color component interpolation unit 1350 performs direction determination processing for all G pixel positions (x, y) and all low-sensitivity pixel positions (x, y) illustrated in FIG. 6A, and the pixel positions (x, y). It is determined whether the image structure in can be said to be either the vertical direction or the horizontal direction. As simple direction determination here, the determination process using the following formula | equation (13) and (14) is performed, for example. When CV-CH <-th, the image structure is determined as “vertical direction”. When CV-CH> th, the image structure is determined as “horizontal direction”. When -th ≦ CV-CH ≦ th, It is determined that the image structure is “not in any direction”. The predetermined threshold th may be set to about several times the fluctuation width of the pixel value due to noise.
CV = | R (x, y-1) −R (x, y + 1) | (13)
CH = | R (x-1, y) −R (x + 1, y) | (14)

In step S340, the color component interpolation unit 1350 interpolates R components of all G pixel positions (x, y) and all low sensitivity pixel positions (x, y) based on the direction determination result, for example, by the following processing. Generate. When the image structure is determined to be “vertical direction”, R components at all G pixel positions (x, y) and all low-sensitivity pixel positions (x, y) are generated by interpolation using Expression (15). When it is determined that the image structure is “lateral direction”, R components at all G pixel positions (x, y) and all low-sensitivity pixel positions (x, y) are generated by interpolation using Expression (16). When it is determined that the image structure is “not in any direction”, R components at all G pixel positions (x, y) and all low sensitivity pixel positions (x, y) are generated by interpolation using Expression (17).
R (x, y) = (R (x, y-1) + R (x, y + 1)) / 2 (15)
R (x, y) = (R (x-1, y) + R (x + 1, y)) / 2 (16)
R (x, y) = (R (x-1, y) + R (x + 1, y) + R (x, y-1) + R (x, y + 1)) / 4 (17)

  The color component interpolation unit 1350 interpolates and generates B components at the R pixel position (x, y), all G pixel positions (x, y), and all low sensitivity pixel positions (x, y) by the same process as described above. To do.

In step S345, the color component interpolation unit 1350, based on the image structure direction determination result at all the B pixel positions (x, y) described above, for example, the reference color difference Cr at the B pixel position (x, y) by the following processing. Is generated by interpolation. When it is determined that the image structure is “oblique first direction”, the reference color difference Cr at the B pixel position (x, y) is generated by interpolation using Expression (18). When it is determined that the image structure is the “diagonal second direction”, the reference color difference Cr at the B pixel position (x, y) is generated by interpolation using the equation (19). When it is determined that the image structure is “not in any direction”, the reference color difference Cr at the B pixel position (x, y) is generated by interpolation using the equation (20).
Cr (x, y) = (Cr (x-1, y-1) + Cr (x + 1, y + 1)) / 2 (18)
Cr (x, y) = (Cr (x-1, y + 1) + Cr (x + 1, y-1)) / 2 (19)
Cr (x, y) = (Cr (x-1, y-1) + Cr (x + 1, y + 1) + Cr (x-1, y + 1) + Cr (x + 1, y-1) ) / 4 (20)

In step S345, the color component interpolating unit 1350 performs, for example, the following process based on the direction determination result of the image structure at all G pixel positions (x, y) and all low-sensitivity pixel positions (x, y). A reference color difference Cr at (x, y) is generated by interpolation. When it is determined that the image structure is “vertical direction”, the reference color difference Cr at all G pixel positions (x, y) and all low sensitivity pixel positions (x, y) is generated by interpolation using Expression (21). When it is determined that the image structure is “horizontal direction”, the reference color difference Cr at all G pixel positions (x, y) and all low sensitivity pixel positions (x, y) is generated by interpolation using Expression (22). When it is determined that the image structure is “not in any direction”, the reference color difference Cr at all G pixel positions (x, y) and all low-sensitivity pixel positions (x, y) is generated by interpolation using Expression (23).
Cr (x, y) = (Cr (x, y-1) + Cr (x, y + 1)) / 2 (21)
Cr (x, y) = (Cr (x-1, y) + Cr (x + 1, y)) / 2 (22)
Cr (x, y) = (Cr (x-1, y) + Cr (x + 1, y) + Cr (x, y-1) + Cr (x, y + 1)) / 4 (23)

  The color component interpolation unit 1350 interpolates the reference color difference Cb at the R pixel position (x, y), all G pixel positions (x, y), and all low sensitivity pixel positions (x, y) by the same process as described above. Generate.

  In step S350, the high-luminance region color component generation unit 1360 in the control device 130 calculates the first high-luminance region G component by subtracting the reference color difference Cr from the R component for all pixel positions. For all pixel positions, the second G component for the high brightness area is calculated by subtracting the reference color difference Cb from the B component. Then, the first and second high luminance region G components are averaged to generate a high luminance region G component.

  In step S355, the color component interpolation unit 1350 in the control device 130 obtains and outputs G components at all R, B, G, and D pixel positions (x, y). For all RB pixel positions (x, y), the color component interpolation unit 1350 performs the following interpolation processing.

First, the color component interpolation unit 1350 determines whether the image structure related to the G component at all RB pixel positions (x, y) is the vertical direction or the horizontal direction. As an example of simple processing, for example, using the following formulas (24) and (25), when CV-CH <-th, the image structure is determined as “vertical direction”, and CV-CH> th The image structure is determined as “horizontal direction”, and when −th ≦ CV−CH ≦ th, the image structure is determined as “not in any direction”. As shown in FIGS. 2A and 2B, the vertical direction is the y direction and the horizontal direction is the x direction. The predetermined threshold th may be set to about several times the fluctuation width of the pixel value due to noise.
CV = | G (x, y-1) −G (x, y + 1) | (24)
CH = | G (x-1, y) −G (x + 1, y) | (25)

Next, based on the direction determination result, G components at all RB pixel positions (x, y) are generated by interpolation, for example, by the following process. When it is determined that the image structure is “vertical direction”, the G component at the pixel position (x, y) is generated by interpolation using Expression (26). When it is determined that the image structure is “lateral direction”, the G component at the pixel position (x, y) is generated by interpolation using the equation (27). When it is determined that the image structure is “not in any direction”, the G component at the pixel position (x, y) is generated by interpolation using Expression (28).
G (x, y) = (G (x, y-1) + G (x, y + 1)) / 2 (26)
G (x, y) = (G (x-1, y) + G (x + 1, y)) / 2 (27)
G (x, y) = (G (x-1, y) + G (x + 1, y) + G (x, y-1) + G (x, y + 1)) / 4 (28)

  In step S355, for the G pixel position, the color component interpolation unit 1350 in the control device 130 outputs the G component value of the G pixel arranged in the G pixel position obtained in step S310. For the D pixel position, the G component value at the D pixel position obtained in step S325 is output. In this way, G components at all pixel positions of the image sensor 120 can be generated.

  In step S360, the reference color component selection unit 1370 in the control device 130 selects and outputs the G component for the pixel position where the G component generated in step S355 is less than the saturation level. The reference color component selection unit 1370 selects and outputs the high-luminance region G component generated in step S350 for the pixel position where the G component generated in step S355 reaches the saturation level. In this way, the G component at all R, B, G, and D pixel positions (x, y) is determined.

In step S365, the image generation unit 1380 in the control device 130 is interpolated in step S345 with the reference color difference Cr in all R pixel positions and the reference color difference Cb in all B pixel positions generated in steps S330 and S335. The reference color difference Cr at all B, G pixel positions, the reference color difference Cb at all R, G pixel positions, and all the R, B, G, D pixel positions (x, y) selected in step S360. Based on the G component, RB components are generated at all pixel positions by the following equations (29) and (30).
R = Cr + G (29)
B = Cb + G (30)

  In step S365, the image generation unit 1380 generates an RGB image based on the G component, R component, and B component at all pixel positions generated as described above. The image generation unit 1380 further performs well-known gradation correction (not shown), saturation enhancement, edge enhancement, and the like on the generated RGB image and outputs the result.

  The image sensor 120 according to the present embodiment has G pixels that receive the G component (green component) of the subject light 510 and output a G component value corresponding to the G component. The image sensor 120 receives the R component (red component) and the B component (blue component) of the subject light, respectively, and the R component value and the B component value corresponding to the R component (red component) and the B component (blue component), respectively. R and B pixels that respectively output. The image sensor 120 has a D pixel that receives the G component of the subject light 510 with a lower sensitivity than the G pixel and outputs a low-sensitivity G component value. The number of D pixels is smaller than any of G pixels, R pixels, and B pixels. The adjacent pixel surrounding the R pixel or the adjacent pixel surrounding at least one other R pixel closest to the R pixel includes the D pixel. The adjacent pixel surrounding the B pixel or the adjacent pixel surrounding at least one other B pixel closest to the B pixel includes the D pixel. The adjacent pixels surrounding the D pixel include the G pixel and do not include other D pixels. Since the D pixel is positioned in the vicinity of the R pixel and the B pixel, it is easy to detect saturation at the pixel position of the R pixel and the B pixel by referring to the D pixel in the vicinity thereof. Since at least two G pixels, not necessarily D pixels, are always adjacent to the periphery of the D pixel, low-sensitivity pixel smoothing with respect to the pixel value of the D pixel can be performed in step S325. Since the number of D pixels is smaller than that of R pixels, B pixels, and G pixels, the influence on the reproducibility of the low luminance region is small.

  The imaging device 100 in the present embodiment includes the above-described imaging element 120 and the control device 130. The control device 130 includes a sensitivity correction unit 1330. The sensitivity correction unit 1330 multiplies the low-sensitivity G component value output from the image sensor 120 by a predetermined number K, thereby reducing the low-sensitivity G component value in the D pixel. The G component value corresponding to the G component in the D pixel, which is obtained when the D pixel receives the G component with the same sensitivity as the G pixel, is corrected. The control device 130 includes an image generation unit 1380. The image generation unit 1380 includes the R component value and the B component value in the R pixel and the B pixel, the G component value in the G pixel, and the sensitivity correction in the D pixel. Based on the G component value, the saturation level of the G component value, and the magnitude relationship between the G component value after sensitivity correction in the D pixel and the saturation level of the G component value, each of the G pixel, R pixel, and B pixel An image is generated by generating a G component value, an R component value, and a B component value. Since the saturation of the G component value is taken into account, it is possible to maintain the reproducibility of the high luminance region in image generation.

  The imaging apparatus 100 according to the present embodiment further includes a reference color difference generation unit 1340. The reference color difference generation unit 1340 generates reference color differences Cr and Cb in the R pixel and the B pixel based on the R component value in the R pixel, the B component value in the B pixel, and the G component value in the G pixel. Based on the reference color differences Cr and Cb in the pixel, the reference color differences Cr and Cb in the G pixel are generated. The reference color difference generation unit 1340, when the G component value after sensitivity correction in the D pixel near the R pixel and the B pixel is equal to or higher than the saturation level, and the reference color difference Cr in the R pixel and the B pixel by the reference color difference generation unit 1340 When all the G component values used to generate Cb have reached the saturation level, the R pixel is determined based on the difference between the G component value after sensitivity correction and the saturation level in the D pixel in the vicinity of the R pixel and the B pixel. And the reference color differences Cr and Cb in the B pixel are corrected. The image generation unit 1380 calculates the R component value and the B component value in each pixel of the G pixel, the R pixel, and the B pixel based on the G component value and the reference color differences Cr and Cb in each pixel of the G pixel, the R pixel, and the B pixel. To generate. Since the saturation of the G component value is taken into account, it is possible to maintain the reproducibility of the high luminance region in image generation.

  The configuration of the image sensor 120 in the present embodiment is not limited to the configuration shown in FIG. FIG. 7 is a diagram illustrating a modified example of the configuration of the image sensor 120. In FIG. 7A, with respect to the R pixel that is the target pixel arranged at the pixel position (x, y) of the Bayer array, three adjacent pixel positions (x The G pixels in −3, y) and (x + 3, y) are replaced with D pixels. In addition, the G pixel at the pixel positions (x, y−3) and (x, y + 3) adjacent to each other in the vertical direction on both sides of the same column as the target pixel is replaced with the D pixel. The number of D pixels is smaller than any of G pixels, R pixels, and B pixels. The adjacent pixels surrounding the R pixel as the target pixel do not include the D pixel, but the pixel position (x−2, y), (x + 2) as at least one other R pixel closest to the R pixel as the target pixel , Y), (x, y−2) or adjacent pixels surrounding (x, y + 2) include D pixels.

  In FIG. 7B, for the R pixel that is the pixel of interest arranged at the pixel position (x, y) in the array shown in FIG. 2A, the D pixel that is arranged in the same row as the pixel of interest. Is returned to the G pixel. That is, with respect to the row in which the R pixel is arranged, the row in which the D pixel is arranged and the row in which the D pixel is not arranged are alternately arranged in the vertical direction every other row. The number of D pixels is smaller than any of G pixels, R pixels, and B pixels. The adjacent pixels surrounding the R pixel that is the target pixel do not include the D pixel, but the pixel position (x, y−2) or (x) that is at least one other R pixel closest to the R pixel that is the target pixel , Y + 2) includes D pixels in adjacent pixels.

  As shown in FIG. 8, the computer program product for the control device 130 of the imaging device 100 to execute the imaging processing in each of the above-described embodiments and modifications is a program storage medium 2000 (for example, a CD-ROM) or communication. The data processing apparatus 1500 (for example, a PC) is provided through a data signal transmitted through the network 4000 (for example, the Internet). The imaging device 100 reads an imaging processing program from the data processing device 1500 via communication.

  The data processing device 1500 is provided with an imaging processing program via the program storage medium 2000. Alternatively, the imaging processing program may be provided in the following supply form. The data processing device 1500 has a connection function with the communication network 4000. The server 3000 is a computer that provides an imaging processing program, and stores the imaging processing program in a storage medium such as a hard disk. The server 3000 places the imaging processing program read from the hard disk on a carrier wave as a data signal, and transfers it to the data processing device 1500 via the communication network 4000. As described above, the imaging processing program is preferably supplied by various forms of computer program products including provision as data signals via the program storage medium 2000 and the communication network 4000.

100 imaging device, 110 lens, 120 imaging device, 130 control device,
210 low-sensitivity pixel, 220 semiconductor substrate, 230 photoelectric conversion element,
240 micro lens, 250 color filter,
260 shading mask, 265 opening,
270 light shielding member, 280 light quantity reduction filter,
500 subjects, 510 subject light,
1310 Imaging control unit, 1320 White balance correction unit, 1330 Sensitivity correction unit,
1340 reference color difference generation unit, 1350 color component interpolation unit,
1360 color component generation unit for high brightness area, 1370 reference color component selection unit,
1380 image generator,
1500 data processing device, 2000 program storage medium,
3000 servers, 4000 communication networks

Claims (9)

A first pixel that receives a first color component of subject light and outputs a first component value corresponding to the first color component;
A second pixel that receives a second color component of subject light and outputs a second component value corresponding to the second color component;
A third pixel that receives the first color component of the subject light with a lower sensitivity than the first pixel and outputs a third component value;
The number of pixels of the third pixel is less than any of the first pixel and the second pixel,
The adjacent pixel surrounding the second pixel, or the adjacent pixel surrounding at least one other second pixel closest to the second pixel includes the third pixel,
The imaging device, wherein the adjacent pixel surrounding the third pixel includes the first pixel and does not include the other third pixel. The imaging device according to claim 1,
The third pixel receives the first color component of the subject light in a light receiving region that is narrower than the first pixel, thereby receiving the third component value with low sensitivity. An image sensor characterized by the above. The imaging device according to claim 1,
The third pixel includes a light amount reduction filter, and transmits the first color component of the subject light through the light amount reduction filter with low sensitivity and outputs the third component value. An image pickup device characterized by: The imaging device according to any one of claims 1 to 3,
By multiplying the third component value output from the image sensor by a predetermined number of times, the third component value in the third pixel is obtained with the sensitivity that the third pixel is equal to the first pixel. Sensitivity correction means for correcting to a fourth component value corresponding to the first color component in the third pixel, obtained when the first color component is received;
Saturation of the second component value in the second pixel, the first component value in the first pixel, the fourth component value in the third pixel, and the first component value Based on the level and the magnitude relationship between the fourth component value and the saturation level, the first component value and the second component value in each pixel of the first pixel and the second pixel are An imaging apparatus comprising: an image generation unit configured to generate an image by generating the image. The imaging apparatus according to claim 4,
A reference color difference in the second pixel is generated based on the second component value in the second pixel and the first component value in the first pixel, and the reference color difference in the second pixel is generated. Further comprising color difference generating means for generating a reference color difference in the first pixel based on
The color difference generation unit is configured such that when the fourth component value in the third pixel in the vicinity of the second pixel is equal to or higher than the saturation level, and the reference color difference in the second pixel by the color difference generation unit When all of the first component values used for generating the saturation level have reached the saturation level, the difference between the fourth component value and the saturation level in the third pixel in the vicinity of the second pixel And correcting the reference color difference in the second pixel,
The image generation means calculates the second component value in each pixel of the first pixel and the second pixel, and the first component value in each pixel of the first pixel and the second pixel. And an image pickup device generated based on the reference color difference. The imaging apparatus according to claim 5,
The second component value corresponding to the second color component in the first pixel is output by the image sensor in the second pixel in the vicinity of the first pixel. And generating the first component value corresponding to the first color component in the second pixel by the imaging device in the first pixel in the vicinity of the second pixel. Color component interpolation means for generating an interpolation based on the first component value,
The second component value corresponding to the second color component in the first pixel, which is generated by interpolation by the color component interpolation generation means, and the second pixel value output by the image sensor. Based on the second component value corresponding to the second color component and the reference color difference in each pixel of the first pixel and the second pixel, the first pixel and the second pixel High-luminance area color component generating means for generating a high-luminance area color component corresponding to the first color component in each pixel of
In the first pixel, the first component value corresponding to the first color component output by the image sensor is interpolated and generated by the color component interpolation means in the second pixel. When the first component value corresponding to one color component does not reach the saturation level, the first component value is set. When the first component value reaches the saturation level, the high value is set. A reference color component selection means for selecting a color component value for the high luminance area of the color component for the luminance area as a reference color component;
The image generation means is configured to determine the first pixel in each pixel of the first pixel and the second pixel based on the reference color difference and the reference color component in each pixel of the first pixel and the second pixel. An image pickup apparatus that generates the image by generating a component value and the second component value. In the imaging device according to claim 5 or 6,
The fourth component value in the third pixel used when the reference color difference is generated by the color difference generation unit is referred to the first pixel around the third pixel, and MaxMin filter processing is performed. An image pickup apparatus, further comprising: a low-sensitivity pixel smoothing unit that performs clipping. In the imaging device according to any one of claims 4 to 7,
An imaging apparatus, further comprising: a white balance correction unit that corrects the second component value by multiplying the second component value output from the imaging element by a white balance correction value. A first pixel that receives a first color component of subject light and outputs a first component value corresponding to the first color component; and a second pixel that receives a second color component of subject light and A second pixel that outputs a second component value corresponding to the first color component, and a third component value that receives the first color component of the subject light with lower sensitivity than the first pixel. And the third pixel has a smaller number of pixels than either the first pixel or the second pixel, and the third pixel is the other of the other pixels. The third component value output from the image sensor not adjacent to the third pixel is multiplied by a predetermined number, whereby the third component value in the third pixel is determined by the third pixel. The first color component in the third pixel obtained when the first color component is received with the same sensitivity as the pixel A sensitivity correction process for correcting the fourth component value corresponding,
Saturation of the second component value in the second pixel, the first component value in the first pixel, the fourth component value in the third pixel, and the first component value Based on the level and the magnitude relationship between the fourth component value and the saturation level, the first component value and the second component value in each pixel of the first pixel and the second pixel are An imaging processing program that causes a computer to execute image generation processing for generating an image by generating the image.

Images