JP2018195961A - Image signal processing method, image signal processing device, and imaging device - Google Patents

Abstract

An object of the present invention is to provide an image signal processing method and an imaging apparatus that reduce the time required to read an image signal by reducing the data amount of the image signal while using a layered image signal and suppressing the occurrence of false colors. can do. Kind Code: A1 An image signal processing method for processing a stacked image signal detected using a stacked solid-state imaging device in which three or more photoelectric conversion layers are stacked on a substrate in one pixel, wherein the stacked solid-state image signal An image signal readout unit that reads image signals of two layers for each pixel of the image sensor, and an image signal of three or more layers detected by the photoelectric conversion layer laminated on the pixel and the image signals of the two layers read by the image signal readout unit. an image signal estimating unit for estimating an image signal in a layer of a difference from the read image signal, wherein the estimated image signal estimated by the image signal estimating unit is the second image signal read by the image signal reading unit. The image signal processing, wherein the two-layer read-out image signal estimated using the layer read-out image signal and read out by the image signal reading unit is a combination of two or more sets of the photoelectric conversion layers. Method. [Selection drawing] Fig. 1

Description

  The present invention relates to an image signal processing method, an image signal processing device, and an imaging device that use a plurality of photoelectric conversion layers having a plurality of light receiving portions and process image signals read from the respective light receiving portions of the photoelectric conversion layers.

  Recently, there is an increasing demand for recording high-quality movies and images. That is, it is necessary to record an image in which a false color is suppressed while leaving a sense of resolution.

  In view of this, Patent Document 1 and Patent Document 2 disclose inventions related to image signal processing and an imaging apparatus that suppresses false colors while leaving a sense of resolution.

  Patent Document 1 discloses that the same pixel is formed by laminating three layers of a photoelectric conversion layer that detects red (R), a photoelectric conversion layer that detects green (G), and a photoelectric conversion layer that detects blue (B). A multilayer solid-state imaging device capable of simultaneously detecting three color signals of red (R), green (G), and blue (B) is disclosed.

  In Patent Document 2, R (red), Gr (green positioned next to R in one direction of the pixel array), Gb (B in the one direction) output via an image sensor having a Bayer array color filter. The luminance (Y) is generated from the sum of the R, Gr, Gb, and B pixel signals, and the R image signal is determined from the sum of the Gr and Gb image signals. And the B image signal are subtracted to generate a first color difference, and a difference between the R image signal and the B image signal is calculated to generate a second color difference, and the luminance value, the first color difference, and the second color difference are calculated. By performing false color suppression for each of the colors and generating color information by converting the first color difference and the second color difference in which the false color is suppressed into a predetermined color space, the false color can be effectively prevented without losing resolution. An invention is disclosed that can suppress the image quality and can generate a high-quality color image.

US Pat. No. 5,965,875

JP 2009-211962 A

  As the amount of data read from the image sensor increases, the time required for reading increases. As a result, since it is necessary to process the increased data, problems such as an increase in power consumption and a rolling shutter phenomenon occur. In addition, when attempting to record a high-quality moving image or image, the amount of data increases. As a result, similarly, the time required for reading from the image sensor increases, power consumption increases, and a rolling shutter phenomenon occurs.

  The invention described in Patent Document 1 can simultaneously detect three color signals of red (R), green (G), and blue (B) with the same pixel. Therefore, there is no reduction in resolution. Unlike the Bayer type solid-state imaging device, light is not detected only in the light receiving portion of the color corresponding to the color filter and the other colors are not interpolated, so that no false color is generated. In other words, it is possible to obtain an image with no sense of false color and leaving a sense of resolution. On the other hand, since the multilayer solid-state imaging device described in Patent Document 1 can detect three colors with one pixel, in the case of an imaging device having the same number of pixels, an image signal (hereinafter, referred to as a multilayer solid-state imaging device) is detected. The stacked image signal) has a larger amount of information than the image signal detected by the Bayer solid-state imaging device (hereinafter referred to as Bayer image signal). Therefore, there is a problem that the time required for reading the image signal increases.

  The invention described in Patent Document 2 uses a Bayer type solid-state imaging device. In the Bayer type solid-state imaging device, one light receiving portion is arranged for one pixel, and each light receiving portion is arranged with one of filters from red (R), green (G), and blue (B), and each light receiving portion. Each detect one color of light. Therefore, the color information other than the color filters stacked in each light receiving unit is obtained by interpolation and calculation from the image signal detected by the peripheral light receiving unit. For this reason, there is a problem that the color becomes false.

  The present invention has been made in view of such a situation. While using a stacked image signal, the amount of image signal data is reduced, thereby reducing the time required to read out an image signal, and false color. An object of the present invention is to provide an image signal processing method, an image signal processing apparatus, and an imaging apparatus that also suppress the above-described problem.

A first invention for solving the above-described problem is to process a stacked image signal detected using a stacked solid-state imaging device in which three or more photoelectric conversion layers are stacked on the top of a substrate. In the image signal processing method, for each pixel of the stacked solid-state imaging device, an image signal reading unit that reads out two layers of image signals, three or more image signals detected by a photoelectric conversion layer stacked on the pixels, and the An image signal estimation unit that estimates an image signal of a layer different from the read image signal of the two layers read by the image signal reading unit, and the estimated image signal estimated by the image signal estimation unit is the image Estimated using the read image signal of the two layers read by the signal reading unit,
2. The image signal processing method according to claim 1, wherein the read image signals of the two layers read by the image signal reading unit are a combination of two or more sets of the photoelectric conversion layers.

  A second invention for solving the above-described problem is an image signal processing method according to the first invention, wherein the stacked solid-state imaging device includes three photoelectric conversion layers, and each photoelectric conversion layer The image signal processing method is characterized in that the color detected is red (R), green (G), and blue (B).

  A third invention for solving the above-described problem is an image signal processing method according to the first or second invention, wherein 1 of the read image signals of the two layers read to the image signal read unit is 1 The image signal of the layer is a green (G) image signal processing method.

  A fourth invention for solving the above-described problem is an image signal processing method according to any one of the first to third inventions, wherein the estimated image signal is transmitted from a peripheral pixel of the pixel to the image signal. An image signal processing method characterized by being estimated using an image signal having the same color as the estimated image signal among the read image signals of the two layers read by the reading unit.

  A fifth invention for solving the above-described problem is an image signal processing method according to any one of the first to third inventions, wherein the estimated image signal is generated from the peripheral pixels of the pixel by the pixel. Calculating the color difference of the two-layer image signal between the image signal having the same color as the one of the two-layer read image signal read by the image signal reading unit and the image signal having the same color as the estimated image signal, By adding an average value of color differences to an image signal having the same color as one of the read image signals of the two layers read from the pixel, the image signal estimation unit has the photoelectric conversion layer stacked on the pixel. An image signal processing method characterized by estimating an image signal detected but not read by the image signal reading unit.

  A sixth invention for solving the above-described problem is an image signal processing method according to any one of the first to third inventions, wherein the estimated image signal is transmitted from the peripheral pixels of the pixel to the pixel. The color ratio of the image signal of the two layers of the image signal of the same color as the one of the read image signals of the two layers read by the image signal reading unit and the image signal of the same color as the estimated image signal is calculated for each pixel. The image signal estimation unit detects the photoelectric conversion layer stacked on the pixel by adding the average value of the color ratio to one of the two layers of read image signals read from the pixel. An image signal processing method, wherein the image signal reading unit estimates an image signal that has not been read.

  A seventh invention for solving the above-described problems is an imaging apparatus, comprising the image signal processing method according to any one of the first to sixth inventions.

  An eighth invention for solving the above-described problem is an image signal processing device, which is detected using a stacked solid-state imaging device in which three or more photoelectric conversion layers are stacked on the top of a substrate. In the image signal processing apparatus that processes a stacked image signal, an image signal reading unit that reads out two layers of image signals for each pixel of the stacked solid-state imaging device, and the 2 that is read out by the image signal reading unit. An image signal rearrangement unit that rearranges the read image signals of the layers into Bayer type image signals, and two or more sets of the image signals of the two layers read by the image signal read unit An image signal processing apparatus characterized by the combination of

  According to the present invention, an image signal processing method and an image that reduce generation of a false color while reducing the time required to read out the image signal by reducing the data amount of the image signal while using the stacked image signal. A signal processing device and an imaging device can be provided.

It is the block diagram which showed the structure of the imaging device to which the image processing method which concerns on this invention is applied. It is a flowchart regarding Example 1 of the image signal of this invention. It is a flowchart regarding Example 2 of this invention. It is a figure explaining image signal rearrangement It is a flowchart explaining image signal estimation. It is a figure explaining the said pixel and a surrounding pixel.

  The best mode for carrying out the present invention will be described below with reference to the accompanying drawings. In addition, this invention is not limited by this embodiment.

  FIG. 1 is a configuration diagram showing a configuration of an imaging apparatus to which an image processing method according to the present invention is applied.

  Reference numeral 100 denotes an imaging apparatus. The imaging apparatus 100 according to the present embodiment is a digital camera, captures a subject, and stores the captured image data. The imaging apparatus 100 has a configuration in which the optical system 200 can be replaced.

  Reference numeral 101 denotes a stacked solid-state imaging device. The stacked solid-state imaging device 101 is a CMOS or CCD that converts light detected through the optical system 200 into an image signal.

  In this embodiment, the stacked solid-state imaging device 101 uses a stacked solid-state imaging device in which three photoelectric conversion layers are stacked in one pixel. However, the photoelectric conversion layer stacked on one pixel of the stacked solid-state imaging device 101 is not limited to three layers, and may be four layers, five layers, or more.

  Here, the image signal will be described.

  An image signal detected by the multilayer solid-state imaging device is defined as a multilayer image signal. An image signal detected by the Bayer type solid-state imaging device is a Bayer type image signal. The solid-state image sensor of the present invention is a stacked image sensor. Therefore, the image signal detected by the stacked solid-state imaging device is a stacked image signal. The laminated image signal detected by the laminated solid-state imaging device 101 can be converted from a laminated image signal to a Bayer image signal by performing signal processing by image signal conversion in the FPGA 102 or the like. is there.

  Reference numeral 1011 denotes a read control unit. The read control unit 1011 outputs a signal for controlling the drive timing of the stacked solid-state imaging device 101. Thus, horizontal driving and vertical driving for each pixel are controlled, and two layers of image signals from each pixel are read out.

  Reference numeral 102 denotes an FPGA. The FPGA 102 constitutes an image signal estimation unit 1021, an image signal synthesis unit 1022, and an image signal rearrangement unit 1024 with internal circuits. An ASIC or the like can be used as an intermediate device instead of the FPGA 102.

  Reference numeral 1021 denotes an image signal estimation unit. The image signal estimation unit 1021 estimates the missing one-layer image signal using the two-layer image signal read out by each pixel. A method for estimating the image signal will be described later.

  Reference numeral 1022 denotes an image signal synthesis unit. The image signal combining unit 1022 combines the two-layer image signal read by the read control unit 1011 and the one-layer image signal estimated by the image signal estimation unit 1021 into one. The synthesized image signal is a three-layer image signal. Therefore, a stacked image signal is obtained.

  Reference numeral 1024 denotes an image signal rearrangement unit. The image signal rearrangement unit 1024 converts the two-layer stacked image signal read by the read control unit 1011 into a Bayer image signal.

  Reference numeral 103 denotes a CPU. The CPU 103 configures a signal processing unit 1031 inside. Specifically, for the image signal detected by the stacked solid-state imaging device 101 and subjected to various image signal processes configured in the FPGA 102, the signal processing unit 1031 performs various image signal processes such as gamma correction and noise. Remove noise. Further, various devices such as an AE mechanism and an AF mechanism (not shown) are also controlled.

  Reference numeral 104 denotes an SDRAM. The SDRAM 104 is connected to the CPU 103. The SDRAM 104 is also connected to the DSP 107.

  Reference numeral 105 denotes an external storage device. The external storage device 105 is a medium for storing captured images. Examples of the external storage device 105 are represented by, for example, an SD memory card (registered trademark), a multimedia card (registered trademark), an xD picture card (registered trademark), and a smart media (registered trademark) that are detachable from the imaging device. Various recording media such as a semiconductor memory card, a portable small hard disk, a magnetic disk, and a magneto-optical disk can be used.

  Reference numeral 106 denotes an external display device. The external display device 106 can display an image immediately after shooting, an image read from the external storage device 105, and the like. The external display device 106 also displays various menu screens for manual setting such as camera operation mode, white balance, image pixel count, and sensitivity, and a user interface that allows manual setting according to user operations. Display the screen for As the external display device 106, for example, a liquid crystal or an organic EL can be used.

  Reference numeral 107 denotes a DSP. The DSP 107 is connected to the SDRAM 104 and performs image signal processing and the like.

  Reference numeral 200 denotes an optical system. The optical system 200 includes a lens CPU and a diaphragm (not shown) in addition to the lens optical system.

  The image signal processing performed by the imaging apparatus according to the present invention will be described with reference to FIG. 2 and FIG. 2 is a flowchart for the first embodiment, and FIG. 3 is a flowchart for the second embodiment.

  In step # 1, the readout control unit 1011 controls an image signal read from each pixel by the multilayer solid-state imaging device 101. The control of the reading control unit 1011 in step # 1 will be described later.

  In step # 2, the image signal rearrangement unit 1024 rearranges the image signal read from each pixel from the stacked solid-state imaging device 101 under the control of the readout control unit 1011 in step # 1 into a Bayer type image signal. A specific image signal rearrangement method will be described later.

  In step # 4, the image signal rearranged into the Bayer image signal is sent from the image signal rearrangement unit 1024 in step # 2. The signal processing unit 1031 performs image signal processing. A specific image signal processing method will be described later.

  In step # 6, the SDRAM 104 temporarily stores the image signal processed in various image signals in step # 4. The data is read from the SDRAM 104, displayed as a preview image on the external display device 106, and stored as a captured image in the external storage device 105.

  Next, step # 3 “image signal estimation” and step # 5 “various signal processing” of the second embodiment will be described instead of step # 2 “image signal rearrangement” of the first embodiment.

  In step # 3, the image signal estimation unit 1021 estimates a one-layer image signal that is deficient from the two-layer image signal sent from the multilayer solid-state imaging device 101 under the control of the readout control unit 1011. Among the pixels adjacent to the pixel of the image signal to be estimated (hereinafter referred to as the pixel), an image signal of the same color as the image signal of the one layer in which the pixel is insufficient is used. Estimate the image signal of the layer. A specific image signal estimation method will be described in the second embodiment.

  In step # 5, the one-layer image signal estimated by the image signal estimation unit 1021 in step # 3 and the two-layer image signal read by the read control unit 1011 are sent, and the image signal synthesis unit 1022 is sent. After the two signals are combined, various image signal processings are performed by the image signal processing unit 1031 and the DSP 107.

  The image signal readout # 1, image signal rearrangement # 2, and image signal processing # 4 in the flow of the present invention shown in FIG. 2 described above will be described in more detail below with reference to FIG.

  Hereinafter, the reading of the image signal in step # 1 will be described.

  In the control of the reading control unit 1011 in step # 1, two layers of image signals are selected and read out from the three layers of image signals in each pixel of the multilayer solid-state imaging device 101. There are three groups of combinations of two-layer image signals to be selected: red (R) and green (G), blue (B) and green (G), and red (R) and blue (B). From these three groups, two layers of image signals of green (G) and red (R) from pixels in all columns in odd rows and blue (B) and green (G) from pixels in all columns in even rows Are read out respectively. Green (G) and red (R) are group 1 and blue (B) and green (G) are group 2.

  In step # 1, the reading control unit 1011 has completed reading of two layers of image signals from all pixels, and thus the process proceeds to step # 2.

  Next, the image signal rearrangement in step # 2 will be described with reference to FIG. Note that the numbers in FIG. 4 represent coordinates, and the coordinates are represented by (row, column).

  In step # 2, the arrangement of the image signals read from the respective pixels of the stacked solid-state imaging device 101 is rearranged by the read control unit 1011 in step # 1. FIG. 4A is a diagram illustrating a part of a state in which the readout control unit 1011 reads out two layers of image signals from each pixel of the multilayer solid-state imaging device 101. The image signal rearrangement unit 1024 rearranges these two layers of image signals into a Bayer-type image signal array as shown in FIG.

  A specific sorting method will be described. The green color (G) and the red color (R) stacked on the coordinates (1, 1) in FIG. 4A are red (R) and the coordinates (1, 1) in FIG. Green (G) is respectively arranged in 2). Next, blue (B) and green (G) stacked on the coordinates (2, 1) in FIG. 4A are green (G) and coordinates (2, 1) in FIG. Blue (B) is arranged at (2, 2). Similarly, green (G) and red (R) stacked at coordinates (1, 2) in FIG. 4A are red (R), and coordinates (1, 3) in FIG. Blue (B) and green (G) stacked on the coordinates (2, 2) in FIG. 4A are arranged in the coordinates (1, 4) in FIG. 2, 3) are arranged in green (G), and the coordinates (2, 4) are arranged in blue (B).

  As described above, the image signal rearrangement unit 1024 rearranges the stacked two-layer image signals into Bayer-type image signals for all the pixels read from the multilayer solid-state imaging device 101. When rearranged, in the two-layer image signal of FIG. 4 (a), the image signal of 2 × (pixels) × 2 (pixels) in length × width becomes the Bayer type image signal of FIG. 4 (b). The vertical × horizontal is an image signal of 2 (pixels) × 4 (pixels). That is, it becomes a horizontally long image signal.

  In this embodiment, when rearranging the stacked two-layer image signals into Bayer-type image signals, the image signals are expanded in the row direction. However, in the same manner as when the two layers of image signals read from each pixel by the reading control unit 1011 are rotated 90 ° counterclockwise from the present embodiment in step # 1, blue (( B), green (G), and two-layer image signals of green (G) and red (R) are read from pixels in all rows of even columns, respectively, and then the two-layer image signals are developed in the column direction to be Bayer. It is also possible to rearrange them into image signals of a type.

  Since the image signal rearrangement unit 1024 has completed the rearrangement of the image signals of all the pixels, the process proceeds to step # 4.

  In order to rearrange into Bayer-type image signals, green (G), blue (B), and red (R) must be in a ratio of 2: 1: 1. Therefore, the pattern of the image signal read by the reading control unit 1011 in step # 1 is 2 of the two types of image signals of the blue (B) and green (G) patterns and the green (G) and red (R) patterns. It is a pattern.

  Next, step # 4 will be described.

  In step # 4, the image signal processing unit 1031 performs suitable image signal processing on the image signal sent from step # 2. Specifically, the image signals sent from step # 2 are rearranged by the image signal rearrangement unit 1024 to become a Bayer-type image signal in FIG. However, the rearranged Bayer-type image signal of this embodiment is horizontally long because it is developed in the row direction. Therefore, the image signal processing 1031 adjusts the vertical and horizontal directions by performing adjustment processing. Also, when rearranging into a Bayer type image signal in step # 2, unlike the present embodiment, if it is expanded in the column direction, it becomes a vertically long image signal, but the image is the same as in the case of a horizontally long image signal. The signal processing unit 1031 and the DSP 107 perform various image signal processing such as gamma correction, white balance correction, noise removal, and demosaic processing. After the image signal processing unit 1031 performs image signal processing, the process proceeds to step # 6.

  Next, Example 2 will be described. In the second embodiment, step # 3 “image signal estimation” is performed instead of step # 2 “image signal rearrangement” in the first embodiment. Along with this change, the processing content of step # 4 “image signal processing” in the first embodiment has been changed, so that the processing is changed to step # 5. Since Step # 1 and Step # 6 are the same as those in the first embodiment, description thereof is omitted.

  Hereinafter, step # 3 will be described with reference to FIGS.

  FIG. 5 is a flowchart of image signal estimation in step # 3. FIG. 6 is a diagram illustrating the pixel and peripheral pixels. The middle pixel in FIG. 6 is the relevant pixel, FIG. 6 frame (a) is an adjacent pixel, and the same frame (b) is a neighboring pixel.

  In step # 31, 8 pixels ((a) in FIG. 6) (hereinafter referred to as adjacent pixels) in contact with the pixel are set as the search pixel range in the search range of the image signal of one layer lacking in the pixel.

  In step # 32, the image signal estimation unit 1021 has a one-layer image signal that is insufficient for the pixel within the search pixel range set in step # 31 or # 34 (hereinafter referred to as an insufficient pixel signal). It is discriminated whether or not it has. Specifically, when the process proceeds to step # 32 for the first time, any of the adjacent pixels of the pixel used by the image signal estimation unit 1021 to estimate within the search pixel range set in step # 31 is a missing image signal. Is determined. That is, both the row direction and the column direction are searched from the pixel. If the process proceeds to step # 32 after passing through step # 34, the search pixel range set in step # 34 is similarly searched and determined. As a result, if there is an insufficient image signal in the search pixel range, the process proceeds to step # 33. If there is no insufficient image signal in the search pixel range, the process proceeds to step # 34.

  In step # 33, the image signal estimation unit 1021 reads the insufficient image signal from the pixel having the insufficient image signal located within the search pixel range. Then, using the read image signal, the image signal estimation unit 1021 estimates an insufficient image signal. If the insufficient image signal is estimated, the process proceeds to # 35.

  Step # 34 widens the search pixel range by n pixels. If n = 1, this step is a new search for a range that is expanded by one pixel both in the row direction and in the column direction from FIG. 6 (a), which is a neighboring pixel, to FIG. 6 (b), which is a neighboring pixel. This is the pixel range.

  Specifically, in step # 31, the search pixel range of the insufficient image signal is set as the frame (a) in FIG. 6 that is an adjacent pixel, and as a result of the search in step # 32, the shortage image signal is included in the search pixel range. If not, the present step # 34 is a frame (b) in FIG. 6 which is a neighboring pixel obtained by expanding the search pixel range by n = 1 pixel.

  Step # 35 determines whether or not the estimation of the insufficient image signal has been completed for all the pixels of the image sensor. If completed, the process proceeds to RETURN, and thus proceeds to step # 5. If not completed, the process proceeds to step # 32.

  Step # 5 will be described.

  The image signal combining unit 1022 receives the one-layer image signal estimated in step # 3 and the two-layer image signal read from the pixel in step # 1. The estimated image signal of one layer and the image signal of two layers read from the pixel are combined by the image signal combining unit 1022 to create a stacked image signal. Thereafter, the image signal processing unit 1031 and the DSP 107 perform various image signal processing such as gamma correction, white balance correction, and noise removal. After performing image signal processing, the process proceeds to step # 6.

  Step # 6 is the same as that in the first embodiment, and a description thereof will be omitted.

  Next, step # 32 will be described with reference to FIG. In the case of FIG. 6, the insufficient image signal of the pixel in the middle is red (R). Step # 32 determines whether or not there is a pixel having red (R) in the search pixel range. Therefore, when the process proceeds to step # 32 for the first time, among the adjacent pixels, the pixels in the upper and lower rows of the pixel have red (R), so the process proceeds to step # 33.

  On the other hand, when all the pixels in the frame (a) in FIG. 6 are blue (B) and green (G), that is, when there is no insufficient image signal in the adjacent pixels, there is no insufficient image signal. The process proceeds to step # 34.

  Next, a method for estimating an insufficient image signal performed in step # 33 will be described.

  Specifically, in Example 2, since the image signal has already been read from each pixel in Step # 1, the insufficient image signal of the pixel is determined by the position of the pixel. At this time, if the pixel to be estimated is located in an odd row (in FIG. 6, the second and fourth stages from the top correspond), the two-layer image signals read from the pixel in step # 1 are , Group 1 green (G) and red (R). Therefore, the insufficient image signal, that is, the estimated image signal is blue (B). Next, among the adjacent pixels of the pixel, the image signal of blue (B) has an even-numbered row (in FIG. 6, in which blue (B) and green (G) of group 2 are read out in step # 1. The first, third, and fifth stages from the top correspond to pixels). Accordingly, 6 pixels among the adjacent pixels of the pixel have a blue (B) image signal. The average of the blue (B) image signals of the six pixels is defined as the blue (B) image signal in which the pixels are insufficient.

  Further, when the pixel to be estimated is located in an even-numbered row (in FIG. 6, the first, third, and fifth steps from the top correspond), the two-layer image signals read out in step # 1 are , Group 2 blue (B) and green (G). Therefore, the insufficient image signal, that is, the estimated image signal is red (R). The subsequent procedure is the same as that in the case where the pixel is located in an odd row, and among the adjacent pixels of the pixel, the red (R) image signal is included in the group 1 green (step # 1). G) and red (R) read out pixels (in FIG. 6, the second and fourth stages from the top correspond to pixels). Accordingly, 6 pixels among the adjacent pixels of the pixel have a red (R) image signal. The average of the red (R) image signals of the six pixels is taken as the red (R) image signal for which the pixel is insufficient.

  This is the end of the description of the second embodiment.

  Further, in the second embodiment, as described above, there is the following method as a method for estimating the insufficient image signal from the pixels around the pixel based on the image signal of the same color as the insufficient image signal.

  The insufficient image signal is estimated using the color difference. This will be specifically described below.

  When the missing image signal is blue (B)

  In step # 32, the image signal estimation unit 1021 determines whether or not a pixel having the blue (B) that is the insufficient image signal is within the search pixel range. The specific method of the determination method is to know which image signal the read control unit 1011 has read out in step # 1 from each pixel in the search pixel range at the time of this step. Therefore, the image signal estimation unit 1021 can determine whether the pixel shortage signal is included in the pixels within the search pixel range. The pixel having blue (B) is a pixel from which blue (B) and green (G) are read out in step # 1. That is, the presence / absence of pixels having blue (B) and green (G) in adjacent pixels is determined. If the adjacent pixel includes a pixel having blue (B) and green (G), the process proceeds to step # 33. Next, when there is no pixel having blue (B) and green (G) in the adjacent pixels, the process proceeds to step # 34.

  In step # 33, the image signal estimation unit 1021 calculates a color difference of blue (B) -green (G) from the blue (B) and green (G) of each pixel, and calculates the blue (B ) -Green (G) average value is defined as average value a. By adding the average value a to the green color (G) of the pixel, the blue (B) of the insufficient image signal is obtained.

  Other steps are the same as in the second embodiment.

  Next, when the insufficient image signal is red (R)

  In the same manner as when the insufficient image signal is blue (B), in step # 32, the image signal estimation unit 1021 determines whether there is a pixel having red (R) and green (G) in adjacent pixels within the search pixel range. In step # 33, a color difference of red (R) -green (G) is calculated from red (R) and green (G) of each pixel, and the calculated red (R) -green of each pixel. The average value of (G) is set as the average value b, and the average value b is added to the green color (G) of the pixel, thereby setting red (R) of the insufficient image signal.

  Also, when the insufficient image signal is green (G)

  As in the case where the insufficient image signal is blue (B) and red (R), the image signal estimation unit 1021 assigns green (G) and blue (B) to adjacent pixels within the search pixel range in step # 32. In step # 33, a color difference of green (G) -blue (B) is calculated from the green (G) and blue (B) of each pixel, and the calculated green of each pixel is determined. The average value of (G) -blue (B) is defined as an average value c, and the average value c is added to the blue color (B) of the pixel, thereby obtaining the green color (G) of the insufficient image signal.

  Further, the insufficient image signal is estimated using the color ratio. The estimation method using the color difference described so far differs from the calculation method used when estimating the insufficient image signal. This will be specifically described below.

  When the missing image signal is blue (B)

In step # 32, the image signal estimation unit 1021 determines whether or not there is a pixel having blue (B) and green (G) in the adjacent pixels as in the case of the color difference.
In step # 33, blue (B) / green (G) is calculated from blue (B) and green (G) of each pixel, and the average value of the calculated blue (B) / green (G) is averaged. Let it be the value d. This average value d is added to the green color (G) of the pixel to obtain the blue color (B) of the insufficient image signal.

  When the missing image signal is red (R)

In step # 32, the image signal estimation unit 1021 determines whether there is a pixel having red (R) and green (G) in adjacent pixels.
In step # 33, red (R) / green (G) is calculated from red (R) and green (G) of each pixel, and the average value of the calculated red (R) / green (G) is averaged. Let it be the value e. The average value e is added to the green color (G) of the pixel to obtain the red color (R) of the insufficient image signal.

  When the missing image signal is green (G)

  In step # 32, the image signal estimation unit 1021 determines whether there is a pixel having blue (B) and red (R) in adjacent pixels. In step # 33, green (G) / blue (B) is calculated from green (G) and blue (B) of each pixel, and the average value of the calculated green (G) / blue (B) is averaged. Let it be the value f. The average value f is added to the blue color (B) of the pixel to obtain the green color (G) of the insufficient image signal.

  As described above, the search pixel range is set as the adjacent pixel in step # 32 when estimating the insufficient image signal based on the color difference / color ratio. If the process proceeds to step # 32 via step # 34, step # 32 is performed. It goes without saying that the range set at 34 is searched.

  As described above, in the second embodiment described so far, the image signal estimation is performed using the search pixel range in both the row direction and the column direction. However, only the column direction may be targeted. Specifically, the pixel range for searching for the deficient pixel signal in step # 32 is set to only the column direction from the pixel, and when the search pixel range is expanded in step # 34, only the column direction is set.

  Since the subsequent flow is the same as that of the second embodiment, a description thereof will be omitted.

  From the above, according to the present invention, since two layers of image signals are read out from each pixel, the amount of reading can be reduced compared to reading out three layers of image signals. By reducing the read amount, the time required for reading the image signal is reduced.

  In the embodiments of the present invention, the description has been given using the stacked solid-state imaging device in which three layers are stacked. However, it goes without saying that the present invention is not limited to three layers, and there is no problem even if more stacked solid-state imaging devices are used. Yes. Even if the stacked solid-state imaging device has four or more layers, the read control unit 1011 reads two image signals.

  In addition, by setting one of the two image signals read by the reading control unit 1011 to green (G), the green color (G) detected by the stacked solid-state imaging device is changed to green (G) having high visual sensitivity of human eyes. The image signal G) is used. As a result, the image signal finally stored in each embodiment is an image signal with a sense of resolution.

  Further, by using the green (G) image signal detected by the stacked solid-state imaging device, accurate luminance information is used for Bayer-type rearrangement and image signal estimation. Therefore, even if rearranged into a Bayer image signal or estimated image signal, the image signal has few false colors.

  Further, it is possible to suppress the rolling shutter phenomenon, which is image distortion, by reducing the time required for reading the image signal.

  Further, in the case of a Bayer type solid-state imaging device, one layer of image signal is read from each pixel, and the remaining two layers of image signals are estimated and generated from the read one layer of image signal. On the other hand, in the present invention, two layers of image signals are read out from each pixel, and the remaining one layer of image signals is estimated and generated from the read out two layers of image signals. From the above, the present invention can suppress the generation of false colors in an image signal that is estimated and produced.

  In addition, the present invention makes it possible to use an image signal processing circuit for processing a Bayer image signal by converting a stacked image signal into a Bayer image signal. Therefore, it is not necessary to manufacture an image signal processing circuit for a stacked type image signal, so that the cost can be suppressed.

DESCRIPTION OF SYMBOLS 100 Imaging device 101 Multilayer solid-state image sensor 1011 Reading control part 102 FPGA
1021 Image signal estimation unit 1022 Image signal synthesis unit 1024 Image signal rearrangement unit 103 CPU
1031 Various signal processing units 104 SDRAM
105 External storage device 106 External display device 107 DSP
200 Optical system

Claims (8)

In an image signal processing method for processing a laminated image signal detected using a laminated solid-state imaging device in which three or more photoelectric conversion layers are laminated on a substrate on one pixel,
An image signal readout unit that reads out two layers of image signals for each pixel of the stacked solid-state imaging device;
An image signal estimation unit for estimating an image signal of a difference layer between the image signals of three or more layers detected by the photoelectric conversion layer stacked on the pixel and the read image signals of the two layers read by the image signal reading unit;
Have
The estimated image signal estimated by the image signal estimating unit is estimated using the read image signals of the two layers read by the image signal reading unit,
The image signal processing method characterized in that the two layers of image signals read out by the image signal reading unit include a combination of two or more sets of the photoelectric conversion layers.   2. The stacked solid-state imaging device includes three photoelectric conversion layers, and the colors detected by each photoelectric conversion layer are red (R), green (G), and blue (B). The image signal processing method as described.   3. The image signal according to claim 1, wherein among the read image signals of the two layers read out to the image signal read unit, the image signal of the first layer is a green (G) image signal. Processing method. The estimated image signal is
Of the read image signals of the two layers read from the peripheral pixels of the pixel to the image signal reading unit,
The image signal processing method according to claim 1, wherein the image signal processing method is estimated using an image signal having the same color as the estimated image signal. The estimated image signal is
Two-layer image signals of an image signal having the same color as one of the two-layer read image signals read from the pixel by the image signal reading unit from the peripheral pixels of the pixel and an image signal having the same color as the estimated image signal For each pixel,
By adding the average value of the color difference to an image signal having the same color as one of the read image signals of the two layers read from the pixel,
The image signal estimator is
The image signal processing according to claim 1, wherein an image signal detected by the photoelectric conversion layer stacked on the pixel but not read by the image signal reading unit is estimated. Method. The estimated image signal is
Two-layer image signals of an image signal having the same color as one of the two-layer read image signals read from the pixel by the image signal reading unit from the peripheral pixels of the pixel and an image signal having the same color as the estimated image signal For each pixel,
By integrating the average value of the color ratio to one of the read image signals of the two layers read from the pixel,
The image signal estimator is
The image signal processing according to claim 1, wherein an image signal detected by the photoelectric conversion layer stacked on the pixel but not read by the image signal reading unit is estimated. Method.   An image pickup apparatus comprising the image signal processing method according to claim 1. In an image signal processing apparatus that processes a stacked image signal detected using a stacked solid-state imaging device in which three or more photoelectric conversion layers are stacked on the top of a substrate in one pixel,
An image signal readout unit that reads out two layers of image signals for each pixel of the stacked solid-state imaging device;
An image signal rearrangement unit that rearranges the read image signals of the two layers read by the image signal read unit into a Bayer-type image signal;
Have
2. The image signal processing apparatus according to claim 2, wherein the two layers of image signals read by the image signal reading unit include two or more combinations.

Images