JP2015219284A - IMAGING DEVICE AND IMAGING DEVICE CONTROL METHOD - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pickup apparatus and an image pickup apparatus corresponding to at least two or more pixels having color filters having different spectral transmittances for one microlens, and capable of performing appropriate image output. And a method for driving an imaging device, wherein at least two or more pixels having color filters having different spectral transmittances correspond to one microlens, and the first microlens corresponds to the first microlens. An image sensor in which a first pixel having a spectral transmittance and a second pixel having the first spectral transmittance corresponding to a second microlens adjacent to the first microlens are adjacent to each other; Reordering means for performing a reordering process for generating image data by changing the order of data output from the image sensor. [Selection] Figure 2

Description

  The present invention relates to a technique for performing phase difference type focus detection using signals of a plurality of pixels arranged on an imaging surface of a solid-state imaging device.

  Conventionally, in a solid-state imaging device, there is a technique of performing phase difference type focus detection by dividing a photodiode focused by one microlens in one pixel. In Patent Document 1, each photodiode is configured to receive light of a different pupil plane of the imaging lens by dividing the photodiode in one pixel into two. As a result, the focus of the imaging lens is detected by comparing the outputs of the two photodiodes. Further, there is a concept of using such a solid-state imaging device not only for focus detection but also for an output image.

  Further, in Patent Document 2, by having a plurality of color filter arrangements in one pixel, another color is exposed to a plurality of photodiodes in the pixel.

JP 2001-083407 A JP 2007-317951 A

  However, in the method described in Patent Document 2, there is no clear description as to how image output is performed in each focus state, user setting, and shooting mode.

  In view of the above problems, in the present invention, an image sensor that enables appropriate image output from an image sensor corresponding to at least two or more pixels of a color filter having different spectral transmittances for one microlens. It is another object of the present invention to provide a method for driving an image sensor.

  In order to solve the above-described problem, in the imaging apparatus of the present invention, at least two or more pixels having color filters having different spectral transmittances correspond to one microlens, and the first microlens corresponds to the first microlens. An imaging device in which a first pixel having a spectral transmittance of 1 and a second pixel having the first spectral transmittance corresponding to a second microlens adjacent to the first microlens are adjacent to each other; And rearrangement means for performing rearrangement processing for generating image data by changing the arrangement order of the data output from the image sensor.

  In the imaging apparatus of the present invention, at least two pixels having color filters having different spectral transmittances correspond to one microlens, and the first spectral transmittance corresponding to the first microlens is obtained. Corresponding to a different microlens and an imaging device in which the first pixel having the first pixel and the second pixel having the first spectral transmittance corresponding to the second microlens adjacent to the first microlens are adjacent And adding means for adding pixel signals from pixels having the same spectral transmittance.

  According to the present invention, at least two or more pixels having color filters having different spectral transmittances with respect to one microlens can perform an appropriate image output from the corresponding imaging device. It is done.

It is a figure showing roughly the whole image sensor composition of this embodiment. It is a figure which shows the outline of the pixel structure of the image pick-up element of this embodiment. It is a figure which shows the outline of the pixel structure of the image pick-up element of this embodiment. It is a figure which shows typically the imaging relationship of an object. It is a figure which illustrates the focus detection of a phase difference system typically. It is a figure which shows typically the rearrangement process of the imaging device of this embodiment. It is a figure which shows typically the addition process of the imaging device of this embodiment. It is a figure which shows typically the rearrangement process of the imaging device of this embodiment. It is a figure which shows typically the addition process of the imaging device of this embodiment. It is a figure which shows the outline of a process of the imaging device of this embodiment. It is a figure which shows typically the rearrangement process of the imaging device of this embodiment. It is a figure which shows a to-be-photographed object.

(First embodiment)
In the present embodiment, it is considered that an image output having a pixel configuration including a plurality of photodiodes corresponding to one microlens and a plurality of color filters is performed according to a focus state. First, in such an image sensor, the image output described in Patent Document 2 causes a problem in an image output when the image is not focused. This will be described below with reference to FIG.

  FIG. 4A and FIG. 4B respectively show a conceptual diagram when the focus is achieved and a conceptual diagram when the focus is not achieved. Reference numeral 401 denotes a cross section of the pixel array. Reference numeral 402 denotes a microlens, reference numeral 403 denotes a color filter, and reference numerals 404 and 405 denote photodiodes. An area where the microlens 402 is shared is defined as one pixel. Reference numeral 406 denotes a photographing lens. The light emitted from the object passes through each region of the photographing lens 406 around the optical axis 409 and forms an image on the image sensor. The optical axis 409 passes through the microlens 402. Here, reference numerals 407 and 408 denote partial areas of the exit pupil of the photographing lens. In FIG. 4, the exit pupil and the center of the photographic lens are the same. The outermost rays of the light passing through the region 407 by the light emitted from the object are indicated by 410 and 411, and the outermost rays of the light passing through the region 408 by the light emitted from the object are indicated by 412 and 413, respectively. As shown in FIG. 4A, when the object is in focus, all the light beams passing through the exit pupil centered on the optical axis 409 pass through the microlens 402 and reach the photodiodes 404 and 405. . In the state shown in FIG. 4A, since the light incident on the photodiodes 404 and 405 is light generated from the same object, the outputs of the photodiodes 404 and 405 are outputs corresponding to the light transmitted through the color filter. In the state shown in FIG. 4A, there is no problem even if the output of the photodiode is used as an image as it is.

  However, when there is no focus as shown in FIG. 4B, blur light is incident on the image sensor. In FIG. 4B, the image is incident on the image pickup device as a blurred image extending over three pixels. However, in one microlens, an incoming light beam is changed depending on a region of the photodiode. In FIG. 4B, light emitted from different objects is incident on the photodiode 404 and the photodiode 405. For this reason, when an image is simply generated using the output of a photodiode having a different area for the microlens, there is a problem that the color of the blurred image changes. Regardless of this problem, the prior art document 2 does not clearly describe the problem of how to process pixels with an irregular arrangement order.

  Therefore, in the present embodiment, pixel rearrangement processing is performed for an image that is in focus, and addition processing is performed for an image that is out of focus, thereby realizing appropriate image output in each focus state. It is characterized by that.

  Hereinafter, the present embodiment will be described with reference to the drawings. FIG. 1A is a diagram showing an outline of the image sensor of the present embodiment. In FIG. 1A, the image sensor includes a pixel array 101, a vertical selection circuit 102 that selects a row in the pixel array 101, and a horizontal selection circuit 104 that selects a column in the pixel array 101. Further, the pixel array 101 may include a readout circuit 103 that reads out signals of pixels selected by the vertical selection circuit 102 and the horizontal selection circuit 104. In addition to the components shown in the figure, the imaging device includes, for example, a timing generator or a control circuit that provides timing to the vertical selection circuit 102, the horizontal selection circuit 104, the signal reading unit 103, and the like.

  Typically, the vertical selection circuit 102 sequentially selects a plurality of rows of the pixel array 101, and the horizontal selection circuit 104 sequentially selects a plurality of pixels constituting the row selected by the vertical selection circuit 102. A plurality of columns of the pixel array are sequentially selected.

  The pixel array 101 is configured by arranging a plurality of pixels in a two-dimensional array in order to provide a two-dimensional image. The configuration of the pixel will be specifically described later.

  FIG. 1B illustrates an embodiment in which the image sensor according to the present embodiment is applied to a digital camera that is an image capturing apparatus.

  In FIG. 1B, reference numeral 105 denotes a lens unit that forms an optical image of a subject on the image sensor 109. The lens unit 105 includes a focus lens, a zoom lens, and a diaphragm (light-shielding member). The lens driving device 106 performs zoom control, focus control, Aperture control is performed. Reference numeral 107 denotes a mechanical shutter which is controlled by a shutter driving device 108.

  Reference numeral 109 denotes a solid-state image sensor for taking in a subject imaged by the lens unit 105 as an image signal, for example, a CCD sensor or a CMOS sensor. An image processing unit 110 converts an analog signal output from the solid-state imaging device 1305 into an image signal that is a digital signal, or performs various corrections on the image signal stored in the memory unit 112 or compresses data. Circuit. A timing generation circuit 111 is a driving unit that outputs various timing signals to the solid-state imaging device 109 and the image processing circuit 110. A control circuit 113 controls various calculations and the entire image pickup apparatus, and 112 is a memory for temporarily storing image data. Reference numeral 114 denotes a recording medium control interface unit for performing recording or reading on the recording medium, and reference numeral 115 denotes a removable recording medium such as a semiconductor memory for recording or reading image data. Reference numeral 116 denotes a display unit such as a liquid crystal for displaying various information and captured images.

  Next, the operation of the digital camera at the time of shooting in the above configuration will be described.

  When the main power supply is turned on, the power supply for the control system is turned on, and the power supply for the image pickup system circuit such as the image pickup signal processing circuit 1306 is turned on.

  Then, when a release button (not shown) is pressed, distance calculation is performed based on data from the image sensor, and a defocus amount is calculated by the control circuit 113 based on the distance measurement result. After driving the lens unit by the lens driving device 106 based on the defocus amount, it is determined whether or not the lens unit is in focus. When it is determined that the lens unit is not in focus, the lens unit is driven again to perform distance measurement. In addition to the data from the solid-state image sensor 109, the distance measurement calculation may be performed by separately providing a distance measurement device that performs distance measurement by means such as phase difference detection or infrared rays.

  Then, after the in-focus state is confirmed, the photographing operation starts. When the photographing operation is finished, the image signal output from the solid-state image sensor 1305 is subjected to image processing by the image processing circuit 110 and written to the memory by the control circuit 113. In the photographing signal processing circuit, rearrangement processing, addition processing, and selection processing thereof are performed. The data stored in the memory 112 is recorded on a removable recording medium 115 such as a semiconductor memory through the recording medium control I / F unit 114 under the control of the control circuit 113.

  Further, the image may be processed by directly inputting to a computer or the like through an external I / F unit (not shown).

  FIG. 2 is a diagram showing an outline of the pixel configuration of the image sensor of the present embodiment. 201, 202, 203, and 204 represent pixels. One pixel has a microlens 205. One pixel has four photodiodes, and a plurality of color filters having different spectral transmittances are arranged. Reference numerals 206, 209, 211, 212, 214, 215, 216, 217, 218, 219, 220, 221, and 222 are photodiodes having a G filter disposed thereon. Reference numerals 207 and 210 denote B filters, and 208 and 213 denote photodiodes having an R filter disposed thereon. All the color filters are arranged at all positions of the photodiodes in the pixel in any pixel. For example, when the pixels 201, 202, 203, and 204 are basically arranged and arranged in a two-dimensional manner by repeating this, all the R, G, and B color filters are arranged in the upper right, upper left, lower right, and lower left in one pixel. It is arranged in any area.

  Here, the shape is such that four photodiodes and three color filters are arranged in one pixel, but this embodiment is not limited to this form. For example, as shown in FIG. There may be a form in which two photodiodes are provided in one pixel. Further, as shown in FIG. 3B, a shape in which two color filters are arranged in one pixel may be employed. In other words, in the present embodiment, a single microlens has a plurality of photodiodes, and the photodiodes have different color filters, and the color filters are arranged in all regions together with the peripheral pixels. If it is a structure, the form will not be limited.

  The operation of the image sensor having the pixel configuration as shown in FIGS. 2 and 3 will be described with reference to FIG. FIG. 4A and FIG. 4B respectively show a conceptual diagram when the focus is achieved and a conceptual diagram when the focus is not achieved. Reference numeral 401 denotes a cross section of the pixel array. Reference numeral 402 denotes a microlens, reference numeral 403 denotes a color filter, and reference numerals 404 and 405 denote photodiodes. An area where the microlens 402 is shared is defined as one pixel. Reference numeral 406 denotes a photographing lens. The light emitted from the object passes through each region of the photographing lens 406 around the optical axis 409 and forms an image on the image sensor. The optical axis 409 passes through the microlens 402. Here, reference numerals 407 and 408 denote partial areas of the exit pupil of the photographing lens. In FIG. 4, the exit pupil and the center of the photographic lens are the same. The outermost rays of the light passing through the region 407 by the light emitted from the object are indicated by 410 and 411, and the outermost rays of the light passing through the region 408 by the light emitted from the object are indicated by 412 and 413, respectively. As shown in FIG. 4A, when the object is in focus, all the light beams passing through the exit pupil centered on the optical axis 409 pass through the microlens 402 and reach the photodiodes 404 and 405. . In this case, since the light incident on the photodiodes 404 and 405 is light generated from the same object, the photodiodes 404 and 405 output according to the color filter. When the focus is not as shown in FIG. 4B, a blurred image extending over three pixels is formed in FIG. 4B, but depending on the location of the photodiode sharing the microlens, The incoming light flux changes. In other words, light generated from another object enters. Taking advantage of this characteristic, the phase difference is detected. The method of phase detection is known, but will be described briefly. In the region within the pixel, the phase can be detected by obtaining the correlation data between the lines with the left data as the first line and the right data as the second line. FIG. 5 shows line data when a point light source is imaged. FIG. 5A shows data of the first line and the second line in the case of FIG. The horizontal axis represents the pixel position, and the vertical axis represents the output. When focus is achieved, the first line and the second line overlap. FIG. 5B shows the line data in the case of FIG. At this time, the first line and the second line have a phase difference, and the pixel position is shifted. If this shift amount 501 is calculated, it can be seen how much the shift is from the in-focus state. The focus can be adjusted by detecting the phase in this way and driving the lens.

  As the line data, for example, the data 206, 214, and 218 shown in FIG. 2 may be collected as the first line, and the data 211, 217, and 221 may be collected as the second line. Further, paying attention to the left and right, 206, 212, 214, 216, 220 may be used as the first line, or data added to the upper and lower lines may be used. For example, since 222 and 220 are data on the same left side, they may be added and used. Further, in order to detect the phase in the vertical direction as well as the left and right, the line data may be in the vertical direction instead of the horizontal direction.

  Next, image data generation in these pixel arrangements will be described.

  When focus is achieved, the plurality of photodiodes sharing the microlens receive light emitted from the same object, and may be used as image data as they are. In this case, RGB can be arranged regularly by changing the order of the image data. This will be described with reference to FIG. 6 shows the arrangement order of the image data in the pixel arrangement shown in FIG. In order to show the microlens sharing relationship in an easy-to-understand manner, the data of pixels belonging to the same microlens are circled. Color filter arrangement with regularity in a 2x2 basic pattern by replacing the rows including 206 and 207 with the rows including 208, 209, and 212, and the columns including 210 and 212 with the columns including 207 and 209 become. According to the above method, the image data can be rearranged in an appropriate order.

  FIG. 7 shows a method of generating image data by another method. The upper diagram of FIG. 7 shows the arrangement order of the image data in the pixel arrangement shown in FIG. In order to show the microlens sharing relationship in an easy-to-understand manner, the data of pixels belonging to the same microlens are circled. Adjacent color filters of the same color have different microlenses, but by adding them, one microlens can be handled almost equivalently as a pixel having a single color filter. For example, by adding data of 701, 702, 703, and 704 having a B filter, the data is handled as pixel data of one B filter.

  Since the 2 × 2 basic pattern has a regular color filter arrangement, it can be handled by the same image processing method as the data in the lower diagram of FIG. That is, the added data is handled as one pixel. In the present embodiment, it is possible to obtain an appropriate image in which image data is arranged in an appropriate order and the color does not shift.

  6 and 7, the pixel having four photodiodes in one microlens shown in FIG. 2 has been described. However, two photodiodes as shown in FIG. As long as there is no problem. For example, in the color filter arrangement shown in FIG. 3A, the rearrangement processing described in FIG. 6 is shown in FIG. Further, in the color filter arrangement shown in FIG. 3A, the addition processing described with reference to FIG. 7 is as shown in FIG.

  These processes will be described with reference to FIG.

  FIG. 10A illustrates the rearrangement process described with reference to FIG. The data output from the image sensor is rearranged into an arrangement with 2 × 2 as a basic pattern by the rearrangement unit included in the image processing circuit 110, and is generated as image data. The generated image data is subjected to processing such as gamma correction and color matrix processing by the image processing means.

  FIG. 10B illustrates the addition process described with reference to FIG. In the data output from the solid-state image sensor 109, the pixels to be subjected to addition processing are pixels adjacent to each other on the image sensor. Pixel signals from pixels (photodiodes) corresponding to different microlenses and having the same spectral transmittance (color filter) are added by an adding means (for example, an adding circuit) included in the image processing circuit 110, and an image Generated as data. The addition process may be an average process. The generated image data is subjected to processing such as gamma correction and color matrix processing by the image processing circuit 110.

  In FIG. 10C, the data output from the image sensor determines what kind of processing is performed by the selection means (for example, a switching circuit) included in the image processing circuit 110. When rearrangement is selected by the selection unit, the rearrangement unit applies the rearrangement processing described with reference to FIG. 6 and outputs the data as image data. The generated image data is subjected to image processing such as gamma correction and color matrix processing by the image processing circuit 110. When composition is selected by the selection means, the addition process described with reference to FIG. 7 is applied. When the addition process is performed, the size of the image data is reduced. For example, when the addition process shown in FIG. 7 is performed, the number of pixel data is ¼. The image processing in the image processing circuit 110 requires information on the number of image data. Therefore, the selection means transmits the number of image data by changing the selection flag in order to transmit to the processing circuit subsequent to the image processing circuit 110 whether the selection means has performed sorting or composition processing. For example, when the addition process is performed, the selection flag is set to 0, and when the synthesis process is performed, the selection flag is set to 1. The processing circuit subsequent to the selection unit and the addition unit of the image processing circuit 110 detects the flag and performs image processing according to the size of the image data.

  In order to obtain the same image data size (number of pixels) as the processing by the rearranging means, a rearrangement means (for example, a resizing circuit) may be provided after the synthesizing means. The rearrangement unit rearranges the image data synthesized by the synthesis unit.

  FIG. 11 shows processing in the rearrangement means. In the rearrangement means, it is possible to make the same color filter arrangement and the same image data size as the rearrangement means by rearranging the data from the image data synthesized by the synthesis means in a different area. Become. In FIG. 11, for example, this is realized by dividing the R1 region into data for four pixels and rearranging the data. In this way, it may be simply replaced, or an average value of surrounding data may be used. For example, the data of 1101 may not be the data of R14, but may be an average value of R12, R14, R32, and R34. Similarly, 1102 is not R14 data, but may be an average value of R14 and R34.

  The determination by the selection means may be determined by the user. Further, it is detected from the focus detection information detected by the control circuit 113 whether or not the camera is in focus, and the camera is in focus (the focus state is a defocus amount smaller than a predetermined amount). When it is an image, rearrangement processing is performed. When the image is not in focus (in an out-of-focus state, with a defocus amount greater than a predetermined amount), addition processing is performed. In addition, the focus detection information at that time may be information obtained from the phase difference in the above-described imaging device, may be obtained from another phase difference detection device, or may be obtained from a known contrast evaluation method. It may be obtained from the image of

  As described above, rearrangement processing is performed when the captured image is in focus, and addition processing is performed when the captured image is not in focus. On the other hand, there are often shooting scenes in which a subject that is in focus and a subject that is not in focus are mixed in the same shot image. FIG. 12 shows a subject to be photographed. A subject 1201 is in focus, and a subject 1202 is not in focus. In such a case, it is possible to change the area for performing the rearrangement process and the area for the addition process in the screen. The determination is performed by the selection means described in FIGS. 10 (c) and 10 (d). It is also possible to switch between rearrangement processing and addition processing according to the set shooting mode. For example, it has a moving image shooting mode and a still image shooting mode. When it is in the moving image shooting mode, addition processing is performed by the adding means, and when it is in the still image shooting mode, rearrangement processing is performed by the rearranging means. In the case of an imaging device with an interchangeable lens, switching may be performed according to the photographing lens.

(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is recorded is supplied to the system or apparatus. The computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.

  Further, by making the program code read by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, the functions of the above-described embodiments are realized by the processing. Is also included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

  In addition, the present invention is not limited to devices such as digital cameras, but includes built-in or external connection of imaging devices such as mobile phones, personal computers (laptop type, desktop type, tablet type, etc.), game machines, etc. It can be applied to any device. Therefore, the “imaging device” in this specification is intended to include any electronic device having an imaging function.

Claims (12)

A first pixel having a first spectral transmittance corresponding to a first microlens, wherein at least two or more pixels having color filters having different spectral transmittances correspond to one microlens; An image sensor in which a second pixel having the first spectral transmittance corresponding to a second microlens adjacent to the microlens is adjacent;
An imaging apparatus comprising: a rearranging unit that performs rearrangement processing for generating image data by changing an arrangement order of data output from the imaging element. A first pixel having a first spectral transmittance corresponding to a first microlens, wherein at least two or more pixels having color filters having different spectral transmittances correspond to one microlens; An image sensor in which a second pixel having the first spectral transmittance corresponding to a second microlens adjacent to the microlens is adjacent;
An image pickup apparatus comprising: addition means for adding pixel signals from pixels having the same spectral transmittance corresponding to different microlenses.   As a process to be applied to the pixel signal output from the image sensor, a selection unit that selects any one of the addition processes for adding the same spectral transmittance regions corresponding to the microlenses different from the rearrangement process is provided. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized.   The imaging apparatus according to claim 3, wherein the selection by the selection unit is changed according to a focus state.   The imaging apparatus according to claim 4, wherein the selection unit selects the rearrangement process when in an in-focus state, and selects the addition process when in an out-of-focus state.   The imaging apparatus according to claim 3, wherein the selection by the selection unit is changed according to a mode.   7. The imaging apparatus according to claim 6, wherein the selection unit performs addition processing by the addition unit when in the moving image shooting mode, and performs the rearrangement processing when in the still image shooting mode.   The imaging apparatus according to claim 2, wherein the pixels to be subjected to the addition processing are adjacent to each other on the imaging element. A first pixel having a first spectral transmittance corresponding to a first microlens, wherein at least two or more pixels having color filters having different spectral transmittances correspond to one microlens; A control method of an imaging apparatus having an imaging element adjacent to a second pixel having the first spectral transmittance corresponding to a second microlens adjacent to the microlens,
A control method for an imaging apparatus, comprising: a rearrangement step for performing rearrangement processing for generating image data by changing an arrangement order of data output from the image sensor. A first pixel having a first spectral transmittance corresponding to a first microlens, wherein at least two or more pixels having color filters having different spectral transmittances correspond to one microlens; A control method of an imaging apparatus having an imaging element adjacent to a second pixel having the first spectral transmittance corresponding to a second microlens adjacent to the microlens,
An image pickup apparatus control method comprising an addition step of adding pixel signals from pixels having the same spectral transmittance.   A computer-executable program in which a procedure of a control method for an imaging apparatus according to claim 9 is described.   A computer-readable storage medium storing a program for causing a computer to execute each step of the control method of the imaging apparatus according to claim 9 or 10.

Images