JP2013223210A - Image pickup treatment apparatus, image pickup treatment method, and program - Google Patents

Abstract

An image with reduced color blur is obtained while securing a light amount and improving S / N.
An imaging processing apparatus is a single-plate color solid-state image pickup device having both pixels read out at a first frame rate and a second frame rate lower than the first frame rate, and the single-plate color solid-state image pickup device includes: The horizontal lines in which the pixels read out at the first and second frame rates are alternately arranged at a ratio of 3: 1 and the horizontal lines in which the pixels are alternately arranged at a ratio of 1: 3 are alternately arranged in the vertical direction. The imaging processing apparatus also includes an image separation unit 401 that separates an input color image into RGB three-color components, single-color image processing units 402 and 403 that generate an output image of the first frame rate from input images of each color, and 404 and an image integration unit 405 that generates a color image from the processing result image of each color. In the single color image processing unit, constraints on spatial smoothness of pixel values are set on the same horizontal line or two adjacent horizontal lines. This is applied to pixels of the same color on the line.
[Selection] Figure 4

Description

  The present application relates to an imaging processing apparatus that generates a moving image, an imaging processing method, and an image processing program.

  In the conventional imaging processing apparatus, the number of pixels of the imaging element tends to increase for the purpose of achieving high resolution. However, since there is a limit to increasing the size of the entire image sensor, each pixel must be downsized. On the other hand, as the pixel size is reduced, the amount of light incident on one pixel of the image sensor decreases. As a result, the signal-to-noise ratio (S / N) of each pixel is reduced, and it is difficult to maintain the image quality.

  Patent Document 1 uses three image sensors that detect red, green, and blue light, respectively, and processes signals obtained by controlling exposure time, thereby achieving high resolution, high frame rate, and high sensitivity. It is disclosed that image restoration is realized. In this technology, two types of resolution image sensors are used. One high-resolution image sensor reads pixel signals with long exposure, and the other low-resolution image sensor reads pixel signals with short exposure. Thereby, the light quantity was ensured.

  Furthermore, Patent Document 2 extends the method of Patent Document 1 to a single-panel camera, and even with a single-panel camera, it maintains image quality while suppressing a decrease in the signal-to-noise ratio (S / N) of each pixel. Is disclosed.

JP 2009-105992 A International Publication No. WO2010 / 090025

  However, the color component image captured at the high resolution and the low frame rate is an image including motion blur when the subject is moving. Embodiments of the present invention provide an imaging processing technique that can reduce deterioration in image quality caused by motion blur.

  In one embodiment, the imaging processing apparatus includes: a first pixel unit in which two first color pixels, one second color pixel, and one third color pixel are arranged to form a Bayer array; Two first color pixels, one second color pixel, and a second pixel unit in which one third color pixel is arranged so as to form a Bayer array are alternately arranged in rows and columns. A single-plate color imaging device, wherein one of the two first color pixels in the first pixel unit, the second color pixel and the third color pixel, and the two in the second pixel unit. A pixel value is read at a first frame rate from a first color pixel located in the same row as the one of the first color pixels in the first pixel unit. Of the two first color pixels in the pixel unit. And the second frame having a pixel value lower than the first frame rate from the other of the two first color pixels, the second color pixel, and the third color pixel in the second pixel unit. Based on a single-plate color imaging device that can be read at a rate, a pixel value read at the first frame rate, and a pixel value read at the second frame rate, the first And an image processing unit that generates a new moving image having a frame rate of 1.

  In one embodiment, the image processing unit includes a pixel value read from the first color pixel at the first frame rate, and a pixel read from the first color pixel at the second frame rate. A first monochrome image processing unit that generates a first color moving image having a first frame rate based on the value, a pixel value read from the second color pixel at the first frame rate, and A second single-color image processing unit that generates a second color moving image of the first frame rate based on a pixel value read from the second color pixel at the second frame rate; Based on the pixel value read from the three-color pixel at the first frame rate and the pixel value read from the third-color pixel at the second frame rate, the first frame rate Third monochrome image for generating the third color moving image And a processing unit, to generate the color moving image from the first, second and third moving image.

  In one embodiment, the image processing unit includes the first color moving image output from the first single color image processing unit and the second color moving image output from the second single color image processing unit. And a memory for storing the third color moving image output from the third single-color image processing unit.

  In one embodiment, the image processing unit includes the moving image of the first color component, the moving image of the second color component, or the moving image of the second color component obtained based on the pixel value read at the first frame rate. A first difference between the moving image of the third color component and the moving image obtained by sampling the new moving image at the first frame rate and the second moving image are read at the second frame rate. The moving image of the first color component, the moving image of the second color component or the moving image of the third color component obtained based on the obtained pixel value, and the new moving image are converted into the second moving image. The second difference from the moving image obtained by sampling at the frame rate and the change in the distribution of pixel values in the new moving image become smaller, or the change in the pixel value in the new moving image becomes constant. The smaller the constraint condition terms Set the expression, a moving image that minimizes the evaluation formula is determined as the new moving image.

  In one embodiment, the image processing unit sets the constraint condition term for each of the first color pixel, the second color pixel, and the third color pixel.

  In one embodiment, the image processing unit calculates the constraint condition term for the first color pixel using pixel values in two rows of the single-plate color solid-state imaging device.

  In one embodiment, the image processing unit uses the pixel value in one row and / or one column of the single-plate color solid-state imaging device as the constraint condition term for the second color pixel and the third color pixel. To calculate.

  In one embodiment, the image processing unit includes a weighting factor multiplied by each of the first difference, the second difference, and the constraint condition term.

  In one embodiment, the first color pixel, the second color pixel, and the third color pixel are a green pixel, a red pixel, and a blue pixel, respectively.

  In one embodiment, the image processing unit includes the moving image of the first color component, the moving image of the second color component, or the first moving image obtained based on the pixel value read at the first frame rate. A motion detection unit that detects a motion of the subject using a moving image of three color components; and a constraint condition term relating to the motion distribution of the new moving image is set using the detection result of the motion, A constraint condition term relating to the motion distribution is included in the evaluation formula.

  In one embodiment, the image processing method performs image processing based on a pixel value obtained from a single-plate color imaging device, and the single-plate color imaging device includes two first color pixels, one pixel A first pixel unit in which a second color pixel and one third color pixel are arranged to form a Bayer array, two first color pixels, one second color pixel, and one A single-plate color image sensor in which second pixel units arranged so that third color pixels form a Bayer array are alternately arranged in rows and columns, and the two pixel units in the first pixel unit One of the first color pixels, the second color pixel and the third color pixel, and one of the two first color pixels in the second pixel unit, the first color pixel in the first pixel unit A first color pixel located in the same row as said one of Read pixel values at a first frame rate, the other of the two first color pixels in the first pixel unit, the other of the two first color pixels in the second pixel unit, the second Pixel values can be read out from the two-color pixel and the third color pixel at a second frame rate lower than the first frame rate, and the single-plate color image pickup device at the first frame rate. Obtaining a read pixel value and a pixel value read at the second frame rate; the pixel value read at the first frame rate; and the second frame rate. And generating a new moving image having the first frame rate based on the pixel value read in step (b).

  In one embodiment, the step of generating a new moving image of the first frame rate is performed for each color, and then includes the step of generating a color image of the first frame rate.

  In one embodiment, the image processing program is an image processing program for performing image processing based on pixel values obtained from a single-plate color image sensor, and the single-plate color image sensor includes two first colors. A first pixel unit in which pixels, one second color pixel, and one third color pixel are arranged so as to form a Bayer array, two first color pixels, and one second color pixel , And a second pixel unit in which one third color pixel is arranged so as to form a Bayer array, and is a single-plate color imaging device in which the first pixel unit is alternately arranged in rows and columns. One of the two first color pixels, one of the second color pixel and the third color pixel, and one of the two first color pixels in the second pixel unit. The one of the first color pixels; The pixel values are read out from the first color pixels located in the same row as the first frame rate, the other of the two first color pixels in the first pixel unit, and the two in the second pixel unit. The pixel value can be read out from the second color pixel and the third color pixel at a second frame rate lower than the first frame rate. Based on the pixel value read at the frame rate and the pixel value read at the second frame rate, a computer generates a new moving image at the first frame rate.

  In one embodiment, the step of generating a new moving image of the first frame rate is performed for each color, and then includes the step of generating a color image of the first frame rate.

  According to the present invention, for each of a plurality of colors, images having different frame rates are obtained and image processing is performed, so that it is possible to reduce deterioration in image quality due to motion blur.

1 is a block diagram showing a configuration of an imaging processing apparatus 100 according to Embodiment 1 of the present invention. The figure which shows an example of the pixel arrangement | sequence of the single-plate color image pick-up element 102 The figure which shows the example of arrangement | positioning of the single time exposure image and long time exposure image in this invention FIG. 3 is a diagram illustrating an internal configuration of a color image processing unit 105 according to the first embodiment. Flow chart showing processing procedure of conjugate gradient method The figure which shows the structural example of the R image process part 402 for performing the process shown in FIG. FIG. 4 is a diagram illustrating an example of an array of R pixels extracted by the image separation unit 401 according to the first embodiment. FIG. 6 is a diagram illustrating an example of an array of G pixels extracted by the image separation unit 401 according to the first embodiment. The figure which shows the applicable range of the smoothness constraint with respect to G pixel of Embodiment 1. FIG. 5 is a diagram illustrating an example of an array of B pixels extracted by the image separation unit 401 according to the first embodiment. FIG. 5 is a diagram illustrating an example of an internal configuration of a color image processing unit 105 according to the second embodiment. (A) And (b) is a figure which shows the base frame and reference frame when performing motion detection by block matching FIG. 10 is a diagram illustrating another internal configuration example of the color image processing unit 105 according to the second embodiment. The figure which shows the example of the pixel arrangement | sequence of Embodiment 3. The figure which shows the applicable range of the smoothness constraint with respect to R pixel of Embodiment 1. The figure which shows the applicable range of the smoothness constraint with respect to B pixel of Embodiment 1. The figure which shows the example of a structure of the single-plate color solid-state image sensor 102. The figure which shows the detailed structure of the pixel array 1701 of the single-plate color solid-state image sensor 102. The figure which shows the other example of pixel arrangement | positioning of the pixel array 1701 of the single-plate color solid-state image sensor 102. FIG. The top view which shows the example of pixel arrangement | positioning in the single-plate color image pick-up element with which the imaging processing apparatus in 1 aspect of this invention is provided.

  In the above-described prior art, a G component image is captured with long exposure. For this reason, for example, when the movement of the subject does not work well in the RB component image, such as when the subject moves larger than the detection range defined by the system or when the subject moves complicatedly with deformation of the subject, the G component The motion blur is not completely removed and remains in the restored image. The same applies to the case where the motion detection does not work well in the RB component image, such as when the subject moves complicatedly with deformation of the subject. If the motion blur of the G component is not completely removed and remains in the restored image, green or magenta color blur, which is a complementary color thereof, occurs. For this reason, although the image quality of the obtained moving image is high in many cases, in some areas where it is difficult to detect motion, the color blur may be seen on the restored image, and there is room for improvement in image quality. there were. Also in Patent Document 2, further improvement of S / N is desired.

  An imaging processing apparatus according to an aspect of the present invention includes a single-plate color imaging element and an image processing unit. In this single-plate color imaging device, as illustrated in FIG. 20, the first pixel units and the second pixel units are alternately arranged in rows and columns. In the first pixel unit, two first color pixels (1S, 1L), one second color pixel (2S), and one third color pixel (3S) are arranged so as to form a Bayer array. Has been. In the second pixel unit, two first color pixels (1S, 1L), one second color pixel (2L), and one third color pixel (3L) are arranged so as to form a Bayer array. Has been. In some embodiments, the first color pixel, the second color pixel, and the third color pixel correspond to a green pixel, a red pixel, and a blue pixel, respectively. The three colors are not limited to green, red, and blue, and may be complementary colors thereof. The first pixel unit and the second pixel unit are merely elements constituting a periodic two-dimensional array of pixels, and no special boundary is provided between the pixel units. Alternatively, each pixel unit may be relatively moved by, for example, one pixel while the actual pixel arrangement is fixed.

  One of the two first color pixels (1S, 1L) in the first pixel unit (1S), the second color pixel (2S) and the third color pixel (3S), and the two first colors in the second pixel unit From the first color pixel (1S) located in the same row as the one (1S) of the first color pixels in the first pixel unit, which is one of the color pixels (1S, 1L), the pixel value is the first frame rate ( S: Short).

  Also, the other of the two first color pixels (1S, 1L) in the first pixel unit (1L), the other of the two first color pixels in the second pixel unit (1L), and the second color pixel (2L). ) And the third color pixel (3L), pixel values are read out at a second frame rate (L: Long) lower than the first frame rate.

  The image processing unit generates a new moving image having the first frame rate based on the pixel value read at the first frame rate and the pixel value read at the second frame rate. The image processing unit according to the embodiment generates a new moving image having the first frame rate based on the pixel value read at the first frame rate and the pixel value read at the second frame rate. The process to generate is executed for each color. A color image having the first frame rate can be obtained from the moving image having the first frame rate obtained for each color.

  Hereinafter, embodiments of an imaging processing apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a block diagram illustrating a configuration of an imaging processing apparatus 100 according to the present embodiment. The imaging processing apparatus 100 includes an optical system 101, a single plate color imaging element 102, a readout control unit 103, a control unit 104, and a color image processing unit 105. Hereinafter, each component of the imaging processing apparatus 100 will be described in detail.

  The optical system 101 is, for example, a camera lens, and forms an image of a subject on the image plane of the image sensor.

  The single-plate color solid-state imaging device 102 is an imaging device in which a color filter array of red (R), green (G), and blue (B) is attached to each pixel. FIG. 2 shows an example of the single-plate color image sensor 102. Corresponding to each pixel (photodiode) of the single-plate image pickup device, an array of color filters called a Bayer array is mounted. In the drawing, “R”, “Gr”, “Gb”, and “B” indicate “red”, “green”, and “blue” filters, respectively. Here, “Gr” means a G component color filter arranged in the same row as the R component color filter, and “Gb” means a G component arranged in the same row as the B component color filter. A color filter is shown.

  Each pixel of the single-plate color solid-state imaging device 102 receives light (optical image) connected by the optical system 101 through a red, green, or blue filter provided corresponding to the light. Each pixel performs photoelectric conversion on a pixel basis, and outputs a pixel value (pixel signal) corresponding to the amount of light incident on each pixel. An image for each color component is obtained from pixel signals of the same color component that are captured at the same frame time. A color image is obtained from images of all the color components.

  In the following description and drawings, pixels that detect red, green, and blue light are represented by “R”, “G”, and “B”, respectively. A pixel signal output from each pixel of the image sensor has a pixel value related to any of R, G, and B colors. Hereinafter, pixels that detect light of red, green, and blue color components are referred to as R pixel, G pixel, and B pixel, respectively. In addition, pixels at positions where “Gr” and “Gb” in FIG. 3 are referred to as Gr pixels and Gb pixels, respectively. That is, the “G pixel” here is a generic term for Gr pixel and Gb pixel.

  As is apparent from this description, each pixel of the image sensor is a unit of the image sensor that receives light transmitted through the color filters of each color and outputs a signal corresponding to the intensity of the received light.

  The readout control unit 103 independently sets the exposure time (or readout timing) of the charge (pixel value) of each pixel of the single-plate color imaging element 102 in the arrangement shown in FIG. The reading method is as described in detail below. The read operation of the read control unit 103 is executed based on control by the control unit 104.

  FIG. 3 shows an example of the arrangement of short-time exposure images and long-time exposure images in the present embodiment. The pixel signal is read out in units of lines. In FIG. 3, the subscript S for R, Gr, Gb, and B indicates short-time exposure, and the subscript L indicates long-time exposure.

  In this specification, “short-time exposure” means exposure for one frame time during general moving image shooting. For example, “short-time exposure” corresponds to exposure with an exposure time of 1/30 seconds when the frame rate is 30 frames / second (30 fps). When the frame rate is 60 frames / second (60 fps), it is 1/60 seconds. On the other hand, long exposure means exposure for a time longer than the time required for exposure of one frame, for example, about 2 to 10 frames.

  FIG. 17 shows an example of the configuration of the single-plate color solid-state imaging device 102 according to this embodiment. The single-plate color solid-state imaging device 102 includes a pixel array 1701, a G component AD conversion unit 1702, a G component HSR 1703, an RB component AD conversion unit 1704, an RB component HSR 1705, and an imaging control unit 1706. The pixel array 1701 is mounted with the color filter arrangement and the long exposure pixel / short exposure pixel arrangement shown in FIG. 3, and photoelectrically converts incident light of each color component. The G component AD converter 1702 performs analog-digital conversion on the output signal of the G pixel obtained from the pixel array 1701. The G component HSR 1703 outputs the output of the G component AD converter 1702 to the outside of the single-plate color solid-state image sensor 102. The RB component AD conversion unit 1704 performs analog-digital conversion on the output signals of the R and B pixels obtained from the pixel array 1701. The RB component HSR 1705 converts the output of the RB component AD conversion unit 1704 into a single-plate color solid-state imaging device. Output outside 102. The imaging control unit 1706 supplies readout signals corresponding to the long exposure time and the short exposure time to the pixel array 1701.

  FIG. 18 shows an example of a detailed configuration of the pixel array 1701. In FIG. 18, a pixel (subscript L) that is imaged with long exposure is connected with a long-time readout signal line 1801 and is read out with long-time exposure and a low frame rate. For short-time exposure (subscript S) pixels, a short-time readout signal line 1802 is connected and readout is performed at a short-time exposure and a high frame rate. The long-time readout signal line 1801 and the short-time readout signal line 1802 are connected to the imaging control unit 1706 in FIG. 17, and receive readout signals corresponding to the respective exposure times from the imaging control unit 1706. In this configuration, since the long-time readout signal line 1801 or the short-time readout signal line 1802 is shared by two adjacent pixels in the upper and lower sides, two horizontal pixel columns can be read out at the same time, and the reading speed can be increased. it can. Further, in the configuration of FIG. 18, the G signal output line 1803 and the RB signal output line 1804 are separately provided in order to prevent the output signals of the pixels simultaneously read out from being mixed with each other.

  Refer to FIG. 1 again.

  The color image processing unit 105 receives the short-exposure image data and the long-exposure image data, and performs image processing on the image data to estimate pixel values (for example, R, G, and B pixel values) at each pixel. Generate high-resolution color moving images.

  FIG. 4 shows an example of a detailed configuration of the color image processing unit 105. The color image processing unit 105 includes an image separation unit 401, an R image processing unit 402, a G image processing unit 403, a B image processing unit 404, and an image integration unit 405. As shown in FIG. 4, the image separation unit 401 converts an input image including all color components of R, Gr, Gb, and B into an image including only R component pixels (hereinafter referred to as an R component image) and an image including only the G component. An image including pixels (G component image) and an image including only B component pixels (B component image) are decomposed. In FIG. 4, the GrL pixel and the GbL pixel in FIG. 3 are collectively referred to as GL, and the GrS pixel and the GbS pixel in FIG. 3 are collectively referred to as GS. The R image processing unit 402 generates a pixel value of the output image at each R pixel position by performing image processing from the pixel values of the RL pixel and the RS pixel. The G image processing unit 403 generates a pixel value of the output image at each G pixel position by performing image processing from the pixel values of the GL pixel and the GS pixel. The B image processing unit 404 generates the pixel value of the output image at each B pixel position by performing image processing from the pixel values of the BL pixel and the BS pixel.

  The image integration unit 405 integrates the color component images output from the R image processing unit 402, the G image processing unit 403, and the B image processing unit 404, and generates a final color image. A specific example of the image integration unit 405 is a memory. Each data output from the R image processing unit 402, the G image processing unit 403, and the B image processing unit 404 can be recorded in the memory. A color image can be obtained by reading each color data from the memory. Therefore, the final generation of the color image means not aligning the color image but aligning the data of each color constituting the color image. The data of each color may be stored separately in different areas in the same memory, or may be stored separately in different memories. Further, the data of each color may be stored in the memory in a mixed form.

  Next, detailed operations of the R image processing unit 402, the G image processing unit 403, and the B image processing unit 404 in FIG. 4 will be described. The contents of each of these processes are the same except for a part, and hereinafter, the R image processing unit 402 will be described as an example. After the description of the processing contents in the R image processing unit 402, differences between the processing in the G image processing unit 403 and the B image processing unit 404 from the R image processing unit 402 will be described.

ここで、Ｈ_1Rは長時間露出のサンプリング過程、Ｈ_2Rは短時間露出のサンプリング過程、ｆ_Rは新たに生成すべき高空間解像度かつ高時間解像度のＲ画像である。また、λ_LRおよびλ_SRは重み係数である。単板カラー撮像素子１０２によって撮像されたＲの短時間露出画像をｇ_SR、長時間露出画像をｇ_LRとする。Ｍはべき指数、Ｑは生成すべき画像ｆが満たすべき条件、すなわち拘束条件である。 <R Pixel Value Generation Processing for Each Pixel>
The R image processing unit 402 calculates an R pixel value by minimizing an evaluation function (evaluation expression) _JR represented by the following expression.

Here, H _1R is a long-exposure sampling process, H _2R is a short-exposure sampling process, and f _R is a high spatial resolution and high temporal resolution R image to be newly generated. Also, λ _LR and λ _SR are weighting factors. The short exposure image R picked up by the single-chip color image sensor 102 g _SR, the long exposure image and g _LR. M is a power index and Q is a condition to be satisfied by the image f to be generated, that is, a constraint condition.

  The value of the exponent M in (Expression 1) is not particularly limited, but 1 or 2 is preferable from the viewpoint of the amount of calculation.

The first term of (Equation 1) is the g _LR obtained by sampling the R image f _R to be generated by the long exposure sampling process H _1R and the actual long exposure image. Means the calculation of the difference between If the long exposure sampling process H _1R is determined in advance and f _R that minimizes this difference is obtained, it can be said that f _R is the best match with g _LR obtained by the long exposure. Similarly, for the second term, f _R that minimizes the difference can be said to be the best match with g _SR obtained by short-time exposure. The third term is a constraint condition (constraint condition term) for uniquely determining the solution f _R.

Then, R image f _R that minimizes the evaluation function J _R (Equation 1), both g _LR and g _sR obtained by exposure and short exposure prolonged said to comprehensively well satisfactory. The image generation unit 202 calculates the R image f _R that minimizes Equation 1, thereby generating a pixel value of the R image with improved S / N.

J _R minimization in (Equation 1) is performed by obtaining an R image f _R where ∂J _R / ∂f _R = 0. This calculation, and a M = 2 in equation (1), when Q is a quadratic form of f _R are partial differential ∂J _R / ∂f _R by f _R of the above J _R is f _R 1 Since it becomes the following equation, it becomes a linear optimization problem, resulting in the calculation of simultaneous equations A _R f _R = b _R. That is, when (Equation 1) is partially differentiated by f _R , the following is obtained.

ここでＱ_Mは列数がｆ_Rの要素数に等しい行列であり、||・||はベクトルのノルムを表す。（数２）に（数３）を代入し、右辺＝０とおいて変形することで以下の連立方程式を得、Ｒ画像ｆ_Rを計算できる。

In (Equation 2), assuming that Q is a quadratic form of f _R , rewrite as follows.

Here, Q _M is a matrix whose number of columns is equal to the number of elements of f _R , and || · || represents the norm of the vector. By substituting (Equation 3) into (Equation 2) and transforming with the right side = 0, the following simultaneous equations can be obtained, and the R image f _R can be calculated.

  On the other hand, when M = 1, when M = an integer of 3 or more, and when M is not an integer, a nonlinear optimization problem occurs.

FIG. 5 shows a processing procedure of the conjugate gradient method. When the calculation becomes a linear optimization problem, the R image f _R that minimizes the evaluation function _JR can be calculated by the procedure of the conjugate gradient method shown in FIG. FIG. 6 shows a configuration example of the R image processing unit 402 for performing the processing shown in FIG. The R image processing unit 402 includes a coefficient calculation unit 501, a vector calculation unit 502, and a calculation unit 503. Note that the G image processing unit 403 and the B image processing unit 402 in FIG. 4 can be realized in the same manner as the configuration of the R image processing unit 402 in FIG.

The coefficient calculation unit 501 calculates a coefficient matrix A _R. The vector calculation unit 502 calculates a constant vector b _R. The calculation unit 503 solves the simultaneous equations of A _R f _R = b _R using the matrix A _R and the vector b _R obtained by the calculation. The process illustrated in FIG. 5 is mainly the process of the calculation unit 503. When the matrix A _R and the vector b _R are obtained and an initial value and an end condition are given from the control unit 104, the processing of FIG. 5 is started. The end condition in FIG. 5 is when the residual r _{k + 1} at the (k + 1) th step shown in FIG. 5 becomes sufficiently small.

Thus, an R image f _R that minimizes the evaluation function J _R is obtained. Since the algorithm shown in FIG. 5 is known, detailed description thereof is omitted.

  Hereinafter, (Equation 1) will be described in more detail.

The images f _R , g _LR and g _SR are vertical vectors whose elements are the pixel values of the moving image. Hereinafter, the vector notation for an image means a vertical vector in which pixel values are arranged in the raster scan order, and the function notation means a spatio-temporal distribution of pixel values. As a pixel value, in the case of a luminance value, one value may be considered per pixel. The number of elements of f _R is, for example, 2000 × 1000 × 30 = 60000000 if the moving image to be generated is 2000 pixels wide, 1000 pixels long, and 30 frames.

The number of vertical and horizontal pixels of the R image f _R and the number of frames used for signal processing are set by the color image processing unit 105 or the R image processing unit 402.

The long exposure sampling process H _1R is a matrix in which the number of rows is equal to the number of elements of g _{LR and} the number of columns is equal to the number of elements of f _R. The short-time exposure sampling process H _2R is a matrix whose number of rows is equal to the number of elements of g _{SR and} whose number of columns is equal to the number of elements of f _R.

Since the amount of information relating to the number of pixels of a moving image (for example, 2000 pixels wide × 1000 pixels in height) and the number of frames (for example, 30 frames) is too large in a computer that is currently in widespread use, f (1) is minimized. _R cannot be determined in a single process. In this case, the moving image f _R to be generated can be calculated by repeating the process of obtaining a part of f _R for the temporal and spatial partial regions.

Next, formulation of the long exposure sampling process H _1R will be described using a simple example. In FIG. 7, an image having a width of 4 pixels (x = 1 to 4), a height of 4 pixels (y = 1 to 4), and two frames (t = 1, 2) is displayed by a single-plate color imaging element 102 having a Bayer arrangement. The pixel arrangement when an image is captured and only the R component image is extracted by the image separation unit 401 is shown. This is equivalent to applying the processing in the image separation unit 401 to the left half 4 × 4 pixel region in the pixel array shown in FIG. 3. In the following, the g _LR imaging process in the case where the image is captured for a long time in the pixel array of FIG. 7 (the pixel having “RL” in FIG. 3) is accumulated for two frames.

Elements of the obtained R image f _R are expressed as in the following (Equation 5) including t = 1 and 2.

In (Expression 5), R _{111 to} R ₃₃₂ indicate R pixel values in each pixel, and three subscripts indicate x, y, and t values in order.

Of the above (Equation 5), R ₁₁₁ and R _{112 at the} position (1, 1) and R ₃₃₁ and R ₃₃₂ at the position (3, 3) are necessary for obtaining the long-time exposure. Therefore, a matrix for extracting only those pixel values can be defined as follows.

The matrix H _1R is set in consideration of the long exposure sampling process. That is, in order to obtain a long-exposure pixel at the position (1, 1), it is necessary to add each pixel value at t = 1, 2 at the position (x, y) = (1, 1). Further, in order to obtain a long-exposure pixel at the position (3, 3), it is necessary to add the pixel values of t = 1, 2 at the position (x, y) = (3, 3). According to these, the imaging process of the long exposure image is formulated as follows.

Next, the formulation of the sampling process H _2R for a short exposure will be described using a simple example. As in the previous example, consider the case of the arrangement of R pixels shown in FIG.

  In this case, since the exposure is performed for a short time, the pixel value of the short-time exposure pixel Rs in the left half of the pixel array in FIG. 3 is required for t = 1 and 2.

The elements of the obtained R image f _R are as shown in (Equation 5). Of these, what is necessary for obtaining the short-time exposure is R ₃₁₁ and R ₃₁₂ for the position (3, 1), the position (1, 3) is R ₁₃₁ and R ₁₃₂ . Therefore, a matrix for extracting only those pixel values can be defined as follows.

The first to fourth columns of (Equation 8) relate to time t = 1, and the fifth to eighth columns relate to time t = 2. According to these, the imaging process of the short time exposure line is formulated as follows.

(Equation 7) and (Equation 9) show the process of obtaining g _LR and g _SR by imaging the R image f _R by changing the exposure time with a Bayer array single-plate imaging device. Conversely, the problem of generating f _R from g _LR and g _SR is generally called an inverse problem. In the absence of the constraint condition Q, there are an infinite number of f _R that minimize the following (Equation 10).

This can be easily explained by the fact that (Equation 10) holds even if an arbitrary value is entered as the pixel value that is not sampled. Therefore, the R image f _R cannot be uniquely solved by minimizing (Equation 10).
Therefore, in order to obtain a unique solution for the R image f _R , a constraint condition Q is introduced. As Q, a smoothness constraint on the distribution of pixel values f _R is given. Hereinafter, how to give this constraint will be described.

First, as a constraint of smoothness regarding the distribution of pixel values of the R image f _R , the constraint equation of (Equation 11) or (Equation 12) is used.

As shown in (Equation 1), the constraint condition Q also constitutes a part of the evaluation function _JR . Therefore, in order to make the value of the evaluation function _JR as small as possible, it is preferable that the constraint condition Q is also small.

Regarding (Equation 11), the smaller the change in the distribution of the pixel value f _R , the smaller the Q value. Regarding (Equation 11), the value of Q decreases as the change in the distribution of the pixel value f _R is constant. These are the constraints of smoothness on the distribution of pixel values f _R.

  In addition, || represents the norm of the vector in (Equation 10). The value of the power exponent m is preferably 1 or 2 from the viewpoint of the amount of computation, like the power exponent M in (Equation 1) and (Equation 10).

Furthermore, as a constraint of smoothness regarding the distribution of the pixel values of the image f, the constraint condition is adapted according to the gradient of the pixel values of f _R by using either Q in the following (Equation 13) or (Equation 14). May be changed. Note that the gradient is obtained by first derivative of the distribution function f _R of the pixel values of the image f _R for the position.

In (Equation 13) and (Equation 14), w (x, y) is a gradient function of pixel values and is a weighting function for the constraint condition. For example, when the power sum of the gradient component of the pixel value shown in (Equation 15) is large, the value of w (x, y) is small, and in the opposite case, the value of w (x, y) is large. By doing so, the constraint condition can be adaptively changed according to the gradient of f _R.

ここで、ベクトルｎ_maxおよび角度θは方向微分が最大になる方向であり、下記（数１７）によって与えられる。

By introducing such a weight function, it is possible to prevent the newly generated image f _R from being smoothed more than necessary. Further, the weighting function w (x, y) may be defined by the magnitude of the power of the directional differentiation shown in (Expression 16) instead of the square sum of the components of the luminance gradient shown in (Expression 15).

Here, the vector n _max and the angle θ are directions in which the directional differential is maximized and are given by

The problem of solving (Equation 1) by introducing smoothness constraints on the distribution of pixel values of the moving image f _R as shown in (Equation 11), (Equation 12), and (Equation 13) to (Equation 17). Can be calculated using a known solution, for example, a solution to a variational problem such as a finite element method.

  Although a plurality of constraint condition expressions have been described here, the optimal combination of these may be determined according to the status of the apparatus such as cost, operation scale, circuit scale, and the like. For example, what has the best image quality may be selected. “The image quality is the best” can be determined based on the subjectivity of the person who confirms the video, for example.

An example of the evaluation function J _R is shown in (Equation 18).

The evaluation function J _R is defined as a function of the red image constituting the high resolution color image (target image) f _R to be generated. Here, the first term and the second term on the right side of (Equation 18) are the same as those when the exponent M in (Equation 1) is 2.

  The sum of squares of the pixel value difference at the corresponding pixel position of the low resolution image and the input image is set as the evaluation condition of the evaluation function (the first term and the second term on the right side of (Equation 18)). That is, these evaluation conditions represent the size of a difference vector between a vector having each pixel value included in the reduced resolution image as an element and a vector having each pixel value included in the input image as an element.

The third term on the right side of (Equation 18) describes the constraint on smoothness regarding the distribution of the pixel values of the image f _R , and this may take into account the pixel arrangement. In this case, as a constraint on the smoothness regarding the distribution of the pixel values of the R image f _R , the second-order partial differentiation of x and y is, for example, a filter of three coefficients of 1, −2, and 1, and the square thereof is 1, It becomes a filter of five coefficients of -4, 6, -4, and 1. In the present embodiment, in consideration of the smoothness in the x direction, in the pixel arrangement of FIG. 3, the smoothness constraint is calculated between the R pixels adjacent in the horizontal direction as shown in FIG. This formulation can be expressed as a quadratic norm of the product of a certain matrix Q _SR and f _R as shown in (Equation 18), or can be directly calculated from the elements of f _R. For the matrix Q _SR (several 17), wherein the second-order partial differential filter (coefficients 1, -2, 1) Q _SR1, the square of the one corresponding to (coefficient 1, 4,6, 4,1 ) _Is represented as Q _SR2 , and the square of the quadratic norm of the product of Q _SR1 and f _R and Q _SR2 and f _R can be expressed directly from the elements of f _R as follows: It becomes a shape.

In ( _Expression 20) and ( _Expression 21), R _xyt indicates the R pixel value at the position (x, y) and time t. Further, by adjusting the weighting coefficients λ _LR and λ _SR in the evaluation function _JR in (Equation 1), the image quality of the generated image can be adjusted. That is, when the noise is random, the noise level for long exposure is smaller than the noise level for short exposure. For this reason, when λ _{LR is set} to a large weight, priority is given to the constraint condition relating to long-time exposure, so that the noise suppression effect is increased and the S / N can be further improved. On the other hand, lambda case of a larger weight of _SR, in order to prioritize constraints concerning short exposure, the noise suppression effect is not so large, the high frequency component such as edge information remained, to generate an image having a resolution feeling Can do. The values of λ _LR and λ _SR may be determined so that the PSNR (Peak Signal to Noise Ratio) of the generated image is the best when the pixel values read out from all pixels by three-plate photography or the like can be prepared in advance. Further, the weight value may be determined by analyzing the shooting scene.

This may be performed, for example, by detecting the luminance value of the image. When the luminance value of the image is low, it can be estimated that the noise level is high. Therefore, when the average luminance value of the image is lower than the threshold value, the noise suppression effect is improved by setting the magnitude of λ _LR to be larger than λ _SR, and the average luminance value of the image is If it is lower, the size of λ _SR is set to be larger than λ _LR so that high-frequency components such as edge information remain.

Further, for example, it may be performed by detecting temporal or spatial luminance variations. When the variation in luminance is large, it can be estimated that the noise level is high. Therefore, when the luminance variation is higher than the threshold value, the noise suppression effect is improved by setting the size of λ _LR to be larger than λ _SR, and the luminance variation is lower than the threshold value. , Λ _SR is set to be larger than λ _LR so that high-frequency components such as edge information remain.

As described above, by determining the weighting coefficient in the evaluation function _JR depending on the shooting scene, it is possible to always generate an image with optimum image quality.

  In this embodiment, it is assumed that the number of accumulated frames in a long-exposure image is determined in advance. This determination method will be described in detail. If the noise is random noise and the number of accumulated frames is increased, the noise level of the long exposure becomes smaller than the noise level of the short exposure. Therefore, the noise suppression effect is increased and the S / N can be further improved. On the other hand, if the number of accumulated frames is reduced, an image closer to the short-time exposure image is generated, so the noise suppression effect is not so great, but a high-resolution component such as edge information remains is generated. Can do. Therefore, for example, the user may specify the number of frames to be stored. This is realized by having a user interface in the control unit 104. The user interface may have a known configuration such as a manual aperture adjustment function or a manual exposure adjustment function in the camera. As described above, since the image quality changes depending on the number of accumulated frames, having such an interface makes it possible to generate an image that matches the user's preference.

The processing in the G image processing unit 403 differs from the processing in the R image processing unit 402 in the form of the sampling process and the smoothness constraint on the space. FIG. 8 is a diagram of a pixel array when only the G pixels are extracted by the image separation unit 401 from the left half pixel group of the pixel array shown in FIG. 3 in the same manner as the description of the R image processing unit 402 described above. At this time, if the G image to be obtained is f _G , f _G is expressed by the following equation.

Of the above (Equation 22), in order to obtain long-time exposure, G ₂₁₁ and G _{212 at} position (2,1), G ₄₁₁ and G _{412 at} position (4,1), position (2, G ₂₃₁ and G ₂₃₂ in 3), G ₄₃₁ and G ₄₃₂ in position (4, 3). Therefore, a matrix for extracting only those pixel values can be defined as follows.

The matrix H _1G is set in consideration of the sampling process of long exposure. That is, in order to obtain a long-exposure pixel at the position (2, 1), it is necessary to add each pixel value at t = 1, 2 at the position (x, y) = (2, 1). Further, in order to obtain a long-exposure pixel at the position (4, 1), it is necessary to add the pixel values of t = 1, 2 at the position (x, y) = (4, 1). The same applies to the pixels with long exposure at the positions (x, y) = (2, 3) and (4, 3). According to these, the imaging process of the long exposure image is formulated as follows.

Next, regarding the formulation of the sampling process H _2G for a short exposure, consider the case of the arrangement of G pixels shown in FIG.

  In this case, since the exposure is performed for a short time, the pixel values of the short-time exposure pixels Gs in the left half of the pixel array in FIG. 3 are required for t = 1 and 2.

The elements of the G image f _G obtained are as shown in (Equation 22). Of these, what is necessary for obtaining the short-time exposure is G ₁₂₁ and G ₁₂₂ for the position (1, 2), and the position (3, 3). G ₃₂₁ and G ₃₂₂ for 2), the position (l, 4) G ₁₄₁ and G ₁₄₂ for, for the location (3,4) is a G ₃₄₁ and G _342. Therefore, a matrix for extracting only those pixel values can be defined as follows.

The first to eighth columns of (Equation 25) relate to time t = 1, and the ninth to sixteenth columns relate to time t = 2. According to these, the imaging process of the short time exposure line is formulated as follows.

In addition, the smoothness constraint regarding the distribution of the pixel values of the image f _G can be expressed in the same manner as (Expression 19) to (Expression 21) of the R image processing unit 402. At this time, a filter of three coefficients of 1, −2, 1 of second-order partial differentiation is applied to a zigzag type extending over two adjacent rows as shown in FIG. That is, when the smoothness constraint term is calculated directly from the G pixel value as in (Equation 20), the smoothness constraint Q _G1 of the G pixel in the pixel arrangement of FIG. .

Similarly to the processing in the R image processing unit 402, the square of the second-order partial differential filter is a filter of five coefficients 1, -4, 6, -4, and 1. Similarly, when performing the zigzag filter operation as shown in FIG. 9, if the equation corresponding to (Equation 21) is established for the G image, the filter operation takes the form of (Equation 28).

In ( _Equation 27) and ( _Equation 28), G _xyt represents the G pixel value at the position (x, y) and time t.

The processing in the B image processing unit 404 differs from the processing in the R image processing unit 402 in the form of the sampling process and the smoothness constraint on the space. FIG. 10 shows pixels when only the B pixel is extracted by the image separation unit 401 from the left half pixel group of the pixel array shown in FIG. 3, as in the description of the R image processing unit 402 and the G image processing unit 403. FIG. At this time, assuming that the B image to be obtained is f _B , f _B is expressed by the following equation.

Of the above (Equation 29), B ₂₂₁ and B _{222 at the} position (2, 2) and B ₄₄₁ and B ₄₄₂ at the position (4, 4) are required to obtain the long-time exposure. Therefore, a matrix for extracting only those pixel values can be defined as follows.

The matrix H _1B is set in consideration of the sampling process of long exposure. That is, in order to obtain a long-exposure pixel at the position (2, 2), it is necessary to add the pixel values at t = 1, 2 at the position (x, y) = (2, 2). Further, in order to obtain a long-exposure pixel at the position (4, 4), it is necessary to add the pixel values of t = 1, 2 at the position (x, y) = (4, 4). According to these, the imaging process of the long exposure image is formulated as follows.

Next, regarding the formulation of the sampling process H _2B for short exposure, consider the case of the arrangement of B pixels shown in FIG. In this case, since the exposure is performed for a short time, the pixel values of the short-time exposure pixels Bs in the left half of the pixel array in FIG. 3 are required for t = 1 and 2.

Elements of the obtained B image f _B are as shown in (Equation 29). Of these, what is necessary for obtaining a short-time exposure is B ₄₂₁ and B ₄₂₂ at positions (4, 2) and positions (2, 4). B ₂₄₁ and B ₂₄₂ in FIG. Therefore, a matrix for extracting only those pixel values can be defined as follows.

The first to fourth columns of (Equation 32) relate to time t = 1, and the fifth to eighth columns relate to time t = 2. According to these, the imaging process of the short time exposure line is formulated as follows.

Further, as a constraint on smoothness regarding the distribution of pixel values of the image f _B , a second-order partial differential filter is applied in the horizontal direction to B pixels belonging to the same row as shown in FIG. When a formula similar to (Expression 20) in the R image processing section 402 or (Expression 27) in the G image processing section 403 is established, the smoothness constraint Q _B1 of the B pixel can be expressed in the form shown in (Expression 34). .

Similarly to the processing in the R image processing unit 402 and the G image processing unit 403, when considering the square of the second-order partial differential filter, the horizontal filter operation is given in the form of (Equation 28).

In ( _Expression 34) and ( _Expression 35), B _xyt indicates the B pixel value at the position (x, y) and time t.

  Note that the arrangement of the color filters in the pixel array 1701 of the single-plate color solid-state imaging device 102 in the present invention is not limited to that shown in FIG. The four pixels surrounded by the broken line in FIG. 2 may be arranged as shown in FIG. 19, for example, and the arrangement of the long exposure pixels and the short exposure pixels may be as shown in FIG.

As described above, according to the first embodiment, the function of individually setting the exposure time for each pixel is added to the single-plate image sensor, and a new image is generated from the input image accumulated for each pixel. Thus, it is possible to estimate and generate an image with less motion blur while ensuring a light amount at the time of imaging and at a high resolution and a high frame rate.
Further, by performing R / G / B processing separately, it is possible to estimate and generate an image with little motion blur without degrading the image even for a subject having a negative color correlation.

  Furthermore, since the present invention applies smoothness constraint using adjacent short-time exposure pixels and long-time exposure pixels, motion blur of long-time exposure pixels can be easily reduced, and S / N can be further improved. it can. In addition, the number of horizontal pixel columns related to the smoothness constraint calculation, that is, the smoothness constraint calculation should be maintained by applying the smoothness constraint in a zigzag pattern that extends in one horizontal direction or at most two lines. The amount of pixel signal data can be reduced, and the amount of memory required for computation can be saved.

In the R image processing unit 402, the G image processing unit 403, and the B image processing unit 404 according to the present embodiment, the output image f _R , f _G, or f _B is obtained by solving the simultaneous equations obtained by (Equation 4). However, the solution of the simultaneous equations is not limited to the conjugate gradient method shown in FIG. Hereinafter, the R image processing unit 402 is taken as an example, and examples of other solutions are shown.

ここで、解くべき連立方程式を（数３６）のようにおく。

When M = 2 in (Expression 1) and the second expression of (Expression 21) is used as Q, the evaluation expression J _R is a quadratic expression of f _R. The calculation of f _R that minimizes the evaluation formula results in the calculation of simultaneous equations for f _R by (Equation 36).

Here, the simultaneous equations to be solved are set as (Equation 36).

In (Expression 37), since f _R has elements for the number of pixels to be generated (the number of pixels in one frame × the number of frames to be processed), the calculation amount of (Expression 37) is usually extremely large. As a method for solving such a large-scale simultaneous equation, a method (an iterative method) for converging the solution f _R by an iterative calculation such as a conjugate gradient method or a steepest descent method is generally used.

  However, in (Equation 37), since the evaluation functions are only the constraint term of the sampling process and the spatial smoothness constraint term, the processing does not depend on the content. By utilizing this, the inverse matrix of the coefficient matrix A of the simultaneous equations (Equation 37) can be calculated in advance, and by using this, image processing can be performed by a direct method.

When the smoothness constraint shown in (Expression 21) is used, the second-order partial differentiation of x and y becomes, for example, a filter of three coefficients 1, -2, and 1 as shown in (Expression 19), and its square Becomes a filter of five coefficients 1, -4, 6, -4, and 1. These coefficients can be diagonalized by sandwiching a coefficient matrix between horizontal Fourier transform and inverse transform. Similarly, the restriction on the sampling process of long exposure can be diagonalized by sandwiching a coefficient matrix between time-direction Fourier transform and inverse Fourier transform. That is, the matrix can be set as Λ as in (Equation 38).

As a result, the number of non-zero coefficients per row can be reduced as compared with the coefficient matrix A. As a result, the calculation of the inverse matrix Λ ⁻¹ of Λ becomes easy. Then, according to (Equation 39) and (Equation 40), f _R can be obtained with a smaller calculation amount and circuit scale without performing repeated calculations.

  Note that the R image processing unit 402, the G image processing unit 403, and the B image processing unit 404 in the present embodiment do not have to have separate hardware configurations, and one single-color image processing unit performs each processing in a time division manner. May be performed.

(Embodiment 2)
In the imaging processing apparatus according to the present invention, as an image processing method in the R image processing unit 402, the G image processing unit 403, and the B image processing unit 404, an image included in the output image f (f _R , f _G , f _B ) is processed. Smoothness constraints on the motion distribution can be used. In the present embodiment, a processing method using the constraint on the smoothness of the motion distribution will be described.

  FIG. 11 shows the configuration of the color image processing unit 105 according to the present embodiment. The functions of the image separation unit 1101 and the image integration unit 1108 in the figure are the same as those of the image separation unit 401 and the image integration unit 405 in the first embodiment. In the present embodiment, an R motion detector 1103, a G motion detector 1105, and a B motion detector 1107 are added to the R image generator 1102, the G image generator 1104, and the B image generator 1106, respectively.

  The processes performed by the R motion detection unit 1103, the G motion detection unit 1105, and the B motion detection unit 1107 in FIG. 11 are basically the same. Details will be described below by taking the R motion detection unit 1103 as an example. The R motion detection unit 1103 receives an image (short-time exposure image) formed by a pixel signal obtained by being exposed for one frame period. The R motion detection unit 1103 uses a short-time exposure image to detect motion (optical flow) from pixel values photographed with short-time exposure using existing known techniques such as block matching, gradient method, and phase correlation method. . Examples of known techniques include P.I. Anandan. “Computational framework and an algorithm for the measurement of visual motion”, International Journal of Computer Vision, Vol. 2, pp. 283-310, 1989 is known.

  FIGS. 12A and 12B show a base frame and a reference frame when motion detection is performed by block matching. The R motion detection unit 1103 sets a window area A shown in FIG. 12A in a reference frame (an image at time t when attention is given to obtain motion), and is similar to the pattern in the window area. Search for a pattern in the reference frame. As the reference frame, for example, a frame next to the frame of interest is often used.

As shown in FIG. 12B, the search range is normally set in advance with a certain range C based on the position B where the movement amount is zero. In addition, the degree of similarity (degree) of the pattern is determined by the residual flat saturation (SSD: Sum of Square Differences) shown in (Equation 41) and the residual absolute value sum (SAD: Sum of Absorbed Differences) shown in (Equation 42). ) Is calculated as an evaluation value.

  In (Equation 41) and (Equation 42), I (x, y, t) is a spatio-temporal distribution of images, that is, pixel values, and x, yεW is a pixel included in the window region of the reference frame. The coordinate value of

  The R motion detection unit 1103 searches for a group (u, v) that minimizes the evaluation value by changing (u, v) within the search range, and uses this as a motion vector between frames. . Specifically, by sequentially shifting the set position of the window area, a motion is obtained for each pixel or each block (for example, 8 pixels × 8 pixels), and a motion vector is generated.

  For the distribution of the value of (u, v) in the vicinity of (u, v) that minimizes (Equation 41) or (Equation 42) obtained as described above, a linear or quadratic function is obtained. By applying (a known technique known as an equiangular fitting method or a parabolic fitting method), motion detection with sub-pixel accuracy is performed.

Next, the movement of the image included in the R image f _R , the G image f _G, or the B image f _B of the processing contents in the R image generation unit 1102, the G image generation unit 1104, and the B image generation unit 1106 in the present embodiment. The smoothness constraint relating to the distribution of the image will be described using the R image generation unit 1102 as an example.

ここで、ｕは動画像ｆ_Rから得られる各画素についての動きベクトルのｘ方向の成分を要素とする縦ベクトル、ｖは動画像ｆから得られる各画素についての動きベクトルのｙ方向の成分を要素とする縦ベクトルである。 The smoothness constraint on the distribution of motion images contained in the R image f _R, using the following equation (43) or equation (44).

Here, u is a vertical vector whose element is the x-direction component of the motion vector for each pixel obtained from the moving image f _R , and v is the y-direction component of the motion vector for each pixel obtained from the moving image f. A vertical vector as an element.

The smoothness constraint on the distribution of image motion obtained from f _R is not limited to (Equation 43) and (Equation 44), but for example, the direction of the first or second floor shown in (Equation 45) or (Equation 46). It is good also as differentiation.

ここで、ｗ（ｘ，ｙ）は、ｆ_Rの画素値のこう配に関する重み関数と同一であり、（数１５）に示す画素値のこう配の成分のべき乗和、または、（数１６）に示す方向微分のべき乗によって定義される。 Further, as shown in (Equation 47) to (Equation 50), the constraint conditions of (Equation 43) to (Equation 46) may be adaptively changed according to the gradient of the pixel value of f _R. As a result, the effect of smoothness constraint on the edge (border where the luminance change is discontinuous) in the image can be reduced compared to the flat part of the image, so that the edge in the image can be reproduced more clearly. Can do.

Here, w (x, y) is the same as the weight function related to the gradient of the pixel value of f _R , and is represented by the power sum of the gradient component of the pixel value shown in (Equation 15) or (Equation 16). Defined by the power of the directional derivative.

By introducing such a weight function, it is possible to prevent the motion information of f _{R from} being smoothed more than necessary, and as a result, the newly generated image f _R is smoothed more than necessary. Can be prevented.

(Number 43) - as shown in equation (50), with respect to the problems solved by introducing the smoothness constraint on the distribution of motion obtained from the image f _R (Formula 1), the smoothness of f _R Complicated calculation is required as compared with the case of using constraints. This is because the image f _R to be newly generated and the motion information (u, v) depend on each other.

This problem can be calculated using a known solution, for example, a variational solution using an EM algorithm or the like. At that time, the image f _R to be newly generated and the initial value of the motion information (u, v) are necessary for the repeated calculation.

The initial value of f _R, may be used interpolated and enlarged image of the input image. On the other hand, as the motion information (u, v), motion information obtained by calculating (Equation 41) to (Equation 43) in the motion detection unit 201 is used. As a result, as described above, the color image processing unit 105 introduces the constraint on smoothness regarding the motion distribution obtained from the image f as shown in (Equation 43) to (Equation 50) (Equation 1). By solving the above, the image quality of the processing result can be improved.

ここで、Ｑ_fはｆ_Rの画素値のこう配に関する滑らかさの拘束、Ｑ_uvはｆ_Rから得られる画像の動きの分布に関する滑らかさの拘束、λ₁およびλ₂はＱ_fとＱ_uvの拘束に関する重みである。 The processing in the R image generation unit 1102 includes any one of the constraints on smoothness relating to the distribution of pixel values shown in (Equation 11) to (Equation 16), (Equation 19), and (Equation 20) of the first embodiment. Any one of the constraints on smoothness relating to the motion distribution shown in (Equation 43) to (Equation 50) can be combined and used simultaneously as in (Equation 51).

Where Q _f is the smoothness constraint on the gradient of the pixel value of f _R , Q _uv is the smoothness constraint on the motion distribution of the image obtained from f _R , and λ ₁ and λ ₂ are the Q _f and Q _uv It is a weight related to restraint.

In addition, the movement-related constraints are not limited to those relating to the smoothness of the motion vector distribution shown in (Equation 43) to (Equation 50), but the residual between corresponding points (the pixel value between the start point and the end point of the motion vector). You may make it make this small by making an evaluation value into (difference). The residual between corresponding points can be expressed as (Equation 52) when f _R is expressed as a function f _R (x, y, t).

Considering the entire image with f _R as a vector, the residual in each pixel can be expressed as a vector as shown in (Formula 53) below.

The sum of squares of the residuals can be expressed as shown below (Formula 53).

In (Expression 53) and (Expression 54), H _m is a matrix of the number of elements of the vector f _R (the total number of pixels in space-time) × f _R. In H _m , in each row, only elements corresponding to the viewpoint and end point of the motion vector have non-zero values, and other elements have zero values. When the motion vector has integer precision, the elements corresponding to the viewpoint and the end point have values of −1 and 1, respectively, and the other elements are 0.

  The problem of solving (Equation 1) by introducing both the smoothness constraint relating to the distribution of pixel values and the smoothness constraint relating to the distribution of image motion is also a variational problem using a known solution (for example, an EM algorithm). Can be used.

ここで、λ₃は拘束条件Ｑ_mに関する重みである。 (Equation 54) may be set as Q _m and the constraint condition may be changed to (Equation 55) in combination with (Equation 51).

Here, λ ₃ is a weight related to the constraint condition Q _m .

  In this embodiment, when motion detection is performed from the R short-time image, the G short-time image, and the B short-time image obtained by the image separation unit 1101 in FIG. 11, for example, each color component image is subjected to interpolation enlargement processing such as linear interpolation. What is necessary is just to make up for the missing part.

  In the motion detection in this embodiment, as shown in FIG. 11, motion information is obtained separately for each color from an image obtained by interpolating and enlarging a short-time exposure image of each of R, G, and B color components. Alternatively, for example, a luminance image is generated from a short-time exposure image of all the R, G, and B color components, and motion information of all the color components is obtained at once on the luminance image and used for processing each color component. May be. In this case, for example, as shown in FIG. 13, motion detection is performed before the image separation unit 1301 separates the RGB color components, and the motion information obtained there can be supplied to the image generation unit for each color component.

When the motion vector has sub-pixel accuracy, a plurality of elements corresponding to a plurality of pixels near the end point have values according to the value of the sub-pixel component of the motion vector.
By using the motion information extracted from the RGB short-time exposure moving image by the R motion detecting unit 1103, the G motion detecting unit 1105 and the B motion detecting unit 1107, or the motion detecting unit 1302 by the method described above, The color image processing unit 105 can increase the spatial and temporal resolution of RGB moving images captured by the array of image sensors.

(Embodiment 3)
The image processing according to the image pickup processing apparatus according to the present invention is not limited to the method of separately processing each RGB color component as described in the first and second embodiments, but may process all RGB color components at once. Any configuration may be used.

  In the present embodiment, the configuration of an imaging processing apparatus when processing such three color components simultaneously will be described.

  Image processing performed by the imaging processing apparatus according to the present embodiment is the same as that of the first and second embodiments in that simultaneous equations are solved (for example, (Equation 4) in the first embodiment). The difference from these examples is that the configuration of the present embodiment does not have the image separation unit 401 or 1101, 1301 used in the first and second embodiments, the form of the expression describing the sampling process is different, and the space The shape of the smoothness constraint of typical pixel values is different. In this embodiment as well, information about movement in an image can be used as in the second embodiment. At this time, the detection of the movement of each pixel can be realized by the same method as that used in the configuration of FIG.

FIG. 14 is a diagram in which the left half pixel group of the pixel array shown in FIG. 3 is extracted, and it is now considered that the values of these pixels are obtained by image processing. In the present embodiment, the value of 4 × 4 pixels shown in the figure is obtained at a time. At this time, if the RGB image to be obtained is f, f is expressed as follows.

Of the above (Equation 56), in order to obtain long exposure, R ₁₁₁ and R ₁₁₂ at position (1, 1), G ₂₁₁ and G _{212 at} position (2, 1), position (4, 4) are required. G ₄₁₁ and G ₄₁₂ in 1), B ₂₂₁ and G _{222 in} position (2,2), G ₂₃₁ and G _{232 in} position (3,2), R ₃₃₁ and R in position (3,3) ₃₃₂ , G ₄₃₁ and G ₄₃₂ at position (4, 3), B ₄₄₁ and G ₄₄₂ at position (4, 4). Therefore, a matrix for extracting only those pixel values can be defined as follows.

According to this, the imaging process of the long exposure image is formulated as follows.

Next, regarding the formulation of the sampling process H ₂ for short-time exposure, consider the case of the pixel array shown in FIG.

Elements of the obtained RGB image f are as shown in (Equation 56). Among them, R ₃₁₁ and R ₃₁₂ are required for the position (3, 1), and positions (1, 2) are required for obtaining the short-time exposure. G ₁₂₁ and G ₁₂₂ for position (3, 2), G ₃₂₁ and G ₃₂₂ for position (4, 2), B ₄₂₁ and B ₄₂₂ for position (4, 2), R ₁₃₁ and R ₁₃₂ for position (1, 3), G ₁₄₁ and G ₁₄₂ for position (4,1), G ₂₄₁ and G ₂₄₂ for position (2,4), and G ₃₄₁ and G ₃₄₂ for position (3,4). Therefore, a matrix for extracting only those pixel values can be defined as follows.

The 1st to 16th columns of (Equation 59) relate to the time t = 1, and the 17th to 32nd columns relate to the time t = 2. According to these, the imaging process of the short-time exposure pixel is formulated as follows.

  Also in the present embodiment, when obtaining each pixel value, it is possible to apply a spatial smoothness constraint condition of pixel values. In the configuration of the present embodiment, the pixel values of all the color components of RGB are obtained at once, but the spatial smoothness constraint at this time is a method of processing each color component individually as in the first embodiment ( (Equation 20), (Equation 21), (Equation 27), (Equation 28), (Equation 34), (Equation 35) in the first embodiment is also possible, and it is different from the space where three colors of RGB are stretched. It is also possible to convert to a three-dimensional space and apply on that space. Hereinafter, a case where the spatial smoothness constraint is applied after converting the RGB space to another space will be described.

As the smoothness constraint regarding the distribution of the pixel values of the RGB image f, the constraint equation of (Equation 61) or (Equation 62) is used.

This constraint condition Q constitutes a part of the evaluation function J. The evaluation function J has the same form as in (Equation 1) of the first embodiment, but since the pixel values of all the RGB colors are targeted, the evaluation function _JR in (Equation 1) is written as J.

In (Equation 61) and (Equation 62), || represents the norm of the vector. The value of the power index m is preferably 1 or 2 from the viewpoint of the amount of computation, as in the description in the first embodiment. Here, C _i is obtained by converting the R, G, and B values of the element of f by the following equation.

In (Equation 63), C ₁ , C ₂ , and C ₃ are changed from the first principal component to the third principal component of the RGB pixel value distribution in the general image, so that the regularization parameter for the smoothness constraint is obtained. This adjustment can be made easier than in the RGB space. That is, the C ₁ component is almost equal to the luminance, the C ₂ and C ₃ components can be regarded as two color components, and the influence of each term of the smoothness constraint on the newly generated image by adjusting λ _ci individually. Can be controlled. The values of λ _C1 to λ _C3 may be determined so that the PSNR (Peak Signal to Noise Ratio) of the generated image is the best when the pixel values read out from all pixels by three-plate photography or the like can be prepared in advance.

ここで、ｆ_maxは画素値の最大値（例えば、８ビット時２５５、１０ビット時１０２３）であり、σは正解画像と生成画像との差の標準偏差である。 PSNR is defined by (Equation 64).

Here, f _max is the maximum pixel value (for example, 255 for 8 bits and 1023 for 10 bits), and σ is the standard deviation of the difference between the correct image and the generated image.

On the other hand, in other cases, λ _ci may be determined while looking at the image quality newly generated by manual operation based on the above values.

The partial differential values ∂C _i / ∂x, ∂C _i / ∂y, ∂ ² C _i / ∂x ² , and ∂ ² C _i / ∂y ² are obtained by differential development based on pixel values in the vicinity of the target pixel. For example, approximate calculation can be performed by (Equation 65).

Difference expansion is not limited to the above (Equation 65), and other pixels in the vicinity may be referred to, for example, as shown in (Equation 66).

(Equation 66) is averaged in the vicinity of the calculated value of (Equation 65). Thereby, although spatial resolution falls, it can make it hard to receive the influence of a noise. Furthermore, as an intermediate between them, weighting may be performed with α in a range of 0 ≦ α ≦ 1, and the following formula may be employed.

  The difference expansion calculation method may be performed by predetermining α according to the noise level so that the image quality of the processing result is further improved, or in order to reduce the circuit scale and the calculation amount as much as possible ( Equation 65) may be used.

ここで、ベクトルｎ_minおよび角度θは１階の方向微分の２乗が最小になる方向であり、下記（数６９）によって与えられる。

Note that the smoothness constraint regarding the distribution of the pixel values of the image f is not limited to (Equation 61) and (Equation 62). For example, the absolute value of the absolute value of the second-order directional differentiation shown in (Equation 68) is expressed as follows. It may be used.

Here, the vector n _min and the angle θ are directions in which the square of the first-order directional differential is minimized, and are given by the following (Equation 69).

Furthermore, as a constraint of smoothness regarding the distribution of the pixel values of the image f, any one of the following (Equation 70) to (Equation 72) Q is used, and the constraint condition is adaptive according to the gradient of the pixel values of f. It may be changed to. The gradient is obtained by first-order differentiation of the distribution function f of the pixel value of the image f with respect to the position.

In (Equation 70) to (Equation 72), w (x, y) is a gradient function of the pixel value, and is a weighting function for the constraint condition. For example, when the power sum of the gradient component of the pixel value shown in (Equation 73) below is large, the value of w (x, y) is small, and in the opposite case, the value of w (x, y) is large. Then, the constraint condition can be adaptively changed according to the gradient of f.

  By introducing such a weight function, it is possible to prevent the newly generated image f from being smoothed more than necessary.

ここで、ベクトルｎ_maxおよび角度θは方向微分が最大になる方向であり、下記（数７５）によって与えられる。

The weight function w (x, y) may be defined by the magnitude of the power of the directional differentiation shown in (Expression 74) instead of the square sum of the components of the luminance gradient shown in (Expression 73).

Here, the vector n _max and the angle θ are directions in which the directional differential is maximized, and are given by the following (Equation 75).

  When introducing the constraint of smoothness regarding the distribution of pixel values of the moving image f as shown in (Equation 61), (Equation 62), and (Equation 66) to (Equation 72), Similarly, it can be calculated using a known solution, for example, a solution to a variational problem such as a finite element method.

  According to the present embodiment, correlation between each color component can be introduced into the image processing as shown in (Equation 63), and an image with higher image quality and less motion blur is estimated particularly for a subject having a positive color correlation. Can be generated.

  As described above, according to the imaging processing apparatus of the embodiment of the present invention, the short time exposure image is compared with the short time exposure image in which the pixel signal obtained by exposure with the exposure time corresponding to the predetermined frame rate is read. By arranging pixels to be read with a lower frame rate and longer exposure time than the exposure image, the amount of light accumulated during imaging can be increased, and an image with a higher S / N can be obtained. Also, compared to the case of performing motion blur removal processing for images read out at a low frame rate and long exposure for all pixels, by arranging pixels with short exposure and performing image restoration processing on the readout signal, It is possible to obtain an image signal in which color blur caused by the movement of the subject is suppressed. In the embodiment in which the smoothness constraint is applied in a horizontal direction or zigzag extending across at most two lines, the number of horizontal pixel columns related to the smoothness constraint calculation, that is, the smoothness constraint should be retained. The amount of pixel signal data can be reduced, and the amount of memory required for computation can be saved.

  The imaging apparatus and processing apparatus of the present invention are useful for high-resolution imaging at low light amounts and imaging with small pixels. The present invention can also be applied as an image processing program. The image processing program is a program for causing a computer to execute the steps of the image processing method, for example, and may be recorded on a tangible recording medium instead of temporarily.

DESCRIPTION OF SYMBOLS 100 Image pick-up processing apparatus 101 Optical system 102 Single plate color image sensor 103 Reading control part 104 Control part 105 Image processing part 401 Image separation part 402 R image processing part 403 G image processing part 404 B image processing part

Claims (14)

A first pixel unit in which two first color pixels, one second color pixel, and one third color pixel are arranged so as to form a Bayer array, two first color pixels, A single-plate color imaging device in which a plurality of second color pixels and a second pixel unit in which one third color pixel is arranged so as to form a Bayer array are alternately arranged in rows and columns. ,
One of the two first color pixels in the first pixel unit, the second color pixel and the third color pixel, and one of the two first color pixels in the second pixel unit, A pixel value is read at a first frame rate from a first color pixel located in the same row as the one of the first color pixels in the first pixel unit, and the two first color pixels in the first pixel unit are read out. And the second color pixel from the other of the two first color pixels in the second pixel unit, the second color pixel and the third color pixel have a pixel value lower than the first frame rate. A single-plate color image sensor that can be read at a frame rate;
An image processing unit that generates a new moving image at the first frame rate based on the pixel value read at the first frame rate and the pixel value read at the second frame rate. When,
An imaging processing apparatus comprising: The image processing unit
A first frame rate based on a pixel value read from the first color pixel at the first frame rate and a pixel value read from the first color pixel at the second frame rate. A first monochrome image processing unit for generating a first color moving image of
The first frame based on the pixel value read from the second color pixel at the first frame rate and the pixel value read from the second color pixel at the second frame rate. A second monochromatic image processing unit for generating a second color moving image of the rate;
Based on the pixel value read from the third color pixel at the first frame rate and the pixel value read from the third color pixel at the second frame rate, the first frame A third monochromatic image processing unit for generating a rate third color moving image;
With
The imaging processing apparatus according to claim 1, wherein the color moving image is generated from the first, second, and third moving images. The image processing unit
The first color moving image output from the first single color image processing unit, the second color moving image output from the second single color image processing unit, and the output from the third single color image processing unit The imaging processing apparatus according to claim 2, further comprising a memory that stores the third color moving image that has been recorded. The image processing unit
A moving image of the first color component, a moving image of the second color component, or a moving image of the third color component obtained based on the pixel value read at the first frame rate; A first difference from a moving image obtained by sampling the new moving image at the first frame rate;
A moving image of the first color component, a moving image of the second color component, or a moving image of the third color component obtained based on a pixel value read at the second frame rate; A second difference from a moving image obtained by sampling the new moving image at the second frame rate;
A constraint condition term that decreases as the change in the distribution of pixel values in the new moving image decreases or as the change in the pixel values in the new moving image becomes constant;
Set an evaluation formula containing
The imaging processing apparatus according to claim 1, wherein a moving image that minimizes the evaluation formula is obtained as the new moving image.   The imaging processing apparatus according to claim 4, wherein the image processing unit sets the constraint condition term for each of the first color pixel, the second color pixel, and the third color pixel.   The imaging processing apparatus according to claim 5, wherein the image processing unit calculates the constraint condition term for the first color pixel using pixel values in two rows of the single-plate color solid-state imaging device.   The image processing unit calculates the constraint condition term for the second color pixel and the third color pixel using pixel values in one row and / or one column of the single-plate color solid-state imaging device. Item 7. The imaging processing apparatus according to Item 5 or 6. The image processing unit
The imaging processing apparatus according to claim 1, wherein the evaluation formula includes a weighting factor multiplied by each of the first difference, the second difference, and the constraint condition term.   The single-plate color solid-state imaging device according to any one of claims 1 to 8, wherein the first color pixel, the second color pixel, and the third color pixel are a green pixel, a red pixel, and a blue pixel, respectively. The image processing unit
Using the moving image of the first color component, the moving image of the second color component, or the moving image of the third color component obtained based on the pixel value read at the first frame rate, A motion detector for detecting the motion of the subject;
The constraint condition term related to the motion distribution of the new moving image is set using the motion detection result, and the constraint condition term related to the motion distribution is included in the evaluation formula. The imaging processing apparatus described. An image processing method for performing image processing based on pixel values obtained from a single-plate color imaging device,
The single-plate color image sensor is
A first pixel unit in which two first color pixels, one second color pixel, and one third color pixel are arranged so as to form a Bayer array, two first color pixels, A single-plate color imaging device in which a plurality of second color pixels and a second pixel unit in which one third color pixel is arranged so as to form a Bayer array are alternately arranged in rows and columns. ,
One of the two first color pixels in the first pixel unit, the second color pixel and the third color pixel, and one of the two first color pixels in the second pixel unit, A pixel value is read at a first frame rate from a first color pixel located in the same row as the one of the first color pixels in the first pixel unit, and the two first color pixels in the first pixel unit are read out. And the second color pixel from the other of the two first color pixels in the second pixel unit, the second color pixel and the third color pixel have a pixel value lower than the first frame rate. Can be read at the frame rate,
Obtaining a pixel value read at the first frame rate by the single-plate color image sensor and a pixel value read at the second frame rate;
Generating a new moving image at the first frame rate based on the pixel value read at the first frame rate and the pixel value read at the second frame rate; ,
An image processing method including:   The image processing method according to claim 11, wherein the step of generating a new moving image at the first frame rate is executed for each color, and then includes a step of generating a color image at the first frame rate. An image processing program for performing image processing based on pixel values obtained from a single-plate color imaging device,
The single-plate color image sensor is
A first pixel unit in which two first color pixels, one second color pixel, and one third color pixel are arranged so as to form a Bayer array, two first color pixels, A single-plate color imaging device in which a plurality of second color pixels and a second pixel unit in which one third color pixel is arranged so as to form a Bayer array are alternately arranged in rows and columns. ,
One of the two first color pixels in the first pixel unit, the second color pixel and the third color pixel, and one of the two first color pixels in the second pixel unit, A pixel value is read at a first frame rate from a first color pixel located in the same row as the one of the first color pixels in the first pixel unit, and the two first color pixels in the first pixel unit are read out. And the second color pixel from the other of the two first color pixels in the second pixel unit, the second color pixel and the third color pixel have a pixel value lower than the first frame rate. Can be read at the frame rate,
Based on the pixel value read at the first frame rate and the pixel value read at the second frame rate, a new moving image at the first frame rate is generated on the computer. An image processing program.   The image processing program according to claim 11, wherein the step of generating a new moving image at the first frame rate is executed for each color, and then includes a step of generating a color image at the first frame rate.

Images