JP2010066925A - Learning apparatus, image processing apparatus, learning method, image processing method, and program - Google Patents

Abstract

An image pickup image is improved in image quality in consideration of characteristics of an image sensor corresponding to a pixel position.
A plurality of pixels included in a first image, and a plurality of pixels corresponding to pixel positions in the vicinity of the pixel of interest in the second image of higher quality than the first image are used as prediction taps. A prediction tap extraction unit to extract; a class classification unit that determines a class of the pixel of interest according to a pixel position of the pixel of interest in the second image; and a pre-acquired teacher image having the image quality of the second image And the student image having the image quality of the first image generated by applying different filters to the teacher image according to the pixel position, and predicting the pixel value of the target pixel from the pixel value of the prediction tap And a coefficient calculation unit that calculates a prediction coefficient used for each class determined by the class classification unit.
[Selection] Figure 7

Description

  The present invention relates to a learning device, an image processing device, a learning method, an image processing method, and a program.

  In the process of imaging light in the real world using an image sensor mounted on a digital camera or the like, there are various factors that degrade the quality of an image signal obtained as a result of imaging. For example, focus shift, subject movement, image sensor noise, or signal conversion in accordance with a pixel array such as a Bayer array are examples of factors that degrade the quality of an image signal.

  Therefore, technology development for removing noise included in the image signal or improving the image quality by image processing inside the camera has been in progress. For example, Patent Document 1 below discloses a technique called class classification adaptive processing that adaptively improves the quality of an image signal for each class classified according to the properties of the image signal.

JP 2004-260399 A

  However, in the conventional method, for example, the noise characteristics of the device are adaptively learned, and thereby high image quality can be achieved. However, unless the image quality is improved by trial and error through parameter tuning, There has been no example in which the sensor characteristics according to the pixel position in the sensor are appropriately considered. For example, the condensing characteristic of the image sensor is closely related to the aperture ratio of the pixel, and the condensing characteristic of the image sensor differs depending on the position of the pixel even within the same light receiving surface.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to improve the image quality of a captured image in consideration of the characteristics of the image sensor according to the pixel position. A new and improved learning device, image processing device, learning method, image processing method, and program are provided.

  In order to solve the above-described problem, according to one aspect of the present invention, a different filter is applied to a pre-acquired teacher image having a second image quality higher than that of the first image according to the pixel position. A student image generation unit that generates a student image having the image quality of the first image, and a plurality of pixels included in the student image, the pixels in the vicinity of the target pixel to be noted when the teacher image is predicted From a prediction tap extraction unit that extracts a plurality of pixels corresponding to positions as prediction taps, a class classification unit that determines a class of the pixel of interest according to the pixel position of the pixel of interest, and a pixel value of the prediction tap A learning device comprising: a coefficient calculation unit that calculates a prediction coefficient used for predicting a pixel value of the target pixel in the teacher image for each of the classes determined by the class classification unit. It is provided.

  The class classification unit may determine the class according to a distance from a center position of the teacher image to a pixel position of the target pixel.

  In addition, the first image is a captured image captured by an image sensor, and the student image generation unit uses a filter that reflects a specific light collection characteristic of the image sensor according to a pixel position as the teacher image. The student image may be generated by applying to the above.

  The student image generation unit may generate the student image by continuously changing the characteristics of the filter according to pixel positions.

  Further, the coefficient calculation unit may further calculate a prediction coefficient corresponding to a pixel position between representative pixel positions representing each class by linear interpolation using the prediction coefficient calculated for each class. Good.

  The first image may be a captured image captured using a camera including an image sensor, and the class classification unit may further include characteristics of the camera when the teacher image is captured by the camera, or The class may be determined according to camera parameters representing the state.

  The learning apparatus further includes a class tap extraction unit that extracts a plurality of pixels included in the student image and corresponding to a pixel position in the vicinity of the target pixel as a class tap, The class classification unit may determine the class according to a pixel value pattern of the class tap and a pixel position of the target pixel.

  In order to solve the above-described problem, according to another aspect of the present invention, a plurality of pixels included in a first image and the vicinity of a target pixel noted in a second image having a higher quality than the first image. A prediction tap extraction unit that extracts a plurality of pixels corresponding to the pixel position of the target pixel as a prediction tap; a class classification unit that determines a class of the target pixel according to a pixel position in the second image of the target pixel; A prediction coefficient used for predicting the pixel value of the target pixel from the pixel value of the prediction tap, the teacher image acquired in advance having the image quality of the second image, and the teacher image according to the pixel position A storage unit storing a prediction coefficient calculated for each class using a student image having the image quality of the first image generated by applying different filters, a pixel value of the prediction tap, and the Record By the obtained the prediction coefficients for linear combination from parts, the prediction arithmetic unit for calculating a predicted value corresponding to the pixel value of the target pixel, the image processing apparatus comprising a are provided.

  The class classification unit may determine the class according to a distance from a center position of the second image to a pixel position of the target pixel.

  In addition, the first image is a captured image captured by an image sensor, and the student image is applied to the teacher image a filter reflecting a specific light collection characteristic of the image sensor according to a pixel position. An image generated in this manner may be used.

  The student image may be an image generated by applying a filter whose characteristics continuously change according to a pixel position to the teacher image.

  Further, the prediction calculation unit further uses a prediction coefficient corresponding to a pixel position between representative pixel positions representing each class by linear interpolation, using the prediction coefficient for each class acquired from the storage unit. It may be calculated.

  In addition, the first image is a captured image captured using a camera including an image sensor, and the class classification unit further includes characteristics of the camera when the first image is captured by the camera. Or you may determine the said class according to the camera parameter showing a state.

  The image processing apparatus further includes a class tap extraction unit that extracts a plurality of pixels included in the first image and corresponding to pixel positions in the vicinity of the target pixel as class taps. The class classification unit may determine the class according to a pixel value pattern of the class tap and a pixel position of the target pixel.

  In order to solve the above-described problem, according to another aspect of the present invention, a different filter is applied to a pre-acquired teacher image having a second image quality higher than that of the first image according to the pixel position. Thus, a student image generation step for generating a student image having the image quality of the first image, and a plurality of pixels included in the student image, in the vicinity of the target pixel that is noticed when the teacher image is predicted. A prediction tap extracting step of extracting a plurality of pixels corresponding to the pixel position as a prediction tap; a class classification step of determining a class of the target pixel according to the pixel position of the target pixel; and a pixel value of the prediction tap To calculate a prediction coefficient used for predicting the pixel value of the target pixel in the teacher image for each class determined in the class classification step Learning method comprising the steps out, it is provided.

  In order to solve the above-described problem, according to another aspect of the present invention, a computer that controls the learning apparatus is configured to store a pixel position in a teacher image that is acquired in advance and has a second image quality higher than that of the first image. A student image generation unit that generates a student image having the image quality of the first image by applying different filters according to the method, and a plurality of pixels included in the student image when predicting the teacher image A prediction tap extraction unit that extracts a plurality of pixels corresponding to pixel positions in the vicinity of the target pixel of interest as a prediction tap, and a class classification unit that determines a class of the target pixel according to the pixel position of the target pixel And a prediction coefficient used for predicting the pixel value of the target pixel in the teacher image from the pixel value of the prediction tap for each class determined by the class classification unit. A coefficient calculation unit that, a program to function as is provided.

  In order to solve the above-described problem, according to another aspect of the present invention, a plurality of pixels included in a first image and the vicinity of a target pixel noted in a second image having a higher quality than the first image. A prediction tap extraction step of extracting a plurality of pixels corresponding to the pixel position as a prediction tap; a class classification step of determining a class of the target pixel according to a pixel position in the second image of the target pixel; A prediction coefficient used for predicting the pixel value of the target pixel from the pixel value of the prediction tap, the teacher image acquired in advance having the image quality of the second image, and the teacher image according to the pixel position The class classification class is stored in a storage unit storing a prediction coefficient calculated for each class using a student image having the image quality of the first image generated by applying different filters. A prediction coefficient acquisition step for acquiring the prediction coefficient corresponding to the class determined by the step, and a linear linear combination of the pixel value of the prediction tap and the prediction coefficient acquired by the prediction coefficient acquisition step. A prediction calculation step of calculating a prediction value corresponding to the pixel value of the pixel of interest.

  In order to solve the above-described problem, according to another aspect of the present invention, a computer that controls an image processing apparatus is a second pixel that is a plurality of pixels included in a first image and has a higher quality than the first image. A prediction tap extraction unit that extracts a plurality of pixels corresponding to a pixel position in the vicinity of the target pixel of interest in the image as a prediction tap; and according to the pixel position of the target pixel in the second image, A class classification unit for determining a class, a prediction coefficient used for predicting a pixel value of the target pixel from a pixel value of the prediction tap, and a teacher image acquired in advance having the image quality of the second image; A prediction coefficient calculated for each class using a student image having the image quality of the first image generated by applying a different filter to the teacher image according to the pixel position. And a prediction calculation unit that calculates a prediction value corresponding to the pixel value of the target pixel by linearly combining the pixel value of the prediction tap and the prediction coefficient acquired from the storage unit. A program is provided.

  As described above, according to the learning device, the image processing device, the learning method, the image processing method, and the program according to the present invention, the captured image is improved in image quality in consideration of the characteristics of the image sensor according to the pixel position. be able to.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The “best mode for carrying out the invention” will be described in the following order.
1. Overview of image quality improvement considering the characteristics of image sensors 1. First embodiment Second Embodiment 4. 3. Third embodiment Summary

<1. Overview of high image quality considering the characteristics of image sensors>
First, an outline of high image quality in consideration of the characteristics of the image sensor will be described.

  FIG. 1 is a schematic diagram showing an appearance of an image sensor 10 used for a digital still camera or the like as an example. The image sensor 10 may be any sensor that senses light in the real world and generates an electrical signal for each pixel, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

  Referring to FIG. 1, the image sensor 10 has a light receiving surface 12. The light receiving surface 12 typically includes a plurality of pixels that each sense light and generate an electrical signal. FIG. 1 also shows a pixel position P1 at the center of the light receiving surface 12 and a pixel position P2 at the periphery of the light receiving surface 12.

  2 to 5 are characteristic diagrams obtained by actually measuring the condensing characteristics as an example of the characteristics of the image sensor. Among these, FIG.2 and FIG.3 represents the condensing characteristic in the pixel position P1 and pixel position P2 of image sensor S1, respectively. 4 and 5 show the light condensing characteristics at the pixel position P1 and the pixel position P2 of the image sensor S2, respectively.

  2 to 5, each coordinate on a two-dimensional horizontal plane corresponds to the direction of incident light incident on the pixel P1 or the pixel P2. The height of the grid line with respect to the horizontal plane represents the contribution rate of each incident light to the signal level of the electrical signal (pixel signal) generated in the pixel.

  Here, assuming an ideal image sensor, for all pixels, the contribution rate of incident light perpendicularly incident on the pixel, that is, the contribution rate at the center (coordinate (0, 0)) of the characteristic diagram is 1. The contribution ratio of other incident light incident obliquely is preferably zero. However, the condensing characteristic of an actual image sensor shows that each pixel signal is generated while convolving incident light in a range having a certain spread at the incident angle, as can be understood from FIGS. Yes.

  Further, comparing FIG. 2 and FIG. 3 or FIG. 4 and FIG. 5, in any of the image sensors, the condensing characteristic at the pixel position P1 is relatively steep, and the condensing characteristic at the pixel position P2 is relatively gentle. It is. That is, it can be seen that the light condensing characteristics of the image sensor differ depending on the pixel position on the light receiving surface.

  For example, when FIG. 2 is compared with FIG. 4, even if the pixel position on the light receiving surface is the same P1, the light condensing characteristics are different between the image sensor S1 and the image sensor S2. The same applies when FIG. 3 and FIG. 5 are compared. That is, it can be seen that even if the pixel positions are the same, the light condensing characteristics can be changed by different image sensors.

  Such a difference in condensing characteristics of the image sensor is caused by, for example, the shape of the opening of the image sensor or the influence of an on-chip lens. Although not shown in the figure, even when the same individual is used, if the camera parameters representing the characteristics or state of the camera at the time of shooting are different, the light collection characteristics of the image sensor change according to the camera parameters. It is also possible.

  Here, the characteristics of the image sensor as described above can be handled as a filter that acts on the original image input to the image sensor. In that case, for example, the height of the grid line at each coordinate on the characteristic diagrams shown in FIGS. 2 to 5 is a filter coefficient representing the characteristic of the filter. Such filter coefficients can be actually measured as described above.

  Therefore, it is conceivable that a student image is generated from a given teacher image using a virtual filter reflecting the characteristics of the image sensor, and the teacher image is predicted from the generated image. Further, the accuracy of prediction can be improved by adaptive learning by applying the class classification adaptive processing described in Patent Document 1.

  FIG. 6 is an explanatory diagram for explaining learning processing and prediction processing using a virtual filter that reflects the characteristics of the image sensor.

In the learning process, first, a student image IS is generated by applying a filter reflecting characteristics of each actually measured image sensor to a given teacher image IT. Here, it is assumed that the teacher image IT is given with a higher resolution than the student image IS. For example, the pixel value IS _{u, v at} the pixel position (u, v) of the student image IS is calculated by the following equation using the pixel values IT _{u + i, v + j} near the teacher image IT corresponding to the pixel position. .

（１）

(1)

Here, α _{i, j} (u, v) represents individual filter coefficients to be multiplied to each pixel value in the teacher image IT in the filter processing reflecting the characteristics of the image sensor. As described above, the filter coefficient α _{i, j} (u, v) can vary depending on the pixel position (u, v).

  Then, a prediction coefficient for predicting the teacher image IT from the student image IS generated in this way is calculated according to the learning concept in the class classification adaptation process described in Patent Document 1.

  That is, each pixel value of the student image IS is mapped (mapped) to the teacher image IT using a predetermined prediction coefficient. For example, when a linear linear combination model is adopted as a mapping method using a prediction coefficient, the target pixel value y of the teacher image IT is obtained by the following linear linear expression (linear combination).

（２）

(2)

Here, d _n, of the prediction tap consisting of N pixels that have been extracted from the student image IS, representing pixel values of the n-th pixel. Also, w _n represent the prediction coefficients to be multiplied to n-th prediction tap. Note that the target pixel value y may be obtained not by the linear primary expression shown in Expression (2) but by a higher-order expression of the second or higher order.

Here, the true value of the pixel value of the kth pixel of interest is y _k , and the predicted value is y _k ′. Then, the prediction error _ek is expressed by the following equation.

（３）

(3)

Since the predicted value y _k ′ of Expression (3) is obtained according to Expression (2), when the predicted value y _k ′ of Expression (3) is replaced according to Expression (2), the following expression is obtained.

（４）

(4)

However, in Expression (4), dn _{, k} represents the nth pixel value among the prediction taps for the kth pixel of interest.

Here, the optimum prediction coefficient w _n is the concept of the least squares method, for example, it is determined by minimizing (minimum) the sum E of square errors expressed by the following equation as a statistical error.

（５）

(5)

  Here, K represents the total number of pixels of interest when predicting the teacher image IT.

Then, from equation (4) and (5), the optimal prediction coefficients _{w n} to the total sum E of the square errors to a minimum (local minimum), the normal equation represented by the following formula, for example, sweeping-out method (Gaus-Jordan It is calculated | required by solving using the elimination method.

（６）

(6)

That is, in the learning process in FIG. 6, after the student image IS is generated using the given teacher image IT and the filter coefficient α _{i, j} (u, v) corresponding to the pixel position, first, the target pixel position successively, the prediction taps d _n from the student image iS are extracted every. Next, using the prediction taps d _n and the teacher image IT of the target pixel value y _{k (true} value), the normal equation of formula (6) is generated. Then, by solving the normal equation of formula (6), prediction coefficients for predicting the teacher image IT from the student image IS w _n are calculated.

Further, in the prediction process in FIG. 6, using the prediction coefficient w _n which has been calculated by the learning process, from the captured image I1 captured by the image sensor, an original image I2 before imaging is predicted. Here, the captured image I1 corresponds to the student image IS described above, and the original image I2 corresponds to the teacher image IT described above. Prediction of the original image I2 from the captured image I1, after extracting prediction taps d _n from the captured image I1 as with learning processing is performed according to equation (2).

  Up to this point, the outline of high image quality considering the characteristics of the image sensor has been described with reference to FIGS. Hereinafter, in the present specification, the learning apparatus that performs the learning process described with reference to FIG. 6 and the image processing apparatus that performs the prediction process are in line with three embodiments of high image quality in consideration of the characteristics of the image sensor. This will be described in detail.

<2. First Embodiment>
[Learning device]
FIG. 7 is a block diagram showing a logical configuration of the learning device 100 according to the first embodiment of the present invention.

  Referring to FIG. 7, the learning apparatus 100 includes a student image generation unit 110, a characteristic storage unit 112, an image storage unit 116, a target pixel setting unit 120, a class classification unit 142, a prediction tap extraction unit 150, a teacher pixel extraction unit 152, A normal equation generation unit 160, a learning storage unit 162, a coefficient calculation unit 170, and a coefficient storage unit 172 are provided.

  When the teacher image IT is supplied to the learning apparatus 100, the student image generation unit 110 acquires the filter coefficient reflecting the characteristics of the image sensor from the characteristic storage unit 112, and uses the acquired filter coefficient to determine the student image IT from the student image IT. An image IS is generated.

More specifically, the student image generation unit 110 first pays attention to a pixel at a specific pixel position (u, v) in the student image IS. Then, the student image generation unit 110 acquires, for example, the filter coefficient α _{i, j} (u, v) representing the light collection characteristic of the image sensor corresponding to the pixel position (u, v) from the characteristic storage unit 112. Thereafter, the student image generation unit 110 calculates the _target pixel value IS _{u, v} of the student image IS from the filter coefficient α _{i, j} (u, v) and each pixel value of the teacher image IT according to the above-described equation (1). To do. The student image generation unit 110 generates the student image IS by repeating this process for all the pixels of the student image IS to be generated.

The characteristic storage unit 112 stores in advance filter coefficients α _{i, j} that reflect the characteristics of the image sensor to be learned. The filter coefficient α _{i, j} is actually measured according to the pixel position by a test after manufacturing the image sensor, for example.

  Here, for example, the condensing characteristic of the image sensor is generally constant on the circumference of a concentric circle centering on the center of the light receiving surface due to the symmetry of the shape of the opening of the sensor and the on-chip lens. Further, as the distance from the center of the light receiving surface increases, the light collection characteristics continuously change.

Therefore, for example, the characteristic storage unit 112 may store only the filter coefficients α _{i, j} for several pixel positions having different distances from the center of the light receiving surface. Then, the student image generation unit 110 calculates the distance from the center of the light receiving surface for the target pixel position (u, v), and interpolates the discrete filter coefficients acquired from the characteristic storage unit 112 according to the distance. The filter coefficient used for calculating the _target pixel value IS _{u, v} may be calculated. Further, the interpolation method in this case is not limited to linear interpolation, and may be a method using a higher-order interpolation function.

  The image storage unit 116 temporarily stores the student image IS generated by the student image generation unit 110.

The pixel-of-interest setting unit 120 acquires the student image IS from the image storage unit 116, and sequentially sets the pixel-of-interest position S (s, t) used for calculating a prediction coefficient for predicting the teacher image IT from the student image IS. When the target pixel position S is set, the target pixel setting unit 120 inputs the target pixel position S to the class classification unit 142 and determines a class corresponding to the target pixel position S. Further, the pixel of interest setting unit 120 inputs the pixel-of-interest position S and student image IS in the prediction tap extracting unit 150, thereby extracting a prediction tap d _n from the student image IS. In addition, the target pixel setting unit 120 inputs the target pixel position S to the teacher pixel extraction unit 152 and extracts the true value y _k of the teacher pixel at the target pixel position S from the teacher image IT.

  The class classification unit 142 classifies the pixel of interest into one of one or more classes according to the pixel-of-interest position S input from the pixel-of-interest setting unit 120, and sets a class code C representing the class of the pixel of interest. The data is output to the learning storage unit 162.

  The classes classified by the class classification unit 142 are classes classified according to the distance D (S) from the center of the light receiving surface of the image sensor to the target pixel position S, as shown as an example in FIG. Also good.

  Referring to FIG. 8A, four types of classes given class codes C1, C2, C3, and C4 are prepared. For each class code, classification conditions for classification into the class are shown.

  For example, the classification condition of the class code C1 is that the distance D (S) satisfies D (S) <D1. For example, the pixel position P1 in FIG. 8B is classified into the class represented by the class code C1 because the distance from the center of the light receiving surface 12 is smaller than D1. The classification condition for the class code C2 is D1 ≦ D (S) <D2, and the classification condition for the class code C3 is D2 ≦ D (S) <D3. Further, the classification condition of the class code C4 is D3 ≦ D (S). For example, the pixel position P2 in FIG. 8B is classified into the class represented by the class code C4 because the distance from the center of the light receiving surface 12 is greater than D3.

  The class classified by the class classification unit 142 is not limited to the example illustrated in FIG. 8, and may be an arbitrary class classified according to the target pixel position S. For example, the direction of a vector from the center of the light receiving surface to the target pixel position S may be associated with a class.

The prediction tap extracting unit 150 extracts a plurality of pixels adjacent to the pixel of interest position S contained in the student image IS as prediction taps d _n, and outputs the extracted predictive taps d _n to the normal equation generating unit 160.

Figure 9 shows an example of a prediction tap d _n extracted by the prediction tap extracting unit 150. In the example of FIG. 9, the prediction tap extracting unit 150, extracts the target pixel position S (s, t) centered at a total 13 disposed in a so-called diamond pixels _d 1 _{to d 13} as prediction taps _{d n} is doing. Note that the prediction taps d _n extracted by the prediction tap extracting unit 150 is not limited to the example of FIG. 9 may be set at any position or any number of pixels adjacent to the pixel of interest position S.

The teacher pixel extraction unit 152 extracts the pixel value at the target pixel position S input from the target pixel setting unit 120 from the teacher image IT, and outputs it to the normal equation generation unit 160 as a true value y _k . Note that _k in the true value y _k of the teacher pixel is given by making the target pixel position S (s, t) one-dimensional.

Normal equation generating unit 160, by using the true value y _k of the teacher pixels input from the predictive tap d _n and the teacher pixel extracting unit 152 which is input from the prediction tap extracting unit 150 is represented by the formula (6) Add to normal equation.

Here, the addition to the normal equation refers to a process of calculating each matrix and vector element appearing in the equation (6) and setting the calculated result in the normal equation. That is, the normal equation generating unit 160 first performs formula left side of the pixel values between the predictive tap d _n in matrix multiplication and summation (6) (Σ). Then, the normal equation generating unit 160 performs multiplication and summation (sigma) between the true value y _k of the pixel values of the prediction taps d _n in the vector of the right side and the teacher pixels of the formula (6). Then, the normal equation generation unit 160 sets the result of multiplication and summation (Σ) as a formal equation.

  The normal equation generation unit 160 performs the above-described addition for all the target pixel positions S set by the target pixel setting unit 120, and outputs the generated normal equations to the learning storage unit 162.

  The learning storage unit 162 stores the normal equation generated by the normal equation generation unit 160 for each class represented by the class code C determined by the class classification unit 142.

Coefficient calculation unit 170 acquires the normal equation from the learning memory unit 162 for each class as described above, by solving the obtained normal equation to calculate the prediction coefficient w _n for each class. The coefficient calculating unit 170 outputs the prediction coefficient w _n for each calculated class to the coefficient storage unit 172.

Coefficient storage unit 172 stores the prediction coefficient w _n for each input from the coefficient calculator 170 class. Prediction coefficient w _n stored here, in the image processing apparatus to be described later, is used to predict the pixel values of the original image from the pixel values of the prediction taps extracted from the captured image.

[Description of processing flow: Learning processing]
Next, an example of the flow of learning processing by the learning device 100 according to the present embodiment will be described using the flowchart of FIG.

  Referring to FIG. 10, first, a teacher image IT is supplied to the learning apparatus 100 (S202). The teacher image IT is input to the student image generation unit 110 and the teacher pixel extraction unit 152.

  Next, the student image IS is generated by the student image generation unit 110 using the teacher image IT and the filter coefficient acquired from the characteristic storage unit 112 (S204). The student image IS generated here is output to the image storage unit 116 and stored.

  Thereafter, the target pixel setting unit 120 acquires the student image IS from the image storage unit 116, and sets the target pixel position S of interest when predicting the teacher image IT from the student image IS (S206).

  Then, the class classification unit 142 determines a class corresponding to the target pixel position S, and a class code representing the determined class is output to the learning storage unit 162 (S208).

Furthermore, the prediction tap extracting unit 150 is extracted as a plurality of pixels prediction taps d _n located in the vicinity of the target pixel position S from the student image IS, extracted prediction taps d _n is output to the normal equation generating unit 160 (S210).

Further, the teacher pixel extraction unit 152 extracts the true value y _k of the pixel at the target pixel position S from the teacher image IT, and outputs the extracted true value y _k of the target pixel to the normal equation generation unit 160 (S212). ).

Then, the normal equation generating unit 160, by using the true value _{y k} of the prediction tap _{d n} and the pixel of interest, is performed summation to the normal equation (S214). The normal equation generated here is stored in the learning storage unit 162 for each class determined by the class classification unit 142.

  Thereafter, it is determined whether or not the addition to the normal equation has been completed for all the target pixels (S216). Here, if there is a target pixel that has not been added to the normal equation, the process returns to S206, and the target pixel setting unit 120 sets a new target pixel position S. On the other hand, if the addition to the normal equation has been completed for all the target pixels, the process proceeds to S218.

In S218, the coefficient calculation unit 170, a normal equation is obtained from sequential learning storing unit 162 for each class by solving the obtained normal equation, the prediction coefficient w _n for each class are calculated (S218). Calculated here prediction coefficient w _n is stored by the coefficient storage unit 172 for each class.

Then, whether the calculation of the prediction coefficients w _n for all classes been finished it is determined (S220). Here, if the remaining class calculation is not completed in the prediction coefficient w _n is, the process returns to S218, the calculation of the prediction coefficients w _n for the class remaining the coefficient calculation unit 170 is performed. On the other hand, the calculation of the prediction coefficients w _n for all classes if completed, the learning process is terminated.

[Modification]
In the present embodiment has been described for an example of calculating the prediction coefficient w _n by solving the normal equation for all classes. However, instead, to calculate the prediction coefficient w _n by solving the normal equations for some classes for the remaining classes, as described below, the relationship between the representative pixel positions representative of each class The prediction coefficient may be calculated by interpolation according to the above.

  FIG. 11 is an explanatory diagram for describing a representative pixel position representing a class.

  In FIG. 11, as an example, three classes (hereinafter referred to as class C1, class C2, and class C3) represented by class codes C1, C2, and C3 are defined according to the pixel position on the light receiving surface 12 of the image sensor. Has been. Class C1 is a class corresponding to a circular region including the center of the light receiving surface 12. Class C2 is a class that is located between class C1 and class C3 and corresponds to a region in which the distance from the center of light receiving surface 12 is medium. The class C3 is a class that is located outside the class C2 and corresponds to a region farthest from the center of the light receiving surface 12. FIG. 11 shows a representative pixel position Q1 of class C1, a representative pixel position Q2 of class C2, and a representative pixel position Q3 of class C3.

Here, for example, the prediction coefficient w _n of the class C1 and a class C3 is the already calculated by solving the normal equation described above. In that case, for example, by using a linear interpolation method shown in FIG. 12, it is possible to calculate the prediction coefficient w _n of the class C2.

In FIG. 12, the horizontal axis of the graph represents the distance from the center of the light receiving surface 12 of the representative pixel position of each class shown in FIG. 11. The ordinate of the graph represents the coefficient values of the prediction coefficient w _n for each class.

Figure 12 shows the class C1 prediction coefficient _W n of (representative pixel value Q1) (Q1), and the prediction coefficient value _W n (Q3) Class C3 (representative pixel value Q3) are each solved the normal equations Plotted from the calculated results. Then, the prediction coefficient value corresponding to the representative pixel position Q2 on the straight line connecting the two plotted end points becomes the prediction coefficient value W _n (Q2) of the class C2 (representative pixel value Q2). In FIG. 12, the prediction coefficient value of class C2 is calculated by linear interpolation. However, the interpolation method of the prediction coefficient value is not limited to linear interpolation, and may be any method.

Thus, by calculating by interpolation prediction coefficient w _n corresponding to a portion of a class or a part of the pixel positions, it is possible to reduce the amount of computation required for learning by the learning device 100.

  So far, the learning apparatus 100 according to the first embodiment of the present invention has been described. According to the learning device 100 according to the present embodiment, when the image sensor has different characteristics depending on the pixel position, the original image is predicted from the captured image obtained through the imaging process by the image sensor. Prediction coefficients are generated adaptively. Accordingly, it is possible to improve the image quality in consideration of the characteristics of the image sensor by the image processing apparatus described in the next section without performing trial and error image quality tuning.

  Further, the prediction coefficient is calculated for each class classified according to the pixel position of the target pixel when the original image is predicted from the captured image. That is, for example, for two pixels of interest located in the vicinity, only one set of prediction coefficients can be generated. Therefore, it is possible to learn the prediction coefficient with high accuracy while saving the amount of calculation required for learning the prediction coefficient and the storage area used for storing the prediction coefficient.

[Image processing device]
Next, an image processing apparatus that performs prediction processing for predicting an original image before imaging from a captured image using the prediction coefficient generated by the learning device 100 described above will be described. FIG. 13 is a block diagram showing a logical configuration of the image processing apparatus 300 according to the first embodiment of the present invention.

  Referring to FIG. 13, the image processing apparatus 300 includes a target pixel setting unit 320, a class classification unit 342, a prediction tap extraction unit 350, a coefficient storage unit 372, a prediction coefficient acquisition unit 374, and a prediction calculation unit 380.

  The image processing apparatus 300 performs a prediction calculation, which will be described in detail below, on the captured image I1 captured by the image sensor, using the prediction coefficient generated in advance by the learning apparatus 100, so that an original image before imaging is obtained. A corresponding predicted image I2 is generated and output.

  First, when the captured image I1 is supplied to the image processing apparatus 300, the captured image I1 is input to the target pixel setting unit 320.

  The target pixel setting unit 320 sequentially sets, as a target pixel, arbitrary pixels to be predicted from the predicted image I2 corresponding to the original image before being imaged by the image sensor. Then, the target pixel setting unit 320 outputs the pixel position S of the set target pixel in the predicted image I2 to the class classification unit 342. Further, the target pixel setting unit 320 outputs the captured image I1 and the target pixel position S to the prediction tap extraction unit 350.

  The class classification unit 342 classifies the target pixel into one of the classes associated with the prediction coefficient in the coefficient storage unit 372, which will be described later, according to the target pixel position S input from the target pixel setting unit 320. Then, the class classification unit 342 outputs the class code C representing the classified class to the prediction coefficient acquisition unit 374.

  For example, as described with reference to FIG. 8, the prediction coefficient is calculated in the coefficient storage unit 372 for each class divided according to the distance D (S) from the center of the light receiving surface of the image sensor to the target pixel position S. Suppose that it is remembered. In that case, the class classification unit 342 first calculates the distance D (S) from the center of the light receiving surface of the target pixel position S input from the target pixel setting unit 320. Then, the class classification unit 342 determines a class code C corresponding to the distance D (S) obtained as a result of the calculation according to the classification conditions illustrated in FIG. Then, the class classification unit 342 outputs the determined class code C to the prediction coefficient acquisition unit 374.

Coefficient storage unit 372, the prediction coefficient w _n obtained as a result of the above-described learning process by the learning apparatus 100, stored in each class corresponding to the pixel position. The coefficient storage unit 372, for example, reads out the prediction coefficient w _n stored at the address corresponding to the class code C specified by the prediction coefficient acquisition unit 374, and outputs the prediction coefficient acquisition unit 374.

Prediction coefficient acquiring unit 374 acquires the prediction coefficient w _n corresponding to the class code C that is input from the classification unit 342 from the coefficient storage unit 372, and outputs it to the prediction computation unit 380.

The prediction tap extracting unit 350, the target pixel setting section a plurality of pixels adjacent to the pixel of interest position S input from 320 as prediction taps d _n from the captured image I1, extracted prediction computation unit 380 prediction taps d _n were Output to. Here the prediction taps d _n extracted by the prediction tap extracting unit 350, for example, described in relation to FIG. 9, the pixel of interest position S (s, t) rhombic meter around the 13 pixels d and so on _{1 ~d 13.}

Prediction computation unit 380, in accordance with the equation (2) described above, and each coefficient value of the prediction coefficient w _n inputted from the prediction coefficient obtaining unit 374 and each pixel value of the prediction taps d _n inputted from the prediction tap extracting unit 350 Are linearly linearly combined to calculate a predicted value corresponding to the pixel value of the target pixel.

  Such calculation of the predicted value is repeated until all the target pixel values of the predicted image I2 are calculated.

[Description of process flow: Prediction process]
Next, an example of the flow of prediction processing by the image processing apparatus 300 according to the present embodiment will be described using the flowchart of FIG.

  Referring to FIG. 14, first, a captured image I1 is supplied to the image processing apparatus 300 (S402). The captured image I1 is input to the target pixel setting unit 320.

  Thereafter, the target pixel setting unit 320 sets the pixel position S of the target pixel to be predicted in the predicted image I2 (S404). The pixel position S set here is output to the class classification unit 342, and the target pixel position S and the captured image I1 are output to the prediction tap extraction unit 350.

  Then, the class classification unit 342 determines the class of the target pixel according to the target pixel position S, and outputs the class code C representing the determined class to the prediction coefficient acquisition unit 374 (S406).

Moreover, the prediction coefficient acquisition unit 374, the acquired prediction coefficient w _n in association with class code C stored in the coefficient storage unit 372, the obtained prediction coefficient w _n is output to the prediction computation unit 380 ( S408).

Furthermore, the prediction tap extracting unit 350, a plurality of pixels are extracted as prediction taps d _n located in the vicinity of the target pixel position S in the captured image I1, is output to the prediction computation unit 380 (S410).

Then, the prediction computation unit 380, linear combination of the prediction taps _{d n} and prediction coefficient _{w n} is performed according to equation (2), the predicted value of the pixel of interest is calculated (S412).

  Thereafter, it is determined whether or not the calculation of the prediction values for all the target pixels has been completed (S414). Here, if there remains a pixel for which the calculation of the predicted value has not been completed, the process returns to S <b> 404, and a new target pixel is set by the target pixel setting unit 320. On the other hand, if the calculation of the predicted value has been completed for all the pixels of the predicted image I2, the prediction process ends.

  Up to this point, the image processing apparatus 300 according to the first embodiment of the present invention has been described. According to the image processing apparatus 300 according to the present embodiment, when the image sensor has different characteristics depending on the pixel position, the prediction coefficient obtained as a result of learning by the learning apparatus 100 described in the previous section is used. The original image can be predicted from the captured image. Thereby, it is possible to improve the image quality of the captured image in consideration of the characteristics of the image sensor without performing trial and error image quality tuning.

  Here, the example in which the prediction coefficient corresponding to the class determined according to the target pixel position is applied to the linear linear combination by the prediction calculation unit 380 as it is has been described. However, instead, a prediction coefficient corresponding to a plurality of classes is acquired according to the target pixel position, and the prediction coefficient is interpolated according to the relationship between the representative pixel position representing each class and the target pixel position. You may apply. For example, the linear interpolation illustrated in FIG. 12 or a higher-order interpolation method can be used. By doing so, it is possible to perform highly accurate prediction while saving a storage area necessary for storing the prediction coefficient.

<3. Second Embodiment>
In the first embodiment, the example in which the image quality of the captured image is improved considering only the characteristics of the image sensor according to the pixel position has been described. Here, as described above, when the camera parameters at the time of capturing an image are different, the characteristics of the image sensor may be changed according to the camera parameters in addition to the pixel position. Therefore, in the second embodiment of the present invention, an example will be described in which the image quality of a captured image is improved in consideration of camera parameters during imaging.

[Learning device]
FIG. 15 is a block diagram showing a logical configuration of a learning device 500 according to the second embodiment of the present invention.

  Referring to FIG. 15, the learning apparatus 500 includes a student image generation unit 510, a characteristic storage unit 512, a camera parameter acquisition unit 514, an image storage unit 516, a target pixel setting unit 520, a class classification unit 542, a prediction tap extraction unit 150, A teacher pixel extraction unit 152, a normal equation generation unit 160, a learning storage unit 162, a coefficient calculation unit 170, and a coefficient storage unit 172 are provided.

  During learning by the learning device 500, the teacher image IT having a higher quality than the captured image and the camera parameters CP when the teacher image IT is captured are supplied to the learning device 500. Here, the camera parameter is an image sensor such as a focal length (focus), zoom, aperture (iris), or depth of field, among parameters representing the characteristics or state of the camera when an image is taken. It may be any parameter that affects the characteristics of

  15, the prediction tap extraction unit 150, the teacher pixel extraction unit 152, the normal equation generation unit 160, the learning storage unit 162, the coefficient calculation unit 170, and the coefficient storage unit 172 are respectively illustrated in FIG. It has a function equivalent to the content explained in relation to it. Therefore, here, the student image generation unit 510, the characteristic storage unit 512, the camera parameter acquisition unit 514, the image storage unit 516, the target pixel setting unit 520, and the class classification unit 542 will be mainly described.

  In FIG. 15, the camera parameter acquisition unit 514 acquires the camera parameter CP when the teacher image IT is captured and outputs it to the student image generation unit 510.

  When the teacher image IT is supplied, the student image generation unit 510 acquires a filter coefficient corresponding to the camera parameter CP input from the camera parameter acquisition unit 514 from the characteristic storage unit 512. Then, the student image generation unit 510 generates a student image IS from the teacher image IT using the acquired filter coefficient.

  Alternatively, when the teacher image IT is supplied, the student image generation unit 510 recalculates the filter coefficient acquired from the characteristic storage unit 512 to a value according to the camera parameter CP, and uses the recalculated filter coefficient to reinforce the teacher image. A student image IS may be generated from IT.

  For example, the condensing characteristic of an image sensor is generally steep when the subject is in focus, and is gentle when the subject is not in focus. Therefore, the filter coefficient may be recalculated according to the degree of coincidence between the distance to the subject and the focal length in the teacher image IT.

The characteristic storage unit 512 stores in advance filter coefficients α _{i, j} reflecting the characteristics of the image sensor to be learned in association with, for example, the camera parameter CP.

  The image storage unit 516 temporarily stores the student image IS generated by the student image generation unit 510 in association with the camera parameter CP.

  The pixel-of-interest setting unit 520 acquires the camera parameter CP and the student image IS from the image storage unit 516, and sequentially sets the pixel-of-interest position S used for calculating a prediction coefficient for predicting the teacher image IT from the student image IS. Then, the pixel-of-interest setting unit 520 inputs the camera parameter CP and the pixel-of-interest position S to the class classification unit 542. Further, the target pixel setting unit 520 inputs the student image IS and the target pixel position S to the prediction tap extraction unit 150. In addition, the target pixel setting unit 520 inputs the target pixel position S to the teacher pixel extraction unit 152.

  The class classification unit 542 determines any class corresponding to the camera parameter CP and the target pixel position S input from the target pixel setting unit 520, and outputs a class code C representing the determined class to the learning storage unit 162. .

  The class determined by the class classification unit 542 is related to the condition relating to the distance D (S) from the center of the light receiving surface of the image sensor to the target pixel position S and the value of the camera parameter CP, as shown as an example in FIG. It may be a class given according to conditions.

  Referring to FIG. 16, when CP = CP1, class C11 if D (S) <D1, class C12 if D1 ≦ D (S) <D2, and class C13 if D2 ≦ D (S). Given each. Similarly, when CP = CP2, class C21 is given if D (S) <D1, class C22 is given if D1 ≦ D (S) <D2, and class C23 is given if D2 ≦ D (S). . In the case of CP = CP3, class C31 is given if D (S) <D1, class C32 is given if D1 ≦ D (S) <D2, and class C33 is given if D2 ≦ D (S).

Such a structure of the learning device 500, the prediction coefficient w _n for predicting the teacher image IT from the student image IS is adaptively learned for each class corresponding to the target pixel position S and the camera parameters CP.

[Description of processing flow: Learning processing]
Next, an example of the flow of learning processing by the learning device 500 according to the present embodiment will be described using the flowchart of FIG.

  Referring to FIG. 17, first, a teacher image IT is supplied to the learning apparatus 500 (S602). The teacher image IT is input to the student image generation unit 510 and the teacher pixel extraction unit 152.

  Next, the camera parameter acquisition unit 514 acquires the camera parameter CP when the teacher image IT is captured and inputs it to the student image generation unit 510 (S604).

  Then, the student image generation unit 510 generates a student image IS using the teacher image IT, the camera parameter CP, and the filter coefficient acquired from the characteristic storage unit 512 (S606). The student image IS generated here is output to the image storage unit 516 and stored.

  Next, the target pixel setting unit 520 acquires the student image IS from the image storage unit 516, and sets the target pixel position S to be noted when predicting the teacher image IT from the student image IS (S608).

  Then, the class according to the target pixel position S and the camera parameter CP is determined by the class classification unit 542, and the class code C representing the determined class is output to the learning storage unit 162 (S610).

Furthermore, the prediction tap extracting unit 150 is extracted as a plurality of pixels prediction taps d _n located in the vicinity of the target pixel position S from the student image IS, extracted prediction taps d _n is output to the normal equation generating unit 160 (S612).

Further, the teacher pixel extraction unit 152 extracts the true value y _k of the pixel at the target pixel position S from the teacher image IT, and outputs it to the normal equation generation unit 160 (S614).

Then, the normal equation generating unit 160, by using the true value _{y k} of the prediction tap _{d n} and the pixel of interest, is performed summation to the normal equation (S616). The normal equation generated here is stored in the learning storage unit 162 for each class determined by the class classification unit 542.

  Thereafter, it is determined whether or not the addition to the normal equation has been completed for all the target pixels (S618). Here, if there remains a target pixel that has not been added to the normal equation, the process returns to S608, and the target pixel setting unit 520 sets a new target pixel position S. On the other hand, if the addition to the normal equation has been completed for all the target pixels, the process proceeds to S620.

In S620, the coefficient calculation unit 170, a normal equation is obtained in each class from learning and storing unit 162, by solving the obtained normal equation, the prediction coefficient w _n for each class are calculated (S620). Calculated here prediction coefficient w _n is stored by the coefficient storage unit 172 for each class.

Then, whether the calculation of the prediction coefficients w _n for all classes been finished it is determined (S622). Here, if the remaining class calculation is not completed in the prediction coefficient w _n is, the process returns to S620, the calculation of the prediction coefficients w _n for remaining classes are performed. On the other hand, the calculation of the prediction coefficients w _n for all classes if completed, the learning process is terminated.

  So far, the learning apparatus 500 according to the second embodiment of the present invention has been described. According to the learning apparatus 500 according to the present embodiment, when the characteristics of the image sensor according to the pixel position further vary depending on the camera parameters at the time of image capturing, the prediction coefficient for predicting the original image from the captured image is adaptive. Generated. Accordingly, it is possible to improve the image quality of a captured image in consideration of camera parameters by the image processing apparatus described in the next section without performing trial and error image quality tuning.

[Image processing device]
Next, an image processing apparatus that performs a prediction process for predicting an original image before imaging from a captured image using the prediction coefficient generated by the learning apparatus 500 described above will be described. FIG. 18 is a block diagram showing a logical configuration of an image processing apparatus 700 according to the second embodiment of the present invention.

  Referring to FIG. 18, the image processing apparatus 700 includes a camera parameter acquisition unit 714, a pixel-of-interest setting unit 320, a class classification unit 742, a prediction tap extraction unit 350, a coefficient storage unit 372, a prediction coefficient acquisition unit 374, and a prediction calculation unit. 380.

  Of the functional blocks in FIG. 18, the pixel-of-interest setting unit 320, the prediction tap extraction unit 350, the coefficient storage unit 372, the prediction coefficient acquisition unit 374, and the prediction calculation unit 380 have been described with reference to FIG. 13. Has the same function as the content. Therefore, here, the camera parameter acquisition unit 714 and the class classification unit 742 will be mainly described.

  When the captured image I1 is supplied to the image processing apparatus 700, the camera parameter acquisition unit 714 acquires the camera parameter CP when the captured image I1 is captured. As described above, the camera parameter CP may be any parameter that affects the characteristics of the image sensor, such as focal length (focus), zoom, aperture (iris), or depth of field. Then, the camera parameter acquisition unit 714 outputs the acquired camera parameter CP to the class classification unit 742.

  The class classification unit 742 determines a class according to the condition described in relation to FIG. 16 according to the target pixel position S and the camera parameter CP input from the camera parameter acquisition unit 714, and determines the class code C as a prediction coefficient. The data is output to the acquisition unit 374.

  With such a configuration of the image processing apparatus 700, a predicted image I2 corresponding to an original image that has been improved in image quality in consideration of not only the pixel position but also the camera parameter CP from the captured image I1 captured using the image sensor. Generated.

[Description of process flow: Prediction process]
Next, an example of the flow of prediction processing by the image processing apparatus 700 according to the present embodiment will be described using the flowchart of FIG.

  Referring to FIG. 24, first, a captured image I1 is supplied to the image processing apparatus 700 (S802). The supplied captured image I1 is input to the target pixel setting unit 320.

  Next, the camera parameter CP when the captured image I1 is captured is acquired by the camera parameter acquisition unit 714 and output to the class classification unit 742 (S804).

  Thereafter, the target pixel setting unit 320 sets a target pixel to be predicted in the predicted image I2 (S806). The pixel position S of the target pixel set here is output to the class classification unit 742, and the target pixel position S and the captured image I1 are output to the prediction tap extraction unit 350.

  Then, the class classification unit 742 determines a class according to the target pixel position S and the camera parameter CP, and outputs a class code C representing the determined class to the prediction coefficient acquisition unit 374 (S808).

Moreover, the prediction coefficient acquisition unit 374, the acquired prediction coefficient w _n in association with class code C stored in the coefficient storage unit 372, the obtained prediction coefficient w _n is output to the prediction computation unit 380 ( S810).

Furthermore, the prediction tap extracting unit 350, a plurality of pixels are extracted as prediction taps d _n located in the vicinity of the target pixel position S in the captured image I1, is output to the prediction computation unit 380 (S812).

Then, the prediction computation unit 380, linear combination of the prediction taps _{d n} and prediction coefficient _{w n} is performed according to equation (2), the predicted value of the pixel of interest is calculated (S814).

  Thereafter, it is determined whether or not the calculation of the predicted values for all the target pixels has been completed (S816). Here, if there remains a pixel for which the calculation of the predicted value has not been completed, the process returns to S806, and the target pixel setting unit 320 sets a new target pixel. On the other hand, if the calculation of the predicted value has been completed for all the pixels of the predicted image I2, the prediction process ends.

  Up to this point, the image processing apparatus 700 according to the second embodiment of the present invention has been described. According to the image processing apparatus 700 according to the present embodiment, when the image sensor has different characteristics depending on the camera parameter in addition to the pixel position, the learning result obtained by the learning apparatus 500 described in the previous section was obtained. The original image can be predicted from the captured image using the prediction coefficient. Accordingly, it is possible to improve the image quality of the captured image in consideration of the camera parameters without performing trial and error image quality tuning.

  Note that the learning according to the first to third embodiments described in this specification, that is, the calculation of the prediction coefficient in consideration of the characteristics of the image sensor, typically reflects the characteristics of each product more accurately. In addition, it is performed before the shipment after the product is manufactured. In such a case, since learning is performed in a range closed to one product, it is not necessary to consider the type of product at the time of learning.

  However, instead, for example, learning may be performed at once on a plurality of types of image sensors using independent learning modules. For example, as learning processing by the learning apparatus 500 according to the present embodiment, the camera parameter acquisition unit 514 acquires the individual identification number of the image sensor, and determines the class according to the individual identification number in addition to the target pixel position X. Also good. In such a case, a prediction coefficient is calculated for each class corresponding to the target pixel position X and the individual identification number. In the prediction processing by the image processing apparatus 700, the individual identification number is acquired by the camera parameter acquisition unit 714 of the image processing apparatus 700, and the prediction coefficient corresponding to the class determined according to the pixel position and the individual identification number is used. Thus, the original image is predicted from the captured image.

<4. Third Embodiment>
The high image quality of the captured image in consideration of the characteristics of the image sensor can be combined with the class classification according to the pixel value pattern in the vicinity of the target pixel. Therefore, in the third embodiment of the present invention, an example will be described in which a captured image is improved in image quality in consideration of a pattern of pixel values in the vicinity of the target pixel in addition to the pixel position.

[Learning device]
FIG. 20 is a block diagram showing a logical configuration of a learning device 900 according to the third embodiment of the present invention.

  Referring to FIG. 20, the learning apparatus 900 includes a student image generation unit 110, a characteristic storage unit 112, an image storage unit 116, a target pixel setting unit 920, a class tap extraction unit 940, a class classification unit 942, a prediction tap extraction unit 150, A teacher pixel extraction unit 152, a normal equation generation unit 160, a learning storage unit 162, a coefficient calculation unit 170, and a coefficient storage unit 172 are provided.

  Note that here, the pixel-of-interest setting unit 920, the class tap extraction unit 940, and the class classification unit 942 will be mainly described.

  The target pixel setting unit 920 acquires the student image IS generated by the student image generation unit 110 from the image storage unit 116, and sequentially sets the target pixel position S used for calculating the prediction coefficient for predicting the teacher image IT. Then, the pixel-of-interest setting unit 920 outputs the student image IS and the pixel-of-interest position S to the class tap extraction unit 940 and the prediction tap extraction unit 150. Also, the target pixel setting unit 920 outputs the target pixel position S to the teacher pixel extraction unit 152.

The class tap extraction unit 940 extracts the class tap _xn used for class classification of the target pixel from the student image IS and outputs the class tap _xn to the class classification unit 942. Here, the class tap refers to a set of pixels located in the vicinity of the target pixel for performing class classification according to the pattern of the pixel value.

FIG. 21 shows an example of the class tap _xn extracted by the class tap extraction unit 940. In the example of FIG. 21, the class tap extracting unit 940, the target pixel position S (s, t) centered at a total of nine placed in a so-called cross-shaped one by five vertical and horizontal pixels x ₁ ~x ₉ Extracted as class tap _xn . The class tap _xn extracted by the class tap extraction unit 940 is not limited to the example in FIG. 21, and may be an arbitrary position near the target pixel position S or a set of an arbitrary number of pixels.

The class classification unit 942 determines any class corresponding to the pixel value pattern of the class tap _xn input from the class tap extraction unit 940 and the target pixel position S, and learns the class code C representing the determined class The data is output to the storage unit 162.

As a method of classifying the class according to the pixel value pattern of the class tap _xn , for example, ADRC (Adaptive Dynamic Range Coding) can be used. When ADRC is used, each pixel value included in the class tap _xn is subjected to ADRC processing, and as a result, an ADRC code is obtained.

More specifically, for example, in the K-bit ADRC, first, the maximum value MAX and the minimum value MIN of each pixel value included in the class tap _xn are detected. Then, DR = MAX−MIN is set as a local dynamic range of the set of pixel values, and each pixel value included in the class tap _xn is quantized again to K bits based on the dynamic range DR. That is, the minimum value MIN is subtracted from each pixel value included in the class tap _xn , and the subtracted value is divided (quantized) by DR / 2K. Then, a bit string in which the K-bit pixel values obtained as described above are arranged in a predetermined order is obtained as an ADRC code.

For example, when the class tap _xn is subjected to, for example, 1-bit ADRC processing, the pixel value of each pixel constituting the class tap is divided by the average value of the maximum value MAX and the minimum value MIN (rounded down). Thereby, each pixel value is binarized. Then, a bit string in which the binarized pixel values are arranged in a predetermined order is obtained as an ADRC code.

  For example, the class classification unit 942 determines one of the classes according to the ADRC code thus obtained and the target pixel position S, and outputs the class code C representing the determined class to the learning storage unit 162. To do.

  Note that the class classification unit 942 may determine the class not by ADRC processing but by, for example, regarding pixels constituting the class tap as vector components and vector quantization of the vectors.

[Description of processing flow: Learning processing]
Next, an example of the flow of learning processing by the learning apparatus 900 according to the present embodiment will be described using the flowchart of FIG.

  Referring to FIG. 22, first, a teacher image IT is supplied to the learning apparatus 900 (S1002). The teacher image IT is input to the student image generation unit 110 and the teacher pixel extraction unit 152.

  Next, the student image IS is generated by the student image generation unit 110 using the teacher image IT and the filter coefficient acquired from the characteristic storage unit 112 (S1004). The student image IS generated here is output to the image storage unit 116 and stored.

  Thereafter, the target pixel setting unit 920 acquires the student image IS from the image storage unit 116, and sets the target pixel position S of interest when predicting the teacher image IT from the student image IS (S1006).

Then, the class tap extraction unit 940 extracts a plurality of pixels corresponding to the pixel position in the vicinity of the target pixel position S as the class tap _xn and outputs it to the class classification unit 942 (S1008).

Further, the class classification unit 942 determines a class corresponding to the class tap _xn and the target pixel position S, and a class code representing the determined class is output to the learning storage unit 162 (S1010).

Furthermore, the prediction tap extracting unit 150, a plurality of pixels located near the pixel of interest position S from the student image IS are extracted as prediction taps d _n, is output to the normal equation generating unit 160 (S1012).

In addition, the teacher pixel extraction unit 152 extracts the true value y _k of the pixel at the target pixel position S from the teacher image IT and outputs it to the normal equation generation unit 160 (S1014).

Then, the normal equation generating unit 160, by using the true value _{y k} of the prediction tap _{d n} and the pixel of interest, is performed summation to the normal equation (S1016). The normal equation generated here is stored in the learning storage unit 162 for each class determined by the class classification unit 942.

  Thereafter, it is determined whether or not the addition to the normal equation has been completed for all the target pixels (S1018). Here, if there remains a target pixel that has not been added to the normal equation, the process returns to S1006, and the target pixel setting unit 920 sets a new target pixel position S. On the other hand, if the addition to the normal equation has been completed for all the target pixels, the process proceeds to S1020.

In S1020, the coefficient calculation unit 170, a normal equation is obtained in each class from learning and storing unit 162, the prediction coefficient _{w n} for each class are calculated (S1020). Calculated here prediction coefficient w _n is stored by the coefficient storage unit 172 for each class.

Then, whether the calculation of the prediction coefficients w _n for all classes been finished it is determined (S1022). Here, if the remaining classes that calculation is not the end of the prediction coefficient w _n is, the process returns to S1020. On the other hand, the calculation of the prediction coefficients w _n for all classes if completed, the learning process is terminated.

  So far, the learning apparatus 900 according to the third embodiment of the present invention has been described. According to the learning apparatus 900 according to the present embodiment, the prediction coefficient for predicting the original image from the captured image obtained through the imaging process by the image sensor is not only the filter characteristic according to the pixel position, but also the target pixel. Neighboring pixel value patterns are also taken into account and learned adaptively. As a result, the image processing apparatus described in the next section can achieve higher image quality in consideration of the characteristics of the image sensor and at the same time, for example, noise can be removed, for example.

[Image processing device]
Next, an image processing apparatus that performs a prediction process for predicting an original image before imaging from a captured image using the prediction coefficient generated by the learning apparatus 900 described above will be described. FIG. 23 is a block diagram showing a logical configuration of an image processing apparatus 1100 according to the third embodiment of the present invention.

  Referring to FIG. 23, the image processing apparatus 1100 includes a target pixel setting unit 1120, a class tap extraction unit 1140, a class classification unit 1142, a prediction tap extraction unit 350, a coefficient storage unit 372, a prediction coefficient acquisition unit 374, and a prediction calculation unit. 380.

  Here, mainly the pixel-of-interest setting unit 1120, the class tap extraction unit 1140, and the class classification unit 1142 will be described.

  The pixel-of-interest setting unit 1120 sequentially sets arbitrary pixels to be predicted as pixels of interest in the predicted image I2 corresponding to the original image before being imaged by the image sensor. Then, the target pixel setting unit 320 outputs the pixel position S of the set target pixel in the predicted image I2 to the class classification unit 342. In addition, the target pixel setting unit 320 outputs the captured image I1 and the target pixel position S to the class tap extraction unit 1140 and the prediction tap extraction unit 350.

The class tap extraction unit 1140 extracts a plurality of pixels located in the vicinity of the target pixel position S from the captured image I1 as class taps _xn used for class classification of the target pixel, and outputs them to the class classification unit 1142. For example, the class tap extraction unit 1140 extracts nine pixels near the target pixel position S described with reference to FIG. 21 as class taps _xn .

The class classification unit 1142 determines a class according to the target pixel position S and the class tap _xn input from the class tap extraction unit 1140, and outputs the class code C to the prediction coefficient acquisition unit 374. For example, similar to the class classification unit 942 of the learning apparatus 900, the class classification unit 1142 generates an ADRC code from each pixel value included in the class tap _xn by ADRC processing, and the class corresponding to the ADRC code and the target pixel position S The code C is determined.

  With such a configuration of the image processing apparatus 1100, a predicted image corresponding to the original image is considered from the captured image I1 captured using the image sensor in consideration of not only the pixel position but also the pixel value pattern in the vicinity of the target pixel. I2 is generated.

[Description of process flow: Prediction process]
Next, an example of the flow of prediction processing by the image processing apparatus 1100 according to the present embodiment will be described using the flowchart of FIG.

  Referring to FIG. 24, first, a captured image I1 is supplied to the image processing apparatus 1100 (S1202). The supplied captured image I1 is input to the target pixel setting unit 1120.

  Next, the target pixel setting unit 1120 sets the pixel position S of the target pixel to be predicted in the predicted image I2 (S1204). The pixel position S of the target pixel set here is output to the class classification unit 1142, and the target pixel position S and the captured image I1 are output to the class tap extraction unit 1140 and the prediction tap extraction unit 350.

Thereafter, the class tap extraction unit 1140 extracts a plurality of pixels near the target pixel position S as the class tap _xn from the captured image I1 (S1206).

Then, the class classification unit 1142 outputs the class code C representing the class determined according to the target pixel position S and the pattern of each pixel value included in the class tap _xn to the prediction coefficient acquisition unit 374 ( S1208).

Moreover, the prediction coefficient acquisition unit 374, the prediction coefficient _{w n} stored in the coefficient storage unit 372 in association with the class code C is obtained and output to the prediction computation unit 380 (S1210).

Furthermore, the prediction tap extracting unit 350, a plurality of pixels are extracted as prediction taps _{d n} located in the vicinity of the target pixel position S in the captured image I1, is output to the prediction computation unit 380 (S1212).

Then, the prediction computation unit 380, linear combination of the prediction taps _{d n} and prediction coefficient _{w n} is performed according to equation (2), the predicted value of the pixel of interest is calculated (S1214).

  Thereafter, it is determined whether or not the calculation of predicted values for all target pixels has been completed (S1216). If there is a pixel for which the calculation of the predicted value has not been completed, the process returns to S1204. On the other hand, if the calculation of the predicted value has been completed for all the pixels of the predicted image I2, the prediction process ends.

  Up to this point, the image processing apparatus 1100 according to the third embodiment of the present invention has been described. According to the image processing apparatus 1100 according to the present embodiment, imaging by an image sensor is performed using a prediction coefficient based on learning that takes into consideration not only filter characteristics according to pixel positions but also pixel value patterns in the vicinity of the target pixel. An original image is predicted from a captured image obtained through the process. Thereby, when the original image is predicted from the captured image, for example, noise can be removed, so that more effective image quality can be realized.

<5. Summary>
Up to this point, a learning apparatus that learns a prediction coefficient and an image processing apparatus that predicts a high-quality image for each of the three embodiments relating to high image quality in consideration of the characteristics of the image sensor, with reference to FIGS. Separately explained in detail.

  The learning device according to each embodiment may be, for example, an independent learning module connected to the image sensor, the image sensor itself, or a camera (or camera module) equipped with the image sensor. Further, the image processing apparatus according to each embodiment may be, for example, an independent image processing module connected to the image sensor, the image sensor itself, or a camera (or camera module) equipped with the image sensor.

  It does not matter whether a series of processing according to each embodiment is realized by hardware or software. When a series of processes or a part thereof is executed by software, a program constituting the software is executed by using a computer incorporated in dedicated hardware, for example, a general-purpose computer shown in FIG. The

  In FIG. 25, a CPU (Central Processing Unit) 12 controls the overall operation of the general-purpose computer. A ROM (Read Only Memory) 14 stores a program or data describing a part or all of a series of processes. A RAM (Random Access Memory) 16 temporarily stores programs and data used by the CPU 12 during execution of processing.

  The CPU 12, ROM 14, and RAM 16 are connected to each other via the bus 20. An input / output interface 22 is further connected to the bus 20.

  The input / output interface 22 is an interface for connecting the CPU 12, ROM 14, and RAM 16 to the input device 30, output device 32, storage device 34, communication device 36, and drive 40.

  The input device 30 receives an instruction and information input from a user via an input device such as a button, a switch, or a lever. The output device 32 outputs information to the user via a display device such as a liquid crystal display or OLED (Organic Light Emitting Diode), or an audio output device such as a speaker.

  The storage device 34 is configured by, for example, a hard disk drive or a flash memory, and stores programs, program data, image data, and the like. The communication device 36 performs communication processing via a communication port such as a USB (Universal Serial Bus). For example, a removable medium 42 is attached to the drive 40.

  When the series of processes according to the first to third embodiments is executed by software, for example, a program stored in the ROM 14 or the storage device 34 shown in FIG. 25 is read into the RAM 16 at the time of execution and executed by the CPU 12. Is done.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to the example which concerns. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  For example, the learning process or the prediction process according to the first to third embodiments does not necessarily have to be executed in the order described in the flowchart. Each processing step may include processing executed in parallel or individually independently.

It is a schematic diagram which shows the external appearance of an image sensor. It is a characteristic view which shows an example of the condensing characteristic in the center part of an image sensor. It is a characteristic view which shows an example of the condensing characteristic in the peripheral part of an image sensor. It is a characteristic view which shows the other example of the condensing characteristic in the center part of an image sensor. It is a characteristic view which shows the other example of the condensing characteristic in the peripheral part of an image sensor. It is explanatory drawing which shows the outline | summary of the image quality improvement process which considered the characteristic of the image sensor. It is a block diagram which shows the structure of the learning apparatus which concerns on 1st Embodiment. It is explanatory drawing which shows an example of the class classification | category according to a pixel position. It is explanatory drawing which shows an example of the prediction tap extracted from the vicinity of an attention pixel. It is a flowchart which shows an example of the learning process which concerns on 1st Embodiment. It is explanatory drawing which shows an example of the representative pixel position for every class. It is explanatory drawing which shows an example of the interpolation of the prediction coefficient value using a representative pixel position. 1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment. It is a flowchart which shows an example of the prediction process which concerns on 1st Embodiment. It is a block diagram which shows the structure of the learning apparatus which concerns on 2nd Embodiment. It is explanatory drawing which shows an example of the class classification | category according to a pixel position and a camera parameter. It is a flowchart which shows an example of the learning process which concerns on 2nd Embodiment. It is a block diagram which shows the structure of the image processing apparatus which concerns on 2nd Embodiment. It is a flowchart which shows an example of the prediction process which concerns on 2nd Embodiment. It is a block diagram which shows the structure of the learning apparatus which concerns on 3rd Embodiment. It is explanatory drawing which shows an example of the class tap extracted from the vicinity of an attention pixel. It is a flowchart which shows an example of the learning process which concerns on 3rd Embodiment. It is a block diagram which shows the structure of the image processing apparatus which concerns on 3rd Embodiment. It is a flowchart which shows an example of the prediction process which concerns on 3rd Embodiment. And FIG. 18 is a block diagram illustrating a configuration example of a general-purpose computer.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image sensor 12 Light-receiving surface 100,500,900 Learning apparatus 110,510 Student image generation part 112,512 Characteristic memory | storage part 514 Camera parameter acquisition part 116,516 Image memory | storage part 120,520,920 Target pixel setting part 940 Class tap extraction Unit 142, 542, 942 class classification unit 150 prediction tap extraction unit 152 teacher pixel extraction unit 160 normal equation generation unit 162 learning storage unit 170 coefficient calculation unit 172 coefficient storage unit 300, 700, 1100 image processing device 714 camera parameter acquisition unit 320 1120 attention pixel setting unit 1140 class tap extraction unit 342, 742, 1142 class classification unit 350 prediction tap extraction unit 372 coefficient storage unit 374 prediction coefficient acquisition unit 380 prediction calculation unit

Claims (18)

A student image that generates a student image having the image quality of the first image by applying a different filter to the previously acquired teacher image having a higher image quality of the second image than the first image according to the pixel position With a generator;
A prediction tap extraction unit that extracts, as prediction taps, a plurality of pixels included in the student image and corresponding to pixel positions in the vicinity of a target pixel of interest when predicting the teacher image;
A class classification unit that determines a class of the target pixel according to a pixel position of the target pixel;
A coefficient calculation unit that calculates a prediction coefficient used for predicting the pixel value of the target pixel in the teacher image from the pixel value of the prediction tap for each class determined by the class classification unit;
A learning apparatus comprising:   The learning apparatus according to claim 1, wherein the class classification unit determines the class according to a distance from a center position of the teacher image to a pixel position of the target pixel. The first image is a captured image captured by an image sensor,
3. The student image generation unit generates the student image by applying a filter reflecting a specific light collection characteristic of the image sensor according to a pixel position to the teacher image. The learning apparatus in any one of.   The learning apparatus according to claim 1, wherein the student image generation unit generates the student image by continuously changing characteristics of the filter according to a pixel position.   The coefficient calculation unit further calculates a prediction coefficient corresponding to a pixel position between representative pixel positions representing each class by linear interpolation using the prediction coefficient calculated for each class. The learning device according to any one of? The first image is a captured image captured using a camera including an image sensor,
The learning apparatus according to claim 1, wherein the class classification unit further determines the class according to a camera parameter representing a feature or state of the camera when the teacher image is captured by the camera. The learning device further includes a class tap extraction unit that extracts a plurality of pixels included in the student image and corresponding to a pixel position in the vicinity of the target pixel as a class tap,
The class classification unit determines the class according to a pixel value pattern of the class tap and a pixel position of the target pixel;
The learning device according to claim 1. A prediction tap for extracting, as a prediction tap, a plurality of pixels included in the first image and corresponding to pixel positions near the target pixel of interest in the second image of higher quality than the first image. An extractor;
A class classification unit that determines a class of the target pixel according to a pixel position of the target pixel in the second image;
A prediction coefficient used for predicting the pixel value of the target pixel from the pixel value of the prediction tap, the teacher image acquired in advance having the image quality of the second image, and the teacher image according to the pixel position A storage unit storing a prediction coefficient calculated for each class using a student image having the image quality of the first image generated by applying different filters;
A prediction calculation unit that calculates a prediction value corresponding to the pixel value of the target pixel by linearly combining the pixel value of the prediction tap and the prediction coefficient acquired from the storage unit;
An image processing apparatus comprising:   The image processing apparatus according to claim 8, wherein the class classification unit determines the class according to a distance from a center position of the second image to a pixel position of the target pixel. The first image is a captured image captured by an image sensor,
10. The student image according to claim 8, wherein the student image is an image generated by applying, to the teacher image, a filter that reflects a unique light collection characteristic of the image sensor according to a pixel position. The image processing apparatus described.   The image processing apparatus according to claim 8, wherein the student image is an image generated by applying a filter whose characteristics continuously change according to a pixel position to the teacher image.   The prediction calculation unit further calculates a prediction coefficient corresponding to a pixel position between representative pixel positions representing each class by linear interpolation, using the prediction coefficient for each class acquired from the storage unit. The image processing apparatus according to claim 8. The first image is a captured image captured using a camera including an image sensor,
The image processing apparatus according to claim 8, wherein the class classification unit further determines the class in accordance with a camera parameter representing a feature or state of the camera when the first image is captured by the camera. The image processing device further includes a class tap extraction unit that extracts a plurality of pixels included in the first image and corresponding to a pixel position in the vicinity of the target pixel as a class tap,
The class classification unit determines the class according to a pixel value pattern of the class tap and a pixel position of the target pixel;
The image processing apparatus according to claim 8. A student image that generates a student image having the image quality of the first image by applying a different filter to the previously acquired teacher image having a higher image quality of the second image than the first image according to the pixel position Generation step;
A prediction tap extracting step of extracting, as prediction taps, a plurality of pixels included in the student image and corresponding to pixel positions in the vicinity of a target pixel of interest when predicting the teacher image;
A class classification step for determining a class of the target pixel according to a pixel position of the target pixel;
A coefficient calculation step for calculating, for each class determined in the class classification step, a prediction coefficient used to predict the pixel value of the target pixel in the teacher image from the pixel value of the prediction tap;
Learning methods including. The computer that controls the learning device:
A student image that generates a student image having the image quality of the first image by applying a different filter to the previously acquired teacher image having a higher image quality of the second image than the first image according to the pixel position With a generator;
A prediction tap extraction unit that extracts, as prediction taps, a plurality of pixels included in the student image and corresponding to pixel positions in the vicinity of a target pixel of interest when predicting the teacher image;
A class classification unit that determines a class of the target pixel according to a pixel position of the target pixel;
A coefficient calculation unit that calculates a prediction coefficient used for predicting the pixel value of the target pixel in the teacher image from the pixel value of the prediction tap for each class determined by the class classification unit;
Program to function as A prediction tap for extracting, as a prediction tap, a plurality of pixels included in the first image and corresponding to pixel positions near the target pixel of interest in the second image of higher quality than the first image. An extraction step;
A class classification step of determining a class of the target pixel according to a pixel position of the target pixel in the second image;
A prediction coefficient used for predicting the pixel value of the target pixel from the pixel value of the prediction tap, the teacher image acquired in advance having the image quality of the second image, and the teacher image according to the pixel position Determined by the class classification step from a storage unit storing a prediction coefficient calculated for each class using a student image having the image quality of the first image generated by applying different filters. A prediction coefficient acquisition step of acquiring the prediction coefficient according to the class;
A prediction calculation step of calculating a prediction value corresponding to the pixel value of the target pixel by linearly combining the pixel value of the prediction tap and the prediction coefficient acquired in the prediction coefficient acquisition step;
An image processing method including: A computer that controls the image processing device:
A prediction tap for extracting, as a prediction tap, a plurality of pixels included in the first image and corresponding to pixel positions near the target pixel of interest in the second image of higher quality than the first image. An extractor;
A class classification unit that determines a class of the target pixel according to a pixel position of the target pixel in the second image;
A prediction coefficient used for predicting the pixel value of the target pixel from the pixel value of the prediction tap, the teacher image acquired in advance having the image quality of the second image, and the teacher image according to the pixel position A storage unit storing a prediction coefficient calculated for each class using a student image having the image quality of the first image generated by applying different filters;
A prediction calculation unit that calculates a prediction value corresponding to the pixel value of the target pixel by linearly combining the pixel value of the prediction tap and the prediction coefficient acquired from the storage unit;
Program to function as

Images