JP2018195918A - Signal processing apparatus, signal processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a signal processing device, a signal processing method and a program capable of reducing false resolution and generating an image signal having a high resolution. A signal processing device interpolates a first color signal with respect to a pixel of interest corresponding to a second color signal of an image signal by using a first color signal in pixels surrounding the pixel of interest. The first color signal is interpolated by using the second color signal and the color difference signal in the pixel of interest with respect to the first interpolation means (301) and the pixel of interest corresponding to the second color signal of the image signal. A color signal interpolated by the first interpolation means and a color signal interpolated by the second interpolation means based on the inclinations of the second interpolation means (302) and the color difference signals in the plurality of directions of the image signal. It has a first synthesis means (304) for synthesizing. [Selection diagram] Fig. 3

Description

  The present invention relates to a signal processing device, a signal processing method, and a program.

  In Patent Document 1, the signal processing apparatus discriminates the correlation in the vertical direction or the horizontal direction when interpolating the G signal which is the main component of the luminance signal, and in the vertical direction when there is a correlation in the vertical direction. LPF processing is performed, and when there is a correlation in the horizontal direction, LPF processing is performed in the horizontal direction. In addition, the signal processing apparatus performs two-dimensional LPF processing when there is no correlation in the vertical direction or the horizontal direction, so that the vertical and horizontal edges are not blurred. However, in Patent Document 1, when there is no vertical or horizontal correlation, two-dimensional LPF processing is performed. For example, an edge in a diagonal direction is blurred compared to an edge in a vertical direction and a horizontal direction.

  Therefore, in Non-Patent Document 1, as a method for calculating the G signal at the position of the R or B pixel, the color difference signal between the pixel of interest and the same color as the pixel of interest in the up, down, left, and right directions including the pixel of interest. Are calculated respectively. In Non-Patent Document 1, a G signal is calculated by adding a color difference signal obtained by synthesizing a color difference signal in each direction to a target pixel. At this time, in Non-Patent Document 1, an image with high resolution is generated regardless of the direction by synthesizing the color difference signals in each direction according to the inclination of the color difference signal in the corresponding direction.

Japanese Patent No. 3862506

“Gradient based threshold free color filter array interpolation”, I. Pekkucuksen, Y. Altunbasak, ICIP 2010

  However, in Non-Patent Document 1, in the bands near the Nyquist frequency in the horizontal direction and the vertical direction, the gradient of the color difference signal in each direction becomes a small value, so it is possible to accurately determine which direction is the horizontal direction or the vertical direction. It may not be possible to determine and false resolution may occur.

  An object of the present invention is to provide a signal processing device, a signal processing method, and a program capable of generating an image signal with reduced false resolution and high resolution.

  The signal processing apparatus according to the present invention interpolates the first color signal by using the first color signal in pixels around the target pixel with respect to the target pixel corresponding to the second color signal of the image signal. The first color signal is obtained by using the second color signal and the color difference signal in the target pixel with respect to the target pixel corresponding to the second color signal of the image signal. And a color signal interpolated by the first interpolation means and a color interpolated by the second interpolation means based on respective inclinations of the plurality of color difference signals of the image signal. First combining means for combining signals.

  It is possible to reduce false resolution and generate an image signal with high resolution.

It is a figure which shows 1 unit of a primary color Bayer arrangement | sequence. It is a block diagram which shows the structural example of a signal processing apparatus. It is a block diagram which shows the structural example of a G interpolation circuit. It is a block diagram which shows the structural example of a 1st G interpolation circuit. It is a flowchart which shows an example of the flow of a process of the 1st G interpolation circuit. It is a block diagram which shows the structural example of the 2nd G interpolation circuit. It is a flowchart which shows an example of the flow of a process of the 2nd G interpolation circuit. It is a block diagram which shows the structural example of a synthetic | combination coefficient calculation circuit. It is a figure which shows an example of the input-output characteristic of a coefficient calculation circuit.

  FIG. 1 is a diagram showing one unit of a Bayer array of an image sensor according to an embodiment of the present invention. The image sensor has the Bayer array color filter of FIG. 1, and outputs an R (red) signal, a G1 (green) signal, a G2 (green) signal, and a B (blue) signal by photoelectric conversion. The imaging element has a plurality of pixels arranged in a matrix, and each pixel is provided with one color filter of three colors of red (R), green (G), and blue (B). The image sensor is, for example, a CMOS image sensor, and is used in an imaging device. The imaging device can be applied to a smartphone, a tablet, an industrial camera, a medical camera, and the like in addition to a digital camera and a video camera.

  Since an image sensor having a Bayer array can obtain only one color signal of R, G, and B in each pixel, when obtaining all the color signals of R, G, and B in each pixel, Interpolation processing needs to be performed by the signal processing device 200 of FIG. For example, the signal processing apparatus 200 can obtain an interpolation value by setting all the color signal levels other than the color signal to be interpolated to 0 and performing a two-dimensional low-pass filter (LPF) process on each pixel. However, by simply performing a two-dimensional LPF process, the frequency characteristics of the interpolated image loses high frequency components, and the image becomes blurred. Hereinafter, the signal processing apparatus 200 that can reduce blurring of an image will be described.

  FIG. 2 is a block diagram illustrating a configuration example of the signal processing device 200 according to the embodiment of the present invention. Hereinafter, a signal processing method of the signal processing device 200 will be described. The signal processing device 200 includes a WB circuit (white balance circuit) 201, a G interpolation circuit 202, an R interpolation circuit 203, a B interpolation circuit 204, an APC circuit 205, a luminance signal generation circuit 206, an adder 207, Have The image sensor outputs an image signal obtained by converting the R signal, the G1 signal, the G2 signal, and the B signal from analog to digital to the signal processing device 200. The WB circuit 201 receives a digital image signal from the image sensor, corrects the white balance of the image signal, and outputs the corrected image signal. The image signal includes an R signal, a G1 signal, a G2 signal, and a B signal. The R pixel is an R signal pixel. The B pixel is a B signal pixel. The G pixel is a pixel of the G1 signal or the G2 signal.

  The G interpolation circuit 202 receives the image signal output from the WB circuit 201 and calculates the G signal at the position of the R pixel and the B pixel by interpolation at each pixel position of the image sensor. Details of the processing of the G interpolation circuit 202 will be described later. The R interpolation circuit 203 receives the output signal of the WB circuit 201 and calculates the R signal at the position of the G pixel and the B pixel of the image sensor by interpolation. The B interpolation circuit 204 receives the output signal of the WB circuit 201 and calculates the B signal at the positions of the R pixel and the G pixel of the image sensor by interpolation. For example, the R interpolation circuit 203 and the B interpolation circuit 204 perform interpolation by two-dimensional LPF processing.

The APC circuit 205 generates an aperture correction signal for the G signal output from the G interpolation circuit 202. The adder 207 adds the aperture correction signal output from the APC circuit 204 and the G signal output from the G interpolation circuit 202, and outputs the addition result. The luminance signal generation circuit 206 generates the luminance signal Y according to Equation (1) based on the G signal, R signal, and B signal output from the adder 207, R interpolation circuit 203, and B interpolation circuit 204, respectively. .
Y = 0.3R + 0.59G + 0.11B (1)

  FIG. 3 is a block diagram illustrating a configuration example of the G interpolation circuit 202. The G interpolation circuit 202 includes a first G interpolation circuit 301, a second G interpolation circuit 302, a synthesis coefficient calculation circuit 303, and a synthesis circuit 304. The first G interpolation circuit 301 interpolates the G signal in the target pixel, which is the R pixel or the B pixel of the image signal, using the G signal of the pixels around the target pixel, and outputs the first G signal. The second G interpolation circuit 302 interpolates the G signal in the pixel of interest which is the R pixel or B pixel of the image signal using the R or B signal of the pixel of interest and the color difference signal, and outputs the second G signal. . The synthesis circuit 304 interpolates the first G signal interpolated by the first G interpolation circuit 301 and the second G interpolation circuit 302 based on the respective gradients of the color difference signals in the vertical direction and horizontal direction of the image signal. The synthesized second G signal is synthesized. Details thereof will be described later.

  FIG. 4 is a block diagram illustrating a configuration example of the first G interpolation circuit 301. The first G interpolation circuit 301 includes a 0 insertion circuit 401, an HV interpolation circuit 402, an H interpolation circuit 403, a V interpolation circuit 404, subtracters 405 and 406, a selector 407, a vertical / horizontal discrimination circuit 408, An absolute value circuit 409, a multiplier 410, and an adder 411 are included.

  FIG. 5 is a flowchart showing a processing flow of the first G interpolation circuit 301. In step S500, the 0 insertion circuit 401 sets the signal levels of the R pixel and the B pixel in the input image signal to 0. Next, in step S501, the HV interpolation circuit 402 calculates a G signal by performing interpolation processing in the horizontal direction and the vertical direction on the image signal output from the 0 insertion circuit 401. At this time, the HV interpolation circuit 402 uses, for example, a filter having a coefficient of (1, 2, 1) / 2 for the interpolation processing in the horizontal direction and the vertical direction. The HV interpolation circuit 402 can generate a G signal in which false resolution of the Nyquist frequency in the horizontal direction and the vertical direction is suppressed by performing interpolation processing using the above-described filter.

Next, in step S502, the vertical / horizontal discrimination circuit 408 determines whether the pixel signal to be interpolated has a large correlation in the vertical direction or a large correlation in the horizontal direction, that is, whether the image has a vertical stripe pattern or a horizontal stripe pattern. Is determined. Specifically, the vertical / horizontal discrimination circuit 408 determines whether the vertical stripe or horizontal stripe is based on the difference between the signal level difference of the pixel in the horizontal direction centered on the pixel of interest and the signal level difference of the pixel in the vertical direction centered on the pixel of interest. A vertical / horizontal discrimination signal for discriminating between these is generated. For example, when the Y coordinate and the X coordinate of the pixel of interest are (i, j), the vertical / horizontal discrimination circuit 408 calculates the vertical / horizontal discrimination signal diffHV _{i, j} when the pixel of interest is an R pixel according to equation (2).
diffHV _{i, j} = diffH _{i, j} −diffV _{i, j}
diffH _{i, j} = | G _{i, j-1} −G _{i, j + 1} | + | 2 × R _{i, j} −R _{i, j−2} −R _{i, j + 2} |
diffV _{i, j} = | G _{i−1, j} −G _{i + 1, j} | + | 2 × R _{i, j} −R _{i−2, j} −R _{i + 2, j} |
... (2)

  Here, the vertical / horizontal discrimination signal diffHV is obtained by subtracting the level difference diffV of the pixels in the vertical direction from the level difference diffH of the pixels in the horizontal direction. That is, when the vertical / horizontal discrimination signal diffHV is negative, the image has a horizontal stripe pattern, and when the vertical / horizontal discrimination signal diffHV is positive, the image has a vertical stripe pattern. Further, the vertical / horizontal discrimination circuit 408 clips the vertical / horizontal discrimination signal diffHV with a certain threshold value, and further normalizes it, thereby setting the range of the vertical / horizontal discrimination signal diffHV to a range from −1.0 to 1.0. . The formula for calculating the vertical / horizontal discrimination signal diffHV is not limited to the above formula (1), and may be calculated from, for example, a difference signal between adjacent pixels.

  The vertical / horizontal discrimination circuit 408 can discriminate vertical stripes and horizontal stripes up to the Nyquist frequency in the horizontal direction and vertical direction by generating the vertical / horizontal discrimination signal diffHV based on the difference signal between adjacent pixels as described above. However, when there is a level difference between R, G, and B pixels as in a chromatic color subject, the level difference due to color is erroneously discriminated as a vertical stripe or a horizontal stripe, so it is necessary to limit it to a monochrome subject.

  The vertical / horizontal discrimination circuit 408 proceeds to step S503 if the vertical / horizontal discrimination signal diffHV is negative, that is, if the image is a horizontal stripe pattern, and if the vertical / horizontal discrimination signal diffHV is positive, that is, if the image is a vertical stripe pattern. Advances the process to step S504.

  In step S503, first, the H interpolation circuit 403 performs an interpolation process in the horizontal direction on the image signal output from the 0 insertion circuit 401. At this time, the H interpolation circuit 403 uses a filter with a coefficient of (1, 4, 6, 4, 1) / 8, for example, for the horizontal interpolation process. Next, the subtracter 405 subtracts the output signal of the HV interpolation circuit 402 from the output signal of the H interpolation circuit 403. The selector 407 selects the output signal from the subtracter 405. Thereafter, the first G interpolation circuit 301 advances the process to step S505.

  In step S504, first, the V interpolation circuit 404 performs interpolation processing in the vertical direction on the image signal output from the 0 insertion circuit 401. At this time, the V interpolation circuit 404 uses, for example, a filter with a coefficient of (1, 4, 6, 4, 1) / 8 for the interpolation process in the vertical direction. Next, the subtracter 406 subtracts the output signal of the HV interpolation circuit 402 from the output signal of the V interpolation circuit 404. The selector 407 selects the output signal of the subtracter 406. Thereafter, the first G interpolation circuit 301 advances the process to step S505.

  In step S505, first, the absolute value conversion circuit 409 outputs a signal obtained by converting the vertical / horizontal discrimination signal diffHV output from the vertical / horizontal discrimination circuit 408 into an absolute value. Next, the multiplier 410 multiplies the signal selected by the selector 407 and the signal output from the absolute value circuit 409.

  Next, in step S506, the adder 411 adds the output signal of the HV interpolation circuit 402 and the output signal of the multiplier 410, and outputs a first G signal.

  As a result, when the vertical / horizontal discrimination circuit 408 is difficult to discriminate vertical stripes or horizontal stripes in the frequency band near the Nyquist frequency in the horizontal direction or vertical direction, that is, when the vertical / horizontal discrimination signal diffHV is small, the output of the multiplier 410 The value becomes smaller. As a result, since the ratio of the output signal of the HV interpolation circuit 402 to the output signal of the adder 411 is increased, the first G interpolation circuit 301 is suppressed from false resolution of the Nyquist frequency in the horizontal direction and the vertical direction. The G signal can be output.

  The vertical / horizontal discrimination circuit 408 generates the vertical / horizontal discrimination signal diffHV based on the difference signal between adjacent pixels. As a result, the first G interpolation circuit 301 can interpolate the frequency band near the Nyquist frequency according to the direction of the vertical stripe or the horizontal stripe when the vertical stripe horizontal stripe can be discriminated up to the Nyquist frequency in the horizontal direction and the vertical direction. It can be performed.

  FIG. 6 is a block diagram illustrating a configuration example of the second G interpolation circuit 302. The second G interpolation circuit 302 includes a G pixel V interpolation circuit 601, a G pixel H interpolation circuit 602, an R and B pixel V interpolation circuit 603, an R and B pixel H interpolation circuit 604, and a V color difference calculation circuit 605. And an H color difference calculation circuit 606. Further, the second G interpolation circuit 302 includes a V color difference inclination calculation circuit 607, an H color difference inclination calculation circuit 608, an N filter circuit 609, an S filter circuit 610, a W filter circuit 611, and an E filter circuit 612. Have Further, the second G interpolation circuit 302 includes an N weight calculation circuit 613, an S weight calculation circuit 614, a W weight calculation circuit 615, an E weight calculation circuit 616, a synthesis circuit 617, and an adder 618. .

FIG. 7 is a flowchart showing the flow of processing of the second G interpolation circuit 302. In step S701, the G pixel V interpolation circuit 601 generates a G signal by performing an interpolation process in the vertical direction on the input image signal. Specifically, the G pixel V interpolation circuit 601 outputs the G pixel signal as it is as the G signal when the target pixel is the G pixel, and the vertical direction when the target pixel is the R pixel or the B pixel. A G signal is generated by performing an interpolation process. For example, when the X coordinate and the Y coordinate of the pixel of interest in the image are (j, i), the G pixel V interpolation circuit 601 determines that the G signal Gv _{i, j} is expressed by Equation (3) when the pixel of interest is an R pixel. Generate.
Gv _{i, j} = (G _{i−1, j} + G _{i + 1, j} ) / 2 + (2 × R _{i, j} −R _{i−2, j} −R _{i + 2, j} ) / 4
... (3)

  Note that the G pixel V interpolation circuit 601 generates a G signal by the same method as described above even when the target pixel is a B pixel.

In step S702, the G pixel H interpolation circuit 602 generates a G signal by performing an interpolation process on the input image signal in the horizontal direction. Specifically, the G pixel H interpolation circuit 602 outputs the G pixel signal as it is as a G signal when the target pixel is a G pixel, and in the horizontal direction when the target pixel is an R pixel or a B pixel. A G signal is generated by performing an interpolation process. For example, when the pixel of interest is an R pixel, the G pixel H interpolation circuit 602 generates a G signal Gh _{i, j} using Expression (4).
Gh _{i, j} = (G _{i, j-1} + G _{i, j + 1} ) / 2 + (2 × R _{i, j} −R _{i, j−2} −R _{i, j + 2} ) / 4
... (4)

  Note that the G pixel H interpolation circuit 602 generates a G signal by the same method as described above even when the target pixel is a B pixel.

In step S703, the R and B pixel V interpolation circuit 603 generates an R signal or a B signal by performing interpolation processing on the input image signal in the vertical direction. Specifically, when the target pixel is an R pixel or a B pixel, the R and B pixel V interpolation circuit 603 outputs the signal of the R pixel or the B pixel as it is as an R signal or a B signal. When the target pixel is a G pixel and the vertical direction of the target pixel is an R pixel, the R and B pixel V interpolation circuit 603 generates an R signal by performing interpolation processing in the vertical direction. When the vertical direction of the pixel of interest is a B pixel, a B signal is generated by performing interpolation processing in the vertical direction. For example, when the target pixel is a G pixel and the vertical direction of the target pixel is an R pixel, the R and B pixel V interpolation circuit 603 generates an R signal Rv _{i, j} by Expression (5).
Rv _{i, j} = (R _{i−1, j} + R _{i + 1, j} ) / 2 + (2 × G _{i, j} −G _{i−2, j} −G _{i + 2, j} ) / 4
... (5)

  The R and B pixel V interpolation circuit 603 generates a B signal by the same method as described above when the vertical direction of the target pixel is a B pixel.

In step S704, the R and B pixel H interpolation circuit 604 generates an R signal or a B signal by performing an interpolation process on the input image signal in the horizontal direction. Specifically, when the target pixel is an R pixel or a B pixel, the R and B pixel H interpolation circuit 604 outputs an R pixel or B pixel signal as it is as an R signal or a B signal. When the target pixel is a G pixel and the horizontal direction of the target pixel is an R pixel, the R and B pixel H interpolation circuit 604 generates an R signal by performing an interpolation process in the horizontal direction. When the horizontal direction of the target pixel is a B pixel, a B signal is generated by performing interpolation processing in the horizontal direction. For example, when the target pixel is a G pixel and the horizontal direction of the target pixel is an R pixel, the R and B pixel H interpolation circuit 604 generates an R signal Rh _{i, j} using Expression (6).
Rh _{i, j} = (R _{i, j-1} + R _{i, j + 1} ) / 2 + (2 × G _{i, j} −G _{i, j−2} −G _{i, j + 2} ) / 4
... (6)

  Note that the R and B pixel H interpolation circuit 604 generates a B signal by the same method as described above when the vertical direction of the target pixel is a B pixel. Moreover, although the method of Formula (3)-(6) was taken as the interpolation method of step S701-S704, it is not limited to this, For example, you may linearly interpolate the same color pixel to each direction.

Next, in step S705, the V color difference calculation circuit 605 subtracts the R signal or B signal output from the R and B pixel V interpolation circuit 603 from the G signal output from the G pixel V interpolation circuit 601 to obtain the vertical direction. A color difference signal is generated. For example, in the pixel of interest (j, i), the signal output from the G pixel V interpolation circuit 601 is the G signal Gv _{i, j} , and the signal output from the R, B pixel V interpolation circuit 603 is the R signal Rv _{i, j.} It is. In that case, the V color difference calculation circuit 605 generates the color difference signal Diff_v in the vertical direction by Expression (7).
Diff_v _{i, j} = Gv _{i, j} −Rv _{i, j} (7)

  Note that the V color difference calculation circuit 605 generates a color difference signal in the same manner as described above even when the signal output from the R and B pixel V interpolation circuit 603 is a B signal.

Next, in step S706, the H color difference calculation circuit 606 subtracts the R signal or the B signal output from the R and B pixel H interpolation circuit 604 from the G signal output from the G pixel H interpolation circuit 602, and performs horizontal processing. A color difference signal is generated. For example, in the pixel of interest (j, i), the signal output from the G pixel H interpolation circuit 602 is the G signal Gh _{i, j} , and the signal output from the R, B pixel H interpolation circuit 604 is the R signal Rv _{i, j.} It is. In this case, the H color difference calculation circuit 606 generates a horizontal color difference signal Diff_h by Expression (8).
_{Diff_h i, j = Gh i,} j -Rh i, j ··· (8)

  The H color difference calculation circuit 606 generates a color difference signal by the same method as described above even when the signal output from the R and B pixel H interpolation circuit 604 is a B signal.

In step S707, the V color difference inclination calculation circuit 607 calculates the inclination of the color difference in the vertical direction based on the color difference signal output from the V color difference calculation circuit 605. Specifically, in the pixel of interest (j, i), the V color difference inclination calculation circuit 607 generates a vertical color difference inclination signal Grad_v based on the color difference signal Diff_v output from the V color difference calculation circuit 605 using the equation (9). ).
Grad_v _{i, j} = | Diff_v _{i−1, j} −Diff_v _{i + 1, j} | (9)

  Note that the equation for calculating the inclination signal Grad_v is not limited to the above equation (9). For example, the V color difference inclination calculation circuit 607 may calculate the inclination signal Grad_v based on the respective differences between the target pixel and the upper and lower adjacent pixels. The V color difference inclination calculation circuit 607 outputs the calculated vertical color difference inclination signal Grad_v to the N weight calculation circuit 613, the S weight calculation circuit 614, and the synthesis coefficient calculation circuit 303, respectively.

Next, in step S708, the H color difference inclination calculation circuit 608 calculates the color difference inclination in the horizontal direction based on the color difference signal output from the H color difference calculation circuit 606. Specifically, in the pixel of interest (j, i), the H color difference inclination calculation circuit 608 generates a horizontal color difference inclination signal Grad_h based on the color difference signal Diff_h output from the H color difference calculation circuit 606 using the equation (10). ).
Grad_h _{i, j} = | Diff_h _{i, j−1} −Diff_h _{i, j + 1} | (10)

  Note that the equation for calculating the inclination signal Grad_h is not limited to the above equation (10). For example, the H color difference inclination calculation circuit 608 may calculate the inclination signal Grad_h based on the respective differences between the target pixel and the left and right adjacent pixels. The H color difference inclination calculation circuit 608 outputs the calculated horizontal color difference inclination signal Grad_h to the W weight calculation circuit 615, the E weight calculation circuit 616, and the synthesis coefficient calculation circuit 303, respectively.

  In step S709, the N filter circuit 609 performs an upward filter process on the color difference signal output from the V color difference calculation circuit 605. Specifically, the N filter circuit 609 uses the color difference signal Diff_v output from the V color difference calculation circuit 605 to calculate the result Diff_n of the upward filter processing on the pixel of interest (j, i) using Expression (11). To do.

  In step S710, the S filter circuit 610 performs a downward filter process on the color difference signal output from the V color difference calculation circuit 605. Specifically, the S filter circuit 610 uses the color difference signal Diff_v output from the V color difference calculation circuit 605 to calculate the result Diff_s of the downward filter processing on the pixel of interest (j, i) using Expression (12). To do.

  In step S711, the W filter circuit 611 performs leftward filter processing on the color difference signal output from the H color difference calculation circuit 606. Specifically, the W filter circuit 611 uses the color difference signal Diff_h output from the H color difference calculation circuit 606 to calculate the left-direction filter processing result Diff_w of the pixel of interest (j, i) using Expression (13). To do.

  In step S712, the E filter circuit 612 performs rightward filter processing on the color difference signal output from the H color difference calculation circuit 606. Specifically, the E filter circuit 612 uses the color difference signal Diff_h output from the H color difference calculation circuit 606 to calculate the right-direction filter processing result DIFF_e of the pixel of interest (j, i) using Expression (14). To do.

  In step S 713, the N weight calculation circuit 613 uses the vertical color difference inclination signal Grad_v output from the V color difference inclination calculation circuit 607 to calculate the upward weight Wn using Expression (15).

  In step S714, the S weight calculation circuit 614 uses the vertical color difference inclination signal Grad_v output from the V color difference inclination calculation circuit 607 to calculate the downward weight Ws using Expression (16).

  In step S715, the W weight calculation circuit 615 uses the horizontal color difference inclination signal Grad_h output from the H color difference inclination calculation circuit 608 to calculate the left weight Ww by Expression (17).

  In step S716, the E weight calculation circuit 616 uses the horizontal color difference gradient signal Grad_h output from the H color difference gradient calculation circuit 608 to calculate the right weight We using Equation (18).

  In step S717, the combining circuit 617 combines the color difference signals Diff_n, Diff_s, Diff_w, and Diff_e based on the weights Wn, Ws, Ww, and We, and calculates the color difference signal Diff_mix by Expression (19).

  In step S718, when the target pixel is an R pixel or a B pixel with respect to the image signal output from the WB circuit 201, the adder 618 outputs the color difference signal Diff_mix synthesized by the synthesis circuit 617 corresponding to the target pixel. Add and output the second G signal. The output signal of the adder circuit 618 is the output signal of the second G interpolation circuit 302. As a result, the second G interpolation circuit 302 can calculate a high-resolution G signal with respect to the pixel position of the R pixel or the B pixel regardless of the direction.

  FIG. 8 is a block diagram illustrating a configuration example of the synthesis coefficient calculation circuit 303 in FIG. The composite coefficient calculation circuit 303 includes a maximum value selection circuit 801 and a coefficient calculation circuit 802. The maximum value selection circuit 801 receives the gradient signal Grad_v in the vertical color difference and the gradient signal Grad_h in the horizontal direction from the second G interpolation circuit 302, and either one of the gradient signals Grad_v and Grad_h for each pixel. The larger one is selected and output as the maximum value max.

  The coefficient calculation circuit 802 receives the maximum value max from the maximum value selection circuit 801, the first G signal output from the first G interpolation circuit 301, and the second value output from the second G interpolation circuit 302. A synthesis coefficient α for synthesizing the G signals is calculated. Specifically, first, the coefficient calculation circuit 802 compares the maximum value max with a predetermined threshold value, and determines whether or not the maximum value max is smaller than the predetermined threshold value. When the maximum value max is smaller than the predetermined threshold, the coefficient calculation circuit 802 calculates the synthesis coefficient α so as to output the G signal output from the first G interpolation circuit 301. When the maximum value max is smaller than a predetermined threshold, it means an area where the horizontal and vertical color difference gradients Grad_v and Grad_h are both small and the direction cannot be determined based on the color difference gradient.

  FIG. 9 is a diagram illustrating an example of input / output characteristics of the coefficient calculation circuit 802, and illustrates an example of a conversion table from the maximum value max to the composite coefficient α. The coefficient calculation circuit 802 generates a synthesis coefficient α for synthesizing the G signals output from the first G interpolation circuit 301 and the second G interpolation circuit 302 based on the maximum value max output from the maximum value selection circuit 801. calculate. In FIG. 9, the horizontal axis represents the maximum value max, and the vertical axis represents the synthesis coefficient α. The coefficient calculation circuit 802 sets the synthesis coefficient α to 1.0 when determining that the maximum value max is smaller than the predetermined threshold Th1 and the direction cannot be determined based on the gradient of the color difference. If the coefficient calculation circuit 802 determines that the maximum value max is greater than the predetermined threshold Th2 and the direction can be determined based on the gradient of the color difference, the coefficient calculation circuit 802 sets the combination coefficient α to 0.0. In addition, when the maximum value max is between the threshold value Th1 and the threshold value Th2, the coefficient calculation circuit 802 outputs a composite coefficient α that linearly ranges between 0.0 and 1.0 according to the maximum value max. To do. The synthesis coefficient α output from the coefficient calculation circuit 802 is output to the synthesis circuit 304 as the output of the synthesis coefficient calculation circuit 303.

Next, the synthesis circuit 304 in FIG. 3 will be described. The synthesis circuit 304 synthesizes the first G signal from the first G interpolation circuit 301 and the second G signal from the second G interpolation circuit 302 based on the synthesis coefficient α from the synthesis coefficient calculation circuit 303. The final G signal is output. The G signal output from the synthesis circuit 304 is an output signal of the G interpolation circuit 202. The synthesis circuit 304 outputs the final G signal G_sig _{i, j} when the X coordinate and Y coordinate of the pixel of interest in the image are (j, i). Specifically, the synthesis circuit 304 uses the first G signal G1_sig _{i, j} , the second G signal G2_sig _{i, j} , and the synthesis coefficient α _{i, j} in the pixel of interest to generate a final G signal G_sig _{i. , j} is calculated by equation (20).
G_sig _{i, j} = α × G1_sig _{i, j} + (1.0−α) × G2_sig _{i, j}
... (20)

  When the synthesis coefficient α is 0.0, that is, when the direction can be determined based on the gradient of the color difference, the synthesis circuit 304 generates the second G that can generate a high-resolution image regardless of the direction. The signal G2_sig is output as the final G signal G_sig. Further, when the synthesis coefficient α is 1.0, that is, when the direction cannot be determined based on the gradient of the color difference, the pseudo resolution of the Nyquist frequency in the horizontal direction or the vertical direction is suppressed. The first G signal G1_sig is output as the final G signal G_sig. The first G signal G1_sig is a signal in which false resolution of the Nyquist frequency in the horizontal direction or the vertical direction is suppressed in a frequency band such as the Nyquist frequency in the horizontal direction or the vertical direction.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

301 1st G interpolation circuit, 302 2nd G interpolation circuit, 304 composition circuit

Claims (10)

First interpolation means for interpolating the first color signal with respect to the target pixel corresponding to the second color signal of the image signal by using the first color signal in pixels around the target pixel;
Second interpolation means for interpolating the first color signal with respect to the target pixel corresponding to the second color signal of the image signal by using the second color signal and the color difference signal in the target pixel. When,
A first synthesis for synthesizing the color signal interpolated by the first interpolation means and the color signal interpolated by the second interpolation means based on respective inclinations of the color difference signals in a plurality of directions of the image signal. And a signal processing apparatus. The second interpolation means includes
Third interpolation means for interpolating each color signal in a first direction with respect to the image signal;
Fourth interpolation means for interpolating each color signal in a second direction with respect to the image signal;
First color difference calculation means for calculating a color difference signal in the first direction of the image signal interpolated by the third interpolation means;
Second color difference calculation means for calculating a color difference signal in the second direction of the image signal interpolated by the fourth interpolation means;
First inclination calculating means for calculating an inclination of the color difference signal in the first direction calculated by the first color difference calculating means;
Second inclination calculating means for calculating an inclination of the color difference signal in the second direction calculated by the second color difference calculating means;
Based on the inclination of the color difference signal in the first direction calculated by the first inclination calculation means and the inclination of the color difference signal in the second direction calculated by the second inclination calculation means, the first Second combining means for combining a signal based on the color difference signal in the first direction calculated by the color difference calculating means and a signal based on the color difference signal in the second direction calculated by the second color difference calculating means. The signal processing apparatus according to claim 1, further comprising:   The second interpolation means includes first addition means for adding a signal corresponding to the pixel of interest out of the signals synthesized by the second synthesis means and the second color signal in the pixel of interest. The signal processing apparatus according to claim 2.   The first synthesizing unit is configured to calculate the gradient of the color difference signal in the first direction calculated by the first gradient calculation unit and the color difference signal in the second direction calculated by the second gradient calculation unit. 4. The first color signal interpolated by the first interpolation means and the first color signal interpolated by the second interpolation means are synthesized based on an inclination. The signal processing apparatus as described. The second interpolation means includes
First filter means for performing filter processing in the first direction on the color difference signal in the first direction calculated by the first color difference calculation means;
Second filter means for performing filter processing in the second direction on the color difference signal in the second direction calculated by the second color difference calculation means;
5. The method according to claim 2, wherein the second synthesizing unit synthesizes the color difference signal filtered by the first filter unit and the color difference signal filtered by the second filter unit. The signal processing device according to claim 1.   The first synthesizing unit may determine the first combination when the larger one of the inclination of the color difference signal in the first direction and the inclination of the color difference signal in the second direction is smaller than a first threshold. When the color signal interpolated by the interpolation means is output and the larger one of the gradient of the color difference signal in the first direction and the gradient of the color difference signal in the second direction is larger than the second threshold value. The signal processing apparatus according to claim 1, wherein the color signal interpolated by the second interpolation means is output. The first interpolation means includes
Fifth interpolation means for interpolating the image signal in the first direction and the second direction;
Sixth interpolation means for performing interpolation in the first direction on the image signal;
The signal processing apparatus according to claim 1, further comprising: a seventh interpolation unit that performs interpolation on the image signal in the second direction.   The signal processing apparatus according to claim 2, wherein the first direction is a vertical direction, and the second direction is a horizontal direction. The first interpolation means interpolates the first color signal with respect to the pixel of interest corresponding to the second color signal of the image signal by using the first color signal in the pixels around the pixel of interest. A first interpolation step;
Using the second color signal and the color difference signal in the target pixel, the first color signal is obtained by the second interpolation unit with respect to the target pixel corresponding to the second color signal of the image signal. A second interpolation step to interpolate;
The first synthesizing unit synthesizes the color signal interpolated in the first interpolation step and the color signal interpolated in the second interpolation step based on the slopes of the plurality of color difference signals of the image signal. And a first synthesizing step.   The program for functioning a computer as each means of the signal processing apparatus of any one of Claims 1 thru | or 8.

Images