JP2013115650A - Image processing device, image processing method, and computer program - Google Patents

Abstract

The present invention provides an image processing apparatus that uses a smaller amount of memory than a conventional multiple-resolution super-resolution method and achieves high resolution with a small number of low-resolution images.
An image processing apparatus selects a reference image and a non-reference image from a plurality of images having a Bayer array, and enlarges the reference image to generate an enlarged reference image and a storage image. The image processing apparatus determines whether the target pixel of the enlarged standard image has the same color as the reference pixel of the non-standard image. If the target pixel has the same color, the reference pixel is inserted into the target pixel. In addition, when the target pixel of the enlarged standard image is not the same color as the reference pixel of the non-standard image, the image processing apparatus determines a reference pixel to be inserted based on a predetermined evaluation value, and stores the determined reference pixel for storage Insert into the reference pixel of the image.
[Selection] Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and a computer program.

  A plurality of super-resolutions that create one high-resolution image using a plurality of low-resolution images have been proposed. The multiple-image super-resolution technique is used in a digital camera to improve the image quality of electronic zoom. Conventionally, an image processing apparatus has created a single high-resolution image as follows using a low-resolution image obtained by an image sensor having a Bayer array color filter. In order to create a single high-resolution image from a Bayer-array low-resolution image, the image processing apparatus creates an image having the same resolution as the output for each color component, and processes multiple images for each image. Execute.

  In Patent Document 1, when a high-resolution image is created for each color component from a plurality of low-resolution images, a plurality of G components are subjected to super-resolution processing, and one R-component and B-component are processed. An image processing system that performs simple interpolation processing on an image to create a high-resolution image is disclosed.

JP 2002-112007 A

  In the conventional technique in which an image having the same resolution as the output is generated for each color component and a plurality of super-resolution processes are performed, there is a problem that a large amount of memory is used. In addition, since this conventional technique requires an image for each color component at the same resolution as the output, there is a problem that the number of low-resolution images necessary for interpolation of missing pixels increases. Further, the image processing system disclosed in Patent Document 1 has a problem that the effect of increasing the resolution greatly differs for each color component.

  It is an object of the present invention to provide an image processing apparatus that uses a smaller amount of memory than a conventional multiple-resolution super-resolution method and achieves high resolution with a small number of low-resolution images.

  An image processing apparatus according to an embodiment of the present invention includes a photographing unit that captures a plurality of images having a predetermined first pixel arrangement, and a reference image that is a single image from the plurality of photographed images. And selecting an image selection unit that uses an image other than the reference image among the plurality of images as a non-reference image, and converting the coordinates of the pixels included in the reference image into the coordinates of the pixels included in the non-reference image. A conversion coefficient calculating means for calculating a coordinate conversion coefficient and storing the calculated coordinate conversion coefficient in a storage means; an image enlarging means for enlarging the reference image at a predetermined enlargement magnification to obtain an enlarged reference image; Among the pixels included in the enlarged reference image, pixels arranged on the first pixel array are defined as first non-missing pixels, and pixels not arranged on the first pixel array are defined as first missing pixels. Determine the expansion group An enlarged reference image storage means for storing an image in the storage means, and pixels of the reference image that the enlarged reference image has that are not arranged on the same pixel array as the pixel array that the reference image has A storage image that is generated as a non-missing pixel and generates a storage image having a pixel other than the second non-missing pixel as a second missing pixel among the pixels of the enlarged reference image, and stores the image in the storage unit Using the generation means, the magnification of the reference image, and the coordinate conversion coefficient, the coordinates of the first missing pixel of the enlarged reference image are converted into reference coordinates, and the pixels of each non-standard image are converted. Of these, neighboring pixel determining means for determining a pixel closest to the reference coordinate as a neighboring pixel corresponding to the first missing pixel, a predetermined second pixel array, and the first missing pixel Coordinates and Based on the color determination means for determining the color of the first missing pixel and performing a color determination process for determining whether the determined color of the first missing pixel is the same as the color of the neighboring pixel; When it is determined that the color of the first missing pixel is the same as the color of the neighboring pixel, the neighboring pixel having the closest distance to the corresponding reference coordinate among the neighboring pixels of the non-reference image is the first. First reference pixel determining means for determining the reference pixel of the first missing pixel, and when it is determined that the color of the first missing pixel is not the same as the color of the neighboring pixel, the color of the first missing pixel and the neighborhood Evaluation value calculation means for calculating an evaluation value of the neighboring pixel according to the color of the pixel and the distance between the neighboring pixel of the non-standard image and the first reference coordinate; and the neighboring pixel of the non-reference image Among the neighboring pixels having the largest calculated evaluation value, the second reference image A second reference pixel determining means for determining as a prime; a first pixel inserting means for inserting the first reference pixel into the first missing pixel; and a second reference pixel as the second missing pixel. Second pixel insertion means for inserting.

  According to the image processing apparatus of the present invention, the amount of memory used can be reduced as compared with the conventional multi-image super-resolution method, and high resolution can be achieved with a small number of low-resolution images.

It is a figure which shows the structural example of the image processing apparatus of this embodiment. It is a figure which shows the structure of a part of image processing part. 3 is a flowchart illustrating image processing according to the first exemplary embodiment. It is a figure explaining the expansion method of a reference image. It is an example of the flowchart explaining an insertion pixel selection process. It is a figure explaining the calculation process of distance d [i]. It is a figure explaining a color judgment process. It is a figure which shows the flowchart explaining an example of a pixel insertion process. It is a figure explaining calculation of evaluation value e [i]. It is a figure explaining the example of a R, G, B simultaneous process. 10 is a flowchart illustrating image processing according to the second exemplary embodiment. It is a figure explaining the expansion process of a reference image.

  FIG. 1 is a diagram illustrating a configuration example of an image processing apparatus according to the present embodiment. The image processing apparatus generates one high-resolution image from a plurality of low-resolution images. The image processing apparatus includes an optical system 101, an image sensor 102, an A (Analog) / D (Digital) conversion unit 103, an image processing unit 104, a system control unit 105, an operation unit 106, a display unit 107, and a recording unit 108. .

  The optical system 101 includes a lens group including a zoom lens and a focus lens, a diaphragm adjusting device, and a shutter device. The optical system 101 adjusts the magnification, focus position, or light quantity of the subject image that reaches the image sensor 102. The imaging element 102 is a photoelectric conversion element such as a CCD or a CMOS sensor that photoelectrically converts a light beam of a subject that has passed through the optical system 101 and converts it into an electrical signal. CCD is an abbreviation for Charge Coupled Device. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor.

  In the present embodiment, the image sensor 102 is a single-plate image sensor having a Bayer array color filter. That is, the image sensor 102 functions as an imaging unit that captures a plurality of images having a predetermined first pixel array (Bayer array).

  The A / D conversion unit 103 converts the input video signal into a digital image. In addition to normal signal processing, the image processing unit 104 performs super-resolution processing using a plurality of input images. The image processing unit 104 can perform similar image processing not only on the image output from the A / D conversion unit 103 but also on the image read from the recording unit 108. The system control unit 105 is a control function unit that controls and controls the overall operation of the image processing apparatus. Based on the luminance value obtained from the image processed by the image processing unit 104 and the instruction transmitted from the operation unit 106, drive control of the optical system 101 and the image sensor 102 is also performed.

  The display unit 107 includes a liquid crystal display and an organic EL (Electro Luminescence) display. The display unit 107 displays an image generated by the image sensor 102 and an image read from the recording unit 108. The recording unit 108 has a function of recording an image. The recording unit 108 includes, for example, an information recording medium using a package containing a rotary recording body such as a memory card on which a semiconductor memory is mounted or a magneto-optical disk. This information recording medium may be detachable from the recording unit 108. A bus 109 is used for exchanging images among the image processing unit 104, the system control unit 105, the display unit 107, and the recording unit 108.

  FIG. 2 is a diagram illustrating a partial configuration of the image processing unit. FIG. 2 shows a processing unit related to the super-resolution processing executed by the image processing unit 104. The image processing unit 104 includes a reference image selection unit 201, a memory unit 202, a coordinate conversion coefficient calculation unit 203, an image generation unit 204, an inter-pixel distance calculation unit 205, an evaluation value calculation unit 206, a missing pixel determination unit 207, and a pixel insertion unit. 208, an RGB synchronization unit 209, and a YUV conversion unit 210. The image processing method of the present embodiment is realized by the function of the processing unit included in the image processing unit 104 illustrated in FIG. The computer program of this embodiment is characterized by causing a computer to execute this image processing method.

  The reference image selection unit 201 selects one image as a reference image from a plurality of images input to the image processing unit 104, and uses an image other than the reference image as a non-reference image among the plurality of images. It functions as a selection means. The memory unit 202 functions as a storage unit that stores image data related to processing executed by the reference image selection unit 201 to the RGB synchronization unit 209. The coordinate conversion coefficient calculation unit 203 functions as an alignment unit that aligns the non-reference image with respect to the reference image and stores the alignment result in the memory unit 202. Specifically, the coordinate conversion coefficient calculation unit 203 functions as a conversion coefficient calculation unit. That is, the coordinate conversion coefficient calculation unit 203 calculates a coefficient of a coordinate conversion expression that converts the coordinates of the pixel included in the reference image into the coordinates of the pixel included in the non-reference image, and the alignment coefficient is calculated as the alignment result. And

  The image generation unit 204 functions as an image enlarging unit that generates an enlarged reference image by enlarging the reference image at a predetermined enlargement magnification. In addition, the image generation unit 204 determines a pixel arranged on the first pixel array, that is, on the Bayer array, among the pixels included in the enlarged reference image as the first non-missing pixel. In addition, the image generation unit 204 determines pixels that are not arranged on the first pixel array as the first missing pixels. The image generation unit 204 functions as an enlarged reference image storage unit that stores the enlarged reference image in the memory unit 202.

  The image generation unit 204 functions as a storage image generation unit that generates a storage image and stores it in the memory unit 202. The storage image includes, as second non-missing pixels, pixels that are not arranged on the same pixel array as the pixel array (Bayer array) included in the reference image among the pixels of the reference image included in the enlarged reference image. The storage image has pixels other than the second non-missing pixel among the pixels of the enlarged reference image as the second missing pixel.

  The inter-pixel distance calculation unit 205 converts the coordinates of the first missing pixel included in the enlarged reference image into reference coordinates using the enlargement magnification of the reference image and the coordinate conversion formula in the memory unit 202. The inter-pixel distance calculation unit 205 calculates a distance d between the reference pixel and the reference coordinate. The reference coordinates are coordinates on the non-standard image corresponding to the missing pixels included in the enlarged standard image. The reference pixel is a pixel (neighboring pixel) that is closest to the reference coordinate among the pixels of the non-standard image.

  Specifically, the inter-pixel distance calculation unit 205 converts the coordinates of the missing pixels included in the enlarged reference image into reference coordinates by using the enlargement magnification of the reference image and the coordinate conversion formula in the memory unit 202. Then, the inter-pixel distance calculation unit 205 determines a reference pixel for each non-standard image, and calculates a distance d between the determined reference pixel and reference coordinates. That is, the inter-pixel distance calculation unit 205 functions as a neighboring pixel determining unit that determines a pixel having the closest distance to the reference coordinate as a neighboring pixel corresponding to the first missing pixel among the pixels of each non-standard image. To do.

  Further, the inter-pixel distance calculation unit 205 functions as a color determination unit that executes the following color determination process. That is, the inter-pixel distance calculation unit 205 determines the color of the first missing pixel based on a predetermined second pixel array (for example, a Bayer array) and the coordinates of the first missing pixel, It is determined whether the determined color of the first missing pixel is the same as the color of the reference pixel (neighboring pixel). Then, when the inter-pixel distance calculation unit 205 determines that the color of the first missing pixel is the same as the color of the reference pixel, the distance from the corresponding reference coordinate among the reference pixels included in the non-standard image is It functions as a first reference pixel determining unit that determines the closest reference pixel as the first reference pixel.

  The evaluation value calculation unit 206 functions as an evaluation value calculation unit that calculates the evaluation value of the neighboring pixel when it is determined that the color of the first missing pixel is not the same as the color of the reference pixel (neighboring pixel). The evaluation value calculation unit 206 calculates an evaluation value according to the color of the first missing pixel, the color of the neighboring pixel, and the distance d between the neighboring pixel and the first reference coordinate included in the non-reference image. Calculated as an evaluation value. In addition, the evaluation value calculation unit 206 functions as a second reference pixel determining unit that determines, as a second reference pixel, a neighboring pixel having the largest calculated evaluation value among neighboring pixels included in the non-standard image.

  The missing pixel determination unit 207 reads the enlarged reference image from the memory unit 202 and determines whether the target pixel of the read enlarged reference image, that is, the pixel to be processed is a missing pixel. The missing pixel determination unit 207 reads the storage image from the memory unit 202 and determines whether the target pixel of the read storage image, that is, the pixel to be processed is a missing pixel. After determining whether the pixel of interest of the enlarged reference image is a missing pixel, the missing pixel determination unit 207 determines whether the pixel of interest of the storage image at the same position as the pixel of interest of the enlarged reference image is a missing pixel. to decide.

  The pixel insertion unit 208 is a first pixel insertion unit that inserts the first reference pixel into the first missing pixel when the missing pixel determination unit 207 determines that the target pixel of the enlarged reference image is the missing pixel. Function as. The pixel insertion unit 208 also inserts the second reference pixel into the second missing pixel when the missing pixel determination unit 207 determines that the target pixel of the storage image is the missing pixel. It functions as an insertion means. The pixel insertion unit 208 stores the enlarged standard image and the storage image in which the reference pixel is inserted in the memory unit 202.

  The RGB synchronization unit 209 reads from the memory unit 202 the enlarged reference image in which the first reference pixel is inserted into the first missing pixel. In addition, the RGB synchronization unit 209 reads from the memory unit 202 the storage image in which the second reference pixel is inserted into the second missing pixel. The RGB synchronization unit 209 functions as a color synchronization unit that performs color synchronization processing based on the read enlarged reference image and storage image. The YUV conversion unit 210 converts the RGB signal of the pixels of the image after the color synchronization processing into a YUV signal and outputs it as an output image.

Example 1
FIG. 3 is a flowchart illustrating image processing according to the first embodiment. In the first embodiment, the image processing apparatus uses a plurality of low-resolution images of the Bayer array to generate a high-resolution image of the Bayer array in which the number of pixels is expanded three times in the horizontal and vertical directions, thereby realizing high resolution. To do.

  First, the A / D conversion unit 103 inputs n low-resolution images to the image processing unit 104 (step S301). Here, the n images are images that are continuously captured in a handheld state. Note that the input low-resolution image is not limited to an image captured in a handheld state, but has a configuration in which the photographer intends to move (panning or zooming) or mechanically moves the camera optical device or image sensor. May be obtained.

  Next, the reference image selection unit 201 selects, as a reference image, an image to be a high resolution image by super-resolution processing (step S302). The remaining images are stored in the memory unit 202 as non-reference images. Here, the reference image selection unit 201 sets the first input image as a reference image, and the remaining n−1 images as non-reference images. Note that the reference image selection criterion is not limited to this method, and the reference image selection unit 201 may select from images other than the first input image.

  Next, the coordinate conversion coefficient calculation unit 203 calculates a coefficient of a coordinate conversion formula necessary for alignment between the reference image and the non-reference image (step S303). In this embodiment, a projective transformation formula is used as the coordinate transformation formula. If the coordinates of a predetermined position (for example, the center position) of the image are (0, 0), the coordinates (x ′, y ′) of the non-reference image corresponding to the coordinates (x, y) of the target pixel of the reference image. The expression for projective transformation representing) is the following expression (1).

基準画像の着目画素の座標（ｘ、ｙ）に対応する非基準画像の座標（ｘ’，ｙ’）を参照座標と呼称する。式（１）の射影変換係数（座標変換係数）の算出方法については、基準画像と非基準画像間でパターンマッチングによって求めた動きベクトルから連立方程式を用いて解くのが一般的である。しかし、座標変換係数算出部２０３が、撮影時のジャイロデータを用いて、射影変換係数を求めるようにしてもよい。

The coordinates (x ′, y ′) of the non-standard image corresponding to the coordinates (x, y) of the target pixel of the standard image are referred to as reference coordinates. The calculation method of the projective transformation coefficient (coordinate transformation coefficient) in Expression (1) is generally solved using simultaneous equations from motion vectors obtained by pattern matching between the reference image and the non-reference image. However, the coordinate conversion coefficient calculation unit 203 may obtain the projective conversion coefficient using the gyro data at the time of shooting.

  Next, the coordinate conversion coefficient calculation unit 203 stores the calculated coordinate conversion coefficients in the memory unit 202 by the number of non-reference images (step S304). The stored coordinate conversion coefficient is used in step S306.

  Next, the image generation unit 204 enlarges the reference image three times in the horizontal direction and in the vertical direction (step S305). In the present embodiment, the image generation unit 204 enlarges the reference image while maintaining the Bayer arrangement. Note that the process of step S305 may be performed before the process of step S303.

  FIG. 4 is a diagram for explaining the reference image enlargement method in step S305. FIG. 4A shows a reference image before enlargement. This reference image has a Bayer array with a horizontal pixel number W and a vertical pixel number H. The image generation unit 204 enlarges the reference image shown in FIG. 4A three times in each of the horizontal direction and the vertical direction, and generates an enlarged reference image. FIG. 4B is an enlarged reference image. Of the pixels included in the enlarged reference image, the image generation unit 204 sets a pixel included in the reference image, that is, a pixel arranged on the Bayer array as a first non-missing pixel. Of the pixels included in the enlarged reference image, the image generation unit 204 sets pixels other than the pixels included in the reference image, that is, pixels that are not arranged on the Bayer array as the first missing pixels. The image generation unit 204 inserts 0 as a value in the first missing pixel and stores the enlarged reference image in the memory unit 202.

  As shown in FIG. 4B, when a reference image having a Bayer array with the number of horizontal pixels W × 2 and the number of vertical pixels H × 2 is created, among the pixels of the reference image before enlargement, the G component pixels are non-missing pixels. Can be left as. However, since the pixels of the R component and the B component are respectively located at the position of the G component of the enlarged reference image, when the pixels are left as non-missing pixels, the enlarged reference image cannot maintain the Bayer array. Here, when performing the color synchronization processing later, it is better that there is more information on the non-missing pixels. Therefore, it is necessary to store the R and B pixels separately from the enlarged reference image.

  That is, the image generation unit 204 generates a storage image having a horizontal pixel number W × 2 and a vertical pixel number H × 2, as shown in FIG. 4C, separately from the enlarged reference image in which the Bayer array is maintained. To do. The image generation unit 204 sets the R component and B component pixels shown in FIG. 4C among the pixels of the storage image as the second non-missing image. Also, the image generation unit 204 sets pixels other than the R component and B component pixels shown in FIG. 4C among the pixels of the storage image as the second non-missing image. The image generation unit 204 inserts 0 as a value into the second missing pixel, and stores the storage image in the memory unit 202.

  Returning to FIG. 3, in step S306, the image processing apparatus selects the first reference pixel candidate pix to be inserted and the second reference pixel candidate sub_pix from the n−1 non-standard images (inserted pixel). Perform selection process).

  FIG. 5 is an example of a flowchart for explaining the inserted pixel selection process in step S306 of FIG. First, the inter-pixel distance calculation unit 205 initializes a variable i (step S501). The variable i is identification information for uniquely identifying n−1 non-reference images. In this embodiment, the (i + 1) th non-reference image is the non-reference image [i], and the coordinate conversion coefficient corresponding to the non-reference image [i] is the coordinate conversion coefficient [i]. Note that d [i], pix [i], e [i], sub_pix [i], and color_id [i] used in and after step S505 are d, pix, e corresponding to the non-reference image [i], respectively. , Sub_pix, color_id.

  Next, the inter-pixel distance calculation unit 205 reads the non-reference image [i] from the memory unit 202 (step S502). Subsequently, the inter-pixel distance calculation unit 205 reads the coordinate conversion coefficient [i] corresponding to the non-reference image [i] from the memory unit 202 (step S503).

  Next, the reference coordinates of the non-standard image [i] corresponding to the coordinates (x, y) of the target pixel of the standard image using the coordinate conversion coefficient [i] read by the inter-pixel distance calculation unit 205 in step S503. (X ′, y ′) is calculated (step S504). Specifically, the inter-pixel distance calculation unit 205 calculates the reference coordinates using the projective transformation formula shown by the formula (1) described above.

  Next, the inter-pixel distance calculation unit 205 performs the pixel (nx, y) of the non-standard image [i] that is closest to the reference coordinates (x ′, y ′) of the non-standard image [i] calculated in step S504. ny) is a reference pixel (neighboring pixel). Then, the inter-pixel distance calculation unit 205 calculates the distance d [i] between the reference pixel and the reference coordinate (step S505).

  FIG. 6 is a diagram for explaining the calculation process of the distance d [i] in step S505 in FIG. FIG. 6A shows an enlarged image. FIG. 6B shows a non-reference image. For the coordinates (x, y) of the pixel of interest 601 on the enlarged reference image in FIG. 6A, the reference coordinates 602 of the corresponding non-reference image (FIG. 6B) are (x ′, y ′). is there.

  The inter-pixel distance calculation unit 205 determines a pixel 603 that includes the reference coordinate 602 and is located closest to the reference coordinate 602 as a reference pixel corresponding to the reference coordinate 602. If the coordinates of the reference pixel 603 are (nx, ny), the inter-pixel distance calculation unit 205 calculates the distance d between the reference pixel and the reference coordinates using the following equation (2).

上記のようにして、画素間距離算出部２０５は、非基準画像［ｉ］の参照座標（ｘ’，ｙ’）に対応する非基準画像［ｉ］の参照画素（ｎｘ，ｎｙ）を決定する。また、画素間距離算出部２０５は、非基準画像［ｉ］の参照座標（ｘ’，ｙ’）に対応する距離ｄ［ｉ］を算出する。

As described above, the inter-pixel distance calculation unit 205 determines the reference pixel (nx, ny) of the non-standard image [i] corresponding to the reference coordinates (x ′, y ′) of the non-standard image [i]. . The inter-pixel distance calculation unit 205 calculates a distance d [i] corresponding to the reference coordinates (x ′, y ′) of the non-standard image [i].

  Returning to FIG. 5, the inter-pixel distance calculation unit 205 determines whether or not the reference pixel of the non-standard image [i] is the same color as the target pixel of the enlarged standard image (executes a color determination process) (step S506). If the target pixel has the same color as the reference pixel, the process proceeds to step S507. If the target pixel is not the same color as the reference pixel, the process proceeds to step S508.

  FIG. 7 is a diagram for explaining the color determination process. FIG. 7 shows the color of each pixel of the enlarged reference image having the horizontal pixel number W × 3 and the vertical pixel number H × 3. That is, this enlarged reference image has a Bayer array. In order to replace a missing pixel with a non-missing pixel, it must be replaced with a pixel of a color described at the position of the missing pixel.

The coordinates of the upper left pixel of the enlarged reference image are (0, 0), and the coordinates of the lower right pixel are (2W-1, 2H-1). The inter-pixel distance calculation unit 205 determines the color of the pixel of interest (x, y) by the following equation (3).
R if (x & 1) = 0 and (y & 1) = 0
G if (x & 1) = 1 and (y & 1) = 0, or (x & 1) = 0 and (y & 1) = 1, B if (x & 1) = 1 and (y & 1) = 1
... Formula (3)

  The above equations (x & 1) and (y & 1) are equations for determining whether the x-coordinate and y-coordinate are even or odd, respectively. When the values of (x & 1) and (y & 1) are 0, the x and y coordinates are even, and when the value is 1, the x and y coordinates are odd.

  Note that the inter-pixel distance calculation unit 205 also sets the coordinates of the upper left pixel to (0, 0) and the coordinates of the lower right pixel for the color of the reference pixel (nx, ny) of the non-standard image [i]. As (W−1, H−1), the determination is made using Equation (3). Then, the inter-pixel distance calculation unit 205 determines whether the determined color of the target pixel (x, y) is the same color as that of the reference pixel (nx, ny). That is, the inter-pixel distance calculation unit 205 performs color determination processing using a Bayer array as the second pixel array.

  Returning to FIG. 5, the inter-pixel distance calculation unit 205 sets the reference pixel of the non-standard image [i] determined to be the same color as the target pixel of the standard image as pix [i], and is calculated in step S <b> 505. i] and stored in the memory unit 202 (step S507). Then, the process proceeds to step S510.

  In step S508, the evaluation value calculation unit 206 determines the color of the target pixel of the enlarged reference image and the color of the reference pixel of the non-reference image [i] determined not to be the same as the color of the target image, in step S505. An evaluation value e [i] is calculated according to the calculated distance d [i]. The evaluation value e [i] is a reference value serving as a reference for selection as the second reference pixel with respect to the reference pixels (neighboring pixels) of the non-standard image [i]. A reference pixel with a higher evaluation value is easily selected as the second reference pixel. Further, the evaluation value increases as the distance d from the reference coordinate is closer.

  FIG. 9 is a diagram illustrating calculation of the evaluation value e [i]. The evaluation value calculation unit 206 calculates an evaluation value e [i] corresponding to the distance d [i] according to the graph shown in FIG. 9 (A), (B), or (C). The horizontal axis of the graph shown in FIG. 9 is the distance d [i], and the vertical axis is the evaluation value e [i]. R, G, and B in the graph are the colors of the reference pixels of the non-standard image [i] corresponding to each graph line.

  FIG. 9A is a graph when the target pixel of the standard image is R. G in this graph is the distance d [i] and the evaluation value e when the reference image of the non-standard image is G. The relationship with [i] is shown. B in this graph indicates the relationship between the distance d [i] and the evaluation value e [i] when the reference image of the non-standard image is B.

  FIG. 9B is a graph when the pixel of interest of the base image is G, and R and B in this graph are distances d [i] when the reference image of the non-base image is R or B. And the evaluation value e [i].

  FIG. 9C is a graph when the target pixel of the standard image is B, and G in this graph is the distance d [i] and the evaluation value e when the reference image of the non-standard image is G. The relationship with [i] is shown. R in this graph indicates the relationship between the distance d [i] and the evaluation value e [i] when the reference image of the non-standard image is R.

  In this embodiment, in order to be able to replace many missing pixels in the storage image with non-missing pixels of G having a large contribution ratio of the luminance system signal, when the reference pixel is G, R, B A graph is set in advance so that the evaluation value [i] is larger than that in the above case. For example, as in the graphs shown in FIGS. 9A and 9C, the evaluation value e [i] is higher in G than in the case where the color of the reference pixel of the non-standard image [i] is R or B. Set the graph to be larger.

  Returning to FIG. 5, the evaluation value calculation unit 206 sets the color of the reference pixel of the non-standard image [i] as color_id [i] and the pixel value as sub_pix [i], and the evaluation value e [i] calculated in step S508. At the same time, it is stored in the memory unit 202 (step S509). A value from 0 to 2 is substituted for color_id [i]. The value “0” of color_id [i] indicates the G component. The value “1” of color_id [i] indicates the R component. The value “2” of color_id [i] indicates the B component.

  Next, the evaluation value calculation unit 206 determines whether the processing has been completed for all the n−1 non-reference images (step S508). If there is a non-reference image that has not been processed, the evaluation value calculation unit 206 increments i (step S509) and returns to step S502. If the processing has been completed for all the non-reference images, the process proceeds to step S512.

  Next, the evaluation value calculation unit 206 selects, as e_max, the value of the evaluation value e [i], which is the maximum value among the evaluation values e [i] stored in the memory unit 202 (step S512). When there is no evaluation value e [i] stored in the memory unit 202, the evaluation value calculation unit 206 substitutes a settable minimum value (for example, “0”) into e_max.

  Next, the evaluation value calculation unit 206 selects the reference pixel sub_pix [i] corresponding to e_max (e [i]) selected in step S512 as a non-missing pixel candidate sub_pix (step S513). When there is no e [i] stored in the memory unit 202, the evaluation value calculation unit 206 substitutes 0 for the value of sub_pix.

  Next, the evaluation value calculation unit 206 selects the color “color_id [i]” of the reference pixel corresponding to “e_max” selected in step S512 as “color_id” indicating the color of the non-missing pixel candidate sub_pix (step S514). When there is no e [i] stored in the memory unit 202, the evaluation value calculation unit 206 substitutes 0 for the value of color_id.

Next, the evaluation value calculation unit 206 selects the value of the distance d [i], which is the minimum value among the distances d [i] stored in the memory unit 202, as d_min (step S515). When there is no distance d [i] stored in the memory unit 202, the evaluation value calculation unit 206 determines that d_min
Is substituted with a settable maximum value (for example, “1”).

  Next, the evaluation value calculation unit 206 selects the reference pixel pix [i] corresponding to the distance d_min selected in step S515 as a non-missing pixel candidate pix (step S516). When there is no distance d [i] stored in the memory unit 202, the evaluation value calculation unit 206 substitutes 0 for the value of pix. Subsequently, the evaluation value calculation unit 206 outputs e_max, sub_pix, color_id, d_min, and pix (step S517).

  Returning to FIG. 3, the pixel insertion unit 208 executes a pixel insertion process (step S307). Specifically, the pixel insertion unit 208 performs processing to replace the value of the first missing pixel of the enlarged reference image with the value of pix that is the first reference image. In addition, the pixel insertion unit 208 performs processing to replace the value of the second missing pixel in the storage image with the value of sub_pix that is the second reference image.

  FIG. 8 is a flowchart illustrating an example of the pixel insertion process. First, the pixel insertion unit 208 determines whether the target pixel of the reference image is a missing pixel (step S801). If the target pixel of the reference image is a missing pixel, the process proceeds to step S802. If the target pixel of the reference image is not a missing pixel, the process proceeds to step S804.

  In step S802, the pixel insertion unit 208 determines whether the value of d_min is less than the threshold value TH_d (step S802). The threshold value TH_d is determined in advance. A value obtained by an empirical rule or experiment may be used as the threshold value TH_d.

  In step S803, the pixel insertion unit 208 replaces the pixel of interest of the reference image with a non-missing pixel candidate pix. In step S804, the pixel insertion unit 208 does not replace the target pixel of the reference image with the non-missing pixel candidate pix. Then, the process proceeds to step S805.

  The processing in steps S805 to S808 is processing performed on the target pixel of the storage image. The position of the target pixel of the storage image to be processed is the same as the position of the target pixel of the reference image that has been processed in steps S801 to S804.

  First, the pixel insertion unit 208 determines whether the target pixel of the storage image is a missing pixel (step S805). If the target pixel of the image for storage is a missing pixel, the process proceeds to step S806. If the target pixel of the image for storage is not a missing pixel, the process proceeds to step S808.

  In step S806, the pixel insertion unit 208 determines whether the value of e_max is greater than or equal to the threshold value TH_e (step S806). The threshold value TH_d is determined in advance. A value obtained by an empirical rule or experiment may be used as the threshold value TH_d. If the value of e_max is greater than or equal to the threshold value TH_e, the process proceeds to step S807. If the value of e_max is less than the threshold value TH_e, the process proceeds to step S808.

  In step S807, the pixel insertion unit 208 replaces the target pixel of the storage image with a non-missing pixel candidate sub_pix (step S807). In step S808, the pixel insertion unit 208 does not replace the target pixel of the image for storage with the candidate pixel pix of the non-missing pixel. Then, the process ends.

  Returning to FIG. 3, the pixel insertion unit 208 determines whether all pixels in the reference image and the storage image have been processed (step S308). When all the pixels in the reference image and the storage image have been processed, the process proceeds to step S309. If there are unprocessed pixels in the reference image and the storage image, the target pixel is changed, and the process returns to step S306.

  Next, the RGB synchronization unit 209 executes R, G, B synchronization processing (color synchronization processing) based on the enlarged reference image and the storage image after the pixel insertion processing (step S309).

  FIG. 10 is a diagram for explaining an example of the R, G, B synchronization processing. FIG. 10A is an enlarged reference image after pixel insertion processing. In this example, the enlarged reference image is a Bayer array image having a horizontal pixel number W × 2 and a vertical pixel number H × 2. FIG. 10B shows an image for storage after pixel insertion processing.

  The RGB synchronization unit 209 performs R, G, B synchronization processing using the enlarged reference image shown in FIG. 10A and the storage image shown in FIG. Through the image shown in (E). Specifically, the RGB synchronizer 209 creates an image having a horizontal pixel number W × 2 and a vertical pixel number H × 2 for each of R, G, and B. When performing the R, G, B synchronization processing, non-missing pixels of the enlarged reference image and the storage image are set to the pixels of the plane having the horizontal pixel number W × 2 and the vertical pixel number H × 2. .

  When there is no non-missing pixel of the same color in the enlarged reference image or the storage image for the setting target pixel, the RGB synchronization unit 209 linearly sets the pixel value of the pixel to the non-missing pixel around the pixel. Created by interpolated values. In this embodiment, linear interpolation is used, but the RGB synchronization unit 209 may perform synchronization processing by adaptive interpolation.

Returning to FIG. 3, the YUV conversion unit 210 generates a YUV image based on the image after the R, G, B synchronization processing (step S <b> 310). In step S310, the YUV conversion unit 210 converts RGB signals into YUV signals for each pixel using R, G, and B images obtained by the R, G, and B synchronization processing. Thereby, a YUV image having a horizontal pixel number W × 2 and a vertical pixel number H × 2 is created. The YUV conversion unit 210 converts RGB signals into YUV signals using the following equation (4).
Y = 0.299R + 0.587G + 0.114B
U = −0.169R−0.331G + 0.500B
V = 0.500R-0.419G-0.081B Formula (4)

  The YUV conversion unit 210 ends the super-resolution processing using the YUV image generated in step S310 as an output signal.

  The image processing apparatus according to the present embodiment uses a plurality of Bayer-array low-resolution images to generate a Bayer-array high-resolution image in which the number of pixels is increased three times in the horizontal and vertical directions, thereby realizing high resolution. Super-resolution processing can be performed. In other words, the image processing apparatus according to the present embodiment stores one enlarged reference image in the memory unit 202, and uses the non-reference image having the Bayer array to detect missing pixels of the enlarged reference image. Fill the image so that the Bayer array is maintained. Therefore, according to the image processing apparatus of the present embodiment, the amount of memory used can be reduced and the number of sheets can be reduced compared to the conventional multi-resolution super-resolution method that creates an image having the same resolution as the output for each color component. High resolution can be achieved with a low resolution image.

  In addition, the image processing apparatus according to the present embodiment includes, as non-missing pixels, pixels that are not arranged on the same pixel array as the pixel array (Bayer array) included in the reference image among the pixels of the reference image included in the enlarged reference image. A storage image is generated and stored separately from the enlarged reference image. The image processing apparatus determines a reference pixel to be inserted based on a predetermined evaluation value and saves the determined reference pixel when the color of the pixel of interest of the enlarged standard image is not the same as the color of the reference pixel of the non-standard image Is inserted into the target pixel of the image. Then, the image processing apparatus executes R, G, B synchronization processing based on the enlarged reference image and the storage image after the pixel insertion processing. As described above, the image processing apparatus according to the present embodiment executes the R, G, B synchronization processing using the enlarged reference image and the storage image having the pixels of the reference image as non-missing pixels. Higher resolution can be realized than when the R, G, B synchronization processing is executed using only the image.

  Note that the image processing method of the present embodiment can be applied not only when the enlargement ratio of the reference image is doubled but also when the reference image is enlarged three times or four times in the horizontal and vertical directions.

(Example 2)
The image processing apparatus according to the second exemplary embodiment changes the reference image enlargement method according to the number of low-resolution images input to the image processing unit 104. The configuration of the image processing apparatus according to the second embodiment is the same as that of the image processing apparatus according to the first embodiment. The configuration of the image processing unit of the second embodiment is the same as the configuration of the image processing unit of the first embodiment.

  FIG. 11 is a flowchart illustrating image processing according to the second embodiment. Steps S1101 to S1104 are the same as steps S301 to S304 in FIG.

  In step S1104, the image generation unit 204 determines whether the number of low-resolution images input in step S1101 is less than the threshold value TH1 (step S1105). The threshold value TH1 is determined in advance. A value obtained by an empirical rule or experiment may be used as the threshold value TH1.

  If the number of low resolution images input in step S1101 is less than the threshold TH1, the process proceeds to step S1107. If the number of low resolution images input in step S1101 is not less than the threshold TH1, the image generation unit 204 determines whether the number of low resolution images input in step S1101 is less than the threshold TH2 (step S1106). . The threshold value TH2 is larger than the threshold value TH1 and is determined in advance. A value obtained by an empirical rule or experiment may be used as the threshold value TH2.

  If the number of low-resolution images input in step S1101 is less than the threshold value TH2, the process proceeds to step S1108. If the number of low resolution images input in step S1101 is not less than the threshold TH2, the process proceeds to step S1109.

  In step S1107, the image generation unit 204 executes an enlargement process using the first enlargement method for the reference image. The enlargement process using the first enlargement method is a process for generating a first enlarged reference image by enlarging the reference image at a predetermined enlargement magnification so that the Bayer arrangement of the reference image is maintained. .

  FIG. 12 is a diagram for explaining the reference image enlargement process. FIG. 12A shows a reference image before enlargement. When the image generation unit 204 executes the enlargement process using the first enlargement method, the image generation unit 204 performs the reference processing so that the Bayer array included in the reference image is maintained as illustrated in FIG. Enlarge the image twice. As a result, a first enlarged reference image of one plane is generated. Pixels not hatched in FIG. 12B are non-missing pixels.

  In step S1108, the image generation unit 204 executes an enlargement process using the second enlargement method for the reference image. When performing an enlargement process using the second enlargement method, the image generation unit 204 separates the reference image into a G component image and an RB component image. The image generation unit 204 enlarges the G component image twice in both the horizontal and vertical directions to generate a second enlarged reference image. The image generation unit 204 enlarges the image of the RB component of the reference image twice in both the horizontal and vertical directions so that the R component and the B component are arranged in a checkered pattern, whereby the third enlarged reference image Is generated.

  FIGS. 12C and 12D show two-plane enlarged reference images generated using the second enlargement method. FIG. 12C is a second enlarged reference image. The second enlarged reference image has a pixel array in which pixels of only the G component are arranged. FIG. 12D is a third enlarged reference image. The third enlarged reference image has a pixel array in which the R component and the B component are arranged in a checkered pattern.

  As shown in FIG. 12C, the second enlarged reference image has more G component non-missing pixels than the first enlarged reference image shown in FIG. Therefore, the high-resolution image of the G component generated based on the second enlarged reference image is higher than the high-resolution image of the G component generated based on the first enlarged reference image shown in FIG. Increases resolution.

  In step S1109, the image generation unit 204 executes an enlargement process using the third enlargement method for the reference image. When performing the enlargement process using the third enlargement method, the image generation unit 204 creates an image in which the reference image is separated for each color component. Then, the image generation unit 204 generates the fourth, fifth, and sixth enlarged reference images by enlarging each created image in the horizontal and vertical directions at a predetermined magnification. Specifically, the image generation unit 204 generates the fourth enlarged reference image by magnifying the G component image twice. The image generation unit 204 generates a fifth enlarged reference image by magnifying the R component image twice. The image generation unit 204 generates a sixth enlarged reference image by magnifying the B component image twice.

  FIGS. 12E to 12G show three-plane enlarged reference images generated using the third enlargement method. FIG. 12E is a fourth enlarged reference image. The fourth enlarged reference image has a pixel array in which pixels of only the G component are arranged. The fifth enlarged reference image has a pixel array in which pixels of only the R component are arranged. The sixth enlarged reference image has a pixel array in which pixels of only the B component are arranged.

  The processing after step S1110 in FIG. 11 is the same as the processing after step S306 in FIG. Note that the color determination process performed in the insertion pixel selection process in step S1110 is performed as follows. The inter-pixel distance calculation unit 205 performs color determination processing on the first enlarged reference image using the Bayer array as the second pixel array.

  For the second enlarged reference image, the inter-pixel distance calculation unit 205 performs color determination processing using an array in which pixels of only the G component are arranged as the second pixel array. That is, the color of the pixel of interest in the second enlarged reference image is all set to G and then compared with the color of the reference pixel. For the third enlarged reference image, the inter-pixel distance calculation unit 205 performs color determination processing using a pixel array in which R components and B components are arranged in a checkered pattern as the second pixel array. That is, the inter-pixel distance calculation unit 205 determines the color of the pixel of interest of the third enlarged reference image as a color determined according to the pixel arrangement in which the R component and the B component are arranged in a checkered pattern, Compare with the color of the reference pixel.

  The inter-pixel distance calculation unit 205 executes color determination processing for the fourth enlarged reference image using an array in which pixels of only the G component are arranged as the second pixel array. That is, the color of the pixel of interest in the second enlarged reference image is all set to G and then compared with the color of the reference pixel. For the fifth enlarged reference image, the inter-pixel distance calculation unit 205 performs color determination processing using an array in which only R component pixels are arranged as the second pixel array. That is, the color of the pixel of interest in the second enlarged reference image is all set to R and then compared with the color of the reference pixel. For the sixth enlarged reference image, the inter-pixel distance calculation unit 205 performs color determination processing using an array in which pixels of only the B component are arranged as the second pixel array. That is, the color of the pixel of interest in the second enlarged reference image is all set to B and then compared with the color of the reference pixel.

  By executing the color determination process based on the enlargement process result using the third enlargement method, the following effects can be obtained. The color of the reference pixel is the same color as one of the target pixels at the same position in each of the three-plane enlarged reference images. Therefore, when the target pixel is a missing pixel, a reference pixel is inserted into the target pixel of any plane of the three-plane enlarged reference image. Thereby, the missing pixels of the enlarged reference image can be filled efficiently. Furthermore, as the number of low-resolution images increases, the number of non-standard images from which reference pixels are provided increases. Therefore, even if the enlarged standard image has three planes, missing pixels are more likely to be filled. As a result, it is possible to reduce the processing load of the RGB synchronization unit 209 in the R, G, B synchronization process to be executed later.

  According to the image processing apparatus of this embodiment, depending on the number of input low resolution images, the memory usage is reduced, the resolution of the G component is increased, or the load of the R, G, B synchronization processing is increased. Either can be realized.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

DESCRIPTION OF SYMBOLS 101 Optical system 102 Image pick-up element 103 A / D conversion part 104 Image processing part 105 System control part 106 Operation part 107 Display part 108 Recording part

Claims (7)

Photographing means for capturing a plurality of images having a predetermined first pixel arrangement;
An image selection means for selecting one image as a reference image from the plurality of captured images, and setting an image other than the reference image among the plurality of images as a non-reference image;
A conversion coefficient calculation unit that calculates a coordinate conversion coefficient for converting the coordinates of the pixel included in the reference image into the coordinates of the pixel included in the non-reference image, and stores the calculated coordinate conversion coefficient in a storage unit;
Image enlargement means for enlarging the reference image at a predetermined enlargement magnification to obtain an enlarged reference image;
Of the pixels of the enlarged reference image, the pixels arranged on the first pixel array are the first non-missing pixels, and the pixels not arranged on the first pixel array are the first missing pixels. An enlarged reference image storage means for determining and storing the enlarged reference image in a storage means;
Among the pixels of the reference image that the enlarged reference image has, pixels that are not arranged on the same pixel array as the pixel array that the reference image has as a second non-missing pixel, and that the enlarged reference image has A storage image generating unit that generates a storage image having a pixel other than the second non-missing pixel as a second missing pixel, and stores the storage image in a storage unit;
Using the enlargement magnification of the reference image and the coordinate conversion coefficient, the coordinates of the first missing pixel included in the enlarged reference image are converted to reference coordinates, and the reference among the pixels included in each non-reference image Neighboring pixel determining means for determining a pixel having the closest distance to the coordinates as a neighboring pixel corresponding to the first missing pixel;
Based on a predetermined second pixel array and the coordinates of the first missing pixel, the color of the first missing pixel is determined, and the determined color of the first missing pixel is the color of the neighboring pixel. Color determination means for performing color determination processing to determine whether the color is the same;
When it is determined that the color of the first missing pixel is the same as the color of the neighboring pixel, the neighboring pixel having the closest distance to the corresponding reference coordinate among the neighboring pixels included in the non-standard image is the first pixel. First reference pixel determining means for determining one reference pixel;
When it is determined that the color of the first missing pixel is not the same as the color of the neighboring pixel, the color of the first missing pixel, the color of the neighboring pixel, and the neighboring pixel that the non-reference image has Evaluation value calculating means for calculating an evaluation value of the neighboring pixels according to a distance from the first reference coordinate;
A second reference pixel determining unit that determines a neighboring pixel having the largest calculated evaluation value as a second reference pixel among neighboring pixels included in the non-reference image;
First pixel insertion means for inserting the first reference pixel into the first missing pixel;
An image processing apparatus comprising: a second pixel insertion unit that inserts the second reference pixel into the second missing pixel. Based on the enlarged reference image in which the first reference pixel is inserted into the first missing pixel and the storage image in which the second reference pixel is inserted into the second missing pixel, The image processing apparatus according to claim 1, further comprising color synchronization means for performing synchronization processing. The image processing apparatus according to claim 1, wherein the image enlargement unit enlarges the reference image at the enlargement magnification so that the first pixel arrangement is maintained. The image processing apparatus according to any one of claims 1 to 3, wherein the first and second pixel arrays are Bayer arrays. The first pixel array is a Bayer array;
The image enlargement means enlarges the reference image by any one of the first enlargement method, the second enlargement method, and the third enlargement method according to the number of the taken images.
The first enlargement method generates the first enlarged reference image by enlarging the reference image at the enlargement magnification so that the Bayer array of the reference image is maintained.
In the second enlargement method, the reference image is separated into a G component image and an RB component image, the G component image is enlarged in the horizontal and vertical directions at the enlargement magnification, and the second enlarged reference image is obtained. And generating a third enlarged reference image by enlarging the image of the RB component of the reference image in the horizontal and vertical directions at the enlargement magnification so that the R component and the B component are arranged in a checkered pattern. And
In the third enlargement method, an image obtained by separating the reference image for each color component is created, and each of the created images is enlarged in the horizontal direction and the vertical direction at the enlargement magnification. An enlarged reference image of
The color determining means is
For the first enlarged reference image, the color determination process is performed using a Bayer array as the second pixel array,
For the second enlarged reference image, the color determination process is executed using an array in which pixels of only the G component are arranged as the second pixel array, and for the third enlarged reference image, the second enlarged reference image The color determination process is executed using a pixel array in which R component and B component are arranged in a checkered pattern as the pixel array of 2.
For the fourth enlarged reference image, the color determination process is executed using an array in which pixels of only the G component are arranged as the second pixel array, and for the fifth enlarged reference image, The color determination process is executed using an array in which pixels of only the R component are arranged as the second pixel array, and only the B component is arranged as the second pixel array in the sixth enlarged reference image. The image processing apparatus according to claim 1, wherein the color determination process is executed using an array to be processed. Capturing a plurality of images having a predetermined first pixel arrangement;
Selecting a reference image as one image from the plurality of captured images, and setting an image other than the reference image as a non-reference image among the plurality of images;
Calculating a coordinate conversion coefficient for converting the coordinates of the pixel included in the reference image into the coordinates of the pixel included in the non-reference image, and storing the calculated coordinate conversion coefficient in a storage unit;
Enlarging the reference image at a predetermined enlargement magnification to obtain an enlarged reference image;
Of the pixels of the enlarged reference image, the pixels arranged on the first pixel array are the first non-missing pixels, and the pixels not arranged on the first pixel array are the first missing pixels. And storing the enlarged reference image in a storage means,
Among the pixels of the reference image that the enlarged reference image has, pixels that are not arranged on the same pixel array as the pixel array that the reference image has as a second non-missing pixel, and that the enlarged reference image has Generating a storage image having a pixel other than the second non-missing pixel as the second missing pixel, and storing it in the storage unit;
Using the enlargement magnification of the reference image and the coordinate conversion coefficient, the coordinates of the first missing pixel included in the enlarged reference image are converted to reference coordinates, and the reference among the pixels included in each non-reference image Determining a pixel closest to the coordinate as a neighboring pixel corresponding to the first missing pixel;
Based on a predetermined second pixel array and the coordinates of the first missing pixel, the color of the first missing pixel is determined, and the determined color of the first missing pixel is the color of the neighboring pixel. A step of performing color determination processing for determining whether the color is the same;
When it is determined that the color of the first missing pixel is the same as the color of the neighboring pixel, the neighboring pixel having the closest distance to the corresponding reference coordinate among the neighboring pixels included in the non-standard image is the first pixel. Determining as one reference pixel;
When it is determined that the color of the first missing pixel is not the same as the color of the neighboring pixel, the color of the first missing pixel, the color of the neighboring pixel, and the neighboring pixel that the non-reference image has Calculating an evaluation value of the neighboring pixels according to a distance from the first reference coordinates;
Determining a neighboring pixel having the largest calculated evaluation value among neighboring pixels included in the non-reference image as a second reference pixel;
Inserting the first reference pixel into the first missing pixel;
Inserting the second reference pixel into the second missing pixel. An image processing method comprising:   A computer program causing a computer to execute the image processing method according to claim 6.

Images