JP2018160746A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

Color noise is sufficiently reduced while retaining local color information so that a point light source or the like in a night view does not lose color.
An image processing apparatus that performs reduction processing on input image data to generate reduced image data of a plurality of hierarchies with reduced resolution, the input image data, and the reduced image data Correction means for correcting the pixel value of the pixel of interest to a value with reduced color noise, and the image data other than the reduced image data with the lowest resolution among the image data subjected to the correction other layers A composition ratio deriving unit for deriving a composition ratio with the image data, and among the image data subjected to the correction, an area to which each pixel belongs is local to the image data excluding reduced image data having the lowest resolution. Local chromaticity deriving means for deriving local chromaticity indicating the degree of color area, and attention is made among the image data subjected to the correction based on the composition ratio and the local chromaticity Characterized by comprising a larger synthesizing means for synthesizing the image data of the image data and its one of a hierarchy, a.
[Selection] Figure 8

Description

  The present invention relates to an image processing technique for reducing color noise.

  Imaging devices such as digital cameras require high-quality shot images, but in low-light random noise, such as low-frequency random noise, in a dark place or at night when a sufficient S / N ratio cannot be obtained. The image quality is greatly degraded by (color noise). Particularly in recent years, there is a high demand for high-sensitivity shooting, and it is desired that a high-quality shot image can be obtained with little noise even in a dark place or at night. Therefore, in order to suppress the color noise, a smoothing process using an averaging filter, a Gaussian filter, or the like, or a color noise reduction process using an order statistical filter such as a median filter is performed. However, when a smoothing process or order statistical filter is used, it is necessary to design the number of taps of the filter to be large enough to reduce a large range of noise (low frequency noise), which causes an increase in circuit scale. End up. In this regard, there has been proposed a technique for obtaining the same effect without increasing the number of taps by performing filter processing after reducing the input image (see Patent Document 1). In the method disclosed in Patent Document 1, first, a plurality of low resolution images having a resolution reduced stepwise is generated from an input image, and color noise suppression processing is performed on each of the input image and the low resolution image. Thereafter, the resolution of the low-resolution image is returned, and weighted composition is performed based on the composition ratio coefficient. In this case, when it is determined that the pixel of interest is a pixel belonging to an area with little color change, the composition ratio coefficient is determined so that the composition ratio of the image from the lower layer becomes high.

JP 2012-104968 A

  When color noise is reduced by using the method of Patent Document 1 on image data with a lot of noise, such as when shooting at high sensitivity, color noise is reduced in the input image itself or a low resolution image with a small reduction magnification. There is a problem that it remains without being sufficiently suppressed. This problem can be dealt with by increasing the weight of a low-resolution image having a large reduction ratio and compositing. However, this time another problem of color loss, specifically, local color information is lost as the reduction ratio increases, and local color information such as a night scene point light source is lost in the synthesized image. The problem will occur. Therefore, an object of the present invention is to sufficiently reduce color noise while retaining local color information so as not to lose color.

  An image processing apparatus according to the present invention performs reduction processing on input image data to generate reduced image data of a plurality of hierarchies with reduced resolution, and attention in the input image data and the reduced image data Correction means for correcting the pixel value of a pixel to a value with reduced color noise, and image data other than the reduced image data with the lowest resolution among the image data subjected to the correction, images in other layers A composition ratio deriving unit for deriving a composition ratio with the data, and among the image data subjected to the correction, an area to which each pixel belongs is a local color with respect to image data excluding reduced image data having the lowest resolution. Local chromaticity deriving means for deriving local chromaticity indicating the degree of the area, and based on the composition ratio and the local chromaticity, the image data of interest among the corrected image data Data and the enlarged synthesizing means for synthesizing the image data of the one of a hierarchy, and further comprising a.

  According to the present invention, it is possible to sufficiently reduce color noise while retaining local color information.

It is a figure which shows an example of the hardware constitutions of an image processing apparatus. (A) is a block diagram showing an example of a logical configuration related to noise reduction processing of the image processing apparatus, and (b) is a block diagram showing an internal configuration of a color noise reduction processing unit. FIG. 3 is a sequence diagram illustrating a flow of the entire processing realized by the logical configuration illustrated in FIG. It is a figure which shows the specific example of the filter used for a low-pass filter process. It is a figure which shows the specific example of how to apply the low-pass filter with respect to the attention pixel. It is a flowchart which shows the flow of each process in a color noise reduction process. It is a flowchart which shows the flow of a resolution reduction process. 6 is a flowchart illustrating a flow of an enlargement / combination process according to the first embodiment. FIG. 10 is a block diagram illustrating an internal configuration of a color noise reduction processing unit in Embodiment 2. It is a figure explaining similarity determination of hue using a formula (10)-a formula (12). 12 is a flowchart illustrating a flow of an enlargement / combination process according to the second embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

  FIG. 1 is a diagram illustrating an example of a hardware configuration of the image processing apparatus according to the present embodiment. The image processing apparatus 100 is a PC, for example, and includes a CPU 101, a RAM 102, an HDD 103, a general-purpose interface (I / F) 104, a monitor 108, and a main bus 109. The general-purpose I / F 104 connects an imaging device 105 such as a camera, an input device 106 such as a mouse and keyboard, and an external memory 107 such as a memory card to the main bus 109.

  The CPU 101 implements various processes as described below by operating various software (computer programs) stored in the HDD 103. First, the CPU 101 activates an image processing application stored in the HDD 103, expands it in the RAM 102, and displays a user interface (UI) on the monitor 108. Subsequently, various data stored in the HDD 103 and the external memory 107, image data acquired by the imaging device 105, user instructions from the input device 106, and the like are transferred to the RAM 102. Furthermore, according to the processing in the image processing application, the data stored in the RAM 102 is arithmetically processed based on a command from the CPU 101. The result of the arithmetic processing is displayed on the monitor 108 or stored in the HDD 103 or the external memory 107. Note that image data stored in the HDD 103 or the external memory 107 may be transferred to the RAM 102. Further, image data transmitted from the server via a network (not shown) may be transferred to the RAM 102.

  In the present embodiment, a description will be given of an aspect in which, in the image processing apparatus 100 having the above-described configuration, based on a command from the CPU 101, image data is input to an image processing application and image data with reduced color noise is output. It shall be.

(Logical configuration of image processing device)
FIG. 2A is a block diagram illustrating an example of a logical configuration related to noise reduction processing of the image processing apparatus 100. The image processing apparatus 100 includes a signal distribution unit 201, a color noise reduction processing unit 202, a luminance noise reduction processing unit 203, and a signal integration processing unit 204.

  A color signal of color image data (a color signal expressed in RGB color space) input from the imaging device 105, HDD 103, or external memory 107 is first input to the signal conversion processing unit 201. Here, the input color image data may be intermediate image data generated in the imaging device 105. For example, RAW image data, which is original image data obtained by digitizing an image captured by an image sensor, or RAW image data after correcting defective pixels by correcting defective pixels may be used. Furthermore, RAW image data after various correction processes such as false color suppression and low-pass filter processing may be used. When inputting a color signal of RAW image data, an RGB color signal may be generated using a known demosaic technique. In addition, in the case of RAW image data having a Bayer array, a method of handling 2 × 2 pixels (RGGB) as one pixel is also conceivable. Although the description will be made on the assumption that the color image data in the present embodiment is an RGB color space, the present invention is not limited to this. For example, the color image data may be converted into an L * a * b * color space.

  The signal conversion processing unit 201 generates a luminance signal representing the luminance component (Y) and a color difference signal representing the color difference components (Cr and Cb) from a RGB color signal of the input color image data by a known conversion formula. Here, the spatial frequency of the noise of the luminance signal (hereinafter referred to as luminance noise) and the spatial frequency of the noise of the color difference signal (hereinafter referred to as color noise) are different from each other, and the spatial frequency of the noise of the color difference signal is lower. By generating a YCrCb color signal from the RGB color signal related to the input color image data by this processing, optimum noise reduction processing is performed for each of the luminance signal representing the luminance component and the color difference signal representing the color difference component. be able to. The input RGB color signal and the YCrCb color signal generated by the signal conversion process are sent to the color noise reduction processing unit 202 and the luminance noise reduction processing unit 203.

  The color noise reduction processing unit 202 performs a process of reducing noise in the color difference signal based on the RGB color signal and the YCrCb color signal. Details of the color noise reduction processing will be described later. The luminance noise reduction processing unit 203 performs processing for reducing noise in the luminance signal. The luminance noise reduction processing unit 203 may use a general noise reduction process.

  The signal integration processing unit 204 performs processing for integrating the color signal whose color noise has been reduced by the color noise reduction processing unit 202 and the color signal whose luminance noise has been reduced by the luminance noise reduction processing unit 203. The signal integration processing unit 204 outputs color image data in which both color noise and luminance noise are reduced, which is generated as a result of the signal integration processing. The integrated color image data is output to the monitor 108, HDD 103, or the like. In addition, for example, the data may be output to the external memory 107 connected to the general-purpose I / F 104, an external server (not shown), a printer, or the like.

(Details of color noise reduction processing section)
Next, the color noise reduction processing unit 202 will be described in detail. FIG. 2B is a block diagram illustrating an internal configuration of the color noise reduction processing unit 202 according to the present embodiment. As shown in FIG. 2B, the color noise reduction processing unit 202 includes a reduction processing unit 210, a correction unit 220, a composition ratio derivation unit 230, a local chromaticity derivation unit 240, and an enlargement / combination processing unit 250. . FIG. 3 is a sequence diagram showing the flow of the entire process realized by the logical configuration shown in FIG. Hereinafter, each unit of the color noise reduction processing unit 202 will be described with reference to FIGS. 2B and 3.

<Reduction processing unit>
The reduction processing unit 210 performs processing (reduction processing 302a to 302d in FIG. 3) for reducing the resolution on the input image data, and generates reduced image data. In the example shown in FIG. 3, the resolution of the input image is reduced to 1/2 times, the reduction processing 302a to be ¼ times, the reduction processing 302b to be ¼ times, the reduction processing 302c to be ８ times, and 1/16 times, respectively. A reduction process 302d is performed to generate four types of reduced image data having different resolutions. The type of reduced image data is not limited to this example, and may be limited to 1/8 times, or further reduced to 1/32 times. In general, when reducing image data, aliasing noise occurs unless low-pass filter processing is performed, and the aliasing noise pattern appears in the finally output image data. Therefore, when performing the reduction process, for example, an algorithm including an average pixel method or other low-pass filter processing is used, or a bi-linear method or the like is applied after performing low-pass filter processing in advance. The filter used for the low-pass filter processing is determined based on the reduction magnification. For example, when the reduction magnification is 1/2, a filter as shown in FIG. 4 is used. Note that the size and coefficient of the filter are not limited to the filters shown in FIG. Hereinafter, input image data and a plurality of reduced image data generated from the input image data are collectively referred to as “multi-resolution image data”. The reduced image data generated by the reduction process is stored in the RAM 102 for use in the correction process described later.

<Correction unit>
The correction unit 220 corrects the color signal of the pixel of interest so as to reduce the color noise on the above-described multi-resolution image data based on preset correction parameters (correction processes 303a to 303e in FIG. 3). To do. Here, the correction parameter means a predetermined reference area for the target pixel and each threshold value for determining whether to average. It is desirable to change these to appropriate values according to the amount of noise. Therefore, for example, it is determined in advance according to the photographing sensitivity, or is adaptively determined based on information on the target pixel and its surrounding pixels. In the correction processing, the value of the color signal (RGB signal here) of the target pixel of the input color image data is corrected for each color component using, for example, the following formulas (1) to (3).

  In the above formulas (1) to (3), R [i], G [i], and B [i] represent pixels that satisfy the following formula (4) in a predetermined reference region for the target pixel. That is, for the target pixel, the smoothing process is performed using only the pixel that satisfies the condition of the following expression (4) in the predetermined reference region. The predetermined reference area includes a target pixel and adjacent pixels surrounding the target pixel. For example, the predetermined reference area may be a 3 × 3 pixel area or a larger 10 × 10 pixel area. M is the number of pixels that satisfy the following expression (4) in the reference region for the target pixel. Note that since the pixel of interest itself is always included, M ≧ 1.

In the above formula (4), ∧ represents the logical product (the AND) (hereinafter, the same), Th _r represents the threshold for the R signal, Th _g is a threshold for the G signal, Th _b is a threshold value for B signals, respectively . R _diff , G _diff , and B _diff represent the difference between each pixel value in the reference area and the pixel value to be compared, and are expressed by the following equations (5) to (7), respectively.

For the pixel values R _comp , G _comp , and B _comp to be compared in the above formulas (5) to (7), the pixel value of the target pixel may be used as it is. Alternatively, the pixel value of the target pixel may be a result of applying a low pass filter to the pixel value of the target pixel. FIG. 5 is a diagram illustrating a specific example of how to apply a low-pass filter to a target pixel. In the example of FIG. 5, a result of applying a low pass filter to the pixel value of the target pixel as “pixel values to be compared R _comp , G _comp , B _comp ” using the target pixel and four pixels adjacent in the vertical and horizontal directions. Is derived. If there is an effect similar to that of the low-pass filter, the weighting and the number of adjacent pixels to be used may be freely determined.

  Note that the correction processing shown here is merely an example, and any other method may be used as long as it is a method of correcting the color signal value so that the color noise is reduced. The multi-resolution image data in which the color noise has been reduced by the correction process is stored in the RAM 102 for use in a synthesis ratio derivation process and a local chromaticity derivation process described later.

<Composition ratio deriving section>
The composition ratio deriving unit 230 derives a composition ratio with the lower layer image data for each image data included in the multi-resolution image data whose color noise has been reduced by the correction process (composition ratio derivation process 304a in FIG. 3). To 304d). However, as is apparent from the sequence diagram of FIG. 3, the image data in the lowest hierarchy (reduced image data reduced to 1/16 times in this embodiment) has no lower-order image data. Not subject to this process. Here, the combination ratio is determined according to the degree of abrupt change in the color signal. In the case of the present embodiment, the composition ratio can be derived by performing edge detection using the RGB signal of the target pixel as an input. Specifically, for example, using a spatial filter having a filter coefficient [−1, 2, −1] and applying the filter in the horizontal direction and the vertical direction, the one that is considered to have a high degree of color edge, Mapping to a coefficient K between 0 and 1 by a predetermined function. A coefficient K having a value closer to 1 is output for a filter application result considered to have a higher degree of color edge, and a coefficient K having a value closer to 0 is output to a filter application result having a lower degree of color edge. This coefficient K becomes the composition ratio.

  Note that a luminance signal or a color difference signal may be used for edge detection. Further, the filter coefficient and the determination method are not limited to those described above. If a combination ratio of a large value is obtained when the degree of color edge is considered to be high and a combination ratio of a small value is obtained when the degree of color edge is considered to be low, it may be derived in any way. The derived composition ratio data is stored in the RAM 102 for use in the enlargement composition process described later.

<Local chromaticity deriving unit>
The local chromaticity deriving unit 240 is a process for deriving local chromaticity for each pixel for each image data excluding the lowest layer among the multi-resolution image data processed by the correcting unit 220 (local chromaticity in FIG. 3). Calculation processes 305a to 305d) are performed. Here, the local chromaticity means a degree indicating that the region to which the pixel belongs is a local color region having a color different from the surrounding color such as a point light source. In the present embodiment, the number of pixels actually referred to in the correction process (smoothing process) (hereinafter referred to as “actual reference pixels”), that is, the number of pixels satisfying the above-described expression (4), Local chromaticity is assumed. Therefore, in this embodiment, the local chromaticity is defined as the number of actual reference pixels.

  Here, the actual reference pixel number is the number of pixels used in the target pixel correction process among the pixels constituting the predetermined reference region, and therefore, the number of pixels similar to the target pixel existing around the target pixel. It can be said that it represents a number. Therefore, the local chromaticity defined as described above becomes large when there are many similar pixels around the pixel of interest such as a flat portion where the color is uniform, and around the pixel of interest like a point light source. It becomes smaller when similar pixels are hardly found. That is, the local chromaticity becomes so small that the target pixel is considered to be a pixel constituting a local color region such as a point light source. In this embodiment, the number of actual reference pixels actually used in the smoothing process is the local chromaticity, but the present invention is not limited to this. Any evaluation index may be used as long as the pixel of interest is a pixel constituting a local color region. The derived local chromaticity information is stored in the RAM 102 for use in the priority use determination process described later.

<Enlarged synthesis processing unit>
The enlargement / combination processing unit 250 derives two types of image data having different resolutions from the multi-resolution image data processed by the correction unit 220 using the combination ratio derived by the combination ratio deriving unit 230 and the local chromaticity deriving unit 240. Based on the obtained local chromaticity, a composition process is performed. The enlargement / combination processing unit 250 further includes an enlargement processing unit 251, a priority use determination unit 252, and a pixel value composition unit 253. Hereinafter, each part which comprises the expansion synthetic | combination process part 250 is demonstrated.

≪Enlargement processing part≫
The enlargement processing unit 251 performs processing for enlarging the image data of the highest layer in the multi-resolution image data (input image data itself. The image data of each layer excluding the input image 301 in FIG. In this case, the enlargement process is performed on the image data processed by the correction unit 220 in the lowest hierarchy, and the pixel value synthesis process described later in the other lower hierarchy. In the example of the sequence diagram of Fig. 3, the enlargement process 306a is the lowest level image data that has been subjected to the correction process 303e, and the enlargement process 306b is the object. The resultant image data has been subjected to pixel value synthesis processing 308a, which will be described later, and the target of the enlargement processing 306c is the pixel value described later. The combined image data is subjected to the composition processing 308b, and the target of the enlargement processing 306d is the combined image data subjected to the pixel value composition processing 308c described later. This method is not limited to the bilinear method, and for example, the nearest neighbor method, the bicubic method, the Lanczos method, or the like may be used.

≪Priority use judgment part≫
The priority use determination unit 252 determines whether to use image data in the upper layer preferentially when combining the target image data and the image data in the next lower layer (in FIG. 3). Preferential use determination processing 307a to 307d) is performed. The priority use determination in the present embodiment is made based on the local chromaticity derived by the local chromaticity deriving unit 240. Here, the “image data in the next lower layer” is image data having a resolution one step lower than the target image data, and is enlarged by the enlargement processing unit 251 to the same resolution as the target image data. Refers to image data. An example of the sequence diagram in FIG. 3 will be described.

  In the priority use determination process 307a, the image data reduced to 1/8 resolution of the input image data and the resolution reduced to 1/16 resolution of the input image data (the same resolution was obtained by the enlargement process). A determination is made when combining with image data. In the preferential use determination process 307b, the image data reduced to 1/4 resolution of the input image data and the resolution reduced to 1/8 resolution of the input image data (the same resolution is obtained by the enlargement process). A) A determination is made when combining with the image data. In the priority use determination processing 307c, the image data reduced to 1/2 the resolution of the input image data and the resolution reduced to 1/4 the resolution of the input image data (the same resolution is obtained by the enlargement processing). A) A determination is made when combining with the image data. In the preferential use determination process 307d, determination is performed when the input image data itself is combined with the image data that has been reduced to half the resolution of the input image data (having the same resolution as a result of the enlargement process). Is made.

  Then, this preferential use determination process is performed by fitting the local chromaticity of the image data of interest on the upper layer side in the following equation (8) in units of pixels.

In the above formula (8), th _deg is a threshold value set based on a local color region (subject) to be left in the shooting scene. For example, this threshold value is adjusted according to the assumed size of the local color region, such as increasing th _deg when it is desired to prevent color loss of a point light source having a relatively large size. This threshold value th _deg is determined and held in advance based on the ISO sensitivity (noise amount) of the camera. Alternatively, different threshold values may be applied depending on the mode selection according to the shooting scene, such as a larger value than the normal mode in the night view mode. Note that the priority use determination based on the above formula (8) is performed only on image data (input image data or image data reduced to 1/2 to 1/4 of its resolution) having a relatively higher hierarchy than a predetermined resolution. You may apply to. This is because local color information such as a point light source exists in the upper image data. When the condition of the above equation (8) is satisfied, the composition ratio is changed so that the ratio of the image data of interest on the upper layer side in the pixel value composition processing is increased (for example, 100% is adopted). become. The changed composition ratio data is stored in the RAM 102.

≪Pixel value composition part≫
The pixel value composition unit 253 calculates the pixel value of the image data of interest and the pixel value of the image data (the same resolution as the image data of interest) after enlarging the image data of the next lower layer at each stage. A synthesis process is performed based on the synthesis ratio derived by the synthesis ratio derivation process. In FIG. 3, pixel value synthesis processing 308a to 308d corresponds to this. Specifically, the combined pixel value Ipost is obtained by using the following formula (9), where Idown is the pixel value on the lower layer side, Iup is the pixel value on the upper layer side, and u is the combination ratio. .

For example, when the composition ratio u = 0.1, the lower layer pixel value I _down = 3405, and the upper layer pixel value I _up = 3621, the combined pixel value I _post is “3427” from the above equation (9). Become. Note that the pixel values here may be in any format, YcrCb values, or RGB values.

  As described above, the composition ratio u is a coefficient representing how much higher-layer image data is used, and is basically derived by the composition ratio deriving unit 230 and appropriately changed depending on the result of the preferential use determination process. It will be. In the case of the present embodiment, the color image data after synthesis, which is the result of the pixel value synthesis processing 308a to 308c, is temporarily stored in the RAM 102 and provided to the enlargement processing 306b to 306d, respectively. On the other hand, the combined image data that is the result of the pixel value combining processing 308d is output as it is (see the sequence diagram of FIG. 3). The output mode at this time includes, for example, display on the monitor 108, storage on the HDD 103, storage on the external memory 107 or an external server (not shown) via the general-purpose I / F 104, and printing on a printer (not shown). There is something.

(Color noise reduction processing flow)
Next, a rough flow of each process in the color noise reduction processing unit 202 will be described. FIG. 6 is a flowchart showing the flow of each process in the color noise reduction process. This series of processing is realized by the CPU 101 loading a program stored in the HDD 130 into the RAM 102 and executing it.

  When image data is input to the color noise reduction processing unit 202, in step 601, the reduction processing unit 210 performs the above-described reduction processing on the acquired input image data to generate multi-resolution image data. Details of the reduction processing will be described later.

  In step 602, the correction unit 220 performs a correction process for correcting the color signal of the pixel of interest so that the color noise is reduced with respect to the image data of each layer in the generated multi-resolution image data.

  In step 603, the composition ratio deriving unit 230 derives a composition ratio with the image data of the lower layer for each image data excluding the lowest layer among the multi-resolution image data in which the color noise is reduced by the correction process. Perform processing.

  In step 604, the local chromaticity deriving unit 240 performs a process of deriving local chromaticity for each image data on a specific upper layer side among the multi-resolution image data in which the color noise is reduced by the correction process.

  In step 605, the enlargement / combination processing unit 250 performs, based on the combination ratio derived in step 603 and the local chromaticity derived in step 604, for the multi-resolution image data in which color noise has been reduced by the correction process. The above enlargement / synthesizing process is performed. Details of the enlargement / synthesis processing will be described later.

  The above is a rough flow of each process in the color noise reduction processing unit 202.

(Reduction processing flow)
Next, details of the reduction process (step 601) in the flow of FIG. 6 described above will be described. FIG. 7 is a flowchart showing the flow of the reduction process.

  In step 701, the reduction processing unit 210 acquires a reduction processing parameter. The reduction processing parameter includes information on the depth of the multi-resolution layer and the minimum resolution that is a fraction (a multiple of 2) of the input image data. Such reduction processing parameters are set in advance and stored in the HDD 103 or the like. In the case of this embodiment, the minimum resolution is 1/16 of the input image data in all five layers (1 ×, 1/2 ×, 1/4 ×, 1/8 ×, 1/16 ×). The content reduction processing parameter is set and saved in advance, and is acquired by reading it from the HDD 103 or the like.

  In step 702, the reduction processing unit 210 determines a magnification to be applied to the reduction processing based on the reduction processing parameter acquired in step 701. In the case of the present embodiment in which the contents of the reduction processing parameters are all five layers and the minimum resolution is 1/16 of the input image data, there are four types of 1/2, 1/4, 1/8, and 1/16. Are sequentially determined.

  In step 703, the reduction processing unit 210 performs processing for reducing the image data to be processed to the magnification determined in step 702.

  In step 704, the reduction processing unit 210 determines whether all reduction processing based on the reduction processing parameters acquired in step 701 has been completed. In the case of the present embodiment, it is determined whether or not the reduction processing has been completed for the four types of magnifications of 1/2, 1/4, 1/8, and 1/16. If all the reduction processes corresponding to the acquired reduction process parameters have been completed as a result of the determination, this process ends. On the other hand, if there is an unprocessed magnification, the process returns to step 702 to determine the next magnification and continue the reduction process.

(Expansion synthesis processing flow)
Next, details of the enlargement / synthesis processing (step 605) in the flow of FIG. 6 described above will be described. FIG. 8 is a flowchart illustrating the flow of the enlargement / synthesizing process according to the present embodiment.

  In step 801, the enlargement / synthesis processing unit 250 acquires all image data (multi-resolution image data) in which color noise has been reduced by the correction processing in step 602 described above.

  In step 802, the enlargement / synthesis processing unit 250 acquires the above-described reduction processing parameter and synthesis processing parameter. Here, the composition processing parameter means the composition ratio derived in step 603 described above and the local chromaticity derived in step 604 described above.

  In step 803, the enlargement processing unit 251 of the enlargement / combination processing unit 250 applies the above-described enlargement processing (this embodiment) to the image data of the lowest hierarchy (smaller resolution) among the plurality of image data acquired in step 801. Then double).

  In step 804, the enlargement / combination processing unit 250 sets image data having the minimum resolution among unprocessed image data among the plurality of image data acquired in step 801 as one image data to be subjected to the pixel value synthesis processing. select. In this case, the unprocessed image data does not include image data in the lowest hierarchy.

  In step 805, the pixel value composition unit 253 of the enlargement composition processing unit 250 selects a pixel to be subjected to pixel value composition processing. Specifically, in the case of the first routine, the pixel value composition processing target for the image data enlarged in step 803 and the image data selected in step 804 having the same resolution as the enlarged image data. Is selected. In the case of the second and subsequent routines, pixel value composition processing is performed on the image data enlarged in step 811 to be described later and the image data selected in step 804 having the same resolution as the enlarged image data. A target pixel is selected.

  In subsequent steps 806 and 807, the above-described priority use determination process for the pixel selected in step 805 is performed in the priority use determination unit 252 of the enlargement / synthesis processing unit 250. That is, it is determined whether to preferentially use the image data selected in Step 804 (the image data on the upper layer side) from the two image data to be subjected to the pixel value synthesis processing.

In step 806, it is determined using the above-described equation (8) whether image data on the upper layer side is used preferentially. If the local chromaticity on the upper layer side is smaller than the threshold value th _{deg as} a result of the determination, the process proceeds to step 807 to preferentially use the image data on the upper layer side. On the other hand, if the local chromaticity on the upper layer side is greater than or equal to the threshold th _deg , the process proceeds to step 808.

  In step 807, the composition ratio obtained in the composition ratio derivation process (step 603 described above) is changed to a composition ratio that employs, for example, 100% of the pixel value of the image data on the upper layer side.

In step 808, the pixel value composition unit 253 of the enlargement composition processing unit 250 performs the above-described pixel value composition processing based on the currently set composition ratio. Thereby, a pixel value _Ipost obtained by combining the selected pixel values in the two image data is obtained.

  In step 809, the enlargement / synthesis processing unit 250 determines whether the synthesis processing for all pixels has been completed. If processing for all the pixels has been completed, the process proceeds to step 810. On the other hand, if there is an unprocessed pixel, the process returns to step 805 to select the next pixel as a pixel to be processed and continue the process.

  In step 810, the enlargement / combination processing unit 250 has performed enlargement processing on all image data (that is, all reduced image data except input image data) other than the image data of the highest hierarchy in the multi-resolution image data. Determine if. As a result of the determination, if there is reduced image data that has not been enlarged, the process proceeds to step 811. On the other hand, if the enlargement process has been performed on all the reduced image data, this process ends.

  In step 811, the enlargement processing unit 251 of the enlargement / combination processing unit 250 performs the above-described enlargement processing (twice in the present embodiment) on the image data after the combining processing obtained in step 808. After executing the enlargement process, the process proceeds to step 804.

  The above is the content of the enlargement / synthesizing process according to the present embodiment.

  As described above, according to this embodiment, it is possible to sufficiently reduce color noise while maintaining local color information.

  In the first embodiment, the local color region such as a point light source is changed by changing the composition ratio so that the weight of the image data of interest on the upper layer side is increased at a location where the local chromaticity is lower than a predetermined threshold. It is possible to obtain an image in which residual color noise is effectively reduced while leaving However, if the weight of the image data on the upper layer side is increased at all locations where the local chromaticity is low, colorful color noise may occur. This is because it is impossible to distinguish whether the portion determined to be a local color area based on local chromaticity is actually due to a local subject or noise. Therefore, a second embodiment will be described in which the above-described adverse effects are prevented as much as possible by evaluating the similarity of colors between the upper layer and the lower layer. In the following description, the characteristic configuration of the present embodiment will be described in detail, and the description of the common parts with the first embodiment will be omitted or simplified.

(Details of color noise reduction processing section)
FIG. 9 is a block diagram illustrating an internal configuration of the color noise reduction processing unit 202 ′ according to the present embodiment. The color noise reduction processing unit 202 ′ includes a reduction processing unit 210, a correction unit 220, a composition ratio derivation unit 230, a local chromaticity derivation unit 240, an enlargement / combination processing unit 250 ′, and a color similarity determination unit 900.

  The enlargement / synthesis processing unit 250 ′ of the present embodiment adds the two types of image data having different resolutions of the multi-resolution image data processed by the correction unit 220 to the above-described color similarity determination in addition to the combination ratio and local chromaticity. The combining process is performed in consideration of the determination result in the unit 900.

  As in the first embodiment, the priority use determination unit 252 ′ of the present embodiment preferentially uses the image data of the upper layer when combining the image data of interest with the image data of the next lower layer. Processing for determining whether or not to perform (upper hierarchy priority determination processing 307a-307d in FIG. 3) is performed. The difference from the first embodiment is that the priority use determination process is performed based on both the local chromaticity derived by the local chromaticity deriving unit 240 and the determination result by the color similarity determining unit 900.

≪Color similarity judgment part≫
First, the color similarity determination unit 900 generates a color difference signal from the RGB color signal of the image data whose color noise has been reduced by the correction unit 220. The generation of the color difference signal is performed for the image data of interest and the image data of the next lower layer that has been enlarged to the same resolution as the image data of interest by the enlargement process. The color difference signals (Cr and Cb) can be generated by a known conversion formula as described above. Further, Cr may be obtained by RG and Cb may be obtained by BG. The generated color difference signal is stored in the RAM 102.

  Next, the color similarity determination unit 900 is based on the color difference signal of the image data of interest and the color difference signal of the image data that is one level below the image data and is enlarged to the same resolution as the image data of interest. Thus, the color similarity between the upper layer side and the lower layer side is determined for the target pixel. In this determination, rough color similarity is evaluated. The reason is that if it is strictly evaluated, a local subject such as a point light source that wants to leave the color area will not be determined. On the other hand, if it is not evaluated at all, the noise will be determined as a local color area and colorful. This is because a large color noise is generated. Specifically, for example, the following equation (10) is used.

In the above equation (10), Cr _down and Cb _down represent the color difference signals on the lower layer side, and Cb _up and Cr _up represent the color difference signals on the upper layer side. This equation (10) evaluates the similarity of colors depending on whether the result of the polar coordinate conversion of the color space of the color difference signal is within the same specific range on the lower layer side and the upper layer side. . That is, when Cr and Cb are regarded as vectors on the xy coordinate plane, it is evaluated whether or not the quadrant to which the conversion result on the upper hierarchy side matches the quadrant to which the conversion result on the lower hierarchy side matches (FIG. 10). See). Thereby, it is possible to determine the rough similarity of the hues of the color signals between the upper layer side and the lower layer side.

  Alternatively, the following formulas (11) to (13) may be used instead of the above formula (10).

The above equations (11) and (12) mean that coring is performed when the value of each color difference signal on the lower layer side is equal to or less than a predetermined value close to 0, and the result of _{coring is expressed} as Cr _downCMP , It is represented by Cb _downCMP . As a result, when the signal value that is likely to be erroneously determined due to the influence of noise is near zero, there is an aim of exceptionally excluding it from the target of the color similarity determination based on the above equation (13).

  The determination method is not limited to these examples as long as the similarity of the hues of the color signal on the lower layer side and the color signal on the upper layer side can be evaluated. For example, the similarity may be determined based on whether the distance between the lower layer side conversion result and the upper layer side conversion result is within a predetermined threshold. In the case of this determination method, even if the result of the polar coordinate conversion of the color space of the color difference signal is located near the intersection 1000 of the Cr axis and the Cb axis, the quadrants do not match between the lower layer side and the upper layer side. If the distance between the two is short, it can be determined that they are similar. And when judging the similarity of a color by distance in this way, the misjudgment prevention by the above-mentioned coring is considered simultaneously. The color similarity determination result obtained in this manner is stored in the RAM 102 for use in the priority usage determination process in the priority usage determination unit 252 '.

≪Priority use judgment process≫
The priority use determination unit 252 ′ determines whether to use image data in the upper layer preferentially, the local chromaticity derived by the local chromaticity derivation unit 240, and the color similarity in the color similarity determination unit 900 described above. The determination is made based on the determination result. Specifically, first, as in the case of the first embodiment, a comparison process between the local chromaticity on the upper layer side and a predetermined threshold value th _deg is performed. As a result of the threshold comparison process, if the local chromaticity on the upper layer side is smaller than the threshold th _deg , then, referring to the result of the color similarity determination described above, the colors on the upper layer side and the lower layer side are similar. If so, the composition ratio is changed so as to increase the ratio of the upper layer side during image composition.

  By adding the color similarity condition as described above, it is possible to increase the composition ratio on the upper layer side by limiting to the portions where the colors are somewhat similar on the lower layer side and the upper layer side. The changed composition ratio data is stored in the RAM 102.

(Expansion synthesis processing flow)
Next, details of the enlargement / synthesizing process (step 605) of this embodiment will be described. FIG. 11 is a flowchart illustrating the flow of the enlargement / synthesizing process according to the present embodiment.

  Steps 1101 to 1106 correspond to steps 801 to 806 in the flow of FIG. That is, multi-resolution image data with reduced color noise is acquired (step 1101), and then reduction processing parameters and synthesis processing parameters are acquired (step 1102). In this case, the synthesis processing parameters of this embodiment include the result of color similarity determination in addition to the synthesis ratio and local chromaticity. Then, enlargement processing is performed on the image data of the lowest hierarchy in the acquired multi-resolution image data (step 1103). Subsequently, the image data with the minimum resolution among the unprocessed image data among the acquired multi-resolution image data is selected as one image data to be subjected to the pixel value synthesis process (step 1104). When image data is selected, a pixel to be subjected to pixel value synthesis processing is selected (step 1105).

  In subsequent steps 1106 to 1108, the above-described priority use determination process for the pixel selected in step 1105 is performed in the priority use determination unit 252 ′ of the enlargement / synthesis processing unit 250 ′.

First, in step 1106, it is determined whether or not the above equation (9) is satisfied. As a result of the determination, if the location chromaticity on the upper layer side is smaller than the threshold value th _deg , the process proceeds to step 1107. On the other hand, if the local chromaticity on the upper layer side is greater than or equal to the threshold th _deg , the process proceeds to step 1109.

  In step 1107, 252 ′ refers to the result of the color similarity determination included in the synthesis processing parameter acquired in step 1102, and performs processing separation. Specifically, if it is a determination result that there is color similarity between the upper layer side and the lower layer side, the process proceeds to step 1108 to preferentially use the image data on the upper layer side. On the other hand, if it is determined that there is no color similarity, the process proceeds to step 1109. In step 1108, the composition ratio obtained in the composition ratio derivation process (step 603 described above) is changed to a composition ratio that employs, for example, 100% of the pixel value of the image data on the upper layer side.

  The subsequent steps 1109 to 1112 correspond to the steps 808 to 811 of the flow of FIG. 8 and are not different from each other, and will not be described.

  The above is the content of the enlargement / synthesizing process according to the present embodiment. Here, the color similarity determination unit 900 is an independent component, and the determination result is referred to by the priority use determination unit 252 '. However, the present embodiment is not limited to this. For example, the function of the color similarity determination unit 900 may be incorporated in the priority use determination unit 252 '.

  As described above, in this embodiment, it is possible to increase the composition ratio on the upper layer side by limiting to places where the colors are somewhat similar on the lower layer side and the upper layer side. As a result, it is possible to obtain an image that retains local color information such as a point light source of a night scene while suppressing the occurrence of colorful color noise and effectively reducing the color noise.

<Other examples>
In the first and second embodiments, an example in which processing is performed by an image processing application has been described. However, these may be a method of processing image data captured by an imaging device on image processing hardware in the imaging device. Absent. Further, the image data may be processed on the server device by transmitting the image data from the client device to the image processing application on the server device.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (14)

Reduction processing means for performing reduction processing on the input image data to generate reduced image data of a plurality of layers with reduced resolution;
Correction means for correcting the pixel value of the pixel of interest in the input image data and the reduced image data to a value with reduced color noise;
Composition ratio deriving means for deriving a composition ratio with image data of other layers for image data excluding reduced image data having the lowest resolution among the image data subjected to the correction;
Of the corrected image data, local color for deriving local chromaticity indicating the degree to which the area to which each pixel belongs is a local color area for image data excluding reduced image data with the lowest resolution Degree derivation means;
Based on the synthesis ratio and the local chromaticity, an enlargement synthesis unit that synthesizes the image data of interest among the image data subjected to the correction and the image data of the layer below it,
An image processing apparatus comprising: The synthesis means includes
Enlargement processing means for enlarging the image data;
Whether to use the image data of interest preferentially when combining the image data of interest and the image data of the next lower layer is determined based on the local chromaticity, and preferentially the When it is determined that the image data of interest is used, priority use determination means for changing the composition ratio so as to increase the weight of the image data of interest;
The pixel value of the image data of interest and the pixel value of the image data which is image data of the next lower layer and enlarged to the same resolution as the image data of interest by the enlargement processing unit Pixel value synthesizing means for synthesizing based on
The image processing apparatus according to claim 1, further comprising:   The preferential use determining means determines that the target image data is preferentially used when the local chromaticity calculated for the target image data is smaller than a predetermined threshold. 2. The image processing apparatus according to 2.   The image processing apparatus according to claim 3, wherein the predetermined threshold value is set to a larger value as the size of the assumed local color region is larger.   5. The image processing apparatus according to claim 2, wherein the preferential use determination unit performs the determination using image data having a resolution equal to or higher than a predetermined resolution as the image data to be noticed. The color difference signal of the pixel of interest of the image data of interest, and the color difference signal of the pixel of interest of the image data that is the image data one layer below it and enlarged to the same resolution as the image data of interest by the enlargement process Further comprising color similarity determination means for determining the color similarity of
The preferential use determination unit determines to use the image data of interest preferentially when the local chromaticity is smaller than the predetermined threshold and the color similarity determination unit determines that the colors are similar. The image processing apparatus according to claim 3, wherein the image processing apparatus is an image processing apparatus.   The color similarity determination means, when the result of the polar coordinate conversion of the color space of the color difference signals on the lower layer side and the upper layer side is within the same specific range on the lower layer side and the upper layer side, the colors are similar The image processing apparatus according to claim 6, wherein the determination is performed.   The color similarity determination means is that the colors are similar when the distance between the lower layer side and the upper layer side is within a predetermined threshold as a result of the polar coordinate conversion of the color space of the color difference signals on the lower layer side and the upper layer side. The image processing apparatus according to claim 6, wherein the determination is performed.   5. The color similarity determination unit according to claim 2, wherein the color similarity determination unit does not determine the color similarity when the value of the color difference signal in the next lower layer is equal to or less than a predetermined value. The image processing apparatus according to item 1. The correction in the correction means is a smoothing process using a reference region composed of adjacent pixels surrounding the target pixel,
The local chromaticity derived by the local chromaticity deriving unit is represented by the number of pixels actually referred to in the smoothing process among the pixels constituting the reference region. The image processing apparatus according to any one of claims 1 to 9. The input image data is image data having RGB color components,
The correction unit corrects the value of the color signal of the target pixel for each color component using Expressions (1) to (3),

In the equations (1) to (3), R [i], G [i], and B [i] represent pixels that satisfy the equation (4) in the reference region for the pixel of interest, and M is the pixel of interest. Represents the number of pixels satisfying the equation (4) in the reference region for

In Equation (4), ∧ represents a logical product (AND), Th _r represents a threshold for the R signal, Th _g represents a threshold for the G signal, Th _b represents a threshold for the B signal, and R _diff , G _diff , B _diff represents a difference between each pixel value in the reference region and a pixel value to be compared.
The image processing apparatus according to claim 10. The pixel value synthesizing means obtains a pixel value after synthesis using the following equation (5),

In the equation (5), I _post represents the pixel value after synthesis, I _down represents the pixel value of the image data on the lower layer side, I _up represents the pixel value on the upper layer side, and u represents the pixel value on the upper layer side. Representing a coefficient as the composition ratio representing how much image data is used,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Performing a reduction process on the input image data to generate reduced image data of a plurality of layers with a reduced resolution;
Correcting the pixel value of the pixel of interest in the input image data and the reduced image data to a value with reduced color noise;
Deriving a composition ratio with image data of other layers for image data excluding reduced image data with the lowest resolution among the image data subjected to the correction;
Deriving local chromaticity indicating the degree to which the area to which each pixel belongs is a local color area for image data excluding reduced image data with the lowest resolution among the corrected image data; ,
Based on the composition ratio and the local chromaticity, synthesizing the image data of interest among the image data subjected to the correction and the image data of a layer below it;
An image processing method comprising:   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 12.

Images