TWI659396B - Method and image processing device for image reconstruction in multimodal noise suppression - Google Patents

Abstract

An image reconstruction method and an image processing device for suppressing multi-mode noise. The method includes the following steps. First, an input image including a plurality of pixels is received, and the pixels include a plurality of damaged pixels and a plurality of clean pixels. The input image is then input to an image reconstruction model to obtain a reconstructed image. The image reconstruction model is associated with the global environment, regional environment, related environment, and three coefficients of each pixel. The global environment is associated with all clean pixels and regional environment. Blocks made up of neighboring pixels that are related to each pixel. The related environment is related to the clusters of similar pixels of each pixel. These coefficients are related to the global environment, regional environment, and related environment and are related to the image. The training phase of the reconstructed model is obtained by a semi-supervised learning method.

Description

Image reconstruction method and image processing device for suppressing multi-mode noise

The invention relates to an image reconstruction method and an image processing device thereof, and in particular to an image reconstruction method and an image processing device for suppressing multi-mode noise.

Interferences in image acquisition such as transmission errors, bit errors, and noise sensing tend to cause impulse noise (IN) in the acquired images and cause poor visual effects. Suppose a clean image without noise can be represented as , When the first Pixels Damage due to impulse noise can be expressed by equation (1): among them , First Pulse noise at several locations, The probability of noise damage. Generally speaking, taking an 8-bit grayscale image as an example, the pixel value of a pixel damaged by salt and pepper noise (SPN) is equal to the maximum value (ie, 255) or the minimum value (ie, 0).

A known way to remove salt and pepper noise from an image is to first detect it using a binary mask represented by equation (2), for example: among them When it is 0, it means Of pixels are damaged by salt and pepper noise, and When it is 1, the representative is located The pixels are no noise. However, this method cannot effectively remove noise in highly damaged images (that is, the proportion of noise is higher than 80%).

Another conventional method is to use matrix factorization to decompose the image into content components and noise components, thereby eliminating noise in the image. However, this method does not take into account the correlation between image blocks and causes errors.

In view of this, the present invention provides an image reconstruction method and an image processing device for suppressing multi-mode noise, which can effectively repair a damaged image with high-density noise to reconstruct an image with good visual effects. .

In an embodiment of the present invention, the image reconstruction method is suitable for an image processing apparatus and includes the following steps. First, an input image is received, where the input image includes a plurality of pixels, and the pixels include a plurality of damaged pixels with noise and a plurality of clean pixels without noise. Then, the input image is input to the image reconstruction model to obtain the reconstructed image output by the image reconstruction model. The image reconstruction model is associated with the global environment, regional environment, related environment, first coefficient, second coefficient, and Three coefficients, the global environment is related to all clean pixels, the regional environment is related to the block made up of neighboring pixels of each pixel, and the related environment is related to the cluster of similar pixels of each pixel. The first coefficient, The second coefficient and the third coefficient are respectively related to the global environment, the regional environment, and the related environment and are obtained by the semi-supervised learning method during the training phase of the image reconstruction model.

In an embodiment of the present invention, the image processing apparatus includes a memory and a processor, wherein the processor is coupled to the memory. The memory is used to store data and images. The processor is configured to receive an input image, input the input image to an image reconstruction model, and obtain a reconstructed image output from the image reconstruction model. The input image includes multiple pixels, and the pixels include multiple damaged pixels with noise. And multiple clean pixels without noise, the image reconstruction model is related to the global environment, regional environment, related environment, first coefficient, second coefficient, and third coefficient of each pixel, and the global environment is related to all clean pixels. The regional environment is related to the block formed by the neighboring pixels of each pixel. The related environment is related to the cluster of similar pixels of each pixel. The first coefficient, the second coefficient, and the third coefficient are related to the global environment. , Regional environment and related environment and obtained during the training phase of the image reconstruction model by a semi-supervised learning method.

In order to make the above features and advantages of the present invention more comprehensible, embodiments are hereinafter described in detail with reference to the accompanying drawings.

FIG. 1 is a block diagram of an image processing apparatus according to an embodiment of the present invention, but this is only for convenience of description and is not intended to limit the present invention. First, FIG. 1 first introduces all components and configuration relationships of the image processing device, and detailed functions will be disclosed together with FIG. 2.

Referring to FIG. 1, the image processing apparatus 100 includes at least a memory 110 and a processor 120. The processor 120 is coupled to the memory 110. The image processing device 100 may be an external or built-in electronic device such as a personal computer, a notebook computer, a digital camera, a digital camera, a web camera, a smart phone, a tablet computer, a driving recorder, a car audio and video system, and the like.

The memory 110 is used to store video images and data, and may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), Flash memory, hard disk, or other similar devices or a combination of these devices.

The processor 120 is configured to execute the proposed image enhancement method, which may be, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessors, digital signals, Processor (digital signal processor (DSP), programmable controller, application specific integrated circuits (ASIC), programmable logic device (programmable logic device (PLD) or other similar device or these devices) combination.

FIG. 2 is a flowchart of an image reconstruction method according to an embodiment of the present invention, and the method flow of FIG. 2 may be implemented by each element of the image processing apparatus 100 of FIG. 1.

Please refer to FIG. 1 and FIG. 2 at the same time. First, the processor 120 of the image processing apparatus 100 receives the input image (step S202), and inputs the input image to the image reconstruction model (step S204). The image reconstruction model here is based on the assumption that there is a positive correlation between pixels in the image and other pixels with specific weights. It can be used in global context, local context, and related environments ( social context) and other attributes.

Global environment In other words, it is obtained from all clean pixels in the damaged image globally using multivariate interpolation of inverse-distance weighting, and is used to reconstruct each pixel To restore the entire damaged image. When given a known clean pixel with no noise When Corrupted pixels with noise at each location It can be reconstructed by equation (3): among them First Clean pixels at locations, Is the inverse distance weight and can be calculated by equation (4): among them Is a power parameter and can be set to 2.75, however the invention is not limited to this.

Regional environment In other words, it can calculate weights on the assumption that a pixel has a high probability of being close to the average brightness value of its neighboring pixels. Therefore, the average interpolation of neighboring pixels in a block can be used to generate the regional environment of a noisy pixel, and can be expressed by equation (5): among them Pixels Centered block Pixels in Block The number of pixels in the.

Relevant context In general, because general images often have repeated composition structures, they are used to describe non-regional repeated content. Taking the schematic diagram of the input image Img shown in FIG. 3A according to an embodiment of the present invention as an example, blocks 311-314, blocks 321-322, blocks 331-334, and blocks 341-343 have similar compositions, respectively. Structure and relevance. Based on this, blocks with similar composition structure can be grouped into a cluster. Taking the grouping diagram shown in FIG. 3B according to an embodiment of the present invention as an example, blocks 311 to 314, blocks 321 to 322, blocks 331 to 334, and blocks 341 to 343 will be combined into groups C1, C2, C3, and C4. Here, a similar non-regional block can be used to obtain the weight of the distance within the class (within-class scatter) by clustering interpolation of the relevant environment of the noise pixels, as shown in equation (6): among them Is the distance weight in the scatter within the category, and can be calculated by equation (7): among them Pixels Centered block Belongs to Groups, and First The number of blocks in a cluster. In this embodiment, the distribution within a category is obtained by using K-means clustering.

In this embodiment, the first Pixels The image reconstruction model can be expressed by equation (8): among them , , Are linear coefficients, Is the random error of the model and is Gaussian (i.e., ). The image reconstruction model proposed by equation (8) can effectively repair the noisy images damaged by salt and pepper noise, and the following will use semi-supervised learning to obtain a large number of noisy images Three optimized coefficients , , And variables .

As image reconstruction involves three topics, high-density noise, sparse training samples, and multimodal density noise, in order to effectively remove noise from the image , Here we will use the cost function of equation (9) The minimization problem is described: The cost function will be explained in three stages below .

Cost function of image reconstruction model in the first stage Effectively manages images damaged by high-density noise. Here we will standardize the learning model with two principles: 1) For each clean pixel , The estimated value of X is close to the original value; and 2) for each noise pixel Adjacent pixels The difference between the estimated pixel values (i.e., ) Need and versus The spatiotemporal distance between them is directly proportional. According to these principles, the first term of the cost function And the second Can be defined by equation (10) and equation (11) respectively: among them Is a space factor and can be calculated using equation (12): among them Can be set to 1.25.

In addition, since most of the noise pixels in the damaged image are randomly distributed, the average brightness value of the clean pixels in the damaged image will be close to the average brightness value of the clean image. Therefore, the third term of the cost function It can be defined by equation (13): among them as well as The number of clean pixels and the total number of pixels in the damaged image. Combining the above three terms, the cost function can be expressed by equation (14): Among them in the first stage, Is a regularization parameter and can be set to 0.3.

In the second phase, based on an image database with a large number of images The number of images without noise in the image is limited, so those marked clean in the training set will be extremely sparse. Image library here Can be divided into a collection of damaged images And labeled clean image collection .

So the cost function here Can target clean image collections Use semi-supervised learning to reconstruct clean images. Therefore, the cost function can be further expressed by equation (15): among them and Will make the reconstructed pixels close to the labeled version, and can be represented by equation (17): Therefore, even if the image collection is not completely non-destructive, the parameters in the image reconstruction model can still be learned by the high similarity between the reconstructed image and the labeled clean image.

In the third stage, due to the image database The noise density of the damaged image is different. Based on the average brightness value of the reconstructed image will be close to the average brightness value of the clean pixels in the damaged image collection. The cost function can further add a fifth term , And can be expressed by equation (18): among them as well as Image database The number of clean pixels in and the number of all pixels.

Combining the above five terms, the cost function It can be expressed by equation (19): among them As mentioned earlier, this cost function Image reconstruction model Based on these three topics, the noise image is reconstructed, and the processor 120 will obtain the reconstructed image output by the image reconstruction model (step S206), and the reconstructed image achieves good visual effects.

Image database Sparsely labeled clean images, image reconstruction models Each input pixel can be Fixed into clean pixels. Here, maximum likelihood estimation (MLE) can be applied to the cost function Learn coefficients through semi-supervised learning , , And variables .

In detail, the salt and pepper noise effect can be randomly combined into the pixels used as training at a ratio of 0% to 90%. In an example, the pixels used for training may be 109,379,580+ pixels obtained from the LabelMe database, including 52,324,300+ noise pixels and 34,055,270+ noise-free pixels. The goal here is to obtain the joint conditions that can constitute a likelihood function such as equation (21): among them Is the probability density function, For damaged pixels Reconstructed pixels, Is the number of pixels in the image database. Assuming that the random error of each reconstructed pixel is independent, equation (21) can be written as equation (22):

Likelihood function here Logarithm to get the optimal coefficient , , And variables To make the likelihood function It can reach the maximum value, which can be expressed by equation (23): Then, you can add Relative coefficient , , Perform partial differential calculations and update iteratively than equation (24): among them , An operator that can set the value of the left term to the value of the right term. Here, with the variable Separately Performing partial differential calculations will give equation (25): Next, if equation (25) is set to zero, the variable can be obtained using the maximum likelihood estimation method. The estimated value, as in equation (26): On the other hand, in equation (24), (among them ) Can be expressed by equation (27), equation (28), and equation (29):

In the previous example, after 20 iterations, an optimized coefficient can be obtained , , And variables .

In simple terms, the linear coefficients in the image reconstruction model of equation (8) , , And variables It can be obtained by using the flowchart of the parameter estimation method shown in FIG. 4 according to an embodiment of the present invention.

Please refer to FIG. 4. First, the processor 120 receives a training image set. Input image (Step S402). Next, the processor 120 will start to initialize the parameters: coefficients ,coefficient ,coefficient , Number of iterations (Step S404). After that, the processor 120 will calculate the variables according to equation (26) (Step S406). Then, during the iteration, the processor 120 will update each image according to equation (27) First Pixels (Step S408), updating each image according to equation (28) First Pixels (Step S410) and updating each image according to equation (29) First Pixels (Step S412). After that, the processor 120 will determine whether the number of iterations has reached the maximum number of iterations. , which is (Step S414). If not, the processor 120 will return to step S408 to perform the iterative process again. If yes, the processor 120 will output the newly updated coefficients , , And variables (Step S416), and the process steps of the parameter estimation method are completed.

FIG. 5 is a functional block diagram of an image reconstruction method according to an embodiment of the present invention.

Please refer to FIG. 5, I _IN is an input image with 80% noise density, and I ′ is an image without noise for reference. After the processor 120 inputs the input image I _IN to the aforementioned image reconstruction model IRM, a restored reconstructed image I _OUT will be obtained.

In summary, the image reconstruction method and image processing device for suppressing multi-mode noise proposed by the present invention can utilize the semi-supervised method on the premise of considering the global environment, local environment and related environment of pixels. The image reconstruction model trained by the learning method can effectively repair the damaged image with high-density noise to reconstruct an image with good visual effects.

Although the present invention has been disclosed as above with the examples, it is not intended to limit the present invention. Any person with ordinary knowledge in the technical field can make some modifications and retouching without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be determined by the scope of the attached patent application.

100: Image processing device 110: Memory 120: Processors S202-S206, S402-S416: Steps 311-314, 321-323, 331-334, 341-343: Blocks C1, C2, C3, C4: Grouping I _IN : input image I _OUT : output image I ': original image IRM: image reconstruction model

FIG. 1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a flowchart of an image reconstruction method according to an embodiment of the present invention. FIG. 3A is a schematic diagram of an input image according to an embodiment of the present invention. FIG. 3B is a schematic diagram of clustering according to an embodiment of the present invention. FIG. 4 is a flowchart of a parameter estimation method according to an embodiment of the present invention. 5 is a functional block diagram of an image reconstruction method according to an embodiment of the present invention;

Claims (9)

An image reconstruction method for suppressing multi-mode noise is applicable to an image processing device. The method includes the following steps: receiving an input image, wherein the input image includes a plurality of pixels, and the pixels include a plurality of pixels with noise. Damaged pixels and multiple clean pixels without noise; input the input image to the image reconstruction model during the prediction stage of the image reconstruction model, where the image reconstruction model is associated with the global environment, regional environment, A related environment, a first coefficient, a second coefficient, and a third coefficient, the global environment is associated with all the clean pixels, and the regional environment is associated with a block formed by a plurality of adjacent pixels of each of the pixels, the The related environment is related to a cluster formed by a plurality of similar pixels of each of the pixels, the first coefficient, the second coefficient, and the third coefficient are respectively related to the global environment, the regional environment, and the related environment, and Obtained during the training phase of the image reconstruction model by a semi-supervised learning method; and obtaining a reconstructed image output by the image reconstruction model, wherein each of the images is This image reconstruction model _{is: f (x i) = θ} 0 g (x i) + θ 1 l (x i) + θ 2 s (x i) + ε (x i) where x _i is located i th Position pixel, f ( x _i ) is the image reconstruction model of the pixel x _i , g ( x _i ) is the global environment of the pixel x _i , and l ( x _i ) is the regional environment of the pixel x _i , S ( x _i ) is the relevant environment of the pixel x _i , θ _{0 is} the first coefficient, θ _{1 is} the second coefficient, θ _{2 is} the third coefficient, and ε ( x _i ) is the pixel x The random error of _i is also Gaussian, where ε ( x _i ) ~ G (0, σ ² ). The method according to item 1 of the scope of the patent application, wherein the global environment g ( x _i ) of the pixel x _i is obtained from all the clean pixels by multivariate interpolation using an inverse distance weighting method. The method according to item 2 of the scope of patent application, wherein the global environment g ( x _i ) of the pixel x _i is: Where x ^nf is the clean pixel, x _j is the clean pixel at the j- th position, λ _J is the inverse distance weight and is: Where β is a power parameter. The method as defined in claim 3 of the scope of the item, wherein the pixel x _i in the local environment L (x _i) is calculated based on the average of the pixels adjacent to the interpolation plurality of pixels of the x _i. The method according to item 4 of the scope of patent application, wherein the regional environment l ( x _i ) of the pixel x _i is: Where x _k is a pixel in a block Ω centered on the pixel x _i , and ω is the number of multiple pixels in the block Ω. The method according to item 5 of the scope of patent application, wherein the relevant environment s ( x _i ) of the pixel x _i is obtained by using the clustering difference value to obtain the scattered distance weights within the category. The method according to item 6 of the scope of patent application, wherein the relevant environment s ( x _i ) of the pixel x _i is: Where ω _k is the distance weight that is scattered within the category and is: Where Ω ( x _k ) is the c- th cluster to which the block Ω centered on the pixel x _k is, and ω is the number of all blocks in the c- th cluster. The method according to item 1 of the scope of patent application, wherein the training phase is a cost function of the image reconstruction model based on high-density noise, sparse training samples, and multi-mode density noise, and uses a clean The image and the multiple noise images are trained using the semi-supervised learning method to optimize the cost function of the first coefficient, the second coefficient, and the third coefficient, wherein the number of the labeled clean images is large The number of noise images. An image processing device includes: a memory for storing images and data; and a processor coupled to the memory for: receiving an input image, wherein the input image includes a plurality of pixels, and the pixels include a plurality of pixels Noisy pixels and multiple clean pixels without noise; input the input image to the image reconstruction model during the prediction phase of the image reconstruction model, where the image reconstruction model is related to the global environment, regional environment, A related environment, a first coefficient, a second coefficient, and a third coefficient, the global environment is associated with all the clean pixels in the input image, and the regional environment is associated with a plurality of neighboring pixels of each of the pixels Block, the related environment is associated with a cluster formed by a plurality of similar pixels of each of the pixels, the first coefficient, the second coefficient, and the third coefficient are associated with the global environment, the region, respectively The environment and the related environment are obtained by a semi-supervised learning method during the training phase of the image reconstruction model; and obtaining the input image output by the image reconstruction model Reconstructed image, wherein the image reconstruction of the model of each pixel _{is: f (x i) = θ} 0 g (x i) + θ 1 l (x i) + θ 2 s (x i) + ε (x _i ) where x _i is the pixel at the i- th position, f ( x _i ) is the image reconstruction model of the pixel x _i , g ( x _i ) is the global environment of the pixel x _i , and l ( x _i ) Is the regional environment of the pixel x _i , s ( x _i ) is the relevant environment of the pixel x _i , θ _{0 is} the first coefficient, θ _{1 is} the second coefficient, and θ _{2 is} the third coefficient, ε ( x _i ) is a random error of the pixel x _i and is a Gaussian distribution, where ε ( x _i ) ~ G (0, σ ² ).