CN104867110A - Lattice Boltzmann model-based video image defect repairing method - Google Patents

Abstract

The invention discloses a video image defect repair method based on a lattice Boltzmann model. The operation steps of this method are: 1) input the object to be repaired; 2) judge the format of the repaired object, if it is an image, go to step 5) to repair, otherwise go to step 3); 3) judge the format of the input video signal, if it is an analog The video is first converted into digital video through the video capture card, and then enter step 4), otherwise directly transfer to step 4); 4) decompose the video into frames; 5), use the proposed lattice Boltz for each frame of the decomposed image Repair by Mann method; 6) Determine the required output format, if necessary, synthesize video, otherwise output image directly. The lattice Boltzmann repair method updates the particle density of each node in the model through the collision and migration process, and outputs if the number of iterations is reached or the threshold condition is met. The restoration method is simple to implement and has good parallelism, and can improve restoration efficiency while maintaining good restoration quality.

Description

Video Image Defect Repair Method Based on Lattice Boltzmann Model

technical field

The invention belongs to the field of video image processing, and proposes a method for repairing video image defects based on a lattice wave model.

Background technique

With the development of digital technology, the preservation and restoration of digital video has become an important means of preservation and restoration of old archives, old photos, and old film and television materials. The restoration of digital video is to use the spatial structure information of the video frame or the continuity between the video frames to repair and fill the specified damaged area in the video, so that the video has spatial smoothness and temporal continuity, and improves the viewing quality of the video. , is widely used in many situations, such as removing redundant objects in the scene, repairing spots or scratches in old film and television materials, and solving the flickering problem of old film during playback, etc.

Video inpainting technology is an extension of digital image inpainting technology. At present, the main video inpainting technology is similar to image inpainting technology. However, for image inpainting problems at home and abroad, the partial differential equation (Partial Differential Equation) is mainly used. Equation, PDE) method and method based on texture synthesis.

The restoration method based on partial differential equations was originally based on the principle of manual restoration, using the surrounding information of the damaged area of the image to gradually spread to the damaged area along the direction of the isoluminance line to achieve the effect of smooth repair. Damaged images with complex texture structures can be better repaired. Chan and Shen proposed a total variational model (TV model) for inpainting, (Chan T, Shen J.2002.Mathematical models for local non-texture inpainting[J].SIAM Journal on Applied Mathematics, 62(3):1019-1043.) The TV repair method finds the minimum value of the energy functional.

The TV model will have discontinuity problems when repairing large areas, destroying the principle of visual connectivity, and the Curvature Driven Diffusion (CDD) model is used for repair (T. Chan, J. Shen. Non-Texture Inpaintings by Curvature-Driven Diffusions [C]. JVCIR, 2001, 12(4):436-449) do not suffer from discontinuities, embedding curvature into the diffusion coefficient. The CDD model can realize the repair of larger damaged areas, and when the repaired area is at the edge, a better repair effect can be obtained. Both the TV model and the CDD model can achieve better repair effects, but from the perspective of computational efficiency, the solution of partial differential equations requires numerical calculations, and the algorithm implementation is more complicated, takes a long time, and has low implementation efficiency.

The image repair method based on texture synthesis, that is, the image completion technology, uses information other than the boundary to be repaired, and uses the method of searching for matching blocks to gradually copy and fill the boundary of the damaged area (Zhang Hongying, Peng Qicong. A review of digital image restoration technology [J ]. Chinese Journal of Image and Graphics, 2007, 12(1): 1-10).

This repair method can realize the repair of large area damage, and can get better texture characteristics through matching filling. However, this kind of method tends to produce obvious color and structure inconsistency inside the repaired area, the repairing effect is not ideal, and its own calculation is time-consuming, resulting in low efficiency of image repairing.

The lattice Boltzmann model has the advantages of clear physical thought, simple boundary processing and fast parallel calculation, and it is a discrete space model, which is especially suitable for digital image processing. From the microscopic model of the lattice Boltzmann model, the evolution equation of the lattice Boltzmann model is designed, and finally the macroscopic partial differential equation that meets the requirements of image processing can be obtained. Therefore, the lattice Boltzmann model method provides a way to achieve fast, efficient and accurate image processing.

Contents of the invention

The purpose of the present invention is to propose a novel and fast cross-platform restoration implementation scheme based on Lattice Boltzmann model to solve the problem of discontinuous repair area and image restoration in the prior art. The problem of low efficiency, this method aims to obtain better repair efficiency while maintaining a better repair effect, and can be used to repair various defects in the image, such as spots, scratches, and target removal. And with this repair method as the core, a repair system that can run on a variety of software and hardware platforms has been developed, making this method more widely used.

In order to achieve the above object, the present invention adopts the following technical solutions:

The invention proposes a restoration implementation scheme based on the lattice Boltzmann model, which can run on hardware computing platforms including PCs and servers, and various software operating systems including Android, Linux, Mac, and Windows. Its main steps are as follows:

1), input images containing defects ;

2) Determine the format of the repaired object, if it is an image repair, go to step 5) to repair, otherwise go to step 3);

3), determine whether the input is digital video or analog video, if it is analog video, first convert it to digital video through the video capture card, and then enter step 4); otherwise, directly enter step 4);

4) Decompose the video into frames such as 25 frames per second;

5) Use the proposed Lattice Boltzmann model to repair each decomposed image;

6) Determine the required output format, if the required output is video, synthesize the video, otherwise directly output the image .

For the repair of the lattice Boltzmann model mentioned above, the present invention proposes an anisotropic diffusion lattice Boltzmann model (D2Q9 model), and using this model to realize defect repair is to use the area to be repaired and the surrounding information Geometric information such as gradient or curvature is used to design the diffusion coefficient, and the diffusion evolution speed of particles is controlled according to the geometric characteristics of the image itself to achieve a smooth repair process. Because all kinds of videos can be decomposed into images, without loss of generality, the present invention takes the spot and scratch image of repairing bmp format as an example, and experiments are carried out on a PC equipped with Windows operating system, and the repair results are shown in Fig. 9 , 10, Data records are shown in Figure 1 2 and Figure 1 3. The main flow chart of the repair is shown in Figure 3. The present invention combines the characteristics of the diffusion coefficient in the TV model and the CDD model macroscopic equation, and is designed to apply to the lattice Boltzmann model for the spots and scratches that appear in images and videos. Diffusion coefficient, so as to improve the repair efficiency while maintaining a good repair effect. For images with artifacts that have been read in , its main steps are as follows:

1) Judging and detecting the damaged area , and dilate the detected area to get , the preprocessed image ;

2) Establish a lattice Boltzmann model and set the initial equilibrium distribution function ;

3) Establish the iteration times of the lattice Boltzmann model evolution equation , and the iteration termination condition and iteration termination threshold ;

4), Design the diffusion coefficient of the lattice Boltzmann model evolution equation ;

5), according to the D2Q9 model in the lattice Boltzmann model, update the equilibrium state distribution function in the evolution equation of the lattice Boltzmann model ;

6), calculate the migration process in the lattice Boltzmann model: ;

7) Calculate the collision process in the lattice Boltzmann model:

8) Update the particle density at each node of the model , and peripheral information of the area to be repaired;

9) Judging whether the iteration termination condition is satisfied, if not, proceed to step 10), otherwise the repair ends;

10), judge whether the number of iterations reaches N, if it does not reach the number of iterations N, go to step 4), and repeat steps 4)-9), until the number of iterations N is reached, the repair is completed;

In the above step 3), it is proposed to use the iteration termination condition to judge whether to stop the iteration of the evolution equation of the lattice Boltzmann model. The commonly used iteration termination condition is:

in, is a preset threshold, and is the number of pixels in the damaged area of the input image. The present invention optimizes the above-mentioned iteration termination condition, divides the absolute value of the pixel difference in the area to be repaired before and after each iteration into five segments, and assigns a weight to each segment, which is called the difference weight here, as shown in Figure 5 .

Before and after each iteration, the greater the pixel difference value, the greater the chance of continuous iteration, so the difference weight represents the importance of the evolution of the pixel point at the pixel difference value during the evolution process. Here the difference weight is set to , and the iteration termination condition proposed by the present invention is:

in, It will be pre-set according to different images to be repaired and the requirements for repair quality.

In the above step 4), it is proposed to design the diffusion coefficient of the evolution equation of the lattice Boltzmann model. In TV model and CDD model, the diffusion coefficients are:

and

in, is the gradient of the image to be repaired, is a curvature function. In the speckle distortion image, most of the spots are approximately circular or oval, and the spot area is small. In the scratch distortion image, the scratches are mostly long strips, neither of which will cause the loss of most textures. Meanwhile, in order to improve repair efficiency, the design of diffusion coefficient only includes gradient information. The diffusion coefficient is ,in is non-negative monotonically decreasing, and and .

The present invention proposes three different diffusion coefficient models and their combinations, and the diffusion coefficients are as follows:

, ,

in, is the gradient of the image to be repaired, is a preset threshold, is a positive value. Analyze the curves of the above three diffusion coefficients, as shown in Figure 6 , and combine the three diffusion coefficients to obtain different diffusion coefficients:

in, Taking different values will correspond to different diffusion coefficients, which will have different effects on image restoration.

Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant technical progress: because the lattice Boltzmann model has a parallel structure, it is easy to realize parallel computing, and compared with the texture synthesis method, the spot repair method is improved. The efficiency can be better applied to the restoration of speckle distortion in old film and television materials. Compared with the classic PDE repair model, it has a better repair effect, overcomes the discontinuity of the repair area, and can also improve the repair efficiency.

Description of drawings

Figure 1 : The vector directions of the lattice Boltzmann model based on D2Q9;

Figure 2 : The next repair operation flow chart of the video image based on the Lattice Boltzmann model;

Figure 3 : A flowchart of the Lattice Boltzmann model inpainting method for defects in images or videos;

Figure 4 : Schematic diagram of the structure of the two-dimensional lattice Boltzmann model D2Q9 model composed of discretized networks;

Figure 5 : Segmentation of the pixel difference in the area to be repaired, and the weight distribution of the difference in each segment;

Fig. 6 : the graph of three kinds of diffusion coefficients and combination thereof that this invention proposes, wherein middle The values are all set to 10, in the combination curve set to ;

Figure 7 : Speckle distortion grayscale image, the black dots in the figure are speckles;

Figure 8 : Old photos with scratches and distorted images, the white stripes are scratches;

Figure 9 : Inpainting results of speckle-distorted grayscale images, where Fig. which in turn corresponds to a diffusion coefficient of Lattice Boltzmann model inpainting results;

Fig. 10 : Repair results of old photo scratches, where Fig. which in turn corresponds to a diffusion coefficient of Lattice Boltzmann model inpainting results;

Figure 1 1: Repair software running display diagram ;

Fig. 1 2 is Table 1 : for the speckle distortion of the grayscale image, the number of iterations and repair time required for repairing the lattice Boltzmann model with different diffusion coefficients;

Figures 1 and 3 are Table 2 : For the scratch distortion of old photos, the number of iterations and repair time required for the repair of the lattice Boltzmann model with different diffusion coefficients.

Detailed ways

The present invention will be described in detail below with reference to accompanying drawing and in conjunction with embodiment:

Embodiment 1: See Fig. 1 and Fig. 2 , this operation steps based on Lattice Boltzmann method: 1), input an image containing defects ;2), determine the format of the repair object, if it is an image repair, go to step 5) to repair, otherwise go to step 3); 3), judge whether the input is digital video or analog video, if it is an analog video, first pass through the video capture card Convert to digital video, and then go to step 4), otherwise, go directly to step 4); 4), decompose the video into frames; 5), use the proposed lattice Boltzmann method to repair each decomposed image; 6) Determine the required output format, if the required output is video, synthesize the video, otherwise directly output the image .

The video includes digital or analog video in MPEG-1, MPEG-4, AVI, RM, ASF, WMV, MOV or MKV formats; the image includes analog or digital images obtained from various remote sensing imaging or medical instrument imaging Image, the format of which includes bmp, jpg, gif, or png; the sources of defects include color distortion caused by imaging equipment failure, fading and creases of old photos, playback flicker caused by film aging, spots on old video materials, scratches. The lattice Boltzmann model: it is realized by embedding the edge cutoff function of the image into the relaxation factor of the lattice Boltzmann model to solve the diffusion equation. The commonly used two-dimensional lattice Boltzmann model includes D2Q5—two-dimensional Diffusion in 5 directions, D2Q9, three-dimensional lattice Boltzmann model including D3Q7, D3Q15. The video image defect repair based on the lattice Boltzmann model can run on a hardware computing platform including a PC or a server, and a software operating system including Android, iOS, Linux or Windows.

The specific operation steps of the lattice Boltzmann repair method are as follows:

1) Judging and detecting the damaged area , and dilate the detected area to get , the preprocessed image

2) Establish the LB model and set the initial equilibrium distribution function ;

3) Establish the number of iterations of the LB evolution equation , and the iteration termination condition and iteration termination threshold ;

4) Design the diffusion coefficient of the LB evolution equation ;

5), according to the D2Q9 model in the LB model, update the equilibrium state distribution function in the LB evolution equation ;

6) Calculate the migration process in the LB model: ;

7) Calculate the action process in the LB model:

;

8) Update the particle density at each node of the model , and peripheral information of the area to be repaired;

9), judging whether the iteration termination condition is met, if not, proceed to step 10), otherwise the repair ends;

10), judge whether the number of iterations reaches N, if it does not reach the number of iterations N, go to step 4), and repeat steps 4)-9), until the number of iterations N is reached, the repair is completed;

The edge cut-off function is a non-negative monotonically decreasing function, which can use image gradient information and curvature information to slow down the diffusion speed at the boundary and accelerate the diffusion speed at the smooth place, so that the diffusion speed and edge retention ability are both improved during the image diffusion process. Has better performance. The relaxation factor controls the diffusion rate to realize nonlinear diffusion, so as to protect image edge characteristics and suppress noise; the design of equilibrium distribution function realizes anisotropic diffusion or isotropic diffusion.

Embodiment 2: This embodiment is implemented on the premise of the technical solution of the present invention, and repairs the spot distorted image and the old photo scratch distorted image respectively, as shown in FIGS. 7 and 8 . Detailed embodiments are given below.

The lattice Boltzmann model D2Q9 model is composed of a uniform grid, where the nodes of the grid are regarded as a cell, corresponding to the pixel points in the image . The gray value at the pixel corresponds to the particle density in the cell . at the same time, described in the vector direction The particle density distribution function on , where the vector direction representation at the node is shown in Figure 1 :

The D2Q9 model has 9 discrete velocities, and the diffusion evolution equation of microscopic particles in the model is:

in, represented at the node The particle diffusion coefficient at , and the relaxation factor in the lattice Boltzmann model The relationship between is:

is the equilibrium distribution function, where the relationship between the equilibrium distribution function and the particle density at each node is:

Example 1: Repairing a grayscale image with speckle defects

Lattice Boltzmann model restoration for distorted videos or images, for images to be repaired that have been obtained in advance , because the selected implementation object is an image, so the present invention directly adopts the restoration process as shown in Figure 3 , and the method mainly includes the following steps:

1) Perform damaged area detection on the input image to obtain the area to be repaired , and dilate the detected area to get , the preprocessed image ;

in, is the entire area of the input image;

2) Establish a lattice Boltzmann model and set the initial equilibrium distribution function ;

3) Preset the number of iterations of the evolution equation of the L model =500 and iteration termination threshold =0.03;

4) Design the diffusion coefficient of the evolution equation of the lattice Boltzmann model, and ,in . when When , calculate the combination of the three diffusion coefficients , respectively apply the above four diffusion coefficients to the evolution equation of the lattice Boltzmann model, and compare the restoration effect;

5) According to the D2Q9 model in the lattice Boltzmann model, as shown in Figure 4 , update the equilibrium state distribution function in the evolution equation of the lattice Boltzmann model ;

6) Calculate the migration process in the lattice Boltzmann model: ;

7) Calculate the collision process in the lattice Boltzmann model:

8) Update the particle density at each node of the model , and update the peripheral information of the area to be repaired, the updated image is:

9) Judging whether the iteration threshold condition is met, if the iteration termination condition is not met, go to step 10), otherwise the repair ends;

10), judge whether the number of iterations reaches 500, if not, go to step 4), and repeat steps 4)-9), until the number of iterations reaches 500, the repair is completed;

After the repair is completed, the repair effect is output, as shown in Figure 9 . The hardware conditions for the above implementation are processor Intel® Core™ i5-2450M, main frequency 2.50GHz, memory 4G; software configuration is Window7 system and matlab2012b; for the experiment of speckle distortion of grayscale images, four different diffusion coefficients need to be iterated The times and iteration time are shown in Fig . 12.

Example 2: Restoring old photos with scratches

Figure 8 shows the image of scratch distortion, and the implementation steps are the same as above. The restoration results obtained by using the above four diffusion coefficients to repair the lattice Boltzmann model are shown in Fig . The number of iterations required and the repair time are compared.

Experiments show that, for the restoration of image and video defects, the restoration scheme based on the Lattice Boltzmann model can obtain better restoration results and improve the restoration efficiency. This method can be better applied to engineering applications.

Claims (7)

1. A quick and novel cross-platform video image flaw repairing method based on a lattice Boltzmann model is characterized in that the video image flaw repairing operation steps are as follows:

1) inputting the image containing the flaw；

2) Judging the format of the repaired object, if the image is repaired, turning to the step 5) to repair, and if not, entering the step 3);

3) judging whether the input video is digital video or analog video, if the input video is analog video, converting the input video into digital video by a video capture card, and then turning to the step 4), otherwise, directly entering the step 4);

4) decomposing the video into frames;

5) repairing each decomposed frame of image by using a proposed lattice Boltzmann method;

6) judging the required output format, if the required output is video, synthesizing the video, otherwise, directly outputting the image。

2. The method for repairing the defects of the video image based on the lattice Boltzmann model is as follows:

the video comprises digital or analog video in MPEG-1, MPEG-4, AVI, RM, ASF, WMV, MOV or MKV format; the images comprise various remote sensing imaging or analog or digital images obtained by imaging of a medical instrument, and the formats of the images comprise bmp, jpg, gif or png; the sources of defects include color distortion caused by failure of imaging equipment, fading and creasing of old photos, play flicker caused by aging of films, spots and scratches of old video data.

3. The method for repairing the defects of the video image based on the lattice Boltzmann model as claimed in claim 1, wherein:

the lattice boltzmann model: solving the diffusion equation is realized by embedding the edge cut function of the image into the relaxation factors of a lattice Boltzmann model, wherein the commonly used two-dimensional lattice Boltzmann model comprises D2Q5, two-dimensional 5-direction diffusion, D2Q9, and the three-dimensional lattice Boltzmann model comprises D3Q7 and D3Q 15.

4. The method for repairing the defects of the video image based on the lattice Boltzmann model as claimed in claim 1, wherein:

the video image flaw repairing based on the lattice Boltzmann model can be operated in a hardware computing platform comprising a PC or a server and a software operating system comprising Android, iOS, Linux or Windows.

5. The method for repairing video and image based on lattice Boltzmann model according to claim 1, wherein the specific operation steps of the step 5) are as follows:

1) determining and detecting the damaged areaAnd expanding the detected region to obtainPre-processing to obtain an image

2) Establishing an LB model and setting an initial equilibrium state distribution function；

3) Establishing the iteration number of the LB evolution equationAnd an iteration end condition and an iteration end threshold；

4) Designing diffusion coefficient of LB evolution equation；

5) Updating the equilibrium state distribution function in the LB evolution equation according to the D2Q9 model in the LB model；

6) Calculating the migration process in the LB model:；

7) calculating the action process in the LB model:

；

8) updating the particle density at each node of the modelAnd peripheral information of the area to be repaired;

9) judging whether an iteration termination condition is met, if the iteration termination condition is not met, entering a step 10), and if not, finishing the repair;

10) judging whether the iteration number reaches N, if not, turning to the step 4), and repeating the steps 4) -9) until the iteration number N is reached, and finishing the repair.

6. The method for repairing video image based on lattice Boltzmann model as recited in claim 3, wherein:

the edge cut-off function is a non-negative monotone decreasing function, and can utilize image gradient information and curvature information to reduce the diffusion speed at the boundary and accelerate the diffusion speed at the smooth part, so that the diffusion speed and the edge retention capacity have better performance in the image diffusion process.

7. The method for repairing video image based on lattice Boltzmann model as recited in claim 3, wherein:

the relaxation factor controls the diffusion rate and realizes the nonlinear diffusion so as to achieve the purposes of protecting the edge characteristic of the image and inhibiting noise; the design of the equilibrium state distribution function realizes anisotropic diffusion or isotropic diffusion.