CN105005977B - A kind of single video frame per second restored method based on pixel stream and time prior imformation - Google Patents

Abstract

The invention discloses a single video frame rate restoration method based on pixel stream and time prior information. Firstly, the single video to be restored is obtained and the observed pixel stream is constructed, and the single video is expressed as a matrix form of the observed pixel stream; and then established The degradation model of the observed pixel stream and the probability estimation formula of the original pixel stream are derived from this to derive the restoration formula of the original pixel stream with time prior information; The data-driven method is determined; finally, the restored pixel streams restored one by one are combined into a restored video in matrix form, which is the final restored high frame rate video. The restoration method of the present invention adopts a single video, which makes video collection convenient and the restoration process is simple; the frame rate restoration is based on the pixel flow probability statistics framework, and introduces time prior information, and adopts a data-driven method to determine the time prior model, which not only improves video fidelity It also effectively eliminates the smearing of video frames.

Description

A Single Video Frame Rate Restoration Method Based on Pixel Flow and Time Prior Information

technical field

The invention relates to a video restoration method, in particular to a single video frame rate restoration method based on pixel stream and time prior information, and belongs to the technical field of computer image and video processing.

Background technique

Most of the traditional video restoration methods aim at the problem of low spatial resolution of the video, and improve the spatial resolution of the video and restore the spatial detail information of the video by performing spatial restoration on a single video frame by frame or performing complementary spatial information reconstruction on multiple videos. On the basis of years of extensive and in-depth research, the majority of scientific researchers have proposed various video restoration methods, which can solve the problem of low spatial resolution to a certain extent, or meet the basic application requirements.

However, even if the above-mentioned targeted video restoration methods restore the video spatial detail information, there are still problems such as low temporal resolution and missing frame information, so that video flickering, pause or jitter effects may occur.

Therefore, unlike commonly used video restoration techniques, in addition to involving spatial resolution improvement and spatial detail information recovery, video restoration should also focus on temporal resolution (ie, frame rate) and temporal detail information to further improve video quality or visual effects. At present, some scholars at home and abroad have noticed this problem and proposed a frame rate restoration method for video.

Existing video frame rate restoration methods generally follow two paths.

One is to collect multiple videos of the same scene at the same time, and restore the video frame rate by fusing redundant/complementary frame information of multiple videos. However, this path will be constrained by video acquisition conditions, such as whether the number of devices is sufficient and whether the models are uniform; and it involves synchronization and time registration of multiple videos, which is more complicated to implement.

Another relatively simple path for frame rate restoration is to restore the frame rate of the video through inter-frame interpolation with only one video. However, the assumption of interpolation functions (such as linear functions, spline functions, quadratic functions, etc.) required for inter-frame interpolation of single video is relatively random, making the fidelity of interpolated frames not high; and even The minimum mean square error is used to achieve interpolation, but it still cannot solve the tailing phenomenon of the video frame. This phenomenon is mainly caused by the long exposure time of the acquisition device, which makes the high-speed moving object appear in the image as a blurred phenomenon along its trajectory. .

Contents of the invention

The main purpose of the present invention is to overcome the deficiencies in the prior art and provide a single video frame rate restoration method based on pixel flow and time prior information, which is especially suitable for video image restoration of high-speed moving objects.

The technical problem to be solved by the present invention is to provide a single video frame rate restoration method based on pixel stream and time prior information, which is convenient for video collection, simple in the restoration process, reliable in restoration results, and strong in practicability. The complex program process of synchronous acquisition or time registration can be obtained, and the video fidelity can be greatly improved, and the tailing phenomenon caused by long exposure time can be effectively eliminated, which is extremely valuable in industry.

In order to achieve the above object, the technical scheme adopted in the present invention is:

A single video frame rate restoration method based on pixel flow and time prior information, characterized in that it comprises the following steps:

Step (1) Obtain the video to be restored: A single video I={I(i)|i∈N} is obtained through video collection as the video to be restored, where I(i) is the frame of the video, and i is the time sequence for each frame number;

Step (2) Design the construction method of the pixel stream to be restored: sort by frame, connect the pixels I _mn (i) at the same coordinates (m, n) in each frame of the single video in series to form the observed pixel stream I _mn ={I _mn (i)|i∈N} as the pixel stream to be restored;

Step (3) Express the single video as the matrix form of the observed pixel stream: according to the construction method of step (2), construct the observed pixel stream one by one according to the order of coordinates in each frame of the single video, and combine the constructed observed pixel streams, A single video can be expressed in the form of a matrix of observed pixel streams as Each element in its matrix is a constructed observation pixel stream;

Step (4) Establish the degradation model of the observed pixel stream: I _mn =DBH _mn +E, where D is the temporal downsampling matrix, B is the temporal fuzzy matrix for simulating the exposure time, H _mn is the original pixel stream, and E is The noise vector of the added Gaussian distribution;

Step (5) calculates the probability estimation formula of the original pixel stream _Hmn :

According to the Bayesian probability statistics rule, the probability estimation formula of the original pixel stream _Hmn is calculated as

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; ;</span>$

Step (6) calculates the restoration formula of the original pixel stream _Hmn :

According to the degradation model of the observed pixel stream built in step (4) and the probability estimation formula of the original pixel stream H _mn obtained in step (5), the restoration formula of the original pixel stream H _mn is obtained by logarithmic calculation:

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; DBH</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

in, To restore the pixel stream, logP(H _mn ) represents the time prior information item of the original pixel stream, and α is an optimization parameter;

Step (7) restore the observed pixel stream: use the restoration formula of the original pixel stream H _mn obtained in step (6), restore the single video in the matrix form of the observed pixel stream obtained in step (3) one by one in the order of subscripts , each I _mn in the matrix is restored by the following steps to obtain the restored pixel stream

Step (7-1) Determine the time prior model P(H _mn ) of the observed pixel flow and judge whether it is Gaussian or Laplacian:

According to the observed pixel flow I _mn , adopt a data-driven method to determine that the time prior model P(H _mn ) of the observed pixel flow is Gaussian or Laplace type Among them, Γ represents the high-pass operator of the signal;

Step (7-2) derives the partial derivative equation based on the restoration formula of the original pixel stream H _mn :

After determining the time prior model P(H _mn ), based on the restoration formula of the original pixel stream H _mn obtained in step (6)

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; DBH</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

The partial derivative equation is derived as $\frac{\&part;}{\&part;h_{m\mathrm{no}}}\&lsqb;\log P\left(h_{m\mathrm{no}}\right)\&rsqb;+{\mathrm{\&alpha;B}}^T{\mathrm{D.}}^T(I_{m\mathrm{no}}-{\mathrm{DBH}}_{m\mathrm{no}})=0;$

Step (7-3) performs linear interpolation on the observed pixel stream I _mn to obtain the pixel stream as the iteration initial value;

Step (7-4) Use the conjugate gradient method to iteratively solve the partial derivative equation obtained in step (7-2) to obtain the restored pixel stream

Step (8) is combined to obtain the restored video: restore the restored pixel stream in step (7) one by one The restored video H is combined in matrix form as output, and the restored video H is expressed in matrix form as

The present invention is further set as: the video collection in the step (1) can use a camera to shoot the moving scene to obtain a single video.

In the present invention, it is further set that: the optimization parameter α in the step (6) is set by trial and error to obtain a value.

The present invention is further set to: in the step (7-1), determine the time prior model P(Hmn) of the observed pixel flow and judge whether it is Gaussian or Laplace-type, and judge it through a data-driven method, including the following step,

Step (7-1-1) Perform linear interpolation on the observed pixel stream _Imn to obtain the pixel stream

Step (7-1-2) establish pixel stream Qualcomm version The Gaussian prior model of

${\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; G</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&Gamma;</span>}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}{\mathrm{<span\; class="notranslate">\; \&pi;\&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; G</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}}\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; \{</span><span\; class="notranslate">\; -</span>\frac{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Gamma;</span>}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; G</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; \}</span>$

and the Laplace prior model $P_L\left(\mathrm{\&Gamma;}{\hat{h}}_{m\mathrm{no}}^0\right)={\left(2{\mathrm{\&sigma;}}_L\right)}^{-K}\exp \{-\frac{\left|\right|\mathrm{\&Gamma;}{\hat{h}}_{m\mathrm{no}}^0|{|}_1^1}{{\mathrm{\&sigma;}}_L}\},$

Among them, σ _G and σ _L represent the standard deviation of Gaussian prior model and Laplace prior model respectively, and K represents the dimension;

Step (7-1-3) utilizes the Gaussian prior model and the Laplace prior model built in step (7-1-2), in Under the known premise, according to the maximum likelihood rule,

estimated standard deviation

Step (7-1-4) finds and value, compare the size of the two, if more than the Then it is determined that the time prior model of the observed pixel flow is Gaussian; otherwise, it is determined that the time prior model of the observed pixel flow is Laplace type.

Compared with prior art, the beneficial effect that the present invention has is:

Collecting a single video and performing a pixel stream-based restoration method can not only get rid of the constraints of the conditions required for multi-video restoration, such as whether the number of devices is sufficient, whether the model is uniform, etc.; and does not involve the complexity of multi-video simultaneous acquisition or time registration. The program process, the path is concise, and the restoration link is simplified; the frame rate restoration is realized through the probability statistics framework of the original pixel stream, which avoids the randomness of the assumed interpolation function, and can greatly improve the video fidelity; at the same time, the time prior information is introduced into the probability statistics In the framework, the time prior model is determined in a data-driven manner, the credibility of the prior model is improved, and the tailing phenomenon caused by long exposure time is effectively eliminated.

The above content is only an overview of the technical solution of the present invention. In order to understand the technical means of the present invention more clearly, the present invention will be further described below in conjunction with the accompanying drawings.

Description of drawings

Fig. 1 is a flow chart of the inventive method;

Fig. 2 is a schematic diagram of a construction method of a pixel stream to be restored;

Fig. 3 is the schematic diagram that the observed pixel flow of construction is combined into a single video in matrix form;

Figure 4 is a schematic side view of a single video expressed in the form of a matrix of observed pixel streams;

Fig. 5 is a diagram of the relationship between the original pixel stream and the observed pixel stream;

Fig. 6 is a flowchart of restoring the observed pixel stream to the original pixel stream;

Fig. 7 is a flow chart of determining a temporal prior model of observed pixel streams based on a data-driven approach.

detailed description

Below in conjunction with accompanying drawing of description, the present invention will be further described.

As shown in Figure 1, the present invention provides a single video frame rate restoration method based on pixel stream and time prior information. First, the single video to be restored is obtained, and the observed pixel stream is constructed as the pixel stream to be restored, and the single video is represented as is the matrix form of the observed pixel stream; then the degradation model of the observed pixel stream and the probability estimation formula of the original pixel stream are established, and the restoration formula of the original pixel stream containing time prior information is derived; and then the observed pixel stream is restored one by one, The time prior models used in the restoration are all determined through data-driven methods; finally, the restored pixel streams restored one by one are combined into a restored video in the form of a matrix, which is the final restored high frame rate video. The specific steps are as follows:

Step (1) Obtain the video to be restored: use the camera to shoot the active scene to complete the video acquisition to obtain a single video I={I(i)|i∈N} as the video to be restored, where I(i) is the frame of the video , i is the number of each frame in chronological order.

Step (2) Design the construction method of the pixel stream to be restored: sort by frame, connect the pixels I _mn (i) at the same coordinates (m, n) in each frame of the single video in series to form the observed pixel stream I _mn ={I _mn (i)|i∈N} is used as the pixel stream to be restored, as shown in Figure 2.

Step (3) Express the single video as the matrix form of the observed pixel stream: according to the construction method of step (2), construct the observed pixel stream one by one according to the order of coordinates in each frame of the single video, and combine the constructed observed pixel streams, As shown in Figure 3, a single video can be expressed in the form of a matrix of observed pixel streams as

Each element in the matrix is a structured observation pixel stream; FIG. 4 is a schematic side view of a single video expressed in the form of a matrix of observation pixel streams.

Step (4) Establish the degradation model of the observed pixel stream: I _mn =DBH _mn +E, where D is the temporal downsampling matrix, B is the temporal fuzzy matrix for simulating the exposure time, H _mn is the original pixel stream, and E is A Gaussian-distributed noise vector to join.

Step (5) calculates the probability estimation formula of the original pixel stream _Hmn :

According to the Bayesian probability statistics rule, the probability estimation formula of the original pixel stream _Hmn is calculated as

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; ;</span>$

The restoration of the original pixel stream is realized through the framework of probability and statistics, instead of using interpolation to estimate the original pixel stream H _mn , so that the fidelity of the restoration result is higher.

Step (6) calculates the restoration formula of the original pixel stream _Hmn :

According to the degradation model of the observed pixel stream built in step (4) and the probability estimation formula of the original pixel stream H _mn obtained in step (5), the restoration formula of the original pixel stream H _mn is obtained by logarithmic calculation:

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; DBH</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

in, To restore the pixel stream, logP(H _mn ) represents the time prior information item of the original pixel stream, and α is an optimization parameter, which can be set by trial and error.

The addition of time prior information in the above step (6) can effectively eliminate the tailing phenomenon of the video.

Because the smearing phenomenon is essentially the result of temporal convolution of the original pixel stream H _mn by the temporal fuzzy matrix B representing the exposure time, as shown in Figure 5, each pixel value of I _mn is mainly composed of the convolution of several pixels of H _mn and obtained; therefore, deconvolution is naturally required to eliminate smearing.

If only the minimum mean square error The deconvolution method to restore the original pixel stream H _mn ignores the time prior information, which is essentially a likelihood estimation, so it is difficult to decompose the implicit convolution and form in the observed pixel stream, Then it is difficult to eliminate the tailing phenomenon.

Step (7) restores the observed pixel stream:

Using the restoration formula of the original pixel stream H _mn obtained in step (6), the single video obtained in step (3) expressed in the matrix form of the observed pixel stream is restored one by one in the following order, and each I _mn in the matrix is shown in the figure The steps shown in 6 are restored to obtain the restored pixel stream

Step (7-1) Determine the time prior model P(H _mn ) of the observed pixel flow and judge whether it is Gaussian or Laplacian:

According to the observed pixel flow I _mn , adopt a data-driven method to determine that the time prior model P(H _mn ) of the observed pixel flow is Gaussian or Laplace type Among them, Γ represents the high-pass operator of the signal; specifically, through the steps shown in Figure 7, using the data-driven method to determine the time prior model can make the model closer to the data characteristics of the original pixel stream, thus avoiding the artificial assumption of the model. Randomness.

Step (7-1-1) Perform linear interpolation on the observed pixel stream _Imn to obtain the pixel stream

Step (7-1-2) establish pixel stream Qualcomm version The Gaussian prior model of

${\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; G</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&Gamma;</span>}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}{\mathrm{<span\; class="notranslate">\; \&pi;\&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; G</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}}\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; \{</span><span\; class="notranslate">\; -</span>\frac{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Gamma;</span>}{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; G</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; \}</span>$

and the Laplace prior model $P_L\left(\mathrm{\&Gamma;}{\hat{h}}_{m\mathrm{no}}^0\right)={\left(2{\mathrm{\&sigma;}}_L\right)}^{-K}\exp \{-\frac{\left|\right|\mathrm{\&Gamma;}{\hat{h}}_{m\mathrm{no}}^0|{|}_1^1}{{\mathrm{\&sigma;}}_L}\},$

Among them, σ _G and σ _L represent the standard deviation of Gaussian prior model and Laplace prior model respectively, and K represents the dimension;

Step (7-1-3) utilizes the Gaussian prior model and the Laplace prior model built in step (7-1-2), in Under the known premise, according to the maximum likelihood rule,

estimated standard deviation ${\widehat{\mathrm{\&sigma;}}}_G=\sqrt{\frac{\left|\right|\mathrm{\&Gamma;}{\hat{h}}_{m\mathrm{no}}^0|{|}_2^2}{K}},{\widehat{\mathrm{\&sigma;}}}_L=\frac{\left|\right|\mathrm{\&Gamma;}{\hat{h}}_{m\mathrm{no}}^0|{|}_1^1}{K};$

Step (7-1-4) finds and value, compare the size of the two, if more than the Then it is determined that the time prior model of the observed pixel flow is Gaussian; otherwise, it is determined that the time prior model of the observed pixel flow is Laplace type.

Step (7-2) derives the partial derivative equation based on the restoration formula of the original pixel stream H _mn :

After determining the time prior model P(H _mn ), based on the restoration formula of the original pixel stream H _mn obtained in step (6)

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; DBH</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

The partial derivative equation is derived as $\frac{\&part;}{\&part;h_{m\mathrm{no}}}\&lsqb;\log P\left(h_{m\mathrm{no}}\right)\&rsqb;+{\mathrm{\&alpha;B}}^T{\mathrm{D.}}^T(I_{m\mathrm{no}}-{\mathrm{DBH}}_{m\mathrm{no}})=0.$

Step (7-3) performs linear interpolation on the observed pixel stream I _mn to obtain the pixel stream as the iteration initial value.

Step (7-4) Use the conjugate gradient method to iteratively solve the partial derivative equation obtained in step (7-2) to obtain the restored pixel stream

Step (8) is combined to obtain the restored video: restore the restored pixel stream in step (7) one by one The restored video H is combined in matrix form as output, and the restored video H is expressed in matrix form as

The innovation of the present invention lies in the fact that a single video is used to achieve convenient video acquisition and simple frame rate recovery in the recovery process. It is based on the pixel flow probability statistics framework, and introduces time prior information, and adopts a data-driven method to determine the time prior model. It not only improves the video fidelity, but also effectively eliminates the smearing phenomenon of the video frame.

The basic principles, main features and advantages of the present invention have been shown and described above. Those skilled in the industry should understand that the present invention is not limited by the above-mentioned embodiments. What are described in the above-mentioned embodiments and the description only illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have Variations and improvements are possible, which fall within the scope of the claimed invention. The protection scope of the present invention is defined by the appended claims and their equivalents.

Claims (4)

1. A method for single video frame rate restoration based on pixel stream and temporal prior information, comprising the steps of:

acquiring a video to be restored: acquiring a single video I ═ { I (I) | I ∈ N } through video acquisition, wherein I (I) is a frame of the video, and I is a number of each frame in time sequence;

step (2) designing a construction method of a pixel stream to be restored: according to frame ordering, locating pixels I of single video each frame at same coordinate (m, n)_mn(i) Are connected in series to form an observation pixel stream I_mn＝{I_mn(i) I ∈ N as a pixel stream to be restored;

step (3) represents the single video as a matrix form of the observed pixel stream: according to the construction method in the step (2), the observation pixel streams are constructed one by one according to the coordinate sequence in each frame of the single video, and the constructed observation pixel streams are combined, so that the single video can be expressed as the observation pixel streams in a matrix formEach element in the matrix is a constructed observation pixel stream;

step (4) establishing a degradation model of the observation pixel flow: i is_mn＝DBH_mn+ E, where D is the time down-sampling matrix, B is the time-blurring matrix, used to simulate the exposure time, H_mnE is the added noise vector of Gaussian distribution;

calculating to obtain an original pixel stream H in step (5)_mnThe probability estimation equation of (1):

according to the Bayes probability statistical rule, calculating to obtain the original pixel flow H_mnIs estimated as

${\hat{H}}_{mn}=\underset{H_{mn}}{\mathrm{argmax}}P\left(H_{mn}\right|I_{mn})=\underset{H_{mn}}{\mathrm{argmax}}\&lsqb;P\left(I_{mn}\right|H_{mn})P\left(H_{mn}\right)\&rsqb;;$

Calculating to obtain an original pixel stream H_mnThe restoration formula of (2):

according to the degradation model of the observed pixel flow established in the step (4) and the original pixel flow H obtained in the step (5)_mnIs obtained by logarithmic calculation to obtain the original pixel stream H_mnIs restored to the formula

${\hat{H}}_{mn}=\underset{H_{mn}}{\arg \max }\left(\log P\right(H_{mn})-\mathrm{\&alpha;}||I_{mn}-{\mathrm{DBH}}_{mn}|{|}_2^2),$

Wherein,to restore the pixel stream, logP (H)_mn) A temporal prior information item representing the original pixel stream, α being an optimization parameter;

and (7) restoring the observation pixel stream: utilizing the original pixel stream H obtained in the step (6)_mnThe single video frequency expressed in the matrix form of the observation pixel stream obtained in the step (3) is restored one by one according to the subscript sequence, and each I in the matrix_mnAre restored by the following steps to obtain a restored pixel stream

Step (7-1) of determining a temporal prior model P (H) of the observed pixel stream_mn) And judging it to be of Gaussian or Laplace type:

from the observed pixel stream I_mnDetermining a temporal prior model P (H) of the observed pixel stream in a data-driven manner_mn) Is of Gaussian typeOr laplace typeWherein a high pass operator of the signal is represented;

step (7-2) is based on the original pixel stream H_mnThe restitution of (c) derives the partial derivative equation:

determining a temporal prior model P (H)_mn) Then, based on the original pixel stream H obtained in step (6)_mnOf the restoration type ${\hat{H}}_{mn}=\underset{H_{mn}}{\arg \max }\left(\log P\right(H_{mn})-\mathrm{\&alpha;}||I_{mn}-{\mathrm{DBH}}_{mn}|{|}_2^2),$

Deriving a partial derivative equation of $\frac{\&part;}{\&part;H_{mn}}\&lsqb;\log P\left(H_{mn}\right)\&rsqb;+{\mathrm{\&alpha;B}}^T{D}^T(I_{mn}-{\mathrm{DBH}}_{mn})=0;$

Step (7-3) for observing pixel stream I_mnPerforming linear interpolation to obtain pixel streamAs an iteration initial value;

step (7-4) iterative solution is carried out on the partial derivative equation obtained in step (7-2) by using a conjugate gradient method, and a restored pixel stream is obtained

And (8) combining to obtain a restored video: the restored pixel stream restored one by one in the step (7)The restored video H is combined in a matrix form as output, and the restored video H is expressed in a matrix form as

2. The method of claim 1, wherein the method for restoring single video frame rate based on pixel stream and temporal prior information comprises: the video acquisition in the step (1) can use a camera to shoot a moving scene to obtain a single video.

3. The method of claim 1, wherein the method for restoring single video frame rate based on pixel stream and temporal prior information comprises: and (4) adopting a trial and error method to set the optimization parameter alpha in the step (6) for value taking.

4. The method of claim 1, wherein the method for restoring single video frame rate based on pixel stream and temporal prior information comprises: determining a temporal prior model P (H) of the observed pixel stream in said step (7-1)_mn) Judging whether the signal is Gaussian or Laplace, and judging by a data driving method, comprising the following stepsIn the step of,

step (7-1-1) for observing the pixel stream I_mnLinear interpolation is carried out to obtain pixel stream

Step (7-1-2) of establishing a pixel streamHigh-pass plate ofGaussian prior model of $P_G\left(\mathrm{\&Gamma;}{\hat{H}}_{mn}^0\right)={\left(2{\mathrm{\&pi;\&sigma;}}_G^2\right)}^{-\frac{K}{2}}\exp \{-\frac{\left|\right|\mathrm{\&Gamma;}{\hat{H}}_{mn}^0|{|}_2^2}{2{\mathrm{\&sigma;}}_G^2}\}$

And laplacian prior model $P_L\left(\mathrm{\&Gamma;}{\hat{H}}_{mn}^0\right)={\left(2{\mathrm{\&sigma;}}_L\right)}^{-K}\exp \{-\frac{\left|\right|\mathrm{\&Gamma;}{\hat{H}}_{mn}^0|{|}_1^1}{{\mathrm{\&sigma;}}_L}\},$

Wherein σ_GAnd σ_LRespectively representing the standard deviation of a Gaussian prior model and a Laplace prior model, and K representsThe dimension of (a);

step (7-1-3) utilizes the Gauss prior model and the Laplace prior model built in step (7-1-2) to solve the problem that the prior model is not a good modelGiven the known premises, according to the maximum likelihood rule,

estimate the standard deviation ${\widehat{\mathrm{\&sigma;}}}_G=\sqrt{\frac{\left|\right|\mathrm{\&Gamma;}{\hat{H}}_{mn}^0|{|}_2^2}{K}},$ ${\widehat{\mathrm{\&sigma;}}}_L=\frac{\left|\right|\mathrm{\&Gamma;}{\hat{H}}_{mn}^0|{|}_1^1}{K};$

Determination of step (7-1-4)Andcomparing the magnitudes of the two values, ifIs greater thanThen judging that the time prior model of the observed pixel flow is Gaussian; and otherwise, judging that the time prior model of the observed pixel flow is of a Laplace type.