CN112770118B - Video frame image motion estimation method and related equipment - Google Patents

Abstract

Embodiments of the present invention disclose a motion estimation method and related equipment for video frame images, wherein the method includes: acquiring a first frame image in a target video, dividing the first frame image into multiple image blocks; The first center image block of the Determining multiple second candidate image blocks in the second candidate search set; using the first center image block to perform precise search for multiple second candidate image blocks, and determining the motion estimation result of the first center image block according to the precise search result; The motion estimation result of the first frame image is determined according to the motion estimation results of the plurality of image blocks of the first frame image. The embodiments of the present invention improve the overall efficiency and accuracy of motion estimation.

Description

Video frame image motion estimation method and related equipment

technical field

The present invention relates to the technical field of video frame interpolation, and in particular, to a video frame image motion estimation method and related equipment.

Background technique

In the motion estimation process of video frame insertion, a frame of video image needs to be divided into non-overlapping image blocks. Similar matching blocks, and then calculate the motion vector.

Various motion estimation algorithms exist. The optical flow method calculates the motion vector independently for each pixel to obtain the optical flow field for motion estimation. The pixel recursion method obtains the motion vector by recursively updating the predicted value for each pixel. The motion estimation algorithm based on block matching searches the image blocks of adjacent frames in the image through the physical distance between pixels, and finds the best result according to the matching rule. The simplest block matching algorithm is the full search method, which searches for matching blocks in sequence within a given search range, such as the entire image. The 3-Dimension Recursive Search (3DRS) method is a block matching algorithm. Its current block inherits the motion vectors of adjacent blocks. According to the matching criteria, the cost value of the current block and the block corresponding to each candidate vector is calculated. , find the most similar block by comparing the cost value.

However, using the 3DRS algorithm for motion estimation of video frame images has problems such as a small search range, does not consider the overall motion of the video, and does not perform accurate search, and the 3DRS algorithm is combined with other existing algorithms for motion estimation, which is computationally complex. , motion estimation takes a long time and so on.

SUMMARY OF THE INVENTION

The embodiments of the present invention provide a motion estimation method and related equipment for video frame images, which can increase the search range of motion estimation without requiring a large amount of calculation and improve the overall efficiency and accuracy of motion estimation.

In a first aspect, an embodiment of the present invention provides a motion estimation method for a video frame image, the method includes: obtaining a first frame image in a target video, dividing the first frame image into multiple image blocks; obtaining multiple images The first center image block in the block; the first center image block is used to perform a rough search for a plurality of first candidate image blocks in the first candidate search set, and the second center image block is determined according to the rough search result, wherein the first The candidate image blocks are determined according to the image blocks in the adjacent frame images of the first frame image; the multiple second candidate image blocks in the second candidate search set are determined according to the second center image block, and the second candidate image blocks are determined according to the second center image block. The step distance between the image block and other image blocks is determined, and the other image blocks and the second center image block are located in the same frame of image; the first center image block is used to perform an accurate search for multiple second candidate image blocks, according to the precise search As a result, the motion estimation result of the first central image block is determined; the motion estimation result of the first frame image is determined according to the motion estimation results of multiple image blocks of the first frame image.

It can be seen that in this embodiment of the present application, when performing motion estimation on image blocks in a video frame image, the full search method is used to estimate the motion of each image block on the frame image, and the search process for each image block is performed twice The search is a rough search and a precise search, respectively. The first candidate search set of the rough search is determined according to the image blocks of the adjacent frame images of the frame image currently undergoing motion estimation, and the second candidate search set of the precise search is obtained based on the result of the rough search. Image blocks within a preset step size, the process expands the search range and improves the search results through two searches. In addition, the process of obtaining the search set is simple in calculation, which improves the efficiency of the search process.

In a second aspect, an embodiment of the present invention provides a motion estimation apparatus, the apparatus includes: an acquisition module for acquiring a first frame of image in a target video; a processing module for dividing the first frame of image into multiple images block, to obtain the first center image block in the multiple image blocks; the processing module is further configured to use the first center image block to perform a rough search for the multiple first candidate image blocks in the first candidate search set, according to the rough The search result determines the second center image block, wherein the first candidate image block is determined according to the image blocks in the adjacent frame images of the first frame image; the processing module is further configured to determine the second candidate search block according to the second center image block Multiple second candidate image blocks in the set, the second candidate image block is determined according to the step distance between the second center image block and other image blocks, and the other image blocks and the second center image block are located in the same frame of image; this process The module is also used to use the first central image block to perform an accurate search for a plurality of second candidate image blocks, and determine the motion estimation result of the first central image block according to the accurate search result; The motion estimation result determines the motion estimation result of the first frame image.

In a third aspect, an embodiment of the present invention provides a motion estimation device, including: a processor and a memory;

The processor is connected to a memory, wherein the memory is used for storing program codes, and the processor is used for calling the program codes to execute the motion estimation method for a video frame image according to the first aspect.

In a fourth aspect, an embodiment of the present application provides a chip system, including: a processor, where the processor is coupled to a memory, and the memory is used to store programs or instructions, and when the programs or instructions are executed by the processor, the chip system enables the above-mentioned first A method in an aspect or any possible implementation of the first aspect.

In a fifth aspect, an embodiment of the present invention provides a computer storage medium, where the computer storage medium stores a computer program, and the computer program includes program instructions, and when executed by a processor, the program instructions are executed as in the first aspect The motion estimation method of the video frame image.

In a sixth aspect, an embodiment of the present application provides a computer program product that, when the computer reads and executes the computer program product, causes the computer to execute the method in the first aspect or any possible implementation manner of the first aspect.

Description of drawings

In order to explain the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention, which are of great significance to the art For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic diagram of a full search matching method provided by an embodiment of the present application;

FIG. 2A provides a flowchart of a method for estimating motion of a video frame image according to an embodiment of the present application;

FIG. 2B is a schematic diagram of dividing an image block of a first frame of an image according to an embodiment of the present application;

2C is a schematic diagram of a process for acquiring multiple first candidate image blocks according to an embodiment of the present application;

2D is a flowchart of a motion vector clustering provided in an embodiment of the present application;

2E is a schematic diagram of a process for determining multiple second candidate image blocks according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a motion estimation apparatus according to an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a motion estimation device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

It should be understood that the terms "first", "second" and the like in the description and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes For other steps or units inherent to these processes, methods, products or devices.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present invention. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the described embodiments of the present invention may be combined with other embodiments.

First, the technical terms that may be involved in the embodiments of this application are introduced.

Motion Estimation (ME): A frame of image is divided into many non-overlapping image blocks, and the displacement of all pixels in the image block is considered to be the same, and each image block is within a given search range of the reference frame. The process of finding the most similar image block, that is, the matching block, according to certain matching criteria, and obtaining the relative offset of the spatial position of the two image blocks.

Motion Vector (MV): The relative offset between the matching block and the current block (image block used for matching search) is the motion vector.

Motion vector field: The motion vectors of all image blocks in a frame of image constitute a motion vector field.

Motion compensation: The process of reconstructing intermediate frames that did not originally exist through the original frame and motion information.

Video frame interpolation: In a video sequence, through motion estimation and motion compensation, new frames are generated to improve the temporal resolution of the video.

Interpolated frame: A new frame generated by the video interpolation method between two frames of a video sequence.

The frame interpolation process of video sequences based on motion estimation and motion compensation needs to use the information of the adjacent frames before and after to estimate the motion of the interpolated frames relative to them. The quality of the interpolated frame depends on the accuracy of the motion estimation. In order to obtain high-quality interpolated frames, it is necessary to select a motion estimation algorithm that can obtain the true motion vector field. Some of the existing motion estimation algorithms have high computational complexity and are not easy to implement, such as optical flow field; the full search method is the simplest block matching algorithm, within a given search range, such as the entire image, search for matching As shown in Figure 1, for the image block in the current frame, the matching block is searched in the search range in the reference frame and then the motion vector is obtained. The search algorithm has high precision but the calculation amount is too large; in the limited search range Searching for matching blocks reduces the computational complexity, but a given image block cannot well represent the true motion of the object.

Based on the above description, please refer to FIG. 2A. FIG. 2A provides a flowchart of a video frame image motion estimation method according to an embodiment of the present application. As shown in FIG. 2A, the method includes the following steps:

101. Obtain a first frame of image in the target video, and divide the first frame of image into multiple image blocks;

102. Obtain a first center image block in multiple image blocks;

103. Use the first center image block to perform a rough search for multiple first candidate image blocks in the first candidate search set, and determine a second center image block according to the rough search result, wherein the first candidate image block is based on the first frame Image blocks in adjacent frame images of the image are determined;

104. Determine a plurality of second candidate image blocks in the second candidate search set according to the second center image block, the second candidate image blocks are determined according to the step distance between the second center image block and other image blocks, and the other image blocks is located in the same frame image as the second central image block;

105. Use the first center image block to perform an accurate search for a plurality of second candidate image blocks, and determine the motion estimation result of the first center image block according to the precise search result;

106. Determine a motion estimation result of the first frame of image according to the motion estimation results of the multiple image blocks in the first frame of image.

In the embodiment of the present application, the target video is a video that needs to be framed, and the first frame image in the target video image is obtained, and the first frame image is a frame image with an unknown moving direction, and it is attempted to perform a video after the first frame image. Frame interpolation, even if it is said that the second frame image is the next frame image adjacent to the first frame image, in order to determine how to perform video frame interpolation between the first frame image and the second frame image, the first frame image needs to be interpolated. Performing motion estimation, specifically, acquiring a running vector corresponding to the motion of the first frame of image to the second frame of image.

The first frame of image is divided into a plurality of image blocks, please refer to FIG. 2B . FIG. 2B is a schematic diagram of dividing the first frame of image into image blocks according to an embodiment of the present application. As shown in (a) of FIG. 2B , you can is to divide the first frame image into a plurality of image blocks of the same size and shape, or, as shown in (b) in FIG. 2B , may also be to divide the first frame image into image blocks of different sizes according to pixel values, for example, for the frame image In the area with larger pixel value (darker color), divide smaller image blocks, that is, perform more detailed division, and divide larger image blocks for the area with smaller pixel value (lighter color) in the frame image, That is, a rougher division is made. This division is based on the assumption that darker areas have more detail and therefore require more detailed matching.

In the embodiment of the present application, the method of dividing the first frame of image into a plurality of image blocks of the same size and shape is taken as an example for description, and the first frame of image is divided into several rectangles of the same size according to the preset size, and A rectangle corresponds to an image block. Any one of the image blocks can be selected as the first central image block for motion estimation. As shown in (a) of FIG. 2B , the image block C is the first central image block selected in this embodiment of the present application.

Then use the first center image block to search and match the image blocks in the reference frame, determine the matching block with the highest degree of matching with the first center image block, calculate and obtain the motion vector between the first center block and the matching block, and according to the The motion vector completes the motion estimation for the first central image block. The reference frame can be a forward reference frame or a backward reference frame. The forward reference frame represents the image frame corresponding to the current frame (the image frame currently requiring motion estimation) at the previous moment of the current moment, and the backward reference frame represents the image frame corresponding to the current moment. The image frame to which the current frame may move at the next moment, the search match between the current frame and the forward reference frame indicates the motion estimation at the previous moment at the current moment, and the search match between the current frame and the backward reference frame indicates the next moment for the current moment. Motion estimation at one moment. In the embodiments of the present application, motion estimation is described by taking the search and matching with the backward reference frame as an example for description.

In the embodiment of the present application, using the first central image block to search and match the image blocks in the reference frame (the next frame image adjacent to the first frame image) specifically includes: using the first central image block A rough search of a plurality of first candidate image blocks in a candidate search set, the second central image block is determined according to the rough search result, wherein the first candidate image block is determined according to the image blocks in the adjacent frame images of the first frame image ; Use the second center image block to carry out an accurate search for a plurality of second candidate image blocks in the second candidate search set, and determine the motion estimation result of the first frame image according to the precise search result, wherein the second candidate image block is based on the first frame image. The step distance between the second center image block and other image blocks is determined, and the other image blocks and the second center image block are located in the same frame of image.

According to the above description, in the process of searching and matching the first central image block with the image blocks in the reference frame, two searches are required. The first is a rough search to determine the second central image block, and the second is a rough search. For precise search, it is used to finalize the motion estimation result. The rough search is actually to perform a matching operation between the first central image block and a plurality of first candidate image blocks in the first candidate search set, and then determine a second central image block according to the matching result. The second central image block may be a matching block with the highest matching degree with the first central image block among the first candidate image blocks, or may be an image block obtained after adjustment on the basis of the matching block. The first candidate image block is determined according to the image blocks in the adjacent frame images of the first frame image, for example, the first candidate image block may be the adjacent image blocks of the first central image block corresponding to the adjacent frame images Image block, the adjacent frame image may be the previous frame image or the next frame image of the first frame image, and the corresponding image block represents the image block corresponding to the same pixel image on different frame images at different times. Alternatively, the first candidate image block may be an image block or the like that may correspond to the first central image block predicted according to the motion vector of the image block of the previous frame on the image of the next frame.

After the second center image block is determined according to the rough search result, multiple second candidate image blocks in the second candidate search set are determined according to the second center image block, and the second center image block is the lower image block adjacent to the first frame image. An image block on one frame of image (which can be named as the second frame of image), that is to say, after a rough search, it is determined that when the first central image block is transformed from the first frame of image to the second frame of image at the next moment, Possibly at the location of the second center image patch. Further, the second center image block may not be the image block with the highest degree of matching with the first center image block on the second frame image. Therefore, multiple second candidate image blocks on the second frame image may be further acquired, and then Obtain the matching block with the highest matching degree with the first central image block from the second candidate image block, and determine the final possible position of the first central image block when the first frame image is transformed to the second frame image at the next moment .

Each image block in the first frame of image can be selected as the first central image block, and then the above method is used for motion estimation, and finally the motion estimation result of the first frame of image is determined according to the motion vector of each image block, and then the motion estimation result is determined. The position for video interpolation.

It can be seen that, in the embodiment of the present application, by acquiring the first frame image in the target video image, and then acquiring the first center image block in the first frame image, a rough search and a precise search are performed for the first center image block, wherein the rough search is performed. The image block used for the matching search during the search is determined according to the image blocks in the adjacent frame images of the first frame image, the image block used for the matching search during the precise search is determined according to the second center image block determined by the rough search, and the second center image block determined by the rough search. The matching result between the second center image block and the image block within the preset step size is determined, and finally the image block with the highest matching degree with the first center image block is obtained as the final matching block, and then the motion estimation result of the first center image block is determined. This process widens the search scope and improves the search results with two searches. In addition, the process of obtaining candidate image blocks is simple in calculation, which improves the efficiency of the search process.

Optionally, the method further includes determining a plurality of first candidate image blocks in the first candidate search set. Please refer to FIG. 2C for details. FIG. 2C is a schematic diagram of a process of acquiring multiple first candidate image blocks according to an embodiment of the present application. As shown in (b) of FIG. 2C , f _n is the first frame of image, that is, is the current frame, in which the image block C is the first central image block obtained. First, on the current frame f _n , the first image block S1 and the second image block S2 on the left and above the image block C have completed motion estimation search, and the space is closer to the image block C than other blocks, so the prediction vectors of S1 and S2 are selected as the spatial prediction vectors, that is to say, as shown in (c) in FIG. 2C, according to the corresponding corresponding The predicted motion vector of , obtains its respective third image block S1 ′ and fourth image block S2 ′ on the second frame image f _n+1 as two first candidate image blocks. The second frame image f _n+1 is the next frame image adjacent to f _n .

In addition, as shown in (a) of FIG. 2C, f _n-1 is the previous frame image of the current frame f _n , which is called the third frame image, and the right adjacent image block C' in f _n-1 is selected The fifth image block T1 and the lower adjacent sixth image block T2 are used as temporal prediction vectors, where the image block C′ is the image block corresponding to the image block C in f _n on the previous frame of image. According to the continuity of object motion in the image, assuming that image block C has the same motion vector as image blocks T1 and T2, the corresponding image blocks of T1 and T2 in f _n are T1' and T2', which are also the right side of image block C. For the adjacent image blocks and the adjacent image blocks below, T1' and T2' keep the motion vectors of T1 and T2 on the previous frame of image, and the seventh image block T1" and the corresponding seventh image block T1" on f _n+1 can be obtained. Eight image blocks T2" are also used as the other two first candidate image blocks.

In addition, the image block Zero corresponding to the same position of the image block C in f _n+1 is obtained as a zero-point image block. The same position means that the coordinate position of the image block Zero in f _n+1 is the same as the coordinate position of the image block C in f _n . The zero point image block is also a first candidate image block.

For areas with complex motion, some image blocks in the temporal and spatial neighborhoods can also be added to improve the frame insertion quality, so some global motion vectors in the temporal domain can be added as a supplement to the first candidate search set.

The global motion vector mainly considers that for areas with complex motion, the first candidate image block determined according to the adjacent image blocks of the first central image block is used for searching, and the required variables may not be captured. Therefore, by increasing the global motion vector method To compensate, the global motion vector corresponding to the previous frame of image is used for the rough search process of the first central image block.

The global motion vector represents the motion vector of the category with a larger number after the motion vector of each image block in the whole frame image is classified by a certain method. In order to find the motion vectors for most of the image blocks, the motion vectors in the previous frame of image can be classified.

Please refer to FIG. 2D for details. FIG. 2D is a flow chart of a motion vector clustering provided in an embodiment of the present application. As shown in FIG. 2D , the global motion vector of the third frame image (the previous frame image of the first frame image) The acquisition process includes the following steps:

201. Initialize the center of the class as a zero vector, the number of classes k=1, the selected class x=0, and a distance D is set;

202. Determine whether the number of selected categories x is less than 4;

203. If so, calculate the distance d between the motion vector in a row of image blocks and the center vector of the existing class, and compare the distance d with the distance D;

204. If the distance d is less than or equal to the distance D, the motion vector of the image block and the existing class are divided into a class, and after adding a new motion vector, the number of motion vectors in the class +1, expressed as count++;

205. If the distance d is greater than the distance D, classify the motion vector of the image into a new class, the number of classes is k++; determine whether the total number of classes k is less than K, if so, execute step 206; if not, then stop clustering;

207, add a new motion vector in a class, recalculate the average value of the motion vector in the current class, and update the average value to a new class center vector mv_c, and execute step 202;

208. After the clustering is completed, obtain the target class whose number of motion vectors in the class exceeds 1/8 of the number of image blocks in each row of the frame image, and the number of motion vectors ranks in the top four;

209. Obtain the target center vector mv_c of the target class, and determine the global image block on f _n+1 according to the target center vector and the first center image block.

According to the above description, the motion vector of each image block of the third frame image is clustered to obtain several classification categories, and the total number of the several classification categories cannot be greater than K. In this embodiment of the present application, K can be set to a maximum of 16. In some cases, if the third frame image is an image with a smaller area, K may be a value smaller than 16. Select the target class in which the number of motion vectors exceeds 1/8 of the number of image blocks in each line of the third frame image, and the number of motion vectors ranks in the top four, and the number of image blocks in each line is the horizontal division of the third frame image. The number of image blocks. To obtain the top four target classes, the center vectors of the four target classes can be obtained, and according to the four center vectors, the four global image blocks corresponding to the first center image block on the second image frame can be obtained, and also Image patches in the set are searched for the first candidate.

According to the above description, the first candidate search set includes a total of 9 first candidate image blocks. After obtaining all the first candidate image blocks in the first candidate search set, the first center image block and multiple first candidate image blocks are respectively subjected to a matching operation (that is, a rough search) to obtain a degree of matching with the first center image block. The highest image block, as the second center image block. The matching algorithm between images includes gray-based matching algorithm and feature-based matching algorithm, and feature-based matching method includes mean absolute deviation (MAD) algorithm, sum of absolute differences (SAD) algorithm. ) algorithm, sum of squared difference (SSD) algorithm, etc. The SAD algorithm is used in the embodiments of the present application, and the first central image block and the first candidate image block are determined by calculating the sum of the absolute values of the differences between the pixel values in the pixel blocks corresponding to each of the first central image block and the first candidate image block. The matching degree of the image block, the larger the SAD value, the lower the matching degree of the two image blocks. This process has the advantage of low computational complexity and can ensure high rough search efficiency. Finally, the image block with the highest matching degree with the first central image block among the first candidate image blocks is obtained as the second central image block.

In an optional case, in order to give a weighted average to the continuously obtained actual data and the original predicted data to make the prediction result closer to the actual situation, when calculating the matching degree between the first candidate image block and the first central image block, According to the SAD algorithm, the sum of the absolute values of the difference between the two pixel values (the first absolute error sum) is obtained, and then the smoothing term smoothness1 is added, that is, the first absolute error sum and the smoothing term are summed to obtain the first absolute error sum and the smoothing term. An estimated value, the smoothing term is specifically:

smoothness1=∑|mv _c1 -mv _neighbor | (1)

where mv _c1 represents the motion vector of the first candidate image block, mv _neighbor represents the motion vector of 8 adjacent image blocks of the first central image block on the first frame image, and the motion vectors of S1 and S2 are spatial prediction vectors, The motion vectors of T1 and T2 are temporal prediction vectors, and the motion vector of Zero is a zero vector.

Finally, the matching degree between the first central image block and the first candidate image block is determined according to the first estimated value, and the larger the first estimated value is, the lower the matching degree is.

It can be seen that in this embodiment of the present application, when selecting the first central image block to roughly search for multiple first candidate image blocks in the corresponding first candidate search set, the temporal prediction vector combined with the adjacent first central image block is considered The image block obtained from the image block, and the image block obtained by combining the spatial prediction vector with the adjacent image blocks of the first center image block, and the image block corresponding to the same position of the first center image block on the next frame of image, and the image block according to the previous frame The global image patch determined by the global motion vector of the image. This process fully considers the candidate image blocks that may correspond to the adjacent image blocks of the first central image block in various situations, and also considers the global motion vector of the previous frame image, which improves the obtained representative of the first candidate search set. This improves the reliability of rough search results.

After the rough search is completed, the precise search is further performed. According to the previous process, the multiple second candidate image blocks in the second candidate search set used in the precise search are determined by the steps between the second central image block and other image blocks. Long distance ok. The second central image block and other image blocks are all located on the second frame image, the step distance represents the straight-line distance between image blocks, and the step distance between adjacent image blocks is 1.

Specifically, please refer to FIG. 2E. FIG. 2E is a schematic diagram of a process for determining multiple second candidate image blocks according to an embodiment of the present application. The second central image block is an image block C0, as shown in (a) of FIG. 2E. As shown, the image block whose step distance from the second center image block is the first distance is obtained as the first step image block, and the first distance can be, for example, 1, that is, the image whose step distance from the second center image block is 1 The block is used as the first long image block, specifically the image block marked as 1 in the figure, including 8 image blocks adjacent to the image block C0. Then, the 8 image blocks are respectively matched with the image block C0, and the image matching algorithm described above can also be used to obtain the first-length image block with the highest matching degree with the image block C0, which is the third center image block 1- C0, and then obtain the image block with a step distance of 1 from the image block 1-C0 as the second step size image block, which is the image block marked 1' in the figure. In fact, the second step size image block is the same as the first step size. The long image blocks overlap, and the overlapping part is used as the first step length image block, and is not repeatedly recorded as the second step length image block.

Then, as shown in (b) of FIG. 2E , an image block whose step distance from the second central image block is a second distance is obtained as a third step size image block, and the second distance can be 3, that is, the image block with the second distance from the second center image block is acquired The image block with the step size distance of 3 in the center image block is used as the third step size image block, which is specifically the 8 image blocks marked as 3 in the figure. The 8 image blocks are respectively matched with the image block C0, and the third step size image block with the highest matching degree with the image block C0 is obtained as the fourth center image block 3-C0, and then the step with the image block 3-C0 is obtained. The image block whose long distance is 3 is used as the fourth step size image block, which is the image block marked 3' in the figure. Similarly, the fourth step size image block includes image blocks that overlap with the third step size image block, and the overlapping portion is used as the third step size image block, and is not repeatedly recorded as the fourth step size image block.

The second center image block obtained according to the above method, the first step image block, the second step image block, the third step image block and the fourth step image block form a plurality of second image blocks in the second candidate search set candidate image blocks. Then use the first central image block to perform an accurate search for multiple second candidate image blocks, that is, perform image matching between the second central image block and each image block in the multiple second candidate image blocks, and use the matching The method can be, for example, the gray-scale-based matching algorithm or the feature-based matching algorithm described above, in particular, the SAD algorithm can be used, which can improve the matching efficiency. Finally, the image block with the highest matching degree with the first central image block among the plurality of second candidate image blocks is obtained, which is called the final matching block.

Optionally, when calculating the degree of matching between the second candidate image block and the first central image block, the sum of the absolute values of the pixel value differences between the two (the second absolute error sum) is obtained according to the SAD algorithm. and, the smoothing term smoothness2 and the distance difference distance can also be added, that is, the second absolute error sum, the smoothing term and the distance difference are summed to obtain the second estimated value, and the corresponding formula is:

smoothness2=∑|mv _c2 -mv _neighbor | (2)

distance=max(|x|,|y|) (3)

Where mv _c2 represents the motion vector of the second candidate image block, mv _neighbor represents the motion vector of 8 adjacent image blocks of the first central image block on the first frame image, and the motion vector of the second candidate image block can also be time Prediction vector or spatial prediction vector. The distance difference (first distance difference) is the maximum value of the absolute value of the offset in the x or y direction of the motion vector between the first central image block and the second candidate image block.

Finally, the matching degree between the first central image block and the second candidate image block is determined according to the second estimated value, and the larger the second estimated value is, the lower the matching degree is.

It can be seen that, in the embodiment of the present application, when selecting the first central image block to precisely search for multiple second candidate image blocks in the corresponding second candidate search set, the step distance is the first distance and the second distance respectively. The second step size image block and the fourth step size image block are obtained, and then the exact search results are obtained according to the matching results of the first center image block and the second step size image block and the fourth step size image block. This process can be further The search range in the motion estimation process is increased, the number of search image blocks is increased, and the process of acquiring candidate image blocks is simple, thereby improving the efficiency and accuracy of motion estimation.

Based on the description of the above embodiments of the motion estimation method for video frame images, an embodiment of the present invention further discloses a motion estimation apparatus. Referring to FIG. 3 , FIG. 3 is a schematic structural diagram of a motion estimation apparatus provided by an embodiment of the present invention. The motion estimation apparatus 300 includes:

An acquisition module 301 is used to acquire the first frame of image in the target video;

a processing module 302, configured to divide the first frame image into multiple image blocks, and obtain the first central image block in the multiple image blocks;

The processing module 302 is further configured to use the first central image block to perform a rough search for a plurality of first candidate image blocks in the first candidate search set, and determine the second central image block according to the rough search result, wherein the first candidate image The block is determined according to the image blocks in the adjacent frame images of the first frame image;

The processing module 302 is further configured to determine a plurality of second candidate image blocks in the second candidate search set according to the second center image block, and the second candidate image blocks are based on the step distance between the second center image block and other image blocks. It is determined that the other image blocks and the second center image block are located in the same frame of image;

The processing module 302 is further configured to use the first center image block to perform an accurate search for multiple second candidate image blocks, and determine the motion estimation result of the first center image block according to the precise search result; The motion estimation result of the block determines the motion estimation result of the first frame image.

It can be seen that in this embodiment of the present application, when performing motion estimation on image blocks in a video frame image, the full search method is used to estimate the motion of each image block on the frame image, and the search process for each image block is performed twice The search is a rough search and a precise search, respectively. The first candidate search set of the rough search is determined according to the image blocks of the adjacent frame images of the frame image currently undergoing motion estimation, and the second candidate search set of the precise search is obtained based on the result of the rough search. Image blocks within a preset step size, the process expands the search range and improves the search results through two searches. In addition, the process of obtaining the search set is simple in calculation, which improves the efficiency of the search process.

Optionally, the processing module 302 is further configured to determine multiple first candidate image blocks in the first candidate search set, and is specifically configured to:

Obtain the first motion vector of the first image block and the second motion vector of the second image block, and determine, according to the first image block and the first motion vector, the corresponding motion of the first image block when the first frame of image moves to the second frame of image The third image block is to determine, according to the second image block and the second motion vector, the fourth image block corresponding to the second image block when the first frame of image moves to the second frame of image, wherein the first image block is the first central image block The left adjacent image block of , the second image block is the adjacent image block above the first central image block, and the second frame image is the next frame image adjacent to the first frame image;

Obtain the third motion vector of the fifth image block moving from the position on the third frame image to the current position, and the fourth motion vector of the sixth image block moving from the position on the third frame image to the current position, and according to the fifth The image block and the third motion vector determine the seventh image block corresponding to the fifth image block when the image of the first frame moves to the image of the second frame, and the image of the first frame is determined to move to the second frame according to the sixth image block and the fourth motion vector The eighth image block corresponding to the sixth image block in the image, wherein the fifth image block is the adjacent image block on the right side of the first central image block, and the sixth image block is the adjacent image block below the first central image block, The third frame image is the previous frame image adjacent to the first frame image;

Obtain the zero point image block corresponding to the first center image block in the second frame image, and the coordinate position of the zero point image block in the second frame image is the same as the coordinate position of the first center image block in the first frame image;

Obtaining the global motion vector of the third frame of image, the global motion vector is obtained according to the motion vector clustering corresponding to the plurality of image blocks divided into the third frame of image;

According to the global motion vector, the corresponding image block of the first central image block on the second frame image is obtained as the global image block;

The third image block, the fourth image block, the seventh image block, the eighth image block, the zero point image block and the global image block form a plurality of first candidate image blocks in the first candidate search set. Optionally, the rough search result is a matching result between the first center image block and a plurality of first candidate image blocks, and determining the second center image block according to the rough search result, including:

An image block with the highest matching degree with the first central image block among the first candidate image blocks is determined as the second central image block.

Optionally, the processing module is further configured to determine the degree of matching between the first central image block and the plurality of first candidate image blocks, specifically for:

Calculate and obtain the first absolute error sum of the first central image block and the first candidate image block, where the first absolute error sum is a plurality of pixels in the first central image block and the first The sum of the absolute values of the difference between the pixel values between the multiple pixels to be matched in the candidate image block;

Summing the first absolute error sum and a first smoothing term to obtain a first estimated value, and the first smoothing term is based on the motion vector of the first candidate image block and the neighbor of the first central image block The difference value sum determination of the motion vector of the image block;

The degree of matching between the first central image block and the first candidate image block is determined according to the first estimated value.

Optionally, the processing module 302 is further configured to determine multiple second candidate image blocks in the second candidate search set, and is specifically configured to:

acquiring an image block whose step distance from the second center image block is the first distance as the first step length image block;

Determine the third center image block according to the matching result between the second center image block and the first-length image block;

acquiring an image block whose step distance from the third central image block is the first distance as a second step image block;

acquiring an image block whose step distance from the second center image block is a second distance as a third step image block, and the second distance is greater than the first distance;

Determine the fourth center image block according to the matching result of the second center image block and the third step size image block;

acquiring an image block whose step distance from the fourth central image block is the second distance as the fourth step image block;

The second central image block, the first-sized image block, the second-sized image block, the third-sized image block and the fourth-sized image block form a plurality of second candidate image blocks in the second candidate search set.

Optionally, the precise search result is a matching result between the first central image block and a plurality of second candidate image blocks, and the motion estimation result of the first central image block is determined according to the precise search result, including:

The final matching block is determined according to the precise search result, and the final matching block is the image block with the highest matching degree with the first central image block among the plurality of second candidate image blocks;

The final motion vector between the first central image block and the final matching block is obtained by calculation as the motion estimation result of the first central image block.

Optionally, the processing module 302 is further configured to determine the degree of matching between the first central image block and the plurality of second candidate image blocks, and is specifically configured to:

Calculate and obtain a second absolute error sum of the first central image block and the second candidate image block, where the second absolute error sum is a plurality of pixels in the first central image block and the second absolute error sum The sum of the absolute values of the difference between the pixel values between the multiple pixels to be matched in the candidate image block;

Summing the second absolute error sum, the second smoothing term, and the first distance difference to obtain a second estimated value, the second smoothing term is based on the motion vector of the second candidate image block and the first The difference sum of the motion vectors of the adjacent image blocks of the central image block is determined, and the first distance difference is determined according to the x or y direction offset of the motion vector between the first central image block and the second candidate image block. The maximum value of the absolute value of the shift is determined;

The degree of matching between the first central image block and the first candidate image block is determined according to the first estimated value.

It is worth noting that, for the specific function implementation manner of the motion estimation apparatus, reference may be made to the description of the above motion estimation method, which will not be repeated here. Each unit or module in the motion estimation device may be combined into one or several other units or modules, respectively or all, or some of the units or modules may be further split into functionally more functional units. It is composed of multiple small units or modules, which can realize the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above units or modules are divided based on logical functions. In practical applications, the function of one unit (or module) can also be implemented by multiple units (or modules), or the functions of multiple units (or modules) are implemented by one unit. (or module) implementation.

Each device and product described in the above embodiments includes modules/units, which may be software modules/units or hardware modules/units, or may be partly software modules/units and partly hardware modules/units. For example, for each device or product of an application or integrated chip, each module/unit included in the product may be implemented by hardware such as circuits, or at least some modules/units may be implemented by software programs, which run on the integrated processing inside the chip. The remaining part of the modules/units can be implemented by hardware such as circuits; for each device and product corresponding to or integrated with chip modules, each module/unit included can be implemented by hardware such as circuits. The unit may be located in the same piece of the chip module (such as a chip, a circuit module, etc.) or in different components, and at least part/unit may be implemented by a software program that runs on the remaining part of the integrated processor module inside the chip module/ The unit can be implemented in hardware such as circuits; for each device or product that corresponds to or integrates with the terminal, the modules/units it contains can be implemented in hardware such as circuits, and different modules/units can be located in the same component in the terminal (for example, Chips, circuit modules, etc.) or different components, or at least some of the modules/units can be implemented in the form of software programs, which run on the processor integrated inside the terminal, and the remaining sub-modules/units can be implemented in hardware such as circuits.

Based on the descriptions of the foregoing method embodiments and apparatus embodiments, an embodiment of the present invention further provides a motion estimation device. Please refer to FIG. 4 , which is a schematic structural diagram of a motion estimation device provided by an embodiment of the present invention. As shown in FIG. 4 , the above-mentioned motion estimation apparatus 300 can be applied to the motion estimation apparatus 400 , and the motion estimation apparatus 400 may include: a processor 401 , a network interface 404 and a memory 405 , in addition, the motion estimation apparatus 400 Also included: a user interface 403, and at least one communication bus 402. Among them, the communication bus 402 is used to realize the connection and communication between these components. The user interface 403 may include a display screen (Display) and a keyboard (Keyboard), and the optional user interface 403 may also include a standard wired interface and a wireless interface. Optionally, the network interface 404 may include a standard wired interface and a wireless interface (eg, a WI-FI interface). The memory 405 may be high-speed RAM memory or non-volatile memory, such as at least one disk memory. The memory 405 can optionally also be at least one storage device located away from the aforementioned processor 401 . As shown in FIG. 4 , the memory 405 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

In the motion estimation device 400 shown in FIG. 4 , the network interface 404 can provide a network communication function; the user interface 403 is mainly used to provide an input interface for the user; and the processor 401 can be used to call the device control stored in the memory 405 An application program to implement the steps of the above-mentioned motion estimation method for video frame images.

It should be understood that the motion estimation apparatus 400 described in the embodiments of the present invention can perform the motion estimation method for the video frame image described above, and can also perform the description of the motion estimation apparatus described above, which is not repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated.

In addition, it should be pointed out here that an embodiment of the present invention further provides a computer storage medium, and the computer storage medium stores a computer program executed by the aforementioned video processing apparatus, and the computer program includes a program instruction, when the processor executes the program instruction, it can execute the description of the video processing method described above, and therefore, it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiments involved in the present invention, please refer to the description of the method embodiments of the present invention.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. During execution, the processes of the embodiments of the above-mentioned methods may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM) or the like.

The above disclosures are only preferred embodiments of the present invention, and of course, the scope of the rights of the present invention cannot be limited by this. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims (14)

1. A method for motion estimation of a video frame image, the method comprising:

acquiring a first frame image in a target video, and dividing the first frame image into a plurality of image blocks;

obtaining a first central image block of the plurality of image blocks;

determining a plurality of first candidate image blocks in the first candidate search set, specifically including:

acquiring a first motion vector of a first image block and a second motion vector of a second image block, determining a third image block corresponding to the first image block when the first frame image moves to the second frame image according to the first image block and the first motion vector, determining a fourth image block corresponding to the second image block when the first frame image moves to the second frame image according to the second image block and the second motion vector, wherein the first image block is a left adjacent image block of a first central image block, the second image block is an upper adjacent image block of the first central image block, and the second frame image is a next frame image adjacent to the first frame image;

acquiring a third motion vector of a fifth image block moving from a position on a third frame image to a current position and a fourth motion vector of a sixth image block moving from the position on the third frame image to the current position, determining a seventh image block corresponding to the fifth image block when the first frame image moves to a second frame image according to the fifth image block and the third motion vector, determining an eighth image block corresponding to the sixth image block when the first frame image moves to the second frame image according to the sixth image block and the fourth motion vector, wherein the fifth image block is a right adjacent image block of the first central image block, the sixth image block is a lower adjacent image block of the first central image block, and the third frame image is a previous frame image adjacent to the first frame image;

acquiring a zero-point image block corresponding to the first center image block in the second frame image, wherein the coordinate position of the zero-point image block in the second frame image is the same as the coordinate position of the first center image block in the first frame image;

acquiring a global motion vector of the third frame of image, wherein the global motion vector is obtained according to motion vector clusters corresponding to a plurality of image blocks divided by the third frame of image;

acquiring a corresponding image block of the first central image block on the second frame image according to the global motion vector to be used as a global image block;

the third image block, the fourth image block, the seventh image block, the eighth image block, the zero image block and the global image block form a plurality of first candidate image blocks in the first candidate search set;

performing a rough search for the plurality of first candidate image blocks in a first candidate search set by using the first center image block, and determining a second center image block according to a rough search result, wherein the first candidate image block is determined according to image blocks in adjacent frame images of the first frame image;

determining a plurality of second candidate image blocks in a second candidate search set according to the second central image block, wherein the second candidate image blocks are determined according to step length distances between the second central image block and other image blocks, and the other image blocks and the second central image block are located in the same frame of image;

adopting the first central image block to perform accurate search aiming at the plurality of second candidate image blocks, and determining a motion estimation result of the first central image block according to an accurate search result;

and determining the motion estimation result of the first frame image according to the motion estimation results of the plurality of image blocks of the first frame image.

2. The method of claim 1, wherein the coarse search result is a match between the first central image block and the first candidate image blocks, and wherein determining a second central image block according to the coarse search result comprises:

and determining the image block with the highest matching degree with the first central image block in the first candidate image blocks as the second central image block.

3. The method according to claim 2, further comprising determining a degree of matching between the first center image block and the plurality of first candidate image blocks, specifically comprising:

calculating to obtain a first absolute error sum of the first central image block and the first candidate image block, wherein the first absolute error sum is a sum of absolute values of differences between a plurality of pixel points in the first central image block and a plurality of pixel points to be matched in the first candidate image block;

summing the first absolute error sum and a first smooth term to obtain a first estimated value, wherein the first smooth term is determined according to the difference sum of the motion vector of the first candidate image block and the motion vector of the adjacent image block of the first central image block;

and determining the matching degree between the first central image block and the first candidate image block according to the first estimated value.

4. The method according to any one of claims 1 to 3, wherein the determining a plurality of second candidate image blocks in a second candidate search set according to the second center image block specifically includes:

acquiring an image block with a first distance from the step length of the second central image block as a first step length image block;

determining a third central image block according to the matching result of the second central image block and the first step length image block;

acquiring an image block with the step length distance from the third central image block being the first distance as a second step length image block;

acquiring an image block with a step length distance from the second central image block as a second distance which is greater than the first distance as a third step length image block;

determining a fourth central image block according to the matching result of the second central image block and the third step length image block;

acquiring an image block with the step length distance from the fourth central image block being the second distance as a fourth step length image block;

the second center image block, the first step length image block, the second step length image block, the third step length image block and the fourth step length image block form a plurality of second candidate image blocks in the second candidate search set.

5. The method of claim 4, wherein the exact search result is a matching result of the first central image block and the plurality of second candidate image blocks, and wherein determining the motion estimation result of the first central image block according to the exact search result comprises:

determining a final matching block according to the accurate searching result, wherein the final matching block is the image block with the highest matching degree with the first central image block in the plurality of second candidate image blocks;

and calculating to obtain a final motion vector between the first central image block and the final matching block as a motion estimation result of the first central image block.

6. The method according to claim 5, further comprising determining a degree of matching of the first center image block with the plurality of second candidate image blocks, specifically comprising:

calculating to obtain a second absolute error sum of the first central image block and the second candidate image block, wherein the second absolute error sum is the sum of absolute values of differences of pixel values between a plurality of pixel points in the first central image block and a plurality of pixel points to be matched in the second candidate image block;

summing the second sum of absolute errors with a second smooth term determined according to a sum of differences between the motion vector of the second candidate image block and the motion vector of an adjacent image block to the first center image block, and a first distance difference determined according to a maximum value of an absolute value of x or y direction offset of the motion vector between the first center image block and the second candidate image block, to obtain a second estimated value;

and determining the matching degree between the first central image block and the second candidate image block according to the second estimated value.

7. A motion estimation apparatus, characterized in that the apparatus comprises:

the acquisition module is used for acquiring a first frame image in a target video;

the processing module is used for dividing the first frame image into a plurality of image blocks to obtain a first central image block in the plurality of image blocks;

the processing module is further configured to determine a plurality of first candidate image blocks in the first candidate search set, and is specifically configured to:

acquiring a first motion vector of a first image block and a second motion vector of a second image block, determining a third image block corresponding to the first image block when the first frame image moves to the second frame image according to the first image block and the first motion vector, determining a fourth image block corresponding to the second image block when the first frame image moves to the second frame image according to the second image block and the second motion vector, wherein the first image block is a left adjacent image block of a first central image block, the second image block is an upper adjacent image block of the first central image block, and the second frame image is a next frame image adjacent to the first frame image;

acquiring a third motion vector of a fifth image block moving from a position on a third frame image to a current position and a fourth motion vector of a sixth image block moving from the position on the third frame image to the current position, determining a seventh image block corresponding to the fifth image block when the first frame image moves to a second frame image according to the fifth image block and the third motion vector, determining an eighth image block corresponding to the sixth image block when the first frame image moves to the second frame image according to the sixth image block and the fourth motion vector, wherein the fifth image block is a right adjacent image block of the first central image block, the sixth image block is a lower adjacent image block of the first central image block, and the third frame image is a previous frame image adjacent to the first frame image;

acquiring a zero-point image block corresponding to the first center image block in the second frame image, wherein the coordinate position of the zero-point image block in the second frame image is the same as the coordinate position of the first center image block in the first frame image;

acquiring a global motion vector of the third frame of image, wherein the global motion vector is obtained according to motion vector clusters corresponding to a plurality of image blocks divided by the third frame of image;

acquiring a corresponding image block of the first central image block on the second frame image according to the global motion vector to be used as a global image block;

the third image block, the fourth image block, the seventh image block, the eighth image block, the zero image block and the global image block form a plurality of first candidate image blocks in the first candidate search set;

the processing module is further configured to perform a coarse search on a plurality of first candidate image blocks in the first candidate search set by using the first center image block, and determine a second center image block according to a coarse search result, where the first candidate image block is determined according to image blocks in adjacent frame images of the first frame image;

the processing module is further configured to determine, according to the second central image block, a plurality of second candidate image blocks in a second candidate search set, where the second candidate image block is determined according to step length distances between the second central image block and other image blocks, and the other image blocks and the second central image block are located in the same frame of image;

the processing module is further configured to perform an accurate search for the plurality of second candidate image blocks by using the first central image block, and determine a motion estimation result of the first central image block according to an accurate search result; and determining the motion estimation result of the first frame image according to the motion estimation results of the plurality of image blocks of the first frame image.

8. The apparatus of claim 7, wherein the coarse search result is a matching result between the first center image block and the first candidate image blocks, and wherein determining a second center image block according to the coarse search result comprises:

and determining the image block with the highest matching degree with the first central image block in the first candidate image blocks as the second central image block.

9. The apparatus according to claim 8, wherein the processing module is further configured to determine a degree of matching between the first center image block and the plurality of first candidate image blocks, and is specifically configured to:

calculating to obtain a first absolute difference sum of the first central image block and the first candidate image block, wherein the first absolute difference sum is a sum of absolute values of differences of pixel values between a plurality of pixel points in the first central image block and a plurality of pixel points to be matched in the first candidate image block;

summing the first absolute error sum and a first smooth term to obtain a first estimated value, wherein the first smooth term is determined according to the difference sum of the motion vector of the first candidate image block and the motion vector of the adjacent image block of the first central image block;

and determining the matching degree between the first central image block and the first candidate image block according to the first estimation value.

10. The apparatus according to any of claims 7 to 9, wherein the processing module is further configured to determine a plurality of second candidate image blocks in the second candidate search set, and specifically configured to:

acquiring an image block with a first distance from the step length of the second central image block as a first step length image block;

determining a third central image block according to the matching result of the second central image block and the first step length image block;

acquiring an image block with the step length distance from the third central image block being the first distance as a second step length image block;

acquiring an image block with a second distance from the step length of the second central image block as a third step length image block, wherein the second distance is greater than the first distance;

determining a fourth central image block according to the matching result of the second central image block and the third step length image block;

acquiring an image block with the step length distance from the fourth central image block being the second distance as a fourth step length image block;

the second center image block, the first step length image block, the second step length image block, the third step length image block and the fourth step length image block form a plurality of second candidate image blocks in the second candidate search set.

11. The apparatus of claim 10, wherein the exact search result is a matching result of the first central image block and the plurality of second candidate image blocks, and wherein the determining the motion estimation result of the first central image block according to the exact search result comprises:

determining a final matching block according to the accurate searching result, wherein the final matching block is the image block with the highest matching degree with the first central image block in the plurality of second candidate image blocks;

and calculating to obtain a final motion vector between the first central image block and the final matching block as a motion estimation result of the first central image block.

12. The apparatus according to claim 11, wherein the processing module is further configured to determine matching degrees of the first center image block and the plurality of second candidate image blocks, and is specifically configured to:

calculating to obtain a second absolute error sum of the first central image block and the second candidate image block, wherein the second absolute error sum is the sum of absolute values of differences of pixel values between a plurality of pixel points in the first central image block and a plurality of pixel points to be matched in the second candidate image block;

summing the second sum of absolute differences with a second smooth term determined according to the sum of differences between the motion vector of the second candidate image block and the motion vector of the adjacent image block to the first central image block, and a first distance difference determined according to the maximum value of the absolute values of x-or y-direction offsets of the motion vectors between the first central image block and the second candidate image block, to obtain a second estimated value;

and determining the matching degree between the first central image block and the second candidate image block according to the second estimated value.

13. A motion estimation device, characterized by comprising: a processor and a memory;

the processor is coupled to a memory, wherein the memory is configured to store program code and the processor is configured to invoke the program code to perform the video frame image motion estimation method of any of claims 1-6.

14. A computer storage medium, characterized in that it stores a computer program comprising program instructions which, when executed by a processor, perform a video frame image motion estimation method according to any one of claims 1 to 6.

Images