TWI428024B - Frame rate up conversion system and method - Google Patents

Description

System and method for increasing frame rate

The present invention relates to the improvement of the frame rate, and more particularly to a spatial interpolation process and smoothing for the inner frame.

Frame rate up conversion (FRUC) is often applied to the display of digital images (such as digital TV), which can generate one or more inner illustration frames between two adjacent original frames to enhance the display frame. The rate, for example, is increased from 60 Hz to 120 Hz or even 240 Hz. The inner frame is usually generated by interpolation techniques of motion compensation (MC). The first figure shows a block-based motion estimation/compensation technique that generates an inset frame according to the previous frame A and the current frame B. First, the action of the macroblock (MB) of the current frame B relative to the corresponding macroblock of the previous frame A is estimated. Next, the inner frame is obtained based on the motion estimation.

For the inner frame created by the block-based motion compensation, a cracked (or split) region is often generated, and there is no motion vector in the region. Furthermore, for block-based motion compensation, there is usually a side effect along the boundary between adjacent blocks. To overcome the problem of rupture areas, traditional systems or methods use line mitigation A line-buffer stores the pixels of the current block and some pixels of the previous block and the next block. For example, for an 8x8 block-based system or method, ten line buffers are needed to store the 8 lines of the current block, the last line of the previous block, and the first line of the next block. Since accessing the pixels of ten line buffers takes a lot of time, conventional systems or methods cannot be used for display of live images. In addition, ten line buffers can cause an increase in circuit area and cost.

In view of the fact that traditional systems and methods cannot effectively solve the problem of rupture zone and boundary effect, it is urgent to propose a novel system and method to produce an inner frame without rupture zone and boundary effect in an economical and effective way.

In view of the above, one of the objects of the embodiments of the present invention is to provide a system and method for improving frame rate up conversion (FRUC), which uses less buffer resources to repair and smooth the inner illustrations. frame.

According to an embodiment of the invention, the elevated frame rate system comprises a motion estimation (ME) unit and a three-line buffer based motion compensation unit. The motion estimation unit generates at least one motion vector (MV) according to the continuous frame input. The motion compensation unit generates an inner frame according to the motion vector, the reference frame and the current frame, thereby generating a frame output whose frame rate is higher than the frame rate of the frame input.

21‧‧‧Action Estimation Unit

22‧‧‧Action Compensation Unit

221‧‧‧ time inserting unit

222‧‧‧ Space Interpolation Unit

223‧‧‧Smoothing unit

2221‧‧‧ memory

2222‧‧‧Three-wire buffer

2223‧‧‧Space Interpolation Processor

31-32‧‧‧Steps

321-323‧‧‧Steps

3222-3223‧‧‧Steps

P1, p2, p3, p4, pc, b1, bc‧‧ ‧ pixels

The first figure shows an example of generating an inner frame according to the previous frame and the current frame.

Figure 2A shows a block diagram of the system for increasing the frame rate in accordance with an embodiment of the present invention.

FIG. 2B is a flow chart showing a method for increasing the frame rate according to an embodiment of the present invention.

Figure 3A shows a detailed block diagram of the motion compensation (MC) unit of the second A diagram in the embodiment of the present invention.

FIG. 3B is a detailed flowchart showing the steps of generating the inset frame in the second B diagram in the embodiment of the present invention.

Figure 4A shows a detailed block diagram of the spatial interpolation unit of the third A diagram in the embodiment of the present invention.

Figure 4B shows a detailed flow chart of the steps of interpolating the space of the third B diagram to repair the rupture zone in the embodiment of the present invention.

The fifth A and fifth B diagrams illustrate storing the last line of the previous block, the current line, and the first line of the next block in the three-line buffer.

The sixth figure shows an example of performing spatial interpolation by spatially interpolating the processor.

The seventh figure shows an example of performing smoothing.

The second A diagram shows a system block diagram of a frame rate up conversion (FRUC) according to an embodiment of the present invention. FIG. 2B is a flow chart showing a method for increasing the frame rate according to an embodiment of the present invention. The boost frame rate system mainly includes a motion estimation unit (ME) 21 and a motion compensation (MC) unit 22. In step 31, the motion estimation unit 21 receives a continuous frame input having an original frame rate (for example, 60 Hz) for generating an motion vector (motion Vector, MV) or motion vector map (MV map). In step 32, the motion compensation unit 22 (particularly a motion compensation unit based on a triple-line buffer) is based on an action vector/action vector map, a reference frame (eg, a previous frame or a subsequent frame). And the current frame to generate an inner frame, thereby producing a continuous frame output with a frame rate increase (eg, 120 Hz). In this embodiment, block-based motion compensation is employed.

The third A diagram shows a detailed block diagram of the action compensation unit 22 of the second A diagram in the embodiment of the present invention. The third B diagram shows a detailed flowchart of the inset frame generating step (step 32) in the second B diagram in the embodiment of the present invention. In the present embodiment, the motion compensation unit 22 includes a temporal interpolation unit 221, a spatial interpolation unit 222, and a smoothing unit 223. In step 321, the time interpolation unit 221 generates a time frame according to the action vector/action vector map, the reference frame and the current frame (where the reference frame and the current frame are available from the motion estimation unit 21 or the frame). Save memory to get). In view of the fact that there are often broken (or split) regions in the illustration frame during the time period caused by the block-based motion compensation, in step 322, the spatial interpolation unit 222 is used to perform spatial interpolation for the time illustration frame. To repair the ruptured area. Details on the repair of the rupture zone will be explained later in this manual. In addition, since the block-based motion compensation has a boundary effect on the boundary between the blocks, in step 323, the smoothing unit 23 is used to perform smoothing along the inter-block boundary of the illustration frame in the space to reduce Border effect. Details on block boundary smoothing will be explained later in this manual.

The fourth A diagram shows a detailed block diagram of the spatial interpolation unit 222 of the third A diagram in the embodiment of the present invention. Figure 4B shows a detailed flow chart of the step of interpolating the space of the third B diagram to repair the rupture zone (step 322) in the embodiment of the present invention. In the present embodiment, the spatial interpolation unit 222 includes a memory 2221, a three-line buffer 2222, and a spatial interpolation processor 2223. Remember The memory 2221 provides lines of some pixel blocks. The three-line buffer 2222 includes three line buffers for storing the current line to be processed, the last line of the previous block, and the first line of the next block (step 3222). Then, spatial interpolation is performed on the current line using the spatial interpolation processor 2223 according to the first line of the previous (upper end adjacent) block and the first line of the next (lower end adjacent) block (step 3223). . Since this embodiment uses only three-line buffers for spatial interpolation (and smoothing), this example can significantly reduce hardware resources and speed up interpolation (and smoothing) compared to conventional systems and methods.

Figure 5A shows an example in which the last line of the previous block N-1 is stored in the buffer 1, and the current line of the current block N is stored in the buffer 2 and the first block of the next block N+1 The line is stored in the buffer 3, and the block N-1, the block N and the block N+1 are consecutive blocks in the vertical direction of the image. For the same block N, one line below covers the contents of the buffer 2 each time. As another example shown in FIG. 5B, after processing the last line of block N, the first line of block N+1 becomes the current line. Since the current line has been previously stored in the buffer 3, it is not necessary to retrieve it from the memory 2221 again. Furthermore, the last line remaining in the block N of buffer 2 will be the last line of the previous block N. At the same time, the first line of the block N+2 is retrieved by the memory 2221 and stored in the buffer 1. For the same block N+1, the content of the buffer 3 (not the buffer 2 shown in FIG. 5A) is covered by one line at a time. Thereby, the examples shown in the fifth A diagram and the fifth B diagram can be repeatedly executed in all the blocks.

The sixth diagram shows an example of spatial interpolation by the spatial interpolation processor 2223 (step 3222). In one embodiment, spatial interpolation is performed on the pixels pc of the current line according to the pixel p1 of the last line of the previous block and the pixel p2 of the first line of the next block. For example, the value of pixel pc can be calculated from the following formula: Pc=[p1*n1+p2*n2]/(n1+n2), where n1 and n2 are the weighting of the pixels p1 and p2, respectively.

In another embodiment, spatial interpolation is performed on the pixels pc of the current line according to the following four pixels: the pixel p1 of the last line of the previous block, the pixel p2 of the first line of the next block, and the left side of the pixel The pixel p3 of the neighboring block and the pixel p4 of the adjacent block of the right side.

Next, smoothing processing is performed on the in-space illustration frame by the smoothing unit 223 (step 323). In this embodiment, low-pass filtering (LPF) is employed to smooth the block boundaries, thereby reducing the boundary effect. The seventh figure shows an example of performing smoothing. In this embodiment, the pixel bc of the current line is compared with the pixel b1 of the last line of the previous block to smooth the pixel bc of the current line. For example, the value of the smoothed pixel bc' can be calculated by the following equation: bc' = [b1 * n1 + bc * n2] / (n1 + n2), where n1 and n2 are the weighting of the pixels b1 and bc, respectively.

The above description is only the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention; all other equivalent changes or modifications which are not departing from the spirit of the invention should be included in the following Within the scope of the patent application.

222‧‧‧ Space Interpolation Unit

2221‧‧‧ memory

2222‧‧‧Three-wire buffer

2223‧‧‧Space Interpolation Processor

Claims (12)

A system for increasing a frame rate, comprising: an action estimating unit, inputting according to a continuous frame to generate at least one motion vector; and a motion compensation unit based on a three-line buffer, the motion compensation unit according to the motion vector a reference frame and a current frame to generate an inner frame, thereby generating a frame output, the frame rate being higher than the frame rate input by the frame; wherein the motion compensation unit comprises: a time Interpolating unit, according to the motion vector, the reference frame and the current frame to generate a time frame; and a spatial interpolation unit, performing spatial interpolation on the illustrated frame during the time to generate an in-space illustration The space interpolation unit includes: a memory for providing a plurality of lines of the pixel block; the three-line buffer includes three line buffers for respectively storing a current line of the current block, and a front a last line of one block and a first line of the next block; and a spatial interpolation processor, according to the stored last line of the previous block and the next block Lines to perform interpolation of the space within the current line. The system for increasing the frame rate as described in claim 1 further includes a smoothing unit that performs smoothing processing on the inset frame of the space along a periphery of the adjacent block. A system for increasing the frame rate as described in claim 1, wherein the line buffer storing the current line is covered by the next current line. The system for increasing the frame rate according to claim 1, wherein during the first period, the three line buffers comprise: a first buffer for storing the last block of the previous block N-1. a second buffer for storing the current line of the current block N; and a third buffer for storing the first line of the next N+1; wherein the block N-1, The block N and the block N+1 are consecutive blocks of an image in the vertical direction. The system for increasing the frame rate as described in claim 4, wherein during the second period, the first buffer is used to store a first line of a block N+2, and the second buffer is used to store the area. The last line of the block N, and the third buffer is used to store the current line of the block N+1, wherein the block N, the block N+1 and the block N+2 are in the vertical direction of the image Continuous block. The system for increasing the frame rate according to the first aspect of the patent application, according to one pixel of the last line of the previous block and one pixel of the first line of the next block, one of the current lines The pixels perform spatial interpolation. A method for increasing a frame rate, comprising: performing motion estimation (ME) according to continuous frame input to generate at least one motion vector (MV); and performing according to the motion vector, a reference frame, and a current frame A three-line buffer-based motion compensation to generate an inner frame, whereby a frame output is generated, the frame rate being higher than the frame rate input by the frame; wherein the motion compensation step includes: according to the action Vector, the reference frame and the current frame, inserted during the execution time to generate a moment An in-picture frame; and performing spatial interpolation on the inset frame to generate a space inset frame; wherein the spatial interpolation step comprises: providing a plurality of lines of the pixel block; and presenting a current line of the current block The last line of the previous block and the first line of the next block are respectively stored in the three line buffer; and according to the stored last line of the previous block and the next block The first line performs spatial interpolation on the current line. The method for increasing the frame rate as described in claim 7 of the patent application further includes smoothing the illustrated frame in the space along one of the adjacent blocks. The method for increasing the frame rate as described in claim 7 further includes a step in which the current line covers the contents of the line buffer storing the current line. For the method of increasing the frame rate according to the seventh aspect of the patent application, during the first period, the line buffer storing step includes: storing the last line of the previous block N-1 to a first buffer. Storing the current line of the current block N to a second buffer; and storing the first line of the next block to a third buffer; wherein the block N-1, the block N And the block N+1 is a continuous block of an image in the vertical direction. The method for increasing the frame rate as described in claim 10, during the second period, the line buffer storing step includes: storing the first line of a block N+2 to the first buffer; Saving the last line of the block N to the second buffer; and storing the current line of the block N+1 to the third buffer; wherein the block N, the block N+1 and the Block N+2 is a contiguous block of the image in the vertical direction. The method for increasing the frame rate according to claim 7, wherein the spatial interpolation step comprises: according to one of the last line of the previous block and one of the first lines of the next block. A pixel that performs spatial interpolation on a pixel of the current line.