JP2016048810A - Frame rate conversion device and frame rate conversion method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide frame rate conversion device and method capable of solving a problem that an output picture may become an unnatural picture by merely reducing the number of frames when an output picture of a low frame rate is generated from an input picture of a high frame rate.SOLUTION: A frame rate conversion device for converting an input picture having a frame rate to an output picture having a lower frame rate than the frame rate of the input picture has a dividing unit for performing area division on frames constituting an input picture, a vector calculator for calculating a motion vector of each area from plural frames constituting an input picture, a frame selector for selecting one or plural frames constituting an input picture used to generate one frame constituting an output picture on the basis of an output frame rate and the magnitude of the motion vector of each area, and a frame generator for generating each area of the frame constituting the output image by using the frame selected every area.SELECTED DRAWING: Figure 2

Description

  The invention of the present application relates to a frame rate conversion device and a conversion method that further suppress the reduction of the output video quality even when the output video frame rate is lower than the input video frame rate.

  Conventionally, various frame rate conversions have been studied. As a typical example, there is a conversion method between 24 fps and 30 fps as a technique for maintaining compatibility of frame rates between movies and TVs, for example.

  Patent Document 1 discloses a technique for increasing the illuminance of a frame by adding a plurality of frames when the illuminance at the time of shooting is not sufficient. At that time, a motion vector is calculated for each region into which the frame is divided, and the corresponding portions of a plurality of frames are added together for a portion having a large motion. On the other hand, for the portion with small movement, the same portion of the same frame is overlapped and added. This increases the illuminance of the image.

  In Patent Document 2, in order to efficiently encode video, the amount of video encoding is reduced by thinning out frames using motion vectors during video encoding.

JP 2006-310784 A JP 2007-295142 A

  If the frame rate of the input video signal is simply thinned out, the video displayed by the output video signal may not be displayed smoothly. This is because the frame is simply thinned out, so that the difference between the frames may become large, so that the viewer feels an unnatural image.

  The present disclosure relates to a frame rate conversion apparatus and a frame rate conversion method in which an output video is displayed as a more natural video when the frame rate of the output video signal is lower than the frame rate of the input video signal. I will provide a.

  A frame rate conversion device according to the present disclosure is a frame rate conversion device that converts a frame rate of an input video into an output video having a frame rate lower than the frame rate of the input video, and divides a frame constituting the input video into regions. An output video is configured based on a division unit, a vector calculation unit that calculates a motion vector of each region from a plurality of frames constituting the input video, an output frame rate, and the size of the motion vector for each region. A frame selection unit that selects one or a plurality of frames of the input video used to generate one frame for each region, and each region of the frames that constitute the output video using the frame selected for each region A frame generation unit for generating

  The frame rate conversion device or the frame rate conversion method according to the present disclosure enables an output video to be displayed as a more natural video.

The figure which shows the system configuration example demonstrated in this Embodiment 1. The figure which shows the function structural example of the frame rate conversion apparatus demonstrated in this Embodiment 1. FIG. The figure which shows the example of a relationship between the frame rate of the video signal input into the frame rate conversion apparatus demonstrated in this Embodiment 1, and the frame rate of the video signal to output. The flowchart which shows the process example of the frame rate conversion apparatus demonstrated in this Embodiment 1. The figure which shows the example of a relationship between the flame | frame input by the frame rate conversion apparatus demonstrated in this Embodiment 1, and the flame | frame output. The figure which shows the comparative example with the video frame which the frame rate conversion apparatus demonstrated in this Embodiment 1 outputs, and a prior art The graph which shows the example of a relationship between the motion vector at the time of the frame rate conversion apparatus demonstrated in this Embodiment 1 combining a flame | frame, and a synthetic | combination number of sheets. The graph which shows another example of the relationship between the motion vector at the time of the frame rate conversion apparatus demonstrated in this Embodiment 1 combining a flame | frame, and a synthetic | combination number of sheets. The graph which shows the example of relationship between the time when the frame rate conversion apparatus demonstrated in this Embodiment 1 synthesize | combines a frame, and the change of the weighting of a synthesis | combination. The graph which shows another example of the relationship between the time when the frame rate conversion apparatus demonstrated in this Embodiment 1 synthesize | combines a frame, and the change of the weight of a synthesis | combination.

  Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of already well-known matters and repeated descriptions for substantially the same configuration may be omitted. This is to avoid the following description from becoming unnecessarily redundant and to facilitate understanding by those skilled in the art.

  The inventor (s) provides the accompanying drawings and the following description in order for those skilled in the art to fully understand the present disclosure, and is intended to limit the subject matter described in the claims. Not what you want.

(Embodiment 1)
FIG. 1 is a diagram illustrating an example of a system configuration in the present embodiment. The system includes a high-speed camera 100, a frame rate conversion device 110, and a video display device 120.

  The high-speed camera 100 captures a subject at a higher frame rate. Here, the high speed may be anything that can be captured at a frame rate higher than the frame rate displayed by the video display device 120. In the present embodiment, the following description will be made assuming that the high-speed camera 100 captures an object at 240 fps as an example.

  The frame rate conversion apparatus 110 receives a video signal captured by the high speed camera 100 as an input. The frame rate conversion device 110 outputs a video signal obtained by converting the video signal to a frame rate lower than the frame rate of the input video signal.

  The video display device 120 displays the video signal output from the frame rate conversion device 110 to the viewer. In the present embodiment, as an example, video display device 120 displays an input video signal at 60 fps.

  As described above, the frame rate conversion apparatus 110 uses 240 fps as the input video signal and 60 fps as the output video signal, and this embodiment will be described below.

  FIG. 2 is a block diagram illustrating a functional configuration example of the frame rate conversion apparatus 110 described in the present embodiment. The frame rate conversion apparatus 110 includes a video memory 200, a dividing unit 210, a motion vector calculating unit 220, a frame selecting unit 230, and a frame generating unit 240.

  The video memory 200 temporarily holds a plurality of frames of an input video signal. The video memory 200 outputs a held video frame in response to a request from a division unit 210, a frame generation unit 240, and the like which will be described later.

  The dividing unit 210 takes out one frame from the video memory 200 and divides the video plane of the frame into a plurality of areas. The dividing unit 210 may divide the image plane by dividing the area into a predetermined size of N × M (N and M are natural numbers) or according to the contents of the input image. It is also possible to dynamically change the dividing method.

  The motion vector calculation unit 220 reads from the video memory 200 other frames that are temporally mixed with the frame divided by the dividing unit 210. The motion vector calculation unit 220 calculates a motion vector for each region divided by the division unit 210 using the frame to be divided and other frames read from the video memory 200.

  The frame selection unit 230 determines the number of input frames to be used for generating one output frame for each area according to the motion vector for each divided area calculated by the motion vector calculation 220. Basically, the number of frames to be used is increased as the calculated motion vector is larger. Conversely, the smaller the calculated motion vector size, the smaller the number of frames used.

  The frame generation unit 240 reads out the corresponding frame from the video memory 200 according to the number of frames to be used determined by the frame selection unit 230 for each divided region of the frame, and is designated for each corresponding divided region of the frames. One output frame is generated using the frame.

  FIG. 3 illustrates the relationship between the frame rate of the input video to the frame rate conversion apparatus 110 and the frame rate of the output video to be output. In FIG. 3, the horizontal axis represents the time axis, and the vertical axis represents the relationship with the video signal.

  In this embodiment, as an explanation example, the input video signal is 240 fps and the output video signal is 60 fps. Therefore, in FIG. 3, it can be seen that the number of frames of the input video signal is four times that of the output video signal.

  Originally, since the input video signal has a frame rate of 240 fps, when a moving image is particularly displayed at 240 fps, the viewer can view a smoother video. If one frame is simply selected from four consecutive 240 fps images, the number of frames is simply reduced to ¼ to generate a 60 fps image, and 3/4 of the information amount originally possessed by 240 fps is deleted. The Rukoto. Therefore, the generated 60 fps video may be unnatural for the viewer due to the information being deleted. The generated 60 fps video has a time difference of 3 frames between one frame and the following frame, and the amount of change for each frame increases. On the other hand, each frame has a smaller amount of information (than a normal picture taken at 60 fps). Such a point may cause a viewer to feel an unnatural image.

  In the present embodiment, the quality of a low frame rate video generated by improving the frame rate conversion to improve the above points is further improved.

  FIG. 4 is an example of a flowchart showing the flow of processing performed by each functional component in the frame rate conversion apparatus 110 described in FIG.

  (Step S401) The dividing unit 210 divides the frame surface into a plurality of regions by a predetermined method for the frame n of the input video signal.

  (Step S402) The motion vector calculation unit 220 calculates a motion vector for each region of the frame n divided by the dividing unit 210. Specifically, the motion vector calculation unit 220 reads the frame n + 1, the frame n + 2, and the frame n + 3 that are temporally subsequent to the frame n from the video memory 200. The motion vector calculation unit 220 calculates a motion vector of the region using these frames.

  In this embodiment, the motion vector calculation unit 220 calculates a motion vector using three frames following the frame n that is a target for calculating the motion vector, but the invention of the present application is not limited to this. Not. Frame n-1, temporal frame n-2, etc. that are temporally prior to frame n may be used. Furthermore, it is possible to use both frames before and after time.

  Furthermore, in the above embodiment, the motion vector calculation unit 220 calculates motion vectors using a total of four frames n to n + 3, but the invention of the present application is not limited to this. A larger number of frames may be used, or a smaller number of frames may be used.

  (Step S403) The frame selection unit 230 compares the magnitude of the motion vector calculated by the motion vector calculation unit 220 with a predetermined threshold T1. When the magnitude of the motion vector is larger than the threshold value T1, the subsequent processing is moved to step S405. On the contrary, when the magnitude of the motion vector is equal to or smaller than the threshold value T1, the subsequent processing is moved to step S404.

  In the present embodiment, the frame selection unit 230 has a different processing flow based on the magnitude of the motion vector, but the invention of the present application is not limited to this. Not only the magnitude of the motion vector but also the direction of motion may be considered. That is, the frame selection unit 230 may use any index as long as it determines the number of input frames used to generate subsequent output frames using the motion vector.

  (Step S404) When the frame selection unit 230 determines that the magnitude of the motion vector is equal to or less than the threshold value T1, the frame generation unit 240 uses the current processing target frame n as it is as an output frame for the region. Will be used. That is, in the area of frame n, the contents of the area of frame n are adopted as they are.

  (Step S405) If the frame selection unit 230 determines that the size of the motion vector is larger than the threshold T1, the frame generation unit 240 is used not only to calculate the current processing target frame n but also the motion vector. A new image is generated for the region using four frames of other frames n + 1, n + 2, and n + 3.

  (Step S406) When the processing in steps S402 to S405 has been completed for all the divided areas of frame n, the frame rate conversion apparatus 110 ends the processing. On the other hand, when there is an unprocessed area, the frame rate conversion apparatus 110 performs the process from step S402 on the unprocessed area.

  FIG. 5 is a diagram showing an example in which a part for combining a plurality of frames (frames n to n + 3) and a part using only frame n are determined for each frame region according to the flowchart of FIG.

  5A, the frame n includes a background portion 500, a person portion 510, and a ball portion 520.

  The background portion 500 is determined as having no motion by the motion vector calculation unit 220.

  The human part 510 is determined to be relatively small (threshold value T1 or less) by the motion vector calculation unit 220 although there is movement.

  The ball portion 520 is moved by the motion vector calculation unit 220 and its size is determined to be relatively large (greater than the threshold value T).

  FIG. 5B is a diagram illustrating a frame generated as an output frame by the frame generation unit 240. In FIG. 5B, in the region 530 including the background portion 500 and the person portion 510 in FIG. 5A, only the frame n is used for generating an output frame.

  A region 540 in FIG. 5B is a region including the ball portion 520 in FIG. In the region 540, an output frame is generated using the frame n, the frame n + 1, the frame n + 2, and the frame n + 3. Therefore, in the area 540, the balls shown in the frame n, the frame n + 1, the frame n + 2, and the frame n + 3 are displayed in an overlapping manner.

  In this way, for a portion where the magnitude of the movement is larger than a predetermined magnitude, one frame is generated using a plurality of frames. Thereby, when a video is converted from a high frame rate to a low frame rate by frame rate conversion, a portion with motion is generated by synthesizing a plurality of frames, and thus is displayed redundantly. This overlapped display is expressed as an “afterimage” representation of the moving part. Therefore, even a low-rate video can be viewed as a more natural video for the viewer.

  FIG. 6 shows the video of the frame rate conversion device acquired in the above description (FIG. 6A) and the video when the frame rate conversion is performed by the conventional method (FIGS. 6B and 6C). FIG.

  In FIG. 6A, since the region 600 (ball portion) with a large motion vector is generated by combining a plurality of frames of the input high frame rate video, the subject (ball) is synthesized. Are displayed multiple times. When a viewer views a video having such a frame, the viewer can feel an afterimage on a plurality of subjects and feel a moving subject as a natural video.

  In FIG. 6B, an output video is generated by thinning out frames so as to convert from an input high frame rate video to a low frame rate. In this case, all the subjects displayed in the frame display the frame for a longer time than the shooting (generation) time per frame generated when the high frame rate is generated. When a viewer views such a video, there is a possibility that a moving part may be felt as an unnatural video.

  In FIG. 6C, the entire frame of the output video is generated by combining a plurality of frames of the input video. In this case, not only a portion having a motion but also a portion having a small motion is synthesized from a plurality of input video frames. When an output frame is generated by combining multiple frames for a subject with small or no motion, the generated subject may be affected by errors that occur between the multiple frames, resulting in an image with unclear outlines. . Therefore, the sharpness of the outline of the entire video after frame rate conversion may be lowered.

  FIG. 7 is a diagram illustrating a method in which the frame generation unit 240 generates an output frame in accordance with the instruction content of the frame selection unit 230. In the graph of FIG. 7, the horizontal axis indicates the magnitude of the motion vector calculated by the motion vector calculation unit 220, and the vertical axis indicates the number of frames used when generating the frame.

  In the example of FIG. 7, when the magnitude of the motion vector is larger than the predetermined threshold T1, an output frame is generated using M input frames for the area where the motion vector is calculated, and conversely, the threshold T1 In the following cases, it means that an output frame is generated using only one input frame.

  The example of FIG. 8 is a diagram showing another example of FIG. In FIG. 8, thresholds are set to T1, T2... TM (T1 <T2 <... <TM), and when the magnitude of the motion vector is T1 or less, one input frame is When the magnitude of the motion vector is greater than T1 and less than or equal to T2, the number of input frames used for the frame to be generated is changed according to the magnitude of the motion vector, such as two input frames. Also good. In this case, it is general that the number of input frames to be used increases as the size of the motion vector increases.

  FIG. 9 is a diagram showing respective composition rates (blend rates) of input frames when the frame generation unit 240 generates an output frame using one or a plurality of frames according to the instruction content of the frame selection unit 230. It is. In the graph of FIG. 9, the horizontal axis represents the intra-frame space axis, and the vertical axis represents the weighting of the frame synthesis process.

  The frame generation unit 240 generates an output frame using only the frame n (one input frame) when it is equal to or less than the intra-frame time axis X0. When the intra-frame time axis is larger than X0, the frame generation unit 240 generates an output frame by combining a plurality of frames (frames n to n + 3 in the example of FIG. 9) selected by the frame selection unit 230. At this time, the frame generation unit 240 sets the weight (blend rate) of each frame to a ratio as shown in the example of FIG. 9 to generate a frame. In the example of Fig. 9, it means that frames n to n + 3 are combined with almost equal weight.

  The example of FIG. 10 shows an example in which the blend ratio changes with the intra-frame space axis. When the intra-frame space axis is X0-Δ or less, only one frame is used. When the space axis in the frame is larger than X0 + Δ, frames n to n + 3 are combined with a substantially equivalent blend rate. When the intra-frame space axis is greater than X0−Δ and less than or equal to X0 + Δ, the frame generation unit 240 generates an output frame using each frame at a blend rate that linearly interpolates the two blend rates. .

  In this case, since the blend ratio of the frames changes smoothly as the intra-frame space axis changes, the viewer can view the frame rate converted video as a more natural video.

  The configuration of the frame rate conversion apparatus described above is summarized as follows. The dividing unit divides the frame surface of the video frame constituting the input video into a plurality of regions. In each region divided by the dividing unit, the motion vector calculating unit calculates a motion vector in the region using the frame and a temporally previous or subsequent frame. The frame selection unit selects the number of frames of the input video that the subsequent frame generation unit uses when generating the frame, based on the magnitude of the motion vector calculated by the motion vector calculation unit. Specifically, the number of frames to be used may be selected by comparing the size with a predetermined reference value, or the number of frames to be used substantially in proportion to the size. It is also possible to increase the amount of The frame generation unit generates a frame for output using the frame selected by the frame selection unit. At this time, the frame generation unit generates a video having a frame rate lower than the frame rate of the input video. When generating an output frame using the frame selected by the frame selection unit, the frame generation unit may change the blend ratio of frames according to other conditions. One output frame is generated by repeating the above contents in all areas divided by the dividing unit.

  As described above, the generated output video has a lower frame rate than the frame of the input video. When the frame rate is converted from a high frame rate to a low frame rate, the amount of information as video is generally reduced, so that the video may be difficult to see or unnatural for the viewer. However, in the case of the contents described in this embodiment, output frames are generated using a plurality of input video frames in a moving part, so that information loss due to a decrease in frame rate is suppressed. It becomes possible. In addition, since the amount of information that is lost by lowering the frame rate is low for a portion with little motion such as a stationary subject, there is no sense of incongruity even if the frame rate of the output frame is reduced. As described above, in this embodiment, even if frame rate conversion is performed from a high frame rate to a low frame rate, an output video can be a more natural video.

  In the present embodiment, a case has been described in which the above processing is implemented as hardware using the functional configuration shown in FIG. 2, but the invention of the present application is not limited to this. By implementing the processing flowchart shown in FIG. 4 and the like by a program, the contents described in the present embodiment can also be realized as a frame rate conversion method.

  In this case, the processing flow described in FIG. 4 and the like is performed by the CPU, and the above frame rate conversion method is realized in a hardware device including the video memory 200 and a memory for realizing the processing flow. Is possible.

  As described above, the embodiments have been described as examples of the technology in the present disclosure. For this purpose, the accompanying drawings and detailed description are provided.

  Accordingly, among the components described in the accompanying drawings and the detailed description, not only the components essential for solving the problem, but also the components not essential for solving the problem in order to illustrate the above technique. May also be included. Therefore, it should not be immediately recognized that these non-essential components are essential as those non-essential components are described in the accompanying drawings and detailed description.

  Moreover, since the above-mentioned embodiment is for demonstrating the technique in this indication, a various change, replacement, addition, abbreviation, etc. can be performed in a claim or its equivalent range.

  The technology in the present application can be used in a video processing device that performs video frame rate conversion, a video playback device, and the like.

DESCRIPTION OF SYMBOLS 100 High-speed camera 110 Frame rate conversion apparatus 120 Image | video display apparatus 200 Image | video memory 210 Dividing part 220 Motion vector calculation part 230 Frame selection part 240 Frame generation part

Claims (6)

A frame rate conversion device for converting a frame rate of an input video into an output video having a frame rate lower than the frame rate of the input video,
A dividing unit that divides a frame constituting the input video;
A vector calculation unit for calculating a motion vector of each region from a plurality of frames constituting the input video;
Based on the output frame rate and the size of the motion vector for each area, one or more frames of the input video used to generate one frame constituting the output video are set for each area. A frame selector to select;
Using a frame selected for each region, a frame generation unit for generating each region of the frame constituting the output video;
A frame rate conversion device comprising: The frame selection unit has a larger number of frames to select for a region with a large motion vector than a number of frames to select for a region with a small motion vector.
The frame rate conversion apparatus according to claim 1. When the number of selected frames is different between adjacent regions, the frame generation unit smoothly changes the weight of the selected frame near the boundary of the region,
The frame rate conversion apparatus according to claim 1 or 2. A frame rate conversion device for converting a frame rate of an input video into an output video having a frame rate lower than the frame rate of the input video,
A division step of dividing a frame constituting the input video;
A vector calculation step of calculating a motion vector of each region from a plurality of frames constituting the input video;
Based on the output frame rate and the size of the motion vector for each area, one or more frames of the input video used to generate one frame constituting the output video are set for each area. A frame selection step to select;
A frame generation step of generating each area of the frame constituting the output video using the frame selected for each area;
A frame rate conversion method comprising: In the frame selection step, the number of frames to be selected for a region having a large motion vector is greater than the number of frames to be selected for a region having a small motion vector.
The frame rate conversion method according to claim 4. In the frame generation step, when the number of selected frames is different between adjacent regions, the weight of the selected frame smoothly changes in the vicinity of the boundary of the region.
The frame rate conversion method according to claim 4 or 5.