JP2009239698A - Video image converting device, and video image converting method - Google Patents

Abstract

To realize video conversion with little image quality degradation while searching for a motion vector only once even when video correlation is low.
A motion vector search unit 5 obtains a first motion vector MV from a frame of an input video. The resolution conversion unit 3 performs resolution conversion processing using the first motion vector. The motion vector conversion unit 6 converts the first motion vector into a pair of second motion vectors MVc according to the frame rate conversion condition. The frame rate conversion unit 4 creates a new frame using a continuous frame and a pair of second motion vectors, and inserts them between the frames. As the pair of second motion vectors MVc, a backward motion vector starting from the past frame and a forward motion vector starting from the current frame are used.
[Selection] Figure 1

Description

  The present invention relates to a video conversion apparatus and video conversion method for converting the resolution and frame rate of a moving image.

  As quality enhancement techniques for moving images, there are resolution conversion for spatially increasing the number of pixels and frame rate conversion for increasing the number of frames in terms of time. Regarding resolution conversion, not only a simple pixel interpolation method, but also synthesizing a plurality of video frames adjacent in the time direction using motion vectors between frames, so that one video having a resolution higher than the input video can be obtained. There is a technique for outputting (see, for example, Patent Document 1). Hereinafter, this processing is referred to as “super-resolution processing”.

  Regarding frame rate conversion, not only a simple method of repeatedly outputting the same frame, but also a technique for searching for a motion vector from an input video signal, generating a new insertion frame based on the searched motion vector, and inserting it between the frames (See, for example, Patent Document 2).

  Furthermore, regarding the motion vector used when converting and re-encoding the moving image encoding method, the motion vector of the video before conversion is reused as it is, or the motion vector in the video after conversion is searched and used. (See, for example, Patent Document 3).

JP 2007-324789 A Japanese Patent Application Laid-Open No. 11-112939 Japanese Patent Laid-Open No. 2005-236484

  In general, when video conversion processing using motion vectors is performed in combination, it is necessary to search independently for the motion vectors used for each processing. For example, even when the resolution conversion described in Patent Document 1 and the frame rate conversion described in Patent Document 2 are combined, it is necessary to search for a motion vector at each stage because the configuration of adjacent video frames is different. If one motion vector is used as it is as the other motion vector, a large deterioration in image quality occurs when the correlation of the video is low.

  However, motion vector search in video processing has a heavy processing load. Therefore, performing motion vector search twice for video processing for one frame complicates the processing and increases the circuit scale and cost of the processing circuit.

  With the technique described in Patent Document 3, when the correlation between motion vectors is high, the motion vector can be searched once by using the motion vector before conversion as the motion vector after conversion. . However, if the correlation is low, the converted motion vector is searched separately, and the processing load remains the same.

  In view of the above problems, an object of the present invention is to realize video conversion with little image quality degradation while searching for a motion vector only once even when video correlation is low.

  The present invention is a video conversion apparatus that performs resolution conversion and frame rate conversion on an input video, a resolution conversion unit that performs resolution conversion on the input video, and a frame that converts the frame rate of the video after resolution conversion A rate conversion unit, a motion vector search unit that obtains a first motion vector from consecutive frames of the input video, and a pair of frames by converting the first motion vector according to the frame rate conversion condition performed by the frame rate conversion unit. A motion vector conversion unit for generating the second motion vector. The resolution conversion unit performs resolution conversion processing using the input video and the first motion vector, and the frame rate conversion unit generates a new frame using the continuous frame after the resolution conversion and the pair of second motion vectors. Create and insert between successive frames.

  Here, the motion vector conversion unit, as a pair of second motion vectors, the backward motion vector from the corresponding pixel of the past frame of the input video to the pixel of the frame to be inserted and the frame of the frame to be inserted from the corresponding pixel of the current frame of the input video. A forward motion vector toward the pixel is created, and the frame rate conversion unit creates a new frame using the past frame and the backward motion vector, and the current frame and the forward motion vector.

  The present invention is a video conversion method for performing resolution conversion and frame rate conversion on an input video, a step of obtaining a first motion vector from consecutive frames of the input video, and a first motion on the input video. A step of performing resolution conversion using a vector; a step of converting a first motion vector to create a pair of second motion vectors according to a frame rate conversion condition; and a continuous frame after resolution conversion; Creating a new frame using the pair of second motion vectors, and inserting the frame between successive frames to convert the frame rate.

  According to the present invention, high-quality video can be obtained without increasing the circuit scale in video conversion processing including resolution conversion and frame rate conversion.

  A video conversion apparatus according to the present invention includes a resolution conversion unit that performs resolution conversion on an input video, a frame rate conversion unit that converts a frame rate of the video after resolution conversion, and a motion vector search unit that obtains a motion vector from the input video And a motion vector conversion unit that converts the obtained motion vector and supplies the converted motion vector to the frame rate conversion unit. Hereinafter, the embodiment will be described with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of a video conversion apparatus according to the present invention.
The video conversion apparatus 10 includes a decoding unit 1, a scaling unit 2, a resolution conversion unit 3, a frame rate conversion unit 4, a motion vector search unit 5, and a motion vector conversion unit 6. Here, it is assumed that an input video P0 encoded by a predetermined method such as MPEG-2 which is an international standard for moving images is input, and the decoding unit 1 can view this video signal (hereinafter referred to as a frame). ). The scaling unit 2 enlarges or reduces the size of the frame in the horizontal and vertical directions according to the screen size of the display device connected to the output destination of the video conversion device 10 to obtain the frame P1. The decoding unit 1 and the scaling unit 2 can be omitted depending on the input video signal and the specifications of the display device.

  The motion vector search unit 5 searches for a motion vector MV (first motion vector) between the frames from two consecutive frames P1 in units of pixels. The resolution conversion unit 3 refers to the current and past two consecutive frames P1 and the search motion vector MV obtained by the motion vector search unit 5, and performs resolution conversion processing for converting the number of pixels of the video (for example, super solution) Image processing) to generate a video P2.

  The motion vector conversion unit 6 receives the motion vector MV (first motion vector) obtained by the motion vector search unit 5, and based on the interpolation mode signal Sc designated by an external control unit or the like, a new motion vector MVc. Conversion to (second motion vector). The second motion vector consists of a pair of a forward motion vector and a backward motion vector. The frame rate conversion unit 4 refers to the continuous frames of the video P2 generated by the resolution conversion unit 3 and the pair of second motion vectors MVc obtained by the motion vector conversion unit 6, and a frame corresponding to the interpolation mode signal Sc And insert, so-called frame rate conversion, and output video P3. The interpolation mode signal Sc is a control signal for designating an operation mode of input / output frame rate conversion. The operation mode includes frame rate conversion such as 24 Hz → 60 Hz, 50 Hz → 60 Hz, and 60 Hz → 120 Hz. By the way, as a setting condition of the interpolation mode signal Sc, it is possible to set the input side and the output side to the same frame rate as 60 Hz → 60 Hz. In this case, the frame rate conversion unit 4 outputs the input frame as it is without performing the frame insertion operation.

  As described above, in the video conversion apparatus according to the present embodiment, the motion vector is used in each of the resolution conversion unit 3 and the frame rate conversion unit 4. The resolution conversion unit 3 uses the first motion vector MV searched by the motion vector search unit 5, while the frame rate conversion unit 4 uses the second motion vector MVc converted by the motion vector conversion unit 6. I did it. Therefore, since the motion vector search process which is originally required twice is only required once, the processing time and the circuit scale can be simplified.

  In this embodiment, as a configuration for performing frame rate conversion after resolution conversion, the burden of resolution conversion (for example, super-resolution processing) is reduced. However, a configuration for performing resolution conversion after frame rate conversion is performed. It is good. Even in such a case, the motion vector search process can be performed only once by providing the motion vector conversion unit.

  FIG. 2 is a diagram illustrating a specific example of frame rate conversion according to the present embodiment. The frame rate indicates the number of video frames transmitted or displayed per second, and is defined by the video standard for each broadcasting system. In the following description, the video frequency (number of vertical synchronizing signals per second) is expressed in Hz. By increasing the frame rate, it is possible to achieve smooth subject movement and improve video quality.

  (A) is a case where 60 Hz video P1 is converted into 120 Hz video P3. As the interpolation mode signal Sc, a control signal for instructing frame rate conversion from 60 Hz to 120 Hz is output to the frame rate conversion unit 4 and the motion vector conversion unit 6. The resolution converter 3 receives the 60 Hz video P1, performs video conversion based on a predetermined video processing algorithm, and outputs a 60 Hz video P2. Also, the 60 Hz video P1 is input to the motion vector search unit 5 to obtain and output a 60 Hz motion vector MV for each pixel. The motion vector conversion unit 6 converts the 60 Hz motion vector MV obtained by the motion vector search unit 5 into a 120 Hz motion vector MVc for 120 Hz video. The frame rate conversion unit 4 uses the 120 Hz motion vector MVc output from the motion vector conversion unit 6 to convert the 60 Hz video P2 output from the resolution conversion unit 3 into a 120 Hz video P3 and outputs it.

  (B) is a case where 24 Hz video P1 is converted into 60 Hz video P3. In this case, a control signal that instructs to perform frame rate conversion from 24 Hz to 60 Hz is used as the interpolation mode signal Sc. The resolution conversion unit 3 receives the 24 Hz video P1, performs resolution conversion, and outputs a 24 Hz video P2. Also, the 24 Hz video P1 is input to the motion vector search unit 5 and a 24 Hz motion vector MV is output. The motion vector conversion unit 6 converts the 24 Hz motion vector MV into a 60 Hz motion vector MVc. The frame rate conversion unit 4 converts the 24 Hz video P2 into the 60 Hz video P3 using the 60 Hz motion vector MVc, and outputs it.

FIG. 3 is a diagram illustrating an example of an internal configuration of the resolution conversion unit 3.
The input video P <b> 1 is a video that has been decoded by the preceding decoding unit 1 and then enlarged or reduced by the scaling unit 2. Each frame of the input video P1 is sequentially stored in the frame memory 31, and the frame delayed by one frame period by the frame delay unit 30 is stored in the frame memory 32. The motion vector search unit 5 refers to the two frames P1a and P1b stored in the frame memories 31 and 32, obtains a motion vector MV between the frames, and sends it to the pixel interpolation unit 33.

  The pixel interpolation unit 33 performs resolution conversion processing (pixel interpolation processing) that changes the number of pixels with reference to the two frames P1a and P1b stored in the frame memories 31 and 32 and the motion vector MV. The resolution-converted video is temporarily stored in the frame memory 34 and then sent out as output video P2 at the same frame rate as the input video P1.

  As an example of the processing of the pixel interpolation unit 33, for example, there is super-resolution processing using a motion vector. Super-resolution processing using motion vectors uses multiple frames of video, aligns frames using motion vectors determined for each pixel, and uses virtual motion with decimal precision for the original video. A sample point and its virtual pixel data are acquired. This is a method for generating video having a resolution higher than that of the original video while reducing aliasing distortion by obtaining video data to be actually displayed from the virtual pixel data.

FIG. 4 is a diagram illustrating an example of an internal configuration of the frame rate conversion unit 4.
The input video P2 is a video sent from the previous resolution conversion unit 3. Each frame of the input video P2 is sequentially stored in the frame memory 41, and a frame delayed by one frame period by the frame delay unit 40 is stored in the frame memory. The motion vector conversion unit 6 converts the first motion vector MV output from the motion vector search unit 5 to generate a second motion vector MVc for frame rate conversion. That is, the second motion vector MVc is used to generate a frame to be inserted for frame rate conversion. The conversion coefficient table 7 stores coefficients necessary for calculating the second motion vector MVc from the first motion vector MV.

  The frame interpolation unit 43 reads two frames P2a and P2b that are temporally adjacent from the frame memories 41 and 42, and uses the second motion vector MVc generated by the motion vector conversion unit 6 to generate two frames. A frame to be inserted between P2a and P2b is generated. The number of frames to be inserted between frames is uniquely determined from the relationship between the frame rates of the input / output video. For example, if the input / output frame rate is 60 Hz / 120 Hz, the number of inserted frames is one, and if it is 24 Hz / 60 Hz, the number of inserted frames is two. The pixel position in the insertion frame is determined from the relationship between the motion vector MV of the operation target pixel and the position of the insertion frame on the time axis.

  The inserted frame generated by the frame interpolation unit 43 is temporarily stored in the frame memory 44. The switching unit 45 selects and saves either the frame memory 42 in which the original video is stored or the frame memory 43 in which the inserted frame is stored in accordance with the video output cycle determined by the output side frame rate. The read frame is read and sent out as output video P3. The control unit 46 transmits an interpolation mode signal Sc designated from the outside to the motion vector conversion unit 6 and the switching unit 45, and controls the conversion of the motion vector and the output timing of the inserted frame.

  FIG. 5 is a diagram schematically showing motion vector search and motion vector conversion. Here, for simplicity, a case where one frame Fc is inserted in order to convert the frame rate to twice the original video (for example, 60 Hz → 120 Hz) is shown. The motion vector MV (first motion vector) between the current frame F2 and the past frame F1 obtained by the motion vector search unit 5 and the motion vectors MVca and MVcb (second motion vectors) converted by the frame rate conversion unit 4 ) Relationship.

  The motion vector search unit 5 searches for a motion vector MV between two consecutive frames in units of pixels. The motion vector conversion unit 6 receives the motion vector MV obtained by the motion vector search unit 5, and based on the input / output frame rate setting value specified by the interpolation mode signal Sc, a motion vector MVc for frame rate conversion. Convert to The frame rate conversion unit 4 creates a new frame Fc using the current frame F2, the past frame F1, and the motion vector MVc of the input video, and inserts it between the frames F1 and F2.

  In order to create the insertion frame Fc, the motion vector search unit 5 searches for a motion vector that passes through each pixel of the insertion frame Fc. Here, when attention is paid to a pixel Qc in the insertion frame Fc, a pixel corresponding to the pixel Qc is searched from subjects in the past frame F1 and the current frame F2. Here, it is assumed that the pixel Qa in the frame F1 corresponds to the pixel Qb in the frame F2. At that time, a pixel movement amount connecting the three points of the pixel Qa, the pixel Qc, and the pixel Qb is defined as a first motion vector MV of the pixel Qc. This motion vector MV is referred to in the processing of the resolution conversion unit 3.

  Subsequently, the motion vector conversion unit 6 converts the first motion vector MV into two second motion vectors MVca and MVcb. MVca is a motion vector from the pixel Qa of the past frame F1 to the target pixel Qc of the insertion frame Fc, and is referred to as a “rear (motion) vector”. MVcb is a motion vector from the pixel Qb of the current frame F2 to the target pixel Qc of the insertion frame Fc, and is referred to as a “forward (motion) vector”. That is, the second motion vectors MVca and MVcb can be obtained by multiplying the first motion vector MV by a conversion coefficient (time ratio from the past frame F1 and the current frame F2) indicating the insertion position of the insertion frame Fc.

  Thereafter, the frame interpolation unit 43 refers to the pixel Qa that is the starting point of the backward motion vector MVca and the pixel Qb that is the starting point of the forward motion vector MVcb, and generates a pixel Qc on the insertion frame Fc. In this case, since new pixels are generated using the pixel data of both the past frame F1 and the current frame F2, even when the correlation between the frames is low, it is possible to interpolate smoothly in video frames. Not only is the displacement of the pixel position of the interpolation frame small, but smooth image quality can be obtained even when the color and brightness of the corresponding pixel fluctuate.

  It supplements about the vector notation of FIG. Here, for conceptual explanation, the first motion vector MV is represented as a vector notation from the past frame F1 to the current frame F2. That is, it is expressed as a three-dimensional pixel movement amount with the passage of time in addition to the horizontal and vertical movements in the frame. However, since the movement of the time axis only means the positioning of the frame Fc, the motion vector used for the actual calculation is the pixel movement amount in the two-dimensional direction within the same frame. When the motion vector is converted into a two-dimensional representation in the frame, MV is equivalent to the motion vector MV 'from the pixel Qa' to the pixel Qb in the current frame F2. Further, MVca and MVcb are equivalent to a motion vector MVca ′ from the pixel Qa ″ to the pixel Qc and a motion vector MVcb ′ from the pixel Qb ″ to the pixel Qc in the insertion frame Fc.

  In the following, specific examples of motion vector conversion and frame rate conversion will be described with reference to FIGS. Here, the case where the frame rate is converted from 60 Hz to 120 Hz is taken up.

  FIG. 6 is a diagram showing an overall outline of frame rate conversion. In frame rate conversion from 60 Hz to 120 Hz, two frames are output for one frame input. For this reason, the detected first motion vector for one frame is converted into a second motion vector for two frames.

  (A) shows a 60 Hz video to be input, which is input in the order of frame F1 and frame F2. Hereinafter, the conversion operation is performed in units of one frame period Tw.

  (B) shows a 60 Hz video motion vector (first motion vector). A motion vector MV corresponding to the motion of each pixel is obtained by referring to two temporally adjacent frames and comparing pixel values in a predetermined range. In order to simultaneously refer to two adjacent frames, two frames of the current frame read from the input 60 Hz video and the previous frame read out one time before (that is, 60 Hz video delayed by one frame) are held. The 60 Hz motion vector MV is detected by comparing the frames. As a result, the motion vector MV1 is obtained at the timing of the frame F1, and the motion vector MV2 is obtained at the timing of the frame F2.

  (C) shows a state in which the 60 Hz motion vector MV is converted into a 120 Hz motion vector MVc (second motion vector). The 60 Hz motion vector MV1 of the frame F1 is converted into two 120 Hz motion vectors MVc1a and MVc1b by the conversion process of the motion vector conversion unit 6 described later. Similarly, the 60 Hz motion vector MV2 is converted into two 120 Hz motion vectors MVc2a and MVc2b.

  (D) shows a state in which the frame rate is converted from 60 Hz video to 120 Hz video. The frame rate conversion unit 4 generates 120 Hz video frames Fc1 and Fc2 from 60 Hz video frames F1 and F2 using 120 Hz motion vectors MVc1a and MVc1b. Similarly, 120 Hz motion vectors MVc2a and MVc2b are used to generate 120 Hz video frames Fc3 and Fc4 from 60 Hz video frames F2 and F3. However, since the first frames Fc1 and Fc3 of the period Tw output the frames F1 and F2 of 60 Hz video as they are, the frames Fc2 and Fc4 are newly inserted.

  FIG. 7 is a diagram showing a procedure for converting a motion vector in FIG. For example, in order to convert the 60 Hz motion vector MV1 (first motion vector) of the frame F1 into 120 Hz motion vectors MVc1a and MVc1b (second motion vectors), two coefficients KA1 and KB1 for motion vector conversion are used. The 120 Hz motion vector MVc1a (rear vector) is obtained by multiplying the 60 Hz motion vector MV1 and the first coefficient KA1. The 120 Hz motion vector MVc1b (forward vector) is obtained by multiplying the 60 Hz motion vector MV1 and the second coefficient KB1. Similarly, 120 Hz motion vectors MVc2a and MVc1b are obtained by multiplying the 60 Hz motion vector MV2 of the frame F2 by two conversion coefficients KA2 and KB2.

  FIG. 8 is a diagram showing a motion vector conversion table at the time of 60 Hz → 120 Hz conversion.

  The conversion coefficient table 7 stores conversion coefficients used for motion vector conversion, and refers to them during conversion. In the conversion procedure, the result obtained by multiplying the 60 Hz motion vector (first motion vector) by the first coefficient of the motion vector conversion coefficient is multiplied by the rear vector of the 120 Hz motion vector (second motion vector) and the second coefficient. Let it be a forward vector of a 120 Hz motion vector. Here, since two frames are generated from one frame, step no. 1 and step no. The procedure shown in 2 is repeated to obtain a 120 Hz motion vector.

  Step No. 1 corresponds to the frame Fc1 of FIG. 6, and the 60 Hz video input frame F1 is output as it is as a 120 Hz video frame, so no motion vector conversion is performed. Step No. 2 corresponds to the frame Fc2 of FIG. 6 and stores the result of multiplying the 60 Hz motion vector MV1 by the first coefficient KA1 as a 120 Hz motion vector MVc1a and the result of multiplying MV1 by the second coefficient KB1 as a 120 Hz motion vector MVc1b.

  The first coefficient and the second coefficient used for motion vector conversion are numerical values representing the difference on the time axis expressed by the 60 Hz motion vector and the 120 Hz motion vector as linear weight components. Note that the values of the first coefficient and the second coefficient are determined according to rate conversion conditions as described later.

Hereinafter, an arithmetic expression for motion vector conversion using the first coefficient and the second coefficient will be described. Here, the 60 Hz motion vector is referred to as a first motion vector group, and the 120 Hz motion vector is referred to as a second motion vector group. Consider the case of converting from a first motion vector group to a second motion vector group. At this time, it is assumed that the frame display period corresponding to one motion vector in the second motion vector group is included in the frame display period corresponding to one motion vector of the first motion vector group as the conversion source. In this case, two motion vectors of the second motion vector group are calculated from one motion vector of the first motion vector group by using the following formula (1). Here, MVn is one vector of the first motion vector group, MVcna is one vector of the second motion vector (rear vector) group between the past frame F2 and the insertion frame Fc, and MVcnb is the current frame F1 and the insertion frame One vector of the second motion vector (forward vector) group between Fc, KAm is the first coefficient, and KBm is the second coefficient. n is an input frame number and m is an insertion frame number.
MVcna = KAm × MVn, MVcnb = KBm × MVn (1)
However, step no. When the display time of the frame of the input video (60 Hz video) and the frame of the output video (120 Hz video) is the same as in 1, the input frame is output as it is, so the above formula (1) is not used.

A method for calculating the second motion vector group in the case of frame rate conversion from 60 Hz to 120 Hz will be described. A mathematical formula used to calculate the motion vector of the second motion vector group is as shown in the following mathematical formula (2). Here, MV1 is the first motion vector, MVc1a is one vector of the second motion vector group between the past frame F2 and the insertion frame Fc, and MVc1b is the second motion vector group between the current frame F1 and the insertion frame Fc. KA1 is the first coefficient, and KB1 is the second coefficient.
MVc1a = KA1 × MV1, MVc1b = KB1 × MV1 (2)
Here, the first coefficient KAm and the second coefficient KBm are set using the following formula (3), respectively. Here, TA and TB are frame intervals divided by frame insertion. TA is the length of a period overlapping with the frame display period corresponding to MVn in the frame display period corresponding to MVcna, and TB is a period overlapping with the frame display period corresponding to MVn among the frame display periods corresponding to MVcnb. Indicates the length. These are determined by the first frame rate f1 and the second frame rate f2.
KAm = TA / (TA + TB), KBm = TB / (TA + TB) (3)
Here, when f1 = 60 Hz and f2 = 120 Hz, since TA = TB, the first coefficient KA1 and the second coefficient KB1 are equal, and are as shown in the following equation (4).
KA1 = KB1 = 1/2 (4)
FIG. 9 is a diagram showing the relationship on the time axis between input / output frames and motion vectors at the time of 60 Hz → 120 Hz conversion. In this case, in a period Tw of the motion vector conversion operation, two frames of 120 Hz video are obtained from one frame of 60 Hz video, and 120 Hz motion vector of two frames from one frame of 60 Hz motion vector (first motion vector). It shows that (second motion vector) is obtained.

  The 60 Hz motion vector MV is created with reference to two adjacent frames of the 60 Hz video. For example, MV1 is generated with reference to the frame F1 and the frame F2. Motion vector conversion from the 60 Hz motion vector MV1 to the 120 Hz motion vectors MVc1a and MVc1b is obtained by multiplying the 60 Hz motion vector MV1 by conversion coefficients KA1 and KB1. This is as described in the equations (1) to (4) and FIGS.

  The procedure for obtaining a 120 Hz image from a 60 Hz image is as follows. First, the frame F1 which is the starting point of the period Tw is output as it is as a frame Fc1 of 120 Hz video. The same applies to frames Fc3 and Fc5, and 60 Hz video input frames F2 and F3 are output as they are.

  Since the frame Fc2 of the 120 Hz video does not coincide with any display time axis of the 60 Hz video, an insertion frame is created and output. In this case, reference is made to 60 Hz video frames F1 and F2 that are close to each other on the time axis, and an insertion frame Fc2 is created by a predetermined algorithm using the 120 Hz motion vectors MVc1a (rear vector) and MVc1b (forward vector). To do. Each pixel in the insertion frame Fc2 is synthesized by moving corresponding pixels in the frames F1 and F2 according to the motion vectors MVc1a and MVc1b, and determining a new pixel value. The same applies to the insertion frame Fc4, and motion vectors MVc2a and MVc2b are used.

  Thus, in frame rate conversion, video conversion is performed using 60 Hz video and 120 Hz motion vectors. A new frame Fc2 to be inserted at that time is created by using the two second motion vectors MVc1a and MVc1b with reference to the two frames F1 and F2. Therefore, even when the video correlation between the frames F1 and F2 is low, a smoothly continuous frame Fc2 can be generated. This is particularly effective when the color or brightness of the corresponding pixel varies.

  FIG. 10 is a diagram showing a video conversion processing operation flow in the present embodiment. Here, 60 Hz video is input to perform resolution conversion and frame rate conversion to 120 Hz.

  First, the conversion period FN is set from the relationship between the frame rate of the input video and the frame rate of the output video. This corresponds to the maximum number of steps in FIG. 8, and the conversion period FN from 60 Hz to 120 Hz is set to 2 (S100). The cycle counter FCNT corresponds to the current step number in FIG. 8, and FCNT is set to 1 as an initial value (S101). Two continuous frames (current frame and past frame one frame before) are input from the 60 Hz video, and the motion vector search unit 5 obtains a 60 Hz motion vector MVn (first motion vector) (S102). Using the 60 Hz motion vector MVn, the resolution conversion unit 3 performs resolution conversion of the current frame of the input video (S103).

  Thereafter, the process branches according to the value of FCNT (S104). If FCNT is 1 (no in S104), the frame subjected to the resolution conversion process in S103 is output as it is as a frame of 120 Hz video (S105).

  When FCNT is larger than 1 (Yes in S104), the motion vector conversion unit 6 reads the first coefficient KAm and the second coefficient KBm corresponding to FCNT from the motion vector conversion coefficient table. If FCNT = 2, KA1 and KB1 are read (S106). Using KA1 and KB1, the 60 Hz motion vector MVn is converted into 120 Hz motion vectors MVcna and MVcnb (second motion vector) (S107). The frame rate conversion unit 4 creates an insertion frame using two 60 Hz video frames and 120 Hz motion vectors MVcna and MVcnb (S108), converts the frame rate to a 120 Hz video frame rate, and outputs it (S109).

  A process end determination is made (S110), and if it is a continuous execution, 1 is added to the period counter FCNT (S111), and it is determined whether FCNT is larger than the conversion period FN (S112). When FCNT is larger than FN, FCNT is reset to 1 (S113), and the process returns to S102. If the process ends in S110, the video conversion processing operation ends.

  In this embodiment, the case of frame rate conversion from 24 Hz to 60 Hz will be taken up. The configuration of the video conversion device is the same as that in FIGS.

  FIG. 11 is a diagram showing an overall outline of frame rate conversion. In the frame rate conversion from 24 Hz to 60 Hz, 60/24 = 2.5 times, that is, 5 frames are output for 2 frames input. The number of motion vectors is the same as that of frames, and the detected first motion vectors for two frames are converted into second motion vectors for five frames.

  (A) shows a 24 Hz video to be input, which is input in the order of frame F1 and frame F2. Hereinafter, the conversion operation is performed in units of a period Tw of 2 frames.

  (B) shows a motion vector (first motion vector) of a 24 Hz video. A 24 Hz motion vector MV is detected by comparing two frames, the current frame read from the input 24 Hz video and the previous frame read one before. As a result, the motion vector MV1 is obtained at the timing of the frame F1, and the motion vector MV2 is obtained at the timing of the frame F2.

  (C) shows a state in which the 24 Hz motion vector MV is converted into a 60 Hz motion vector MVc (second motion vector). The 24 Hz motion vector MV1 of the frame F1 is converted into four 60 Hz motion vectors MVc1a, MVc1b, MVc1c, and MVc1d. Similarly, the 24 Hz motion vector MV2 is converted into four 60 Hz motion vectors MVc2a, MVc2b, MVc2c, and MVc2d.

  (D) shows a state in which the frame rate is converted from a 24 Hz video to a 60 Hz video. The frame rate conversion unit 4 generates a 60 Hz video frame Fc2 from the 24 Hz video frames F1 and F2 using the 60 Hz motion vectors MVc1a and MVc1b, and generates a 60 Hz video frame Fc3 using the 60 Hz motion vectors MVc1c and MVc1d. To do. Subsequently, a 60 Hz video frame Fc4 is generated from the 24 Hz video frames F2 and F3 using the 60 Hz motion vectors MVc2a and MVc2b, and a 60 Hz video frame Fc5 is generated using the 60 Hz motion vectors MVc2c and MVc2d. These generated frames Fc2, Fc3, Fc4, Fc5 are output in order. However, the first frames Fc1 and Fc6 of the period Tw output the frames F1 and F3 of 24 Hz video as they are.

FIG. 12 is a diagram illustrating a motion vector conversion table at the time of 24 Hz → 60 Hz conversion.
The conversion coefficient table 7 stores conversion coefficients used for motion vector conversion, and refers to them during conversion. In the conversion procedure, the result obtained by multiplying the 24 Hz motion vector (first motion vector) by the first coefficient of the motion vector conversion coefficient is multiplied by the rear vector of the 60 Hz motion vector (second motion vector) and the result obtained by multiplying the second coefficient. A forward vector of a 60 Hz motion vector is assumed. Since it is necessary to generate five frames from two frames, step no. 1 to step no. The 120 Hz motion vector is obtained by repeating the five procedures shown in FIG. In each step, the type of input 24 Hz motion vector and the conversion coefficient are different.

  Step No. Since 1 corresponds to the frame Fc1 of FIG. 11 and the 24 Hz video input frame F1 is output as it is as a 60 Hz video frame, no motion vector conversion is performed. Step No. 2 corresponds to the frame Fc2 of FIG. 11 and stores the result of multiplying the 24 Hz motion vector MV1 by the first coefficient KA1 as the 60 Hz motion vector MVc1a and the result of multiplying MV1 by the second coefficient KB1 as the 60 Hz motion vector MVc1b. Step No. 3 corresponds to the frame Fc3 of FIG. 11 and stores the result of multiplying the 24 Hz motion vector MV1 by the first coefficient KA2 as the 60 Hz motion vector MVc1c and the result of multiplying MV1 by the second coefficient KB2 as the 60 Hz motion vector MVc1d. Similarly, the step No. for 24 Hz motion vector MV2 is as follows. 3, step no. However, the first coefficients KA3 and KA4 and the second coefficients KB3 and KB4 used at that time are different from the values in the previous step.

  The first coefficient and the second coefficient used for motion vector conversion are numerical values representing the difference on the time axis expressed by the 24 Hz motion vector and the 60 Hz motion vector as a linear weight component. The first coefficient and the second coefficient are determined according to rate conversion conditions as will be described later.

  Hereinafter, an arithmetic expression for motion vector conversion using the first coefficient and the second coefficient will be described. Here, the 24 Hz motion vector is referred to as a first motion vector group, and the 60 Hz motion vector is referred to as a second motion vector group. Also in this case, two motion vectors of the second motion vector group can be calculated from one motion vector of the first motion vector group by using the formulas (1) and (2).

A method of calculating the second motion vector group in the case of frame rate conversion from 24 Hz to 60 Hz will be described. The mathematical formula used for calculating the second motion vector group is as shown in the following mathematical formula (5). Here, MV1 is the first motion vector, MVc1a, MVc1c, MVc2a, and MVc2c are one vector of the second motion vector (rear vector) group between the past frame F2 and the insertion frame Fc, MVc1b, MVc1d, MVc2b, and MVc2d Is one vector of the second motion vector (forward vector) group between the current frame F1 and the inserted frame Fc, KA1, KA2, KA3, KA4 are the first coefficients, and KB1, KB2, KB3, KB4 are the second coefficients. .
MVc1a = KA1 × MV1, MVc1b = KB1 × MV1
MVc1c = KA2 × MV1, MVc1d = KB2 × MV1
MVc2a = KA3 × MV2, MVc2b = KB3 × MV2
MVc2c = KA4 × MV2, MVc2d = KB4 × MV2 (5)
Here, the first coefficient KAm and the second coefficient KBm are set using the following formula (6), respectively. Here, TAm and TBm are frame intervals divided by frame insertion. TAm is the length of a period overlapping the frame display period corresponding to MVn in the frame display period corresponding to MVcna or MVcnc, and TBm is the frame display period corresponding to MVn in the frame display period corresponding to MVcnb or MVcnd. Indicates the length of the overlapping period. These are determined by the first frame rate f1 and the second frame rate f2. The values of TAm and TBm differ depending on the insertion frame number m.
KAm = TAm / (TAm + TBm),
KBm = TBm / (TAm + TBm) (6)
Here, in the case of f1 = 24 Hz and f2 = 60 Hz, the first coefficients KA1, KA2, KA3, KA4 and the second coefficients KB1, KB2, KB3, KB4 are as shown in the following formula (7), respectively. It becomes.
KA1 = 2/5, KB1 = 3/5
KA2 = 4/5, KB2 = 1/5
KA3 = 1/5, KB3 = 4/5
KA4 = 3/5, KB4 = 2/5 (7)
FIG. 13 is a diagram showing the relationship on the time axis between input / output frames and motion vectors at the time of 24 Hz → 60 Hz conversion. In this case, in the period Tw of the motion vector conversion operation, 5 frames of 60 Hz video are obtained from 2 frames of 24 Hz video, and 60 Hz motion vector for 5 frames from 24 Hz motion vector (first motion vector) for 2 frames. It shows that (second motion vector) is obtained.

  The 24 Hz motion vector MV is created with reference to two adjacent frames of a 24 Hz video. For example, MV1 is generated with reference to the frame F1 and the frame F2. MV2 is also generated with reference to adjacent frames F2 and F3.

  The motion vector conversion from the 24 Hz motion vector MV1 to the 60 Hz motion vectors MVc1a and MVc1b is obtained by multiplying the 24 Hz motion vector MV1 by conversion coefficients KA1 and KB1. The motion vector conversion from the 24 Hz motion vector MV1 to the 60 Hz motion vectors MVc1c and MVc1d is obtained by multiplying the 24 Hz motion vector MV1 by the conversion coefficients KA2 and KB2. Similarly, 60 Hz motion vectors MVc2a, MVc2b, MVc2c, and MVc2d are obtained from the 24 Hz motion vector MV2. These are as described in equations (5) to (7) and FIG.

  The procedure for obtaining a 60 Hz image from a 24 Hz image is as follows. First, the frame F1 that is the starting point of the period Tw is output as it is as a frame Fc1 of 60 Hz video. The same applies to the frame Fc6, and the 60 Hz video input frame F3 is output as it is.

  Since the frames Fc2 to Fc5 of the 60 Hz video do not coincide with any display time axis of the 60 Hz video, an insertion frame is created and output. In this case, two frames of a 24 Hz video that are close to each other on the time axis are referred to, and an insertion frame is created by a predetermined algorithm using the 60 Hz motion vector. For example, a frame Fc2 of a 60 Hz video is created using the 60 Hz motion vector MVc1a (rear vector) and MVc1b (forward vector) with reference to the frame F1 and the frame F2 that are close to each other on the time axis. The frame Fc3 is created using the 60 Hz motion vectors MVc1c and MVc1d with reference to the frames F1 and F2. The frames Fc4 and Fc5 are created using the 60 Hz motion vectors MVc2a and MVc2b and the 60 Hz motion vectors MVc2c and MVc2d with reference to the frames F2 and F3 that are close to each other on the time axis.

  Thus, in frame rate conversion, video conversion is performed using a 24 Hz video and a 60 Hz motion vector. The new frames Fc2 to Fc5 to be inserted at that time are created using two second motion vectors with reference to two adjacent frames. Therefore, even if the correlation between the images between the two frames is low, it is possible to generate smoothly continuous frames.

  FIG. 14 is a diagram showing a video conversion processing operation flow in the present embodiment. Here, 24 Hz video is input to perform resolution conversion and frame rate conversion to 60 Hz.

  First, the conversion period FN is set from the relationship between the frame rate of the input video and the frame rate of the output video. This corresponds to the maximum number of steps in FIG. 12, and the conversion period FN from 24 Hz to 60 Hz is set to 5 (S200). The cycle counter FCNT corresponds to the current step number in FIG. 8, and FCNT is set to 1 as an initial value (S201). Two continuous frames are input from the 24 Hz video, and the motion vector search unit 5 obtains a 24 Hz motion vector MVn (first motion vector) (S202). Using the 24 Hz motion vector MVn, the resolution conversion unit 3 performs resolution conversion of the current frame of the input video (S203).

  Thereafter, the process branches according to the value of FCNT (S204). If FCNT is 1 (no in S204), the frame subjected to the resolution conversion process in S203 is directly output as a 60 Hz video frame (S205).

  When FCNT is larger than 1 (Yes in S204), the motion vector conversion unit 6 reads the first coefficient KAm and the second coefficient KBm corresponding to FCNT from the motion vector conversion coefficient table. That is, KA1 to KA4 and KB1 to KB4 are read for FCNT = 2 to 5, respectively (S206). Using KAm and KBm, the 24 Hz motion vector MVn is converted into a 60 Hz motion vector MVcna (MVcnc) and MVcnb (MVcnd) (second motion vector) (S207). The frame rate conversion unit 4 creates four insertion frames by using two frames of 24 Hz video and 60 Hz motion vectors MVcna (MVcnc) and MVcnb (MVcnd) (S208), converts the frame rate to 60 Hz video, and outputs it. (S209).

  A process end determination is made (S210). If the execution is continued, 1 is added to the period counter FCNT (S211), and it is determined whether or not the FCNT is larger than the conversion period FN (S212). If FCNT is longer than the conversion cycle FN, FCNT is reset to 1 (S213), and the process returns to S202. If the process ends in S210, the video conversion processing operation ends.

  FIG. 15 is a block diagram showing an example of a video display device (television device) to which the video conversion device of the present invention is applied. The video display device of this embodiment is a television device having a function of receiving and recording a television broadcast and a function of displaying the received video on a display after performing resolution conversion and frame rate conversion.

  The configuration of the video display device 80 includes a video input / output unit 81 that receives a television broadcast, downloads video content via a network, or externally outputs video content recorded by the device, as a video input / output function. The video content storage unit 83 stores the recorded video content, and the recording / playback unit 82 controls the recording and playback operations.

  As described in the first and second embodiments, as the video conversion function of the input video signal, the resolution conversion unit 84 that increases the resolution of the video by super-resolution processing or the like and the frame that performs rate conversion of the video frame frequency It has a rate conversion unit 85, a motion vector search unit 86 and a motion vector conversion unit 87 for executing these video conversions. At that time, the resolution converter 84 uses the first motion vector searched by the motion vector search unit 86, but the frame rate converter 85 uses the second motion vector converted by the motion vector converter 87. Since it is used, the motion vector search process is only required once.

  The display unit 90 displays the converted video. The audio signal processing unit 88 processes the input audio signal according to a predetermined procedure and outputs it from the speaker 89. The user interface unit 91 accepts user operations. The control unit 92 controls the recording operation and the reproduction operation to the recording / reproducing unit 82, and sends an interpolation mode signal for converting the frame frequency to the frame rate converting unit 85 and the motion vector converting unit 87. .

  Each element of the apparatus of the present embodiment may be implemented in the form of being implemented as hardware, or may be implemented in the form of software linked to other hardware functions using a signal processing LSI. . The display unit 90 can be a video display device such as a PDP, liquid crystal, or organic EL.

  In the video display apparatus according to the present embodiment, the video frame rate can be displayed by converting from 60 Hz to 120 Hz, or from 24 Hz to 60 Hz. However, there are a plurality of standards relating to the frame rate, and any frame rate is available. It is also possible to cope with the conversion. For example, conversion from 30 Hz to 60 Hz and 50 Hz to 100 Hz is also possible. Further, the frame rate frequency can also correspond to 23.98 Hz, 29.97 Hz, 59.94 Hz, 119.88 Hz, and the like defined as video frequency standards.

  According to the video display apparatus of the present embodiment, it is possible to realize a function of inputting a video having an arbitrary specification and performing a resolution conversion and a frame rate conversion in accordance with the specification of the display unit and displaying them with a simpler configuration.

The block diagram which shows one Example of the video converter which concerns on this invention. The figure which shows the specific example of the frame rate conversion of a present Example. FIG. 3 is a diagram illustrating an example of an internal configuration of a resolution conversion unit 3. The figure which shows an example of an internal structure of the frame rate conversion part 4. FIG. The figure which shows a motion vector search and motion vector conversion typically. The figure which shows the whole outline | summary of frame rate conversion (60Hz-> 120Hz). The figure which shows the procedure which converts a motion vector. The figure which shows the table for motion vector conversion. The figure which shows the relationship on the time-axis of an input / output frame and a motion vector. The figure which shows the video conversion processing operation flow. The figure which shows the whole outline | summary of frame rate conversion (24Hz-> 60Hz). The figure which shows the table for motion vector conversion. The figure which shows the relationship on the time-axis of an input / output frame and a motion vector. The figure which shows the video conversion processing operation flow. The block diagram which shows an example of the video display apparatus to which the video converter is applied.

Explanation of symbols

  3, 84 ... resolution conversion unit, 4, 85 ... frame rate conversion unit, 5, 86 ... motion vector search unit, 6, 87 ... motion vector conversion unit, 7 ... conversion coefficient table, 10 ... video conversion device, 33 ... pixel Interpolation unit, 43 ... frame interpolation unit, 45 ... switching unit, 80 ... video display device, MV, MVc ... motion vector, KA, KB ... conversion coefficient, Sc ... interpolation mode signal.

Claims (10)

In a video conversion device that performs resolution conversion and frame rate conversion on input video,
A resolution converter for converting the resolution of the input video;
A frame rate converter for converting the frame rate of the video after resolution conversion;
A motion vector search unit for obtaining a first motion vector from successive frames of the input video;
A motion vector conversion unit that converts the first motion vector to create a pair of second motion vectors in accordance with a frame rate conversion condition performed by the frame rate conversion unit;
The resolution conversion unit performs resolution conversion processing using the input video and the first motion vector,
The frame rate conversion unit creates a new frame using the continuous frames after the resolution conversion and the pair of second motion vectors, and inserts them between the continuous frames. . The video conversion device according to claim 1,
The motion vector conversion unit inserts, as the pair of second motion vectors, a backward motion vector from a corresponding pixel of a past frame of the input video to a pixel of a frame to be inserted and a corresponding pixel of a current frame of the input video Create a forward motion vector to the pixel of the frame,
The frame rate conversion unit creates a new frame using the past frame and the backward motion vector, and the current frame and the forward motion vector. In the video conversion device according to claim 1 or 2,
A conversion coefficient table for storing a pair of conversion coefficients for generating the second motion vector according to a frame rate conversion condition and a position of a frame to be inserted;
The motion vector conversion unit reads a corresponding conversion coefficient from the conversion coefficient table and multiplies the first motion vector by the pair of conversion coefficients to convert the first motion vector into the pair of second motion vectors. Video conversion device. In the video conversion device according to claim 3,
The video conversion apparatus, wherein the conversion coefficient is a numerical value representing a difference on the time axis expressed by the first motion vector and the second motion vector as a linear weight component. The video conversion device according to claim 2,
The frame rate conversion unit moves the corresponding pixel of the past frame and the corresponding pixel of the current frame by the backward motion vector and the forward motion vector, respectively, and synthesizes each pixel value to generate a pixel value of the new frame A video conversion apparatus characterized by determining the image quality. In the video conversion method that performs resolution conversion and frame rate conversion on the input video,
Obtaining a first motion vector from successive frames of the input video;
Performing resolution conversion on the input video using the first motion vector;
Generating a pair of second motion vectors by converting the first motion vector according to a frame rate conversion condition;
Creating a new frame using the continuous frames after the resolution conversion and the pair of second motion vectors, and inserting them between the continuous frames to convert the frame rate;
A video conversion method comprising: The video conversion method according to claim 6, wherein
As the pair of second motion vectors, a backward motion vector from the corresponding pixel of the past frame of the input video to the pixel of the frame to be inserted and a forward of the corresponding pixel of the current frame of the input video to the pixel of the frame to be inserted Create a motion vector and
In the step of converting the frame rate, a new frame is created using the past frame and the backward motion vector, and the current frame and the forward motion vector. The video conversion method according to claim 6 or 7,
Depending on the frame rate conversion condition and the position of the frame to be inserted, a pair of conversion coefficients for creating the second motion vector is stored in the conversion coefficient table,
In the step of creating the second motion vector, the corresponding transform coefficient is read from the transform coefficient table, and the first motion vector is multiplied by the pair of transform coefficients to be converted into the pair of second motion vectors. A video conversion method characterized by the above. The video conversion method according to claim 8, wherein
The video conversion method, wherein the conversion coefficient is a numerical value representing a difference on the time axis expressed by the first motion vector and the second motion vector as a linear weight component. The video conversion method according to claim 7, wherein
In the step of converting the frame rate, the corresponding pixels of the past frame and the corresponding pixels of the current frame are moved by the backward motion vector and the forward motion vector, respectively, and the respective pixel values are combined to generate the new frame. A video conversion method characterized by determining a pixel value.

Images