JP2011055278A - Motion information acquiring apparatus and image processing device - Google Patents

Abstract

A motion detection algorithm that uses a high frame rate moving image and can obtain an accurate motion with a small amount of processing is provided.
A motion information acquisition apparatus includes an image sensor that captures an image having a first frame rate including a reference frame and an image having a second frame rate higher than the first frame rate including the reference frame. A motion detection unit 12 that sequentially detects a motion vector as motion information of each block constituting each image of the first frame rate using an image of the first frame rate, an image of the first frame rate, and the corresponding image And a data creation unit 13 for synchronizing the motion vectors of the respective blocks. The data creation unit 13 outputs the synchronized image and motion vector of the first frame rate.
[Selection] Figure 8

Description

  The present invention relates to a motion information acquisition device that acquires motion information from a high frame rate moving image and an image processing device that performs various image processing using the motion information.

  There are many moving image applications using motion information between frames. For example, (a) use of motion information for generating a frame rate conversion (FRC) interpolation frame, (b) use of motion information for camera shake correction, (c) motion vector for inter prediction in video coding (D) use of motion information for alignment for obtaining a high-resolution image from a plurality of frames.

  FIG. 14 is a block diagram illustrating a configuration example of a conventional image processing apparatus. FIG. 14A shows a device configuration when motion information is used to generate an FRC interpolation frame or when motion information is used as an inter prediction motion vector in moving image coding. FIG. 14C shows a device configuration when motion information is used for camera shake correction or when motion information is used as a motion vector for inter prediction in moving picture coding. FIG. 14C shows motion information for camera shake correction. FIG. 14D shows a device configuration in the case of using motion information for alignment for obtaining a high-resolution image from a plurality of frames.

  In FIG. 14, 101 is an imaging unit, 102 is a moving image encoder, 103 is a moving image decoder, 104 is an FRC unit, 105 is a camera shake correction unit, 106 is a high resolution processing unit, 107 is a still image / moving image encoder, 108 Indicates a still image / moving image decoder. In addition, in the case of FIG. 14A, the motion detection unit is provided inside the moving image encoder 102 and the FRC unit 104, and in the case of FIG. The motion detection unit is provided inside the moving image encoder 102 and the camera shake correction unit 105. In the case of FIG. 14C, the motion detection unit is provided inside the camera shake correction unit 105. In the case of FIG. Are provided inside the high resolution processing unit 106.

  As shown in FIG. 14 above, various parts in the apparatus such as the inside of the moving image encoder, the pre-processing of the moving image encoder or the still image encoder, or the post-processing of the moving image decoder according to the application. I was seeking motion vectors separately.

  By the way, with the recent increase in the frame rate of an image sensor, an accurate motion vector can be obtained with a small amount of processing by using strong inter-frame constraints during high-speed imaging. However, if a high frame rate moving image is output from a sensor, the amount of moving image data becomes enormous, so in order to output continuously for a long time, output at a low resolution or limited to only necessary information. Must be output.

  In view of this, a technique has been proposed in which motion detection is performed using a high frame rate moving image inside the image sensor, and after an image captured at a normal frame rate is corrected, the image is output from the imaging unit.

  For example, in Patent Document 1, a frame rate (30 fps or 60 fps) reference frame that uses a moving image for an application is divided into two types, that is, a higher frame rate subframe, and a plurality of local regions of a processing target frame are used. In contrast, a region in the reference frame having a high correlation is searched, a region having the maximum correlation is regarded as a movement destination of the local region, a motion vector is obtained, and a motion image is calculated using the motion vector, and a normal frame is obtained. A technique for correcting and outputting shaking for a moving image acquired at a rate is described.

JP 2004-289709 A

  However, in the technique described in Patent Document 1, the motion detection algorithm itself is a general algorithm that does not use the characteristics of the high frame rate even though the moving image of the high frame rate is used. There is a problem that the processing load is not sufficiently reduced. In the technique described in Patent Document 1, in order to suppress the processing load, for example, the upper limit of the number of local areas is set, and the processing target area in one screen is limited. In addition, although the motion vector is calculated by the imaging unit, the motion vector is used only inside the imaging unit, and therefore can be used only for a specific application (in this case, shake correction).

  The present invention has been made in view of the above circumstances, and provides a motion detection algorithm that uses a high frame rate moving image and can obtain an accurate motion with a small amount of processing. In order to use the motion vector for a plurality of purposes, it is an object to provide an output format of a moving image and a motion vector that guarantees compatibility between the imaging apparatus and an external device.

  In order to solve the above-described problem, the first technical means of the present invention provides an image having a first frame rate composed of a reference frame and an image having a second frame rate higher than the first frame rate, including the reference frame. Imaging means for imaging the image, motion information detection means for sequentially detecting motion information of each block constituting each image at the first frame rate, using the image at the second frame rate, and the first Synchronizing means for synchronizing the image of the frame rate and the motion information of each block corresponding thereto, and output means for outputting the image of the first frame rate and the motion information synchronized by the synchronizing means It is characterized by.

  According to a second technical means, in the first technical means, the motion information detecting means changes a motion search range of the target block based on motion information already detected between adjacent images of the second frame rate. It is characterized by having means.

  The third technical means is the first technical means, wherein the motion information detecting means determines whether each pixel between adjacent images of the second frame rate belongs to a still area or a moving area. Means for evaluating the motion information, and evaluation means for evaluating the correlation between the target block and the reference block using corresponding pixel values of two images. The evaluation means is obtained by the determination means. According to the region to which each pixel belongs, the evaluation value is weighted when the evaluation value is calculated.

  According to a fourth technical means, in the first technical means, the synchronizing means includes an image of the first frame rate and motion information of each block constituting at least one image of the second frame rate. And the output means outputs the image and motion information of the first frame rate synchronized by the synchronization means in a predetermined format.

  A fifth technical means is the fourth technical means, wherein the synchronizing means synchronizes an image of the first frame rate and a motion vector between reference frames constituting the image of the second frame rate. It is characterized by that.

  According to a sixth technical means, in the fourth technical means, the synchronizing means includes an image of the first frame rate, a motion vector between a reference frame and a subframe constituting the image of the second frame rate, and It is characterized by synchronizing.

  A seventh technical means is an image processing apparatus that performs image processing using an image and motion information output from the motion information acquisition apparatus according to any one of the first to sixth technical means. Generating at least a partial image of the second frame rate by interpolation from the images of the frame rate of the second frame rate and the motion information corresponding to the images of the second frame rate located between the images. It is a thing.

  According to the present invention, it is possible to improve the accuracy of motion detection while significantly reducing the processing amount as compared with a case where only a low frame rate moving image is used. In addition, by pre-determining a format for outputting moving images and motion vectors in synchronization with the imaging device, the motion vectors can be used as they are, or modified as appropriate, as the motion vectors required for various image processing at the subsequent stage. can do. Therefore, it is possible to greatly reduce the processing necessary for motion vector detection in the subsequent image processing.

It is a figure which shows an example of the relationship of the motion vector between a reference | standard flame | frame, a sub-frame, and a flame | frame. It is a figure which shows the other example of the relationship of the motion vector between a reference | standard flame | frame, a sub-frame, and a flame | frame. It is a figure for demonstrating a state when a pixel value is read at a high frame rate. It is a figure for demonstrating an example of the search method of a motion vector. It is a figure for demonstrating an example of the thinning-out method of the pixel according to the distance of a reference | standard frame. It is a figure which shows an example of the stationary area | region and moving area | region in a block. It is a flowchart for demonstrating an example of the motion information acquisition method which concerns on this invention. It is a block diagram which shows the structural example of the imaging device corresponded to the motion information acquisition apparatus of this invention. It is a figure which shows the packet structure of auxiliary data. It is a figure which shows the effective image area | region and auxiliary data multiplexing possible area | region in the HDTV signal of 1125/60 system. It is a figure for demonstrating an example of the method of storing the motion vector at the time of setting the space | interval a of a reference frame as 4. FIG. 3 is a block diagram illustrating an example of an image processing apparatus that performs various types of image processing using moving images and motion vectors output from an imaging apparatus. It is a figure for demonstrating a mode that an intermediate | middle frame is interpolated from a reproduction | regeneration frame. It is a block diagram which shows the structural example of the conventional image processing apparatus.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments according to a motion information acquisition device and an image processing device of the invention will be described with reference to the accompanying drawings.

(First embodiment)
1. Motion Detection Using Subframes FIGS. 1 and 2 are diagrams illustrating an example of a relationship between a reference frame, a subframe, and motion vectors between frames. As shown in FIGS. 1 and 2, with respect to the reference frame F (am), past subframes are represented by F (am−1), F (am−2),..., F (a (m−1) +1. ) And the immediately preceding reference frame is F (a (m−1)). In this example, the interval between the reference frames is a frame, and the number of sub-frames is (a-1).

  FIG. 1A shows an example in which motion vectors are sequentially detected from a new reference frame to an old frame in time, and FIG. 1B shows a reference frame F (am) shown in FIG. And F (a (m−1)) and a subframe between them are enlarged. FIG. 2 shows an example in which motion vectors are sequentially detected from a temporally old reference frame to a new frame. This will be described later.

  Here, the pixel value of the subframe can be read at a frame rate higher than the normal frame rate as shown in FIG. 3A by nondestructive reading from an image sensor (not shown). Here, some image sensors are called “high-speed non-destructive CMOS image sensors” that can acquire an image at any time even during imaging. This is a kind of element that accumulates charges when receiving light. However, unlike a normal image sensor, the accumulated charges are not destroyed even if pixel values are read out.

  In FIG. 3, the vertical axis represents pixel values, the horizontal axis represents time, and S represents a difference value from the previous frame. FIG. 3B shows a value obtained by extracting the value of the S portion. For example, when the normal frame rate is 30 fps and the high frame rate is 240 fps, the pixel values in FIG. 3B are values accumulated for 1/240 seconds. The pixel value in FIG. 3B is obtained not only as the difference value of the accumulated intermediate pixel value in FIG. FIG. 3C shows the result of adding the pixel values in FIG. 3B for a time corresponding to the normal frame rate. Although FIG. 3C acquires an image at a high frame rate, the accumulation time of the pixel value is the same accumulation time as the pixel value imaged at the normal frame rate.

  In the following description, the pixel values of the reference frame and subframe may be the difference value between frames (FIG. 3B), or the pixel value at the time of high frame rate imaging, or accumulated for the time corresponding to the reference frame. A value (FIG. 3C) may be used.

Next, a motion vector between two adjacent frames is assumed to be ΔVk = (ΔVxk, ΔVyk) (where k = 1 to a). For example, a motion vector between frames F (am−b) and F (am− (b−1)) (where b = 1 to a) is expressed as ΔVb = (ΔVxb, ΔVyb).
The motion vector between the reference frame F (am) and each frame is Vk = (Vxk, Vyk) (where k = 1 to a). For example, the motion vector between the reference frame F (am) and the frame F (am−b) (where b = 1 to a) is

It is expressed. Finally, Va which is a motion vector between the reference frames F (am) and F (a (m−1)) is obtained. There are the following two search methods.

(Method 1) Method for calculating block matching with reference frame F (am) In this method 1, a moving range with a high frame rate uses a fact that the movement of an object is limited to a very narrow range, so that a search range is obtained. And block matching is calculated between the frame F (am-b) and the reference frame F (am). In FIG. 4A, the target block (i, j) of the reference frame (where i and j are the horizontal and vertical positions on the frame and correspond to the upper left pixel of the block) and F (am− (b− It is assumed that the motion vector Vb-1 = (Vxb-1, Vyb-1) during 1)) is obtained.

  Next, when calculating the matching between the reference frame F (am) and the frame F (am−b), the search range is a predetermined range (± SRx, ± SRy) with (i + Vxb−1, j + Vyb−1) as the center. Explore. That is, in the case of the method 1, the reference frame F (am) and the frame F are referred to while referring to the motion vector Vb-1 between the target block (i, j) of the reference frame and the frame F (am− (b−1)). Matching between (am-b) is sequentially calculated. In this manner, the matching between the reference frame F (am) and the frame F (am-b) is directly calculated within the above range of the frame F (am-b), and the most similar region is the reference frame F (am ) Of the target block (i, j).

  For example, SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference) can be used as the above-described evaluation index for block matching.

  Here, bx and by are horizontal and vertical sizes of the block, F (am, i, j) is a pixel value located at (i, j) on the frame F (am), and (x, y) (where − SRx ≦ x ≦ SRx, −SRy ≦ y ≦ SRy) represents a displacement amount (motion vector between adjacent frames) of the target block (i, j).

(Method 2) Method of calculating block matching between adjacent frames F (am- (b-1)) and F (am-b) In Method 2, block matching is calculated between reference frames. Instead of the method 1, the calculation is performed between two adjacent frames as shown in FIG. In this case, SAD or SSD is expressed by the following equation.

For SAD or SSD obtained by Method 1 or 2,
ΔVb = (ΔVxb, ΔVyb) = (x ′, y ′) (5)
It becomes. Where (x ′, y ′) = arg min SAD (x, y) or (x ′, y ′) = arg min SSD (x, y) (where −SRx ≦ x ≦ SRx, −SRy ≦ y ≦ SRy ). further,
Vb = Vb-1 + ΔVb (6)
It is.

  In Method 1, since the reference frame is always used for block matching, it is not easily affected by noise or the like, and even if a detection error occurs in the process of multiple searches using subframes, There is a merit that it hardly affects the motion vector detection. Also, Method 2 has an advantage that a more accurate motion vector can be easily obtained because the correlation between the two frames for which block matching is performed is high. Further, both methods 1 and 2 have the effect that the amount of motion detection processing can be reduced by using subframes. The search range of motion in each frame is 1 / a in both the horizontal and vertical directions with respect to the search range (± Sx, ± Sy) when direct block matching is performed between the reference frames. The search range of each block is (± SRx, ± SRy) = (± Sx / a, ± Sy / a). By using subframes, the number of frames to be searched is multiplied by a, so the amount of processing necessary to detect the motion between reference frames is (1 / a) · (1 / a) · a = 1 / a Thus, the amount of motion detection processing can be reduced by using subframes.

2. Variable search range In the above example, the search range when subframes are used is fixedly set to 1 / a with respect to the horizontal and vertical directions of the search range (Sx, Sy) directly searched between the reference frames. However, the search range may be adaptively changed using the result of block matching with the immediately preceding subframe. From the relationship between the motion ΔVb−1 = (ΔVxb−1, ΔVyb−1) between the frames in the immediately preceding subframe and a predetermined threshold TH1, the search range of block matching with the next frame (± SRx, ± SRy )

SRx = (1 / a) · Sx if | ΔVxb-1 | ≧ TH1 (7)
Sxl otherwise
SRy = (1 / a) ・ Sy if | ΔVyb-1 | ≧ TH1 (8)
Syl otherwise

  Here, Sxl <(1 / a) · Sx and Syl <(1 / a) · Sy. When the amount of motion in the immediately preceding frame is small, it is assumed that the amount of motion for the next frame is also small, and the search range for motion detection is set small. In the above example, the same value is used for the threshold value TH1 in the horizontal direction and the vertical direction, but different values may be set.

3. Thinning out pixels By thinning out pixels to be processed according to the distance from the reference frame, the processing amount can be further reduced. For example, as shown in FIG. 5A, the subframe F (am-1) is subject to processing for each a pixel in the horizontal and vertical directions, and the subframe F (am-2) is shown in FIG. As shown in FIG. 5 (B), the number of pixels to be processed is changed according to the distance from the reference frame, for example, each a / 2 pixel is to be processed horizontally and vertically. In the example of FIG. 5, the number of pixels to be thinned out a = 4, the block horizontal size bx = 8, and the block vertical size by = 8.

  By combining these thinning-out processes with the variable search range process described above and applying it to the variable search ranges shown in the above formulas (7) and (8), it is possible to further block compared to the case where the search range is not changed. It is also possible to reduce the amount of matching processing.

4). Weighting of moving area In the above description, it is assumed that the inside of one block is an area having a uniform property. Hereinafter, a case where different areas such as a still area and a moving area are included in one block will be described. Based on the magnitude relationship between the difference value d (ix, jy) between frames for which block matching is performed and a predetermined threshold value TH2, the blocks are classified into a still area and a moving area as shown in FIG.

In Method 1 (block matching with a reference frame), d (ix, jy) is calculated by the following equation.
d (ix, jy) = | F (am-b, ix, jy) -F (am, ix, jy) | (9)
In Method 2 (block matching between adjacent frames), d (ix, jy) is calculated by the following equation.
d (ix, jy) = | F (am-b, ix, jy) -F (am-b-1, ix, jy) | (10)

  Here, (ix, jy) indicates the position of the target pixel. If d (ix, jy) <TH2, it is determined that the pixel (ix, jy) exists in the static region, and if d (ix, jy) ≧ TH2, the pixel (ix, jy) is determined to exist in the moving region. To do. As shown in FIG. 6, after classifying each pixel in the block into a static region and a dynamic region, an evaluation method such as SAD and SSD is calculated by changing the weight weight for each pixel in the static region and the dynamic region. . For example, in the case of Method 1, an evaluation method in which a weight weight for each pixel is introduced is represented by the following equation.

  However, weight_m> weight_s. Thereby, it is possible to adjust the evaluation method so that the pixel difference value of the moving region greatly affects. If all pixel values in the block are d (ix, jy) <TH2, the target block is regarded as a still block, the subsequent block matching and evaluation method calculations are omitted, and the motion vector is set to 0. Thus, the processing amount can be reduced.

  The above processing will be described based on the flowchart of FIG. First, the reference frame number m, subframe number b, and block number p are initialized (step S1). Next, the block matching search range is determined by the above-described method (step S2). Next, block matching is performed on the determined search range using the above-described method 1 or method 2 (step S3). That is, in the case of Method 1, between the reference frame F (am) and the frame F (am-b), and in the case of Method 2, between the frames F (am- (b-1)) and F (am-b). Block matching is performed at.

  Then, the above process is performed for all blocks, and it is determined whether or not the process of the last block of one frame is completed (step S4). If it is determined that the process of the last block has not been completed (NO), the block number p is incremented by 1 (step S5), and the process returns to step S2 and is repeated. If it is determined in step S4 that the processing of the last block has been completed (in the case of YES), the same processing is performed on the next subframe.

  Then, the above process is performed on all the subframes between the reference frames, and it is determined whether or not the process of the immediately preceding reference frame is completed (step S6). Here, when it is determined that the processing of the immediately preceding reference frame has not been completed (in the case of NO), the subframe number b is incremented by 1 (step S7), and the processing returns to step S2 and is repeated. If it is determined in step S6 that the processing of the immediately preceding reference frame has been completed (in the case of YES), a motion vector between the immediately preceding reference frames F (a (m−1)) and F (am) is obtained.

  Then, it is determined whether or not the processing of the final reference frame of the sequence has been completed (step S8). Here, when it is determined that the processing of the final reference frame has not been completed (in the case of NO), the reference frame number m is incremented by one (step S9), and the process returns to step S2 and is repeated. If it is determined in step S8 that the process of the final reference frame has been completed (in the case of YES), the series of processes is terminated.

  In this example, after the processing of all the blocks in one frame is completed, the processing proceeds to the processing of the next subframe. However, the processing of all the subframes is performed for each block, and the immediately preceding reference frame F (a ( After obtaining the motion vector between m-1)) and F (am), the method may proceed to the processing of the next block.

In the above example, the motion vectors are sequentially detected from the temporally new (current) reference frame to the old (past) frame based on FIG. 1 described above, but as shown in FIG. In particular, motion vectors can be sequentially detected from an old reference frame to a new frame. In this case, first, a motion vector ΔV ₁ between the reference frame F (a (m−1)) and the subframe F (a (m−1) +1) is obtained, and then the subframe F (a (m−1) +1). ) And the subframe F (a (m−1) +2), the motion vector ΔV ₂ is obtained, and similarly the motion detection process proceeds in a new direction in time. Alternatively, as before, a method of always performing block matching with the reference frame F (a (m−1)) may be used.

  Finally, Va, which is a motion vector from the reference frame F (a (m−1)) to the reference frame F (am), is obtained. If necessary, the motion vector Va is inverted and the reference frame F A motion vector from (am) to the reference frame F (a (m−1)) is calculated.

  Since this method can sequentially process input images, there is an advantage that a required memory amount can be reduced as compared with the method shown in FIG. That is, in the case of FIG. 1, in order to obtain a motion vector between F (a (m−1)) and F (am), all subframes in between are accumulated and after F (am) is acquired. However, in the case of FIG. 2, the motion detection process can be started when F (a (m−1) +1) is acquired.

  As described so far, according to the present invention, it is possible to obtain an accurate motion vector while reducing the processing amount by using a high inter-frame correlation of a high frame rate moving image.

  In the following, a format of a moving image and a motion vector for ensuring data compatibility between devices connected to the imaging apparatus when the motion vector and the moving image described so far are output in synchronization will be described. .

  FIG. 8 is a block diagram illustrating a configuration example of an imaging apparatus corresponding to the motion information acquisition apparatus of the present invention. The imaging apparatus includes an image sensor 11 corresponding to an imaging unit, a motion detection unit 12 corresponding to a motion information detection unit, a data creation unit 13 corresponding to a synchronization unit and an output unit, and an output image selection unit 14. . The image sensor 11 acquires a high frame rate moving image, and the motion detection unit 12 calculates a motion vector between frames. The processing of the motion detection unit 12 is, for example, motion detection processing using a high frame rate moving image as described above, and detects a motion vector between reference frames or subframes. The output image selection unit 14 selects a reference frame from a high frame rate moving image acquired by the image sensor 11. The data creation unit 13 outputs the motion vector calculated by the motion detection unit 12 and the reference frame selected by the output image selection unit 14 in synchronization.

  As the image sensor 11, for example, the above-described “high-speed nondestructive CMOS image sensor” can be applied. According to this, incident light is branched into two systems, and one branched light is split into the first frame. It is possible to pick up an image at a rate and pick up the other branched light at a second frame rate higher than the first frame rate.

  That is, the imaging apparatus according to the present invention includes an image sensor 11 that captures an image having a first frame rate including a reference frame and an image having a second frame rate higher than the first frame rate including the reference frame, A motion detection unit 12 that sequentially detects a motion vector as motion information of each block constituting each image of the first frame rate using an image of the first frame rate, an image of the first frame rate, and the corresponding image A data creation unit 13 that synchronizes the motion vector of each block is provided, and the data creation unit 13 outputs the synchronized image and motion vector of the first frame rate.

  Further, as described above, the motion detection unit 12 may include means for changing the motion search range of the target block based on the motion vector already detected between adjacent images of the second frame rate.

  In addition, the motion detection unit 12 determines whether each pixel of the image at the second frame rate belongs to a still region or a motion region, and corresponds to the two images to obtain motion information. Evaluation means for evaluating the correlation between the target and the reference block using the pixel value to be evaluated, and the evaluation means weights and evaluates when the evaluation value is calculated according to the area to which each pixel obtained by the determination means belongs You may do it. Since the weighting method of this area is as described above, the description thereof is omitted here.

  Further, the data creation unit 13 synchronizes the image of the first frame rate with the motion vector of each block constituting at least one image of the second frame rate corresponding to the first frame rate, and thereby generates a predetermined format. Can be output. For example, in the case of the example of FIG. 1 described above, the data creation unit 13 moves the image of the first frame rate and the motion vector (Vi () between the reference frame and the reference or subframe constituting the image of the second frame rate. i = 1 to a)) may be synchronized, or Vi may be divided into motion vectors (ΔVi (i = 1 to a)) between two adjacent frames and synchronized.

  In the above, it is necessary to determine a format in advance in order to ensure data compatibility between devices connected to the imaging apparatus. As an example of this, a data structure and a transmission method when a motion vector is multiplexed are shown in the 1125/60 system HDTV studio system standard (BTA S-005B). This BTA S-005B describes a data structure for transmitting auxiliary data during the digital blanking period, and the auxiliary data packet structure is shown in FIG.

  In FIG. 9, ADF is an auxiliary data flag (3 words) indicating the start of an auxiliary data packet. The DID is a data identification word (1 word) indicating the type of UDW described later. DBN is a data block number word (1 word) indicating the order of packets having the same DID. DC is a data count word (1 word) indicating the number of words of UDW. UDW is an area in which auxiliary data of 256 words or less is written, and the aforementioned motion vector is written in this area. CS is a checksum value (1 word). Here, one word is 10 bits.

  FIG. 10 is a diagram showing an effective video area and an auxiliary data multiplexing possible area in the 1125/60 HDTV signal. FIG. 10A shows the luminance component, and FIG. 10B shows the color difference component. In the figure, the vertical axis indicates the line number (1125 scanning lines), the horizontal axis (time direction) indicates the video sample number (2200 samples), 15 is an effective video area (area for storing images), and 16 is auxiliary. This is a data-multiplexable area (area for storing auxiliary data packets). In this example, image data of a standard frame rate reference frame is stored in the effective video area 15, and a motion vector corresponding to a high frame rate subframe is stored in the auxiliary data multiplexing possible area 16 in the auxiliary data packet ( UDW) and then record.

The motion vector stored here may be a motion vector between reference frames (Va in FIGS. 1 and 2) or a motion vector between reference frames and subframes (Vk in FIGS. 1 and 2 (where k = 1 to a−1). )) Or a differential motion vector between subframes (ΔVk in FIGS. 1 and 2 (where k = 1 to a−1)) may be used. However, a case where a differential motion vector is stored will be described below. . Further, the combined motion vector may be stored for the subframes for every q frames instead of the motion vectors of all the subframes. For example, when motion vectors are stored in subframes every two frames, ΔV ₁ + ΔV ₂ , ΔV ₃ + ΔV ₄ ,... In FIGS.

  Since the amount of data that can be stored in the auxiliary data multiplexing possible region 16 is limited, the number of subframes that can be stored depends on the block size corresponding to the motion vector and the accuracy of the motion vector. For example, in the example of FIG. 10, the auxiliary data multiplexing possible area 16 includes 458404 words including the luminance component (Y) and the color difference components (Pb, Pr). When the block size is 16x16 and the motion vector is expressed by 5 bits (fixed length code), the amount of motion vector data required for one frame is 8160 words, and 32 auxiliary data packets are required. 8384 words. Accordingly, motion vectors for 54 frames can be stored in the auxiliary data multiplexing possible area 16. If the block size and motion vector accuracy are different, the number of frames that can be stored also changes. The motion vector can also be expressed by a variable length code.

  The motion vector can be obtained by the method 1 or the method 2 described above. The following is an example when Method 1 is used to calculate a motion vector. As shown in FIG. 1, the motion vector of the subframe between F (a (m−1)) and F (am) cannot be calculated unless F (am) is acquired from the imaging device. . Therefore, the data creation unit 13 needs a memory for storing image data for a + 2 frames and a motion vector. An example in the case of a = 4 is shown in FIG.

11, at time 4m + 1, the sub-frame F (4m + 1) during the storage in the memory 5, the reference frame F (4m) and sub-frame F (4m-1) to calculate the motion vector _{V 1} = _{ΔV 1} between Store in memory. At time 4m + 2, the image data of the subframe F (4m + 2) is stored in the memory 3 in which the subframe F (4m-1) has been stored, and the reference frame F (4m) and the subframe F (4m-2) are stored therebetween. The motion vector V ₂ between them is calculated, and ΔV ₂ = V ₂ −V ₁ is stored in the memory. Similarly, at time 4 (m + 1), the motion vector V4 between the reference frames F (4 m) and F (4 (m−1)) is calculated and ΔV ₄ = V ₄ −V ₃ is stored in the memory. On the other hand, the reference frame F (4 (m + 1)) is stored in the memory 1 in which the subframe F (4m-3) has been stored. Here, the image of the reference frame F (4 m) is supplemented with the difference motion vector ΔVk (k = 1 to a) between F (4 m) to F (4 (m−1)) in the effective video area 15 of FIG. 9. The data is stored in the data multiplexing possible area 16 and output to the outside.

  With the above processing, a motion vector synchronized with a moving image can be output from the imaging apparatus. When the subframe interval (number of frames) to be stored in synchronization with the moving image, the block size corresponding to the motion vector, and the accuracy of the motion vector are variable, the UDW of the first auxiliary data packet is not a motion vector. By storing values such as the frame interval, block size, and accuracy, it is possible to notify the external processing apparatus of these pieces of information.

(Second Embodiment)
In the present embodiment, a method for using a motion vector output in synchronization with a moving image in the predetermined format described in the first embodiment will be described.
FIG. 12 is a block diagram illustrating an example of an image processing apparatus that performs various types of image processing using moving images and motion vectors output from the imaging apparatus. In the figure, 21 is an imaging device (motion information acquisition device), 22 is a moving image encoder, 23 is a moving image decoder, 24 is an FRC unit, 25 is a motion vector encoder, 26 is a motion vector decoder, 27 is a camera shake correction unit, 28 Indicates a high resolution processing unit, and 29 indicates a still image / moving image encoder. In the case of the apparatus example in FIG. 12A, the motion detection unit is provided in the moving image encoder 22 and the FRC unit 24. In the case of the apparatus example in FIG. 12B, the motion detection unit is in the moving image encoder 22 and the camera shake correction unit 27. 12C, the motion detection unit is provided in the camera shake correction unit 27 and the moving image encoder 22, and in the case of the example in FIG. 12D, the alignment unit is a high resolution processing unit. 28.

  The configuration shown in FIG. 12 is basically the same as the conventional configuration shown in FIG. 14 described above, but the motion vector obtained by independent calculation is replaced with the motion vector output from the imaging device 21, or The portion to be calculated based on the motion vector output from the imaging device 21 is different. The configuration of the imaging device 21 has been described with reference to FIG. 8, and the motion vector obtained by the imaging device 21 is shown in FIGS. 12A and 12B as the motion detection of the moving image encoder 22 and the motion vector encoder 25. FIG. 12A shows a case where the motion vector is used, and FIG. 12B shows a case where the motion vector is used for motion detection of the camera shake correction unit 27. FIG. 12C shows a case where motion compensation of the camera shake correction unit and pre-processing are used for the motion vector of the moving image encoder 22, and FIG. 12D shows still image or moving image resolution enhancement processing. The case where it is used as necessary movement information for alignment is shown.

  Among these, the apparatus configuration in FIG. 12A will be specifically described as a representative example. In the example of FIG. 12A, the motion vector calculated by the imaging device 21 is used for motion detection of the moving image encoder 22 and frame rate conversion (FRC) in a display device that is an example of an image processing device. The case where it is used is shown. The moving image and the motion vector output from the imaging device 21 are separated into a moving image and a motion vector by the moving image encoder 22, and the motion vector is first or directly processed by the motion detecting unit in the moving image encoder 22. Is used to encode. This encoded data is decoded by the moving picture decoder 23, and the reproduced moving picture is input to the FRC unit 24. By using the motion vector obtained by the imaging device 21, it is possible to significantly reduce the amount of processing related to motion detection in the moving image encoder 22.

  Furthermore, only the motion vector required by the motion vector encoder 25 is encoded in the moving image and the motion vector output from the imaging device 21. For example, when a moving image encoded by the moving image encoder 22 needs a moving image of 60 fps and a display needs 120 fps, an intermediate frame must be interpolated from the reproduction frame as shown in FIG. In this case, a motion vector corresponding to the intermediate frame (120 fps) may be created from the motion vectors for subframes output from the imaging device 21 and encoded. On the display side, this is decoded by the motion vector decoder 26 and input to the FRC unit 24.

  The FRC unit 24 conventionally interpolates an intermediate frame by scaling to, for example, ½ using a motion vector between reference frames, but in the present invention, a motion vector input in synchronization with a moving image is used. Use to interpolate intermediate frames. That is, the FRC unit 24 generates each image at the second frame rate by interpolation from each image at the first frame rate and a motion vector corresponding to the image at the second frame rate located between the images. To do.

  Since this motion vector is a motion vector detected by the imaging device 21 using an actual intermediate frame (120 fps reference frame or subframe), (1) a moving image having no intermediate frame input to the display device is displayed. There is an advantage that a motion detection circuit that is more accurate than the motion vector estimated by using the method and (2) a large amount of processing is unnecessary on the display side.

  When the frame rate of the subframe is higher than the displayable frame rate, for example, 240 fps, the motion corresponding to the intermediate frame (120 fps) among the subframe motion vectors output from the imaging device 21. Only the vectors may be encoded, or the motion vectors of all subframes may be encoded, and the FRC unit 24 may interpolate the frames using only the motion vectors necessary for display. In the case of 240 fps, since there are three subframes, the FRC unit 24 interpolates the frame using only the motion vector of the middle subframe.

  Currently, high-end TVs are equipped with FRC as standard, but these are resolutions up to HD (high definition), and when the resolution is further increased in the future, when viewing 4K and 8K resolution moving images, it is enormous. It is difficult to perform motion detection on an image with a large amount of data. Therefore, the effect of detecting a motion vector on the photographing side is great for future resolution enhancement. In addition to the 2 × speed intermediate frame, it is also possible to interpolate a plurality of frames using a plurality of motion vectors such as 4 × speed and 8 × speed to make the motion appear smoother.

DESCRIPTION OF SYMBOLS 11 ... Image sensor, 12 ... Motion detection part, 13 ... Data preparation part, 14 ... Output image selection part, 15 ... Effective image area | region, 16 ... Auxiliary data multiplexing possible area | region, 21 ... Imaging device (motion information acquisition apparatus), 22 ... moving image encoder, 23 ... moving image decoder, 24 ... FRC unit, 25 ... motion vector encoder, 26 ... motion vector decoder, 27 ... camera shake correction unit, 28 ... high resolution processing unit, 29 ... still image / moving image Encoder.

Claims (7)

Imaging means for capturing an image of a first frame rate composed of a reference frame and an image of a second frame rate higher than the first frame rate including the reference frame;
Motion information detecting means for sequentially detecting motion information of each block constituting each image of the first frame rate using the image of the second frame rate;
Synchronization means for synchronizing the image of the first frame rate and the motion information of each block corresponding thereto;
A motion information acquisition apparatus comprising: an output unit that outputs an image and motion information of a first frame rate synchronized by the synchronization unit.   2. The motion information detecting means comprises means for changing a motion search range of a target block based on motion information already detected between adjacent images of the second frame rate. Motion information acquisition device. The motion information detection means; a determination means for determining whether each pixel between adjacent images of the second frame rate belongs to a still area or a motion area;
Evaluation means for evaluating the correlation between the target and the reference block using corresponding pixel values of the two images in order to obtain the motion information;
The motion information acquisition apparatus according to claim 1, wherein the evaluation unit performs weighting and evaluation when calculating an evaluation value according to a region to which each pixel obtained by the determination unit belongs.   The synchronization unit synchronizes the image of the first frame rate with the motion information of each block constituting at least one second frame rate image, and the output unit synchronizes with the synchronization unit. 2. The motion information acquisition apparatus according to claim 1, wherein the first frame rate image and motion information are output in a predetermined format.   5. The motion information acquisition according to claim 4, wherein the synchronization unit synchronizes an image having the first frame rate with a motion vector between reference frames constituting the image having the second frame rate. apparatus.   5. The synchronization unit according to claim 4, wherein the synchronization unit synchronizes an image at the first frame rate with a motion vector between a reference frame and a subframe constituting the image at the second frame rate. Motion information acquisition device. An image processing device that performs image processing using an image output from the motion information acquisition device according to any one of claims 1 to 6 and motion information,
Generating at least a part of the second frame rate by interpolation from the images of the first frame rate and motion information corresponding to images of the second frame rate located between the images. An image processing apparatus.

Images