TWI439121B - Image processing module and image processing method for 2d/3d image conversion and frame rate comversion - Google Patents

Description

Image processing module and image processing method for 2D image/3D image conversion and frame rate conversion

The present invention relates to an image processing module and an image processing method thereof, and more particularly to a frame speed for converting a 2D image to a 3D image and converting the displayed image by using depth information and dynamic information.

In recent years, with the evolution of optoelectronics and semiconductor technology, the development of flat panel displays such as liquid crystal displays (LCDs) has been promoted. Taking a liquid crystal display as an example, the liquid crystal display has become a mainstream in the market because of its superior characteristics such as high space utilization efficiency, low power consumption, no radiation, and low electromagnetic interference.

In order to improve the picture quality of the liquid crystal display, some manufacturers will increase the frame rate of the liquid crystal display, and the frame rate is increased to generate a plurality of intermediate images based on the plurality of original images and to interpose the intermediate images between the original images. These original images are dynamically estimated before the intermediate images are generated to obtain the dynamic vectors corresponding to each of the original images. Then, the objects in the original image are detected according to the motion vector of each block, and the regions usually located in the same object have the same dynamic vector, and the moving trajectories of the objects along the original images to generate the intermediate images.

In addition, in the case that stereoscopic displays have become the mainstream of modern displays, how to convert two-dimensional images into stereoscopic images has been the focus of research for the industry. In general, in a method of converting a two-dimensional image into a set of stereoscopic images including a left-eye image and a right-eye image, it is usually necessary to first find a corresponding depth map, and the corresponding depth map includes all regions in the two-dimensional image/ The depth value of the pixel. This depth value can be determined by the color and/or brightness of the area/pixel. The objects in the foreground and background are generally detected by the depth values of the regions in the two-dimensional image, and then the objects are located in the foreground/background according to the depth values, and the regions generally located in the same object have the same depth value. Finally, a set of 3D images is generated based on the depth map and the 2D image.

However, when the objects of these original images are not moving clearly, the dynamic vector of each block in these original images will be low, so that the objects cannot be detected correctly. Further, the motion vector of the edge region of the moving object cannot be correctly detected, so that the accuracy of image insertion is accurate. Moreover, when the color or brightness of the moving object is close to the color or brightness of the background/adjacent object, the depth value of the object is approximately the same as the depth value of the background/adjacent object, and the object detection is incorrect and Causes an incorrect set of 3D images. Therefore, a module/method must be proposed to improve the accuracy of 2D/3D image conversion and frame speed conversion.

According to the above, the present invention provides an image processing module and an image processing method thereof, which can detect an object according to depth information and dynamic information, and thereby improve the accuracy of object detection.

An embodiment of the present invention provides an image processing module including a depth obtaining unit, a dynamic estimating unit, and a dynamic compensation unit. The depth obtaining unit receives the plurality of original images and detects depth information of each original image. The dynamic estimation unit receives the original images and is coupled to the depth acquisition unit, and the dynamic estimation unit is configured to detect dynamic information of each original image. When the image processing module is in the standard mode, the dynamic estimation unit adjusts the dynamic information of each original image according to the detected depth information of each original image. The dynamic compensation unit performs image insertion and output of a plurality of display images according to the original image and the dynamic information of each of the adjusted original images.

In an embodiment of the present invention, when the image processing module is in the image conversion mode, the depth obtaining unit adjusts the depth information of each original image according to the dynamic information of each original image, and according to the original image and the adjusted each A depth image of the original image outputs a set of stereo images.

In an embodiment of the invention, the depth obtaining unit includes a depth generating unit, a depth adjuster, and a depth image drawing unit. The depth generation unit receives the original images and detects depth information of each original image. The depth adjuster receives depth information and dynamic information of each original image. When the image processing module is in the image conversion mode, the depth adjuster adjusts the depth information of each original image according to the dynamic information of each original image. The depth image drawing unit is coupled to the depth adjuster and receives the original images. When the image processing module is in the image conversion mode, the image drawing unit outputs a set of stereo images of each original image according to the adjusted depth information of each original image and each original image.

The embodiment of the invention provides an image processing method suitable for an image processing module. The image processing method includes the following steps. Receive multiple original images. Detect depth information for each original image. Detect dynamic information of each original image. When the image processing module is in the standard mode, the dynamic information of each original image is adjusted according to the depth information of each original image, the image insertion is performed, and multiple output signals are output according to the original image and the adjusted dynamic information of each original image. Display images.

In an embodiment of the present invention, the image processing method further includes: adjusting the depth information of each original image according to the dynamic information of each original image when the image processing module is in an image conversion mode, and according to each original image And the adjusted depth information of each original image outputs a set of stereo images of each original image.

In an embodiment of the invention, each of the sets of stereo images includes at least one left eye image and at least one right eye image.

In an embodiment of the invention, the stereo images are generated according to a depth image mapping method.

In an embodiment of the invention, when the image processing module is in the image conversion mode, the original images are respectively two-dimensional images.

In an embodiment of the invention, the display images include the original images and a plurality of inserted images.

Based on the above, the image processing module and the image processing method thereof according to the embodiment of the present invention, when the image processing module is in the standard mode, adjust the dynamic information of each original image according to the depth information of each original image, according to the adjusted The dynamic information of each original image detects the object and is dynamically compensated to output a plurality of display images. Therefore, by comparing the depth information and the dynamic information of each original image with each other and correspondingly adjusting, the accuracy of object detection can be improved.

The above described features and advantages of the present invention will be more apparent from the following description.

FIG. 1 is a schematic diagram of an image processing module according to an embodiment of the invention. Referring to FIG. 1 , in the embodiment, the image processing module 100 includes a depth obtaining unit 110 , a dynamic estimating unit 120 , and a dynamic compensation unit 130 . The depth obtaining unit 110 receives the plurality of original images OI and detects the depth information D of each original image OI. The depth information D of each original image OI may be all or part of the depth image/map of each original image OI. The dynamic estimation unit 120 receives the original image OI and is coupled to the depth obtaining unit 110 and the dynamic information MI for detecting each original image OI. The dynamic information MI of each original image OI may be all or part of the dynamic vector/graph of each original image OI. In detail, each region of these original images has a motion vector, and the motion vector can be obtained according to a dynamic estimation algorithm, such as a Block Matching Algorithm. Also, the above area may be as small as one pixel.

When the image processing module 100 is in the standard mode (ie, indicated by a dotted line), the dynamic estimation unit 120 receives the depth information D of each original image OI from the depth obtaining unit 110. Moreover, the dynamic estimation unit 120 generates the dynamic information MI of each original image OI, and adjusts the dynamic information MI of each original image OI according to the depth information D of each original image OI to output the adjusted dynamic information MI'. In the case of the same original image OI, it is assumed that the distribution state of the depth information D of the original image OI (ie, the distribution position of the region formed by the same depth value) is different from the distribution state of the dynamic information MI (ie, formed according to the same dynamic vector). The distribution position of the region), and then adjusting the dynamic vector of the difference block in the dynamic information MI, so that the distribution state of the dynamic information MI' is approximated to the distribution state of the depth information D thereof.

The dynamic compensation unit 130 receives the original image OI and the adjusted dynamic information MI′ of each original image OI, and detects/cuts each original image OI according to the adjusted dynamic information MI′ of each original image OI. At least one object on the top (ie, multiple regions having the same dynamic vector are treated as one object). Then, the motion compensation unit 130 performs image insertion according to the adjusted dynamic information MI' of each original image OI to generate at least one inserted image. Then, the motion compensation unit 130 outputs a plurality of display images DI. The number of these display images DI may be several times (such as 2 times or 3 times) of the original image OI, and the multiple is determined by the ratio of the frame rate of the display to the input frequency of the picture.

Moreover, the display image DI includes the original image OI and a plurality of inserted images (ie, dynamically compensated to generate a derivative image for insertion between the original images OI), wherein the inserted images are respectively configured in corresponding originals. After the image. For example, when the frame rate is 120 Hz and the picture input frequency is 60 Hz, the number of inserted images will be equal to (ie, 1 times) the number of these original images OI, and the number of these display faces DI in each second. It will be twice as large as the original image OI; when the frame rate is 180 Hz and the picture input frequency is 60 Hz, the number of inserted images will be twice the number of these original images OI, and these display faces DI in each second The number will be three times that of these original images OI.

In this embodiment, the dynamic information MI of each original image OI is adjusted according to the depth information D of each original image OI, which is equivalent to the object adjustment/improvement utilization detected according to the depth information D of each original image OI. An object detected by the dynamic image MI of the original image OI. In detail, the dynamic vector of the edge region of the moving object of the original image OI is usually inaccurate/chaotic, and the inaccurate/chaotic dynamic vector causes the inaccuracy of the inserted image.

2A and 2B are schematic diagrams showing depth information and dynamic information of an original image according to an embodiment of the invention. Referring to FIG. 2A and FIG. 2B , in the embodiment, FIG. 2A is a schematic diagram of depth information of the original image OI, and FIG. 2B is a schematic diagram of dynamic information of the original image OI, wherein the original image OI has multiple blocks. Moreover, the depth obtaining unit 110 and the dynamic estimating unit 120 (shown in FIG. 1) respectively detect the depth value (ie, depth information) and the dynamic vector (ie, dynamic information) of each block. However, in other embodiments of the present invention, the depth value and the dynamic vector of each pixel may be detected, and the present invention is not limited thereto. Here, for the sake of simplicity of the illustration, FIGS. 2A and 2B only indicate the depth values and dynamic vectors of the partial blocks to cope with the later description.

As shown in FIG. 2B, it is assumed that the dynamic vectors containing most of the blocks of the car 240 and the driving 250 are the same (eg, dynamic vectors). ), but the dynamic vectors containing the edge blocks of the car 240 and the driving 250 are confusing. In the present embodiment, the vehicle 240 and the driving 250 (ie, the object) may not be accurately detected by simply referring to the dynamic information of FIG. 2B. In the present embodiment, if the motion vectors of the blocks including the car 240 and the driving 250 are assumed to be the same, the car 240 and the driving 250 are regarded as the same object. In order to correctly detect the object, refer to the depth information shown in Figure 2A. According to FIG. 2A, the area with the depth values of 180, 210, and 230 includes the car 240 and the driving 250. Therefore, the dynamic information including the car 240 and the driving 250 in FIG. 2B can be adjusted to be similar or identical, that is, the car 240 can be included. And the dynamic vector of the edge block of the driving 250 is adjusted to be close to the dynamic vector Thereby, the accuracy of detecting the car 240 and driving 250 can be improved. In addition, the depth value can be 0 to 255 (ie, 8 bits), and the object has a depth value of 255 representing the nearest object. on the contrary. The object has a depth value of 0 indicating the object that is furthest away.

In addition, the image processing module 100 can be switched to the image conversion mode to convert the two-dimensional image into a stereo image (ie, a three-dimensional image). When the image processing module 100 is in the image conversion mode (ie, the dotted line is dotted with a dotted line), the depth obtaining unit 110 generates the depth information D of each original image OI, and adjusts each according to the dynamic information MI of each original image. The depth information D of the original image OI is used to detect the depth values corresponding to the objects in the original image OI and the area of the adjustment object. Then, the depth obtaining unit 110 may output a set of stereo images SI of each original image OI according to each original image OI and the adjusted depth information D' of each original image OI (as shown in FIG. 3). The dynamic compensation unit 130 sequentially transmits each of the sets of stereo images SI, wherein each set of stereo images SI can be generated according to a depth image mapping method, and each set of stereo images SI can include a left eye image and a right Eye image.

In this embodiment, adjusting the depth information D of each original image OI according to the dynamic information MI of each original image OI is equivalent to using each original original according to the object detection detected by the dynamic information MI of each original image OI. The object detected by the depth information D of the image OI.

2A and 2B, it is assumed that the depth values of the blocks including the car 240 in FIG. 2A are different, and the depth values of the blocks including the car 240 are different from the depth values of the blocks including the driving 250. In the present embodiment, if the depth value of FIG. 2A is simply referred to, the vehicle body, the tire, and the driving 250 of the car 240 may be regarded as different objects due to the irregular depth relationship with each other, so that the body and the tire of the car 240 are. And the corresponding displacement amount of the driving 250 in the left eye image and the right eye image will be different. Therefore, the dynamic information shown in FIG. 2B can be referred to. According to Figure 2B, the dynamic vector is The area is the body including the car 240, the tire, and the area where the driver 250 is driven. Therefore, the depth values of the vehicle body including the car 240, the tire, and the block of the driving 250 can be adjusted to be similar or identical. That is, the depth value 180 of the block containing the driver 250 may be adjusted to the depth value 225, and the depth value 210 of the block containing the tire of the car 240 may be adjusted to the depth value 227.

FIG. 3 is a schematic diagram of the depth acquisition unit of FIG. 1 according to an embodiment of the invention. Referring to FIGS. 1 and 3 , the depth obtaining unit 110 includes a depth generating unit 111 , a depth adjuster 112 , and a depth image drawing unit 113 . The depth generating unit 111 receives the original image OI and detects the depth information D of each original image OI. When the image processing module 100 is in the standard mode, the depth generating unit 111 outputs the depth information D of each original image OI to the dynamic estimating unit 120. When the image processing module 100 is in the image conversion mode, the depth generation unit 111 outputs the depth information D of each original image OI to the depth adjuster 112, and the depth adjuster receives the depth information D and each original from the dynamic estimation unit 120. Dynamic information MI of the image OI. The depth adjuster 112 adjusts the depth information D of each original image OI according to the dynamic information MI of each original image OI, and the adjusted depth information D' of each original image OI is output to the depth image drawing unit 113.

The depth image mapping unit 113 is coupled to the depth adjuster 112 and receives the original images OI. When the image processing module 100 is in the standard mode, the depth image drawing unit 113 is turned off. When the image processing module 100 is in the image conversion mode, the depth image drawing unit 113 receives the adjusted depth information D′ of each original image OI, and the depth image drawing unit 113 adjusts the original image OI according to each image. The depth information D' of each of the original image OIs is outputted as a set of stereo images SI. In other embodiments, the depth adjuster 112 can be integrated into the depth generation unit 111.

According to the above, the action of the image processing module 100 can be integrated into an image processing method. 4 is a schematic diagram of an image processing method according to an embodiment of the invention. Referring to FIG. 4, in the embodiment, a plurality of original images are received (step S410). Then, the depth information of each original image is detected (step S420) and the dynamic information of each original image is detected (step S430). When the image processing module is in the standard mode, the dynamic information of each original image is adjusted according to the detected depth information of each original image, and image insertion is performed, and each original image and each original image after adjustment are performed. The dynamic information outputs a plurality of display images (step S440).

When the image processing module is in the image conversion mode, the depth information of each original image is adjusted according to the dynamic information of each original image, and each original image is output according to each original image and the adjusted depth information of each original image. A set of stereoscopic images (step S450). The details of the above steps can be inferred by referring to the description of the image processing module, and will not be described herein.

In summary, in the image processing module and the image processing method of the embodiment of the present invention, when the image processing module is in the standard mode, the depth information of each original image is adjusted according to the depth information of each original image. The object is detected according to the dynamic information of each original image after adjustment, and the plurality of display images are output after dynamic compensation. When the image processing module is in the image conversion mode, the depth information of each original image is adjusted according to the dynamic information of each original image, so as to detect the object according to the adjusted depth information of each original image, and output the data according to the original information. Group stereo image. Therefore, by comparing the depth information and the depth information of each original image and making corresponding adjustments, the accuracy of object detection can be improved.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the invention, and any one of ordinary skill in the art can make some modifications and refinements without departing from the spirit and scope of the invention. The scope of the invention is defined by the scope of the appended claims.

100‧‧‧Image Processing Module

110‧‧‧Deep acquisition unit

111‧‧‧Deep generation unit

112‧‧‧ depth adjuster

113‧‧‧Deep image plotting unit

120‧‧‧Dynamic Estimation Unit

130‧‧‧dynamic compensation unit

240‧‧ ‧ car

250‧‧‧ Driving

DI‧‧‧ display image

D, D’‧‧‧ In-depth information

MI, MI’‧‧‧ Dynamic Information

OI‧‧‧ original image

SI‧‧ stereo image

S410, S420, S430, S440, S450‧‧ steps

FIG. 1 is a schematic diagram of an image processing module according to an embodiment of the invention.

2A and 2B are schematic diagrams showing depth information and dynamic information of an original image according to an embodiment of the invention.

FIG. 3 is a schematic diagram of the depth acquisition unit of FIG. 1 according to an embodiment of the invention.

4 is a schematic diagram of an image processing method according to an embodiment of the invention.

100. . . Image processing module

110. . . Depth acquisition unit

120. . . Dynamic estimation unit

130. . . Dynamic compensation unit

DI. . . Display image

D. . . In-depth information

MI, MI’. . . Dynamic information

OI. . . Original image

SI. . . Stereoscopic image

Claims (12)

An image processing module includes: a depth acquisition unit that receives a plurality of original images for detecting depth information of each of the original images; and a dynamic estimation unit that receives the original images and couples the depth images a unit for detecting dynamic information of each of the original images; and a dynamic compensation unit; wherein the dynamic estimation unit adjusts each of the original images according to the detected depth information of each of the original images Dynamic information, the dynamic compensation unit performs an image insertion and output of a plurality of display images according to each of the original images and the adjusted dynamic information of each of the original images, wherein the depth obtaining unit generates each of the original images. The depth information of the plurality of pixels, the dynamic estimation unit generates dynamic information of the pixels of each of the original images to detect at least one object and adjust depth information of the pixels according to each of the original images The detected dynamic information of the edge pixel of the object is the same as the dynamic information of the internal pixel of the object. The image processing module of claim 1, wherein the display images comprise the original image and the plurality of inserted images. An image processing module includes: a depth acquisition unit that receives a plurality of original images for detecting depth information of each of the original images; and a dynamic estimation unit that receives the original images and couples the depth images a unit for detecting dynamic information of each of the original images; and a dynamic compensation unit; The depth obtaining unit adjusts the depth information of each of the original images according to the dynamic information of each of the original images, and outputs each of the original images and the depth information of each of the adjusted original images. a set of stereoscopic images of the original images, wherein the depth obtaining unit generates depth information of a plurality of pixels of each of the original images, and the dynamic estimating unit generates dynamic information of the pixels of each of the original images. The objects of the internal pixels having the same dynamic information are adjusted to have the same depth information by detecting at least one object and dynamic information of the pixels according to each of the original images. The image processing module of claim 3, wherein the set of stereoscopic images comprises at least one left eye image and at least one right eye image. The image processing module of claim 3, wherein the depth obtaining unit comprises: a depth generating unit that receives the original images to detect depth information of each of the original images; The depth adjustment unit adjusts the depth information and the dynamic information of each of the original images. When the image processing module is in an image conversion mode, the depth adjustment unit adjusts each of the original images according to the dynamic information of each of the original images. And the depth image drawing unit is coupled to the depth adjuster and receives the original image. When the image processing module is in the image conversion mode, the depth image drawing unit is based on each of the adjusted originals. The depth information of the image and each of the original images output the set of stereoscopic images of each of the original images image. The image processing module of claim 3, wherein when the image processing module is in an image conversion mode, the original images are respectively a two-dimensional image. An image processing method is applicable to an image processing module, comprising: receiving a plurality of original images; detecting depth information of a plurality of pixels of each of the original images; and detecting the pixels of each of the original images Dynamic information; when the image processing module is in a standard mode, detecting at least one object according to depth information of the pixels of each of the original images, and adjusting according to depth information of the pixels of each of the original images In each of the original images, the dynamic information of the edge pixels of the object is the same as the dynamic information of the internal pixels of the object, and an image insertion is performed, and each of the original images and the adjusted original images are determined according to each of the original images. The dynamic information outputs multiple display images. The image processing method of claim 7, further comprising: adjusting the depth information of each of the original images according to dynamic information of each of the original images when the image processing module is in an image conversion mode And outputting a set of stereo images of each of the original images according to each of the original images and the adjusted depth information of each of the original images. The image processing method of claim 8, wherein the set of stereoscopic images comprises a left eye image and a right eye image. An image processing method according to claim 9 of the patent application, The stereo image in the group is generated according to a depth image drawing method. The image processing method of claim 8, wherein the original image is a two-dimensional image when the image processing module is in the image conversion mode. The image processing method of claim 7, wherein the display images comprise the original image and the plurality of inserted images.