TWI696981B - Interactive image processing system using infrared cameras - Google Patents

Abstract

An interactive image processing system including a first infrared camera, a second infrared camera, an image processing circuit, a vision processing unit, an image signal processor, a central processing unit, and a memory device is disclosed. The present disclosure calculates depth data according to infrared images generated by the first and second infrared cameras to improve depth quality.

Description

Interactive image processing system using infrared camera

The present disclosure refers to an interactive image processing method, device and media using a depth engine, and particularly to an interactive image processing method, device and media using a depth engine to perform depth calculations first.

In a typical stereoscopic image processing system, the original image generated by the red-green-blue image sensor or camera often requires a series of preprocessing operations, such as image analysis, image reconstruction, and image Image quality enhancement (including automatic white balance, exposure value and contrast correction) and depth calculation, etc., to prepare for subsequent applications.

Then, the reconstructed image and its depth data can be input to a central processing unit for processing, so as to realize the related applications of the interactive interface provided by the video game system, sales stop or other systems, such as virtual reality devices, notes Interactive interface provided by devices such as personal computers, tablet computers, desktop computers, smart phones, interactive projectors, TV set-top boxes, or consumer electronic devices.

However, different light sources can be regarded as noise to the stereoscopic image processing system, which causes the quality of depth calculation to be unstable due to the influence of different light sources. In addition, the quality of depth calculations is poor in scenes lacking texture. Therefore, how to improve the quality of deep computing has become one of the topics in the field.

Therefore, one of the purposes of the present disclosure is to provide a method, device, and media for interactive image processing to improve depth quality.

This disclosure discloses an interactive image processing system, including a first camera for generating a first infrared image; a second camera for generating a second infrared image; and a third camera for generating a color image; An image processing circuit, coupled to the first camera, the second camera and the third camera, for calculating a depth data based on the first infrared image, the second infrared image and the color image; and a vision The processing unit, coupled to the image processing circuit, is used to perform stereo matching on the first infrared image and the second infrared image to generate a gray-scale matching image; and to color the gray-scale matching image and the color image Match to produce a color matching image.

The present disclosure discloses an interactive image processing system, including a first camera for generating a first color infrared image; a second camera for generating a second color infrared image; and an image processing circuit coupled to the first camera And the second camera for calculating a depth data based on the first color infrared image and the second color infrared image; and a visual processing unit, coupled to the image processing circuit, for the first color infrared The image and the second color infrared image are stereo matched to generate a color matching image.

The present disclosure calculates depth data based on infrared images generated by the first infrared camera and the second infrared camera to improve depth quality. Therefore, the accuracy and efficiency of the central processing unit for related applications (such as gesture motion detection and tracking, spatial scanning, object scanning, augmented reality perspective, and simultaneous positioning and mapping) can be further improved to provide better users Experience.

FIG. 1 is a functional block diagram of an interactive image processing system 1 according to an embodiment of the disclosure. The interactive image processing system 1 includes a first camera 11, a second camera 12, an image processing circuit 13, a visual processing unit 14, an image signal processor 15, a central processing unit 16, and a memory 17.

The first camera 11 and the second camera 12 are coupled to the image processing circuit 13 for generating images M1 and M2 to the image processing circuit 13 respectively.

The image processing circuit 13 is coupled to the first camera 11, the second camera 12, and the visual processing unit 14, which can be regarded as a depth hardware engine for calculating a depth data corresponding to the objects recognized from the images M1, M2 D. Specifically, the image processing circuit 13 recognizes at least one object from the images M1 and M2, and then calculates the corresponding at least one object according to a reference value (ie, the distance between the first camera 11 and the second camera 12) Distance, where the depth data D includes the distance corresponding to the identified one or more objects.

In one embodiment, the image processing circuit 13 combines the images M1 and M2 marked with the same synchronization label into a same data packet, wherein the same data packet is marked with a label of the first channel; and the depth data D and a dummy The data DY is combined into a same data packet, wherein the same data packet is marked with a label of a second channel. The first channel is a physical path, and the second channel is a virtual path. In this way, the visual processing unit 14 can distinguish the data packet used for the physical path from the data packet used for the virtual path according to the label of the data packet. In one embodiment, the image processing circuit 13 combines the two of the image M1, the image M2, the depth data D, and the dummy data DY into a data packet labeled with the label of the first channel, and combines the images M1, M2, The other two of the depth data D and the dummy data DY are combined into a data packet marked with a label of the second channel, but it is not limited to this, and those with ordinary knowledge in the art can modify the content of the data packet according to actual needs.

The visual processing unit 14 is coupled to the image processing circuit 13 and the image signal processor 15 for stereo matching the images M1 and M2 according to the depth data D. In addition, the visual processing unit 14 is used to determine at least one captured object with a specific pattern or figure based on the images M1, M2, where the specific pattern may be a gesture.

The image signal processor 15 is coupled to the visual processing unit 14 and the central processing unit 16, and is used to perform automatic white balance and exposure value correction on the original images M1, M2 to improve image quality, which is beneficial to subsequent object recognition and depth calculation Quality. In one embodiment, the image processing circuit 13, the visual processing unit 14, and the image signal processor 15 may be integrated on a single chip.

The central processing unit 16 is coupled to the image signal processor 15 and the memory 17 for generating an operation result based on the images M1, M2 and the corresponding depth data D, wherein the operation result can be used for gesture motion detection and tracking, space Scanning, object scanning, Augmented reality see-through (AR see-through), Six degree of freedom (6-Dof) and Simultaneous Localization and Mapping (SLAM) Related applications.

The memory 17 is coupled to the visual processing unit 14, the image signal processor 15, and the central processing unit 16, and is used to store at least one program code to instruct the corresponding processing unit to perform specific calculations and operations. In an embodiment, the memory 17 may be integrated in the central processing unit 16, and at least one of the visual processing unit 14 and the image signal processor 15 may access the program code from the central processing unit 16 to perform related operations.

Under the framework of the interactive image processing system 1, the present disclosure first uses the image processing circuit 13 (ie, the deep hardware engine) to calculate the depth data D corresponding to the original images M1, M2, instead of using the digital signal processor in the conventional technology To perform software operations. Next, the present disclosure can obtain images M1 and M2 with better image quality through the arithmetic operation of the visual processing unit 14 and the image signal processor 15 and also obtain depth data D with better accuracy. Therefore, the accuracy and efficiency of the central processing unit 16 for related applications (such as gesture motion detection and tracking, space scanning, object scanning, augmented reality perspective, and simultaneous positioning and mapping) can be further improved to provide better use Experience.

FIG. 2 is a functional block diagram of the image processing circuit 13 of the disclosed embodiment. The image processing circuit 13 may be an application-specific integrated circuit (ASIC) for calculating the depth data D corresponding to the object identified from the image.

The image processing circuit 13 includes an image analysis circuit 21, an object capturing circuit 22, an object depth calculation circuit 23, an overlapping object depth calculation circuit 24, and a multiplexer 25.

The image analysis circuit 21 is used to determine whether to adjust the pixel values of the first image M1 and the second image M2 to improve the image quality. For example, when the images M1 and M2 are too dark, the image analysis circuit 21 can increase the exposure values of the first image M1 and the second image M2 to obtain better image quality for subsequent object acquisition operations.

The object capturing circuit 22 is coupled to the image analysis circuit 21, and is used for identifying at least one object from the first image M1 and the second image M2.

The object depth computing circuit 23 is coupled to the object capturing circuit 22, and is used to determine the position of at least one object in the first image M1 and at least one object in the second image M2 according to the distance between the first camera 11 and the second camera 12 The pixel distance between the positions and a triangle positioning method calculate a first depth of at least one object.

The overlapping object depth computing circuit 24 is coupled to the object depth computing circuit 23, and is used to calculate a second depth of two overlapping objects in at least one object, and output depth data D including the first depth and the second depth.

The multiplexer 25 is coupled to the overlapping object depth calculation circuit 24, and is used to output one of the first image M1, the second image M2, and the depth data D according to a control signal.

The present disclosure uses the image processing circuit 13 to calculate the depth data D according to the original images M1 and M2 at the front end of the interactive image processing system 1. This can reduce the depth calculation burden of the digital signal processor.

FIG. 3 is a functional block diagram of the interactive image processing system 3 according to the disclosed first embodiment. The interactive image processing system 3 includes a first camera 11, a second camera 12, an image processing circuit 13, a visual processing unit 14, an image signal processor 15, a central processing unit 16, a memory 37 and a digital The signal processor 38.

The architectures of the interactive image processing systems 1 and 3 are similar, so the same components are represented by the same symbols. The digital signal processor 38 is coupled between the image signal processor 15 and the central processing unit 16 and is used to convert the images M1 and M2 into a stereogram MS according to a fourth code and depth data D. For example, the stereogram MS includes a three-dimensional object, which is projected on a two-dimensional plane.

The memory 37 is coupled to the digital signal processor 38 and is used to store the fourth program code for instructing the digital signal processor 38 to convert the three-dimensional image.

Under the architecture of the interactive image processing system 3, the present disclosure first uses the image processing circuit 13 to calculate the depth data D corresponding to the two original images M1, M2, and then uses the digital signal processor 38 to convert the stereo image to reduce the central processing The calculation burden of the unit 16 (please note that in the embodiment of FIG. 1, the central processing unit 16 is used to perform stereoscopic image conversion). Therefore, the software operation power consumption of the central processing unit 16 can be further reduced to achieve the purpose of power saving.

FIG. 4 is a functional block diagram of the interactive image processing system 4 according to the disclosed first embodiment. The interactive image processing system 4 includes a first camera 41, a second camera 42, a third camera 40, an image processing circuit 43, a visual processing unit 44, an image signal processor 45, a central processing unit 16, a The memory 47, a digital signal processor 48 and an infrared light source 49.

In this embodiment, the first camera 41 and the second camera 42 are infrared cameras, which are used to generate infrared images IR1 and IR2 (in which the image pixels of the infrared images IR1 and IR2 are defined according to gray scale values), and the third camera 40 is a red-green-blue (RGB) camera, used to generate a color image RGB (where the color image RGB pixel is defined according to red, green and blue pixels). The infrared light source 49 is used to provide an ambient light source to the first camera 41 and the second camera 42 to facilitate infrared image conversion.

The image processing circuit 43 is coupled to the first camera 41, the second camera 42 and the third camera 40, and is used to calculate a depth data D according to the infrared images IR1, IR2 and the color image RGB. The image processing circuit 43 further combines the infrared images IR1 and IR2 into a same data packet (also called Infrared side by side), or combines the color image RGB and the depth data D into a same data packet, Or, one of the infrared images IR1 and IR2 and the depth data D are combined into a same data packet.

The visual processing unit 44 is coupled to the image processing circuit 43 for stereo matching the infrared images IR1 and IR2 to generate a gray-scale matching image. The visual processing unit 44 is also used to perform color matching on the gray-scale matching image and the color image RGB to generate a color matching image RGBIR (wherein the image pixels of the color matching image RGBIR are based on red, green, blue and infrared/gray scale Pixels).

The image signal processor 45 is coupled to the visual processing unit 44 and is used to perform automatic white balance and exposure value correction on the color stereo image RGBIR to improve image quality, which is beneficial to the quality of subsequent object recognition and depth calculation.

The digital signal processor 48 is coupled to the image signal processor 45 and is used to convert the color matching image RGBIR into a stereo image MS according to the depth data D.

The central processing unit 16 is coupled to the digital signal processor 48 and the memory 47, and is used to generate an operation result based on the stereogram MS and the corresponding depth data D. The operation result can be used for gesture motion detection and tracking, spatial scanning, Related applications of object scanning, augmented reality perspective, six-dimensional freedom, and simultaneous positioning and mapping.

The memory 47 is coupled to the visual processing unit 44, the image signal processor 45, the digital signal processor 48, and the central processing unit 16, and is used to store code for instructing the corresponding processing unit to perform related software operations.

Under the framework of the interactive image processing system 4, the present disclosure uses two infrared cameras, an infrared light source, and a red-green-blue camera to provide stable depth quality. Therefore, the accuracy and efficiency of the central processing unit 16 processing related applications (such as gesture motion detection and tracking, spatial scanning, object scanning, augmented reality perspective, and simultaneous positioning and mapping) can be further improved to provide better users Experience.

FIG. 5 is a functional block diagram of the interactive image processing system 5 according to the disclosed first embodiment. The interactive image processing system 5 includes a first camera 51, a second camera 52, an image processing circuit 53, a visual processing unit 54, an image signal processor 55, a central processing unit 16, a memory 57, a digital The signal processor 58 and a random dot infrared light source 59.

In this embodiment, the first camera 51 and the second camera 52 are color infrared cameras, which are used to generate color infrared images RGBIR1, RGBIR2 (wherein the color pixels of the color infrared images RGBIR1, RGBIR2 are based on red pixels and green pixels , Blue pixels and gray scale values), the random point infrared light source 59 is used to provide an ambient light source to the first camera 51 and the second camera 52 to facilitate infrared image conversion.

The image processing circuit 53 is coupled to the first camera 51 and the second camera 52, and is used to calculate a depth data D according to the color infrared images RGBIR1, RGBIR2.

The image processing circuit 53 further extracts red pixels, green pixels, and blue pixels from the color infrared images RGBIR1, RGBIR2 to combine the color components of the color infrared images RGBIR1, RGBIR2 into a same data packet, which is also called "red-green" "Blue edge to edge" can be used for augmented reality perspective applications.

The image processing circuit 53 also extracts gray-scale values from the color infrared images RGBIR1, RGBIR2 to combine the infrared components of the color infrared images RGBIR1, RGBIR2 into a same data packet, so it is also called "infrared side-to-side" and can be used for simultaneous positioning And mapping, gesture motion detection and tracking and six-dimensional freedom application.

The image processing circuit 53 further combines the depth data D and the color components of the color infrared image RGBIR1 into a same data packet, so that the spatial scanning and object scanning applications can be performed under the viewing angle of the first camera 51. In one embodiment, the image processing circuit 53 further combines the depth data D and the color components of the color infrared image RGBIR2 into a same data packet, so that the spatial scanning and object scanning applications can be performed under the viewing angle of the second camera 52 .

The visual processing unit 54 is coupled to the image processing circuit 53 for stereo matching the color infrared images RGBIR1 and RGBIR2 based on the angles of view of the first camera 51 and the second camera 52 to generate color matching images RGBD1 and RGBD2, respectively.

The image signal processor 55 is coupled to the visual processing unit 54 for performing automatic white balance and exposure value correction on the color stereograms RGBD1, RGBD2 to improve image quality, which is beneficial to the quality of subsequent object recognition and depth calculation.

The digital signal processor 58 is coupled to the image signal processor 55 and is used to convert the color matching image RGBD1 or RGBD2 into a stereo image MS according to the depth data D.

The central processing unit 16 is coupled to the digital signal processor 58 and the memory 57 to generate an operation result based on the stereogram MS and the corresponding depth data D. The operation result can be used for gesture motion detection and tracking, spatial scanning, Related applications of object scanning, augmented reality perspective, six-dimensional freedom, and simultaneous positioning and mapping.

The memory 57 is coupled to the visual processing unit 54, the image signal processor 55, the digital signal processor 58 and the central processing unit 56, and is used to store code for instructing the corresponding processing unit to perform related software operations.

Under the architecture of the interactive image processing system 5, since the color infrared camera can generate color depth images, the present disclosure can provide a higher frame rate. In addition, the present disclosure can also provide stable depth quality, and the depth quality will not be affected by other light sources. Therefore, the accuracy and efficiency of the central processing unit 16 processing related applications (such as gesture motion detection and tracking, spatial scanning, object scanning, augmented reality perspective, and simultaneous positioning and mapping) can be further improved to provide better users Experience.

The operation of the interactive image processing system 1 can be summarized as an interactive image processing flow 6, as shown in FIG. 6, the interactive image processing flow 6 includes the following steps.

Step 61: Use an image processing circuit to calculate a depth data based on a first camera generated from a first image and a second camera generated from a second image.

Step 62: Using the image processing circuit, combine the first image and the second image into a first data packet labeled with a first label of a first channel, and combine the depth data and a dummy data to label a second channel A second data packet of a second label.

Step 63: Use a visual processing unit to perform stereo matching on the first image and the second image based on the depth data.

Step 64: Use an image signal processor to perform automatic white balance and exposure value correction on the first image and the second image.

Step 65: Use a central processing unit to generate an operation result based on the first image, the second image, and the depth data, where the operation result can be used for gesture motion detection and tracking, spatial scanning, object scanning, augmented reality perspective , Six-dimensional degrees of freedom and related applications of simultaneous positioning and mapping.

For the detailed operation of the interactive image processing flow 6, please refer to the related description in FIG. 1, which will not be repeated here.

The operation of the interactive image processing system 3 can be summarized as an interactive image processing flow 7. As shown in FIG. 7, the interactive image processing flow 7 includes the following steps.

Step 71: Use an image processing circuit to calculate a depth data based on a first camera generated from a first image and a second camera generated from a second image.

Step 72: Using the image processing circuit, combine the first image and the second image into a first data packet labeled with a first label of a first channel, and combine the depth data and a dummy data to label a second channel A second data packet of a second label.

Step 73: Use a visual processing unit to perform stereo matching on the first image and the second image based on the depth data.

Step 74: Use an image signal processor to perform automatic white balance and exposure value correction on the first image and the second image.

Step 75: Use a digital signal processor to convert the first image and the second image into a stereo image.

Step 76: Use a central processing unit to generate an operation result based on the first image, the second image, and the depth data, where the operation result can be used for gesture motion detection and tracking, spatial scanning, object scanning, augmented reality perspective , Six-dimensional degrees of freedom and related applications of simultaneous positioning and mapping.

For the detailed operation of the interactive image processing flow 7, reference may be made to the related description in FIG. 3, which will not be repeated here.

Please note that in the conventional technology, different applications (including gesture motion detection and tracking, space scanning, object scanning, augmented reality perspective, six-dimensional freedom, and simultaneous positioning and mapping applications) can only operate in specific Hardware architecture and platform, because these applications are incompatible with different hardware architectures and platforms. In contrast, the architecture of the interactive image processing system provided by the present disclosure can be applied to all of the above applications, as long as the code and algorithms stored in the central processing unit or memory of the interactive image processing system are executed.

In addition, the central processing unit can access two or more codes from the memory for two or more applications (including gesture motion detection and tracking, spatial scanning, object scanning, augmented reality) Perspective, six-dimensional freedom, and simultaneous positioning and mapping applications) to achieve multi-tasking functions.

In summary, the present disclosure calculates depth data based on infrared images generated by the first infrared camera and the second infrared camera to improve depth quality. Therefore, the accuracy and efficiency of the central processing unit for related applications (such as gesture motion detection and tracking, spatial scanning, object scanning, augmented reality perspective, and simultaneous positioning and mapping) can be further improved to provide better users Experience. The above are only the preferred embodiments of the present invention, and all changes and modifications made in accordance with the scope of the patent application of the present invention shall fall within the scope of the present invention.

1, 3, 4, 5: Interactive image processing system 11, 41, 51: the first camera 12, 42, 52: Second camera 13, 43, 53: Image processing circuit 14, 44, 54: visual processing unit 15, 45, 55: Image signal processor 16, 56: Central processing unit 17, 37, 47, 57: memory 21: Image analysis circuit 22: Object extraction circuit 23: Object depth calculation circuit 24: Depth calculation circuit for overlapping objects 25: Multiplexer 38, 48, 58: digital signal processor 40: Third camera 49: Infrared light source 59: Random point infrared light source 6, 7: Interactive image processing flow 61～65、71～76: steps D: depth data IR1, IR2: infrared image M1: the first image M2: Second image MS: Stereogram RGB: color image RGBIR, RGBD1, RGBD2: color matching images RGBIR1, RGBIR2: color infrared image DY: Fake data

FIG. 1 is a functional block diagram of an interactive image processing system according to an embodiment of the disclosure. FIG. 2 is a functional block diagram of an image processing circuit according to an embodiment of the disclosure. FIG. 3 is a functional block diagram of an interactive image processing system according to an embodiment of the disclosure. FIG. 4 is a functional block diagram of an interactive image processing system according to an embodiment of the disclosure. FIG. 5 is a functional block diagram of an interactive image processing system according to an embodiment of the disclosure. FIG. 6 is a flowchart of an interactive image processing process according to an embodiment of the disclosure. FIG. 7 is a flowchart of an interactive image processing process according to an embodiment of the disclosure.

4: Interactive image processing system

40: Third camera

41: The first camera

42: Second camera

43: Image processing circuit

44: visual processing unit

45: Image signal processor

16: Central Processing Unit

47: Memory

48: digital signal processor

49: Infrared light source

D: depth data

IR1, IR2: infrared image

RGB: color image

RGBIR: color matching image

MS: Stereogram

Claims (15)

An interactive image processing system, including: A first camera for generating a first infrared image; A second camera for generating a second infrared image; A third camera for generating a color image; An image processing circuit, coupled to the first camera, the second camera, and the third camera, for calculating a depth data based on the first infrared image, the second infrared image, and the color image; and A visual processing unit, coupled to the image processing circuit, for stereo matching the first infrared image and the second infrared image to generate a gray-scale matching image; and the gray-scale matching image and the color image Perform color matching to generate a color matching image. The interactive image processing system according to claim 1, wherein the image processing circuit combines the color image and the depth data into a first data packet. The interactive image processing system according to claim 1, wherein the image processing circuit combines the first infrared image and the depth data into a second data packet. The interactive image processing system according to claim 1, wherein the image processing circuit combines the first infrared image and the second infrared image into a third data packet. The interactive image processing system as described in claim 1, further includes: An image signal processor, coupled to the visual processing unit, is used to perform automatic white balance and exposure value correction on the first infrared image and the second infrared image; A central processing unit, coupled to the image signal processor, for generating an operation result; and A digital signal processor, coupled between the image signal processor and the central processing unit, is used to convert the first infrared image and the second infrared image into a three-dimensional image based on the depth data. The interactive image processing system according to claim 5, wherein the central processing unit is used to simultaneously execute a plurality of codes for gesture detection and tracking, spatial scanning, object scanning, augmented reality perspective, and simultaneous positioning and mapping At least one of them. The interactive image processing system as described in claim 1, further includes: An infrared light source is used to provide an ambient light source to the first camera and the second camera for the infrared image converter. An interactive image processing system, including: A first camera is used to generate a first color infrared image; A second camera is used to generate a second color infrared image; An image processing circuit, coupled to the first camera and the second camera, for calculating a depth data based on the first color infrared image and the second color infrared image; and A visual processing unit, coupled to the image processing circuit, is used for stereo matching the first color infrared image and the second color infrared image to generate a color matching image. The interactive image processing system according to claim 8, wherein the image processing circuit captures red pixels, green pixels and blue pixels of the first color infrared image and the second color infrared image to convert the first color The infrared image and the color components of the second color infrared image are combined into a first data packet, and the first data packet is used for augmented reality perspective applications. The interactive image processing system according to claim 8, wherein the image processing circuit captures infrared components of the first color infrared image and the second color infrared image to generate a second data packet, and the second data packet is Used for simultaneous positioning and mapping, gesture motion detection and tracking, and six-dimensional freedom applications. The interactive image processing system according to claim 8, wherein the image processing circuit combines the depth data and the color components of the first color infrared image into a third data packet, and the third data packet is located in the first The application of space scanning and object scanning based on the camera's perspective. The interactive image processing system according to claim 8, wherein the image processing circuit combines the depth data and the color components of the second color infrared image into a fourth data packet, and the fourth data packet is located in the second The application of space scanning and object scanning based on the camera's perspective. The interactive image processing system as described in claim 8 additionally includes: An image signal processor, coupled to the visual processing unit, is used to perform automatic white balance and exposure value correction on the first color infrared image and the second color infrared image; A central processing unit, coupled to the image signal processor, for generating an operation result; and A digital signal processor, coupled between the image signal processor and the central processing unit, is used to convert the first color infrared image and the second color infrared image into a stereo image according to the depth data. The interactive image processing system according to claim 13, wherein the central processing unit is used to simultaneously execute a plurality of codes for gesture detection and tracking, spatial scanning, object scanning, augmented reality perspective, and simultaneous positioning and mapping At least one of them. The interactive image processing system as described in claim 8 additionally includes: A random point infrared light source is used to provide an ambient light source to the first camera and the second camera for infrared image conversion.

Images