TWI889317B - 3d around view monitoring system - Google Patents

Abstract

A three-dimensional around view monitoring system includes multiple photography modules, multiple optical radars, and a computer device. The computer device converts a first coordinate of each pixel of a still image captured by each camera module at the same time point to the corresponding optical radar at the same time through the previously stored and corresponding coordinate conversion matrix. A second coordinate of the point cloud data scanned at the time point, and the color of the second coordinate is equal to the color of the pixel corresponding to the first coordinate to generate a corresponding color point cloud data. The computer device then splices the color point cloud data of the optical radars at the same time point to generate a spliced ​​color point cloud data as a three-dimensional surround image.

Description

3D Surround System

The present invention relates to a surround system, and more particularly to a three-dimensional surround system capable of providing accurate distance information.

Most of the existing vehicle-based around view monitoring (AVM) systems use four cameras installed in the front, rear, left and right directions of the vehicle to capture the vehicle's surroundings and generate four images. The four images are then stitched together by a processor or computing chip to obtain a 360-degree surround image. This known approach will reduce the visibility and recognition of the captured objects when the ambient light is insufficient, and the four images are prone to ghosting or distortion at the joints. Therefore, whether there are other vehicle-based surround view systems to improve the problem of learning technology, or even provide more image information, has become a problem to be solved.

Therefore, an object of the present invention is to provide a three-dimensional surround system that can provide accurate distance information.

Therefore, the present invention provides a three-dimensional surrounding system, which is applicable to a vehicle and includes a plurality of camera modules, a plurality of optical radars, and a computer device. The plurality of camera modules are arranged on the vehicle and take pictures of the surrounding environment of the vehicle to generate a plurality of dynamic images respectively. Each of the dynamic images includes a plurality of static images at different time points. The optical radars are arranged on the vehicle and scan the surrounding environment of the vehicle to generate a plurality of point cloud data at different time points.

The computer device is installed on the vehicle and electrically connected to the camera modules and the optical radars, and stores a plurality of coordinate conversion matrices respectively corresponding to the optical radars. Each of the coordinate conversion matrices is used to convert any point in the static image of the corresponding camera module into a point in the point cloud data of the corresponding optical radar.

The computer device converts a first coordinate of each pixel of the static image of each camera module at the same time point into a second coordinate of the point cloud data of the corresponding optical radar at the same time point through the corresponding coordinate conversion matrix, and sets the color of the second coordinate to be equal to (i.e., set to) the color of the pixel corresponding to the first coordinate to generate a corresponding color point cloud data. The computer device splices the color point cloud data of the optical radars at the same time point to generate a spliced color point cloud data as a three-dimensional panoramic image.

In some implementations, the computer device, based on a setting instruction, also displays at least one distance between at least one point selected by the setting instruction and the vehicle in the three-dimensional panoramic image, so that a user can know the distance between at least one specific object corresponding to the at least one point and the vehicle.

In other embodiments, the computer device executes a calibration procedure of the corresponding coordinate conversion of the camera module and the optical radar in advance. In the calibration procedure, the corresponding camera module and the optical radar respectively shoot and scan a feature map, and the computer device performs feature comparison based on the feature map in the static image and the feature map in the corresponding point cloud data to obtain the corresponding coordinate conversion matrix.

In some implementations, the feature map includes a plurality of checkerboard squares, the computer device compares the corners of the checkerboard squares in the corresponding static image with those in the point cloud data, the coordinate transformation matrix includes a rotation matrix and a translation vector, the first coordinate of the corresponding static image is rotated by the rotation matrix, and then translated to the second coordinate of the corresponding point cloud data by the translation vector.

The effect of the present invention is that by performing coordinate conversion between the corresponding camera module and the optical radar through each coordinate conversion matrix, the color of each pixel of the dynamic images and the static images taken by the camera modules can be displayed at the correct position of the point cloud data scanned by the optical radars, thereby generating a three-dimensional panoramic image that can provide accurate distance information.

Before the present invention is described in detail, it should be noted that similar components are represented by the same reference numerals in the following description.

Referring to FIG. 1 , an embodiment of the three-dimensional surround system of the present invention is applicable to a vehicle 9 and includes four camera modules 31-34, four optical radars 21-24, and a computer device 1.

The four camera modules 31-34 are respectively arranged at the front, rear, left and right sides of the vehicle 9, and use wide-angle lenses to shoot four directions of the vehicle 9 to generate four dynamic images. Each dynamic image includes multiple static images at different time points. In other words, the splicing result of the four dynamic images is a panoramic image belonging to a known technology.

The four optical radars 21-24 are respectively arranged at the front, rear, left, and right sides (surroundings) of the vehicle 9, for example, respectively above the four camera modules 31-34, and respectively scan the four directions of the vehicle 9 to respectively generate a plurality of point cloud data at different time points, that is, each of the optical radars 21-24 will generate a plurality of point cloud data at different time points. Each of the point cloud data includes a time tag, and the time tag indicates the time point corresponding to the point cloud data.

The computer device 1 is, for example, a vehicle computer or other computing device, and includes a processing unit (such as a processor), an input unit, and a display unit, wherein the input unit and the display unit are integrated into a touch screen, for example. The computer device 1 is disposed on the vehicle 9 and is electrically connected to the camera modules 31-34 and the optical radars 21-24 to receive the four dynamic images and the point cloud data, and store four coordinate conversion matrices corresponding to the four optical radars 21-24 respectively.

More specifically, the computer device 1 performs a calibration procedure for coordinate conversion between the coordinate systems of the corresponding camera module and the optical radar, such as 21 and 31, 22 and 32, 23 and 33, and 24 and 34. In the calibration procedure, the corresponding camera module (such as 31) and the optical radar (such as 21) respectively photograph and scan a feature map, and the feature map includes a plurality of chessboard-like squares, such as a plurality of black and white rectangular chessboards, but not limited thereto. The computer device 1 compares the features (i.e., the corners of the chessboard-shaped squares) of the feature map in the static image of the corresponding camera module (such as 31) with the feature map in the point cloud data of the corresponding optical radar (such as 21) to obtain a corresponding coordinate conversion matrix. The coordinate conversion matrix is used to convert any coordinate point in the static image of the corresponding camera module (such as 31) into a coordinate point in the point cloud data of the corresponding optical radar (such as 21).

The coordinate transformation matrix between the static image of each camera module and the corresponding point cloud data of the optical radar includes a rotation matrix R and a translation vector t. The 2D-3D correspondence problem is converted into a 3D-3D matching problem by the Efficient Perspective-n-Point (EPNP) algorithm, and the rotation matrix R and the translation vector t can be obtained by the Iterative Closest Point (ICP) algorithm.

The EPNP algorithm solution process requires the knowledge of n 3D reference points in the world coordinate system (i.e. the coordinate system corresponding to the optical radar) and the reference points in the corresponding 2D image (i.e. the static image), as well as the internal parameters K of the camera (i.e. the camera module). First, four virtual control points c ^w are found in the 3D point cloud p ^w in the world coordinate system. The first point is obtained by the centroid formula of the point cloud. The general selection method for the other three points is to determine the three axes of the point cloud data distribution through principal component analysis (PCA) to solve the other three control points.

Through the relationship between the 3D point cloud and the virtual control point c ^w , the homogeneous center of gravity coordinate α (α is a weight vector) is obtained. Because the relationship between the camera projection model, the homogeneous center of gravity coordinate α, the camera's internal parameters K, and the 2D point coordinates are known, the control point c ^c in the camera coordinate system can be solved through SVD (singular value decomposition). Then, the 3D point p ^c in the camera coordinate system is obtained through the control point c ^c and the corresponding homogeneous center of gravity coordinate α.

Finally, the coordinate transformation matrix R’ and a translation vector t’ are solved by the ICP algorithm for 3D-3D point cloud matching. The transformation matrix required for the final 2D-3D correspondence, including the rotation matrix R and the translation vector t, can be obtained by integrating the 2D-3D transformation process (EPNP algorithm) and the 3D-3D matching process (ICP algorithm).

The computer device 1 converts a first coordinate of each pixel of the static image of each of the photographic modules 31-34 at the same time point into a second coordinate of the point cloud data of the corresponding optical radar 21-24 at the same time point via the corresponding coordinate conversion matrix, and makes the color (such as RGB value or other color value) of the second coordinate equal to (i.e., sets to) the color (such as RGB value or other color value) of the pixel corresponding to the first coordinate to generate a corresponding color point cloud data.

The computer device 1 splices the color point cloud data of the optical radars 21-24 at the same time point to generate a spliced color point cloud data. In other words, the computer device 1 converts the point cloud information that originally has no color information at different time points into the color point cloud information with color information, so that the computer device 1 displays the spliced color point cloud data in sequence according to the time label on the display unit as a three-dimensional panoramic image that can provide accurate distance information.

For example, according to a setting instruction (such as displaying the closest distance of the rear object) input by the user through the touch screen, the computer device 1 also displays in the three-dimensional panoramic image a closest distance between an object located behind the vehicle 9 (i.e., one point of the spliced color point cloud data at the displayed time point) and the vehicle 9. For another example, according to another setting instruction (such as displaying the closest distances of the left and right objects) input by the user through the touch screen, the computer device 1 also displays in the three-dimensional panoramic image two other closest distances between another object located on the left side of the vehicle 9 and another object located on the right side of the vehicle 9 and the vehicle 9, respectively.

It should be particularly noted that in this embodiment, the four optical radars 21-24 are all solid-state optical radars with a wide-angle scanning range. In other embodiments, the number of optical radars may be a single mechanical radar disposed above the vehicle 9 or at a higher position of the vehicle body, or a single mechanical optical radar disposed above the vehicle 9 plus two solid-state optical radars disposed on the left and right sides of the vehicle 9, respectively. In addition, the optical radars may be other pluralities and do not necessarily have to be the same as the number of the camera modules 31-34, as long as the optical radars and the camera modules can obtain the environment of all sides of the vehicle 9 (i.e., the dynamic images and the point cloud data) as the deployment principle.

In addition, in other embodiments, when the features of the chessboard squares included in the feature map are incomplete during the calibration process, for example, due to the low resolution of the hardware causing the corners of the chessboard squares to be partially or completely incomplete, the complete features of the feature map can be obtained through pre-calibration by keeping the corresponding camera module and the optical radar as a combination with unchanged relative positions and by swinging the combination for shooting and scanning.

In summary, by performing coordinate conversion between the corresponding camera module and the optical radar using each coordinate conversion matrix, the color of each pixel of the dynamic images and the static images captured by the camera modules can be displayed at the correct position of the point cloud data scanned by the optical radars. Furthermore, by displaying the stitched color point cloud data and the selected distance information, a three-dimensional surround system capable of providing accurate distance information can be realized, thereby achieving the purpose of the present invention.

However, the above is only an embodiment of the present invention and should not be used to limit the scope of implementation of the present invention. All simple equivalent changes and modifications made according to the scope of the patent application of the present invention and the content of the patent specification are still within the scope of the present patent.

1: Computer device 21~24: Optical radar 31~34: Photographic module 9: Vehicle

Other features and functions of the present invention will be clearly presented in the implementation method with reference to the drawings, wherein: Figure 1 is a block diagram illustrating an implementation example of the three-dimensional ambient system of the present invention.

1:Computer device

21~24: Optical radar

31~34: Photography module

9: Vehicles

Claims (2)

A three-dimensional surrounding system is applicable to a vehicle and comprises: A plurality of camera modules, which are arranged on the vehicle and photograph the surrounding environment of the vehicle to generate a plurality of dynamic images respectively, each of which includes a plurality of static images at different time points; A plurality of optical radars, which are arranged on the vehicle and scan the surrounding environment of the vehicle to generate a plurality of point cloud data at different time points; and A computer device is installed on the vehicle and electrically connected to the camera modules and the optical radars, and stores a plurality of coordinate conversion matrices corresponding to the optical radars, each of which is used to convert any point in the static image of the corresponding camera module into a point in the point cloud data of the corresponding optical radar. The computer device converts a first coordinate of each pixel of the static image of each camera module at the same time point to a second coordinate of the point cloud data of the corresponding optical radar at the same time point through the corresponding coordinate conversion matrix, and sets the color of the second coordinate to be equal to (i.e., set to) the color of the pixel corresponding to the first coordinate to generate a corresponding color point cloud data. The computer device splices the color point cloud data of the optical radars at the same time point to generate a spliced color point cloud data as a three-dimensional panoramic image. The computer device executes a calibration procedure of the corresponding coordinate conversion of the photographic module and the optical radar in advance. In the calibration procedure, the corresponding photographic module and the optical radar respectively photograph and scan a feature map. The computer device performs feature comparison based on the feature map in the static image and the feature map in the corresponding point cloud data to obtain the corresponding coordinate conversion matrix. The feature map includes a plurality of chessboard-like squares. The computer device compares the corners of the chessboard-like squares in the corresponding static image with those in the point cloud data. The coordinate conversion matrix includes a rotation matrix and a translation vector. The first coordinate of the corresponding static image is rotated by the rotation matrix and then translated to the second coordinate of the corresponding point cloud data by the translation vector. The coordinate conversion matrix between the static image of each camera module and the point cloud data of the corresponding optical radar includes the rotation matrix R and the translation vector t. The computer device uses Efficient Perspective N-Point Camera Attitude Estimation (Efficient Perspective N-Point Camera Attitude Estimation) The EPNP (Perspective-n-Point) algorithm converts the 2D-3D correspondence problem into a 3D-3D matching problem, and obtains the rotation matrix R and the translation vector t by the iterative closest point (ICP) algorithm. A three-dimensional surround system as described in claim 1, wherein the computer device, based on a setting instruction, also simultaneously displays in the three-dimensional surround image at least one distance between at least one point selected by the setting instruction and the vehicle, so that a user can know the distance between at least one specific object corresponding to the at least one point and the vehicle.

Images