WO2024114119A1 - Sensor fusion method based on binocular camera guidance - Google Patents

Abstract

Disclosed in the present invention is a sensor fusion method based on binocular camera guidance. In the present invention, a fusion method based on binocular vision guidance is used, wherein a motion trajectory of a target can be established by using visual target information, and the position of the target in a three-dimensional space is then predicted; a data layer fusion method is used, wherein a visual prediction result is used to assist a laser radar in delimiting a detection range, such that the volume of operational data is greatly reduced, and the requirement for the performance of a computing platform is low; and on the basis of a target detection result acquired by means of binocular vision, a target depth can be accurately acquired, target detection information is expanded, and the robustness of target detection is enhanced.

Description

A sensor fusion method based on binocular camera guidance Technical Field

The present invention relates to the technical field of intelligent vehicle environment perception, and in particular to a sensor fusion method based on binocular camera guidance.

Background technique

At present, there are specific research works and engineering applications that use a single type of sensor such as vision or lidar to achieve the function of target detection. Visual sensors are widely used in target detection due to their low cost, easy target identification and rich information data collection. Lidar has the ability to actively detect targets and has high ranging accuracy. In recent years, with the optimization and upgrading of radar working principles, it has the possibility of mass production. With the increasing complexity of traffic conditions, the detection method based on a single sensor is restricted by the complexity of the algorithm and the insufficient amount of detection information, and it is difficult to meet the current requirements for target detection robustness and detection speed;

In order to make up for the shortcomings of the detection effect of a single sensor, the method of using multiple sensors for fusion detection has gradually developed. This detection method can also improve the robustness of the system in complex environments. Therefore, the detection method of multi-sensor information fusion is used more and more frequently in the research of target detection. At present, the data fusion of multiple sensors is divided into three levels: decision layer fusion, feature layer fusion and data layer fusion. In the paper "Outdoor Positioning by Fusion of Binocular Vision and 2D LiDAR", Liu Zhengxuan used binocular vision to detect the relative posture of the vehicle, and combined the LiDAR data at multiple moments in a time window for fusion, and finally obtained a local sub-graph. After denoising and matching, the high-precision positioning of the vehicle was achieved. In the paper "Research on SLAM Method Based on LiDAR and Vision Fusion", Gong Chaoguang designed a classification and fusion framework for LiDAR and binocular cameras. By classifying and matching the extracted visual features with the laser point cloud, the feature line segments are merged according to the matching results to obtain the fused point cloud information. Both papers perform data fusion at the data layer level of binocular cameras and LiDARs, which has high accuracy but is not conducive to engineering implementation;

Patent "CN114463303A" proposes a road based on the fusion of binocular camera and laser radar Target detection method, this method combines LiDAR technology with visual sensors, and combines the YOLOv4 detection model to fuse the radar and binocular vision results, thereby providing a smarter and more accurate technical means for target detection; Patent "CN111340797A" proposes a LiDAR and binocular camera data fusion detection method and system, this method uses the detection results of the binocular camera to fuse with the detection results of the LiDAR, and determines whether the fusion overlap rate is greater than a certain threshold. If it is greater than the set value, the fusion result is output, otherwise the target detection needs to be re-performed. This method improves the measurement accuracy. These two patents fuse the decision layer of the binocular camera and the LiDAR, which has a good engineering effect but is not conducive to performance improvement. Most existing studies use traditional methods to fuse the camera and LiDAR data layer or decision layer for target detection. These methods are difficult to achieve a balance between engineering applicability and detection performance;

Based on the current research status, the present invention proposes a sensor fusion method based on binocular camera guidance, which indirectly realizes the data fusion of binocular camera and lidar at the data layer. It can combine the advantages of binocular camera and lidar sensor, effectively improve the accuracy and robustness of target detection, and meet the needs of engineering applications.

Summary of the invention

The present invention provides a sensor fusion method based on binocular camera guidance, which is used to solve the problems existing in the background technology.

A sensor fusion method based on binocular camera guidance includes the following steps:

Step 1: Based on the visual image information collected by the binocular camera and the 3D point cloud information collected by the lidar, calibrate the left and right cameras that make up the binocular camera, and jointly calibrate the lidar and the left camera;

Step 2: Collect the forward environmental information of the electronically guided rubber-wheeled vehicle through binocular cameras and laser radar;

Step 3: Calculate the depth information of the front environment of the electronic guided rubber-wheeled vehicle through a stereo matching algorithm;

Step 4: Obtain the visual information of the target in the image captured by the camera, including the target category and two-dimensional plane position information, through the target detection neural network based on YOLOv7;

Step 5: Combine the target visual information, environmental depth information and target time information output by the neural network to obtain the target detection result based on stereo vision;

Step 6: Based on the target time information in the target detection result obtained in step 5, establish the target's motion trajectory and predict the target's three-dimensional position information at the laser radar acquisition time;

Step 7: Based on the three-dimensional position information of the target predicted in step 6, segment and cluster the point cloud collected by the lidar, and output the target detection result;

Step 8: Track the target detected by the lidar based on the Kalman filter algorithm and the nearest neighbor algorithm, and correct and output the fused target based on the operation scenario.

The specific steps of performing camera calibration in step 1 are as follows:

The visual image information collected by the binocular camera is calibrated by Zhang Zhengyou method to obtain the left camera internal parameters _ML , distortion coefficient _DL , the right camera internal parameters _MR , distortion coefficient _DR and the external parameters _MLR between the left and right cameras.

The specific steps of performing joint calibration in step 1 are as follows:

The calibration plate is used to perform joint calibration on the three-dimensional point cloud information collected by the laser radar and the two-dimensional position information in the visual image information collected by the left camera to obtain the external parameters M _LC between the laser radar and the left camera.

The calculation method in step 3 is as follows:

The corresponding pixel points of the left and right views in the same scene are matched to obtain the disparity map. The projection matrix Q from the two-dimensional position of the image to the three-dimensional position in space is obtained based on the Bouguet algorithm to calculate the target depth information.

The step four includes: collecting and creating a data set of target images in the operating environment, using the data set with existing target annotations as a training set for the neural network, obtaining the weight coefficient of the neural network through the training model, and then predicting the image captured by the left camera to obtain the target detection result containing the target category and the two-dimensional bounding box information describing the target plane position.

The step five includes: using the calculation method in step three to project the target plane information obtained in step four into three-dimensional space, obtaining the target detection result of the stereoscopic information described by the three-dimensional bounding box, and then combining the image timestamp to obtain the target detection result based on stereoscopic vision.

The step seven comprises:

Use the RANSAC algorithm to segment the ground and extract the non-ground target point cloud. The acquired stereo position information is clustered into point clouds to obtain target point cloud clusters, and the point cloud target detection results of the stereo information described by a three-dimensional bounding box are output. The target information obtained by the binocular camera guides the laser radar to accurately measure the distance to the target;

The steps of point cloud clustering are as follows:

(1) The laser radar obtains the target category and historical trajectory information detected by the binocular camera, models and predicts the spatial position of the target, and assists in defining the detection range;

(2) Select any point M in the point cloud within the detection range, and adaptively set the clustering threshold according to the detection range. Search for k points of M's neighbors based on the KDTree data structure, select points whose distance to point M is less than the clustering threshold and store them in set Q. If the number of points in Q increases, select any point other than point M in Q to update point M. Continue searching until the number of elements in set Q stops increasing.

(3) Repeat steps (1) and (2) for all the detection objects in the image acquired by the binocular camera, and then use the clustering detection algorithm to complete the detection of all targets, and output the target category, point cloud, three-dimensional rectangular box and timestamp.

Before clustering the point cloud collected by the LiDAR, motion distortion compensation is used to eliminate the motion distortion inside the point cloud, as follows:

IMU is used to compensate for the motion distortion of the lidar. While receiving the lidar scanning data, the data and the vehicle attitude angle data collected by the IMU are saved in the same circular queue. The processor calculates the adoption time of each laser point based on the sampling time interval of the lidar and the data timestamp, and searches for two temporally adjacent frames of data in the IMU data queue. The laser points at the same measurement moment are paired with the vehicle motion attitude by using spherical linear interpolation. This allows the coordinates of each laser point of a frame of lidar scanning data to be converted to the corresponding position coordinate system of the electronically guided rubber-wheeled vehicle at the same moment, so that all point clouds collected by the lidar within a period of time are unified to one moment, thereby realizing the motion distortion compensation of the lidar.

The method for tracking the target based on the Kalman filter algorithm and the nearest neighbor algorithm in step eight is as follows:

Based on the target motion model, the Kalman filter group is constructed and initialized, and the relevant parameters are configured to calculate the target. The center of mass position of the point cloud is determined by the Kalman filter group. The center of mass position of the target at the current timestamp is used to predict the center of mass position of the target at the next frame of the point cloud. In the next frame of the point cloud, the nearest neighbor algorithm is used to search for a matching combination of real targets and predicted targets to achieve target association between frames, that is, to obtain the spatial position of the same target in two adjacent frames. The target state of the Kalman filter group is updated using the spatial position and timestamp information of the target in the next frame of the point cloud, and then the target state is predicted. The above process is repeated to track the target.

Compared with the prior art, the present invention has the following beneficial effects: the present invention adopts a binocular vision-guided fusion method, which can use visual target information to establish its motion trajectory, and then predict its position in three-dimensional space; adopts a data layer fusion method, uses visual prediction results to assist laser radar in defining the detection range, greatly reduces the amount of computational data, and has low requirements on computing platform performance;

The target detection results obtained based on binocular vision can accurately obtain the target depth, expand the target detection information and enhance the robustness of target detection.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a sensor fusion method of the present invention;

FIG2 is a flow chart of a sensor fusion method of the present invention;

Figure 3 is a flow chart of LiDAR point cloud clustering;

Figure 4 is the effect diagram after point cloud clustering;

Figure 5 is a schematic diagram of target tracking.

Detailed ways

A specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation.

As shown in FIG. 1 to FIG. 5 , a sensor fusion method based on binocular camera guidance provided by an embodiment of the present invention includes the following steps:

Step 1: Based on the visual image information collected by the binocular camera and the 3D point cloud information collected by the lidar, calibrate the left and right cameras that make up the binocular camera, and jointly calibrate the lidar and the left camera;

The fusion method proposed in the present invention first performs target detection and network tracking based on a binocular camera. A binocular camera is generally composed of two horizontally placed left and right cameras. The two cameras image the target respectively. Since there is a baseline of a certain length between the cameras, the imaging position of the target will be different. The positional relationship between the target and the camera can be calculated using the difference in imaging position and the baseline length.

The specific steps for camera calibration in step 1 are as follows:

The visual image information collected by the binocular camera is calibrated by Zhang Zhengyou method to obtain the left camera internal parameters _ML , distortion coefficient _DL , the right camera internal parameters _MR , distortion coefficient _DR and the external parameters _MLR between the left and right cameras;

The specific steps for joint calibration in step 1 are as follows:

The calibration plate is used to jointly calibrate the three-dimensional point cloud information collected by the laser radar and the two-dimensional position information in the visual image information collected by the left camera to obtain the external parameters M _LC between the laser radar and the left camera, and then the measurement values in each coordinate system are converted to the train coordinate system;

Binocular correction dedistorts and aligns the views captured by the two cameras according to the relative position relationship obtained during camera calibration and the internal parameter data of the left and right cameras, so that the coordinates of the imaging origins of the left and right views are consistent, the optical axes of the two cameras are parallel, and the epipolar lines are aligned and coplanar with the imaging planes of the left and right cameras;

Step 2: Collect the forward environmental information of the electronically guided rubber-wheeled vehicle through binocular cameras and laser radar;

Step 2 is performed on the basis of keeping the positions of the binocular camera and the laser radar unchanged;

Step 3: Calculate the depth information of the front environment of the electronic guided rubber-wheeled vehicle through a stereo matching algorithm;

The calculation method in step 3 is as follows: match the pixel points corresponding to the left and right views in the same scene to obtain the disparity map, obtain the projection matrix Q from the two-dimensional position of the image to the three-dimensional position of the space based on the Bouguet algorithm, and calculate the target depth information;

Step 4: Obtain the visual information of the target in the image captured by the camera, including the target category and two-dimensional plane position information, through the target detection neural network based on YOLOv7;

The above step 4 is as follows: collect and create a dataset of target images in the operating environment, use the dataset with existing target annotations as the training set of the YOLOv7 neural network, obtain the weight coefficient of the neural network through the training model, and then predict the images captured by the left camera to obtain the target category and description target. The target detection result of the two-dimensional bounding box information of the plane position, the present invention uses the target detection algorithm based on deep learning to detect the target in a single image, which can identify traffic participants such as pedestrians and vehicles, and classify them to obtain the target detection result based on stereo vision;

In addition, YOLOv7 adopted in the present invention improves the detection speed and accuracy by reforming multiple internal architectures based on YOLOv5; YOLOv7 mainly uses MP structure and ELAN, which can perform deep learning more efficiently by controlling the shortest gradient path and deeper network structure; the loss function of YOLOv7 is divided into three main parts: target confidence loss, coordinate loss and classification loss, which mainly performs data candidate matching in the matching strategy process, retains the data with the smallest loss, and realizes high-precision target detection;

Step 5: Combine the target visual information, environmental depth information and target time information output by the neural network to obtain the target detection result based on stereo vision;

The above step 5 uses the calculation method in step 3 to project the target plane information obtained in step 4 into three-dimensional space, obtain the target detection result of the three-dimensional information described by the three-dimensional bounding box, and then combine it with the image timestamp to obtain the target detection result based on stereoscopic vision.

Step 6: Based on the target time information in the target detection result obtained in step 5, establish the target's motion trajectory and predict the target's three-dimensional position information at the laser radar acquisition time;

Step 7: Based on the three-dimensional position information of the target predicted in step 6, segment and cluster the point cloud collected by the lidar, and output the target detection result;

The above step seven is specifically as follows: In combination with the operating environment characteristics of the electronic guided rubber-wheeled vehicle, the RANSAC algorithm is used to segment the ground, and the non-ground target point cloud is extracted. The point cloud is clustered based on the stereo position information obtained in step six to obtain the target point cloud cluster, and the point cloud target detection result of the stereo information described by the three-dimensional bounding box is output. The target information obtained by the binocular camera is used to guide the lidar to accurately measure the distance to the target; generally, the binocular camera is used to obtain the category and position of the target, and the sampling delay of the image and the processing time of the target detection algorithm need to be considered. At present, the lidar point cloud clustering mainly adopts the Euclidean clustering method for clustering. This clustering method believes that in all the point cloud data collected by the lidar, the distance between any two points in the point cloud of the same object will be less than a certain value. Therefore, points with a distance less than a certain value can be clustered. This clustering method has high accuracy, but its clustering speed is not advantageous. Although this method uses KD-Tree to speed up the data search speed, it is still insufficient in high-speed changing scenes. The present invention innovatively uses binocular cameras to guide laser radars to cluster targets, which can not only improve the clustering speed, but also have better results on the accuracy and robustness of the final recognition results. The steps of point cloud clustering are as follows:

(1) The laser radar obtains the target category and historical trajectory information detected by the binocular camera, models and predicts the spatial position of the target, and assists in defining the detection range;

(2) Select any point M in the point cloud within the detection range, and adaptively set the clustering threshold according to the detection range. Search for k points of M's neighbors based on the KDTree data structure, select points whose distance to point M is less than the clustering threshold and store them in set Q. If the number of points in Q increases, select any point other than point M in Q to update point M. Continue searching until the number of elements in set Q stops increasing.

(3) Repeat steps (1) and (2) for all the detection objects in the image acquired by the binocular camera, and then use the clustering detection algorithm to complete the detection of all targets, and output the target category, point cloud, three-dimensional rectangular box and timestamp.

In addition, when the laser radar follows the movement of the rubber-wheeled vehicle, the electronically guided rubber-wheeled vehicle moves at high speed, and the reference posture of the rubber-wheeled vehicle is different at different times, resulting in motion distortion of the laser point cloud collected by the laser point. At the same time, if the laser radar scanning frequency is low, the data of the laser point is not obtained instantaneously, which will also cause motion distortion of the laser point cloud. Therefore, before clustering the point cloud collected by the laser radar, motion distortion compensation is used to eliminate the motion distortion inside the point cloud, as follows:

IMU is used to compensate for the motion distortion of the laser radar. While receiving the laser radar scanning data, the data and the vehicle attitude angle data collected by the IMU are saved in the same circular queue. The processor calculates the adoption time of each laser point according to the sampling time interval of the laser radar and the data timestamp, and searches for two frames of data adjacent in time in the IMU data queue. The laser points at the same measurement moment are paired with the vehicle motion posture by using spherical linear interpolation. This allows the coordinates of each laser point of a frame of laser radar scanning data to be converted to the corresponding position coordinate system of the electronic guided rubber-wheeled vehicle at the same moment, so that all point clouds collected by the laser radar within a period of time are unified to one moment. Realize motion distortion compensation for lidar.

Step 8: Track the target detected by the lidar based on the Kalman filter algorithm and the nearest neighbor algorithm, and correct and output the fused target based on the operation scenario;

The method for tracking the target based on the Kalman filter algorithm and the nearest neighbor algorithm in the above step eight is as follows:

Based on the target motion model, a Kalman filter group is constructed and initialized, and related parameters are configured. The centroid position of the target point cloud is calculated. Based on the Kalman filter group, the centroid position of the target at the next frame point cloud is predicted by the centroid position of the target at the current timestamp. In the next frame point cloud, a matching combination of real targets and predicted targets is searched based on the nearest neighbor algorithm to achieve target association between frames, that is, the spatial position of the same target in two adjacent frames is obtained. The target state of the Kalman filter group is updated using the spatial position and timestamp information of the target in the next frame point cloud, and then the target state is predicted. The above process is repeated to achieve target tracking. The present invention proposes a sensor fusion framework guided by a binocular camera, and provides a fusion method of a sensor data layer guided by a binocular camera, which takes into account target detection performance and engineering applications, realizes the complementary advantages between sensors with different functions, and improves the intelligence of the target detection system. Compared with the 3D target detection system, the visual target detection system based on deep learning is mature and has lower performance requirements on the computing platform.

It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit and essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims (9)

A sensor fusion method based on binocular camera guidance, characterized in that it includes the following steps: Step 1: Based on the visual image information collected by the binocular camera and the 3D point cloud information collected by the lidar, calibrate the left and right cameras that make up the binocular camera, and jointly calibrate the lidar and the left camera; Step 2: Collect the forward environmental information of the electronically guided rubber-wheeled vehicle through binocular cameras and laser radar; Step 3: Calculate the depth information of the front environment of the electronic guided rubber-wheeled vehicle through a stereo matching algorithm; Step 4: Obtain the visual information of the target in the image captured by the camera, including the target category and two-dimensional plane position information, through the target detection neural network based on YOLOv7; Step 5: Combine the target visual information, environmental depth information and target time information output by the neural network to obtain the target detection result based on stereo vision; Step 6: Based on the target time information in the target detection result obtained in step 5, establish the target's motion trajectory and predict the target's three-dimensional position information at the laser radar acquisition time; Step 7: Based on the three-dimensional position information of the target predicted in step 6, segment and cluster the point cloud collected by the lidar, and output the target detection result; Step 8: Track the target detected by the lidar based on the Kalman filter algorithm and the nearest neighbor algorithm, and correct and output the fused target based on the operation scenario. The sensor fusion method based on binocular camera guidance according to claim 1 is characterized in that the specific steps of performing camera calibration in step 1 are as follows: The visual image information collected by the binocular camera is calibrated by Zhang Zhengyou method to obtain the left camera internal parameters _ML , distortion coefficient _DL , the right camera internal parameters _MR , distortion coefficient _DR and the external parameters _MLR between the left and right cameras. The sensor fusion method based on binocular camera guidance according to claim 1 or 2 is characterized in that the specific steps of performing joint calibration in step 1 are as follows: The 3D point cloud information collected by the laser radar and the visual image information collected by the left camera are compared by the calibration plate. The two-dimensional position information in the information is jointly calibrated to obtain the external parameters M _LC between the laser radar and the left camera. The sensor fusion method based on binocular camera guidance according to claim 1, characterized in that the calculation method in step three is as follows: The corresponding pixel points of the left and right views in the same scene are matched to obtain the disparity map. The projection matrix Q from the two-dimensional position of the image to the three-dimensional position in space is obtained based on the Bouguet algorithm to calculate the target depth information. A sensor fusion method based on binocular camera guidance as described in claim 1, characterized in that step four includes: collecting and creating a data set of target images in the operating environment, using the data set with existing target annotations as a training set for the neural network, obtaining the weight coefficient of the neural network through the training model, and then predicting the image captured by the left camera to obtain a target detection result containing the target category and the two-dimensional bounding box information describing the target plane position. A sensor fusion method based on binocular camera guidance as described in claim 1, characterized in that the step five includes: using the calculation method in step three to project the target plane information obtained in step four into three-dimensional space, obtaining the target detection result of the stereo information described by the three-dimensional bounding box, and then combining the image timestamp to obtain the target detection result based on stereo vision. The sensor fusion method based on binocular camera guidance according to claim 1, characterized in that the step seven comprises: The RANSAC algorithm is used to segment the ground and extract the non-ground target point cloud. The point cloud is clustered based on the stereo position information obtained in step 6 to obtain the target point cloud cluster. The point cloud target detection result of the stereo information described by the three-dimensional bounding box is output. The target information obtained by the binocular camera is used to guide the laser radar to accurately measure the distance to the target. The steps of point cloud clustering are as follows: (1) The laser radar obtains the target category and historical trajectory information detected by the binocular camera, models and predicts the spatial position of the target, and assists in defining the detection range; (2) Select any point M in the point cloud within the detection range, and adaptively set the clustering threshold according to the detection range. Search for k points of M’s neighbors based on the KDTree data structure, and filter out the points that are close to point M. The points whose distance is less than the clustering threshold are stored in the set Q. If the number of points in Q increases, any point other than point M is selected in Q to update point M. The search continues until the number of elements in the set Q stops increasing. (3) Repeat steps (1) and (2) for all the detection objects in the image acquired by the binocular camera, and then use the clustering detection algorithm to complete the detection of all targets, and output the target category, point cloud, three-dimensional rectangular box and timestamp. The sensor fusion method based on binocular camera guidance as claimed in claim 7 is characterized in that before clustering the point cloud collected by the laser radar, the motion distortion inside the point cloud is eliminated by using motion distortion compensation, which is specifically as follows: IMU is used to compensate for the motion distortion of the lidar. While receiving the lidar scanning data, the data and the vehicle attitude angle data collected by the IMU are saved in the same circular queue. The processor calculates the adoption time of each laser point based on the sampling time interval of the lidar and the data timestamp, and searches for two temporally adjacent frames of data in the IMU data queue. The laser points at the same measurement moment are paired with the vehicle motion attitude by using spherical linear interpolation. This allows the coordinates of each laser point of a frame of lidar scanning data to be converted to the corresponding position coordinate system of the electronically guided rubber-wheeled vehicle at the same moment, so that all point clouds collected by the lidar within a period of time are unified to one moment, thereby realizing the motion distortion compensation of the lidar. The sensor fusion method based on binocular camera guidance according to claim 1 is characterized in that the method for tracking the target based on the Kalman filter algorithm and the nearest neighbor algorithm in step eight is as follows: Based on the target motion model, a Kalman filter group is constructed and initialized, and related parameters are configured. The center of mass position of the target point cloud is calculated. The center of mass position of the target at the next frame point cloud is predicted based on the Kalman filter group through the center of mass position of the target at the current timestamp. In the next frame point cloud, a matching combination of real target and predicted target is searched based on the nearest neighbor algorithm to achieve target association between frames, that is, the spatial position of the same target in two adjacent frames is obtained. The target state of the Kalman filter group is updated using the spatial position and timestamp information of the target in the next frame point cloud, and then the target state is predicted. The above process is repeated to achieve target tracking.