CN115222815A - Obstacle distance detection method, device, computer equipment and storage medium - Google Patents

Abstract

The application relates to an obstacle distance detection method, an obstacle distance detection device, a computer device, a storage medium and a computer program product. The method comprises the following steps: acquiring original point cloud data and image data corresponding to the original point cloud data; performing ground point cloud detection based on the original point cloud data to obtain ground point cloud data, and generating a target ground point cloud area based on the ground point cloud data; detecting a contact line between the obstacle and the ground based on the image data to obtain an obstacle contact line; projecting based on the target ground point cloud area to obtain a ground point cloud projection area, and determining an obstacle projection contact line corresponding to an obstacle contact line in the ground point cloud projection area; and determining obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud area, and performing distance calculation based on horizontal plane coordinate information in the obstacle contact line point cloud data to obtain an obstacle distance. The method can improve the accuracy of the obstacle distance detection.

Description

Obstacle distance detection method, device, computer equipment and storage medium

technical field

The present application relates to the technical field of automatic driving, and in particular, to a method, apparatus, computer equipment, storage medium and computer program product for detecting distance of obstacles.

Background technique

With the development of autonomous driving technology, obstacles in the surrounding environment of autonomous vehicles need to be detected, and accurate distance information needs to be obtained to plan the route and speed of driving. Existing obstacle ranging usually detects obstacles through image data, but cannot accurately obtain the distance information of obstacles; obstacle detection through lidar can obtain distance information, but due to the point of small objects and long-distance objects. The clouds are sparse, which leads to the problem of low accuracy of obstacle distance detection.

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide an obstacle distance detection method, device, computer equipment, computer readable storage medium and computer program product that can improve the accuracy of obstacle distance detection in response to the above technical problems.

In a first aspect, the present application provides an obstacle distance detection method. The method includes:

Obtain the original point cloud data and the image data corresponding to the original point cloud data;

The ground point cloud detection is performed based on the original point cloud data, the ground point cloud data is obtained, and the target ground point cloud area is generated based on the ground point cloud data;

Based on the image data, the contact line between the obstacle and the ground is detected, and the obstacle contact line is obtained;

Projecting based on the target ground point cloud area, obtaining the ground point cloud projection area, and determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area;

Determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud area, and calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

In a second aspect, the present application also provides an obstacle distance detection device. The device includes:

an acquisition module for acquiring the original point cloud data and the image data corresponding to the original point cloud data;

The ground detection module is used to detect the ground point cloud based on the original point cloud data, obtain the ground point cloud data, and generate the target ground point cloud area based on the ground point cloud data;

The contact line detection module is used to detect the contact line between the obstacle and the ground based on the image data, and obtain the obstacle contact line;

The projection module is used for projecting based on the target ground point cloud area, obtaining the ground point cloud projection area, and determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area;

The calculation module is used to determine the obstacle contact line point cloud data corresponding to the obstacle projected contact line in the target ground point cloud area, and calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Obtain the original point cloud data and the image data corresponding to the original point cloud data;

The ground point cloud detection is performed based on the original point cloud data, the ground point cloud data is obtained, and the target ground point cloud area is generated based on the ground point cloud data;

Based on the image data, the contact line between the obstacle and the ground is detected, and the obstacle contact line is obtained;

Projecting based on the target ground point cloud area, obtaining the ground point cloud projection area, and determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area;

Determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud area, and calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by the processor, the following steps are implemented:

Obtain the original point cloud data and the image data corresponding to the original point cloud data;

The ground point cloud detection is performed based on the original point cloud data, the ground point cloud data is obtained, and the target ground point cloud area is generated based on the ground point cloud data;

Based on the image data, the contact line between the obstacle and the ground is detected, and the obstacle contact line is obtained;

Projecting based on the target ground point cloud area, obtaining the ground point cloud projection area, and determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area;

Determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud area, and calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

Obtain the original point cloud data and the image data corresponding to the original point cloud data;

The ground point cloud detection is performed based on the original point cloud data, the ground point cloud data is obtained, and the target ground point cloud area is generated based on the ground point cloud data;

Based on the image data, the contact line between the obstacle and the ground is detected, and the obstacle contact line is obtained;

Projecting based on the target ground point cloud area, obtaining the ground point cloud projection area, and determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area;

Determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud area, and calculate the distance based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

The above-mentioned obstacle distance detection method, device, computer equipment, storage medium and computer program product detect the ground point cloud data in the original point cloud data, and generate the target ground point cloud area according to the ground point cloud data. Then, the contact line between the obstacle and the ground is detected through the image data, and the target ground point cloud area is projected to obtain the ground point cloud projection area, which ensures that the projection data corresponding to the obstacle contact line exists in the ground point cloud projection area. Through the ground point cloud projection area, the obstacle projected contact line corresponding to the obstacle contact line can be accurately obtained according to the obstacle contact line, and then the accurate obstacle contact line point cloud data can be obtained through the obstacle projected contact line. The point cloud data is calculated to obtain the accurate obstacle distance, thereby improving the accuracy of obstacle distance detection.

Description of drawings

Fig. 1 is the application environment diagram of the obstacle distance detection method in one embodiment;

FIG. 2 is a schematic flowchart of a method for detecting an obstacle distance in one embodiment;

3 is a schematic flowchart of calculating the distance of a target obstacle in one embodiment;

4 is a schematic diagram of a ground point cloud division area in one embodiment;

5 is a schematic flowchart of generating and filling ground point cloud data in one embodiment;

6 is a schematic diagram of an obstacle contact line in one embodiment;

7 is a schematic flowchart of obstacle distance detection in one embodiment;

8 is a structural block diagram of an obstacle distance detection device in one embodiment;

Fig. 9 is the internal structure diagram of the computer device in one embodiment;

Figure 10 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

The obstacle distance detection method provided in this embodiment of the present application can be applied to the application environment shown in FIG. 1 . Wherein, the motion carrier terminal 102 communicates with the server 104 through the network. The data storage system may store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network server. The motion carrier terminal obtains the original point cloud data and the image data corresponding to the original point cloud data; the motion carrier terminal detects the ground point cloud based on the original point cloud data, obtains the ground point cloud data, and generates the target ground point cloud area based on the ground point cloud data; The moving carrier terminal detects the contact line between the obstacle and the ground based on the image data, and obtains the obstacle contact line; the moving carrier terminal projects based on the target ground point cloud area, obtains the ground point cloud projection area, and determines the obstacle in the ground point cloud projection area The obstacle projection contact line corresponding to the object contact line; the motion carrier terminal determines the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud area, based on the horizontal plane coordinate information in the obstacle contact line point cloud data Perform distance calculation to get the obstacle distance. The motion carrier terminal sends the obstacle distance to the server 104 and obtains the motion path, and moves according to the motion path. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, IoT devices and portable wearable devices, and the IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. . The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, or the like. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

In one embodiment, as shown in FIG. 2 , a method for detecting the distance of obstacles is provided, and the method is applied to the moving carrier terminal in FIG. 1 as an example for description. It can be understood that the method can also be applied to The server can also be applied to a system including a terminal and a server, and is realized through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

Step 202: Obtain original point cloud data and image data corresponding to the original point cloud data.

The original point cloud data refers to the collected original point cloud data of the surrounding environment of the motion carrier, and the surrounding environment may include obstacles. The image data refers to the collected original images of the surrounding environment of the moving carrier. The motion carrier refers to a movable object, and the motion carrier can be an unmanned motion carrier or a manned motion carrier.

Specifically, the motion carrier is equipped with a point cloud data acquisition device and an image acquisition device. The cloud data acquisition device and the image acquisition device have a unified coordinate system in advance, and use the same frame rate to synchronously collect data, that is, one frame of image data in the image data corresponds to A frame of point cloud data in the original point cloud data. The motion carrier terminal can acquire the original point cloud data around the motion carrier through the point cloud data acquisition device, and acquire the image data around the motion carrier through the image acquisition device. The motion carrier can be a vehicle, an aircraft, a robot, and the like.

Step 204: Perform ground point cloud detection based on the original point cloud data to obtain ground point cloud data, and generate a target ground point cloud area based on the ground point cloud data.

The ground point cloud data refers to the point cloud data representing the ground in the original point cloud data. The target ground point cloud area refers to the ground point cloud data obtained by filling the ground area in the original point cloud data with point cloud data.

Specifically, the motion carrier terminal uses a preset ground point classification algorithm to detect the original point cloud data, and obtains ground point cloud data in the original point cloud data. Then, the motion carrier terminal scans the ground point cloud data, and when a blank area is detected between the ground point cloud data, the blank area is filled with point cloud data according to the existing ground point cloud data, and the target ground point cloud area is obtained.

Step 206 , detecting the contact line between the obstacle and the ground based on the image data to obtain the obstacle contact line.

The obstacle contact line refers to the contact area between the obstacle and the ground, which is displayed in the form of a line in the image data.

Specifically, the motion carrier terminal uses a preset deep learning network model to detect the contact area between the image data and the ground, and obtains the obstacle contact line output by the model.

Step 208 , projecting based on the target ground point cloud area to obtain the ground point cloud projection area, and determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area.

Among them, projection refers to the process of projecting the 3D point cloud data of the target ground point cloud area to a 2D plane to obtain the point cloud projection data. The point cloud projection data refers to the data projected from the point cloud data to the 2D plane, and the point cloud projection data exists in the form of two-dimensional data. In one embodiment, the corresponding point cloud projection data is obtained by projecting the point cloud data to the same two-dimensional coordinate system as the image data. The ground point cloud projection area refers to the point cloud projection area obtained after the target ground point cloud area is projected. The obstacle projection contact line refers to the point cloud projection data corresponding to the obstacle contact line in the ground point cloud projection area.

Specifically, the motion carrier terminal converts the three-dimensional coordinates in the target ground point cloud area into horizontal plane coordinates according to preset calibration parameters, and obtains the projected point cloud projection data of the target ground point cloud area, that is, the ground point cloud projection area. Then, the mobile carrier terminal finds the corresponding obstacle projection contact line in the ground point cloud projection area according to the coordinate information of the obstacle contact line.

Step 210: Determine the obstacle contact line point cloud data corresponding to the obstacle projected contact line in the target ground point cloud area, and perform distance calculation based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance.

The obstacle contact line point cloud data refers to the point cloud data corresponding to the obstacle projected contact line in the target ground point cloud area. The horizontal plane coordinate information refers to the two-dimensional coordinates representing the horizontal plane among the three-dimensional coordinates of the point cloud data. The obstacle distance refers to the distance between the motion carrier and the obstacle.

Specifically, according to the projection relationship between the target ground point cloud area and the ground point cloud projection area, the motion carrier terminal finds the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud area. Then, the horizontal plane coordinates in the point cloud data of the obstacle contact line are obtained to calculate the distance, and the distance between the moving carrier and the obstacle is obtained.

In the above obstacle distance detection method, the ground point cloud data is detected in the original point cloud data, and the target ground point cloud area is generated according to the ground point cloud data. Then, the contact line between the obstacle and the ground is detected through the image data, and the target ground point cloud area is projected to obtain the ground point cloud projection area, which ensures that the projection data corresponding to the obstacle contact line exists in the ground point cloud projection area. Through the ground point cloud projection area, the obstacle projected contact line corresponding to the obstacle contact line can be accurately obtained according to the obstacle contact line, and then the accurate obstacle contact line point cloud data can be obtained through the obstacle projected contact line. The point cloud data is calculated to obtain the accurate obstacle distance, thereby improving the accuracy of obstacle distance detection.

In one embodiment, as shown in FIG. 3 , a schematic flowchart of calculating the distance to a target obstacle is provided; the method further includes:

Step 302, obtaining the vertical coordinate information corresponding to the point cloud data of the obstacle contact line and the coordinate information corresponding to the obstacle contact line;

Step 304 , perform point cloud coordinate calculation based on the vertical coordinate information and the coordinate information, and obtain the target horizontal plane coordinate information corresponding to the obstacle contact line point cloud data;

Step 306: Perform distance calculation based on the target horizontal plane coordinate information to obtain the target obstacle distance.

The vertical coordinate information refers to the coordinates in the vertical direction in the three-dimensional coordinates of the point cloud data. The coordinate information refers to the two-dimensional plane coordinates corresponding to the image data. The target horizontal plane coordinate information refers to the horizontal plane coordinate information corresponding to the point cloud data of the obstacle contact line obtained by calculating the point cloud coordinates, including the horizontal coordinate information and the vertical coordinate information corresponding to the point cloud data. The target obstacle distance refers to the distance between the moving carrier and the obstacle obtained by using the target horizontal plane coordinate information to calculate the distance.

Specifically, the motion carrier terminal scans the point cloud data of the obstacle contact line, and obtains vertical coordinate information corresponding to the point cloud data of the obstacle contact line when it detects that there is filled ground point cloud data in the point cloud data of the obstacle contact line. Then the moving carrier terminal obtains the coordinate information corresponding to the obstacle contact line, and the coordinate information corresponding to the obstacle contact line may be obtained when the moving carrier terminal detects the contact area between the image data and the ground using a preset deep learning network model. The motion carrier terminal obtains the pre-stored point cloud coordinate calculation formula, uses the vertical coordinate information corresponding to the obstacle contact line point cloud data and the coordinate information corresponding to the obstacle contact line to calculate according to the point cloud calculation formula, and obtains the obstacle contact line point cloud The horizontal plane coordinate information corresponding to the data. The motion carrier terminal takes the calculated horizontal plane coordinate information corresponding to the point cloud data of the obstacle contact line as the target horizontal plane coordinate information. Then the moving carrier terminal uses the target horizontal plane coordinate information to calculate the distance to obtain the target obstacle distance.

When the motion carrier terminal detects that there is no filled ground point cloud data in the obstacle contact line point cloud data, the horizontal plane coordinate information corresponding to the existing ground point cloud data in the obstacle contact line point cloud data is used as the target horizontal plane coordinate information , and use the target horizontal plane coordinate information to calculate the distance to obtain the target obstacle distance.

In a specific embodiment, the point cloud data acquisition device may be a lidar, the image acquisition device may be a monocular camera, and the internal and external parameter matrices may be obtained in advance through a calibration algorithm of the lidar and the monocular camera. The calculation formula of point cloud coordinates is obtained by deriving the internal and external parameter matrix, which is obtained by the calibration algorithm in advance. The calculation formula for projecting the three-dimensional coordinates in the camera coordinate system into the image through the internal parameter matrix is shown in formula (1):

Where fx, fy are the focal length of the camera, ox oy is the origin offset, Xc, Yc, Zc are the three-dimensional points in the camera coordinate system, and U, V are the two-dimensional coordinates of each point in the obstacle contact line.

The calculation formula for transforming the three-dimensional point in the lidar coordinate system into the camera coordinate system through the external parameter matrix is shown in formula (2):

Where M is the rotation matrix, tx, ty, tz are translation vectors, Xw, Yw, Zw are the three-dimensional points in the lidar coordinate system, and Xc, Yc, Zc are the three-dimensional points in the camera coordinate system.

Use formula (1) and formula (2) to derive the point cloud coordinate calculation formula, as shown in formula (3),

Xw=((M11-(V-oy)*(M21)/fy)*((U-ox)*(M22*Zw+tz)/fx-M02*Zw-tx)-(M01-(U-ox) )*(M21)/fx)*((V-oy)*(M22*Zw+tz)/fy-M12*Zw-ty))/((M11-(V-oy)*(M21)/fy) *(M00-(U-ox)*(M20)/fx)-(M01-(U-ox)*(M21)/fx)*(M10-(V-oy)*(M20)/fy));

Yw=((M10-(V-oy)*(M20)/fy)*((U-ox)*(M22*Zw+tz)/fx-M02*Zw-tx)-(M00-(U-ox) )*(M20)/fx)*((V-oy)*(M22*Zw+tz)/fy-M12*Zw-ty))/((M10-(V-oy)*(M20)/fy) *(M01-(U-ox)*(M21)/fx)-(M00-(U-ox)*(M20)/fx)*(M11-(V-oy)*(M21)/fy)) formula (3),

Among them, Xw represents the horizontal coordinate information corresponding to the point cloud data of the obstacle contact line; Yw represents the vertical coordinate information corresponding to the point cloud data of the obstacle contact line. The motion carrier obtains the corresponding vertical coordinate information Zw in the point cloud data of the obstacle contact line, and obtains the coordinate information (U, V) corresponding to the obstacle contact line, and uses Zw and (U, V) to calculate according to formula (3), Obtain the template horizontal plane coordinate information (Xw, Yw) corresponding to the point cloud number of the obstacle contact line, and then use (Xw, Yw) to calculate the target obstacle distance.

In this embodiment, when it is detected that there is filled ground point cloud data in the point cloud data of the obstacle contact line, the point cloud coordinate calculation formula is used to calculate the horizontal plane coordinate information corresponding to the point cloud data of the obstacle contact line, which can reduce the amount of filling. The error of ground point cloud data improves the accuracy of obstacle distance detection.

In one embodiment, step 204, generating a target ground point cloud area based on the ground point cloud data, including:

Divide the ground point cloud data based on the preset division specification to obtain the ground point cloud division area, and the ground point cloud division area includes the ground point cloud area and the area to be filled;

The filled ground point cloud data corresponding to the area to be filled is generated based on the regional ground point cloud data in the ground point cloud area, and the target ground point cloud area is generated based on the regional ground point cloud data and the filled ground point cloud data.

The preset division specification refers to a preset specification for dividing the ground point cloud data into regions. The ground point cloud division area refers to the division area obtained by dividing the ground point cloud data, and the ground point cloud division area includes the ground point cloud data. The ground point cloud area refers to the area where ground point cloud data exists in the ground point cloud division area. The area to be filled refers to the area that needs to be filled with ground point cloud data in the ground point cloud division area. Filled ground point cloud data refers to the ground point cloud data filled into the area to be filled. Regional ground point cloud data refers to the existing ground point cloud data in the ground point cloud area.

Specifically, the motion carrier terminal divides the ground point cloud data into grids of a fixed size according to the preset division specification, for example, divides the ground point cloud data into a 3*3 square grid according to the division specification of 3 meters. Then the motion carrier terminal searches for the grid with ground point cloud data, scans the distribution state of the ground point cloud data in the found grid, and takes the area where the ground point cloud data is distributed in the scanned grid as the ground point cloud The area in the scanned grid that is not distributed with ground point cloud data is used as the area to be filled.

Then, the motion carrier terminal performs estimation according to the ground point cloud data in the ground point cloud area, uses the estimated ground point cloud data as the filled ground point cloud data, and fills the filled ground point cloud data into the area to be filled. The motion carrier terminal can scan each grid with ground point cloud data again, and when detecting that there is no area to be filled, use each grid with ground point cloud data as the target ground point cloud area.

In a specific embodiment, as shown in FIG. 4, a schematic diagram of a ground point cloud division area is provided; frame A represents the divided ground point cloud area obtained by division; frame A-a represents the ground point cloud area; frame A-b represents the area to be filled .

In this embodiment, by dividing the ground point cloud data according to the preset division specification to obtain each ground point cloud division area, and filling the area to be filled in the ground point cloud division area with the point cloud data, a complete ground point cloud can be obtained. point cloud data, so as to ensure that there are three-dimensional coordinates corresponding to the obstacle contact line in the target ground point cloud area, avoid data loss when the target ground point cloud area is projected, and thus improve the accuracy of obstacle distance detection.

In one embodiment, as shown in FIG. 5 , a schematic flowchart of generating filled ground point cloud data is provided; based on the regional ground point cloud data in the ground point cloud area, the filled ground point cloud data corresponding to the area to be filled is generated, including :

Step 502, establishing a regional point cloud coordinate relationship based on the regional ground point cloud data;

Step 504: Obtain the regional horizontal plane coordinate information corresponding to the regional ground point cloud data, and calculate the filling horizontal plane coordinate information corresponding to the area to be filled based on the regional horizontal plane coordinate information and the preset horizontal plane coordinate interval information;

Step 506: Calculate according to the coordinate relationship of the area point cloud based on the coordinate information of the filling plane, and obtain the filling vertical coordinate information corresponding to the area to be filled;

Step 508: Generate filled ground point cloud data corresponding to the area to be filled based on the filled horizontal coordinate information and the filled vertical coordinate information.

Among them, the regional point cloud coordinate relationship refers to the objective physical relationship between the point clouds in the ground point cloud division area. Different ground point cloud division areas correspond to different regional point cloud coordinate relationships. The regional horizontal plane coordinate information refers to the horizontal plane coordinate information corresponding to the ground point cloud data in the ground point cloud region. Filling the ground point cloud data refers to the ground point cloud data estimated according to the regional ground point cloud data, and the filling ground point cloud data is used to fill the area to be filled. The preset horizontal plane coordinate interval information refers to a preset interval value between horizontal plane coordinates used for estimating and filling ground point cloud data. The filled horizontal plane coordinate information refers to the horizontal plane coordinate information corresponding to the filled ground point cloud data. Filling the vertical coordinate information refers to filling the vertical coordinate information corresponding to the ground point cloud data.

Specifically, the motion carrier terminal randomly acquires at least two regional ground point cloud data, establishes the regional point cloud coordinate relationship according to the at least two regional ground point cloud data, and the motion carrier terminal can use the plane equation to establish the regional point cloud coordinate relationship: Ax+By +Cz+D=0, A, B, C, and D represent the coefficients of the plane equation, and the coefficients of the plane equation can be estimated by using algorithms such as least squares method and RANSAC. Then the motion carrier terminal scans the regional ground point cloud data in the ground point cloud area, obtains the regional horizontal plane coordinate information corresponding to the regional ground point cloud data closest to the area to be filled, and then performs the regional horizontal plane coordinate information according to the preset horizontal plane coordinate interval information. Accumulate to obtain multiple filling horizontal plane coordinate information corresponding to the area to be filled. For example, the regional horizontal plane coordinate information corresponding to the regional ground point cloud data is (1, 1), and the preset horizontal plane coordinate interval information is 0.1, then fill the ground point cloud data. The corresponding coordinate information of multiple filling levels are (1.1, 1.1), (1.2, 1.2), (1.3, 1.3), etc., and can also be accumulated according to the coordinates, for example, (1.1, 1.1), (1.1, 1.2) , (1.2, 1.3), (1.3, 1.4), etc., or (1.1, 1.1), (1.2, 1.1), (1.3, 1.2), (1.4, 1.3), etc. The moving carrier terminal uses the estimated filling horizontal plane coordinate information to calculate according to the point cloud coordinate relationship to obtain the filling vertical coordinate information, and then the moving carrier terminal obtains the filling ground point cloud data corresponding to the area to be filled according to the filling horizontal plane coordinate information and the filling vertical coordinate information. , and mark the filled ground point cloud data for subsequent processing of the filled ground point cloud data.

In this embodiment, according to the existing ground point cloud data, the pre-set horizontal plane coordinate interval information and the regional point cloud coordinate relationship are estimated, the filled ground point cloud data can be accurately obtained, and the complete ground point cloud data can be obtained, so that the It is ensured that there are three-dimensional coordinates corresponding to the obstacle contact line in the target ground point cloud area, which avoids the situation of data missing when the target ground point cloud area is projected, thereby improving the accuracy of obstacle distance detection.

In one embodiment, step 208, determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area, including:

Based on the contact line coordinate information corresponding to the obstacle contact line, search for the projection coordinate information that is the same as the contact line coordinate information in the ground point cloud projection area, and obtain the target projection coordinate information;

The obstacle projection contact line is obtained based on the target projection coordinate information.

The contact line coordinate information refers to the two-dimensional coordinate information corresponding to the obstacle contact line in the collected image data. The projection coordinate information refers to the two-dimensional coordinate information in the image data corresponding to the ground point cloud projection area.

Specifically, the moving carrier terminal scans the projection coordinate information in the ground point cloud projection area according to the contact line coordinate information corresponding to the obstacle contact line, obtains the same projection coordinate information as the contact line coordinate information in the ground point cloud projection area, and sets the The obtained projection coordinate information is used as the target projection coordinate information. Then, the moving carrier terminal obtains the obstacle projected contact line according to the target projected coordinate information. Specifically, the number of two-dimensional points in the obstacle contact line is determined by the number of pixels corresponding to the image data, and the number of two-dimensional points in the obstacle projected contact line is determined by the number of two-dimensional points in the obstacle contact line. The number of points is determined. Further, the number of points of the three-dimensional points in the point cloud data of the obstacle contact line is determined by the number of points of the two-dimensional points in the projected contact line of the obstacle.

In a specific embodiment, as shown in FIG. 6, a schematic diagram of an obstacle contact line is provided; in the figure, A represents the obstacle in the collected image data; B represents the ground area in the collected image data; line L represents the contact line between the obstacle and the ground.

In this embodiment, by searching for the target projection coordinate information that is the same as the contact line coordinate information in the ground point cloud projection area, the obstacle projected contact line can be obtained, which can improve the coordinate accuracy of the obstacle projected contact line, thereby improving the accuracy of the obstacle projection contact line. The accuracy of distance detection.

In one embodiment, in step 210, distance calculation is performed based on the horizontal plane coordinate information in the point cloud data of the obstacle contact line to obtain the obstacle distance, including:

Obtain the horizontal plane coordinate information corresponding to each 3D point in the point cloud data of the obstacle contact line;

Based on the horizontal plane coordinate information corresponding to each 3D point, the distance is calculated respectively to obtain the point obstacle distance corresponding to each 3D point;

The shortest point obstacle distance among the point obstacle distances corresponding to each 3D point is taken as the obstacle distance.

Among them, the three-dimensional point represents the component unit that constitutes the point cloud data. The point obstacle distance is the distance between the motion carrier and the three-dimensional point in the obstacle contact line.

D表示障碍物距离。Specifically, the moving carrier terminal acquires the horizontal plane coordinate information corresponding to each three-dimensional point in the point cloud data of the contact line of the obstacle, including the horizontal coordinate information and the vertical coordinate information. Use the horizontal plane coordinate information corresponding to each three-dimensional point to calculate the distance, and obtain the distance of multiple points of obstacles. The motion carrier terminal can use the triangle hypotenuse calculation formula to calculate the point obstacle distance:

D represents the obstacle distance.

Then the motion carrier terminal compares the distances of each point obstacle, and takes the point obstacle distance with the shortest distance as the obstacle distance.

Then the motion carrier takes the front of the motion carrier as 0°, and the counterclockwise direction is the positive direction, and calculates the obstacle azimuth according to the horizontal coordinate information and vertical coordinate information corresponding to the distance of the obstacle, and obtains the position and azimuth of the obstacle relative to the motion carrier. The motion terminal carrier can use the angle calculation formula atan2(y,x) to calculate the position of the obstacle. For example, the calculated obstacle position is +45°, which means that the obstacle is in front of the left of the motion carrier, and the obstacle position is -45°, which means that the obstacle is in front of the right side of the motion carrier.

In this embodiment, by calculating the point obstacle distance of each three-dimensional point, and using the point obstacle distance with the shortest distance as the obstacle distance, the accuracy of the obstacle distance can be improved.

In a specific embodiment, as shown in FIG. 7 , a schematic flowchart of obstacle distance detection is provided; during the driving process of the autonomous vehicle, the on-board terminal obtains the original point cloud data in the 360-degree range of the current frame from the lidar, The 360-degree image data around the autonomous vehicle is obtained from 6 cameras, and the internal and external parameters of the lidar and camera are calibrated.

The vehicle terminal uses ground point classification algorithms such as deep learning and traditional geometric algorithms to find ground points in the original point cloud data, and obtain ground point cloud data. The scene is then divided into a grid of coarse resolution, such as 3 meters per grid, and the plane equation of the grid is calculated for the grid with ground point cloud data. Then determine the missing area of the ground point cloud data in the grid, that is, the area to be filled. The vehicle terminal estimates the missing ground point cloud data according to the nearest existing ground point cloud data in the missing area of the ground point cloud data. On the x and y planes, at a certain interval, such as 0.1, the filled horizontal coordinate information x and y are generated, and then the vehicle terminal calculates the filled vertical coordinate information according to the filled horizontal coordinate information x and y of the grid using the plane equation corresponding to the grid, and obtains Fill in the ground point cloud data, complete the ground point cloud estimation, and obtain the target ground point cloud area.

The on-board terminal uses YOLO, SSD, CenterNet and other target detection models to detect obstacles on the image data collected by the image acquisition device, including vehicles, pedestrians, bicycles, cones and other traffic participating obstacle targets, and outputs obstacles and ground through the detection model. contact line.

The vehicle-mounted terminal projects the target ground point cloud area to obtain the ground point cloud projection area, and then searches the ground point cloud projection area for the target projection coordinate information that is the same as the coordinate information of the obstacle contact line, and obtains the obstacle projected contact line. According to the projection relationship, the obstacle contact line point cloud data corresponding to the obstacle projected contact line is found in the target ground point cloud area, and the obstacle distance is calculated according to the horizontal plane coordinate information in the obstacle contact line point cloud data. The vehicle terminal can re-plan the driving route and driving speed according to the obstacle distance.

It should be understood that, although the steps in the flowcharts involved in the above embodiments are sequentially displayed according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but may be performed at different times The execution order of these steps or phases is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or phases in the other steps.

Based on the same inventive concept, an embodiment of the present application also provides an obstacle distance detection device for implementing the above-mentioned obstacle distance detection method. The implementation solution for solving the problem provided by the device is similar to the implementation solution described in the above method, so the specific limitations in the embodiments of one or more obstacle distance detection devices provided below can refer to the above for obstacle distance detection. The limitation of the method is not repeated here.

In one embodiment, as shown in FIG. 8, an obstacle distance detection device 800 is provided, including: an acquisition module 802, a ground detection module 804, a contact line detection module 806, a projection module 808 and a calculation module 810, wherein:

an acquisition module 802, configured to acquire original point cloud data and image data corresponding to the original point cloud data;

The ground detection module 804 is configured to perform ground point cloud detection based on the original point cloud data, obtain ground point cloud data, and generate a target ground point cloud area based on the ground point cloud data;

The contact line detection module 806 is used to detect the contact line between the obstacle and the ground based on the image data to obtain the obstacle contact line;

The projection module 808 is used for projecting based on the target ground point cloud area, obtaining the ground point cloud projection area, and determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area;

The calculation module 810 is used to determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud area, and perform distance calculation based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance. .

In one embodiment, the obstacle distance detection device 800 further includes:

The target obstacle distance calculation unit is used to obtain the vertical coordinate information corresponding to the point cloud data of the obstacle contact line and the coordinate information corresponding to the obstacle contact line; based on the vertical coordinate information and coordinate information, the point cloud coordinate calculation is performed to obtain the obstacle contact line The target horizontal plane coordinate information corresponding to the point cloud data; the distance calculation is performed based on the target horizontal plane coordinate information to obtain the target obstacle distance.

In one embodiment, the ground detection module 804 includes:

The area division unit is used to divide the ground point cloud data based on the preset division specifications to obtain the ground point cloud division area, and the ground point cloud division area includes the ground point cloud area and the area to be filled; based on the area ground in the ground point cloud area The point cloud data generates the filled ground point cloud data corresponding to the area to be filled, and the target ground point cloud area is generated based on the regional ground point cloud data and the filled ground point cloud data.

In one embodiment, the ground detection module 804 includes:

The filling point cloud unit is used to establish the regional point cloud coordinate relationship based on the regional ground point cloud data; obtain the regional horizontal plane coordinate information corresponding to the regional ground point cloud data, and calculate the corresponding area to be filled based on the regional horizontal plane coordinate information and the preset horizontal plane coordinate interval information. Filling horizontal plane coordinate information; based on the filling plane coordinate information, calculate according to the regional point cloud coordinate relationship to obtain the filling vertical coordinate information corresponding to the area to be filled; based on the filling horizontal plane coordinate information and filling vertical coordinate information, generate the filling ground point corresponding to the area to be filled cloud data.

In one embodiment, the projection module 808 includes:

The coordinate projection unit is used to search for the same projection coordinate information as the contact line coordinate information in the ground point cloud projection area based on the contact line coordinate information corresponding to the obstacle contact line, and obtain the target projection coordinate information; obtain the obstacle based on the target projection coordinate information Projected contact lines.

In one embodiment, the computing module 810 includes:

The distance calculation unit is used to obtain the horizontal plane coordinate information corresponding to each three-dimensional point in the point cloud data of the obstacle contact line; based on the horizontal plane coordinate information corresponding to each three-dimensional point, the distance calculation is performed respectively to obtain the point obstacle distance corresponding to each three-dimensional point; The shortest point obstacle distance among the point obstacle distances corresponding to each 3D point is taken as the obstacle distance.

Each module in the above-mentioned obstacle distance detection device can be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, and the computer device may be a server, and its internal structure diagram may be as shown in FIG. 9 . The computer device includes a processor, a memory, an input/output interface (Input/Output, I/O for short) and a communication interface. Wherein, the processor, the memory and the input/output interface are connected through the system bus, and the communication interface is connected to the system bus through the input/output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The nonvolatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing raw point cloud data and image data. The input/output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. The computer program implements an obstacle distance detection method when executed by the processor.

In one embodiment, a computer device is provided, and the computer device may be a terminal, and its internal structure diagram may be as shown in FIG. 10 . The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input device. Wherein, the processor, the memory and the input/output interface are connected through the system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The input/output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer equipment is used for wired or wireless communication with an external terminal, and the wireless communication can be realized by WIFI, mobile cellular network, NFC (Near Field Communication) or other technologies. The computer program implements an obstacle distance detection method when executed by a processor. The display unit of the computer equipment is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device, the display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a display screen The touch layer covered on the top may also be keys, trackballs or touchpads provided on the casing of the computer equipment, and may also be an external keyboard, touchpad or mouse.

Those skilled in the art can understand that the structures shown in FIGS. 9-10 are only block diagrams of partial structures related to the solution of the present application, and do not constitute a limitation on the computer equipment to which the solution of the present application is applied. A device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is also provided, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the foregoing method embodiments when the processor executes the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, implements the steps in the foregoing method embodiments.

In one embodiment, a computer program product is provided, including a computer program, which implements the steps in each of the foregoing method embodiments when the computer program is executed by a processor.

It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all It is the information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to a memory, a database or other media used in the various embodiments provided in this application may include at least one of a non-volatile memory and a volatile memory. Non-volatile memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetic variable memory (Magnetoresistive Random Memory) Access Memory, MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration and not limitation, the RAM may be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM). The database involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a blockchain-based distributed database, etc., but is not limited thereto. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., and are not limited to this.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the present application should be determined by the appended claims.

Claims (10)

1. an obstacle distance detection method, is characterized in that, described method comprises: obtaining original point cloud data and image data corresponding to the original point cloud data; Perform ground point cloud detection based on the original point cloud data, obtain ground point cloud data, and generate a target ground point cloud area based on the ground point cloud data; Detecting the contact line between the obstacle and the ground based on the image data to obtain the obstacle contact line; Perform projection based on the target ground point cloud area to obtain a ground point cloud projection area, and determine the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area; Determine the obstacle contact line point cloud data corresponding to the obstacle projected contact line in the target ground point cloud area, and perform distance calculation based on the horizontal plane coordinate information in the obstacle contact line point cloud data to obtain the obstacle distance . 2. The method according to claim 1, wherein the method further comprises: acquiring the vertical coordinate information corresponding to the point cloud data of the obstacle contact line and the coordinate information corresponding to the obstacle contact line; Perform point cloud coordinate calculation based on the vertical coordinate information and the coordinate information to obtain the target horizontal plane coordinate information corresponding to the obstacle contact line point cloud data; The distance calculation is performed based on the coordinate information of the target horizontal plane to obtain the target obstacle distance. 3. The method according to claim 1, wherein the generating a target ground point cloud area based on the ground point cloud data comprises: Divide the ground point cloud data based on a preset division specification to obtain a ground point cloud division area, where the ground point cloud division area includes a ground point cloud area and a to-be-filled area; The filled ground point cloud data corresponding to the to-be-filled area is generated based on the regional ground point cloud data in the ground point cloud area, and the target ground point is generated based on the regional ground point cloud data and the filled ground point cloud data cloud area. 4. The method according to claim 3, wherein generating the filled ground point cloud data corresponding to the to-be-filled area based on the regional ground point cloud data in the ground point cloud area comprises: Establish a regional point cloud coordinate relationship based on the regional ground point cloud data; Obtaining the regional horizontal plane coordinate information corresponding to the regional ground point cloud data, and calculating the filling horizontal plane coordinate information corresponding to the to-be-filled area based on the regional horizontal plane coordinate information and the preset horizontal plane coordinate interval information; Calculate based on the coordinate information of the filling plane according to the coordinate relationship of the point cloud of the area, and obtain the filling vertical coordinate information corresponding to the area to be filled; The filled ground point cloud data corresponding to the area to be filled is generated based on the filled horizontal coordinate information and the filled vertical coordinate information. 5 . The method according to claim 1 , wherein the determining the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area comprises: 5 . Based on the contact line coordinate information corresponding to the obstacle contact line, search for the same projection coordinate information as the contact line coordinate information in the ground point cloud projection area to obtain target projection coordinate information; The obstacle projected contact line is obtained based on the target projected coordinate information. 6. The method according to claim 1, wherein the distance calculation is performed based on the horizontal plane coordinate information in the point cloud data of the obstacle contact line to obtain the obstacle distance, comprising: Obtaining the horizontal plane coordinate information corresponding to each three-dimensional point in the point cloud data of the obstacle contact line; Based on the horizontal plane coordinate information corresponding to each of the three-dimensional points, distance calculation is performed to obtain the point obstacle distance corresponding to each of the three-dimensional points; The shortest point obstacle distance among the point obstacle distances corresponding to the three-dimensional points is used as the obstacle distance. 7. An obstacle distance detection device, wherein the device comprises: an acquisition module for acquiring original point cloud data and image data corresponding to the original point cloud data; a ground detection module, configured to perform ground point cloud detection based on the original point cloud data, obtain ground point cloud data, and generate a target ground point cloud area based on the ground point cloud data; a contact line detection module for detecting the contact line between the obstacle and the ground based on the image data to obtain the obstacle contact line; a projection module, configured to perform projection based on the target ground point cloud area to obtain a ground point cloud projection area, and determine the obstacle projection contact line corresponding to the obstacle contact line in the ground point cloud projection area; A computing module, configured to determine the obstacle contact line point cloud data corresponding to the obstacle projection contact line in the target ground point cloud area, and perform distance calculation based on the horizontal plane coordinate information in the obstacle contact line point cloud data , get the obstacle distance. 8. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the method according to any one of claims 1 to 6 when the processor executes the computer program. step. 9. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented. 10. A computer program product comprising a computer program, characterized in that the computer program implements the steps of the method according to any one of claims 1 to 6 when the computer program is executed by a processor.

Images