CN120014574A

CN120014574A - Method and device for correcting drivable area, storage medium, and electronic device

Info

Publication number: CN120014574A
Application number: CN202510046793.9A
Authority: CN
Inventors: 刘富钰; 张德泽; 吴锐; 樊冬雪; 袁聪
Original assignee: Chongqing Chang'an Technology Co ltd
Current assignee: Chongqing Chang'an Technology Co ltd
Priority date: 2025-01-13
Filing date: 2025-01-13
Publication date: 2025-05-16

Abstract

The present invention provides a method and device for correcting a drivable area, a storage medium, and an electronic device, wherein the method includes: obtaining an original image captured by a vehicle camera; calculating the initial boundary point information of the drivable area of the vehicle based on the original image, and calculating the position information of each target in the original image; correcting the initial boundary point information based on the position information to obtain the target boundary point information of the drivable area of the vehicle. Through the embodiments of the present invention, the technical problem of the inaccurate relative position relationship between the boundary points of the drivable area and the target frame generated in the related art is solved, and accurate drivable area information can provide a reliable bounded safety area for the autonomous driving vehicle at the data layer, effectively ensuring the safety of the autonomous driving vehicle.

Description

Correction method and device for drivable area, storage medium and electronic device

Technical Field

The invention relates to the technical field of automatic driving, in particular to a method and a device for correcting a driving area, a storage medium and an electronic device.

Background

In the related art, the development of intelligent driving technology is particularly important to the development of a perception method. The visual-based detection of the area available as an underlying safety support task module is of self-evident importance. When facing to unfamiliar environments, non-white list barriers and long-tail effects existing in various complex scenes, the information of the drivable region in the image can provide a bottom layer and reliable bounded safety region in the current environment for the automatic driving vehicle, so that the safety of the automatic driving vehicle is effectively ensured.

In the propulsion process of the related art and the scheme, the implementation manner of the autopilot travelable area is mainly divided into two kinds. The first scheme of 'multi-sensor fusion', it collects the peripheral information of the vehicle through equipment such as camera, milli-wave radar, laser, radar together, and the scheme can acquire more degree of depth's spatial information because of the joining of laser radar, and is more accurate to object's position, distance and size perception, and because laser radar is self-luminous and not influenced by ambient light. However, the hardware cost of the laser radar is not low, and the multi-sensor fusion has higher calculation power requirement on a calculation chip, so that the cost is not advantageous. The second 'pure vision' solution, which has the advantages of relatively lower cost, more similar to human driving, more abundant environmental information obtained by high-resolution and high-frame-rate imaging technology, is favored by automobile manufacturers or scheme providers, obtains the detection result of the drivable region of the automobile according to an image segmentation detection model, and inputs the detection result into a confidence level network to obtain the confidence level of the drivable region. The current 'pure vision' scheme can drive the area to get based on the image segmentation model, and the dynamic target (such as the targets of vehicles, pedestrians, awls and the like) is obtained by adding proper post-processing according to the image detection model. Therefore, the relative positional relationship between the currently existing drivable region and the target frame in the vehicle coordinate system is in violation of each other.

In view of the above problems in the related art, an efficient and accurate solution has not been found.

Disclosure of Invention

The invention provides a correction method and device of a drivable area, a storage medium and an electronic device, which are used for solving the technical problems in the related art.

According to one embodiment of the invention, a correction method of a drivable area is provided, which comprises the steps of obtaining an original image acquired by a vehicle camera, calculating initial boundary point information of the drivable area of the vehicle according to the original image, calculating position information of each target in the original image, and correcting the initial boundary point information according to the position information to obtain target boundary point information of the drivable area of the vehicle.

Optionally, calculating initial boundary point information of a drivable area of the vehicle according to the original image comprises dividing the original image into a plurality of divided images, wherein each divided image corresponds to one target, merging at least two divided images belonging to the same object category in the plurality of divided images into a connected domain to obtain a plurality of connected domains, screening out area connected domains belonging to the drivable area of the vehicle in the plurality of connected domains, scanning the area connected domains row by row or column by column, determining boundary coordinates of initial boundary points of the drivable area, and converting the boundary coordinates from a pixel coordinate system into a vehicle coordinate system taking vehicle coordinates as an origin.

Optionally, screening the region connected domain belonging to the drivable region of the vehicle from the plurality of connected domains includes screening a road connected domain and an air-ground connected domain from the plurality of connected domains, and determining the road connected domain and the air-ground connected domain as region connected domains belonging to the drivable region of the vehicle.

Optionally, calculating the position information of each target in the original image includes inputting the original image into a target detection model, outputting detection frames of all targets, estimating first coordinate information and first size information of the targets in a vehicle coordinate system according to camera parameters of the camera for each detection frame, correcting the first size information to obtain second size information, and calculating second coordinate information of each target in the vehicle coordinate system according to the camera parameters and the second size information.

Optionally, correcting the first size information to obtain second size information includes identifying a target category of the target, obtaining a reference size range corresponding to the target category, judging whether the first size information is in the reference size range, and correcting the first size information based on the reference size range to obtain second size information if the first size information is not in the reference size range.

Optionally, the initial boundary point information is corrected according to the position information to obtain target boundary point information of a drivable area of the vehicle, wherein the initial value comprises the number of boundary points and the area size, a boundary point confirmation line is generated according to the initial value and the position information, the boundary point confirmation line is a ray extending outwards from the area size by taking the vehicle as a starting point, and the initial boundary point information is corrected based on the boundary point confirmation line to obtain the target boundary point information of the drivable area of the vehicle.

Optionally, generating boundary point confirmation lines according to the initial values and the position information comprises determining center line point coordinates of first targets, wherein the first targets are targets with target sizes smaller than a preset threshold, the position information comprises center line point coordinates and target sizes, generating first number of first boundary point confirmation lines by taking an origin of a vehicle coordinate system as a starting point and extending the center line point coordinates of each first target to generate second number of second boundary point confirmation lines by taking the origin of the vehicle coordinate system as a starting point and extending a second number of rays with the same angle outwards, and generating second number of second boundary point confirmation lines, wherein the sum of the first number and the second number is the number of the boundary points.

Optionally, the correction of the initial boundary point information based on the boundary point confirmation line comprises initializing a first intersection point between the boundary point confirmation line and a frame where the area size is located as a first boundary point, generating an initial line segment by adopting any two adjacent initial boundary points in the initial boundary point information, generating a second boundary point by adopting a second intersection point between the initial line segment and any one boundary point confirmation line, generating a third boundary point according to the position information and the target category, and determining a target boundary point according to the first boundary point, the second boundary point and the third boundary point on each boundary point confirmation line.

Optionally, generating the third boundary point according to the position information and the target category comprises identifying the target category of the target, wherein the target category comprises a non-vehicle category and a vehicle category, determining a central line point as the third boundary point if the target category is the non-vehicle category, wherein the position information comprises the central line point, calculating four vertexes of the target according to a central point, a length, a width and a course angle of the target in a vehicle coordinate system if the target category is the vehicle category, respectively calculating third intersection points between the four vertexes and the boundary point confirmation line, and selecting a third intersection point closest to the vehicle as the third boundary point in the four third intersection points.

Optionally, determining the target boundary point according to the first boundary point, the second boundary point and the third boundary point on each boundary point confirmation line comprises calculating a first distance, a second distance and a third distance between the first boundary point, the second boundary point, the third boundary point and the vehicle on each boundary point confirmation line, judging whether the third distance is larger than the second distance and a fourth distance between the second boundary point and the third boundary point is smaller than a preset threshold value, determining the third boundary point as the target boundary point if the third distance is larger than the second distance and the fourth distance between the second boundary point and the third boundary point is smaller than the preset threshold value, and determining the third boundary point as the target boundary point if the third distance is smaller than or equal to the second distance or the fourth distance between the second boundary point and the third boundary point is larger than or equal to the preset threshold value.

According to another embodiment of the invention, a correction device for a drivable area is provided, which comprises an acquisition module, a calculation module and a correction module, wherein the acquisition module is used for acquiring an original image acquired by a vehicle camera, the calculation module is used for calculating initial boundary point information of the drivable area of the vehicle according to the original image and calculating position information of each target in the original image, and the correction module is used for correcting the initial boundary point information according to the position information to obtain target boundary point information of the drivable area of the vehicle.

The calculation module comprises a segmentation unit, a merging unit, a screening unit and a determining unit, wherein the segmentation unit is used for segmenting the original image into a plurality of segmentation graphs, each segmentation graph corresponds to one target, the merging unit is used for merging at least two segmentation graphs belonging to the same object category in the plurality of segmentation graphs into one connected domain to obtain a plurality of connected domains, the screening unit is used for screening out a region connected domain belonging to a drivable region of the vehicle in the plurality of connected domains, the determining unit is used for scanning the region connected domains row by row or column by column, determining boundary coordinates of an initial boundary point of the drivable region, and converting the boundary coordinates from a pixel coordinate system into a vehicle coordinate system taking a vehicle coordinate as an origin.

Optionally, the screening unit comprises a screening subunit for screening the road connected domain and the air-ground connected domain from the plurality of connected domains, and a determining subunit for determining the road connected domain and the air-ground connected domain as the region connected domain belonging to the drivable region of the vehicle.

Optionally, the calculation module comprises a processing unit, an estimation unit, a correction unit and a calculation unit, wherein the processing unit is used for inputting the original image into a target detection model and outputting detection frames of all targets, the estimation unit is used for estimating first coordinate information and first size information of the targets under a vehicle coordinate system according to camera parameters of the camera, the correction unit is used for correcting the first size information to obtain second size information, and the calculation unit is used for calculating second coordinate information of each target under the vehicle coordinate system according to the camera parameters and the second size information.

Optionally, the correction unit comprises an identification subunit, an acquisition subunit, a judgment subunit and a correction subunit, wherein the identification subunit is used for identifying the target category of the target, the acquisition subunit is used for acquiring the reference size range corresponding to the target category, the judgment subunit is used for judging whether the first size information is in the reference size range, and the correction subunit is used for correcting the first size information based on the reference size range to obtain the second size information if the first size information is not in the reference size range.

Optionally, the correction module comprises a determination unit for determining an initial value of the drivable area, wherein the initial value comprises the number of boundary points and the area size, a generation unit for generating a boundary point confirmation line according to the initial value and the position information, wherein the boundary point confirmation line is a ray which extends outwards from the vehicle at the area size as a starting point, and a correction unit for correcting the initial boundary point information based on the boundary point confirmation line to obtain target boundary point information of the drivable area of the vehicle.

The generating unit comprises a determining subunit, a first generating subunit and a second generating subunit, wherein the determining subunit is used for determining center line point coordinates of first targets, the first targets are targets with target sizes smaller than a preset threshold, the position information comprises center line point coordinates and target sizes, the first generating subunit is used for generating a first number of first boundary point confirming lines by taking an origin of a vehicle coordinate system as a starting point and extending towards the center line point coordinates of each first target, the second generating subunit is used for generating a second number of second boundary point confirming lines by taking the origin of the vehicle coordinate system as a starting point and extending outwards by a second number of rays with the same angle, and the sum of the first number and the second number is the number of the boundary points.

Optionally, the correction unit comprises a processing subunit, a first generation subunit, a second generation subunit and a third generation subunit, wherein the processing subunit is used for initializing a first intersection point between the boundary point confirming line and a frame where the area size is located as a first boundary point, the first generation subunit is used for generating an initial line segment by adopting any two adjacent initial boundary points in the initial boundary point information, the second generation subunit is used for generating a second boundary point by adopting a second intersection point between the initial line segment and any boundary point confirming line, the third generation subunit is used for generating a third boundary point according to the position information and the target category, and the determination subunit is used for determining a target boundary point according to the first boundary point, the second boundary point and the third boundary point on each boundary point confirming line.

Optionally, the third generating subunit is further configured to identify a target class of the target, where the target class includes a non-vehicle class and a vehicle class, determine a center line point as a third boundary point if the target class is a non-vehicle class, where the location information includes the center line point, calculate four vertices of the target according to a center point, a length, a width, and a heading angle of the target in a vehicle coordinate system if the target class is a vehicle class, respectively calculate third intersection points between the four vertices and the boundary point confirmation line, and select a third intersection point closest to the vehicle as a third boundary point from the four third intersection points.

Optionally, the determining subunit is further configured to, for each boundary point confirmation line, calculate the first boundary point, the second boundary point, the third boundary point, and a first distance, a second distance, and a third distance between the third boundary point and the vehicle on the line, determine whether the third distance is greater than the second distance and a fourth distance between the second boundary point and the third boundary point is less than a preset threshold, determine the third boundary point as a target boundary point if the third distance is greater than the second distance and the fourth distance between the second boundary point and the third boundary point is less than the preset threshold, and determine the minimum boundary point of the first distance, the second distance, and the third boundary point as the target boundary point if the third distance is less than or equal to the second distance or the fourth distance between the second boundary point and the third boundary point is greater than or equal to the preset threshold.

According to another aspect of the embodiments of the present application, there is also provided a storage medium including a stored program that performs the above steps when running.

According to another aspect of the embodiment of the application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus, where the memory is used to store a computer program, and the processor is used to execute the steps in the above method by running the program stored on the memory.

Embodiments of the present application also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the steps of the above method.

The invention has the beneficial effects that:

1. The method for correcting the drivable area based on the target detection is provided to solve the problem that the relative position relationship between the edge point of the drivable area and the target frame is inaccurate under a vehicle coordinate system;

2. The reliable bounded safety area can be provided for the automatic driving vehicle in the data layer through accurate information of the drivable area, and the safety of the automatic driving vehicle is effectively ensured.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

FIG. 1 is a block diagram of a hardware architecture of an automobile in accordance with an embodiment of the present invention;

FIG. 2 is a flow chart of a method of correcting a travelable region in accordance with an embodiment of the present invention;

FIG. 3 is a schematic diagram of generating a boundary point validation line in a vehicle coordinate system in accordance with an embodiment of the present invention;

FIG. 4 is a schematic diagram of the correction of target boundary point information for a travelable region in an embodiment of the present invention;

FIG. 5 is a schematic diagram of generating a third boundary point in an embodiment of the invention;

FIG. 6 is a schematic diagram of boundary point correction based on object detection in accordance with an embodiment of the present invention;

Fig. 7 is a block diagram showing the structure of a correction device for a travelable region according to an embodiment of the present invention.

Detailed Description

In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application. It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example 1

The method embodiment provided in the first embodiment of the present application may be performed in an automobile, a server, a processor, a controller, or a similar processing device. Taking an example of running on a car, fig. 1 is a block diagram of a hardware structure of a car according to an embodiment of the present application. As shown in fig. 1, the automobile may include one or more processors 102 (only one is shown in fig. 1) (the processor 102 may include, but is not limited to, a microprocessor MCU or a processing device such as a programmable logic device FPGA) and a memory 104 for storing data, and optionally, a transmission device 106 for communication functions and an input-output device 108. It will be appreciated by those skilled in the art that the configuration shown in fig. 1 is merely illustrative and is not intended to limit the configuration of the vehicle described above. For example, the automobile may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.

The memory 104 may be used to store a car program, for example, a software program of application software and a module, such as a car program corresponding to a method for correcting a drivable area of a car in an embodiment of the present invention, and the processor 102 executes the car program stored in the memory 104, thereby performing various functional applications and data processing, that is, implementing the above method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located with respect to the processor 102, which may be connected to the automobile via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of an automobile. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as a NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is configured to communicate with the internet wirelessly.

In this embodiment, a method for correcting a drivable area is provided, and fig. 2 is a flowchart of a method for correcting a drivable area according to an embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:

step S202, acquiring an original image acquired by a vehicle camera;

Optionally, after the original image is obtained, the multi-frame images collected by the cameras can be spliced and denoised.

Step S204, calculating initial boundary point information of a drivable area of the vehicle according to the original image, and calculating position information of each target in the original image;

The drivable region (freespace) of the vehicle is a safety region for the vehicle to run, for decision-making in the event of autonomous driving or active safety.

And step S206, correcting the initial boundary point information according to the position information to obtain target boundary point information of the drivable area of the vehicle.

Through the steps, the original image acquired by the vehicle camera is acquired; the initial boundary point information of the drivable area of the vehicle is calculated according to the original image, the position information of each target in the original image is calculated, the target boundary point information of the drivable area of the vehicle is obtained by correcting the initial boundary point information according to the position information, the position information of each target in the image is calculated, the initial boundary point information is corrected, a drivable area correction method based on target detection is realized, the technical problem that the relative position relation between the boundary point of the drivable area and a target frame generated in the related technology is inaccurate is solved, a reliable bounded safety area can be provided for the automatic driving vehicle in a data layer through the accurate drivable area information, and the safety of the automatic driving vehicle is effectively ensured.

In one implementation mode of the embodiment, calculating initial boundary point information of a drivable area of the vehicle according to the original image comprises the steps of segmenting the original image into a plurality of segmentation graphs, merging at least two segmentation graphs belonging to the same object category in the plurality of segmentation graphs into one connected domain to obtain a plurality of connected domains, screening out area connected domains belonging to the drivable area of the vehicle in the plurality of connected domains, scanning the area connected domains row by row or column by column, determining boundary coordinates of initial boundary points of the drivable area, and converting the boundary coordinates from a pixel coordinate system into a vehicle coordinate system taking vehicle coordinates as an origin.

The boundary coordinates of the initial boundary points of the drivable area are a set including boundary coordinates corresponding to a plurality of initial boundary points, and optionally, the boundary coordinates after conversion into the vehicle coordinate system may be stored by an initial boundary point list or an initial boundary point matrix.

In one example, screening out the region connected domain belonging to the drivable region of the vehicle from among the plurality of connected domains includes screening out a road connected domain and a space connected domain from among the plurality of connected domains, and determining the road connected domain and the space connected domain as region connected domains belonging to the drivable region of the vehicle.

The travelable region is identified from the original image, and is converted into a vehicle coordinate system, and is corrected later, including the processes of image segmentation, connected domain analysis, coordinate system conversion and the like. The explanation and illustration is herein:

image segmentation, the camera captures the real-time scene image and takes it as the original input. Then, a series of preprocessing operations such as data normalization, clipping and scaling are performed. And then using the deep learning model to carry out image segmentation. The image segmentation model can identify the boundaries of different object categories from the original image and output a label graph with the same size as the original image, wherein each pixel point corresponds to a specific category (such as a vehicle, a pedestrian, a two-wheel vehicle, a cone barrel, a static target, a background and the like).

And the connected domain analysis is used for identifying continuous areas belonging to the same category in the segmented image. At this stage, connected domains are screened, namely only those areas considered to be drivable (such as roads or empty areas) are reserved, and obstacles, static targets and the like are eliminated. And determining boundary points, namely progressively scanning the selected connected domain from bottom to top, and finding out edge pixels as the boundary points of freespace.

Coordinate system conversion, since segmentation and analysis are typically performed in image space, requires conversion of these boundary points from pixel coordinates (typically in pixels on a screen) to the vehicle coordinate system (in meters). The size information of the vehicle defines the actual physical distance of a pixel value mapped to the vehicle coordinate system according to the internal parameters and the external parameters of the camera. The final output is a list or matrix of freespace boundary points in the vehicle coordinate system.

In one implementation mode of the embodiment, calculating the position information of each target in the original image comprises inputting the original image into a target detection model, outputting detection frames of all targets, estimating first coordinate information and first size information of the targets in a vehicle coordinate system according to camera parameters of the camera for each detection frame, correcting the first size information to obtain second size information, and calculating second coordinate information of each target in the vehicle coordinate system according to the camera parameters and the second size information.

And 2D detection of the target is carried out according to the original image acquired by the camera, detection frame information of the target is output, and coordinates and attribute information of the target under a vehicle coordinate system are calculated.

The raw image captured by the camera is input into a target detection model which is trained to identify and locate specific objects (such as vehicles, pedestrians and cones) within the image, while providing the approximate location, size and corresponding confidence score of each detection frame. If the vehicle is a vehicle, the whole frame, the tail frame and the wheel frame are detected at the same time. The pedestrian or the cone can only output one pedestrian detection frame or one cone detection frame.

Optionally, after detecting the detection frame in the original image of the current frame, the history detection frame in the original image of the history frame may be further acquired to perform target tracking, and the authenticity of the detection frame in the original image of the current frame may be determined by combining the history detection frame in the history frame, and the target that does not appear in the current frame and appears in the history frame may be further added to the target output by the current frame. During target tracking, non-maximum suppression (Non-Maximum Suppression, NMS) and Hungary algorithm are used for matching target detection frames between two frames, and the process is based on the cross ratio (Intersection over Union, ioU) and is used for judging whether the detection frame in the current frame corresponds to a certain object in the previous frame. And then combining featuremap (feature map) vectors of the detection frames output by the image detection model, and tracking the detection frames of the front and rear two frames of images according to the cosine similarity of the featuremap vectors.

In one example, correcting the first size information to obtain second size information includes identifying a target category of the target, obtaining a reference size range corresponding to the target category, judging whether the first size information is in the reference size range, and correcting the first size information based on the reference size range to obtain second size information if the first size information is not in the reference size range.

Optionally, the target classes include vehicles, travelling vehicles, traffic cones, stones, etc., and the vehicles can be further classified into cars, trucks, mechanical engineering vehicles, vans, minivans, tricycles, electric vehicles, bicycles, motorcycles, etc.

Optionally, a reference size range of each target category is preset, for example, a reference size range of a car length and width is 4500 x 1800 x 1600 (mm), and if the estimated first size information bit 6500 x 1600 x 1500 (mm) of the car 1 in the target frame detected in the original image is beyond the normal reference size range, there is visual distortion, and the car needs to be corrected to have the reference size range as the maximum allowable size, so that the length of the car 1 is corrected to be within 4500 mm.

In calculating the first coordinate information and the first size information, first, the three-dimensional position (first coordinate information) of the object in the vehicle coordinate system is estimated by combining the position information of the object detection frame with the camera internal and external reference information of the vehicle camera. And secondly, estimating the first size information of the length, the width, the height and the like of the target according to the small hole imaging principle (pinhole camera model) of the camera. The detected size information is then adjusted according to the target type. If the target type is a vehicle, the course angle of the target can be calculated according to the whole frame, the tail frame or the wheel frame.

In the process of correcting the first size information, the accurate position (second coordinate information) of the target under the vehicle coordinate system is reversely solved through the principle of aperture imaging by utilizing the relation between the width and the height of the target in the image and the camera internal reference matrix and combining known information such as focal length. The final output may include positional information (e.g., center point position, centerline position, centroid position, etc.) for each detected target object, and may additionally provide attributes such as length, width, and heading angle for the vehicle class target.

In one implementation mode of the embodiment, the method for correcting the initial boundary point information according to the position information to obtain the target boundary point information of the drivable area of the vehicle comprises the steps of determining an initial value of the drivable area, wherein the initial value comprises the number of boundary points and the area size, generating a boundary point confirming line according to the initial value and the position information, wherein the boundary point confirming line is a ray extending outwards from the vehicle at the area size, and correcting the initial boundary point information based on the boundary point confirming line to obtain the target boundary point information of the drivable area of the vehicle.

And outputting the corrected target boundary point information according to the generated initial boundary point information and the generated position information such as the target frame, the target attribute and the like.

In one example, when setting the initial value, the number of output target boundary points is set to Fnum (e.g., 256 points), and the maximum Area enclosed by the target boundary points in the vehicle coordinate system is set (e.g., the maximum lateral distance is 50m, and the maximum longitudinal distance is 500 m).

In one example, generating boundary point confirmation lines according to the initial values and the position information comprises determining center line point coordinates of first targets, wherein the first targets are targets with target sizes smaller than a preset threshold, the position information comprises center line point coordinates and target sizes, generating first number of first boundary point confirmation lines by taking an origin of a vehicle coordinate system as a starting point and extending the center line point coordinates of each first target to generate second number of second boundary point confirmation lines by taking the origin of the vehicle coordinate system as a starting point and extending a second number of rays with the same angle outwards, and the sum of the first number and the second number is the number of the boundary points.

Fig. 3 is a schematic diagram of generating a boundary point confirmation line in a vehicle coordinate system according to an embodiment of the present invention, wherein the vehicle coordinate system uses the center of the vehicle as an origin, the longitudinal extension direction is the x-axis direction, and the transverse extension direction is the y-axis direction, and since the projections of the pedestrian and the cone under the vehicle coordinate system are smaller, the boundary point confirmation line is generated freespace according to the center point positions of the pedestrian and the cone (see the first boundary point confirmation line dedicated to the pedestrian/cone indicated by the thick dotted line in fig. 3) first. Assuming the coordinates of the centerline points of the pedestrian or cone are (x ₀,y₀), the pedestrian/cone exclusive freespace confirms the line equation as:

If there are Pnum cones and pedestrians, pnum pedestrian/cone exclusive freespace validation lines are generated.

Next, according to the number of points at the output freespace being Fnum, the maximum Area is defaulted freespace in the vehicle coordinate system, and the remaining second boundary point check line is generated. The default maximum Area of freespace under the vehicle coordinate system is equally divided into rays (Fnum-Pnum) of angles (see fig. 3, which shows the second boundary point confirm line by a gray thin dashed line) with the origin of the vehicle coordinate system as the center. The second boundary point validation line equation is y=tan (n×α) ×x;

Wherein,

In one example, correcting the initial boundary point information based on the boundary point confirmation line comprises initializing a first intersection point between the boundary point confirmation line and a frame where the area size is located as a first boundary point, generating an initial line segment by adopting any two adjacent initial boundary points in the initial boundary point information, generating a second boundary point by adopting a second intersection point between the initial line segment and any boundary point confirmation line, generating a third boundary point according to the position information and the target category, and determining a target boundary point according to the first boundary point, the second boundary point and the third boundary point on each boundary point confirmation line.

In this example, the first boundary point is freespace, the second boundary point is freespace, and the third boundary point is freespace, which are taken as an example, each type of boundary point is a set, and its position information in the vehicle coordinate system is stored through a list or a matrix.

Fig. 4 is a schematic diagram of the correction of the target boundary point information of the drivable area according to the embodiment of the present invention, first, an array with dimension Fnum is generated. And initializing to obtain freespace points (freespace 0) according to the intersection point formed by the generated boundary point confirmation line and the default maximum Area of freespace under the set vehicle coordinate system. Next, a new freespace1 is generated from the intersection point between the line segment formed by two adjacent points in the generated initial boundary point and the boundary point confirmation line.

In this example, a new freespace (third boundary point) is generated from the generated target frame center line point of each target and the intersection point with the boundary point confirmation line, which is the closest to the host vehicle. Fig. 5 is a schematic diagram of generating a third boundary point in the embodiment of the present invention, a is a heading angle of a vehicle, W is a width of the vehicle, L is a length of the vehicle, P0 is a center point of the vehicle (which is also an origin of a coordinate system of the vehicle), P1-P4 are four vertexes (corner points) of the vehicle, and F2 is an intersection point of a boundary point confirmation line closest to the vehicle, that is, the generated third boundary point.

In one implementation scenario, the final output freespace points are validated from three points freespace (x ₀,y₀),freespace1(x₁,y₁) and freespace2 (x ₂,y₂) on each freespace validation line. The specific calculation mode is as follows:

Definition R is the distance from any freespace points to the host vehicle, namely freespace, freespace1 and freespace are respectively defined as R ₀、R₁、R₂. If R ₂ > R1 and the pixel coordinates of freespace and freespace points 1 are similar (less than the set threshold), freespace is determined as the final output of freespace points. Otherwise, according to three points freespace (x ₀,y₀),freespace1(x₁,y₁) and freespace (x ₂,y₂) on each boundary point confirm line (including the first boundary point confirm line and the second boundary point confirm line), the nearest point to the host vehicle is selected as freespace points finally output on the freespace confirm line.

A method for correcting a travelable region based on target detection is provided, so as to solve the problem that the relative position relationship between the edge point of the travelable region and a target box frame is inaccurate under a vehicle coordinate system. FIG. 6 is a schematic diagram of boundary point correction based on object detection according to an embodiment of the present invention, including the following steps:

Step 1-calculation of the travelable region:

And calculating boundary point information of a travelable area (freespace) according to the original image information of the camera and the image segmentation (parsing) information, and sending the boundary point information to a position correction module of the 3-freespace points. It mainly comprises the following contents:

First, an image acquired by a camera is input to an image segmentation model, which detects and outputs an image segmentation map, i.e., a type (e.g., type of vehicle, pedestrian, cone, static object, background, etc.) representing each pixel in the image.

Next, connected domains belonging to freespace are screened out from all connected domains of the image segmentation map (parsing map), and then the connected domains are scanned freespace line by line from bottom to top, so as to determine freespace boundary points and boundary point types.

Finally, freespace boundary points are converted from a pixel coordinate system (the unit is a pixel value) to a vehicle coordinate system (the unit is m), and the boundary points are output to the position correction module of the 3-freespace points.

Step2, calculating target position and attribute information:

According to the original image information of the camera and the 2D detection frame information of the target, calculating the position of the target under the vehicle coordinate system and the attribute information thereof, and sending the position and the attribute information to a 3-freespace point position correction module. It mainly comprises the following contents:

Firstly, inputting an image acquired by a camera into an image detection frame model, wherein the model detects a target 2D detection frame, a target type and a confidence level. The target types include vehicles, pedestrians, or cones. If the vehicle is a vehicle, the whole frame, the tail frame and the wheel frame are detected at the same time. The pedestrian or the cone can only have one detection frame.

And secondly, performing IOU matching according to the target detection frame of the previous frame and the current frame detection frame, and performing detection frame tracking of the front frame image and the rear frame image by combining featuremap vectors of the detection frames output by the image detection model.

And then, calculating the external parameter ranging of the target frame under the vehicle coordinate system according to the external parameter ranging principle. And estimating the length, width and height of the target according to the small hole imaging. And correcting the length, width and height of the target according to the type of the target. If the vehicle is the vehicle, the course angle of the vehicle can be calculated according to the whole frame, the tail frame or the wheel frame.

And finally, calculating the position of the target under the vehicle coordinate system according to the width and height of the target and the internal reference distance measurement principle of the camera. The module outputs the location of the center point of the target in the vehicle coordinate system. The vehicle target also outputs the length, width, and heading angle of the target.

Step 3-freespace, correcting the position of the point:

And outputting corrected FREE SPACE point calculation according to the freespace points generated in the step 1 and the target frames and the target attributes generated in the step 2. The final optimized effect is shown in fig. 3, and the invention successfully solves the technical problem of inaccurate relative position relationship between the edge point of the drivable area and the target box under the vehicle coordinate system.

From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.

Example 2

In this embodiment, a device for correcting a drivable area is further provided, and the device is used to implement the foregoing embodiments and preferred embodiments, and is not described again. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

Fig. 7 is a block diagram showing a structure of a correction device for a travelable region according to an embodiment of the present invention, as shown in fig. 7, the device comprising:

an acquisition module 70, configured to acquire an original image acquired by a vehicle camera;

A calculation module 72 for calculating initial boundary point information of a drivable area of the vehicle from the original image, and calculating position information of each target in the original image;

And a correction module 74, configured to correct the initial boundary point information according to the position information, so as to obtain target boundary point information of a drivable area of the vehicle.

It should be noted that each of the above modules may be implemented by software or hardware, and the latter may be implemented by, but not limited to, the above modules all being located in the same processor, or each of the above modules being located in different processors in any combination.

Example 3

An embodiment of the invention also provides a storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.

Alternatively, in the present embodiment, the above-described storage medium may be configured to store a computer program for performing the steps of:

s1, acquiring an original image acquired by a vehicle camera;

s2, calculating initial boundary point information of a drivable area of the vehicle according to the original image, and calculating position information of each target in the original image;

And S3, correcting the initial boundary point information according to the position information to obtain target boundary point information of the drivable area of the vehicle.

Alternatively, in the present embodiment, the storage medium may include, but is not limited to, a USB flash disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, etc. various media in which a computer program may be stored.

An embodiment of the invention also provides an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.

Optionally, the electronic apparatus may further include a transmission device and an input/output device, where the transmission device is connected to the processor, and the input/output device is connected to the processor.

Alternatively, in the present embodiment, the above-described processor may be configured to execute the following steps by a computer program:

s1, acquiring an original image acquired by a vehicle camera;

Alternatively, specific examples in this embodiment may refer to examples described in the foregoing embodiments and optional implementations, and this embodiment is not described herein.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

From the above description of embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus a general purpose hardware platform, or may be implemented by hardware. Based on such understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the related art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the method described in the respective embodiments or some parts of the embodiments.

It is to be understood that the terminology used herein is for the purpose of describing particular example embodiments only, and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," "including," and "having" are inclusive and therefore specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order described or illustrated, unless an order of performance is explicitly stated. It should also be appreciated that additional or alternative steps may be used.

The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for correcting a drivable area, comprising:

acquiring an original image acquired by a vehicle camera;

calculating initial boundary point information of a drivable area of the vehicle according to the original image, and calculating position information of each target in the original image;

And correcting the initial boundary point information according to the position information to obtain target boundary point information of the drivable area of the vehicle.

2. The method of claim 1, wherein calculating initial boundary point information of a drivable area of the vehicle from the raw image comprises:

Dividing the original image into a plurality of divided images, wherein each divided image corresponds to a target;

merging at least two segmentation graphs belonging to the same object class in the plurality of segmentation graphs into one connected domain to obtain a plurality of connected domains;

screening out a region connected domain belonging to a drivable region of the vehicle from among the plurality of connected domains;

And scanning the region connected domain row by row or column by column, determining boundary coordinates of an initial boundary point of the drivable region, and converting the boundary coordinates from a pixel coordinate system into a vehicle coordinate system taking the vehicle coordinate as an origin.

3. The method according to claim 2, wherein screening out the region connected domain belonging to the drivable region of the vehicle among the plurality of connected domains includes:

screening out a road communicating domain and an air-ground communicating domain from the plurality of communicating domains;

the road communicating region and the air-ground communicating region are determined as region communicating regions belonging to a drivable region of the vehicle.

4. The method of claim 1, wherein calculating the location information for each object in the original image comprises:

inputting the original image into a target detection model, and outputting detection frames of all targets;

estimating first coordinate information and first size information of a target in a vehicle coordinate system according to camera parameters of the camera for each detection frame;

Correcting the first size information to obtain second size information;

and calculating second coordinate information of each target in the vehicle coordinate system according to the camera parameters and the second size information.

5. The method of claim 4, wherein modifying the first size information to obtain second size information comprises:

Identifying a target category of the target;

acquiring a reference size range corresponding to the target category;

Judging whether the first size information is in the reference size range or not;

and if the first size information is not in the reference size range, correcting the first size information based on the reference size range to obtain second size information.

6. The method according to claim 1, wherein correcting the initial boundary point information based on the position information to obtain target boundary point information of a drivable region of the vehicle includes:

determining an initial value of the drivable area, wherein the initial value comprises the number of boundary points and the area size;

Generating a boundary point confirmation line according to the initial value and the position information, wherein the boundary point confirmation line is a ray which extends outwards in the area size by taking the vehicle as a starting point;

And correcting the initial boundary point information based on the boundary point confirmation line to obtain target boundary point information of the drivable area of the vehicle.

7. The method of claim 6, wherein generating a boundary point validation line from the initial value and the location information comprises:

determining a center line point coordinate of a first target, wherein the first target is a target with a target size smaller than a preset threshold value, and the position information comprises the center line point coordinate and the target size;

Extending the origin of a vehicle coordinate system to the center line point coordinates of each first target to generate a first number of first boundary point confirmation lines;

and taking the origin of the vehicle coordinate system as a starting point, extending a second number of rays with the same angle outwards, and generating a second number of second boundary point confirmation lines, wherein the sum of the first number and the second number is the number of the boundary points.

8. The method of claim 6, wherein modifying the initial boundary point information based on the boundary point validation line comprises:

initializing a first intersection point between the boundary point confirmation line and a frame in which the region size is located as a first boundary point;

generating an initial line segment by adopting any two adjacent initial boundary points in the initial boundary point information;

Generating a second boundary point by adopting a second intersection point between the initial line segment and any boundary point confirmation line;

Generating a third boundary point according to the position information and the target category;

and determining a target boundary point according to the first boundary point, the second boundary point and the third boundary point on each boundary point confirmation line.

9. The method of claim 8, wherein generating a third boundary point from the location information and a target category comprises:

Identifying a target class of the target, wherein the target class comprises a non-vehicle class and a vehicle class;

If the target class is a vehicle class, calculating four vertexes of the target according to a central point, a length, a width and a course angle of the target under a vehicle coordinate system, respectively calculating third intersection points between the four vertexes and the boundary point confirmation line, and selecting a third intersection point closest to the vehicle as the third boundary point in the four third intersection points.

10. The method of claim 8, wherein determining a target boundary point from the first, second, and third boundary points on each boundary point validation line comprises:

calculating a first distance, a second distance, and a third distance between the first boundary point, the second boundary point, the third boundary point, and the vehicle on each boundary point confirmation line, respectively;

Judging whether the third distance is larger than the second distance and whether a fourth distance between the second boundary point and the third boundary point is smaller than a preset threshold value;

And if the third distance is smaller than or equal to the second distance or the fourth distance between the second boundary point and the third boundary point is larger than or equal to a preset threshold value, determining the smallest boundary point among the first distance, the second distance and the third distance as a target boundary point.

11. A correction device for a drivable area, comprising:

The acquisition module is used for acquiring an original image acquired by the vehicle camera;

A calculation module for calculating initial boundary point information of a drivable area of the vehicle according to the original image, and calculating position information of each target in the original image;

And the correction module is used for correcting the initial boundary point information according to the position information to obtain target boundary point information of the drivable area of the vehicle.

12. The apparatus of claim 11, wherein the correction module comprises:

A determining unit configured to determine an initial value of the drivable area, wherein the initial value includes a boundary point number and an area size;

A generating unit configured to generate a boundary point confirmation line according to the initial value and the position information, where the boundary point confirmation line is a ray extending outward from the host vehicle as a starting point in the area size;

And the correction unit is used for correcting the initial boundary point information based on the boundary point confirmation line to obtain target boundary point information of the vehicle drivable area.

13. A storage medium having a computer program stored therein, wherein the computer program is arranged to perform the method of any of claims 1 to 10 when run.

14. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to run the computer program to perform the method of any of the claims 1 to 10.