CN106503653B - Region labeling method and device and electronic equipment - Google Patents

Abstract

A region labeling method and device and electronic equipment are disclosed. The method comprises the following steps: acquiring image information of a driving environment acquired by an imaging device in the process of generating a training sample for training a machine learning model; acquiring depth information of the driving environment synchronized in time with the image information; and marking an obstacle region in the driving environment in the image information according to the depth information. Therefore, the obstacle area in the driving environment can be automatically marked, and the area marking efficiency is improved.

Description

Area marking method, device and electronic equipment

technical field

The present application relates to the field of assisted driving, and more particularly, to an area marking method, apparatus, electronic device, computer program product, and computer-readable storage medium.

Background technique

In recent years, with the rapid development of the transportation industry (for example, vehicles), traffic accidents have become a global problem. The number of casualties in traffic accidents worldwide is estimated to exceed 500,000 each year. Therefore, automatic control, artificial intelligence, pattern recognition, etc. The technology-integrated assisted driving technology emerges as the times require. Assisted driving technology can provide the user with necessary information and/or warnings while driving the vehicle to avoid dangerous situations such as collision and deviation from the track. In some cases, assisted driving technology may even be used to automatically control vehicle travel.

Driving area detection has always been one of the key parts of assisted driving technology. At present, the most commonly used detection methods are based on machine learning models. In order to ensure the accuracy of the machine learning model, it is necessary to use a large amount of image information of the driving environment as training samples in advance to conduct offline training of the model. Since there are often various obstacles such as vehicles and pedestrians in the driving environment, these obstacle areas need to be marked in the training samples before offline training, so as to reserve the drivable area for vehicles to travel. At present, the labeling of obstacle areas in training samples mainly depends on the user's manual completion, that is, the user needs to manually find various obstacle individuals in a large amount of image information, and label the size and location of each individual. Since the training sample database generally needs to reach the scale of hundreds of thousands, this manual labeling method is very time-consuming, the labor cost is very high, and it is not scalable.

Therefore, existing region labeling techniques are inefficient.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the present application is made. Embodiments of the present application provide an area marking method, apparatus, electronic device, computer program product, and computer-readable storage medium, which can automatically mark obstacle areas in a driving environment.

According to an aspect of the present application, there is provided an area labeling method, comprising: in the process of generating a training sample for training a machine learning model, acquiring image information of a driving environment collected by an imaging device; acquiring image information related to the image information time-synchronized depth information of the driving environment; and marking an obstacle area in the driving environment in the image information according to the depth information.

According to another aspect of the present application, an area labeling device is provided, including: an image acquisition unit configured to acquire image information of a driving environment collected by an imaging device during a process of generating a training sample for training a machine learning model ; a depth acquisition unit for acquiring depth information of the driving environment synchronized with the image information in time; and an obstacle marking unit for marking the driving environment in the image information according to the depth information obstacle area.

According to another aspect of the present application, there is provided an electronic device comprising: a processor; a memory; and computer program instructions stored in the memory, the computer program instructions, when executed by the processor, cause the The processor executes the region labeling method described above.

According to another aspect of the present application, a computer program product is provided, comprising computer program instructions, the computer program instructions, when executed by a processor, cause the processor to execute the above-mentioned area marking method.

According to another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored, the computer program instructions, when executed by a processor, cause the processor to execute the above-mentioned area marking method.

Compared with the prior art, by using the region labeling method, device, electronic device, computer program product and computer-readable storage medium according to the embodiments of the present application, in the process of generating the training sample for training the machine learning model, the Image information of the driving environment collected by the imaging device, obtain depth information of the driving environment that is synchronized with the image information in time, and mark obstacles in the driving environment in the image information according to the depth information object area. Therefore, compared with the situation of manually labeling the obstacle area as in the prior art, the obstacle area in the driving environment can be automatically labeled, which improves the efficiency of area labeling.

Description of drawings

The above and other objects, features and advantages of the present application will become more apparent from the detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present application, constitute a part of the specification, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation to the present application. In the drawings, the same reference numbers generally refer to the same components or steps.

FIG. 1 illustrates a schematic diagram of image information of a driving environment collected by an imaging device according to an embodiment of the present application.

FIG. 2 illustrates a flowchart of a region labeling method according to the first embodiment of the present application.

FIG. 3 illustrates a flowchart of a step of acquiring depth information according to an embodiment of the present application.

FIG. 4 illustrates a flowchart of a step of marking an obstacle according to an embodiment of the present application.

FIG. 5 illustrates a flowchart of a region labeling method according to the second embodiment of the present application.

FIG. 6 illustrates a flowchart of the steps of marking a drivable area according to an embodiment of the present application.

7A illustrates a schematic diagram of combining depth information and user input in the image information shown in FIG. 1 according to an embodiment of the present application, and FIG. 7B illustrates a schematic diagram of the image information shown in FIG. 1 according to an embodiment of the present application Schematic annotated with obstacle areas and drivable areas.

FIG. 8 illustrates a block diagram of a region labeling apparatus according to an embodiment of the present application.

FIG. 9 illustrates a block diagram of an electronic device according to an embodiment of the present application.

Detailed ways

Hereinafter, exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.

Application overview

As mentioned above, in the prior art, the labeling of obstacle regions in training samples mainly relies on manual completion by the user, so there are problems of complicated operation and low efficiency.

In view of this technical problem, the basic idea of this application is to propose a new area labeling method, apparatus, electronic device, computer program product and computer-readable storage medium, which can combine depth information from a depth sensor during the labeling process. , the obstacle area is automatically marked in the image information collected by the imaging device, without the need for manual operation by the user, thereby reducing the cost of marking and improving the speed of marking.

The embodiments of the present application may be applied to various scenarios. For example, the embodiments of the present application can be used to mark the obstacle area in the driving environment where the vehicle is located. For example, the vehicle may be of a different type, which may be a vehicle, aircraft, spacecraft, underwater vehicle, or the like. For the convenience of explanation, the description will be continued by taking a vehicle as an example of a vehicle.

For example, in order to enable the vehicle to determine various obstacles on the road surface as its driving environment during actual driving, and then achieve the purpose of assisting driving, etc., it is necessary to use a large amount of image information of the driving environment as training samples in advance. Machine learning models are trained offline. For this purpose, the test vehicle can be equipped in advance with one or more imaging devices for collecting a large amount of image information about the different driving environments. Of course, the present application is not limited to this. For example, the image information may also come from a surveillance camera set at a fixed position, or directly from the Internet, and so on.

FIG. 1 illustrates a schematic diagram of image information of a driving environment collected by an imaging device according to an embodiment of the present application.

As shown in FIG. 1 , image information obtained by a test vehicle equipped with an imaging device indicates that the vehicle is driving on a road surface as its typical driving environment. There are 3 obstacles (obstruction 1, obstacle 2, and obstacle 3 as other vehicles located at different distances), 4 lane lines (lane line 1, lane line 2 from left to right) on the road surface , lane line 3, and lane line 4), and a boundary line (the boundary line 1 between the road and the grass) and other objects.

The existing obstacle area labeling methods usually require users to find various obstacle individuals in the image information based on human eye recognition, and use the mouse to circle the way to label the size and location of each individual. In general, this obstacle labeling method is simple and effective. However, the sample library used for offline training of machine learning models often includes a large amount of image information. If each image needs to be distinguished and manually identified by the user's eyes, it will be time-consuming and labor-intensive, and there may be leakage due to manual operation. Therefore, the existing obstacle area labeling may not be accurate enough, resulting in errors in subsequent machine learning results, which may lead to wrong judgment of the actual road conditions when the vehicle is used online, resulting in traffic safety hazards. .

To this end, in the embodiments of the present application, in the process of generating the training samples for training the machine learning model, the image information of the driving environment collected by the imaging device is acquired, and all the images that are synchronized in time with the image information are acquired. depth information of the driving environment, and an obstacle area in the driving environment is marked in the image information according to the depth information. Therefore, the embodiments of the present application according to the basic concept can automatically mark the obstacle area in the driving environment, which improves the efficiency of area marking.

Of course, although the embodiments of the present application are described above by taking a vehicle as an example, the present application is not limited thereto. The embodiments of the present application can be applied to, for example, marking obstacle areas in a driving environment where various online electronic devices such as mobile robots and fixed surveillance cameras are located.

Hereinafter, various embodiments according to the present application will be described with reference to the accompanying drawings in conjunction with the application scenario of FIG. 1 .

Exemplary method

FIG. 2 illustrates a flowchart of a region labeling method according to the first embodiment of the present application.

As shown in FIG. 2, the area labeling method according to the first embodiment of the present application may include:

In step S110, in the process of generating training samples for training the machine learning model, image information of the driving environment collected by the imaging device is acquired.

In order to train the machine learning model offline, it is necessary to mark the image information of the driving environment as the training sample in advance, and find the obstacle area in it. For example, a large amount of image information of the driving environment can be collected by one or more imaging devices. For example, in an application scenario where an imaging device is equipped on a test vehicle (or called the current vehicle), the image information of the road surface in the current driving direction of the vehicle can be obtained through the imaging device, for example, as shown in FIG. 1 .

For example, the imaging device may be an image sensor for capturing image information, which may be a camera or an array of cameras. For example, the image information collected by the image sensor may be a sequence of continuous image frames (ie, a video stream) or a sequence of discrete image frames (ie, a set of image data sampled at a predetermined sampling time point) and the like. For example, the camera can be a monocular camera, a binocular camera, a multi-eye camera, etc. In addition, it can be used to capture a grayscale image or a color image with color information. Of course, any other types of cameras known in the art and that may appear in the future can be applied to the present application, and the present application does not specifically limit the way of capturing images, as long as the grayscale or color information of the input image can be obtained . In order to reduce the amount of computation in subsequent operations, in one embodiment, the color image may be grayscaled before analysis and processing.

In step S120, depth information of the driving environment synchronized in time with the image information is acquired.

Before, after, or at the same time as step S110, the depth information of the road surface acquired simultaneously with the image information may be additionally acquired.

For example, the depth sensor may be any suitable sensor, such as a binocular camera that measures depth based on a binocular disparity map or an infrared depth sensor (or a laser depth sensor) that measures depth based on infrared illumination. For example, depth sensors can generate depth information such as depth maps or laser point clouds that can be used to measure the position of obstacles relative to the current vehicle. The depth sensor can collect any suitable depth information related to the distance of the obstacle from the current vehicle. For example, a depth sensor can gather information about how far an obstacle is in front of the current vehicle. Still further, the depth sensor can collect directional information such as information on whether an obstacle is to the right or left of the current vehicle, in addition to distance information. The depth sensor can also collect information about the distance of the obstacle from the current vehicle at various points in time to determine whether the obstacle is moving towards or away from the current vehicle. In the following, the description will continue by taking the laser depth sensor as an example.

FIG. 3 illustrates a flowchart of a step of acquiring depth information according to an embodiment of the present application.

As shown in FIG. 3, step S120 may include:

In sub-step S121, the acquisition time for the imaging device to acquire the image information is determined.

For example, the image information may include various attribute information such as acquisition time. The acquisition time of the image information can be determined through the attribute information.

In sub-step S122, the depth information of the road surface in the driving direction collected by the depth sensor of the current vehicle at the collection time is obtained.

Similarly, the depth information may also include various attribute information such as acquisition time. Through the acquisition time of the image information, the depth information acquired at the same time point can be determined.

It should be noted that the present application is not limited to this. For example, in the acquisition stage of the imaging device and the depth sensor, the acquisition information and the depth information acquired at the same time point can also be stored together as a pair of related information for later acquisition.

Referring back to FIG. 2, in step S130, an obstacle area in the driving environment is marked in the image information according to the depth information.

After the corresponding image information and depth information are obtained, various methods can be used to combine the two to detect obstacles and their regions in the driving environment.

FIG. 4 illustrates a flowchart of a step of marking an obstacle according to an embodiment of the present application.

As shown in FIG. 4, step S130 may include:

In sub-step S131, it is determined whether there is an obstacle on the road surface according to the depth information.

For example, the obstacle may be at least one of: pedestrians, animals, litter, warning signs, barriers, and other vehicles.

Since the laser depth sensor transmits a very short light pulse, and measures the time from when the light pulse is emitted to when it is reflected back by the obstacle, the distance to the object is calculated by measuring the time interval, so according to the laser detected by the sensor, the distance to the object is calculated. The position and return time of the point cloud can determine whether there is an obstacle on the road surface, and the positional relationship between the obstacle and the current vehicle.

In sub-step S132, in response to the existence of an obstacle, a projection area of the obstacle on the road surface is determined in the image information according to the depth information of the obstacle.

Once it is determined that there are obstacles on the road surface according to the laser point cloud, for example, clustering can be performed according to the laser point cloud to roughly identify the number of possible obstacles, and the depth information of each obstacle can be used to classify the obstacles. Corresponding to the image information to determine the projection area of the obstacle on the road surface. It should be noted that although there may be overlaps between multiple obstacles in the image information, due to the driving law of vehicles, each obstacle must be kept at a certain distance in order to maintain safety, which makes the depth of the obstacles. The results of clustering based on information are often more accurate than those based on image information.

Specifically, for example, first, the three-dimensional coordinates of the obstacle relative to the current vehicle may be determined according to the depth information of the obstacle and the calibration parameters of the depth sensor.

Due to manufacturing tolerances, after a depth sensor is installed on a vehicle, each vehicle must perform an independent end-of-line sensor calibration or post-market sensor adjustment in order to determine that the depth sensor is on that vehicle The pitch angle and other calibration parameters are finally used to assist driving and other purposes. For example, the calibration parameter may refer to an extrinsic parameter matrix of the depth sensor, which may include one or more of a pitch angle and a tilt angle of the depth sensor relative to the current vehicle's formal direction. The three-dimensional coordinates of each laser point related to the obstacle, for example, coordinates (x, y, z), can be calculated based on the depth information of the obstacle according to the calibrated pitch angle and the like and a preset algorithm. The three-dimensional coordinates may be absolute coordinates of the obstacle in the world coordinate system, or may be relative coordinates with the reference position of the current vehicle.

Then, the height coordinate z in the three-dimensional coordinates of the obstacle may be set to zero to generate three-dimensional coordinates projected onto the road surface. That is, the three-dimensional coordinates of each laser point related to the obstacle can be modified to (x, y, 0).

Finally, the projected area of the obstacle on the road surface may be determined in the image information according to the projected three-dimensional coordinates and the calibration parameters of the imaging device.

Similar to the depth sensor, due to manufacturing tolerances, after the imaging device is installed on the vehicle, calibration parameters such as the pitch angle of the imaging device on the vehicle also need to be determined first. Therefore, the projected three-dimensional coordinates of each laser point related to the obstacle can be converted into each image in the image information according to the pitch angle of the imaging device relative to the current driving direction of the vehicle and the preset algorithm. coordinates, and the outermost peripheral area (ie, the largest contour area) of each image coordinate is determined as the projected area of the obstacle on the road surface.

In sub-step S133, the projection area is marked as an obstacle area on the road surface.

The projection area of the obstacle on the road surface determined according to the outermost area of each image coordinate can be automatically marked as the obstacle area on the road surface by means of circle selection or the like.

It can be seen that, by using the region labeling method according to the first embodiment of the present application, the image information of the driving environment collected by the imaging device can be obtained during the process of generating the training samples for training the machine learning model, and the image information corresponding to the image can be obtained. The information is time-synchronized with depth information of the driving environment, and an obstacle area in the driving environment is marked in the image information according to the depth information. Therefore, compared with the situation of manually labeling the obstacle area as in the prior art, the obstacle area in the driving environment can be automatically labeled, which improves the efficiency of area labeling.

In the above-mentioned first embodiment, the obstacle area can be automatically marked in the image information collected by the imaging device by combining the depth information from the depth sensor during the marking process. However, in order to achieve assisted driving and other purposes, not only the obstacle area should be marked, but also the drivable area in the entire driving environment should be marked, and the training samples for the machine learning model should be generated based on the above marking results.

In order to solve the above problems, a second embodiment of the present application is proposed based on the first embodiment of the present application.

FIG. 5 illustrates a flowchart of a region labeling method according to the second embodiment of the present application.

As shown in FIG. 5 , the area labeling method according to the second embodiment of the present application may include:

In FIG. 5 , the same reference numerals are used to refer to the same steps as in FIG. 2 . Therefore, steps S110 - S130 in FIG. 5 are the same as steps S110 - S130 in FIG. 2 , and reference may be made to the above description in conjunction with FIGS. 2 to 4 . The difference between FIG. 5 and FIG. 2 is that step S140 and a further optional step S150 are added.

In step S140, a drivable area in the driving environment is marked in the image information according to the user input and the obstacle area.

Before, after, or at the same time as the obstacle area on the road surface is marked in the image information, various methods can also be used to detect the drivable area on the road surface in the image information.

FIG. 6 illustrates a flowchart of the steps of marking a drivable area according to an embodiment of the present application.

As shown in FIG. 6, step S140 may include:

In sub-step S141, user input is received.

The user input may be the boundary position information of the road surface found by the user based on human eye recognition, which may include coordinate input or circle selection input on the image, and the like.

In sub-step S142, a road surface boundary of the road surface is determined according to the user input.

For example, the road surface boundary of the road surface can be marked in the image information according to the boundary position information input by the user. For example, the pavement boundary may be at least one of the following: a curb, a divider, a green belt, a guardrail, a lane line, and the edge of other vehicles.

In sub-step S143, the drivable area on the road surface is marked according to the road surface boundary and the obstacle area.

For example, a road surface area on the road surface may be determined from the road surface boundary, and the obstacle area may be removed from the road surface area to obtain the drivable area.

Next, the effects of the embodiments of the present application will be described through a specific experiment.

7A illustrates a schematic diagram of combining depth information and user input in the image information shown in FIG. 1 according to an embodiment of the present application, and FIG. 7B illustrates a schematic diagram of the image information shown in FIG. 1 according to an embodiment of the present application Schematic annotated with obstacle areas and drivable areas.

Referring to FIG. 7A , during the labeling process, time-synchronized image information and laser sensor information can be acquired. In the image information, the user can mark the road surface shown in Figure 1 through human eye recognition that there are 4 lane lines (lane lines 1 to 4) and 1 boundary line (boundary line 1), which are used as the boundary of the road surface. Candidate ID. Here, the current range of road surfaces that the vehicle can travel on may be determined depending on different assisted driving strategies. For example, when lane line 3 and lane line 4 are solid lines, they can be used as road boundaries to determine the road surface range under normal circumstances, but in emergency situations (such as when there is a warning of possible collision ahead or behind), The road surface range can be determined by taking the maximum physically drivable range, namely, road boundary 1 and road boundary 5 as road boundary. In addition, as shown in FIG. 7A , it can also be detected that there are three laser point clusters (laser point clusters 1 to 3 ) on the road surface based on depth information such as a laser point cloud. Next, the intersections of obstacles 1 to 3 and the ground plane can be obtained by transforming the spatial coordinates of the laser point clusters 1 to 3 and projecting them on the ground plane of the road surface in the image information. Finally, the largest contour area above the intersection can be marked as an obstacle area, that is, a non-drivable area, and the remaining area can be marked as a drivable area, as shown in FIG. 7B , where road boundary 1 and road boundary 5 are marked as The illustration is shown as an example of a road surface boundary.

Referring back to FIG. 5 , next, optionally, in step S150 , the training samples are generated based on image information in which the drivable area is marked.

For example, image information and associated annotation information can be packaged together to generate training samples for subsequent training of machine learning models.

It can be seen that, by using the area labeling method according to the second embodiment of the present application, the obstacle area in the driving environment can be marked in the image information collected by the imaging device according to the depth information collected by the depth sensor, and the obstacle area in the driving environment can also be marked according to the depth information collected by the depth sensor. The user inputs an environmental boundary in the driving environment marked in the image information, determines a drivable area in the driving environment according to the environmental boundary and the obstacle area, and determines the drivable area in the driving environment based on the drivable area marked therein. image information to generate the training samples. Therefore, it is possible to reliably and efficiently detect drivable areas in the driving environment and generate training samples for use by the machine learning model.

Exemplary device

Hereinafter, with reference to FIG. 8 , a region marking apparatus according to an embodiment of the present application will be described.

FIG. 8 illustrates a block diagram of a region labeling apparatus according to an embodiment of the present application.

As shown in FIG. 8 , the region labeling apparatus 100 may include: an image acquisition unit 110 for acquiring image information of the driving environment collected by the imaging device during the process of generating training samples for training the machine learning model; depth an obtaining unit 120, configured to obtain depth information of the driving environment synchronized with the image information in time; and an obstacle labeling unit 130, configured to label the driving environment in the image information according to the depth information obstacle area.

In one example, the image acquisition unit 110 may acquire image information of the road surface in the current driving direction of the vehicle.

In one example, the depth acquisition unit 120 may include: a time determination module for determining the acquisition time for the imaging device to acquire the image information; and a depth acquisition module for acquiring when the depth sensor of the current vehicle is The depth information of the road surface in the driving direction collected at the collection time.

In one example, the obstacle labeling unit 130 may include: an obstacle judging module for judging whether there is an obstacle on the road surface according to the depth information; a projection determination module for responding to the existence of an obstacle, Determine the projection area of the obstacle on the road surface in the image information according to the depth information of the obstacle; and an obstacle labeling module, configured to label the projection area as an obstacle on the road surface object area.

In one example, the projection determination module may determine the three-dimensional coordinates of the obstacle relative to the current vehicle according to the depth information of the obstacle and the calibration parameters of the depth sensor; The height coordinates in the coordinates are set to zero to generate three-dimensional coordinates projected onto the road surface; and the image information is determined according to the projected three-dimensional coordinates and calibration parameters of the imaging device. The projected area of the obstacle on the road surface.

In one example, the obstacle may be at least one of: pedestrians, animals, litter, warning signs, barriers, and other vehicles.

In one example, the area labeling apparatus 100 may further include: a drivable labeling unit (not shown), configured to label the driving environment in the image information according to user input and the obstacle area. drivable area.

In one example, the drivable labeling unit may include: an input receiving module for receiving user input; a boundary determining module for determining a road surface boundary of the road surface according to the user input; and a drivable labeling module , which is used to mark the drivable area on the road surface according to the road surface boundary and the obstacle area.

In one example, the drivable annotation module may determine a pavement area on the road pavement based on the pavement boundary; and remove the obstacle area from the pavement area to obtain the drivable area.

In one example, the region labeling apparatus 100 may further include: a sample generating unit (not shown) for generating the training samples based on image information in which the drivable region is labeled.

The specific functions and operations of the respective units and modules in the above-mentioned area labeling apparatus 100 have been introduced in detail in the area labeling method described above with reference to FIGS. 1 to 7B , and thus, repeated descriptions thereof will be omitted.

As described above, the embodiments of the present application can be applied to the obstacle area in the driving environment where various online electronic devices such as vehicles, mobile robots, fixed surveillance cameras and the like equipped with imaging devices are located. callout. Moreover, the area marking method and the area marking apparatus according to the embodiments of the present application may be directly implemented on the above-mentioned online electronic device. However, considering that online electronic devices often have limited processing capabilities, in order to obtain better performance, the embodiments of the present application can also be implemented in various devices that can communicate with online electronic devices to transmit trained machine learning models to them. off-line electronic devices. For example, the offline electronic device may include, for example, a terminal device, a server, and the like.

Correspondingly, the region marking apparatus 100 according to the embodiment of the present application may be integrated into the offline electronic device as a software module and/or a hardware module, in other words, the electronic device may include the region marking apparatus 100 . For example, the region marking apparatus 100 may be a software module in the operating system of the electronic device, or may be an application program developed for the electronic device; of course, the region marking apparatus 100 may also be a software module of the electronic device. One of many hardware modules.

Alternatively, in another example, the area marking apparatus 100 and the offline electronic device may also be separate devices, and the area marking apparatus 100 may be connected to the electronic device through a wired and/or wireless network, and as agreed data format to transmit interactive information.

Exemplary Electronics

Hereinafter, an electronic device according to an embodiment of the present application will be described with reference to FIG. 9 . The electronic device may be an online electronic device such as a vehicle, a mobile robot, etc., on which an imaging device is equipped, or an offline electronic device capable of communicating with the online electronic device to transmit the trained machine learning model thereto. equipment.

FIG. 9 illustrates a block diagram of an electronic device according to an embodiment of the present application.

As shown in FIG. 9 , the electronic device 10 includes one or more processors 11 and a memory 12 .

Processor 11 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in electronic device 10 to perform desired functions.

Memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory, or the like. The non-volatile memory may include, for example, read only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions, so as to implement the region labeling method of the various embodiments of the present application described above and/or other desired function. Various contents such as image information, depth information, obstacle areas, drivable areas, annotation information, etc. may also be stored in the computer-readable storage medium.

In one example, the electronic device 10 may also include an input device 13 and an output device 14 interconnected by a bus system and/or other form of connection mechanism (not shown). It should be noted that the components and structures of the electronic device 10 shown in FIG. 9 are only exemplary and not limiting, and the electronic device 10 may also have other components and structures as required.

For example, the input device 13 may be an imaging device for capturing image information, which may be stored in the memory 12 for use by other components. Of course, other integrated or discrete imaging devices can also be used to capture the sequence of image frames and send it to the electronic device 10 . For another example, the input device 13 may also be a depth sensor for collecting depth information, and the collected depth information may also be stored in the memory 12 . In addition, the input device 13 may also include, for example, a keyboard, a mouse, a communication network and a remote input device connected thereto, and the like.

The output device 14 may output various information to the outside (eg, a user or a machine learning model), including the determined obstacle area, drivable area, training samples, and the like of the driving environment. The output devices 14 may include, for example, displays, speakers, printers, and communication networks and their connected remote output devices, among others.

Of course, for simplicity, only some of the components in the electronic device 10 related to the present application are shown in FIG. 9 , and components such as buses, input/output interfaces and the like are omitted. Besides, the electronic device 10 may also include any other suitable components according to the specific application.

Exemplary computer program product and computer readable storage medium

In addition to the methods and apparatuses described above, embodiments of the present application may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the "exemplary methods" described above in this specification The steps in the region labeling method according to various embodiments of the present application described in the section.

The computer program product can write program codes for performing the operations of the embodiments of the present application in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc. , also includes conventional procedural programming languages, such as "C" language or similar programming languages. The program code may execute entirely on the user computing device, partly on the user device, as a stand-alone software package, partly on the user computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.

In addition, embodiments of the present application may also be computer-readable storage media having computer program instructions stored thereon, the computer program instructions, when executed by a processor, cause the processor to perform the above-mentioned "Example Method" section of this specification Steps in the region labeling method according to various embodiments of the present application described in .

The computer-readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses or devices, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, advantages, effects, etc. mentioned in the present application are only examples rather than limitations, and these advantages, advantages, effects, etc., are not considered to be Required for each embodiment of this application. In addition, the specific details disclosed above are only for the purpose of example and easy understanding, rather than limiting, and the above-mentioned details do not limit the application to be implemented by using the above-mentioned specific details.

The block diagrams of devices, apparatus, apparatuses, and systems referred to in this application are merely illustrative examples and are not intended to require or imply that the connections, arrangements, or configurations must be in the manner shown in the block diagrams. As those skilled in the art will appreciate, these means, apparatuses, apparatuses, systems may be connected, arranged, configured in any manner. Words such as "including", "including", "having" and the like are open-ended words meaning "including but not limited to" and are used interchangeably therewith. As used herein, the words "or" and "and" refer to and are used interchangeably with the word "and/or" unless the context clearly dictates otherwise. As used herein, the word "such as" refers to and is used interchangeably with the phrase "such as but not limited to".

It should also be pointed out that in the apparatus, equipment and method of the present application, each component or each step can be decomposed and/or recombined. These disaggregations and/or recombinations should be considered as equivalents of the present application.

The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Therefore, this application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims (12)

1. A region labeling method comprises the following steps:

in the process of generating a training sample for training a machine learning model, acquiring image information of a driving environment acquired by an imaging device, wherein the image information comprises gray scale or color information of the driving environment;

acquiring depth information of the driving environment, which is acquired by a depth sensor synchronized with the image information in time, wherein the depth information comprises information indicating distances of obstacles in the driving environment from a current vehicle at different time points; and

labeling in the image information an obstacle area in the driving environment according to the depth information, the labeling including:

detecting laser point clusters existing in a road pavement based on the depth information;

converting the space coordinates of the laser spot cluster and projecting the space coordinates onto a ground plane of a road surface in the image information to obtain an intersection point of the obstacle and the ground plane; and

marking the maximum outline area above the intersection point as an obstacle area,

the depth sensor is a laser depth sensor.

2. The method of claim 1, wherein acquiring image information of the driving environment acquired by the imaging device comprises:

image information of a road surface in a traveling direction of a current vehicle is acquired.

3. The method of claim 2, wherein obtaining depth information of the driving environment acquired by a depth sensor temporally synchronized with the image information comprises:

determining the acquisition time for the imaging device to acquire the image information; and

and acquiring the depth information of the road surface in the driving direction acquired by the depth sensor of the current vehicle at the acquisition time.

4. The method of claim 3, wherein the marking of the obstacle area in the driving environment in the image information in accordance with the depth information comprises:

judging whether an obstacle exists on the road surface according to the depth information;

in response to the existence of an obstacle, determining a projection area of the obstacle on the road surface in the image information according to the depth information of the obstacle; and

labeling the projected area as an obstacle area on the road surface.

5. The method of claim 4, wherein determining a projected area of the obstacle on the road surface in the image information from the depth information of the obstacle comprises:

determining the three-dimensional coordinates of the obstacle relative to the current vehicle according to the depth information of the obstacle and the calibration parameters of the depth sensor;

setting a height coordinate in the three-dimensional coordinates of the obstacle to zero to generate three-dimensional coordinates after projection onto the road surface; and

and determining a projection area of the obstacle on the road surface in the image information according to the projected three-dimensional coordinates and the calibration parameters of the imaging device.

6. The method of claim 4 or 5, wherein the obstacle is at least one of: pedestrians, animals, spills, warning signs, piers, and other vehicles.

7. The method of claim 5, further comprising:

receiving a user input;

determining a road surface boundary of the road surface from the user input; and

marking a drivable area on the road surface according to the road surface boundary and the obstacle area.

8. The method of claim 7, wherein labeling the drivable area on the road surface as a function of the road surface boundary and the obstacle region comprises:

determining a road surface area on the road surface according to the road surface boundary; and

removing the obstacle area from the road surface area to obtain the travelable area.

9. The method of claim 7 or 8, further comprising:

generating the training sample based on image information in which the travelable region is labeled.

10. A region labeling apparatus comprising:

the device comprises an image acquisition unit, a processing unit and a processing unit, wherein the image acquisition unit is used for acquiring image information of a driving environment acquired by an imaging device in the process of generating a training sample for training a machine learning model, and the image information comprises gray scale or color information of the driving environment;

a depth acquisition unit configured to acquire depth information of the driving environment acquired by a depth sensor that is time-synchronized with the image information, the depth information including information indicating distances of obstacles in the driving environment from a current vehicle at different time points; and

an obstacle labeling unit configured to label an obstacle region in the driving environment in the image information according to the depth information, the labeling including:

detecting laser point clusters existing in a road pavement based on the depth information;

converting the space coordinates of the laser spot cluster and projecting the space coordinates onto a ground plane of a road surface in the image information to obtain an intersection point of the obstacle and the ground plane; and

and marking the maximum outline area above the intersection point as an obstacle area.

11. An electronic device, comprising:

a processor;

a memory; and

computer program instructions stored in the memory, which, when executed by the processor, cause the processor to perform the method of any of claims 1-9.

12. A computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1-9.

Images