CN114818967A - Target object classification method, device, vehicle and system - Google Patents

Abstract

The application provides a target object classification method, a target object classification device, a vehicle and a target object classification system, wherein the method comprises the following steps: acquiring current global data, wherein the current global data comprise a current target object aerial view and road information where the current target object is located, the road information comprises road types, and the road types are lanes, sidewalks or pedestrian crossings; acquiring current local data of a current target object, wherein the current local data is obtained by fusing image data and point cloud data, the image data is acquired by adopting an image sensor, and the point cloud data is acquired by adopting a laser radar; and determining the category of the current target object at least according to the current global data and the current local data. The scheme combines the current local data and the current global data to realize the accurate determination of the category of the current target object.

Description

Object classification method, device, vehicle and system

technical field

The present application relates to the technical field of automatic driving, and in particular, to a target object classification method, device, vehicle, system, computer-readable storage medium, and processor.

Background technique

In order to achieve accurate unmanned driving, the perception of the surrounding environment of unmanned vehicles is extremely important, mainly to identify the targets (including social vehicles, pedestrians, buildings, street signs, etc.) in the surrounding environment of unmanned vehicles. , including identifying the type and location of the target.

Existing solutions often use point cloud data collected by lidar to classify objects. However, due to the sparseness and disorder of point cloud data collected by lidar, it is difficult to accurately classify targets by using lidar.

SUMMARY OF THE INVENTION

The main purpose of the present application is to provide a target object classification method, device, vehicle, system, computer-readable storage medium and processor, so as to solve the problem that it is difficult to accurately classify the target objects by using laser radar in the prior art.

In order to achieve the above object, according to an aspect of the present application, a method for classifying objects is provided, the method comprising: acquiring current global data, where the current global data includes a bird's-eye view of the current object and a location where the current object is located. Road information, the road information includes road types, which are roadways, sidewalks or crosswalks; obtain current local data of the current target, wherein the current local data is obtained by fusing image data and point cloud data The image data is acquired by an image sensor, and the point cloud data is acquired by a lidar; the category of the current target is determined at least according to the current global data and the current local data.

Further, determining the category of the current target object at least according to the current global data and the current local data includes: determining a rough category of the current target object according to the current global data, where the rough category includes vehicles and Pedestrian; according to the rough category of the current target object and the current local data of the current target object, determine the fine category of the current target object, and the fine category includes cars, buses, ambulances and construction vehicles.

Further, determining the rough category of the current target according to the current global data includes: determining the size of the projection of the current target according to the bird's-eye view of the current target; according to the size of the projection of the current target and The road information on which the current target is located determines the rough category of the current target.

Further, according to the coarse category of the current target object and the current local data of the current target object, determining the fine category of the current target object includes: extracting the current target object from the current local data of the current target object. The detailed feature of the target object; according to the coarse category of the current target object and the detailed feature of the current target object, the fine category of the current target object is determined.

Further, determining the rough category of the current target object according to the current global data, where the rough category includes vehicles and pedestrians, includes: constructing a global model, wherein the global model is obtained by training with multiple sets of training data, Each set of training data in the multiple sets of training data includes: obtained within a historical time period: historical global data and a rough category of historical objects corresponding to the historical global data; inputting the current global data into the Perform operations in the global model to obtain the rough category of the current target; determine the fine category of the current target according to the rough category of the current target and the current local data of the current target, including: constructing a local model, wherein the global model is obtained by training using multiple sets of training data, and each set of training data in the multiple sets of training data includes acquired within a historical time period: historical local data, determined according to the global model The coarse category of the historical target and the fine category of the historical target; the coarse category of the current target and the current local data of the current target are input into the local model for operation, and the current target is obtained. Subcategory of the target.

Further, determining the category of the current target object at least according to the current global data and the current local data includes: acquiring temperature information of the current target object; The current local data is used to determine the category of the current target.

Further, acquiring the current local data of the current target includes: acquiring a rotation matrix and a translation vector between the coordinate system of the image sensor and the coordinate system of the lidar; according to the rotation matrix and the translation vector , the image data and the point cloud data are fused to obtain the current local data.

Further, determining the category of the target object at least according to the global data and the local data includes: constructing a neural network model, wherein the neural network model is obtained by using multiple sets of training data, and the multiple sets of training data are used for training. Each group of training data in the training data includes: historical global data, historical local data, and historical target categories corresponding to the historical global data and historical local data obtained in a historical time period; according to the neural network model , and determine the category of the current target object corresponding to the global data and the local data.

Further, determining the category of the target object according to at least the global data and the local data includes: constructing a random forest model, wherein the random forest model is obtained by training with multiple sets of training data, and the multiple sets of training data are used for training. Each group of training data in the training data includes: historical global data, historical local data, and historical target categories corresponding to the historical global data and historical local data obtained within a historical time period; according to the random forest model , and determine the category of the current target object corresponding to the global data and the local data.

Further, constructing the random forest model includes: using the historical global data, the first data combination in the historical local data, and the historical target category, and training to obtain a first decision tree; using the historical data The second decision tree is obtained by training the global data, the second data combination in the historical local data and the category of the historical target; using the third data combination in the historical global data and the historical local data and the category of the historical target object, a third decision tree is obtained by training; the first decision tree, the second decision tree and the third decision tree are integrated to obtain the random forest model.

According to another aspect of the present application, an apparatus for classifying objects is provided, the apparatus includes: a first acquiring unit, a second acquiring unit and a determining unit; the first acquiring unit is configured to acquire current global data, the current The global data includes a bird's-eye view of the current target and road information on which the current target is located, the road information includes a road type, and the road type is a roadway, a sidewalk or a crosswalk; the second acquisition unit is used to acquire The current local data of the current target, wherein the current local data is obtained by fusing image data and point cloud data, the image data is obtained by using an image sensor, and the point cloud data is obtained by using a lidar; The determining unit is configured to determine the category of the current target at least according to the current global data and the current local data.

Further, the determining unit includes: a first determining module and a second determining module, the first determining module is configured to determine a rough category of the current target object according to the current global data, and the rough category includes vehicles and Pedestrian; the second determination module is configured to determine the fine category of the current target object according to the rough category of the current target object and the current local data of the current target object, and the fine category includes cars, buses , ambulances and construction vehicles.

According to another aspect of the present application, there is also provided a vehicle comprising one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory, and configured to be executed by the one or more processors, the one or more programs comprising for performing any of the methods described above.

According to another aspect of the present application, there is also provided a system including a vehicle and an image sensor and a lidar mounted on the vehicle, the vehicle including a controller configured to execute any one of the above Methods.

According to another aspect of the present application, a computer-readable storage medium is also provided, and the computer-readable storage medium includes a stored program, wherein when the program runs, the device where the computer-readable storage medium is located is controlled to execute any of the methods described above.

According to another aspect of the present application, a processor is also provided, and the processor is configured to run a program, wherein when the program is run, any one of the above-mentioned methods is executed.

Applying the technical solution of the present application, by acquiring current global data, the current global data includes a bird's-eye view of the current target object and the road information on which the above-mentioned current target object is located, the current local data of the current target object is obtained, and then at least according to the current global data and the above-mentioned The current local data determines the category of the current target. Accurate determination of the class of the current target can be achieved.

Description of drawings

The accompanying drawings that form a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute improper limitations on the present application. In the attached image:

1 shows a flowchart of a method for classifying objects according to an embodiment of the present application;

FIG. 2 shows a schematic diagram of an apparatus for classifying objects according to an embodiment of the present application.

Detailed ways

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only The embodiments are part of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances for the embodiments of the application described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may also be present. Also, in the specification and claims, when an element is described as being "connected" to another element, the element can be "directly connected" to the other element or "connected" to the other element through a third element.

For the convenience of description, some nouns or terms involved in the embodiments of the present application are described below:

As mentioned in the background art, the existing solutions often use point cloud data collected by lidar to classify objects. However, due to the sparseness and disorder of point cloud data collected by lidar, it is difficult to use lidar to classify objects. For accurate classification, in order to solve the problem that it is difficult to accurately classify targets by using lidar in the prior art, a target classification method, device, vehicle, system, computer-readable storage medium and processor are proposed.

According to the embodiments of the present application, a method for classifying objects is provided.

FIG. 1 is a flowchart of a target object classification method according to an embodiment of the present application. As shown in Figure 1, the method includes the following steps:

Step S101, obtaining current global data, the current global data includes a bird's-eye view of the current target and road information where the current target is located, the road information includes a road type, and the road type is a roadway, a sidewalk or a crosswalk;

In the above steps, the bird's-eye view of the current target may be obtained by radar.

Specifically, the roadway is the roadway, and the width of the road for various vehicles to drive together within the same road width is the width of the roadway, also known as the width of a single road.

Specifically, the sidewalk refers to the part of the road that is divided by curbs or guardrails and other similar facilities for pedestrians, and generally has a width of about four meters.

Specifically, a pedestrian crossing refers to a prescribed walking range for pedestrians to cross the lane marked by markings such as zebra crossings or other methods on the roadway. It is to prevent pedestrians from being injured when the vehicle is moving fast, and the markings on the lane designate the place to slow down to allow pedestrians to cross the street.

Step S102, obtaining the current local data of the current target, wherein the current local data is obtained by fusing image data and point cloud data, the image data is obtained by using an image sensor, and the point cloud data is obtained by using a laser radar;

Step S103, at least according to the current global data and the current local data, determine the category of the current target object.

In the above scheme, by obtaining the current global data, the current global data includes the bird's-eye view of the current target and the road information on which the current target is located, and the current local data of the current target is obtained, and then at least according to the current global data and the above-mentioned current local data. , to determine the category of the current target. Accurate determination of the class of the current target can be achieved.

It should be noted that the steps shown in the flowcharts of the accompanying drawings may be executed in a computer system, such as a set of computer-executable instructions, and, although a logical sequence is shown in the flowcharts, in some cases, Steps shown or described may be performed in an order different from that herein.

In an embodiment of the present application, determining the category of the current object according to at least the current global data and the current local data includes: determining a rough category of the current object according to the current global data, where the rough category includes vehicles and pedestrians; according to the coarse category of the current target object and the current local data of the current target object, the fine category of the current target object is determined, and the fine category includes cars, buses, ambulances and construction vehicles. Since the current global data includes the bird's-eye view of the current target and the road information where the current target is located, the rough category of the current target can be roughly determined according to the current global data, and then combined with the local data on the premise of determining the rough category It is possible to accurately determine the category of the current target.

In an embodiment of the present application, determining the rough category of the current target according to the current global data includes: determining the size of the projected area of the current target according to the bird's-eye view of the current target; The size of the area and the road information on which the current target object is located determine the rough category of the current target object. The size of the projected area refers to the projected area of the orthographic projection on the road surface. Specifically, it is found from the bird's-eye view of the current target that the projected area of the current target is large, and it is found that it is located on the driving lane, and it can be preliminarily determined as a vehicle, and then it is determined whether it is a car, a bus, an ambulance or a construction vehicle according to the current local data. . Specifically, it is found from the bird's-eye view of the current target that the projected area of the current target is small, and it can be preliminarily judged that it is a pedestrian when it is found on the sidewalk, and then it is determined that it is a single pedestrian, a pedestrian on a skateboard or a pedestrian based on the current local data. Or a pedestrian with a stroller.

In an embodiment of the present application, determining the fine category of the current object according to the rough category of the current object and the current partial data of the current object includes: extracting from the current partial data of the current object The detailed feature of the current target object; according to the coarse category of the current target object and the detailed feature of the current target object, the fine category of the current target object is determined. Specifically, since the detailed characteristics of cars, buses, ambulances and construction vehicles are different, on the premise of determining the coarse category, combined with the detailed characteristics of the current target, it is possible to realize the detailed characteristics of the current target. Sure.

In some embodiments, determining the rough category of the current target object according to the current global data, where the rough category includes vehicles and pedestrians, includes: constructing a global model, wherein the global model is obtained by training using multiple sets of training data, and the above Each set of training data in the multiple sets of training data includes: the historical global data and the rough category of the historical target object corresponding to the historical global data obtained in the historical time period; the current global data is input into the global model Perform operations to obtain the rough category of the current target; determine the fine category of the current target according to the rough category of the current target and the current local data of the current target, including: constructing a local model, wherein the global The model is obtained by training with multiple sets of training data, and each set of training data in the multiple sets of training data includes: historical local data, the coarse category of the historical target determined according to the above-mentioned global model, and The fine classification of historical objects; the coarse classification of the current object and the current local data of the current object are input into the local model for operation to obtain the fine classification of the current object. The output of the global model is used as the input of the local model, so that the global data and the local data are better integrated, which further ensures the accuracy of target classification.

In an embodiment of the present application, determining the category of the current target object at least according to the current global data and the current local data includes: acquiring temperature information of the current target object; according to the temperature information, the current global data and the The above-mentioned current local data determines the type of the above-mentioned current target object. In the case of a vehicle, it may be determined whether it is an activated vehicle or a non-activated vehicle according to the temperature of the vehicle. For another example, in the case where it is detected that the temperature is within the body temperature range, the probability of determining that it is a pedestrian is greatly increased.

Specifically, a temperature sensor (eg, an infrared temperature detector, etc.) can be used to obtain the temperature information of the current target.

In an embodiment of the present application, acquiring the current local data of the current target includes: acquiring a rotation matrix and a translation vector between the coordinate system of the image sensor and the coordinate system of the lidar; Translate the vector, and fuse the above image data and the above point cloud data to obtain the above current local data. Through coordinate transformation, the coordinate system of the image sensor and the coordinate system of the lidar are placed in the same coordinate system, thereby realizing the fusion of the image data and the above point cloud data.

In an embodiment of the present application, determining the category of the object according to at least the global data and the local data includes: constructing a neural network model, wherein the neural network model is obtained by training using multiple sets of training data, and the above Each set of training data in the multiple sets of training data includes: historical global data, historical local data, and the categories of historical objects corresponding to the above-mentioned historical global data and historical local data obtained in a historical time period; according to the above-mentioned neural network model , to determine the category of the target object corresponding to the global data and the local data. The neural network algorithm can be used to accurately determine the category of the target.

In an embodiment of the present application, determining the category of the object according to at least the global data and the local data includes: constructing a random forest model, wherein the random forest model is obtained by training using multiple sets of training data, and the above Each set of training data in the multiple sets of training data includes: historical global data, historical local data, and categories of historical objects corresponding to the above-mentioned historical global data and historical local data obtained in a historical time period; according to the above random forest model , to determine the category of the target object corresponding to the global data and the local data. Due to the randomness of the training data selection of the random forest algorithm, the accurate determination of the category of the target object can be guaranteed.

In an embodiment of the present application, constructing the above random forest model includes: using the above-mentioned historical global data, the first data combination in the above-mentioned historical local data, and the above-mentioned historical target category, and training to obtain a first decision tree; using The above-mentioned historical global data, the second type of data combination in the above-mentioned historical local data, and the above-mentioned historical target category are trained to obtain a second decision tree; the third type of data combination in the above-mentioned historical global data, the above-mentioned historical local data and the above-mentioned historical global data are used. A third decision tree is obtained through training; the above-mentioned first decision tree, the above-mentioned second decision tree and the above-mentioned third decision tree are integrated to obtain the above-mentioned random forest model. That is, different decision trees are obtained by training with different parameters, and then different decision trees are fused to obtain an accurate random forest model.

The embodiment of the present application further provides a target object classification apparatus. It should be noted that the target object classification apparatus of the embodiment of the present application may be used to execute the method for target object classification provided by the embodiment of the present application. The apparatus for classifying objects provided in the embodiments of the present application will be introduced below.

FIG. 2 is a schematic diagram of a target object classification apparatus according to an embodiment of the present application. As shown in FIG. 2 , the apparatus includes: a first obtaining unit 10, a second obtaining unit 20 and a determining unit 30;

The above-mentioned first obtaining unit 10 is used to obtain the current global data. The above-mentioned current global data includes a bird's-eye view of the current target object and the road information on which the above-mentioned current target object is located. The above-mentioned road information includes a road type, and the above-mentioned road type is a roadway and a sidewalk. or a pedestrian crossing;

In the above steps, the bird's-eye view of the current target may be obtained by radar.

Specifically, the roadway is the roadway, and the width of the road for various vehicles to drive together within the same road width is the width of the roadway, also known as the width of a single road.

Specifically, the sidewalk refers to the part of the road that is divided by curbs or guardrails and other similar facilities for pedestrians, and generally has a width of about four meters.

Specifically, a pedestrian crossing refers to a prescribed walking range for pedestrians to cross the lane marked by markings such as zebra crossings or other methods on the roadway. It is to prevent pedestrians from being injured when the vehicle is moving fast, and the markings on the lane designate the place to slow down to allow pedestrians to cross the street.

The above-mentioned second obtaining unit 20 is used to obtain the current local data of the above-mentioned current target, wherein the above-mentioned current local data is obtained by fusing image data and point cloud data, the above-mentioned image data is obtained by using an image sensor, and the above-mentioned point cloud data is obtained by using an image sensor. obtained by lidar;

The determining unit 30 is configured to determine the category of the current target object at least according to the current global data and the current local data.

In the above solution, the first acquisition unit acquires current global data, the current global data includes a bird's-eye view of the current target and road information on which the current target is located, the second acquisition unit acquires the current local data of the current target, and the determination unit at least according to The current global data and the above-mentioned current local data determine the type of the above-mentioned current target object. Accurate determination of the class of the current target can be achieved.

In an embodiment of the present application, the determination unit includes: a first determination module and a second determination module, the first determination module is configured to determine the rough category of the current target according to the current global data, and the rough category includes Vehicles and pedestrians; the second determining module is used to determine the fine category of the current target according to the rough category of the current target and the current local data of the current target, and the fine category includes cars, buses, and ambulances and construction vehicles. Since the current global data includes the bird's-eye view of the current target and the road information where the current target is located, the rough category of the current target can be roughly determined according to the current global data, and then combined with the local data on the premise of determining the rough category It is possible to accurately determine the category of the current target.

In an embodiment of the present application, the above-mentioned first determination module includes: a first determination sub-module and a second determination sub-module, and the above-mentioned first determination sub-module is configured to determine the current target object according to the bird's-eye view of the above-mentioned current target object. The size of the projected area; the second determining sub-module is configured to determine the rough category of the current target according to the size of the projected area of the current target and the road information on which the current target is located. The size of the projected area refers to the projected area of the orthographic projection on the road surface. Specifically, it is found from the bird's-eye view of the current target that the projected area of the current target is large, and it is found that it is located on the driving lane, and it can be preliminarily determined as a vehicle, and then it is determined whether it is a car, a bus, an ambulance or a construction vehicle according to the current local data. . Specifically, it is found from the bird's-eye view of the current target that the projected area of the current target is small, and it can be preliminarily judged that it is a pedestrian when it is found on the sidewalk, and then it is determined that it is a single pedestrian, a pedestrian on a skateboard or a pedestrian based on the current local data. Or a pedestrian with a stroller.

In an embodiment of the present application, the second determination module includes: an extraction sub-module and a third determination sub-module, and the extraction sub-module is used to extract the current local data of the current target from the current local data of the current target. Detailed features; the third determining sub-module is configured to determine the fine category of the current target according to the coarse category of the current target and the detailed feature of the current target. Specifically, since the detailed characteristics of cars, buses, ambulances and construction vehicles are different, on the premise of determining the coarse category, combined with the detailed characteristics of the current target, it is possible to realize the detailed characteristics of the current target. Sure.

In some embodiments, the first determination module includes a first construction sub-module and a first operation sub-module, and the first construction sub-module is used to construct a global model, wherein the above-mentioned global model is obtained by training using multiple sets of training data, and the above-mentioned Each group of training data in the multiple sets of training data includes: the historical global data and the rough category of the historical target object corresponding to the historical global data obtained in the historical time period; the first operation sub-module is used for the current global data. Input into the global model for operation to obtain the rough category of the current target; the second determination module includes a second construction sub-module and a second operation sub-module, and the second construction sub-module is used to construct a local model, wherein, The above-mentioned global model is obtained by training with multiple sets of training data, and each set of training data in the above-mentioned multiple sets of training data includes: historical local data, the coarse category of the historical target determined according to the above-mentioned global model. , and the detailed classification of historical objects; the second operation sub-module is used to input the rough classification of the above-mentioned current objects and the current local data of the above-mentioned current objects into the above-mentioned local model for operation, and obtain the details of the above-mentioned current objects. category. The output of the global model is used as the input of the local model, so that the global data and the local data are better integrated, which further ensures the accuracy of target classification.

In an embodiment of the present application, the determination unit includes: a first acquisition module and a third determination module, the first acquisition module is used to acquire the temperature information of the current target; the third determination module is used to The temperature information, the current global data and the current local data determine the category of the current target. In the case of a vehicle, it may be determined whether it is an activated vehicle or a non-activated vehicle according to the temperature of the vehicle. For another example, in the case where it is detected that the temperature is within the body temperature range, the probability of determining that it is a pedestrian is greatly increased.

In an embodiment of the present application, the determination unit further includes: a second acquisition module and a processing module, where the second acquisition module is configured to acquire a rotation matrix between the coordinate system of the image sensor and the coordinate system of the lidar and translation vector; the above-mentioned processing module is configured to fuse the above-mentioned image data and the above-mentioned point cloud data according to the above-mentioned rotation matrix and the above-mentioned translation vector to obtain the above-mentioned current local data. Through coordinate transformation, the coordinate system of the image sensor and the coordinate system of the lidar are placed in the same coordinate system, thereby realizing the fusion of the image data and the above point cloud data.

In an embodiment of the present application, the determining unit further includes: a first building module and a fourth determining module, where the first building module is used to build a neural network model, wherein the neural network model uses multiple sets of training data Obtained by training, each group of training data in the above-mentioned multiple groups of training data includes the historical global data, historical local data and the categories of historical objects corresponding to the above-mentioned historical global data and historical local data obtained within a historical time period; The fourth determining module is configured to determine the category of the target object corresponding to the global data and the local data according to the neural network model. The neural network algorithm can be used to accurately determine the category of the target.

In an embodiment of the present application, the determining unit further includes: a second building module and a fifth determining module, where the second building module is used to build a random forest model, wherein the random forest model uses multiple sets of training data Obtained by training, each group of training data in the above-mentioned multiple groups of training data includes the historical global data, historical local data and the categories of historical objects corresponding to the above-mentioned historical global data and historical local data obtained within a historical time period; The fifth determining module is configured to determine the category of the target object corresponding to the global data and the local data according to the random forest model. Due to the randomness of the training data selection of the random forest algorithm, the accurate determination of the category of the target object can be guaranteed.

In an embodiment of the present application, the above-mentioned second building module includes: a first training sub-module, a second training sub-module, a third training sub-module and an integration sub-module, and the above-mentioned first training sub-module is used for adopting the above-mentioned history The global data, the first data combination in the above-mentioned historical local data, and the category of the above-mentioned historical target are trained to obtain a first decision tree; the above-mentioned second training submodule is used for using the above-mentioned historical global data and the first in the above-mentioned historical local data. The two data combinations and the categories of the above-mentioned historical targets are trained to obtain a second decision tree; the above-mentioned third training sub-module is used for using the above-mentioned historical global data, the third data combination in the above-mentioned historical local data, and the above-mentioned historical target objects. The third decision tree is obtained by training; the integration sub-module is used to integrate the first decision tree, the second decision tree and the third decision tree to obtain the random forest model. That is, different decision trees are obtained by training with different parameters, and then different decision trees are fused to obtain an accurate random forest model.

The target object classification device includes a processor and a memory, the above-mentioned acquisition unit, the second acquisition unit and the determination unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions. .

The processor includes a kernel, and the kernel calls the corresponding program unit from the memory. One or more kernels can be set, and by adjusting the kernel parameters, the problem that it is difficult to accurately classify targets by using lidar in the prior art can be solved.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory, such as read only memory (ROM) or flash memory (flash RAM), the memory including at least one memory chip.

According to another aspect of the present application, there is also provided a vehicle comprising one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory, And configured to be executed by the one or more processors, the one or more programs include a method for performing the above-described object classification.

According to another aspect of the present application, there is also provided a system including a vehicle and an image sensor and a lidar mounted on the vehicle, the vehicle including a controller for executing the above-mentioned method for classifying objects.

An embodiment of the present invention provides a computer-readable storage medium, where the computer-readable storage medium includes a stored program, wherein when the program runs, a device where the computer-readable storage medium is located is controlled to perform the target object classification method.

An embodiment of the present invention provides a processor for running a program, wherein the target object classification method is executed when the program is run.

An embodiment of the present invention provides a device. The device includes a processor, a memory, and a program stored in the memory and running on the processor. When the processor executes the program, the processor implements at least the following steps: acquiring current global data, the current global data The data includes a bird's-eye view of the current target and road information on which the current target is located, the road information includes a road type, and the road type is a roadway, a sidewalk or a crosswalk; obtain the current local data of the current target , wherein the current local data is obtained by fusing image data and point cloud data, the image data is obtained by using an image sensor, and the point cloud data is obtained by using lidar; at least according to the current global data and all The current local data is used to determine the category of the current target. The devices in this article can be servers, PCs, PADs, mobile phones, and so on.

The present application also provides a computer program product that, when executed on a data processing device, is suitable for executing a program initialized with at least the following method steps: acquiring current global data, where the current global data includes a bird's-eye view of the current target and all the road information on which the current target object is located, the road information includes the road type, and the road type is a roadway, a sidewalk or a crosswalk; obtain the current local data of the current target object, wherein the current local data is a fusion The image data and point cloud data are obtained, the image data is obtained by using an image sensor, and the point cloud data is obtained by using lidar; at least according to the current global data and the current local data, determine the current Type of target.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory in the form of, for example, read only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media includes both persistent and non-permanent, removable and non-removable media, and storage of information may be implemented by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article of manufacture or apparatus that includes the element.

From the above description, it can be seen that the above-mentioned embodiments of the present application achieve the following technical effects:

1), the target object classification method of the present application, by obtaining the current global data, the current global data includes the current target object bird's-eye view and the road information where the above-mentioned current target object is located, obtain the current local data of the current target object, and then at least according to the current The global data and the above-mentioned current local data determine the type of the above-mentioned current target object. Accurate determination of the class of the current target can be achieved.

2), the target object classification device of the present application, the first acquisition unit acquires current global data, the current global data includes the current target object bird's eye view and the road information where the current target object is located, and the second acquisition unit acquires the current target object. Local data, the determining unit determines the category of the current target object at least according to the current global data and the current local data. Accurate determination of the class of the current target can be achieved.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims (16)

1. a target object classification method, is characterized in that, comprises: Acquiring current global data, where the current global data includes a bird's-eye view of the current target and road information on which the current target is located, where the road information includes a road type, and the road type is a roadway, a sidewalk or a crosswalk; Obtain the current local data of the current target, wherein the current local data is obtained by fusing image data and point cloud data, the image data is obtained by using an image sensor, and the point cloud data is obtained by using lidar ; The category of the current target is determined at least according to the current global data and the current local data. 2. The method according to claim 1, wherein determining the category of the current target object at least according to the current global data and the current local data, comprising: Determine a rough category of the current target object according to the current global data, where the rough category includes vehicles and pedestrians; According to the coarse category of the current object and the current local data of the current object, the fine category of the current object is determined, and the fine category includes a car, a bus, an ambulance and a construction vehicle. 3. The method according to claim 2, wherein determining the rough category of the current target according to the current global data, comprising: Determine the size of the projection of the current target according to the bird's-eye view of the current target; The rough category of the current target is determined according to the projected size of the current target and the road information on which the current target is located. 4. The method according to claim 2, wherein determining the fine category of the current object according to the rough category of the current object and the current local data of the current object, comprising: extracting the detailed features of the current target from the current local data of the current target; The fine category of the current object is determined according to the rough category of the current object and the detailed feature of the current object. 5. The method of claim 2, wherein The rough category of the current target is determined according to the current global data, and the rough category includes vehicles and pedestrians, including: Build a global model, wherein the global model is obtained by training using multiple sets of training data, and each set of training data in the multiple sets of training data includes acquired within a historical time period: the historical global data and the historical global data The coarse category of the historical target corresponding to the data; Inputting the current global data into the global model for operation to obtain the rough category of the current target; According to the rough category of the current target and the current local data of the current target, determine the fine category of the current target, including: Build a local model, wherein the global model is obtained by training using multiple sets of training data, and each set of training data in the multiple sets of training data includes acquired within a historical time period: historical local data, according to the global The coarse category of historical objects determined by the model, and the fine category of historical objects; The coarse category of the current target object and the current local data of the current target object are input into the local model for operation to obtain the fine category of the current target object. 6. The method according to any one of claims 1 to 4, wherein determining the category of the current target object at least according to the current global data and the current local data, comprising: obtain the temperature information of the current target; The category of the current target is determined according to the temperature information, the current global data and the current local data. 7. The method according to any one of claims 1 to 4, wherein acquiring the current local data of the current target comprises: obtaining a rotation matrix and a translation vector between the coordinate system of the image sensor and the coordinate system of the lidar; According to the rotation matrix and the translation vector, the image data and the point cloud data are fused to obtain the current local data. 8. The method according to any one of claims 1 to 4, wherein determining the category of the target object at least according to the global data and the local data, comprising: Construct a neural network model, wherein the neural network model is obtained by training using multiple sets of training data, and each set of training data in the multiple sets of training data includes acquired within a historical time period: historical global data, historical local data and categories of historical objects corresponding to the historical global data and historical local data; According to the neural network model, the category of the current target object corresponding to the global data and the local data is determined. 9. The method according to any one of claims 1 to 4, wherein determining the category of the target object at least according to the global data and the local data, comprising: Build a random forest model, wherein the random forest model is obtained by training using multiple sets of training data, and each set of training data in the multiple sets of training data includes acquired within a historical time period: historical global data, historical local data data and categories of historical objects corresponding to the historical global data and historical local data; According to the random forest model, the category of the current target object corresponding to the global data and the local data is determined. 10. The method according to claim 9, wherein constructing the random forest model comprises: Using the historical global data, the first data combination in the historical local data, and the category of the historical target to train to obtain a first decision tree; Using the historical global data, the second data combination in the historical local data, and the category of the historical target to train to obtain a second decision tree; Using the historical global data, the third data combination in the historical local data, and the category of the historical target, a third decision tree is obtained by training; The first decision tree, the second decision tree and the third decision tree are integrated to obtain the random forest model. 11. A device for classifying objects, comprising: The first acquisition unit is used to acquire current global data, where the current global data includes a bird's-eye view of the current target and road information on which the current target is located, where the road information includes a road type, and the road type is a car dealership road, sidewalk or crosswalk; The second acquiring unit is configured to acquire the current local data of the current target, wherein the current local data is obtained by fusing image data and point cloud data, the image data is acquired by using an image sensor, and the point cloud The data is acquired using lidar; A determining unit, configured to determine the category of the current target object at least according to the current global data and the current local data. 12. The apparatus according to claim 11, wherein the determining unit comprises: a first determining module, configured to determine a rough category of the current target object according to the current global data, where the rough category includes vehicles and pedestrians; The second determination module is configured to determine the fine category of the current target according to the rough category of the current target and the current local data of the current target, and the fine category includes cars, buses, and ambulances and construction vehicles. 13. A vehicle comprising one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the One or more processors execute, the one or more programs comprising for performing the method of any one of claims 1 to 10. 14. A system comprising a vehicle and an image sensor and a lidar mounted on the vehicle, the vehicle comprising a controller for performing the method of any one of claims 1 to 10 . 15. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored program, wherein, when the program is run, a device where the computer-readable storage medium is located is controlled to execute claims 1 to 10 any of the methods described in . 16. A processor, characterized in that the processor is configured to run a program, wherein the method of any one of claims 1 to 10 is executed when the program is run.

Images