CN111316286A - Trajectory prediction method and device, storage medium, driving system and vehicle - Google Patents

Abstract

Provided are a trajectory prediction method and device, a storage medium, a driving system and a vehicle. The method comprises the steps of obtaining global semantic data and global track data of an area where an object to be predicted is located (S202), then fusing the global semantic data and the global track data to obtain global fusion data (S204), extracting features in the global fusion data to obtain global features (S206), and further processing the global features by using a trained track prediction model to obtain a target track of the object to be predicted (S208). The method can realize the track prediction of the moving object by combining the global data, has higher prediction accuracy and reduces the occurrence probability of accidents to a certain extent.

Description

Trajectory prediction method and device, storage medium, driving system and vehicle

technical field

The invention relates to the technical field of intelligent transportation, and in particular, to a trajectory prediction method and device, a storage medium, a driving system and a vehicle.

Background technique

With the development of the field of intelligent transportation, the prediction algorithm for the trajectory of moving objects is of great significance in the field of path planning. By predicting the motion trajectory of the moving object, the path planning can be carried out under the condition that the possible motion trajectory of the moving object in the future is known, which is beneficial to prevent the occurrence of unexpected situations such as collision.

The current trajectory prediction algorithm is generally based on the motion data of the moving object itself, determines the motion model suitable for the moving object according to the category of the moving object, and uses the motion model to process the motion data of the moving object itself, and then uses the post-processing method. By integrating regional semantic information, the motion trajectory of the object to be predicted can be predicted.

The existing trajectory prediction algorithm is based on the motion data of the moving object itself, and cannot predict the trajectory from a global perspective, which easily leads to the intersection of the predicted trajectories of different moving objects, which in turn leads to collisions in the path planning or scheduling based on this. Accidents present a greater safety hazard.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a trajectory prediction method and device, a storage medium, a driving system, and a vehicle, which can realize trajectory prediction of moving objects in combination with global data, have a high prediction accuracy, and reduce accidents to a certain extent. probability of occurrence.

In a first aspect, an embodiment of the present invention provides a trajectory prediction method, including:

Obtain the global semantic data and global trajectory data of the region where the object to be predicted is located;

fusing the global semantic data and the global trajectory data to obtain global fusion data;

Extracting features in the global fusion data to obtain global features;

The global feature is processed by using the trained trajectory prediction model to obtain the target trajectory of the object to be predicted.

In a second aspect, an embodiment of the present invention provides a trajectory prediction device, including:

The acquisition module is used to acquire the global semantic data and global trajectory data of the region where the object to be predicted is located;

a fusion module, configured to fuse the global semantic data and the global trajectory data to obtain global fusion data;

a feature extraction module for extracting features in the global fusion data to obtain global features;

The prediction module is used to process the global feature by using the trained trajectory prediction model to obtain the target trajectory of the object to be predicted.

In a third aspect, an embodiment of the present invention provides a trajectory prediction device, including:

memory;

processor; and

Computer program;

Wherein, the computer program is stored in the memory and configured to be executed by the processor to implement the method of the first aspect.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored,

The computer program is executed by a processor to implement the method as described in the first aspect.

In a fifth aspect, an embodiment of the present invention provides a driving system, including:

a trajectory prediction device for performing the method according to the first aspect;

The motion controller is used to control the movement of the controlled object according to the target trajectory.

In a possible design, the controlled object and the to-be-predicted object are different objects.

In a sixth aspect, an embodiment of the present invention provides a vehicle, including:

The trajectory prediction apparatus according to the second aspect or the third aspect is used to perform the method according to the first aspect.

In a seventh aspect, an embodiment of the present invention provides a vehicle, including:

The driving system of the fifth aspect.

In an eighth aspect, an embodiment of the present invention provides a control device for an unmanned aerial vehicle, including:

The driving system of the fifth aspect.

According to the technical solution provided by the embodiment of the present invention, by acquiring and processing the global semantic data and the global trajectory data of the region where the object to be predicted is located, the global features of the region can be obtained, and then the trained trajectory prediction model is used to perform the global feature analysis. After processing, the target trajectory of the object to be predicted can be obtained. In other words, the technical solution provided by the embodiment of the present invention starts from global semantics and global trajectory. When predicting the trajectory of an object to be predicted, all moving objects in the area are considered. Combined with the global semantic data of the area, to realize the trajectory prediction of any moving object in the area, compared with the prediction method that only considers a single object to be predicted, this scheme has a higher prediction accuracy, and when Performing subsequent path planning or scheduling based on this can also reduce the probability of accidents to a certain extent and provide higher safety.

Description of drawings

FIG. 1 is a schematic top view of a trajectory prediction scene according to an embodiment of the present invention;

2 is a schematic flowchart of a trajectory prediction method provided by an embodiment of the present invention;

3 is a schematic flowchart of another trajectory prediction method provided by an embodiment of the present invention;

4 is a schematic flowchart of another trajectory prediction method provided by an embodiment of the present invention;

5 is a schematic diagram of a cyclic unit structure of a cyclic neural network model provided by an embodiment of the present invention;

6 is a schematic diagram of a model architecture of a recurrent neural network model provided by an embodiment of the present invention;

7 is a schematic diagram of a model architecture of a long short-term memory network model according to an embodiment of the present invention;

8 is a schematic flowchart of another trajectory prediction method provided by an embodiment of the present invention;

9 is a functional block diagram of a trajectory prediction apparatus provided by an embodiment of the present invention;

10 is a schematic diagram of a physical structure of a trajectory prediction apparatus provided by an embodiment of the present invention;

FIG. 11 is a schematic structural diagram of a driving system according to an embodiment of the present invention;

12 is a schematic structural diagram of a vehicle according to an embodiment of the present invention;

FIG. 13 is a schematic structural diagram of another vehicle according to an embodiment of the present invention.

The above-mentioned drawings have shown clear embodiments of the present disclosure, and will be described in more detail hereinafter. These drawings and written descriptions are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the disclosed concepts to those skilled in the art by referring to specific embodiments.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

First, the terms involved in the present invention are explained:

Moving object: refers to a creature or object that can achieve trajectory movement. The moving objects involved in the embodiments of the present invention may include, but are not limited to, at least one of vehicles, animals, people, robots, and unmanned aerial vehicles. Wherein, the vehicle may be an unmanned vehicle, such as an unmanned ground vehicle (Unmanned Ground Vehicle, UGV), or a private vehicle or a public transportation vehicle in an automatic driving mode.

Object to be predicted: One or more moving objects of the target trajectory to be predicted. Among them, a plurality of refers to two or more than two, and this concept will be involved in the following, and will not be repeated here.

Semantic objects: refer to the objects on each semantic concept in the area. Please refer to the scene shown in Figure 1. The semantic objects in this scene include: vehicles, lane lines, and lanes. It can be seen that FIG. 1 is only used as an example. In an actual trajectory prediction scene, the semantic categories of semantic objects also include many kinds, such as: trees, obstacles, railings, signs, people, animals, etc. The semantic categories of semantic objects are not particularly limited.

The Long Short Term Memory (LSTM) model is a variant of the Recurrent Neural Network (RNN) model. Compared with the RNN model, LSTM has longer time-dependent modeling capabilities.

A specific application scenario of the technical solution provided by the embodiment of the present invention is: a trajectory prediction scenario for a moving object.

Further, the technical solutions provided by the embodiments of the present invention can also be specifically applied to a path planning scenario. In this case, the path planning for one or more moving objects can be implemented according to the predicted trajectory.

In addition, the technical solutions provided by the embodiments of the present invention can also be specifically applied to vehicle scheduling scenarios. For example, dispatchable vehicles can be dispatched by predicting the trajectory of other non-dispatchable vehicles or objects.

As described in the background art, the existing trajectory prediction method is only for a single moving object. After the category of the moving object is determined, the motion data of the moving object itself is processed through the motion model corresponding to the category. Thus, the motion trajectory of the moving object is predicted. On the one hand, this prediction method is limited by the object category, and it is necessary to accurately determine the category of the moving object in order to obtain more accurate prediction results with the motion model corresponding to the category; on the other hand, this prediction method only depends on The motion data of the moving object itself is not comprehensively analyzed from a global perspective combined with the current moving environment and the motion of other moving objects. This kind of objects that do not consider other moving or non-moving objects in the area where the moving object is located is very likely to be predicted. The trajectories of two moving objects of the same category intersect, so if path planning or object scheduling is performed based on this, accidents such as collisions are very likely to occur, and there is a greater safety risk.

Based on this, the technical solutions provided by the embodiments of the present invention aim to solve the above technical problems in the prior art, and propose the following solution idea: comprehensively consider the global data of the region where the object to be predicted is located, including global semantic data and global trajectory data, according to This obtains the global feature, and uses the global feature as the input of the trajectory prediction model to obtain the target trajectory of the object to be predicted.

Based on this design, the trajectory prediction method provided by the embodiments of the present invention may be specifically executed in a built-in processor of a moving object or a terminal device held by it, or may be specifically executed in a cloud or a background server.

for example. In a possible scenario, the first processor of the self-driving vehicle can plan the driving route by itself, and the second processor of the self-driving vehicle is used to execute the trajectory prediction method provided by this solution, and is used to predict the trajectory of the predicted trajectory. Input to the first internal processor, so that the first processor can perform subsequent path planning according to the predicted trajectory. The first processor and the second processor may be the same processor, or may be different processors, for example, may be one or two processors in an Advanced Driving Assistant System (ADAS) and, the first processor and the second processor may be part of the overall controller of the vehicle, or may also be a background overall server or a cloud server that controls the driving of the unmanned vehicle.

The technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems will be described in detail below with specific examples. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.

Example 1

The embodiment of the present invention provides a trajectory prediction method. Please refer to FIG. 2 to FIG. 4 , wherein FIG. 2 shows a schematic flowchart of a trajectory prediction method provided by an embodiment of the present invention, and FIG. 3 shows a specific embodiment of the trajectory prediction method provided by the present invention. A schematic diagram of an implementation flow in an application scenario, FIG. 4 is a specific implementation manner of the flow shown in FIG. 3 .

As shown in Figure 2, the method includes the following steps:

S202: Obtain global semantic data and global trajectory data of the region where the object to be predicted is located.

As mentioned above, the object to be predicted is a moving object, which may include, but is not limited to, at least one of the following: a vehicle, an animal, a human, a robot, and an unmanned aerial vehicle. In addition, in this embodiment of the present invention, the number of objects to be predicted is not particularly limited, and may be one or more.

The global semantic data is used to describe the semantic category of each object in the region where the object to be predicted is located. In this case, each moving object or non-moving object is regarded as a semantic object and has its own semantic category. The global trajectory data is used to describe the historical motion coordinates of each moving object in this area. It can be known that, in specific implementation, the global trajectory data may include historical motion coordinates of at least one frame.

FIG. 3 shows a schematic diagram of the implementation flow of this solution when the scenario shown in FIG. 1 is taken as an example. As shown in Figure 3, the driving scene contains three semantics, namely: vehicle, lane and lane line, at this time, the global semantic data that needs to be acquired in the scene, that is, the semantic data of each semantic object; and , it is also necessary to obtain the global trajectory data of the scene, that is, the trajectory data of each moving object: vehicle 1 (as the object to be predicted), vehicle 2 and vehicle 3 .

It should be noted that FIG. 3 is only schematic, and in a specific implementation scenario, the representations of the global trajectory data and the global semantic data are not limited to a single acquisition method, but can be acquired as overall data.

In an implementation scenario as shown in FIG. 4 , the trajectory data of each moving object can be realized by the LSTM model, which will be described in detail later.

S204, fuse the global semantic data and the global trajectory data to obtain global fusion data.

Specifically, as shown in FIG. 3 , the fusion module is used to fuse the global semantic data corresponding to each frame with the global trajectory data to obtain the global fusion data of each frame.

S206, extracting features in the global fusion data to obtain global features.

As shown in Figure 3, the feature extraction module is used to extract global features from the global fusion data. In an implementation scenario as shown in FIG. 4 , this step may be implemented by a Convolutional Neural Networks (CNN) model, which will be described in detail later.

S208, using the trained trajectory prediction model to process the global feature to obtain the target trajectory of the object to be predicted.

The present invention uses the trained trajectory prediction model to predict the motion trajectory of the moving object. At this time, the input of the trajectory prediction model is the global feature, and the output is the motion trajectory of the moving object.

Specifically, the trajectory prediction model provided in this embodiment of the present invention may include, but is not limited to, at least one of the following: an LSTM model, a multi-layer perceptron (Multi-Layer Perception, MLP); wherein the multi-layer perceptron includes : RNN model, gated recurrent neural network (GRU) model. For example, in the implementation scenario shown in Fig. 4, the trajectory prediction is realized by the LSTM model, at this time, Y=LSTM(feature), where Y represents the target trajectory, and feature represents the global feature.

Please refer to Figures 5-7, wherein Figure 5 shows the design logic of a recurrent unit in the RNN model, Figure 6 shows a schematic diagram of the model architecture of the RNN model, and Figure 7 shows a schematic diagram of the model architecture of the LSTM model.

As an effective means of time series modeling, RNN model, compared with ordinary neural network, as shown in Figure 5, the main difference is that the output of the previous frame or the intermediate state is used as the input of the current frame to realize the historical message and integration of temporal relationships. The RNN model shown in Figure 6 can be obtained after the cyclic unit shown in Figure 5 is expanded in time. As shown in Figure 6, the RNN model can realize time series modeling. Therefore, the trajectory prediction step in this scheme can be implemented through the RNN model.

As shown in Figure 7, each repeating module in the LSTM model contains 4 interaction layers, and these 4 interaction layers interact in a special way, so that the output of the previous frame or the intermediate state is used as the input of the current frame, Therefore, compared to the RNN model shown in Figure 6, the LSTM model has better time-dependent modeling capabilities. In addition, considering that the prediction of the trajectory has a strong temporal correlation with the historical data of the moving object, using the LSTM model to realize the trajectory prediction can obtain the trajectory prediction result that is closer to the actual development.

Through the foregoing design, the technical solution provided by the embodiments of the present invention can realize trajectory prediction for any object to be predicted based on global data. Compared with the trajectory prediction method based only on the motion data of the object to be predicted, this solution It has a high prediction accuracy, and when the subsequent path planning or scheduling is performed based on this, it can also reduce the probability of accidents to a certain extent and have higher safety.

Hereinafter, the implementation of the method shown in FIG. 2 will be described in detail.

S202 includes two aspects of global data acquisition: global semantic data and global trajectory data. For the acquisition methods of these two global data, refer to the flow shown in FIG. 8 .

On the one hand, as shown in FIG. 8 , the way of acquiring global semantic data may include the following steps:

S202-12: Acquire a global area image of the area where the object to be predicted is located.

The global region image can be an image obtained in real time, and the image obtained in real time has higher timeliness, and the obtained global semantic data is also more accurate. Specifically, the real-time acquisition method may include, but is not limited to: real-time acquisition of images by an image acquisition device. Wherein, the image acquisition device may be a part of the trajectory prediction apparatus (execution subject of the trajectory prediction method); or, it may have real-time data interaction with the trajectory prediction apparatus. For example, if the trajectory prediction device can be the processor A in the main controller of the vehicle, the image acquisition device can be a camera in the black box of the vehicle, which can directly input the acquired image to the processor A; The device can be a camera set in an area, such as a camera set on a road or a roadside. At this time, the processor A can request and receive the global area image in real time from the camera in the area or the background server of the camera in the area.

In addition to real-time acquisition, global semantic data can also be acquired by invoking the collected data. Specifically, in one implementation scenario, a global area image of the area in the high-precision map can be acquired; in another implementation scenario, a global area image of the area that has been collected in other processors or memory can be acquired. Moreover, this implementation method can only obtain the environmental information of the area, such as non-real-time data of non-moving objects such as roads, signs, lane lines, etc., but cannot obtain the movement of moving objects in the real-time scene. Therefore, in the When the solution is implemented in this way, it is only suitable for trajectory prediction of a single object to be predicted, and cannot be combined with the trajectories of other moving objects to achieve comprehensive prediction, and the prediction accuracy is weak.

It should be noted that, in this embodiment of the present invention, the image to be subsequently processed is an overhead image. Therefore, if the acquired image is not an overhead image in the foregoing implementation manner, an overhead projection of the collected image is also required to satisfy the subsequent processing requirements. The desired overhead image.

In a possible design, the overhead image involved in the embodiment of the present invention may be embodied as: a digital orthophoto map (Digital Orthophoto Map, DOM) image.

In addition, in order to facilitate processing, the shape or size of the region of the "region where the object to be predicted is located" may be further personalized, which is not particularly limited in this embodiment of the present invention. Specifically, taking the object to be predicted as the center, a rectangular area with a certain length and width in the top view can be obtained as the area where it is located. For example, a rectangular area with a length W and a width H as shown in FIG. 1 (or FIG. 3 ) can be obtained Image. For another example, the entire road where the object to be predicted is located may also be used as the region where the object to be predicted is located.

S202-14: Perform semantic recognition on each pixel in the global area image to obtain a semantic category of each pixel.

In a possible design, the semantic recognition of each pixel can be achieved through deep learning. If it is implemented in this way, it is necessary to use the preset pixel sample data to perform deep learning on the pixel semantic recognition model before performing this step, so as to obtain a pixel semantic recognition model that meets the application requirements (which can be achieved by defining the loss function). . In this way, when performing this step, it is only necessary to input the global region image into the pixel semantic recognition model, and the output of the pixel semantic recognition model is the semantic category of each pixel.

In another possible design, the pixel value of each pixel can also be compared with the pixel interval corresponding to each semantic category based on the pixel value of each pixel, so that for any pixel value, the pixel value falls within The semantic category corresponding to the pixel interval of , as the semantic category corresponding to the pixel. The correspondence between each semantic category and the pixel interval can be preset in a custom way.

S202-16: Perform semantic annotation on the global area image according to the semantic category of each pixel to obtain the global semantic information.

Since the semantic category of each pixel has been determined, when performing this step, the global area image can be semantically segmented according to the semantic category of each pixel to obtain a plurality of semantic objects; thus, semantic annotation is performed on each semantic object. to obtain the global semantic information.

Among them, semantic annotation is only used to distinguish the semantic category of each object, and can be labeled in any distinguishable manner. For example, each semantic object may be identified by a different color. Alternatively, as shown in FIG. 1 (or FIG. 3 ), each semantic object may be identified by different shading. It can be known that, after being identified, the semantic objects with the same identification are of the same class of semantic objects.

At this time, it should be noted that, in different implementation scenarios, the labeling method of other moving objects of the same category as the object to be predicted may be the same as or different from the labeling method of the object to be predicted. As shown in FIG. 1 , when the object to be predicted is vehicle 1 , the area where the object to be predicted is located also includes moving objects of the same category: vehicle 2 and vehicle 3 . At this time, if step S208 is implemented through the first trajectory prediction model (the first trajectory prediction model is used to predict the target trajectory of the object to be predicted, the implementation will be described in detail later), as shown in FIG. Distinguish identification with vehicle 2 and vehicle 3, vehicle 1 has one kind of identification, and vehicle 2 and vehicle 3 are another kind of identification. Or, if step S208 is implemented through a second trajectory prediction model (the second trajectory prediction model is used to predict the motion trajectories of all moving objects in the area, and the implementation will be described in detail later), then there is no need to perform To distinguish identification, vehicle 1 , vehicle 2 and vehicle 3 can be identified using the same identification method (the same identification method is not shown in FIG. 1 ).

And, when performing semantic annotation, the grid can be further customized, and each grid can include one or more pixels, and the dividing method can be realized by a preset resolution. For example, the global area image shown in FIG. 1 can be divided into square grids with a length of 20 cm. In this way, when performing subsequent semantic annotations, only the grids need to be labeled. When the grid contains multiple pixels, the implementation of the grid is beneficial to reduce the amount of marking and improve the processing efficiency.

In addition to the foregoing implementation manner, steps S202-14 and S202-16 described in FIG. 8 can also be implemented by a neural network model. That is, before executing S202-14, the semantic recognition model is trained, so that the global region image obtained in step S202-12 is input into the semantic recognition model, and the output of the semantic recognition model is the global region image marked with the semantic category. , and the global semantic data is obtained.

There are no special restrictions on the categories of the aforementioned semantic recognition models and pixel semantic recognition models, which can be realized by using the CNN model or other neural network models, and the sample data of the two need to be marked differently according to the input and output of the model. Design, no more details.

On the other hand, as shown in FIG. 8 , the method of acquiring global trajectory data may include the following steps:

S202-22: Acquire a track point set of each moving object in the region where the object to be predicted is located, where the track point set is formed by collecting coordinate points of the moving objects in a time sequence order.

This step is used to obtain the track point set of each moving object in the current area, wherein each track point set is composed of a plurality of coordinate points of the moving object. In order to facilitate processing, the coordinate points of each moving object can be converted into the same A coordinate point in a coordinate system. For example, the coordinate points in each track point set can be converted into coordinate points in the rectangular coordinate system formed by the two right-angled sides of the rectangular area shown in FIG. 1, and the expression form of each coordinate point is (X, Y), The trajectory point set of each moving object can be represented as {(X _i , Y _i )}, where i is used to represent the time sequence order of the coordinate points.

Specifically, during specific implementation, the aforementioned coordinate point set can be formed by acquiring the coordinate points of each moving object within a time interval whose end point is the current moment. The length of the time interval can be preset as required, for example, the track point set of each moving object within 3s before the current moment can be acquired.

It should be noted that, in this step, the trajectory point set may be actively monitored by the executive body, or may be obtained by requesting data from other processors or collection devices. For example, if the execution subject is the processor A in the main controller of the vehicle 1, the coordinate points of the vehicle 1 can be collected by its own locator, such as GPS, and the coordinate data collected by the locator of the vehicle 1 can be obtained. Send it to the processor A, and the processor A will perform coordinate transformation to obtain the trajectory point set of the vehicle 1; and the trajectory point set of other vehicles can be obtained by requesting other processors, for example, if there is communication with other vehicles , the track point set of other vehicles can be obtained from other vehicles; for example, the track point set of other vehicles can be obtained from the road monitor in the area; in addition, the image of other vehicles can also be collected by itself and the distance from itself can be calculated. , calculate and obtain the set of trajectory points of other vehicles. There may be various implementation manners, which will not be repeated here.

In addition, the data source (or direct acquisition source) of the track point set may be different from the data source (or direct acquisition source) of the global area image in the foregoing S202-12.

S202-24: Perform coding processing on the trajectory point set of each moving object to obtain the trajectory feature of each moving object.

This step can also be implemented by a neural network model, inputting the trajectory point set of each moving object obtained in the aforementioned S202-22 (for example, the trajectory point set of each moving object within 3s) into the coding model (a trained neural network The model, for example, the LSTM model shown in FIG. 4 can be used), and the output of the encoding model is the trajectory feature (encoder) of each moving object.

Specifically, for any moving object, its trajectory feature can be expressed as: encoder=LSTM{(X _i , Y _i )}. Among them, the length of the trajectory feature (encoder) can be assumed to be C, and the value of C is generally a preset empirical value.

S202-26, construct a trajectory tensor according to the trajectory feature of each moving object, as the global trajectory data.

Based on the trajectory features of each moving object obtained in step S202-24, this step can construct a trajectory tensor (tensor) whose size is C*H*W, and store the trajectory features (encoder) of each moving object in the corresponding in the tensor. Specifically, for any moving object, the encoder of the moving object is correspondingly stored in the center position of the moving object. In a possible design, the grid can also be divided in the tensor according to the method shown in Figure 1. In this way, which grid in the Tensor the center position of the moving object is located in, the encoder of the moving object is stored correspondingly. in this grid.

Through the implementation as shown in FIG. 8 , the acquisition of global semantic data and global trajectory data can be achieved. In the aforementioned implementation manner, the global semantic data can be represented as a W*H image, and the global trajectory data can be represented as a tensor with a size of C*H*W. Therefore, the fusion step described in S204 is performed. , the two can be fused into a fused tensor of size (C+1)*H*W. The fusion tensor can be specifically expressed as: tensor((C+1)*H*W).

Based on the global fusion data obtained in the preceding steps, it is only necessary to perform feature extraction on the global fusion data to obtain global features including the object to be predicted. In specific implementation, it can also be implemented through a neural network model. That is, the fusion information is processed by using the trained feature extraction model to obtain the global feature. The feature extraction model involved in the embodiment of the present invention at least includes: a convolutional neural network (Convolutional Neural Networks, CNN) model as shown in FIG. 4 . Similar to the aforementioned method of implementing data processing using a neural network model, it is necessary to use feature extraction samples to train the CNN model before performing this step. The model training process is not repeated here.

Similarly, before executing S208, the training and learning for the trajectory prediction model also needs to be completed. In a specific implementation scenario, the training of the trajectory prediction model (and the neural network model involved in the foregoing implementations) is generally completed before the implementation of this solution, so as to realize the trajectory prediction efficiently in real time. This implementation has High real-time performance is beneficial to realize trajectory prediction in real time, and then realize path planning or scheduling in approximately real time.

When specifically implementing the training of the trajectory prediction model, a first trajectory prediction model for predicting a single moving object (object to be predicted) according to the global feature can be trained. For the object to be predicted, this single prediction method has faster processing efficiency, which is beneficial to path planning and scheduling in real-time scenarios.

Alternatively, it is also possible to train a second trajectory prediction model that predicts all moving objects contained in the region according to the global feature. Wherein, when using the second trajectory prediction model to process the global feature, the global feature is input into the second trajectory prediction model, and the motion trajectories of all moving objects in the region output by the second trajectory prediction model are obtained, and The motion trajectory of the object to be predicted among all the moving objects may be used as the target trajectory. This global prediction method can output the motion trajectories of all moving objects in the area at one time, which is beneficial to the realization of global scheduling, as well as reducing the probability of accidents during the scheduling or path planning process, and improving safety.

In addition, a third trajectory prediction model that predicts multiple (but not all) moving objects included in the region according to the global feature can also be trained. The model training method and the implementation method of S208 are the same as above, and will not be repeated here.

To sum up, based on the different designs of the trained trajectory prediction models, the technical solutions provided by the embodiments of the present invention can not only realize trajectory prediction for a single moving object, but also realize trajectory prediction for multiple moving objects. And the above trajectory prediction model has no dependence on the object category. Through the above trained trajectory prediction model, the trajectory prediction of any category of moving objects can be realized, which has higher flexibility and can also be applied to objects with multiple categories. trajectory prediction in the scene.

In addition, in some special implementation scenarios, it is also possible to train respective trajectory prediction models for various types of moving objects, respectively, as in the existing implementation manner. That is, the technical solutions provided by the embodiments of the present invention can also implement personalized prediction for various types of moving objects. Wherein, when training respective trajectory prediction models for various types of moving objects, the sample data is the relevant data of the objects of this type.

As mentioned above, after the target trajectory of the object to be predicted is obtained through the foregoing implementation manners, the target trajectory can be used for further processing.

In a possible design, motion planning may be performed for the object to be predicted according to the target trajectory. That is, further path planning is implemented according to the predicted trajectory of the object to be predicted.

In another possible design, motion planning may be performed for other moving objects according to the target trajectory. That is, in the process of path planning for one or more other moving objects, a route can be planned according to the predicted trajectory of the object to be predicted, so as to avoid collision or other safety accidents with the object to be predicted.

Furthermore, according to the planned motion path, the scheduling of the moving objects is realized.

It can be understood that, some or all of the steps or operations in the foregoing embodiments are only examples, and other operations or variations of various operations may also be performed in the embodiments of the present application. Furthermore, the various steps may be performed in a different order presented in the above-described embodiments, and may not perform all operations in the above-described embodiments.

Embodiment 2

Based on the trajectory prediction method provided in the above-mentioned first embodiment, the embodiment of the present invention further provides an embodiment of an apparatus for implementing the steps and methods in the above-mentioned method embodiment.

An embodiment of the present invention provides a trajectory prediction apparatus, please refer to FIG. 9 , the trajectory prediction apparatus 600 includes:

an acquisition module 61, configured to acquire global semantic data and global trajectory data of the region where the object to be predicted is located;

A fusion module 62, configured to fuse the global semantic data and the global trajectory data to obtain global fusion data;

Feature extraction module 63, for extracting features in the global fusion data to obtain global features;

The prediction module 64 is configured to process the global feature by using the trained trajectory prediction model to obtain the target trajectory of the object to be predicted.

In a possible design, the acquisition module 61 is specifically used for:

obtaining a global area image of the area where the object to be predicted is located;

Semantic recognition is performed on each pixel in the global area image, respectively, to obtain the semantic category of each pixel;

According to the semantic category of each pixel, semantic annotation is performed on the global area image to obtain the global semantic information.

Wherein, the acquisition module 61 is further specifically used for:

According to the semantic category of each pixel, semantic segmentation is performed on the global area image to obtain a plurality of semantic objects;

Semantic annotation is performed on each semantic object to obtain the global semantic information.

The global area image involved in the embodiment of the present invention is a digital orthophoto DOM image.

In another possible design, the obtaining module 61 is specifically used for:

acquiring the trajectory point set of each moving object in the area where the object to be predicted is located, and the trajectory point set is formed by the coordinate points of the moving object assembled in time sequence;

Encoding the set of trajectory points of each moving object to obtain the trajectory features of each moving object;

According to the trajectory features of each moving object, a trajectory tensor is constructed to serve as the global trajectory data.

In a possible design, the fusion module 63 is specifically used for:

The fusion information is processed by using the trained feature extraction model to obtain the global feature.

Wherein, the feature extraction model involved in the embodiment of the present invention includes at least: a convolutional neural network CNN model.

The trajectory prediction model involved in the embodiment of the present invention includes at least one of the following: a long short-term memory network LSTM model, and a multi-layer perceptron MLP;

Wherein, the multilayer perceptron includes: a recurrent neural network RNN model and a gated recurrent unit GRU model.

In a possible design, the trajectory prediction model is a first trajectory prediction model, and the first trajectory prediction model is used to predict the target trajectory of the object to be predicted.

In another possible design, the trajectory prediction model is a second trajectory prediction model, and the second trajectory prediction model is used to predict the motion trajectories of all moving objects in the area; in this case, the prediction module 64 specifically uses At:

Input the global feature into the second trajectory prediction model, obtain the motion trajectories of all moving objects in the region output by the second trajectory prediction model, and use the The motion trajectory is used as the target trajectory.

In the embodiment of the present invention, the object to be predicted includes at least one of the following: a vehicle, an animal, a person, a robot, and an unmanned aerial vehicle.

In addition, in one or a possible design, the trajectory prediction apparatus 600 may further include:

A planning module (not shown in FIG. 9 ), configured to perform motion planning for the object to be predicted according to the target trajectory.

The trajectory prediction apparatus 600 in the embodiment shown in FIG. 9 can be used to implement the technical solutions of the above method embodiments. For the implementation principle and technical effect, reference may be made to the relevant descriptions in the method embodiments. Optionally, the trajectory prediction apparatus 600 may be Terminal devices or background servers, etc.

It should be understood that the division of each module of the trajectory prediction apparatus 600 shown in FIG. 9 above is only a division of logical functions, and may be fully or partially integrated into a physical entity in actual implementation, or may be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also all be implemented in hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in hardware. For example, the prediction module 64 can be a separately established processing element, or can be integrated in the trajectory prediction device 600, such as a certain chip of the terminal, and can also be stored in the memory of the trajectory prediction device 600 in the form of a program, The functions of the above modules are called and executed by a certain processing element of the trajectory prediction apparatus 600 . The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, and can also be implemented independently. The processing element described here may be an integrated circuit with signal processing capability. In the implementation process, each step of the above-mentioned method or each of the above-mentioned modules can be completed by an integrated logic circuit of hardware in the processor element or an instruction in the form of software.

For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more specific integrated circuits (Application Specific Integrated Circuit, ASIC), or one or more microprocessors (digital) singnal processor, DSP), or, one or more Field Programmable Gate Arrays (Field Programmable Gate Array, FPGA), etc. For another example, when one of the above modules is implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (Central Processing Unit, CPU) or other processors that can invoke programs. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

Moreover, an embodiment of the present invention provides a trajectory prediction apparatus, please refer to FIG. 10 , the trajectory prediction apparatus 600 includes:

memory 610;

processor 620; and

Computer program;

The computer program is stored in the memory 610 and is configured to be executed by the processor 620 to implement the methods described in the above embodiments.

The number of processors 620 in the trajectory prediction apparatus 600 may be one or more, and the processors 620 may also be referred to as processing units, which may implement certain control functions. The processor 620 may be a general-purpose processor or a special-purpose processor, or the like. In an optional design, the processor 620 may also store instructions, and the instructions may be executed by the processor 620, so that the trajectory prediction apparatus 600 executes the trajectory prediction method described in the above method embodiments.

In yet another possible design, the trajectory prediction apparatus 600 may include a circuit, and the circuit may implement the function of sending or receiving or communicating in the foregoing method embodiments.

Optionally, the number of memories 610 in the trajectory prediction apparatus 600 may be one or more, and instructions or intermediate data are stored in the memory 610, and the instructions can be executed on the processor 620 to make the trajectory The prediction apparatus 600 executes the methods described in the above method embodiments. Optionally, other related data may also be stored in the memory 610 . Optionally, processor 620 may also store instructions and/or data. The processor 620 and the memory 610 may be provided separately or integrated together.

In addition, as shown in FIG. 10 , a transceiver 630 is also provided in the trajectory prediction apparatus 600, wherein the transceiver 630 may be called a transceiver unit, a transceiver, a transceiver circuit, or a transceiver, etc., and is used for testing and testing. The device or other terminal device performs data transmission or communication, which is not repeated here.

As shown in FIG. 10 , the memory 610 , the processor 620 and the transceiver 630 are connected and communicated through a bus.

If the trajectory prediction apparatus 600 is used to implement the method corresponding to FIG. 2 , for example, the transceiver 630 can acquire global semantic data and global trajectory data. The processor 620 is used to complete corresponding determination or control operations, and optionally, corresponding instructions may also be stored in the memory 610 . For the specific processing manner of each component, reference may be made to the relevant descriptions of the foregoing embodiments.

In addition, an embodiment of the present invention provides a readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the method described in the first embodiment.

In addition, an embodiment of the present invention provides a driving system, please refer to FIG. 11 , the driving system 800 includes:

A trajectory prediction apparatus 600, configured to execute the method described in any implementation manner of Embodiment 1;

The motion controller 810 is configured to control the movement of the controlled object according to the target trajectory obtained by the trajectory prediction device.

The controlled object and the to-be-predicted object are the same object. For example, in a possible design, in the scenario of the vehicle's own trajectory prediction and route planning, the motion controller can realize the planning of its own driving route according to its own target trajectory predicted by the trajectory prediction device 600; and, further Therefore, the automatic movement of the controlled object can be realized, that is, automatic driving can be realized.

In addition, the controlled object and the to-be-predicted object may be different objects. Taking the aforementioned scenario as an example, the vehicle can predict the trajectory of other vehicles on the road that are closer to itself, so that when executing its own motion control, it can try to avoid contact with other vehicles or moving obstacles (vehicles, people, animals, etc.) In order to reduce the probability of safety accidents, it is beneficial to improve safety.

In another specific implementation scenario, the motion controller 810 can also output the target trajectory obtained by the trajectory prediction device 600, so that the user can use the target trajectory as a reference when driving or controlling the movement of the controlled object. Especially in the scene of multiple objects, it is more beneficial to improve the control security in the multi-object scene by predicting the target trajectory of other objects to be predicted.

Specifically, the controlled objects may include, but are not limited to, at least one of the following: vehicles, animals, people, robots, and unmanned aerial vehicles. In addition, in this embodiment of the present invention, the number of controlled objects is not particularly limited, and may be one or more. For example, the motion controller may be a driving controller of a vehicle, or a flight controller of an unmanned aerial vehicle, etc., which will not be described again.

In addition, embodiments of the present invention provide a vehicle.

Referring to Figure 12, the vehicle 900 includes:

The trajectory prediction apparatus 600 is configured to execute the method described in any one of the implementation manners of the first embodiment.

Alternatively, as shown in FIG. 13, the vehicle 900 includes:

Driving system 800 as shown in FIG. 11 .

In addition, an embodiment of the present invention also provides a control device for an unmanned aerial vehicle.

In a possible design, the control device of the unmanned aerial vehicle includes:

The trajectory prediction apparatus 600 is configured to execute the method described in any one of the implementation manners of the first embodiment.

In another design, the control device of the unmanned aerial vehicle includes:

Driving system 800 .

Specifically, the unmanned aerial vehicle and the control device of the unmanned aerial vehicle can be designed independently, or can be designed in combination (the control device is arranged inside the unmanned aerial vehicle), which is not particularly limited in the embodiment of the present invention.

It can be seen that the control device of the vehicle and the unmanned aerial vehicle is the controlled object that can carry the aforementioned trajectory prediction device. As mentioned in the previous story, in addition to this, it further includes robots or machine toys, etc., which will not be repeated.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by program instructions related to hardware, the aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, execute It includes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

Since each module in this embodiment can execute the method shown in Embodiment 1, for parts that are not described in detail in this embodiment, reference may be made to the relevant description of Embodiment 1.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.

Claims (17)

1. A trajectory prediction method, comprising:

acquiring global semantic data and global track data of an area where an object to be predicted is located;

fusing the global semantic data and the global track data to obtain global fusion data;

extracting features in the global fusion data to obtain global features;

and processing the global characteristics by using a trained track prediction model to obtain the target track of the object to be predicted.

2. The method according to claim 1, wherein the obtaining global semantic data of an area where the object to be predicted is located comprises:

acquiring a global area image of an area where the object to be predicted is located;

performing semantic recognition on each pixel in the global area image to obtain the semantic category of each pixel;

and according to the semantic category of each pixel, performing semantic annotation on the global area image to obtain the global semantic information.

3. The method according to claim 2, wherein the semantically labeling the global region image according to the semantic category of each pixel to obtain the global semantic information comprises:

according to the semantic category of each pixel, performing semantic segmentation on the global area image to obtain a plurality of semantic objects;

and performing semantic annotation on each semantic object to obtain the global semantic information.

4. The method according to claim 2 or 3, wherein the global area image is a Digital Orthophoto (DOM) image.

5. The method according to claim 1, wherein the obtaining global trajectory data of an area where the object to be predicted is located comprises:

acquiring a track point set of each moving object in the area of the object to be predicted, wherein the track point set is formed by collecting coordinate points of the moving objects according to a time sequence;

coding the track point set of each moving object to obtain the track characteristics of each moving object;

and constructing a track tensor according to the track characteristics of each moving object to serve as the global track data.

6. The method according to any one of claims 1-3 and 5, wherein the extracting features in the global fusion data to obtain global features comprises:

and processing the fusion information by using a trained feature extraction model to obtain the global features.

7. The method of claim 6, wherein the feature extraction model comprises at least: convolutional neural network CNN model.

8. The method of claim 1, wherein the trajectory prediction model comprises at least one of: a long and short term memory network LSTM model and a multilayer perceptron MLP;

wherein the multilayer perceptron comprises: a Recurrent Neural Network (RNN) model and a Gated Recurrent Unit (GRU) model.

9. The method according to claim 1 or 8, wherein the trajectory prediction model is a first trajectory prediction model for predicting a target trajectory of the object to be predicted.

10. The method according to claim 1 or 8, wherein the trajectory prediction model is a second trajectory prediction model for predicting the motion trajectories of all moving objects in the area;

the processing the global feature by using the trained track prediction model to obtain the target track of the object to be predicted comprises the following steps:

inputting the global features into the second track prediction model, obtaining the motion tracks of all the moving objects in the area output by the second track prediction model, and taking the motion tracks of the objects to be predicted in all the moving objects as the target track.

11. The method of claim 1, wherein the object to be predicted comprises at least one of: vehicles, animals, humans, robots and unmanned aerial vehicles.

12. The method of claim 1, further comprising:

and performing motion planning on the object to be predicted according to the target track.

13. A trajectory prediction device, comprising:

the acquisition module is used for acquiring global semantic data and global track data of an area where an object to be predicted is located;

the fusion module is used for fusing the global semantic data and the global track data to obtain global fusion data;

the feature extraction module is used for extracting features in the global fusion data to obtain global features;

and the prediction module is used for processing the global characteristics by utilizing a trained track prediction model to obtain the target track of the object to be predicted.

14. A trajectory prediction device, comprising:

a memory;

a processor; and

a computer program;

wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1 to 12.

15. A computer-readable storage medium, having stored thereon a computer program,

the computer program is executed by a processor to implement the method of any one of claims 1 to 12.

16. A driving system, comprising:

trajectory prediction means for performing the method of any one of claims 1 to 12;

and the motion controller is used for controlling the controlled object to move according to the target track acquired by the track prediction device.

17. A vehicle, characterized by comprising:

the driving system of claim 16.

Images