CN108803617A - Trajectory predictions method and device - Google Patents

Abstract

The embodiment of the present invention provides a kind of trajectory predictions method and device, is related to the Local Navigation field of robot and intelligent vehicle, applied to the vehicle for being provided with vehicle-mounted camera, this method includes：It is photographed to ambient enviroment using vehicle-mounted camera, acquisition includes the video sequence of surrounding vehicles and vehicle context.The surrounding vehicles are positioned from the video sequence and extract the historical track information of the surrounding vehicles, and the Scene Semantics information that video sequence progress image segmentation is obtained is as auxiliary information.The historical track information and the auxiliary information are inputted into neural network model, obtain the prediction locus of the surrounding vehicles.The accuracy of prediction track of vehicle can be improved in the trajectory predictions method.

Description

Trajectory prediction method and device

technical field

The invention relates to the field of local navigation of robots and intelligent vehicles, in particular to a trajectory prediction method and device.

Background technique

During vehicle driving, it is very important to predict the future trajectories of other traffic participants to avoid the autonomous vehicle from crashing into other vehicles. Assuming that all traffic participants obey the traffic rules and human drivers can subconsciously predict the future trajectory of the target, for autonomous vehicles, the method of building a model is usually used to predict the future trajectory of other traffic participants.

However, most current works either use static images to extract visual-semantic messages, or adopt an end-to-end structure to learn driving networks, the former ignores temporal continuity in driving situations, and the latter lacks the interpretability of training networks, thus causing The problem of low accuracy in predicting vehicle trajectories.

Contents of the invention

The main purpose of the present invention is to provide a trajectory prediction method and device, which can improve the accuracy of predicting vehicle trajectory.

The trajectory prediction method provided in the first aspect of the embodiment of the present invention is applied to a vehicle equipped with a vehicle-mounted camera, and the method includes: using the vehicle-mounted camera to take pictures of the surrounding environment, and acquiring a video sequence including surrounding vehicles and vehicle backgrounds; Position the surrounding vehicles in the video sequence and extract the historical trajectory information of the surrounding vehicles, and use the scene semantic information obtained by image segmentation of the video sequence as auxiliary information; input the historical trajectory information and the auxiliary information into the neural network network model to obtain the predicted trajectory of the surrounding vehicles.

The trajectory prediction device provided in the second aspect of the embodiment of the present invention is applied to a vehicle equipped with a vehicle-mounted camera, and the device includes: an acquisition module, configured to use the vehicle-mounted camera to take pictures of the surrounding environment, and acquire information including surrounding vehicles and vehicle backgrounds Video sequence; extraction and segmentation module, used to locate the surrounding vehicles from the video sequence and extract the historical track information of the surrounding vehicles, and use the scene semantic information obtained by image segmentation of the video sequence as auxiliary information; output module , for inputting the historical trajectory information and the auxiliary information into the neural network model to obtain the predicted trajectory of the surrounding vehicles.

As can be seen from the above-mentioned embodiments, the video sequence including the surrounding vehicles and the vehicle background is obtained through the vehicle-mounted camera, and the video sequence is image-segmented to obtain scene semantic information, and then the scene semantic information and historical trajectory information are input into the neural network model to obtain the predicted trajectory. Rather than using static images to extract scene semantic information for analysis, the time continuity of the neural network model in this embodiment is ensured, thereby improving the accuracy of predicting vehicle trajectories.

Description of drawings

FIG. 1 is a schematic diagram of the implementation flow of the trajectory prediction method provided by the first embodiment of the present invention;

Fig. 2 is a schematic diagram of the implementation flow of the trajectory prediction method provided by the second embodiment of the present invention;

Fig. 3 is a schematic diagram of the neural network model of the trajectory prediction method provided by the second embodiment of the present invention;

Fig. 4 is a schematic diagram of the application of the trajectory prediction method provided by the second embodiment of the present invention;

Fig. 5 is a schematic structural diagram of a trajectory prediction device provided by a third embodiment of the present invention.

Detailed ways

In order to make the purpose, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts belong to the protection scope of the present invention.

Please refer to FIG. 1 . FIG. 1 is a schematic diagram of the implementation flow of the trajectory prediction method provided by the first embodiment of the present invention, and the method is applied to a vehicle equipped with a vehicle-mounted camera. As shown in Figure 1, the trajectory prediction method mainly includes the following steps:

101. Use the on-board camera to take pictures of the surrounding environment, and acquire a video sequence including the surrounding vehicles and the vehicle background.

Specifically, in the automatic driving process of the vehicle, it is assumed that all traffic participants obey the traffic rules, and the method of building a model is used to predict the future trajectories of other traffic participants. In the process of building the model, it is necessary to obtain the surrounding environment information, so first use the on-board camera on the vehicle to take pictures of the surrounding environment, and obtain a video sequence including the surrounding vehicles and the vehicle background. Wherein, the number of frames per second of the video sequence may be selected according to actual conditions. Wherein, the surrounding vehicles may refer to vehicles that are within a certain distance from the vehicle equipped with the vehicle camera and have potential influence on the vehicle equipped with the vehicle camera, and the range may be 30 meters around the vehicle equipped with the vehicle camera.

102. Locate the surrounding vehicle from the video sequence, extract historical track information of the surrounding vehicle, and use scene semantic information obtained by image segmentation of the video sequence as auxiliary information.

Specifically, the motion in the video sequence is the illusion of motion formed by displaying frames in rapid succession, and the video sequence of each frame is a still image, so the surrounding vehicles are positioned in the video sequence of each frame, and the continuous multi-frame The trajectory information of the surrounding vehicles can be seen in the video sequence, so for the video sequence of the current frame, the historical trajectory information of the surrounding vehicles is obtained from the video sequence of multiple frames in the past.

Wherein, the scene semantic information obtained by image segmentation of the video sequence of each frame is used as auxiliary information. Image segmentation refers to segmenting objects in each frame of the video sequence according to semantic categories and labeling scene semantic information, such as pedestrians, surrounding vehicles, buildings, sky, vegetation, road obstacles, lane lines, road sign information and traffic lights Information, etc., and then identify the drivable area in the video sequence of the current frame. By using the scene semantic information as auxiliary information, it can be robust to the apparent changes of the target.

Optionally, since the areas corresponding to different semantic categories are different feature areas, and the boundaries between different feature areas are edges, edge detection can be used to segment the video sequence of each frame, so as to extract the desired target. Among them, the edge indicates the end of one feature area and the beginning of another feature area. The internal features or attributes of the required target are consistent and inconsistent with the internal features or attributes of other feature areas, such as grayscale, color, or texture. feature.

103. Input the historical trajectory information and the auxiliary information into the neural network model to obtain the predicted trajectory of the surrounding vehicles.

Specifically, the neural network is a complex network system formed by a large number of simple neurons widely interconnected. It is a highly complex nonlinear dynamic learning system with large-scale parallelism, distributed storage and processing, self-organization, self-organization Adaptability and self-learning ability. Therefore, the neural network is used to establish a mathematical model to obtain the neural network model, and the obtained historical trajectory information and auxiliary information are input into the neural network model to obtain the predicted trajectory of the surrounding vehicles.

In the embodiment of the present invention, the video sequence including the surrounding vehicles and the vehicle background is obtained through the vehicle-mounted camera, and the video sequence is image-segmented to obtain scene semantic information, and then the scene semantic information and historical trajectory information are input into the neural network model to obtain the predicted trajectory, Rather than using static images to extract scene semantic information for analysis, the time continuity of the neural network model in this embodiment is ensured, thereby improving the accuracy of predicting vehicle trajectories.

Please refer to FIG. 2 . FIG. 2 is a schematic diagram of the implementation flow of the trajectory prediction method provided by the second embodiment of the present invention, and the method is applied to a vehicle equipped with a vehicle-mounted camera. As shown in Figure 2, the trajectory prediction method mainly includes the following steps:

201. Use the on-board camera to take pictures of the surrounding environment, and acquire a video sequence including surrounding vehicles and vehicle backgrounds.

202. Locate the surrounding vehicle from the video sequence, extract historical track information of the surrounding vehicle, and use scene semantic information obtained by image segmentation of the video sequence as auxiliary information.

203. Input the auxiliary information to the convolutional neural network to obtain spatial feature information.

Specifically, the neural network model includes a convolutional neural network, a first-layer long-short-term memory network, a second-layer long-short-term memory network, and a fully connected layer.

Among them, convolutional neural network is a kind of feedforward neural network. After the video sequence is segmented and labeled, the scene semantic information is obtained as auxiliary information and input to the convolutional neural network to obtain spatial feature information. Auxiliary information is image information, which can be encoded with one-bit effective code, and the number of channels is used as the number of semantic categories, and the auxiliary information is input into a four-layer convolutional neural network. The convolution kernel can be 3*3*4, and the spatial Feature information, the spatial feature information is represented by a 6-dimensional vector.

Among them, as shown in Figure 3, the convolutional neural network includes a convolutional layer, a linear correction unit, a pooling layer, and a dropout layer. Convolutional layers extract features from side information. Linear layers can introduce nonlinear features. The pooling layer can compress the input auxiliary information and extract the main features. Dropout layers can be used to alleviate the overfitting problem.

204. Input the historical trajectory information into the first-layer long-short-term memory network to obtain time feature information. Inputting the spatial feature information and the time feature information into the second-layer long-short-term memory network to obtain joint feature information.

Specifically, the Long-short Term Memory (LSTM) network is a time recurrent network. The historical trajectory information has a certain timing, and there is a certain contextual relationship in the position, that is, the historical trajectory information as a sequence input needs to continuously learn the location characteristics before and after, so the LSTM network is used to train the historical trajectory information and connect the historical frames. The trajectory information of is used to infer the trajectory information of the current frame.

Wherein, as shown in FIG. 3 , the historical trajectory information is input into the first-layer LSTM network to obtain temporal feature information, and the temporal feature information and the spatial feature information obtained in step 203 are input to the second-layer LSTM network to obtain joint representation information. And because the dimension of the three-dimensional space grid is 6, the first layer LSTM network can not only learn the temporal feature information, but also make the dimensions of the temporal feature information and spatial feature information consistent. In practical applications, the number of units in the first layer LSTM network may be 100, and the second layer LSTM network may include LSTM networks with 300 units in both layers.

205. Input the joint feature information into the fully connected layer to obtain the predicted trajectory.

Specifically, each node of the fully connected layer is connected to all the nodes of the previous layer to combine all the features extracted by the previous layer, so the joint representation information is input into the fully connected layer for a series of The output of the neural network model is obtained by matrix multiplication, and the predicted trajectory J of T time steps is obtained. In practical applications, the time T can be 1.6s (unit: second)

Wherein, the neural network model includes the following formula:

J←M _p (h,a):H×A.

Among them, J represents the predicted trajectory, M represents the mapping relationship between H, A and J, H represents the historical trajectory information, A represents the auxiliary information, p represents the surrounding vehicles, h represents the vehicle in the t-th frame video sequence The position information of p, a represents the scene semantic information of vehicle p in the t-th frame video sequence, j represents the position information of vehicle p in the t-th frame video sequence from T+1 frame, and t represents each frame.

Among them, as shown in FIG. 3 , in this embodiment, an image segmentation-long-short-term memory network (Segmentation-Long-short Term Memory, SEG-LSTM) is proposed to fuse multiple streams of historical frames and predict future trajectories of surrounding vehicles.

Among them, the number of layers of the LSTM network, the number of units of each layer of the LSTM network, the number of layers of the convolutional neural network, and the size of the convolution kernel are network hyperparameters, which are determined by cross-validation. The role of cross-validation is to determine the optimal hyperparameters while avoiding model overfitting. Exemplarily, first, the data set is divided into a training set and a test set with a ratio of 5:1. Then the training set is divided into 5 parts, and each part is used as a verification set in turn, and the remaining 4 parts are used as a training set for 5 training and verification, and the corresponding average accuracy can be obtained by using different hyperparameters. Determine its value.

As shown in Figure 4, the video sequence is divided into multiple time-step video sequences by frame, and the position information is obtained from the video sequence of each frame by detection and tracking, and the semantic information is obtained by image segmentation. Subsequently, the position information and semantic information of the same frame are input into the LSTM network for training, and the predicted trajectory is obtained by training multiple historical frames and video sequences of the current frame.

206. Acquire minimum relative distances between the vehicle and each of the surrounding vehicles through the depth camera. According to the minimum relative distance, the two-dimensional spatial prediction trajectory is converted into a three-dimensional spatial prediction trajectory.

Specifically, the predicted trajectory is a predicted trajectory in two-dimensional space, and a depth camera is also set in the vehicle.

Wherein, the two-dimensional spatial prediction trajectory is converted into a three-dimensional spatial prediction trajectory according to the minimum relative distance by the following formula:

Among them, x, y, w, h respectively represent the elements in the pixel bounding box of the two-dimensional spatial prediction trajectory in each frame of video sequence, x _r , y _r , w _r , h _r represent the three-dimensional spatial prediction trajectory in each Elements in the pixel bounding box in a frame of video sequence, f represents the focal length of the depth camera, and d _min represents the minimum relative distance between the vehicle and the surrounding vehicles.

Among them, if the subscript p is ignored, the historical trajectory information and predicted trajectory can be defined as a three-dimensional space occupying grid, namely

H,J∈R ⁶ ={x,y,w,h,d _min ,d _max }

In the formula, d _max represents the maximum distance between the vehicle and the surrounding vehicles.

In the embodiment of the present invention, firstly, a video sequence including surrounding vehicles and vehicle background is acquired through the vehicle-mounted camera, and image segmentation is performed on the video sequence to obtain scene semantic information, and then the scene semantic information and historical track information are input into the neural network model to obtain prediction Trajectories, instead of using static images to extract scene semantic information for analysis, thereby ensuring the time continuity of the neural network model in this embodiment, and improving the accuracy of predicting vehicle trajectories. In addition, the use of convolutional neural network and LSTM network can improve the robustness of surrounding vehicle tracking, and the use of image segmentation to obtain scene semantic information can improve the interpretability of the training process.

Please refer to FIG. 5 . FIG. 5 is a schematic structural diagram of a trajectory prediction device provided by a third embodiment of the present invention, which is applied to a vehicle equipped with a vehicle-mounted camera. As shown in Figure 5, the trajectory prediction device mainly includes:

The acquisition module 301 is configured to use the vehicle camera to take pictures of the surrounding environment, and acquire video sequences including surrounding vehicles and vehicle backgrounds.

The extraction and segmentation module 302 is configured to locate surrounding vehicles from the video sequence and extract historical trajectory information of the surrounding vehicles, and use scene semantic information obtained by image segmentation of the video sequence as auxiliary information.

The output module 303 is used for inputting historical trajectory information and auxiliary information into the neural network model to obtain predicted trajectories of surrounding vehicles.

Further, the neural network model includes a convolutional neural network, a first layer of long-term short-term memory network, a second layer of long-term short-term memory network and a fully connected layer, then,

The output module 303 is also used to input auxiliary information to the convolutional neural network to obtain spatial feature information.

The output module 303 is also used to input the historical trajectory information into the first layer long short-term memory network to obtain time feature information.

The output module 303 is further configured to input the spatial feature information and the temporal feature information into the second-layer long-short-term memory network to obtain joint feature information.

The output module 303 is also used to input the joint feature information into the fully connected layer to obtain the predicted trajectory.

Further, the neural network model includes the following formula:

J←M _p (h,a):H×A.

Among them, J represents the predicted trajectory, M represents the mapping relationship between H, A and J, H represents the historical trajectory information, A represents the auxiliary information, p represents the surrounding vehicles, h represents the position information of the vehicle p in the t-th frame video sequence , a represents the scene semantic information of vehicle p in the t-th frame video sequence, j represents the position information of vehicle p in the t-th frame video sequence from T+1 frame, and t represents each frame.

Further, the predicted trajectory is a two-dimensional space predicted trajectory, and a depth camera is also set in the vehicle,

The acquiring module 301 is further configured to respectively acquire the minimum relative distances between the vehicle and surrounding vehicles through the depth camera.

Then the device further includes a conversion module 304,

The conversion module 304 is configured to convert the two-dimensional spatial prediction trajectory into a three-dimensional spatial prediction trajectory according to the minimum relative distance.

Further, the conversion module 304 is also used to convert the two-dimensional space prediction trajectory into the three-dimensional space prediction trajectory according to the minimum relative distance by the following formula:

Among them, x, y, w, h respectively represent the elements in the pixel bounding box of the two-dimensional spatial prediction trajectory in each frame of video sequence, x _r , y _r , w _r , h _r represent the three-dimensional spatial prediction trajectory in each The element in the pixel bounding box in a frame of video sequence, f represents the focal length of the depth camera, and d _min represents the minimum relative distance between the vehicle and each surrounding vehicle.

For the process of the above-mentioned modules realizing their respective functions, reference may be made to relevant content in the above-mentioned embodiments shown in FIGS. 1 to 4 , which will not be repeated here.

In the embodiment of the present invention, the video sequence including the surrounding vehicles and the vehicle background is obtained through the vehicle-mounted camera, and the video sequence is image-segmented to obtain scene semantic information, and then the scene semantic information and historical trajectory information are input into the neural network model to obtain the predicted trajectory, Rather than using static images to extract scene semantic information for analysis, the time continuity of the neural network model in this embodiment is ensured, thereby improving the accuracy of predicting vehicle trajectories.

In the multiple embodiments provided in this application, it should be understood that the disclosed methods and devices may be implemented in other ways. For example, the above-described embodiments are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or can be Integrate into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication link shown or discussed may be through some interfaces, and the indirect coupling or communication link between modules may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or may be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present invention can be integrated into one processing module. Each module may also exist separately physically, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented not only in the form of hardware, but also in the form of software function modules.

It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence. Because of the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

The above is a description of a trajectory prediction method and device, terminal, and computer-readable storage medium provided by the present invention. For those of ordinary skill in the art, based on the ideas of the embodiments of the present invention, both in terms of specific implementation and application range There will be changes. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. A trajectory prediction method, applied to a vehicle provided with a vehicle-mounted camera, characterized in that the method comprises: Use the on-board camera to take pictures of the surrounding environment, and obtain video sequences including surrounding vehicles and vehicle backgrounds; Locating the surrounding vehicles from the video sequence and extracting the historical trajectory information of the surrounding vehicles, and using the scene semantic information obtained by image segmentation of the video sequence as auxiliary information; Inputting the historical trajectory information and the auxiliary information into the neural network model to obtain the predicted trajectory of the surrounding vehicles. 2. trajectory prediction method as claimed in claim 1, is characterized in that, described neural network model comprises convolutional neural network, first layer long short-term memory network, second layer long short-term memory network and fully connected layer, then said Said inputting the historical trajectory information and the auxiliary information into the neural network model to obtain the predicted trajectory of the surrounding vehicles includes: Inputting the auxiliary information into the convolutional neural network to obtain spatial feature information; Inputting the historical trajectory information into the first layer long short-term memory network to obtain time feature information; inputting the spatial feature information and the temporal feature information into the second-layer long-short-term memory network to obtain joint feature information; The joint feature information is input into a fully connected layer to obtain the predicted trajectory. 3. trajectory prediction method as claimed in claim 1, is characterized in that, described neural network model comprises following formula: J←M _p (h,a):H×A; Among them, J represents the predicted trajectory, M represents the mapping relationship between H, A and J, H represents the historical trajectory information, A represents the auxiliary information, p represents the surrounding vehicles, h represents the tth frame The position information of the vehicle p in the video sequence, a represents the scene semantic information of the vehicle p in the t-th frame of the video sequence, j represents the position information of the vehicle p in the t-th frame of the video sequence from the T+1 frame, and t represents each frame . 4. The trajectory prediction method according to claim 1, wherein the predicted trajectory is a two-dimensional space predicted trajectory, and a depth camera is also arranged in the vehicle, then the method also includes: Obtaining the minimum relative distances between the vehicle and each of the surrounding vehicles through the depth camera; Converting the two-dimensional spatial prediction trajectory into a three-dimensional spatial prediction trajectory according to the minimum relative distance. 5. trajectory prediction method as claimed in claim 4, is characterized in that, by following formula, according to described minimum relative distance, described two-dimensional spatial prediction trajectory is converted into three-dimensional spatial prediction trajectory: Among them, x, y, w, h respectively represent the elements in the pixel bounding box of the two-dimensional spatial prediction trajectory in each frame of video sequence, x _r , y _r , w _r , h _r represent the three-dimensional spatial prediction trajectory in each Elements in the pixel bounding box in a frame of video sequence, f represents the focal length of the depth camera, and d _min represents the minimum relative distance between the vehicle and each of the surrounding vehicles. 6. A trajectory prediction device applied to a vehicle provided with a vehicle-mounted camera, characterized in that the device comprises: The acquisition module is used to use the vehicle camera to take pictures of the surrounding environment, and acquire video sequences including surrounding vehicles and vehicle backgrounds; An extraction and segmentation module, configured to locate the surrounding vehicles from the video sequence and extract historical track information of the surrounding vehicles, and use the scene semantic information obtained by image segmentation of the video sequence as auxiliary information; The output module is used to input the historical trajectory information and the auxiliary information into the neural network model to obtain the predicted trajectory of the surrounding vehicles. 7. trajectory prediction device as claimed in claim 6, is characterized in that, described neural network model comprises convolutional neural network, the first layer of long short-term memory network, the second layer of long short-term memory network and fully connected layer, then, The output module is further configured to input the auxiliary information to the convolutional neural network to obtain spatial feature information; The output module is also used to input the historical trajectory information into the first layer long short-term memory network to obtain time feature information; The output module is further configured to input the spatial feature information and the temporal feature information into the second-layer long-short-term memory network to obtain joint feature information; The output module is further configured to input the joint feature information into the fully connected layer to obtain the predicted trajectory. 8. trajectory prediction device as claimed in claim 6, is characterized in that, described neural network model comprises following formula: J←M _p (h,a):H×A; Among them, J represents the predicted trajectory, M represents the mapping relationship between H, A and J, H represents the historical trajectory information, A represents the auxiliary information, p represents the surrounding vehicles, h represents the tth frame The position information of the vehicle p in the video sequence, a represents the scene semantic information of the vehicle p in the t-th frame of the video sequence, j represents the position information of the vehicle p in the t-th frame of the video sequence from the T+1 frame, and t represents each frame . 9. The trajectory prediction device according to claim 6, wherein the predicted trajectory is a two-dimensional space predicted trajectory, and a depth camera is also arranged in the vehicle, The acquiring module is further configured to respectively acquire the minimum relative distances between the vehicle and each of the surrounding vehicles through the depth camera; Then the device further includes a conversion module, The conversion module is configured to convert the two-dimensional spatial prediction trajectory into a three-dimensional spatial prediction trajectory according to the minimum relative distance. 10. The trajectory prediction device according to claim 9, wherein: The conversion module is further configured to convert the two-dimensional spatial prediction trajectory into a three-dimensional spatial prediction trajectory according to the minimum relative distance through the following formula: Among them, x, y, w, h respectively represent the elements in the pixel bounding box of the two-dimensional spatial prediction trajectory in each frame of video sequence, x _r , y _r , w _r , h _r represent the three-dimensional spatial prediction trajectory in each Elements in the pixel bounding box in a frame of video sequence, f represents the focal length of the depth camera, and d _min represents the minimum relative distance between the vehicle and each of the surrounding vehicles.