CN111814623A - A visual detection method of vehicle lane departure based on deep neural network - Google Patents

Abstract

A visual detection method for vehicle lane departure based on deep neural network, comprising the following steps: (1) using a lane line segmentation algorithm to locate and segment the position of the lane line; (2) cutting out the segmented lane line in the figure, and classifying it with The algorithm classifies the lane lines; (3) compresses the neural network model; (4) calibrates the vehicle-mounted camera. The invention proposes a vehicle lane departure detection method based on a deep neural network, which combines a deep neural network-based lane line semantic segmentation and recognition and a coordinate system perspective conversion algorithm, and is deployed on a driving recorder, with low cost and strong robustness. . The present invention adopts a lightweight ERFnet-SAD network, and performs corresponding pruning and quantization compression on the model, which ensures high speed while ensuring accuracy.

Description

A visual detection method of vehicle lane departure based on deep neural network

technical field

The invention relates to the field of vehicle assisted driving, in particular to real-time lane line real-time semantic segmentation and recognition, and vehicle departure warning.

Background technique

Vehicle safety assisted driving is an important research direction in the current intelligent transportation field. Many companies and research institutions at home and abroad are doing research on related technologies, including vehicle lane departure warning technology. According to the statistics of the traffic department, 50% of traffic accidents are caused by vehicles deviating from the normal driving track. When the vehicle is driving at high speed for a long time, the driver is prone to inattention, which can easily cause the vehicle to deviate from the normal lane, thereby causing traffic accidents. Therefore, developing a vehicle lane departure detection method and proactively reminding drivers of safe driving is an effective means to avoid such accidents.

How to accurately detect, identify, and locate the lane line is the core technical issue of the vehicle lane departure warning system. Nowadays, widely used departure warning systems such as AutoVue system, LDW system, AWS system, etc. are installed in multiple parts of the vehicle. The sensor and camera obtain the lane line information, and then detect and identify the lane line through traditional image processing methods (edge detection, Hough line detection, straight line fitting, etc.) , The problem of background interference is extremely sensitive, which leads to problems such as narrow application range of the system and low detection and recognition accuracy.

Some researchers have tried to use deep learning-based methods, such as VPGnet proposed by Seokju Lee and SCNN proposed by Xingang Pan, using scene segmentation algorithms to segment lane lines and lanes, and then predict the location of the vanishing point. The network model parameters used by the algorithm are large, and the detection speed is slow, so it cannot be applied to the real scene.

SUMMARY OF THE INVENTION

In view of the shortcomings of the existing lane line departure warning system, the present invention proposes a vehicle lane departure detection method based on a deep neural network, which combines the lane line semantic segmentation recognition and coordinate system perspective conversion algorithm based on a deep neural network, and is deployed in driving. On the recorder, the cost is low and the robustness is strong. The invention adopts the lightweight ERFnet-SAD network, and performs corresponding pruning and quantization compression on the model, so as to ensure the accuracy and at the same time the speed is fast.

The technical scheme adopted by the present invention to solve its technical problems is:

A method for visual detection of vehicle lane departure based on deep neural network, comprising the following steps:

(1) Use the lane line segmentation algorithm to locate and segment the position of the lane line;

(2) Cut out the lane lines segmented in the figure, and use the classification algorithm to classify the lane lines;

(3) Neural network model compression;

(4) Calibration of vehicle-mounted camera.

Further, in the step (1), self-attention distillation is used to learn the lightweight lane detection network (Learning Lightweight Lane Detection CNNs by Self Attention Distillation), which is different from other existing deep neural network-based lane line segmentation algorithms (SCNN etc.). ), the number of parameters is 20 times less and the speed is 10 times faster. There are three benchmark datasets for lane line detection (TuSimple, Culane, and BDD100K). Considering the domestic application, the Culane dataset was collected in Beijing, so this program chooses to train on Culane.

Still further, in the step (2), the lane lines segmented in the step (1) are cut out from the image, manually marked again, put into the classification network for training, and the lane lines are divided into yellow solid lines and white solid lines. , yellow dotted line, white dotted line, wireless category 5.

Further, in the step (3), the neural network model is simplified, and the model is pruned and quantized and compressed.

In the step (4), after the lane line segmentation algorithm locates the position of the lane line, the position of the own vehicle needs to be obtained, and the on-board camera is calibrated to convert the image coordinate system.

The beneficial effects of the invention are mainly manifested in: low cost, strong robustness, and high speed while ensuring accuracy.

Description of drawings

Figure 1 is the network architecture diagram of the lane line segmentation algorithm.

Figure 2 is a probability map of lane line segmentation results.

Figure 3 is an example of the Culane dataset

Figure 4 shows Hough line detection on the probability map.

Figure 5 is to cut out each lane line.

Figure 6 is the classification of lane lines.

Figure 7 is a neuron reduction.

Figure 8 is Zhang's calibration template.

Figure 9 is a coordinate system conversion.

Figure 10 is an image perspective transformation.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

1 to 10 , a method for visual detection of vehicle lane departure based on a deep neural network includes the following steps:

(1) The network structure of the lane line segmentation network is shown in Figure 1. It consists of an encoder and a decoder. The encoding part includes 4 Encoders E1, E2, E3, and E4. Each Encoder consists of several convolutions, pooling, BN (BatchNorm) layer, the decoding part includes two branches, and the upper right branch includes two Decoders D1 and D2, which are used to obtain the probability map of lane line segmentation results. As shown in Figure 2, a lane line corresponds to a probability map. The lower right branch P1 is a classification network consisting of a few simple layers of convolution and one fully connected layer, which is used to determine whether there is a lane line in the lane line segmentation probability map. It can be seen in the network structure diagram that each layer of Encoder includes an AT-GEN, which is an attention generator. The attention map generated by the shallow layer is used as the target of the deep layer. On the one hand, richer context information is extracted, and the other is Aspects can better guide deep network training, which is the self-attention distillation mechanism. The total loss function of the network is:

用L₂Loss计算两个attention map的损失。车道线分割实施过程如下：Among them, L _seg is the segmentation loss, using the CrossEntropy function, L _Iou is the loss of calculating the IOU (Intersectionover Union), L _exist is the loss of judging whether the lane line exists, L _distill is the distillation loss, α, β, γ are balance factors, and

Calculate the loss of two attention maps with L ₂ Loss. The implementation process of lane line segmentation is as follows:

a. Dataset preparation:

The present invention uses the existing lane line benchmark data set Culane, including 55 hours of on-board camera video, and extracts more than 100,000 on-board cameras to record urban road images. An example of the data label is shown in Figure 3. We split the dataset into 80000+ graphs as training set and 30000+ graphs as test set. The resolution of each image is 1640*590. Considering that all the vehicles and roads in the images are in the lower part of the image, in order to reduce the negative samples of information in the image, and at the same time improve the training efficiency and the detection speed, the upper part of the image is cut and discarded, and the final input is an image of 1640*350. In order to increase the generalization ability of the network model, the data set is randomly enhanced, including image enlargement, reduction, cropping, contrast enhancement, and noise addition.

b. Initialize the training model:

Initialize the network parameters, resize the input image to 976*208 while ensuring the original aspect ratio of the image, and shuffle it randomly. Set the network initial learning rate η = 0.01, initialization iteration times epoch = 300, batch data volume (BatchSize = 32) and other hyperparameters. Finally, stochastic gradient descent (SGD) is used as the optimizer for network parameter iteration, so hyperparameters such as momentum parameters and weight decay rate parameters are also set, and the loss factors α, β, and γ are set to 0.1.

c. Use the trained model to detect the lane line map:

The video data obtained by the vehicle-mounted device is divided into each frame and detected one by one, and the image resolutions obtained by different vehicle-mounted devices are different, but the present invention uses an adaptive cropping algorithm to uniformly crop the image into an aspect ratio of 1640:350. The detection result outputs 4 probability maps and 4 probability values, which are filtered by the threshold. Assuming that the threshold is set to 0.5, the probability map with a probability value greater than 0.5 is considered to have lane lines, and the probability map is post-processed. The resolution of the output probability map is 976*208 and the height is 208. Initially, the present invention takes a bright spot every 20 lines, which is the point with the largest pixel value in this line, and a total of 10 points are obtained, and then the cubic spline interpolation algorithm is used. A curve is fitted as the lane line. However, many experiments have found that when there are many noise points in the probability map, the fitted curve is very unsatisfactory. Therefore, the present invention first performs Hough line detection and screening on 10 points selected in the above output probability map. Considering that the Hough transform algorithm is time-consuming, in order to improve efficiency, we reduce the length and width of the probability map by 4 times, and adjust it to (976/ 4, 208/4) Do the Hough line detection again, select the Hough line with the highest probability of the Hough line, and calculate the distance from each point to the Hough line. The points with a distance less than 2.5 pixels are left, and the rest throw away. The result is shown in Figure 4, where 4 points far from the Hough line are filtered out. Then restore the graph to 976*208 size, and use the selected points for curve fitting.

(2) In step (1), the coordinate points of each lane line are obtained, and then each line is classified and predicted. The process is as follows:

a. Intercept the lane line:

For each line, we select three points, the coordinates of both ends of the line and the coordinates of the midpoint of the line, and use the minimum circumscribed rectangle function of opencv to detect the minimum circumscribed rectangle of the three points. At this time, a rectangle is obtained (the minimum width is 1 pixel value), widen the rectangle whose width is less than 30 to 30, that is, crop a rectangle with a width of 30 pixels. The rectangular frame obtained at this time may be inclined (not horizontal or vertical, and cannot be cropped), and the image needs to be rotated. This solution uniformly rotates all rectangles to be vertical. And cut line by line, respectively, for classification prediction. The cropping is shown in Figure 5.

b. Train a classifier

There is only one line in each rectangular image that is cut, and each image can only correspond to one category. The classification task is relatively simple. The present invention selects the Resnet18 network as the classification network, and its structure is shown in Table 1:

Table 1

Resnet18 consists of 16 convolutional layers, 1 pooling layer, and 1 fully connected layer, and the loss function uses CrossEntropy Loss. Data set preparation is also required. We cropped more than 3,000 lane lines, manually labeled and classified them, and performed data enhancement (image enlargement, reduction, contrast map enhancement, random noise addition, random cropping, etc.), and expanded the data by more than 20,000 sheets. The image resolution is uniformly adjusted to 224*224, the image is sent to the network for training, the initial learning rate η=0.01, the number of iterations epoch=50, BatchsSize=128, and the stochastic gradient descent optimizer is selected. The classification effect is shown in Figure 6. The letter bx in the figure represents a white dotted line, and bs represents a white solid line.

(3) Model compression:

计算得到每一个聚类中心值，代表cluster的权重。由于同一簇的权重共享一个权重大小，因此只需存储权值簇的索引值。最后为减少精度损失，需要通过微调训练对权重进行补偿。通过模型压缩，车道线分割网络网络参数量减小了一半，运行速度快了一倍。Considering that the present invention is to be transplanted to mobile devices, such devices have limited computing power, storage space, and running memory, while the neural network model has a large volume and a large amount of computation, so the model needs to be compressed. Model compression currently mainly uses channel pruning and weight quantization, and the present invention also compresses the model by combining the methods of pruning and quantization. A. Channel pruning, the purpose is to prune out the unimportant neurons in the neural network, as shown in Figure 7, the last black neurons in the figure are the real effective neurons. Each convolutional layer corresponds to a BN layer. We use the scaling factor gamma in the BN layer as the contribution size factor for evaluating the output of the previous layer, and use the L1 regularization gamma parameter to sparse the network. Finally, the channel with the gamma value of 0 is used. Safe to cut. B. Weight quantization. Weight quantization focuses on the parameters themselves, and does not change the calculation amount of the model, mainly including weight sharing and weight reduction. The method is to use the K-means clustering algorithm to cluster the weight matrix of each layer into several clusters, using

The value of each cluster center is calculated to represent the weight of the cluster. Since the weights of the same cluster share a weight size, it is only necessary to store the index value of the weight cluster. Finally, to reduce the loss of accuracy, the weights need to be compensated by fine-tuning training. Through model compression, the number of network parameters of the lane line segmentation network is reduced by half, and the running speed is doubled.

(4) Calibration of vehicle-mounted instrument camera, the present invention adopts Zhang Zhengyou's calibration plane calibration method, which has the advantages of simple operation, low requirements for equipment and high precision. The basic principle of Zhang's calibration can be expressed by the following formula:

Here, it is assumed that the template plane and the plane of the world coordinate system Z=0 are coincident, the matrix K represents the camera internal parameters, M=[XY 1] ^T is the homogeneous coordinate of the point on the template plane, for the point on the template, the corresponding image plane is obtained by projection point, the homogeneous coordinate of this point is represented by m=[uv 1] ^T , relative to the world coordinate system, the rotation matrix of the camera coordinate system is represented by [r ₁ r ₂ t], and the translation vector is represented by t. make

From the properties of the rotation matrix, it can be known that r ₁ ^T r ₂ =0, Pr ₁ P=Pr ₂ P=1, for each image, the internal parameter matrix has two basic constraints: h ₁ K ^-T K ^-1 h ₂ = 0, h ₁ ^T K - ^T K ^-1 h ₁ =h ₂ ^T K - ^T K ^-1 h ₂ . It has 5 undetermined internal parameters, so when the number of images is not less than 3, K can be calculated by virtue of linear uniqueness.

Zhang's calibration method usually adopts the following steps to complete:

a. Print the prepared template and paste it on a fixed plane;

b. Shoot the template from different angles to obtain the corresponding template image;

c. Perform feature point detection on the captured image;

d. Solve the internal and external parameters of the camera accordingly;

e. Solve the camera distortion coefficient;

f. Optimize and solve the obtained results.

Figure 8 is a classic plane template diagram used in Zhang's calibration. After the camera is calibrated, we can convert the coordinate system of the vehicle camera from the camera coordinate system to the world coordinate system. As shown in Figure 9. The same lane line map conversion effect is shown in Figure 10. From this we can determine the location of the vehicle.

Through the above steps, the position of the lane line in the image and the position of the vehicle can be determined, focusing on the lane lines approaching on both sides of the vehicle, setting a threshold, and sending a warning message to the vehicle when the distance between the vehicle and the lane line is less than a certain threshold system. In this way, it can better warn the vehicle to deviate from the track and assist the vehicle to drive.

Claims (5)

1. a vehicle lane departure visual detection method based on deep neural network, is characterized in that, described method comprises the following steps: (1) Use the lane line segmentation algorithm to locate and segment the position of the lane line; (2) Cut out the lane lines segmented in the figure, and use the classification algorithm to classify the lane lines; (3) Neural network model compression; (4) Calibration of vehicle-mounted camera. 2. A deep neural network-based visual detection method for vehicle lane departure as claimed in claim 1, characterized in that, in the step (1), self-attention distillation is used to learn a lightweight lane detection network, and the Training on Culane. 3. A kind of vehicle lane departure visual detection method based on deep neural network as claimed in claim 1 or 2, is characterized in that, in described step (2), the lane line segmented in step (1) is extracted from the image It is cut out from the middle, manually marked again, and put into the classification network training, and the lane lines are divided into five categories: yellow solid line, white solid line, yellow dotted line, white dotted line, and wireless. 4. A deep neural network-based visual detection method for vehicle lane departure as claimed in claim 1 or 2, wherein in the step (3), the neural network model is simplified, and the model is pruned, quantized and compressed . 5. a kind of vehicle lane departure visual detection method based on deep neural network as claimed in claim 1 or 2, is characterized in that, in described step (4), after the lane line segmentation algorithm locates the position of the lane line, It is necessary to obtain the position of the own vehicle, and use the calibration of the vehicle-mounted camera to convert the image coordinate system.

Images