CN114620218B - Unmanned aerial vehicle landing method based on improved YOLOv algorithm - Google Patents

Abstract

The invention provides an unmanned aerial vehicle landing method based on an improved YOLOv algorithm, wherein a camera module shoots a video of an environment where an unmanned aerial vehicle is located and transmits the video to an onboard computer through a data graph transmission module; the onboard computer uses YOLOv s algorithm, based on a Mosaic data enhancement model, and uses a self-adaptive anchor frame calculation and a self-adaptive picture scaling method to detect whether the real-time video stream corresponds to a flat land, a sloping field, a pit or a stair; if the result is a landing environment, the airborne computer is combined with the ground station and the flight control board to generate a yaw angle for controlling the unmanned aerial vehicle to land; if the detection result is not the landing environment, repeating the detection until the detection result is the landing environment; the ground station transmits the yaw angle into the flight control board through the data graph transmission module, the flight control board controls the unmanned aerial vehicle to land, after the unmanned aerial vehicle is far away from the ground, the unmanned aerial vehicle naturally lands if the unmanned aerial vehicle is flat, and the flight control board drives the electric push rod to extend outwards if the unmanned aerial vehicle is sloping fields, pits or stairs, so that the unmanned aerial vehicle is assisted to land in a corresponding landing environment.

Description

A drone landing method based on improved YOLOv5 algorithm

Technical Field

The present invention relates to the technical field of computer vision and unmanned aerial vehicle control, and in particular to a method for landing an unmanned aerial vehicle based on an improved YOLOv5 algorithm.

Background Art

A drone is an aerial robot that combines remote control and receiving equipment and is controlled by a program. It is active and applied in various fields due to its strong maneuverability, compactness, flexibility, and short mission cycle.

In recent years, with the continuous development of computer technology, many technologies such as computer vision have also been applied in many aspects. For example, in daily shopping mall entrances and exits, real-time detection of whether people are wearing masks, automatic recognition of license plates at highway intersections and parking lots, etc.

The difficulty of real-time detection and recognition of targets lies mainly in the extraction of feature information and real-time processing of information. The YOLO algorithm has solved this problem very well. Since its development, many versions of the YOLO algorithm have appeared. Currently, the best detection effect is the YOLOv5 series, which is a single-stage target detection algorithm. Compared with the previous YOLOv4 version, its real-time recognition accuracy and speed have been greatly improved.

The combination of target detection and recognition and drones has some applications in actual scenarios, but currently there are not many computer visions applied to the landing phase of drones, and the effect is average. There are also no measures to timely identify and enable drones to land smoothly and safely on the ground, so that drones can descend quickly and stably.

Summary of the invention

In view of the problems existing in the prior art, the present invention provides a UAV landing method based on an improved YOLOv5 algorithm, which can timely identify the landing ground of the UAV and enable the UAV to land smoothly and safely on the ground.

In order to achieve the above object, the present invention adopts the following technical solutions:

A method for landing a drone based on an improved YOLOv5 algorithm, wherein an electric push rod is installed at the bottom of a drone frame of the drone, and a camera module, a data graphics transmission module, an onboard computer and a flight control board are installed on the drone frame, comprising the following steps:

Step 1, the camera module captures the video of the drone's environment, which is then transmitted to the onboard computer via the data graphics transmission module;

Step 2: The onboard computer uses the YOLOv5s algorithm, based on the Mosaic data enhancement model, and uses the adaptive anchor frame calculation and adaptive image scaling methods to detect whether the real-time video stream transmitted by the data graphics transmission module corresponds to a flat environment, a slope environment, a pit environment, or a stair environment;

Step 3: If the detection result of step 2 is the landing environment of the drone, the onboard computer combines the ground station and the flight control board to generate a yaw angle for controlling the landing of the drone; if the detection result of step 2 is not the landing environment of the drone, repeat steps 1 and 2 until the detection result is the landing environment of the drone, and the onboard computer combines the ground station and the flight control board to generate a yaw angle for controlling the landing of the drone;

Step 4: The ground station transmits the yaw angle obtained in step 3 to the flight control board through the data graphic transmission module. The flight control board controls the UAV to land. When the UAV is 1 to 2 meters away from the ground, if the landing environment is flat, the UAV will land naturally. If the landing environment is a slope, pit or stairs, the flight control board drives the electric push rod to extend to assist the UAV to land in the corresponding landing environment.

Preferably, the step 2 includes the following specific sub-steps:

Step 2a, first extract the real-time video stream transmitted by the data graphic transmission module into frames of pictures, then mark these pictures and divide them into flat environment, slope environment, pit environment and stair environment, and finally convert and adjust the format to obtain a uniform-sized marked picture file;

Step 2b, sending the annotated image file described in step 2a to the input end of YOLOv5s, performing Focus operation and Ghost operation in sequence to obtain a fused image, fusing information and extracting features of the fused image, and outputting three feature prediction images;

Step 2c, performing a non-maximum suppression operation on the feature prediction map described in step 2b to obtain a prediction box that is closest to the true box probability, and obtain a trained network model;

In step 2d, the onboard computer first extracts the real-time video stream transmitted by the data graphics transmission module into frames of images, then annotates these images and converts and adjusts the format. Finally, the obtained annotated image files of uniform size are sent to the network model trained in step 2c to determine whether the real-time video stream corresponds to a flat environment, a slope environment, a pit environment or a stair environment.

Furthermore, in step 2c, when performing the non-maximum suppression operation, CIoU_Loss is used as the loss function of the bounding box, and CIoU_Loss is shown in the following formula:

Among them, IoU is the ratio of the intersection of the predicted box and the true box to the union of the predicted box and the true box, b, ^bgt represent the center points of the predicted box and the true box respectively, ρ is the Euclidean distance between the center points of the predicted box and the true box, c is the diagonal distance of the minimum closure area that contains both the predicted box and the true box, α is the weight function, and v is the influencing factor.

Furthermore, in step 2b, the FPN structure and the PAN structure are used to fuse information and extract features from the fused image in turn, and three feature prediction maps are output.

Preferably, the camera module is RER-USB4K02AF-V100.

Preferably, the camera module is connected to the input end of the onboard computer through a data graphic transmission module, the output end of the onboard computer is connected to the input end of the flight control board, one end of the data graphic transmission module is connected to the input end of the flight control board, and the model of the data graphic transmission module is HM30.

Preferably, it also includes a model aircraft battery, which is connected to the power supply end of the onboard computer and the power supply end of the flight control board respectively through a 5V2A BEC power supply ammeter.

Preferably, it also includes a GPS positioning module connected to the input end of the flight control board, and the GPS positioning module is used to transmit the collected real-time position information of the drone to the ground station through the communication module.

Preferably, the electric push rod is connected to the output end of the flight control board, and there are 3 to 5 electric push rods.

Preferably, the model of the flight control board is Pixhawk.

Compared with the prior art, the present invention has the following beneficial technical effects:

The invention discloses a method for landing a drone based on an improved YOLOv5 algorithm. A camera module can shoot a video of the real-time environment of the drone, and then transmit the video to an onboard computer through a data graphic transmission module. The onboard computer can detect whether the real-time video stream corresponds to a flat land, a sloped land, a pit or a stair environment. If the result is the landing environment of the drone, the onboard computer can generate a yaw angle for controlling the landing of the drone in combination with a ground station and a flight control board. If the result is not the landing environment of the drone, the drone can continue to fly. The landing environment of the drone can be found through the cooperation of the camera module, the data graphic transmission module and the onboard computer, and the required yaw angle can be generated. The ground station can transmit the yaw angle to the flight control board through the data graphic transmission module, and the flight control board controls the landing of the drone. When the drone approaches the ground, it is necessary to judge the ground environment at this time. If the landing environment is a flat land, the drone can land naturally. If it is a sloped land, a pit or a stair, the flight control board drives an electric push rod to extend outward at this time, so as to assist the drone to land in the corresponding landing environment, so that the drone can land on the ground stably and safely, thereby enhancing the intelligence of the drone. The structure of the present invention is simple to build and has strong portability, which solves the problem of intelligent landing of drones and can greatly improve the work efficiency of drone-related scientific researchers.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a block diagram of the working principle of the quad-rotor UAV according to the present invention.

FIG. 2 is a flowchart of the working process of the quadrotor drone described in the present invention.

FIG3 is a schematic diagram of the structure of the quad-rotor UAV according to the present invention.

In the figure: an electric push rod 6, a brushless DC motor 7, propeller blades 8, a drone frame 9 and stairs 10.

DETAILED DESCRIPTION

The present invention is further described in detail below in conjunction with specific embodiments, which are intended to explain the present invention rather than to limit it.

The present invention discloses a landing method for a quad-rotor drone based on an improved YOLOv5 algorithm. Referring to FIG3 , the drone mainly includes a drone frame 9, an electric push rod 6 is installed at the bottom of the drone frame 9, a camera module, a GPS positioning module, a data graphic transmission module, an onboard computer, a flight control board and a model aircraft battery are installed on the drone frame 9, and a DC brushless motor 7 is installed at the four rotors of the drone frame 9, and a blade 8 is installed at the output end of the DC brushless motor 7 at the upper end. The camera module is specifically RER-USB4K02AF-V100, the data graphic transmission module model is specifically HM30, and the flight control board model is Pixhawk.

Referring to Figure 1, the camera module is connected to the input end of the onboard computer through the data graphic transmission module, and the output end of the onboard computer is connected to the input end of the flight control board. The connection is made using a USB data cable, and the other end of the data graphic transmission module is simultaneously connected to the input end of the flight control board through a DuPont line. The camera module collects the video stream of the drone's environment in real time, and transmits the video stream to the onboard computer through the data graphic transmission module.

The aircraft model battery is connected to the power supply end of the onboard computer and the power supply end of the flight control board through a 5V2A BEC power supply ammeter. A USB data cable is used for connection. The aircraft model battery is used to power the onboard computer and the flight control board. The GPS positioning module is connected to the input end of the flight control board through a DuPont line, which can collect the position information of the quadcopter in real time and transmit it to the ground station software of the flight control board through the communication module. The electric push rod 6 and the brushless DC motor 7 are respectively connected to the output end of the flight control board through DuPont lines.

Refer to Figure 2. The flight control board cooperates with the drone ground station software to generate and control the roll, pitch, and yaw angles of the drone, and drives the DC brushless motor 7 to rotate at different speeds and directions, thereby controlling the drone to fly in a free posture. The onboard computer is installed with the operating environment and operating system of the YOLOv5 algorithm. By running the improved YOLOv5 algorithm, the camera module captures the video of the drone's environment and sends it to the onboard computer through the data graphics transmission module. The onboard computer can quickly detect in real time whether the real-time video stream captured by the camera module is the four trained landing environments. If so, the onboard computer combines the ground station and the control algorithm of the flight control board to estimate the yaw angle used to control the landing of the quadcopter. If not, it continues to search for a landing environment that meets the landing characteristics until it is found. Then the ground The station transmits the yaw angle back to the flight control board through the data graphic transmission module, and then controls the DC brushless motor 7, thereby controlling the landing of the quadcopter. After the quadcopter lands to 1-2m, if the landing environment is a slope, a pit or stairs, the flight control board drives several electric push rods 6 at the bottom of the drone. Figure 3 only shows 3 of them. The electric push rod 6 is an existing structure. The thinner support rod is inserted into the wider support rod. The thinner support rod is extended to different lengths to assist the drone to land autonomously on different grounds. If the landing environment is flat, there is no need to start the electric push rod 6, and the quadcopter can land naturally.

Since the process of training the real-time video stream captured by the camera module into four landing environments that can be recognized by the onboard computer is relatively complicated, it will be explained in detail later.

The present invention provides a landing method for a quad-rotor drone based on an improved YOLOv5 algorithm. When detecting the ground environment, the real-time video stream captured by the camera module is trained into four landing environments that can be recognized by an onboard computer, including the following steps:

Step 1: Improve the real-time detection model of the YOLOv5 algorithm and run it under the deep learning framework based on pytorch. First, extract the video stream captured by the camera module as frames of pictures as the test set and training set of the experimental data. Then use LabelImage to annotate the picture data set and divide it into four different ground environments: flat land, slope, pit, and stairs. Annotate them in the Pascal VOC label format and save them as xml format files.

Step 2: Convert the saved annotation file in XML format into an annotation file in TXT format, and divide it into a test set and a training set in a ratio of 1:4.

Step 3: The improved YOLOv5 algorithm is a single-stage detection algorithm. The existing YOLOv5 detection algorithm is divided into 4 versions of different sizes, namely YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. Since the real-time detection is required to be fast and have a small delay, the YOLOv5s model is selected. This model has the smallest network, the fastest speed, and the smallest memory usage, and is small and precise.

In order to adapt to the above annotation files, the relevant code parameters of the model need to be modified in the following steps. After running, you can enter the training code to perform subsequent weight pre-training and formal training. During the pre-training and formal training process, the model uses Mosaic data enhancement to improve the model's training speed and network accuracy, and uses adaptive anchor box calculation and adaptive image scaling methods.

The main process of this step is to adjust the format of each labeled image file in txt format, unify it into a standard size of 640*640*3, and send it to the input end of YOLOv5s.

Step 4: In the benchmark network stage, some common network feature representations are mainly extracted. The existing YOLOv5 algorithm mainly uses the Focus structure and the CSP structure. The Focus structure mainly crops the input image through the Slice operation, and then outputs a 320*320*12 feature map after the Slice operation and the Concat operation, and then passes through a convolution layer with 32 channels to output a 320*320*32 feature map. The CSP structure is mainly to enhance the learning ability of the convolutional neural network, alleviate the problem of a large number of inference calculations, and reduce memory costs and computing bottlenecks.

The present invention uses the Ghost module to replace the CSP structure. The improved network backbone adopts the Ghost bottleneck composed of the Ghost module to replace the three bottleneck structures in the CSP of the original benchmark network stage, and the Ghost module is used to replace the traditional convolution layer. The detection speed of the weight is improved by modifying the convolution method. The Ghost module is divided into three steps: convolution, Ghost generation and feature map splicing. First, the feature map is obtained using the existing convolution method, and then the feature map of each channel is subjected to Φ operation to generate a Ghost feature map. The Φ operation is similar to a 3×3 convolution. Finally, the feature map obtained in the first step and the Ghost feature map obtained in the second step are connected to obtain the final result output, that is, the working principle description of the Ghost module is completed, and then step five is entered.

The main process of this step is to process the image sent to the input end of YOLOv5s, that is, to perform Focus operation and Ghost operation on it, integrate the features of the image and improve the learning ability of the network, and output the integrated image to the next stage.

Step 5: The FPN structure and PAN structure are used in the neck network stage. The FPN structure is top-down, passing down the strong semantic features of the high-level layers, enhancing the entire pyramid structure of the FPN, but only enhancing the semantic information, without passing the positioning information. The PAN structure is aimed at this point, adding a bottom-up pyramid behind the FPN structure to supplement the FPN structure and pass up the strong positioning features of the low-level layers.

The main process of this step is to use the FPN structure and PAN structure to fuse the image information and extract the features of the output image of the baseline network stage, and output three feature prediction maps of different sizes.

Step 6: At the output end of YOLOv5s, the main purpose is to select the prediction box with the highest probability as the actual output, so it involves the selection of the loss function. This part uses CIoU_Loss as the loss function of the bounding box. To calculate CIoU_Loss, you need to calculate GIoU_Loss first. The calculation method is as follows:

First, calculate the minimum enclosed area of the predicted box and the real box, that is, the area of the minimum box that contains both the predicted box and the real box, then calculate IoU, and then calculate the proportion of the area that does not belong to the predicted box and the real box in the minimum enclosed area of the predicted box and the real box to the minimum enclosed area of the predicted box and the real box, and finally subtract this proportion from IoU to get GIoU_Loss.

In the above formula: GIoU_Loss is the loss function of the bounding box (not the bounding box loss function of YOLOv5), IoU is the ratio of the intersection of the predicted box and the true box to the union of the predicted box and the true box, _Ac is the minimum enclosed area of the predicted box and the true box, and U is the area of the predicted box and the true box in the minimum enclosed area of the predicted box and the true box.

DIoU_Loss is more in line with the mechanism of real box regression than GIoU_Loss. It takes into account the distance, overlap rate and scale between the real box and the predicted box, making the real box regression more stable. Its calculation method is as follows:

In the above formula: b, b ^gt represent the center points of the predicted box and the real box respectively, ρ is the Euclidean distance between the two center points, and c is the diagonal distance of the minimum closure area that can contain both the predicted box and the real box. The bounding box loss function CIoU_Loss of YOLOv5 is defined as follows:

Here α is the weight function and v is the impact factor.

For the screening of prediction boxes, NMS (non-maximum suppression) operation is performed to remove the prediction boxes with small prediction probabilities and leave the prediction boxes with large prediction probabilities. This is repeated step by step until the prediction box with the largest prediction probability is left. This is because CIoU_Loss contains the influence factor v, which involves the information of the real box. YOLOv5 uses weighted non-maximum suppression. Compared with non-maximum suppression, weighted non-maximum suppression is to weight the prediction boxes according to the probability predicted by the network during the process of removing the prediction boxes, and obtain a new prediction box, which is used as the final prediction box.

Therefore, the main process of this part is to perform weighted non-maximum suppression on the three different sizes of feature prediction maps output by the neck network stage, select the prediction box with the closest probability to the true box, and output it as the final result. At this point, all processes of the training part are completed.

Step 7: In the actual test, the unknown image is sent to the trained model, and the model determines the type of the image. The specific process does not involve the real frame. The rest of the test steps are the same as steps 1 to 6 of the above training, proving that the trained model is reliable and can run on the onboard computer.

Therefore, the onboard computer first extracts the real-time video stream transmitted by the data graphics transmission module into frames of pictures, and then annotates these pictures according to the above process for conversion and format adjustment. Finally, the obtained annotated image files of uniform size are sent to the trained network model to accurately determine whether the real-time video stream corresponds to flat ground, sloped ground, pit or stairs.

Claims (8)

1. A drone landing method based on an improved YOLOv5 algorithm, characterized in that an electric push rod is installed at the bottom of the drone frame of the drone, and a camera module, a data graphics transmission module, an airborne computer and a flight control board are installed on the drone frame, comprising the following steps: Step 1, the camera module captures the video of the drone's environment, which is then transmitted to the onboard computer via the data graphics transmission module; Step 2: The onboard computer uses the YOLOv5s algorithm, based on the Mosaic data enhancement model, and uses the adaptive anchor frame calculation and adaptive image scaling methods to detect whether the real-time video stream transmitted by the data graphics transmission module corresponds to a flat environment, a slope environment, a pit environment, or a stair environment; Step 3: If the detection result of step 2 is the landing environment of the drone, the onboard computer combines the ground station and the flight control board to generate a yaw angle for controlling the landing of the drone; if the detection result of step 2 is not the landing environment of the drone, repeat steps 1 and 2 until the detection result is the landing environment of the drone, and the onboard computer combines the ground station and the flight control board to generate a yaw angle for controlling the landing of the drone; Step 4: The ground station transmits the yaw angle obtained in step 3 to the flight control board through the data graphic transmission module. The flight control board controls the drone to land. When the drone is 1 to 2 meters away from the ground, if the landing environment is flat, the drone will land naturally. If the landing environment is a slope, pit or stairs, the flight control board drives the electric push rod to extend outward to assist the drone to land in the corresponding landing environment. The step 2 includes the following specific sub-steps: Step 2a, first extract the real-time video stream transmitted by the data graphic transmission module into frames of pictures, then mark these pictures and divide them into flat environment, slope environment, pit environment and stair environment, and finally convert and adjust the format to obtain a uniform-sized marked picture file; Step 2b, sending the annotated image file described in step 2a to the input end of YOLOv5s, performing Focus operation and Ghost operation in sequence to obtain a fused image, fusing information and extracting features of the fused image, and outputting three feature prediction images; Step 2c, performing a non-maximum suppression operation on the feature prediction map described in step 2b to obtain a prediction box that is closest to the true box probability, and obtain a trained network model; Step 2d, the onboard computer first extracts the real-time video stream transmitted by the data graphic transmission module into frames of pictures, then annotates the pictures, converts them, and adjusts their formats, and finally sends the obtained annotated picture files of uniform size to the network model trained in step 2c to determine whether the real-time video stream corresponds to a flat environment, a slope environment, a pit environment, or a stair environment; In step 2c, when performing the non-maximum suppression operation, CIoU_Loss is used as the loss function of the bounding box. CIoU_Loss is shown in the following formula: Among them, IoU is the ratio of the intersection of the predicted box and the true box to the union of the predicted box and the true box, b, ^bgt represent the center points of the predicted box and the true box respectively, ρ is the Euclidean distance between the center points of the predicted box and the true box, c is the diagonal distance of the minimum closure area that contains both the predicted box and the true box, α is the weight function, and v is the influencing factor. 2. According to the unmanned aerial vehicle landing method based on the improved YOLOv5 algorithm of claim 1, it is characterized in that the step 2b uses the FPN structure and the PAN structure to fuse the information and extract the features of the fused picture in turn, and outputs three feature prediction maps. 3. The unmanned aerial vehicle landing method based on improved YOLOv5 algorithm according to claim 1, wherein the camera module is RER-USB4K02AF-V100. 4. The unmanned aerial vehicle landing method based on the improved YOLOv5 algorithm according to claim 1 is characterized in that the camera module is connected to the input end of the onboard computer through a data graphic transmission module, the output end of the onboard computer is connected to the input end of the flight control board, one end of the data graphic transmission module is connected to the input end of the flight control board, and the model of the data graphic transmission module is HM30. 5. The unmanned aerial vehicle landing method based on improved YOLOv5 algorithm according to claim 1, is characterized in that, it also includes a model aircraft battery, and the model aircraft battery is connected to the power supply end of the onboard computer and the power supply end of the flight control board respectively through a 5V2A BEC power supply type ammeter. 6. The unmanned aerial vehicle landing method based on the improved YOLOv5 algorithm according to claim 1 is characterized in that it also includes a GPS positioning module connected to the input end of the flight control board, and the GPS positioning module is used to transmit the collected real-time position information of the unmanned aerial vehicle to the ground station through the communication module. 7. The drone landing method based on the improved YOLOv5 algorithm according to claim 1 is characterized in that the electric push rod is connected to the output end of the flight control board, and there are 3 to 5 electric push rods. 8. The unmanned aerial vehicle landing method based on the improved YOLOv5 algorithm according to claim 1, wherein the model of the flight control board is Pixhawk.