CN111824406A - A public safety autonomous inspection quadrotor UAV based on machine vision - Google Patents

Abstract

The invention discloses a public safety autonomous inspection quadrotor unmanned aerial vehicle based on machine vision, which belongs to the field of aircraft and consists of a power system, a flight control system, a data processing system and a mobile terminal system. When the system is running, the target model uploaded by the mobile terminal system to the data processing system will analyze and process the real-time image data obtained by the image acquisition module to obtain the target type in the image. In order to judge whether the scene is a cluster of suspicious and dangerous people, and then scan the facial data of the approaching people one by one, obtain the body temperature data through the infrared thermal imager to judge whether the body temperature is normal, and at the same time compare the external features of the face with the pictures in the criminal face database. A comparison is made to determine whether they are fugitives.

Description

A public safety autonomous inspection quadrotor UAV based on machine vision

technical field

The invention belongs to the field of aircraft, and in particular relates to a quadrotor unmanned aerial vehicle for autonomous inspection of public safety based on machine vision.

Background technique

With the rapid development of UAV technology, small and medium-sized UAVs are used in a variety of scenarios due to their portable size, such as search and rescue, aerial photography, natural resource surveys, wild hunting, target monitoring and reconnaissance, etc. Small quad-rotor UAVs can obtain more environmental information by virtue of their advantages of easy control and flexible aerial activities, and their application scenarios are much more than other aircraft. The application of quadrotor UAV based on machine vision includes target tracking, traffic management, air navigation, etc., but at present, the realization of these functions is inseparable from the operator, the degree of automation is very limited, the application of image recognition technology in artificial intelligence to achieve human UAV applications for behavior recognition are even rarer. Only some target recognition technologies use the ground station to process image information and can recognize the target. Although this method can handle complex models, the time delay is much higher than that of the airborne image processing board to process image information.

At present, there are patent documents that propose implementation methods for UAVs in the field of security, such as "a multi-rotor UAV dynamic security system" (publication number CN207078318U) and "a UAV-based security system" ” (publication number CN110766907A), the data of the environment monitored by the UAV is obtained through the UAV equipped with the FPV image system and some functional modules, and then the data is sent to the ground station for the operator to judge whether it is safe. The above two patents use a data transmission system to transmit all the obtained information to the ground station for operators to judge whether it is safe or not. The process of processing the collected images is done manually. This implementation method is time-consuming and labor-intensive, and does not reflect the Autonomous and intelligent drones.

There are also patent documents that propose an implementation method of using face recognition technology to identify criminals and draw criminal whereabouts maps. ), the implementation method of the system described in this patent is to use the facial image information collected by fixed cameras installed on roads and streets to compare with the facial data of criminals provided by the police, so as to find the traces of criminals. Using a large number of fixed cameras to collect face data itself is an operation with a huge amount of information, and it takes a long time from collecting the criminal face data to processing the image data to find the criminal. Compared with the method of discovering a human target and then actively acquiring the target face data, the face data collection method described in this patent has low efficiency, high passivity and high randomness. Secondly, the area covered by the fixed camera is limited, and it has little effect on criminals with high anti-reconnaissance awareness. To sum up, the method for finding criminal traces adopted by the patent implementation method has low efficiency, low reliability, poor adaptability to the environment, and low practicability.

SUMMARY OF THE INVENTION

The purpose of the present invention is to propose a public safety autonomous inspection quadrotor drone based on machine vision, which can automatically search for suspicious persons or fugitive criminals in real-time around the clock, and automatically track and automatically monitor the body temperature of human targets. The ability to detect and autonomously detect clusters of suspicious persons. The realization of these functions does not require human intervention, and can be completed autonomously by the drone itself.

The technical scheme of the present invention is as follows: a public safety autonomous inspection quadrotor UAV based on machine vision, which is composed of a power system, a flight control system, a data processing system and a mobile terminal system; it is characterized in that:

The power system is used to provide flight power for the UAV and power supply for the flight control system and data processing system;

The flight control system is used to control the flight of the quadrotor UAV and provide a stable platform for the UAV to obtain the target image;

The data processing system is used to acquire and process image data, establish communication with the mobile terminal system, and send motion control instructions to the flight control system; the data processing system acquires image data through a high-definition camera and processes it, and can autonomously identify clusters of suspicious persons and automatically find suspicious persons. Targets or criminals can control the flight control system to automatically track the target after finding the target, and can also automatically measure the forehead temperature of the human target through an infrared thermal imager;

The mobile terminal system is used to complete human-computer interaction and information transmission with the data processing system.

Further, the fuselage structure of the four-rotor UAV is composed of a frame, a frame base, and a landing gear from top to bottom. On the end corners, the first quadrilateral plate is suspended on the four rotating shafts; the top of the four rotating shafts is set with a frame, the frame is an X-shaped frame, and the bottom ends of the four rotating shafts are connected to the second quadrilateral plate, the second quadrilateral plate is connected to the bottom of the four rotating shafts. The lower end of the quadrilateral plate is connected to the landing gear; the power system and the flight control system are arranged on the upper end of the first quadrilateral of the frame base, and the data processing system is arranged at the lower end of the first quadrilateral of the frame base.

Further, the power system is composed of a brushless motor, a blade, a high-rate model aircraft lithium battery and a power supply voltage stabilizer board; a brushless motor is installed at the four ends of the frame, and each brushless motor is fixed on the There are paddles, the high-rate lithium battery is arranged on the upper end of the first quadrilateral board of the frame base, and the power voltage stabilizer plate is vertically arranged on the side ends of the first quadrilateral and the second quadrilateral, and the height is balanced with the first quadrilateral.

Further, the flight control system is composed of an embedded flight microcontroller, an IMU integrated sensor, a monocular ultrasonic sensor, an optical flow sensor, and a GPS antenna; the upper part of the rack is suspended with an embedded flight microcontroller, and the embedded flight micro The controller is equipped with an IMU integrated sensor, and a monocular ultrasonic sensor and an optical flow sensor are arranged in parallel at the lower end of the high-rate model lithium battery and the side end of the power supply voltage stabilizer plate. The GPS antenna is extended and suspended at the top through one of the rotating shafts. Among them, the IMU integrated sensor is integrated by a six-axis motion sensor, a magnetometer and a barometer.

Further, the data processing system is composed of an image processing module, an image acquisition device and a wireless signal transceiver, and is used for acquiring and processing image data, establishing communication with the mobile terminal system, and sending motion control instructions to the flight control system; image The acquisition device includes a three-axis self-stabilizing pan/tilt, a high-definition camera, an infrared fill light and an infrared thermal imager, which are used to collect image data; the image processing module is arranged on the upper end of the second quadrilateral and below the first quadrilateral; the second quadrilateral The lower end of the block quadrilateral is provided with a bracket on which a three-axis self-stabilizing pan/tilt, a high-definition camera and an infrared thermal imager are placed. The image processing module uses deep learning image processing algorithms (Lu Jian, He Jinxin, Li Zhe, et al. Overview of object detection based on deep learning [J]. Electro-Optics and Control, 2020, 27(5): 56-63. DOI: 10.3969 /j.issn.1671-637X.2020.05.012.) Process image data, establish communication with the mobile terminal system through the wireless signal transceiver, and send motion control commands to the flight control system through the serial port; the data processing system collects images through the After the device obtains and processes image data, it can autonomously identify suspicious person clusters and automatically find suspicious targets or criminals. After finding the target, it can control the flight control system to automatically track the target. In addition, it can also automatically measure the forehead of the human target through an infrared thermal imager. temperature.

Further, the mobile terminal system is composed of an intelligent terminal and a wireless signal transceiver, which is used to complete human-computer interaction and information transmission with the data processing system. The human-computer interaction includes acquiring the deep learning target image model, the collected target face image data and the control instructions from the operator to the drone through the input device, and processing the image data and image data from the data processing system through the output device. The latter data and UAV status data are displayed to the operator, and the operator is reminded by vibration and playing a prompt tone. The deep learning target image model is a convolutional neural network model containing the depth features of the target image obtained after deep learning of the target image by the deep learning image processing algorithm (Zhou Junyu, Zhao Yanming. A review of the application of convolutional neural networks in image classification and target detection[ J]. Computer Engineering and Application, 2017, 53(013):34-41.), which can be used for target detection and recognition. The information transmission with the data processing system includes sending the information obtained through the input device in the human-computer interaction to the data processing system through the wireless signal transceiver and receiving the image data from the data processing system, the data after image processing and the state of the drone data.

Further, the image processing module can independently identify and track the target offline.

Further, the ability to autonomously identify and track the target offline through the image processing module is specifically:

(1) Training of the deep learning target image model: The convolutional neural network built by the deep learning image processing algorithm performs deep learning on the target image training set containing the target image to obtain the deep learning target image model containing the depth features of the target image, namely A network model containing the depth features of the target image;

(2) Target recognition: The image processing module loads the deep learning target image model obtained after training, and the image acquisition device acquires video data in real time and transmits it to the image processing module. A picture data set is formed, and then the image processing module performs noise filtering and color format conversion on the picture data set to obtain a picture data set containing target features. The confidence of the target and the pixel height, width and pixel coordinate information of the target image in the original image;

(3) Target tracking: The target tracking algorithm adopts the template matching method. The feature template is the target image with high confidence after target recognition. The matching method is the square difference matching method or the correlation coefficient matching method, that is, the feature template and the searched image are matched. Correlation is measured by the squared difference or correlation coefficient between image data matrices.

Further, the specific process of the template matching method used in the target tracking is:

After the image processing module obtains the target feature template, it searches for the image most similar to the feature template in the first frame of pictures collected by the image acquisition device, and outputs the center pixel coordinates and pixel size of the image to the flight control system. , the flight control system uses the pixel coordinates of the image to control the aircraft or the three-axis self-stabilizing pan/tilt to move to the state where the center of the target image remains in the middle of the screen, and uses the relative size of the pixel size of the target image to control the distance between the aircraft and the target. The image processing module adopts Kalman filter (Welch G. Kalman Filter.[J]. Siggraph Tutorial, 2001.) or particle filter (Gool, Luc. Object Tracking withan Adaptive Color-Based Particle Filter[C]/ / 2002:353-360.) to predict the position of the target in the image for the next frame;

While predicting the target position of the next frame, the image processing module updates the image searched in the first frame to a new feature template to search for the target in the next frame of pictures. If the target image is not searched in the first frame of pictures, continue Use the previous feature template to search for the next frame of pictures, until no image similar to the feature template is found in the set number of pictures, re-target the current frame of the picture to obtain a new target feature template ;

In order to solve the tracking drift over time, that is, the position of the tracked target image has a certain deviation from the actual target position, the image processing module uses the target tracking algorithm to process the picture for a period of time and will automatically re-detect the target.

The steps for realizing the functions of automatically finding suspicious persons or criminals and automatically tracking them, automatically performing body temperature safety detection on human targets, and autonomously detecting clusters of suspicious persons are as follows:

(1) You can choose to use a high-performance computer to run a deep learning image processing algorithm, learn a large number of images of human targets to obtain a deep learning target image model of the human target, and then combine the deep learning target image model of the human target with suspicious persons or criminals at the same time. The face data of the target is uploaded from the mobile terminal system to the data processing system of the UAV;

(2) When the UAV is working, after the image acquisition device obtains the image, the image processing module processes and identifies the number of human targets in the image. If the number of human targets is too large, it is judged as a cluster of suspicious persons;

(3) After being identified as a human target, the drone actively moves to a position suitable for collecting face images, uses an infrared thermal imager to obtain the temperature of the human forehead, and uses a high-definition camera to collect face image data. If the temperature of the human forehead exceeds the set threshold , will issue an alarm, and send the collected face image data and the latitude and longitude coordinate data of the drone to the mobile terminal system to judge whether the body temperature is normal, and use the uploaded face data and collected images of suspicious persons or criminals. The data is compared for face recognition. After face recognition, it is determined as a suspicious person or criminal. The drone sends the collected image data and the longitude and latitude coordinate data of the drone to the mobile terminal system and starts to automatically track suspicious persons. If a person or criminal is identified as a non-suspicious person or criminal after face recognition, the same identification will be performed on the next person target.

The beneficial effects of the present invention are: the technology adopts the offline working mode, and does not need to transmit the image data with huge amount of data and complex processing to the ground station for processing through image transmission, but uses the image processing module of the unmanned aerial vehicle to complete the image processing module. The feature recognition of the target in the image can not only avoid many problems such as lost data and delay in the wireless data transmission, but also enable the drone itself to complete the functions required by the present invention in real time, and improve the sensitivity of the drone system. , response speed and scene adaptability. The image acquisition device of the present invention uses a high-definition camera and an infrared thermal imager at the same time, and turns on the infrared fill light at night, so that the high-definition camera can be used to realize the operation of the drone in bright and dark environments, and the infrared thermal imager can also be used. Obtaining the target temperature greatly reduces the impact of the environment on the drone's work. The UAV adopts the mobile terminal system to upload the deep learning target image model and the face image data of suspicious persons or criminals. After the UAV obtains multiple deep learning target image models, the UAV will automatically find the target, which improves the automation and intelligence of the UAV. In the process of recognizing the target face of the person, the UAV adopts the implementation method of actively moving to a suitable position to obtain the face image data, which improves the probability of finding dangerous targets such as criminals.

Description of drawings

Fig. 1 is the unmanned aerial vehicle system structure diagram of the present invention;

Fig. 2 is the front view of the UAV structure of the present invention;

3 is a bottom view of the UAV structure of the present invention;

Fig. 4 is the left side view of the UAV structure of the present invention;

Fig. 5 is the right side view of the UAV structure of the present invention;

Fig. 6 is the UAV safety inspection flow chart of the present invention;

Fig. 7 is the motion control principle diagram of the present invention;

Description of reference numerals: In Figures 2, 3, 4, and 5, the main view direction is the side of the fuselage, that is, the direction facing the camera, 1—power system; 2—flight control system; 3—data processing system; 4—mobile terminal system ;5—frame; 6—frame base; 7—landing gear, 8—rotating shaft, 9—first quadrilateral plate, 10, second quadrilateral plate; 101—brushless motor; 102—blade; 103 —High rate lithium battery for model aircraft; 104—Power voltage regulator board; 201—Embedded flight microcontroller; 202—IMU integrated sensor composed of six-axis motion sensor, magnetometer and barometer; 203—Monocular ultrasonic sensor; 204— Optical flow sensor; 205—GPS antenna; 301—Image processing module; 302—Three-axis self-stabilizing pan/tilt; 303—HD camera; 304—Infrared thermal imager.

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings and examples. As shown in FIGS. 1-7 , the machine vision-based public safety autonomous inspection quadrotor UAV designed by the present invention includes a power system 1 , a flight control system 2 , a data processing system 3 and a mobile terminal system 4 . The fuselage structure of the quadrotor UAV is composed of a frame 5, a frame base 6, and a landing gear 7 from top to bottom. The frame base 6 is two quadrilateral plates, and four rotating shafts 8 are interspersed in the four sides of the two quadrilateral plates. On the end corners, the first quadrilateral plate 9 is suspended on the four rotating shafts 8; the top of the four rotating shafts 8 is erected with a frame 5, the frame 5 is an X-shaped frame, and the bottom ends of the four rotating shafts 8 are connected to the second. A quadrilateral plate 10, the lower end of the second quadrilateral plate 10 is connected to the landing gear 7; the power system 1 and the flight control system 2 are arranged on the upper end of the first quadrilateral 9 of the frame base 6, and the data processing system 3 is arranged on the frame base 6. The lower end of the first quadrilateral 9 .

The power system 1 is composed of a brushless motor 101, a blade 102, a high-rate model aircraft lithium battery 103 and a power supply voltage regulator board 104; a brushless motor 101 is installed at the four ends of the frame 5, and each brushless motor The blade 102 is fixed on the 101, the high-power model aircraft lithium battery 103 is arranged on the upper end of the first quadrilateral plate 9 of the frame base 6, and the power voltage stabilizer plate 104 is vertically arranged on the side of the first quadrilateral 9 and the second quadrilateral 10 end, height and balance of the first block quadrilateral 9.

The power system 1 is used to provide flight power for the UAV, as well as power supply for the flight control system 2 and the data processing system 3 . It includes a power unit, a high-rate lithium battery 103 , a propeller blade 102 , and a power stabilizer plate 104 . Among them, the power device can be an electric drive system with a high-rate model aircraft battery 103 paired with a brushless motor 101 and an ESC, or a direct-drive oil-driven system, which is specifically composed of a fuel tank, an engine, a fuel feeder, a pitch control device, The steering gear is composed, and the power supply voltage stabilization board 104 is used to stabilize the power supply voltage provided by the high-rate lithium battery 103 of the model aircraft to 5V DC voltage to drive the flight control system 2 and the data processing system 3 .

The flight control system 2 consists of an embedded flight microcontroller 201, an IMU integrated sensor 202, a monocular ultrasonic sensor 203, an optical flow sensor 204, a GPS antenna 205, and an acoustic or optical prompting device; the upper part of the frame 5 is suspended in the air with an embedded flight control system. Microcontroller 201, an IMU integrated sensor 202 is arranged on the embedded flight microcontroller 201, a monocular ultrasonic sensor 203 and an optical flow sensor 204 are arranged in parallel at the lower end of the high-rate model lithium battery 103 and the side end of the power supply voltage stabilization board 104, The GPS antenna 205 is extended and suspended at the top through one of the rotating shafts 8, wherein the IMU integrated sensor 202 is integrated by a six-axis motion sensor, a magnetometer and a barometer.

The flight control system 2 is used to control the flight of the quadrotor UAV and provide a stable platform for the UAV to acquire target images. It includes an embedded flight microcontroller 201, an altitude measurement device, an attitude sensor, a data receiving and sending device, a horizontal displacement sensor, a navigation system, and an acoustic or optical prompting device. The flight microcontroller can use FPGA based platforms (Field-Programmable Gate Array), that is, field programmable gate array platforms, or ARM-based platforms, or Atmel-based platforms or Raspberry Pi based platforms. The altitude measurement device is a device that can measure the altitude of the drone. It can use an ultrasonic distance sensor, a barometer, a laser ranging device or a GPS device, or a device that combines various methods to obtain altitude information. The attitude sensor may be a nine-axis attitude sensor composed of a six-axis accelerometer and a three-axis gyroscope. The data receiving and sending device is used to receive the motion control instructions of the UAV, and can use a radio transceiver device of the WiFi protocol or the Bluetooth protocol, or a digital radio communication device or a mobile communication device. The horizontal displacement sensor is used to obtain the horizontal displacement data of the UAV to keep the horizontal stability of the UAV, and an optical flow sensor can be used. The navigation system is used to obtain the longitude, latitude and altitude information of the UAV, and adopts the GPS navigation system. The acoustic or optical prompting device is a device used to send out prompt acoustic or optical information, which can be an optical output device or an acoustic output device, or a combination of the two methods. Here, a speaker and light output device is used.

The data processing system 3 is composed of an image processing module 301 , a three-axis self-stabilizing pan/tilt 302 , a high-definition camera 303 and an infrared thermal imager 304 ; the image processing module 301 is arranged on the upper end of the second quadrilateral 10 and below the first quadrilateral 9 ; The lower end of the second quadrilateral 10 is provided with a bracket, and the bracket is placed with a three-axis self-stabilizing pan/tilt 302, a high-definition camera 303 and an infrared thermal imager 304.

The data processing system 3 is used for acquiring and processing image data, establishing communication with the mobile terminal system 4 and sending motion control commands to the flight control system 2 . The system includes an image processing module 301, an image acquisition device, and a wireless signal transceiver, wherein the wireless signal transceiver can use the data receiving and sending device solution described in the flight control system. The image processing module 301 is used to process image data and establish communication with the mobile terminal system 4. Specifically, NVIDIA's Jetson AI supercomputer, Baidu's EdgeBoard deep learning computing card, Arduino open source hardware board, Huawei HiSilicon artificial intelligence AI can be used. Platform or Shenjian Technology DP-8000 AI development board. The image acquisition device includes a self-stabilizing gimbal, a high-definition camera, an infrared fill light, and an infrared thermal imager. The self-stabilizing gimbal is used to provide a stable platform for the camera and the infrared thermal imager. The monocular camera can also use a multi-eye camera, and the infrared fill light can be turned on at night. No matter the environment is bright or dark, the image data can be obtained through the high-definition camera, so that the drone can work all day. Target image for temperature data.

The mobile terminal system 4 is composed of intelligent terminals and wireless signal transceivers, and is used to complete human-computer interaction and information transmission with the data processing system 3 . The wireless signal transceiver can not only receive the data sent by the on-board data processing system 3 but also send the operation data of the operator. The smart terminal can be a computer or a smart phone installed with smart applications, with input and output devices. Smart applications will provide the operator interface. The specific implementation process is that after the mobile terminal system 4 receives the data sent by the data processing system 3 through the wireless signal transceiver, it will display it to the operator through the application program, and remind the operator by vibrating and playing the prompt sound. The instruction of whether to continue tracking is transmitted to the intelligent terminal through the application program, and the intelligent terminal sends the operation instruction data to the airborne data processing system 3 through the wireless signal transceiver.

The process flow when the present invention works is as follows:

(1) The operator inputs the target face image data and the deep learning target image model through the mobile terminal device, and formulates the flight mission, including planning the flight route, setting the target, and setting the task content and priority. The task content has detection After identification and tracking of escaped criminals, cluster identification of suspicious persons, identification and tracking of persons with abnormal body temperature, and data input and setting, the mobile terminal system 4 sends the task information to the data processing system 3 .

(2) The data processing system 3 will start the self-check of the system after receiving the task information and getting the start instruction. If the self-check is wrong, the data processing system 3 will feed back the error information to the mobile terminal system 4; if the self-check is completed without error , the drone will automatically start and start flying according to the set route.

(3) After the UAV is started, the data processing system 3 sends the route data to the flight control system 2, and the flight control system 2 will control the UAV to fly according to the set route.

(4) During the flight, the data processing system 3 collects the image data of the surrounding environment, and uses the deep learning image processing algorithm to identify the human target in the image, and calculates the density of people, judges whether there is a suspicious person cluster, and sends the identification information. Go to the mobile terminal system 4, and then the operator confirms whether there is a cluster of suspicious people. If the operator is not online, the drone directly calibrates the GPS position of the cluster target (that is, the longitude and latitude position data of the drone at this time), and saves the cluster target. Then send the image information to the mobile intelligent terminal, and continue to fly according to the set route after the execution is completed.

(5) If the tasks of identifying and tracking fugitives and people with abnormal body temperature are set, after the data processing system 3 recognizes it as a human target, it will also calculate the best path to move to the position suitable for obtaining face data, and send flight control instructions to The flight control system 2, after flying to a position suitable for obtaining face data, uses the target face data uploaded by the mobile terminal system 4 to compare with the obtained target face data. If it is confirmed that the target is a fugitive criminal, the data processing system 3 will Send the target image processing result to the mobile terminal system 4 to remind the operator. If the operator is offline, the drone will automatically track the criminal target until the operator issues a cancellation command; while scanning the target's face data, the infrared thermal imager will also acquire The infrared image data of the face, and the temperature of the forehead of the human body is measured. If the temperature is abnormal, the data processing system 4 will also send the target image data, the data after image processing, and the temperature of the human forehead to the mobile terminal system 4. Whether the abnormal forehead temperature target is detected Tracking can be set in task settings, or can be set in real time in the mobile terminal system 4 .

(6) After completing the task according to the route, the drone will automatically return to home.

The operation process of a public safety autonomous inspection quadrotor UAV based on machine vision is shown in Figure 6. When the drone is working, first turn on the power switch on the power supply voltage regulator board 104, and then the drone starts self-checking and initialization of each system. At this time, the three-axis self-stabilizing gimbal 302 also returns to the initial position. After the initialization is completed, the flight control system 2 will prompt with lights and speaker prompts. After the mobile terminal system 4 confirms the start, the drone will automatically take off to the set safe altitude. If the automatic mode has been turned on and the working data has been uploaded, no The man-machine will start to control the three-axis self-stabilizing gimbal 302 to start looking for the target, otherwise it will hover in the air and wait for the command. After the automatic identification tracking mode is turned on, the data processing system 3 will control the three-axis self-stabilizing pan/tilt 302 to rotate to find the target, and the image captured by the high-definition camera 303 will be transmitted to the image processing module 301 in real time. Use the trained model to detect the information. When the target in the picture is identified, the target motion model is quickly established according to the pixel coordinates of the center position of the target in the front and rear frame images and the pixel size information of the target, and is passed through Kalman filter or A filter such as a particle filter predicts the pixel coordinate position of the target in the next frame, and calculates the motion control data of the UAV, and then sends it to the embedded flight microcontroller 201 . The embedded flight microcontroller 201 quickly controls the fuselage to a suitable position where the target can be tracked to obtain target image information. After the image data of the high-definition camera 303 and the infrared thermal imager 304 are acquired, the image processing module 301 performs feature recognition on the acquired image information to determine whether the person is a suspect or a fugitive criminal; whether it is a cluster of suspicious persons; whether the body temperature is Normal; the body temperature is obtained by measuring the temperature of the face and forehead by the infrared thermal imager 304 . If the processing results are valuable, such as identifying criminal suspicious targets, detecting suspicious dangerous cluster behaviors, and finding people with abnormal body temperature, the data processing system 3 will send the image data of the suspicious target, the image recognition data of the image processing module 301, and the drone. GPS position data to the mobile terminal system 4 . After receiving the data, the mobile terminal system 4 presents the data to the operator through the application interface, and the operator confirms whether to perform tracking. If there is no operator's instruction, the drone will track the fugitive criminal in real time and continuously transmit the data to the mobile terminal system 4 .

The flight control principle of the quadrotor unmanned aerial vehicle is shown in FIG. 7 , and the flight control system 2 is composed of a cascade double closed-loop PID controller. The outer loop is position feedback, which can feedback the current yaw angle. The purpose of the outer loop control is to achieve the desired angle. The output of the outer loop PID is used as the angular velocity to the inner loop as the input, that is, the expected value of the inner loop. The inner loop is attitude feedback, and its purpose is to achieve the desired angular velocity, that is, the input of the outer loop, and the output of the inner loop is the speed control parameter of the brushless motor. With the control of these two rings, the movement and positioning of the drone will be much smoother. The output formula of the double closed-loop PID controller is as follows, out(t) is the output formula of the PID controller of the continuous system, out(k) is the output formula of the PID controller of the discrete system, and err(k) is the corresponding value at time k The deviation between the expected value and the actual value, Kp, Ki, Kd are the proportional coefficient, integral coefficient, and differential coefficient, respectively, and T is the data update time period:

During the specific implementation, the quadrotor UAV obtains motion data and attitude data through the attitude sensor of the flight control system 2; at low altitude, the UAV’s height above the ground is obtained from the data obtained by various altitude measurement devices after passing through the Kalman filter. It can be concluded that at high altitude, it is obtained by the GPS module; when the drone is safely inspected, the image processing module 301 in the data processing system 3 uses the trained model to process the image information of each frame of the video in real time, and the image will be obtained after processing. The number of human bodies and the density of people in the model are obtained after deep learning of a large number of labeled target image training sets by convolutional neural networks. The density of people is calculated from the number of human bodies and the relative distance between the human body and the drone, where The relative distance between the human body and the drone is obtained by dividing the number of pixels occupied by the human body and the total number of pixels in the image; the drone will judge whether there is a dangerous group of people through the density of people; The human body target is subjected to face recognition, and the face model data is collected by the operator and uploaded to the mobile terminal system 4, and the mobile terminal system 4 is then sent to the data processing system 3. At the same time as the face recognition, the human body target will be obtained through the infrared thermal imager 304. The head temperature data is obtained, so as to realize the identification of fugitive criminals and the identification of people with abnormal body temperature. When the environment is bright or dark, the image data is obtained through the high-definition camera 303 and the infrared fill light to realize the all-day work of the drone.

The above is a preferred example of the specific implementation of the present invention, but the present invention is not limited to the content disclosed in the example and the accompanying drawings. Therefore, all equivalents or modifications accomplished without departing from the spirit disclosed in the present invention fall into the protection scope of the present invention.

Claims (9)

1. A public safety autonomous inspection quad-rotor unmanned aerial vehicle based on machine vision is composed of a power system, a flight control system, a data processing system and a mobile terminal system; the method is characterized in that:

the power system is used for providing flight power for the unmanned aerial vehicle and providing power supplies for the flight control system and the data processing system;

the flight control system is used for controlling the flight of the quad-rotor unmanned aerial vehicle and providing a stable platform for the unmanned aerial vehicle to acquire a target image;

the data processing system is used for acquiring and processing image data, establishing communication with the mobile terminal system and sending a motion control instruction to the flight control system; the data processing system can automatically identify suspicious personnel clusters and automatically search suspicious targets or criminals after acquiring and processing image data through the high-definition camera, can control the flight control system to automatically track the targets after finding the targets, and can also automatically measure the forehead temperature of a human body target through the thermal infrared imager;

the mobile terminal system is used for completing human-computer interaction and information transmission between the mobile terminal system and the data processing system.

2. A machine vision based public safety autonomous patrol quad-rotor drone according to claim 1; the method is characterized in that: the four-rotor unmanned aerial vehicle body structure comprises a rack, a rack base and an undercarriage from top to bottom, wherein the rack base is two quadrilateral plates, four rotating shafts are inserted into four corners of the two quadrilateral plates, and a first quadrilateral plate is suspended on the four rotating shafts; the top ends of the four rotating shafts are provided with a rack in an X-shaped mode, the bottom ends of the four rotating shafts are connected with a second quadrilateral plate, and the lower end of the second quadrilateral plate is connected with an undercarriage; the power system and the flight control system are arranged at the upper end of the first quadrangle of the rack base, and the data processing system is arranged at the lower end of the first quadrangle of the rack base.

3. A machine vision based public safety autonomous patrol quad-rotor drone according to claim 2; the method is characterized in that: the power system consists of a brushless motor, blades, a high-magnification model airplane lithium battery and a power supply voltage stabilizing plate; a brushless motor is installed respectively to four tip of frame, is fixed with the paddle on every brushless motor, and high magnification model aeroplane and model ship lithium cell sets up the first quadrilateral plate upper end at the frame base, and the power voltage regulator plate sets up at first quadrangle and the quadrangle side of second, and the height is balanced with first quadrangle.

4. A machine vision based public safety autonomous patrol quad-rotor drone according to claim 2; the method is characterized in that: the flight control system consists of an embedded flight microcontroller, an IMU integrated sensor, a monocular ultrasonic sensor, an optical flow sensor and a GPS antenna; the upper portion of the rack is provided with an embedded flying microcontroller in a suspension mode, the embedded flying microcontroller is provided with an IMU integrated sensor, the lower end of the high-magnification aeromodelling lithium battery and the side end of the power supply voltage stabilizing plate are provided with a monocular ultrasonic sensor and a light stream sensor in a parallel mode, the GPS antenna is arranged at the topmost end in a suspension mode through lengthening of one of the rotating shafts, and the IMU integrated sensor is integrated by a six-axis motion sensor, a magnetometer and a barometer.

5. A machine vision based public safety autonomous patrol quad-rotor drone according to claim 2; the method is characterized in that: the data processing system consists of an image processing module, a three-axis self-stabilizing holder, a high-definition camera and a thermal infrared imager; the image processing module is arranged at the upper end of the second quadrangle and below the first quadrangle; the lower end of the second quadrangle is provided with a support, and the support is used for placing a three-axis self-stabilizing holder, a high-definition camera and a thermal infrared imager.

6. A machine vision based public safety autonomous patrol quad-rotor drone according to claim 2; the method is characterized in that: the mobile terminal system is composed of an intelligent terminal and a wireless signal receiver-transmitter, the mobile terminal system receives data sent by the data processing system through the wireless signal receiver-transmitter and then displays the data to an operator, the operator is reminded through vibration and a prompt tone playing mode, whether a continuously tracking instruction is transmitted to the intelligent terminal after the operator receives information, and the intelligent terminal sends operation instruction data to an airborne data processing system through the wireless signal receiver-transmitter.

7. A public safety autonomous patrol quad-rotor unmanned aerial vehicle based on machine vision according to claim 5, characterized in that the target can be autonomously identified and tracked off-line through an image processing module.

8. A machine vision based public safety autonomous patrol quad-rotor drone according to claim 7, characterized in that: the method can be used for carrying out autonomous offline identification and tracking on the target through the image processing module, and specifically comprises the following steps:

(1) training of a deep learning target image model: performing deep learning on a target image training set containing a target image by adopting a convolutional neural network established by a deep learning image processing algorithm to obtain a deep learning target image model containing a target image depth feature, namely a network model containing the target image depth feature;

(2) target identification: the image processing module loads a deep learning target image model obtained after training, simultaneously the image acquisition device acquires video data in real time and transmits the video data to the image processing module, the image processing module converts the video data into pictures of each frame to form a picture data set, then the image processing module performs noise filtering and color format conversion on the picture data set to obtain a picture data set containing target characteristics, and each picture obtained after the processing of the deep learning target image model of the picture data set contains the confidence of a target and the pixel height, width and pixel coordinate information of the target image in an original image;

(3) target tracking: the target tracking algorithm adopts a template matching method, the characteristic template is a target image with high confidence after target identification, and the matching method is a square error matching method or a correlation coefficient matching method, namely, the correlation between the characteristic template and the searched image is measured by using the square error or the correlation coefficient between image data matrixes.

9. A machine vision based public safety autonomous patrol quad-rotor unmanned aerial vehicle according to claim 8, wherein: the specific process of the template matching method used in target tracking is as follows:

searching an image most similar to the characteristic template in a first frame of picture acquired by the image acquisition device at the moment after the image processing module acquires the target characteristic template, outputting the central pixel coordinate and the pixel size of the image in the picture to a flight control system, controlling the airplane or a three-axis self-stabilizing pan-tilt to move to a state that the center of the target image is kept at the middle position of the picture by the flight control system by using the pixel coordinate of the image, controlling the airplane and the target to keep a proper distance by using the relative size of the pixel size of the target image, and predicting the position of the next frame of target in the image by using a Kalman filter or a particle filter by the image processing module;

when the target position of the next frame is predicted, the image processing module updates the image searched by the first frame into a new characteristic template so as to search the target in the next frame of picture, if the target image is not searched in the first frame of picture, the previous characteristic template is continuously utilized to search the next frame of picture, and when the image similar to the characteristic template is not searched in the set pictures with a certain frame number, the target recognition is carried out on the picture of the current frame again so as to obtain the new target characteristic template;

in order to solve the problem that tracking drift is generated along with the time, namely the position of a tracked target image has certain offset with the position of an actual target, the image processing module automatically detects the target again after processing the image for a period of time by using a target tracking algorithm.

Images