CN111401297A - Triphibian robot target recognition system and method based on edge calculation and neural network - Google Patents

Abstract

An amphibious robot target recognition system and method based on edge computing and neural network, mainly composed of neural network and amphibious robot, a camera collects images to be rescued, and transmits them to an internal edge computing control board through a UART serial port, and has been trained The neural network recognizes the target in the picture, so as to output the target recognition map, and transmit it to the main control board to control the GPS communication module to transmit the coordinates to the terminal, identify the target in depth through the neural network, and transplant the neural network to the edge node, through the edge computing It saves the time of rescue and search, and greatly improves the efficiency of work.

Description

An amphibious robot target recognition system based on edge computing and neural network and its method

technical field

The invention belongs to the field of artificial intelligence image processing, in particular to an amphibious robot target recognition system and method based on edge computing and neural network, uses convolutional neural network for target recognition, and is especially suitable for marine garbage search and marine rescue scenarios.

Background technique

The country and society gradually turn their attention to the development and utilization of the ocean to the development of the national economy. However, the phenomenon of emphasising development over safety is still widespread in our society, with relatively weak safety awareness, insufficient awareness of the risks of maritime activities, low level of ship and shore management, relatively low skill level of personnel, and poor technical conditions of offshore facilities or equipment. In addition, natural disasters such as typhoons, cold waves, and dense fog occur frequently.

The existing maritime search and rescue system and mechanism is not perfect, and there are still many problems in the operation process, such as the weak signal at sea and the inability to get timely contact with the terminal, and the decision-making from the terminal cannot be completed immediately. The active and frequent maritime activities in the future will inevitably increase the probability and frequency of maritime dangerous accidents. Maritime rescue work is facing unprecedented challenges, and the demand for maritime search and rescue will become more urgent and show a diversified trend.

In addition, due to the prosperity of my country's tourism industry, a lot of garbage accumulates on the sea level. Many plastic bottles are materials that are difficult to degrade, which seriously damages the ecological balance of the sea. Therefore, the collection of marine garbage is imminent.

At present, most marine rescue systems use robot-assisted terminal systems to complete rescues, and robots do not have the ability to make independent decisions and cannot determine whether there is a danger signal in the current sea area, and the signal transmission with the terminal often takes time. When the signal at sea is weak, the signal The transmission is too slow, resulting in delayed rescue of personnel.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide an amphibious robot target recognition system and method based on edge computing and neural network, which can overcome the disadvantage that the existing edge nodes do not have the decision-making ability, and is a target recognition system with simple structure and easy realization. , and its identification method is simple and easy to implement, and has certain practicability and development prospects.

The technical scheme adopted in the present invention:

An amphibious robot target recognition system based on edge computing and neural network, characterized in that it includes a neural network target recognition subsystem and an amphibious robot; wherein, the amphibious robot has a main control board, an edge computing control board and a GPS in the inner cabin of the amphibious robot (Global Positioning System, Global Positioning System) communication module; a camera is installed on the outside of the amphibious robot, and a camera module is installed on the lower end of the pan/tilt, and the camera passes the collected original image through the UART (Universal Asynchronous Receiver/Transmitter on the main control board). , Universal Asynchronous Receiver Transmitter) serial port is transmitted to the edge computing control board inside the amphibious robot, and the edge computing control board recognizes the target in the picture through the trained neural network, so as to output the target recognition map, and then transmit the output signal to the main control. A neural network target recognition subsystem trained for target recognition is installed on the edge computing control board, which is used for target recognition of people who are in distress in the sea in the to-be-recognized picture.

The main control board is an STM32F4 chip.

The edge computing control board adopts a heterogeneous structure of "CPU (Central Processing Unit, central processing unit) + GPU (Graphic Processing Unit, graphics processing unit)", which is used to realize the calculation of the neural network, wherein the GPU is used to assist the CPU to perform Speed up computation.

The neural network target recognition subsystem is a convolutional neural network structure, consisting of an input layer, a hidden layer and an output layer; the input layer is connected to the output layer through the hidden layer; the input layer is connected to the output layer through the hidden layer, and Each layer is provided with a fully convolutional feature extractor, and a convolution kernel structure is set inside the corresponding feature extractor.

The number of input layers and hidden layers of the neural network target recognition subsystem is not less than one.

The neural network target recognition subsystem is a system implemented by using the YOLOV3 algorithm; the main frame of the YOLOV3 algorithm is Darknet53, and the two-layer network that is not adjacent is connected by using the shortcut method, which solves the problem caused by the increase in the number of Darknet53 layers. Gradient disappears.

The Darknet53 adopts a combination of full convolution and Residual structure.

The hidden layer cannot be too much or too little. For example, the current YOLOV3 uses 53 layers, and the 53 layers are the current optimal solution after repeated experiments. If the number of hidden layers is too small, the network cannot have the necessary layers. On the contrary, if it is too much, it will not only greatly increase the complexity of the network structure, which is especially important for the network implemented by hardware, the network is more likely to fall into a local minimum point in the learning process, and it will make the network The learning rate becomes very slow.

An amphibious robot target recognition method based on edge computing and neural network is characterized in that it comprises the following steps:

(1) Install the Ubuntu system on the edge computing board, and then use Anaconda to build Tensorflow and conduct deep learning training in the Tensorflow framework to build a neural network target recognition subsystem;

(2) The camera collects image information of the marine distress situation, and transmits the collected image information to the edge computing control panel in the interior cabin of the amphibious robot, and performs image preprocessing on it;

(3) Suppose the image information received by the edge computing control board is a picture of S*S size, and the original pixel picture is output as the current pixel picture through the convolution layer, that is, the original pixel picture passes through the convolution kernel in the convolution layer. The filtering process, the obtained pixel image;

The convolutional layer in the step (3) includes an input layer, a hidden layer and an output layer; the input layer is connected to the output layer through the hidden layer, and each layer is provided with a fully convolutional feature extractor, and the corresponding features A convolution kernel structure is arranged inside the extractor; there are at least two feature layers in the hidden layer, and the input signal of each feature layer receives the output signal of the previous feature layer of the feature layer.

(4) The neural network target recognition subsystem in the control board of the edge computing board first performs deep convolution on the image collected in step (2) in the convolutional layer, and performs dimension reduction processing. The data is represented in a low-dimensional space, and it is expected that the variance of the data in the projected dimension is the largest, so as to achieve the purpose of using less data dimensions and retaining the characteristics of more original data points, and reducing the dimension all the time. to the 13th floor;

(5) The edge computing control board calls the Opencv library to process the image. Since the neural network target recognition subsystem has been built and trained in step (1), the image signal will be used for target recognition and processing. The neural network identifies the people killed at sea in the picture,

In the step (5), the identification of the person in the sea in the picture to be identified specifically refers to: the picture to be identified is segmented by the 53rd layer of Darknet53, and then the target bounding box and the class label are regressed and predicted on the identification sample. If the category label of the person is detected, the signal will be returned to make a decision, and each feature layer after segmentation will output a prediction result to predict which person was killed at sea. Finally, the edge computing control board will predict the event according to the degree of confidence. , get the final prediction result, determine whether there are people in distress at sea in the processing picture, make rescue decisions by the edge computing control board, and send the rescue signal to the main control board through the Uart communication interface on the central control board. The control board receives the signal to enter the interrupt service routine, and then rescue.

The working principle of the present invention: the amphibious robot adopts STM32F407 as the main control board of the amphibious robot, and uses the PID to adjust the speed of the motor to perform the steady-state control of the amphibious robot. Self-stabilization in the air, edge computing performs data processing and analysis in real time or faster, so that data processing is closer to the source, rather than an external data center or cloud. The internal edge computing board of the amphibious robot has trained the neural network data in advance, so The edge computing board can make autonomous decisions, and does not need to be transmitted to the terminal for recognition by the neural network, which gives the amphibious robot the ability to make autonomous decisions. The edge computing control board sends the final decision result obtained in step 3 back to the main control board STM32F4 for robotic rescue operations.

Ordinary cloud computing terminals carry out processing and algorithmic decision-making and then transmit commands to robots, while edge computing pushes intelligence and computing closer to actual actions. The completion of marine rescue and garbage search and rescue through the cooperation of robots will greatly improve efficiency. The communication transmission time with the robot is long, so the use of the characteristics of edge computing makes the robot have the ability to make edge decisions, which greatly improves the efficiency. Due to the introduction of edge computing, data processing and analysis can be performed in real time or faster, making data processing closer to the source, Instead of an external data center or cloud, the delay time can be shortened. After the identification of the victims and marine debris at sea, it does not need to be transmitted to the terminal for decision-making, which doubles the efficiency.

The advantages of the present invention: for marine rescue and marine garbage search, the neural network is used to deeply identify the target, and the neural network is transplanted to the edge node to save rescue and search time through edge computing, and greatly improve work efficiency.

Description of drawings

1 is a schematic diagram of the structure of a neural network in a method for amphibious robot target recognition based on edge computing and neural network according to the present invention (where 1 is an input layer, 2 is a hidden layer, and 3 is an output layer).

Fig. 2 is a kind of image processing convolution layer analysis diagram in a kind of amphibious robot target recognition method based on edge computing and neural network according to the present invention (wherein, 4 is the original pixel picture, 5 is the current pixel picture, 6 is the volume layer.)

FIG. 3 is a schematic diagram of a neural network processing flow in an amphibious robot target recognition method based on edge computing and neural network according to the present invention.

FIG. 4 is a schematic diagram of the overall structure of an amphibious robot target recognition system based on edge computing and neural network according to the present invention.

FIG. 5 is a schematic flowchart of an amphibious robot target recognition method based on edge computing and neural network according to the present invention.

Detailed ways

Embodiment: an amphibious robot target recognition system based on edge computing and neural network, as shown in Figure 4, is characterized in that it includes a neural network target recognition subsystem and an amphibious robot; A control board, an edge computing control board and a GPS communication module; a camera is installed on the outside of the amphibious robot, and a camera module is installed on the lower PTZ, and the camera transmits the collected original image to the interior of the amphibious robot through the UART serial port on the main control board The edge computing control board, the edge computing control board recognizes the target in the picture through the trained neural network, so as to output the target recognition map, and then transmit the output signal to the main control board, the edge computing control board is installed with trained The neural network target recognition subsystem for target recognition is used for target recognition of people who are killed in the sea in the to-be-recognized picture.

The main control board is an STM32F4 chip.

The edge computing control board adopts the heterogeneous structure of "CPU+GPU", which is used to realize the calculation of the neural network, wherein the GPU is used to assist the CPU to perform accelerated calculation.

The neural network target recognition subsystem is a convolutional neural network structure, consisting of an input layer 1, a hidden layer 2 and an output layer 3; the input layer 1 is connected to the output layer 3 through the hidden layer 2, as shown in Figure 1; The input layer 1 is connected to the output layer 3 through the hidden layer 2, and each layer is provided with a fully convolutional feature extractor, and the corresponding feature extractor is provided with a convolution kernel structure inside.

The number of input layer 1 and hidden layer 2 of the neural network target recognition subsystem is not less than one.

The neural network target recognition subsystem is a system implemented by using the YOLOV3 algorithm; the main frame of the YOLOV3 algorithm is Darknet53, and the two-layer network that is not adjacent is connected by using the shortcut method, which solves the problem caused by the increase in the number of Darknet53 layers. Gradient disappears.

The Darknet53 adopts a combination of full convolution and Residual structure.

The hidden layer 2 cannot be too much or too little. For example, the current YOLOV3 uses 53 layers, and the 53 layers are the current optimal solution after repeated experiments. If the number of hidden layers 2 is too small, the network cannot It has the necessary learning ability and information processing ability; on the contrary, if it is too much, it will not only greatly increase the complexity of the network structure, which is especially important for the network implemented by hardware. The learning speed of the network becomes very slow.

An amphibious robot target recognition method based on edge computing and neural network, as shown in Figure 3 and Figure 5, is characterized in that it includes the following steps:

(1) Install the Ubuntu system on the edge computing board, and then use Anaconda to build Tensorflow and conduct deep learning training in the Tensorflow framework to build a neural network target recognition subsystem;

(2) The camera collects image information of the marine distress situation, and transmits the collected image information to the edge computing control panel in the interior cabin of the amphibious robot, and performs image preprocessing on it;

(3) Suppose the image information received by the edge computing control board is a picture of size S*S, and the original pixel picture 4 is output as the current pixel picture 5 through the convolution layer 6, that is: the original pixel picture 4 is in the convolution layer. 6 After the filtering process of the convolution kernel, the obtained pixel picture 5 is shown in Figure 2;

The convolution layer 6 in the step (3) includes an input layer 1, a hidden layer 2 and an output layer 3; the input layer 1 is connected to the output layer 3 through the hidden layer 2, and each layer is provided with a full convolution Feature extractor, a convolution kernel structure is arranged inside the corresponding feature extractor; there are at least two feature layers in the hidden layer 2, and the input signal of each feature layer receives the signal of the previous feature layer of the feature layer. output signal.

(4) The neural network target recognition subsystem in the control board of the edge computing board first performs deep convolution on the image collected in step (2) in the convolution layer 6, and performs dimension reduction processing. The data is mapped to a low-dimensional space, and the variance of the data in the projected dimension is expected to be the largest, so as to achieve the purpose of using less data dimensions and retaining the characteristics of more original data points, and has been reducing Dimension to 13 layers;

(5) The edge computing control board calls the Opencv library to process the image. Since the neural network target recognition subsystem has been built and trained in step (1), the image signal will be used for target recognition and processing. The neural network identifies the people killed at sea in the picture,

In the step (5), the identification of the person in the sea in the picture to be identified specifically refers to: the picture to be identified is segmented by the 53rd layer of Darknet53, and then the target bounding box and the class label are regressed and predicted on the identification sample. If the category label of the person is detected, the signal will be returned to make a decision, and each feature layer after segmentation will output a prediction result to predict which person was killed at sea. Finally, the edge computing control board will predict the event according to the degree of confidence. , get the final prediction result, determine whether there are people in distress at sea in the processing picture, make rescue decisions by the edge computing control board, and send the rescue signal to the main control board through the Uart communication interface on the central control board. The control board receives the signal to enter the interrupt service routine, and then rescue.

The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only descriptive, not restrictive, and cannot limit the protection scope of the present invention.

An amphibious robot target recognition system and method based on edge computing and neural network, the system is mainly composed of a neural network and an amphibious robot, a camera module is installed on the outside of the amphibious robot, and the camera module passes the collected original image through UART The serial port is transmitted to the edge computing control board inside the amphibious robot, and the edge computing control board recognizes the target in the picture through the trained neural network, thereby outputting the target recognition map, and then transmitting the output signal to the main control board STM32F4, the main control board STM32F4 It will control the GPS communication module to transmit the coordinates to the terminal.

Build the tensorflow framework through ANACONDA and build the convolutional neural network in tensorflow, and transplant the neural network model to the edge computing control board to realize edge computing, reduce rescue time, and use YOLOV3 algorithm to improve the accuracy of target recognition. First, the camera It will collect image information and transmit the information to the edge computing control board in the amphibious robot for target recognition. First, it will receive S*S size pictures, and it will reduce the dimension to 13 layers through deep convolution;

Darknet53 has a total of 53 layers of convolution, except for the last fully connected layer, which is actually implemented by 1x1 convolution, a total of 52 convolutions are used as the main network, first is a convolution layer kernel of 32 filters, and then It is 5 groups of repeated residual units. Each of these 5 groups of residual units consists of a separate convolutional layer and a set of repeated convolutional layers. The repeated convolutional layers are repeated 1 time, 2 times, 8 times, 8 times, 4 times; in each repeated convolution layer, first perform a 1x1 convolution operation, and then perform a 3x3 convolution operation, the number of filters is first halved, and then restored, a total of 52 floors. The residual calculation is not part of the convolutional layer calculation. 52=1+(1+1*2)+(1+2*2)+(1+8*2)+(1+8*2)+(1+4*2) A single convolutional layer operation is a convolution operation with a step size of 2, so the entire YOLOV3 network has a total of 5 dimensionality reductions of 32 times, that is: 32=2^5, and the final output feature map dimension is 13, so This is the dimensionality reduction to 13 layers;

The Darknet53 framework has a fully convolutional feature extractor in the three layers of 52, 26, and 13, and the corresponding internal convolution kernel structure of the feature extractor. Multiple convolution kernels are interleaved to achieve the purpose, but there is no essential difference. to layer 13; the input of the current feature layer has part of the output from the previous layer. Each feature layer has an output prediction result, and finally performs linear regression on the result according to the confidence level to obtain the final prediction result, and finally transmits the obtained target extraction information back to the main control board STM32F4 for command control.

Although the embodiments and drawings of the present invention are disclosed for illustrative purposes, those skilled in the art will appreciate that various substitutions, changes and modifications are possible without departing from the spirit and scope of the invention and the appended claims Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims (10)

1. A triphibian robot target recognition system based on edge calculation and neural network is characterized in that the triphibian robot target recognition system comprises a neural network target recognition subsystem and a triphibian robot; the triphibian robot inner cabin is internally provided with a main control board, an edge calculation control board and a GPS communication module; the triphibian robot outside is installed camera lower extreme cloud platform and is installed the camera module, the original image that the camera will gather transmits the marginal calculation control panel of triphibian robot inside through the UART serial ports on the main control board, and the marginal calculation control panel carries out the target recognition in the picture through the neural network that trains well to output target identification picture, then transmit output signal for the main control board, install the neural network target identification subsystem that is used for target recognition through training on the marginal calculation control panel for treat among the discernment picture just carry out target recognition at the people that the sea area meets difficulties.

2. The triphibian robot target recognition system based on edge computing and neural network as claimed in claim 1, wherein said master control board is STM32F4 chip; the edge calculation control panel adopts a CPU + GPU heterogeneous structure and is used for realizing calculation of a neural network, wherein the GPU is used for assisting the CPU to perform accelerated calculation.

3. The triphibian robot target recognition system based on edge computing and neural network as claimed in claim 1, wherein the neural network target recognition subsystem is a convolutional neural network structure, consisting of an input layer, a hidden layer and an output layer; the input layer is connected with the output layer through the hidden layer; the input layer is connected to the output layer through the hidden layer, each layer is provided with a full convolution feature extractor, and the corresponding feature extractor is internally provided with a convolution kernel structure.

4. The triphibian robot target recognition system based on edge computing and neural network as claimed in claim 3, wherein the number of input layers and hidden layers of the neural network target recognition subsystem is no less than 1.

5. The triphibian robot target recognition system based on edge computing and neural network as claimed in claim 4, wherein the hidden layer is a 53-layer structure.

6. The triphibian robot target recognition system based on edge computing and neural network as claimed in claim 3, wherein said neural network target recognition subsystem is a system implemented by using YO L OV3 algorithm, and said YO L OV3 algorithm body frame is Darknet 53.

7. The triphibian robot target recognition system based on edge computing and neural networks as claimed in claim 5, wherein said Darknet53 is in the form of a combination of full convolution and Residual structure.

8. A triphibian robot target recognition method based on edge calculation and neural network is characterized by comprising the following steps:

(1) installing an Ubuntu system on an edge computing board, then building a Tensorflow inside an Anaconda and performing deep learning training on a Tensorflow framework, thereby building a neural network target recognition subsystem;

(2) carrying out image information acquisition on the marine distress condition by a camera, transmitting the acquired image information to an edge calculation control panel in an inner cabin of the triphibian robot, and carrying out image preprocessing on the image information;

(3) and (3) setting the image information received by the edge calculation control board as a picture with the size of S × S, and outputting the original pixel point picture as the current pixel point picture through the convolution layer, namely: filtering the original pixel point picture in the convolution layer by a convolution kernel to obtain a pixel point picture;

(4) carrying out deep convolution on the picture collected in the step (2) in a convolution layer by a neural network target recognition subsystem in an edge calculation board control board, carrying out dimension reduction treatment, mapping high-dimensional data to a low-dimensional space for representation through linear projection, and expecting that the variance of the data on the projected dimension is maximum, so as to achieve the purpose of using less data dimensions and retaining the characteristics of more original data points and directly reducing the dimension to 13 layers;

(5) and (3) calling an Opencv library by the edge computing control board to process the image, and since the construction and training of the neural network target recognition subsystem are already carried out in the step (1), the image signal can be used for recognizing and processing the target in the image signal, and people at sea in distress in the image can be recognized by the neural network.

9. The method for identifying a triphibian robot target based on edge computing and neural network as claimed in claim 4, wherein said convolutional layer in step (3) comprises an input layer, a hidden layer and an output layer; the input layer is connected to the output layer through the hidden layer, each layer is provided with a full convolution feature extractor, and a convolution kernel structure is arranged inside the corresponding feature extractor; the hidden layer has at least 2 characteristic layers, and the input end signal of each characteristic layer receives the output signal of the last characteristic layer.

10. The triphibian robot target recognition method based on edge computing and neural network as claimed in claim 4, wherein the step (5) of recognizing the people in distress at sea in the picture to be recognized specifically means: the method comprises the steps of dividing a picture to be recognized by 53 layers of Darknet53, then predicting a target boundary frame and a class label on a recognition sample in a regression mode, if the class label of a person is detected, returning a signal to make a decision, outputting a prediction result by each divided feature layer, predicting which person is a person in distress at sea, finally predicting an event by an edge calculation control board according to the confidence degree to obtain a final prediction result, determining whether the person in distress at sea exists in the processed picture, making a rescue decision by the edge calculation control board, sending a rescue signal to a main control board through a Uart communication interface on the central control board, and enabling the central control board to enter an interrupt service program after receiving the signal and then rescue.

Images