CN111508002A - A small low-flying target visual detection and tracking system and method thereof - Google Patents

Abstract

The invention discloses a small low-flying target visual detection and tracking system and a method thereof. The system comprises: a video data input unit, a video preprocessing unit, a training data construction unit, a detection model training unit, a target comparison and screening unit, and a detection and correction unit , a reference frame initialization unit, a sample library dynamic construction unit, an online learning unit, a position refinement unit, a decision control unit and a tracking result output unit. The method includes: target detection network construction, target comparison and screening; target tracking online learning; dynamic construction of a classifier training sample library, target tracking position refinement; Robust target tracking can be achieved by the occurrence of tracking drift conditions caused by factors such as these. It has the ability to update the reference frame features in time according to the target change, and at the same time, the introduction of the feature point matching algorithm can avoid the wrong tracking caused by updating the reference frame features.

Description

A small low-flying target visual detection and tracking system and method thereof

technical field

The invention relates to the technical field of flight target tracking, in particular to a small low-flying target detection and tracking system based on neural network and online learning and a detection and tracking method thereof.

Background technique

At present, there are some related methods of joint detection and tracking and some tracking methods suitable for low-slow and small targets. Existing methods achieve short-term target tracking through correlation filtering methods, and use neural network-based target detection methods to achieve the function of relocating targets when tracking fails. The relevant patents and research technologies are as follows:

Chinese invention patent, titled: A robust long-term tracking method based on correlation filtering and target detection, application number: CN201910306616.4; the invention realizes target tracking through correlation filtering method, and uses one-stage target detector YOLO for target detection , after obtaining the detection results, the SURF feature point matching method is used to select the candidate frame with the highest number of matching points as the target bounding frame of the re-initialized tracker, and finally achieve the effect of long-term tracking. However, this method does not take into account the extreme imbalance between the target and the background in the detector, and is not suitable for long-term tracking of small targets.

Chinese invention patent, titled: A low-altitude slow-speed UAV tracking method combining correlation filtering and visual saliency, application number: 201910117155.6; it uses correlation filtering in a small search area to obtain a predicted response map to determine the predicted target The center position of , and the saliency detection method is used to determine the scale of the predicted target in a large search area, and a tracking method suitable for low-altitude and slow UAVs is realized. However, this method does not perform further processing after the target tracking fails, and the accuracy needs to be further improved.

Combining the above existing technologies, none of them solves the problem of extreme imbalance of target background in single target detection and tracking, the network performance has not yet reached the optimum, and the tracking accuracy for small targets still has room for further improvement.

SUMMARY OF THE INVENTION

Aiming at the defects of the prior art, the present invention provides a small low-flying target visual detection and tracking system and a method thereof, and solves the defects in the prior art.

In order to realize the above purpose of the invention, the technical scheme adopted by the present invention is as follows:

A small low-flying target detection and tracking system, comprising: a video data input unit, a video preprocessing unit, a training data construction unit, a detection model training unit, a target comparison and screening unit, a detection and correction unit, a reference frame initialization unit, and a sample library Dynamic construction unit, online learning unit, position refinement unit, decision control unit and tracking result output unit.

The video data input unit is used for: inputting several video sequences containing the target, and randomly dividing them into two parts, one part is used for training the target detection model, and the other part is used for the online test of the target tracking model.

The video preprocessing unit is used to: complete the preliminary video preprocessing work according to the needs of the target detection and tracking unit, specifically including deleting video clips that have no targets for a long time, and eliminating small targets that obviously do not meet the low-altitude airspace and fly at a slow speed. Features video clips, etc.

The training data construction unit is used to: ensure the completeness and richness of the training data, construct a training set and a verification set by extracting video frames at equal intervals, and perform data annotation, that is, determine the center position and width of the target in the image. and high information for supervised training of object detection models.

The detection model training unit is used for: creating a pyramid-structured object detection network, and using focal loss to alleviate the problem of object-background imbalance. Stop the training process after observing that the training loss function becomes stable, and save the model file with the best performance during the verification process, which is used to provide reset box information when target tracking fails.

The target comparison and screening unit is used to: use the SURF feature matching algorithm to compare the true value frame of the first frame of the tracked video with the detection result, eliminate false alarms that are obviously not low-slow and small targets, and further ensure long-term stable tracking. robustness.

The detection and correction unit is used to: start the detection and correction unit when the following two situations occur: one is when the confidence level of the tracking frame position is lower than the set threshold, indicating that the current frame target tracking fails, then the detection and correction unit is started; The detection and correction unit is automatically activated at the specified frame interval to ensure that the current tracking target does not differ too much from the reference frame target features.

The reference frame initialization unit is used to: according to the received target position and scale information of the reference frame, cut out the search area by 5 times the size of the target size, and zoom to the specified size 288*288 through the image, and the search area of the specified size is The regional image patch and target location scale information are input to the classifier to dynamically construct the sample library.

The sample library dynamic construction unit is used for: performing data enhancement on the reference frame samples, including basic operations of rotation, scaling, jittering and blurring, and receiving newly added samples during the tracking process.

The online learning unit uses the deep network ResNet18 network to perform feature extraction on the samples stored in the sample library, and then obtains the predicted Gaussian response map after two fully connected layers. The peak point generates the true Gaussian label, in order to narrow the gap between the predicted Gaussian response map and the true Gaussian label as the optimization goal, the parameters of the two fully connected layers are adjusted online to achieve the feature extraction network and the fully connected layer without labels. To achieve the purpose of predicting the Gaussian response map and obtaining the center of the predicted target location.

The position refinement unit is used to: perform position refinement on the initial tracking result obtained by the online learning unit, and obtain a number of The jitter frame is mapped to the search area of the current frame. The precise region of interest pooling layer is used to extract the features of the jitter frame, and the modulation features obtained from the reference frame are spliced. Finally, the confidence of the predicted position of each jitter frame is obtained through the fully connected layer. , and merge the results of the three jitter boxes with the highest confidence, which is the tracking result of the current frame after refinement.

The decision control unit is used to: judge the target tracking state through the relationship between the predicted position confidence of the tracking frame and the set threshold during the tracking process, if the target is stably tracked, then continue to track the next frame, , the detection and correction unit is activated to detect the target of the current frame and update the reference frame to achieve long-term stable tracking of low-slow and small targets.

The tracking result output unit is used for outputting the position and scale information of each frame after traversing all the frames of the video.

The invention also discloses a small low-flying target visual detection and tracking method, comprising the following steps:

Step 1, target detection network construction;

1) Construct the network structure, including: backbone network, classification subnet and regression subnet;

2) Loss function, use focus loss to solve the problem of serious imbalance of positive and negative samples in target detection, and reduce the weight of simple negative samples in training.

Step 2, target comparison and screening;

Before sending it to the detection and correction unit, perform a SURF feature point matching, and perform feature matching between the first frame target of the current video and the detection result. When the number of matching points is greater than the set value, it indicates that the detection result is indeed what needs to be tracked. The target indicates that the detection is successful. At this time, the detection frame is sent to the detection and correction unit for the subsequent process.

Step 3, target tracking online learning;

Predicting the center position of the target includes two parts: initialization classifier and online classification process:

1) Initialize the classifier

For the data-enhanced reference frame in the sample library, the feature extraction network is used to extract features, and at the same time, the target center position of the reference frame is used as the peak value to generate a two-dimensional Gaussian ground truth label ygt of the same size as the feature map, and the classifier is initialized according to the features and labels. Use the least squares optimization algorithm to minimize the distance between the actual value and the true value, and use the Gauss-Newton iteration method to solve the nonlinear least squares problem; the formula is as follows:

In equation (1), x∈{1,...,M} represents the horizontal coordinate of the target center point of the reference frame, M is the width of the feature map; y∈{1,...,N} represents the target center point of the reference frame Vertical coordinate, N is the height of the feature map; σ is the Gaussian bandwidth.

2) Online classification process

(

表示全连接层的映射函数；weight₁,weight₂表示全连接层的权重系数矩阵)，最大响应位置即为当前帧估计的目标中心坐标(x_t，y_t)。在线训练分类器充分考虑被跟踪目标和背景区域，期间不断的更新分类器来估计目标的位置。According to the previous frame (frame t-1) tracking result (x _t-1 , y _t-1 , w _t-1 , h _t-1 ) (where (x _t-1 , y _t-1 ) is the estimated target (w _t-1 , h _t-1 ) is the width and height of the estimated target), the center position of the target in the previous frame is the center of the target search area of the current frame (frame t), and the width is expanded according to the specified ratio k, High, generate the current frame search area (x _t-1 , y _t-1 , k*w _t-1 , k*h _t-1 ). Then, a feature extraction network is used to extract the features f _t of the search area, and after two fully connected layers, a predicted Gaussian response map that is consistent with the size of the search area is generated

(

Represents the mapping function of the fully connected layer; weight ₁ and weight ₂ represent the weight coefficient matrix of the fully connected layer), and the maximum response position is the estimated target center coordinate (x _t , y _t ) of the current frame. The online training classifier fully considers the tracked target and the background area, and the classifier is continuously updated to estimate the position of the target.

Step 4: Dynamically build a classifier training sample library

1) Set the reference frame update interval to T, and call the target detection unit to update the reference frame when the current frame number t is divisible by T, clear all outdated samples in the sample library, and re-initialize the classifier with the updated reference frame. As the tracking process progresses, the newly generated samples are added to the sample library in turn, the purpose is to make the characteristics of the target in the sample library more similar to the sample currently being tracked, which is conducive to accurately estimating the center position of the target.

2) When the predicted position confidence of the tracking frame is less than the set threshold, it indicates that the current frame target tracking fails, and the decision control unit sends the information that the reference frame needs to be re-initialized to the detection and correction unit, and the detection and correction unit receives the detection of the target detection unit. After the frame information, the information is sent to the reference frame initialization unit for data enhancement and other operations, and finally sent to the dynamically constructed sample library.

Step 5, target tracking position refinement;

It includes two parts: feature extraction network and similarity evaluation network:

1) Feature extraction network

The feature extraction uses the ResNet18 network, which balances the retention of the previous template information and the update of the current reference frame information, provides the neural network with features that combine the current and historical states of the target, and improves the stability of tracking. The characteristics of the search area are extracted from the three parts of the image frame at the moment, and sent to the precise area of interest pooling layer respectively, which is used for the similarity evaluation network to calculate the confidence of the predicted position.

2) Similarity evaluation network

The core of the similarity evaluation network is the precise region of interest pooling layer. Its input consists of two parts. The first part is to perform bilinear interpolation on the image feature map extracted by the network with an interpolation coefficient of IC.

IC(x, y, i, j)=max(0,1-|x-i|)×max(0,1-|y-j|) (2)

Map the discrete feature map to a continuous space and get the feature map f(x,y)

并除以矩形框面积，得到的精准感兴趣区域池化(PrROI Pooling)In equations (2) and (3), (x, y) is the center coordinate of the feature map, (i, j) is the coordinate index on the feature map, wi _{, j} is the position (i, j) on the feature map corresponding weights. The second part of the input is the upper left corner coordinates (x ₂ , x ₁ ) and the lower right corner coordinates (y ₂ , y ₁ ) of the rectangle. According to the obtained continuous spatial feature map and the coordinates of the rectangular frame, the precise region of interest pooling operation is performed, and the target features on the image are preserved to the greatest extent, so as to prepare for further comparison of the similarity between the reference target and the historical frame target. Finally, double-integrate the feature map f(x,y)

And divide it by the area of the rectangular box to get the precise region of interest pooling (PrROI Pooling)

After obtaining the features of the accurate region of interest pooling layer, the three features of the reference frame, the intermediate frame and the current frame are spliced, and input to the fully connected layer to output the final position confidence. Compare the similarity between the candidate target and the historical target, and find the largest similar target as the tracking result.

Compared with the prior art, the advantages of the present invention are:

1) The present invention combines detection and tracking methods, which can effectively alleviate the occurrence of tracking drift caused by occlusion, scale change, illumination and other factors of the tracking target, and can achieve robust target tracking.

2) It has the ability to update the reference frame features in time according to the target change, and at the same time, the introduction of the feature point matching algorithm can avoid the erroneous tracking caused by updating the reference frame features.

3) It is suitable for long-term stable tracking of low, slow and small targets in optical aerial remote sensing images.

Description of drawings

1 is a structural block diagram of a small low-flying target detection and tracking system according to an embodiment of the present invention;

2 is a structural diagram of a target detection network according to an embodiment of the present invention;

3 is a flow chart of online learning according to an embodiment of the present invention;

FIG. 4 is a flow chart of position refinement according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below according to the accompanying drawings and examples.

As shown in Figure 1, a small low-flying target visual detection and tracking system includes the following units:

(1) Video data input unit. Several video sequences containing targets are input and randomly divided into two parts, one part is used for training the target detection model, and the other part is used for online testing of the target tracking model.

(2) Video preprocessing unit. According to the needs of the target detection and tracking unit, the preliminary video preprocessing work is completed, including the deletion of video clips that have no target for a long time, and the removal of video clips that obviously do not meet the characteristics of low-slow and small targets.

(3) Training data construction unit. In order to ensure the completeness and richness of the training data, the training set and the verification set are constructed by extracting video frames at equal intervals, and the data is annotated, that is, the center position and width and height of the target in the image are determined, which is used for supervised training. Train an object detection model.

(4) Detection model training unit. Due to the large scale variation of low, slow and small flying targets and the extreme class imbalance in the single target training process, a target detection network with pyramid structure is designed, and focus loss is used to alleviate the problem of target background imbalance. Stop the training process after observing that the training loss function becomes stable, and save the model file with the best performance during the verification process, which is used to provide reset box information when target tracking fails.

(5) Target comparison screening unit. In view of the possibility of false alarms in the results of target detection, the SURF feature matching algorithm is used to compare the ground truth box of the first frame of the tracked video with the detection results, and eliminate false alarms that are obviously not low, slow and small targets, and further ensure long-term stable tracking. robustness.

(6) Detection and correction unit. When the following two situations occur, the detection and correction unit is activated. One is when the confidence level of the tracking frame position is lower than the set threshold, indicating that the current frame target tracking fails, and the detection and correction unit is activated; Ensure that the current tracking target and the reference frame target features are not too far apart.

(7) Reference frame initialization unit. According to the received target position and scale information of the reference frame, the search area is cut out according to 5 times of the target size, and the image is scaled to the specified size 288*288, and the specified size of the search area image block and target position scale information, etc. Input to the classifier dynamically builds the sample library.

(8) Dynamic construction unit of sample library. Perform data augmentation on base frame samples, including basic operations such as rotation, scaling, jittering, and blurring, and receive newly added samples during tracking.

(9) Online learning unit. Use the deep network ResNet18 to extract the features of the samples saved in the sample library, and then obtain the predicted Gaussian response map after two fully connected layers. At the same time, according to the label information in the sample library, the target center is the peak point of the Gaussian distribution to generate the true value Gaussian label, in order to narrow the gap between the predicted Gaussian response map and the true Gaussian label as the optimization goal, the parameters of the two fully connected layers are adjusted online to achieve the predicted Gaussian response map and the fully connected layer without labels. Get the purpose of predicting the center of the target location.

(10) Position refinement unit. Perform position refinement on the initial tracking results obtained by the online learning unit, take the predicted target position center obtained by the online learning unit of the current frame and the target scale width and height obtained in the previous frame as reference, obtain several jitter boxes, and map them to the current frame search On the area, the precise region of interest pooling layer is used to extract the features of the jitter frame, and the modulation features obtained from the reference frame are spliced. Finally, the confidence of the predicted position of each jitter frame is obtained through the fully connected layer, and the three jitters with the highest confidence are jittered. The results of the boxes are merged, that is, the tracking results of the current frame after refinement.

(11) Decision control unit. During the tracking process, the target tracking state is judged by the relationship between the predicted position confidence of the tracking frame and the set threshold. If the target is tracked stably, the tracking of the next frame is continued. If it has been lost, the detection and correction unit is activated to perform the current Frame target detection, and update the reference frame to achieve long-term stable tracking of low-slow and small targets.

(12) Tracking result output unit. After traversing all frames of the video, output the position and scale information of each frame.

A small low-flying target visual detection and tracking method, comprising the following steps:

Step 1, target detection network construction;

The detection network structure for low-slow and small targets mainly includes two parts, one is the neural network structure of the multi-level pyramid structure, and the other is the loss function that alleviates the extreme imbalance of the target background during the training process:

1) Network structure

Backbone network:

The backbone network for object detection uses feature pyramid levels P3 to P7. P3 and P4 in the backbone network are composed of two parts, one of which is calculated from the output of the corresponding feature extraction network ResNet (C3 and C4) through lateral connections, and the other is a top-down approach to deep small-size feature maps. Upsampling reaches the same size as the shallow layer, and then performs stacking and convolution operations so that the number of output channels is still 256. Although the number of channels of the feature map remains unchanged, the information it has is increasing. P5 is computed from the output of the corresponding feature extraction network ResNet (C5) via lateral connections. P6 and P7 are obtained on the basis of C5 through convolutional layers and activation layers, where P1 has a resolution 21 lower than the input image ⁽ ₁ represents the level of the pyramid), and all pyramid levels have C=256 channels.

Classified subnets:

The object classification subnet predicts the probability of object existence at each spatial location for every A = 9 anchors and K object classes. This subnet is a small fully convolutional network connected to each pyramid level of the backbone network, and all pyramid levels share the parameters of this subnet. From a C-channel input feature map at a given pyramid level, the subnet applies four 3×3 convolutional layers, each with C=256 filters and Relu activation function, followed by a A 3×3 convolutional layer, with a sigmoid activation function attached to each spatial position at the end, outputs KA binary predictions.

Return subnet:

The bounding box regression subnet runs in parallel with the object classification subnet, with another small fully convolutional network appended after each pyramid level to regress the offset of each anchor box to possible nearby ground-truth objects. The design subnet of the bounding box regression is the same as the classification subnet, except that it has 4A linear outputs at each spatial position. For A anchor boxes centered at each spatial position, the meaning of 4 is 4 output prediction anchors The relative offset between the coordinate positions of the upper left and lower right corners of the box and the corresponding positions of the ground truth box. The object classification subnet and the box regression subnet share a common structure but use separate parameters. Figure 2 shows the main structure of the target detection network.

2) Loss function

Use Focal Loss (FL)

FL(p _t )=-α _t (1-p _t )γlog(p _t ) (5)

Solve the problem that the proportion of positive and negative samples in target detection is seriously unbalanced, and reduce the weight of simple negative samples in training. In formula (5), FL(.) represents the focal loss; p _t represents the discriminant function value of the target classification probability; α represents the balance factor used to balance the uneven proportion of positive and negative samples; γ represents the classification of difficult and easy samples The balance factor of γ is greater than 0, which will reduce the loss of easy-to-classify samples and make the model pay more attention to the learning of difficult and misclassified samples. Equation (6) is the expression of the probability discriminant function, y is the predicted output label after the activation function, and the value is between 0-1; p represents the probability value of the target belonging to the label labeling category.

Step 2, target comparison and screening;

In order to prevent the tracking failure caused by using background information to update the reference frame when there is no target in the image or the target is detected as a false alarm, before sending it to the detection and correction unit, a SURF feature point matching is performed, and the first SURF feature point of the current video is matched. The frame target and the detection result are feature-matched. When the number of matching points is greater than the set value, it means that the detection result is indeed the target that needs to be tracked, and the detection is successful. At this time, the detection frame is sent to the detection and correction unit for the subsequent process.

Step 3, target tracking online learning;

As shown in Figure 3, the online learning of target tracking realizes the function of predicting the center position of the target in real time during the tracking process. It mainly includes two parts: initialization classifier and online classification process:

1) Initialize the classifier

For the data-enhanced reference frame in the sample library, use the feature extraction network to extract features, and at the same time use the target center position of the reference frame as the peak value to generate a two-dimensional Gaussian ground truth label y _gt of the same size as the feature map,

Initialize the classifier according to the features and labels, use the least squares optimization algorithm to minimize the distance between the actual value and the true value, and use the Gauss-Newton iteration method to solve the nonlinear least squares problem. The basic idea of the Gauss-Newton iteration method is to use the Taylor level The number expansion is used to approximate the nonlinear regression model, and then through multiple iterations, the regression coefficients are revised multiple times, so that the regression coefficients continue to approach the optimal regression coefficients of the nonlinear regression model, and finally the residual sum of squares of the original model is minimized.

2) Online classification process

最大响应位置即为当前帧估计的目标中心坐标(x_t，y_t)。在线训练分类器充分考虑被跟踪目标和背景区域，期间不断的更新分类器来估计目标的位置。As shown in Figure 3, according to the previous frame tracking results (x _t-1 , y _t-1 , w _t-1 , h _t-1 ), where (x _t-1 , y _t-1 ) is the estimated target Center coordinates, (w _t-1 , h _t-1 ) are the width and height of the estimated target, the center position of the target in the previous frame is the center, the width and height are expanded according to the specified ratio k, and the search area is generated from the current frame (x _t-1 , y _t-1 , k*w _t-1 , k*h _t-1 ), and then use the feature extraction network to extract the feature f _t of the search area, and generate a predicted Gaussian with the same size as the search area after two fully connected layers response graph

The maximum response position is the estimated target center coordinates (x _t , y _t ) in the current frame. The online training classifier fully considers the tracked target and the background area, and the classifier is continuously updated to estimate the position of the target.

Step 4: Dynamically build a classifier training sample library

In view of the complex motion state of low, slow and small flying targets, the target scale and target shape will change greatly during the tracking process. If only the first frame of the video is used as the supervision frame, it is obvious that there are dynamic changes that cannot adapt to the target in real time, so we adopt The following two steps to further alleviate this problem are as follows:

One is to set the update interval of the reference frame to T. When t is divisible by T, call the target detection unit to update the reference frame, clear all outdated samples in the sample library, and re-initialize the classifier with the updated reference frame. The newly generated samples are added to the sample library in turn, the purpose is to make the characteristics of the target in the sample library have a high similarity with the sample currently being tracked, which is conducive to accurately estimating the center position of the target.

Second, when the confidence of the predicted position of the tracking frame is less than the set threshold, it means that the target tracking of the current frame fails, and the decision control unit sends the information that the reference frame needs to be re-initialized to the detection and correction unit, and the detection and correction unit receives the detection of the target detection unit. After the frame information, the information is sent to the reference frame initialization unit for data enhancement and other operations, and finally sent to the dynamically constructed sample library.

Step 5, target tracking position refinement

As shown in Figure 4, the position and scale information of the current frame tracking frame is finally determined according to the classifier predicts the center position of the target and the width and height of the target in the previous frame, which is the main function of the target tracking position refinement. It mainly includes two parts: feature extraction network and similarity evaluation network:

1) Feature extraction network

The feature extraction still uses the ResNet-18 network. In order to make full use of the historical information, balance the preservation of the previous template information and the update of the current reference frame information, to provide the neural network with features that combine the current and historical states of the target, and to improve the stability of tracking. The features of the search area are extracted from the current frame, the reference frame and the image frame in the middle of the current frame, and are respectively sent to the precise area of interest pooling layer for the similarity evaluation network to calculate the confidence of the predicted position.

2) Similarity evaluation network

The core of the similarity evaluation network is the precise region of interest pooling layer. Its input consists of two parts. One is the image feature map extracted by the network, where (i, j) are the coordinates on the feature map, and w _{i, j} are The weight of the corresponding position (i, j) on the feature map, using bilinear interpolation with the interpolation coefficient IC

IC(x, y, i, j)=max(0,1-|x-i|)×max(0,1-|y-j|) (8)

Mapping discrete feature maps into a continuous space

The second is the coordinates (x ₂ , x ₁ ) and (y ₂ , y ₁ ) of the upper left and lower right corners of the rectangular box. According to the obtained continuous space feature map and the coordinates of the rectangular frame, the precise region of interest pooling operation is performed,

Retain the target features on the image to the greatest extent, and prepare for further comparison of the similarity between the reference target and the historical frame target. After obtaining the features of the accurate region of interest pooling layer, the three features of the reference frame, the intermediate frame and the current frame are spliced, and input to the fully connected layer to output the final position confidence. Compare the similarity between the candidate target and the historical target, and find the largest similar target as the tracking result.

Those of ordinary skill in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation method of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations without departing from the essence of the present invention according to the technical teaching disclosed in the present invention, and these modifications and combinations still fall within the protection scope of the present invention.

Claims (2)

1. A small low-flying target detection and tracking system, comprising: the system comprises a video data input unit, a video preprocessing unit, a training data construction unit, a detection model training unit, a target comparison screening unit, a detection correction unit, a reference frame initialization unit, a sample library dynamic construction unit, an online learning unit, a position fine modification unit, a decision control unit and a tracking result output unit;

the video data input unit is configured to: inputting a plurality of video sequences containing targets, and randomly dividing the video sequences into two parts, wherein one part is used for training a target detection model, and the other part is used for online testing of a target tracking model;

the video pre-processing unit is configured to: finishing the preliminary video preprocessing work according to the needs of a target detection and tracking unit, specifically comprising deleting video segments without targets for a long time, eliminating video segments obviously not conforming to the characteristics of small targets flying at a slow speed per hour in a low-altitude airspace, and the like;

the training data construction unit is configured to: ensuring the completeness and richness of training data, constructing a training set and a verification set by extracting video frames at equal intervals, and carrying out data labeling, namely determining the central position, width and height information of a target in an image, wherein the information is used for a supervised training target detection model;

the detection model training unit is used for: creating a target detection network with a pyramid structure, and relieving the problem of unbalanced target background by using focus loss; stopping the training process after observing that the training loss function tends to be stable, storing a model file with optimal performance in the verification process, and providing reset frame information when the target tracking fails;

the target alignment screening unit is used for: comparing the first frame true value frame of the tracked video with the detection result by using a SURF (speeded up robust features) feature matching algorithm, eliminating false alarms which are obviously not low-slow small targets, and further ensuring the robustness of long-term stable tracking;

the detection correction unit is used for: starting a detection and correction unit when the following two conditions occur, namely starting the detection and correction unit when the position confidence coefficient of a tracking frame is lower than a set threshold value, which indicates that the target tracking of the current frame fails; secondly, automatically starting a detection and correction unit when a specified frame number interval is reached, and ensuring that the difference between the current tracking target and the target characteristics of the reference frame is not too large;

the reference frame initialization unit is configured to: cutting out a search area according to the received target position and scale information of the reference frame and according to the size 5 times of the target size, zooming the image to the specified size 288 x 288, and inputting the image blocks of the search area with the specified size, the scale information of the target position and the like into a dynamic construction sample library of the classifier;

the sample library dynamic construction unit is used for: performing data enhancement on the reference frame sample, including basic operations of rotation, scaling, dithering and blurring, and receiving a newly added sample in the tracking process;

the online learning unit extracts features of samples stored in a sample library by using a deep network ResNet18 network, obtains a predicted Gaussian response map after passing through two fully-connected layers, generates a true Gaussian label by taking a target center as a peak point of Gaussian distribution according to label information in the sample library, adjusts parameters of the two fully-connected layers on line for an optimization target by reducing a difference between the predicted Gaussian response map and the true Gaussian label, and achieves the purposes of realizing the predicted Gaussian response map and obtaining a predicted target position center through the feature extraction network and the fully-connected layers under the condition of no label;

the position refinement unit is configured to: performing position refinement on an initial tracking result obtained by an online learning unit, obtaining a plurality of shaking frames by taking the center of a predicted target position obtained by the online learning unit of a current frame and the width and height of a target scale obtained by a previous frame as references, mapping the shaking frames onto a search area of the current frame, performing feature extraction on the shaking frames by using an accurate region-of-interest pooling layer, splicing modulation features obtained by a reference frame, and finally obtaining the confidence coefficient of the predicted position of each shaking frame through a full-connection layer, and merging the results of 3 shaking frames with the highest confidence coefficients to obtain the tracking result of the current frame after the refinement;

the decision control unit is configured to: judging a target tracking state through the relation between the confidence coefficient of the predicted position of the tracking frame and a set threshold value in the tracking process, if the target is stably tracked, continuing to track the next frame, and if the target is lost, activating a detection and correction unit to detect the target of the current frame and updating a reference frame so as to realize long-term stable tracking of low and slow small targets;

the tracking result output unit is used for: and after traversing all the frames of the video, outputting the position and scale information of each frame.

2. The detection and tracking method of the small-sized low-flying target detection and tracking system according to claim 1, characterized by comprising the following steps:

step 1, constructing a target detection network;

1) constructing a network structure comprising: a backbone network, a classification subnet and a regression subnet;

2) a loss function, which solves the problem of serious imbalance of the proportion of positive and negative samples in target detection by using focus loss and reduces the weight of simple negative samples in training;

step 2, comparing and screening targets;

before the video is sent to the detection and correction unit, SURF feature point matching is carried out for one time, a first frame target of a current video is subjected to feature matching with a detection result, when the number of matching points is larger than a set value, the detection result is really a target needing to be tracked at present, the detection is successful, and at the moment, a detection frame is sent to the detection and correction unit for subsequent processes;

step 3, target tracking online learning;

predicting the target center position, comprising two parts of an initialization classifier and an online classification process:

1) initialization classifier

For a reference frame subjected to data enhancement in a sample base, extracting features by using a feature extraction network, and generating a two-dimensional Gaussian true value label y with the same size as a feature map by taking the target central position of the reference frame as a peak value_gtInitializing a classifier according to the characteristics and the label, minimizing the distance between an actual value and a true value by using a least square optimization algorithm, and solving a nonlinear least square problem by using a Gauss-Newton iteration method; formula asThe following:

in the formula (1), y ∈ {1, …, M } represents the horizontal direction coordinate of the target center point of the reference frame, M is the width of the feature map, y ∈ {1, …, N } represents the vertical direction coordinate of the target center point of the reference frame, N is the height of the feature map, and sigma is the Gaussian bandwidth;

2) online classification process

Tracking the result (x) from the previous frame (t-1 th frame)_t-1，y_t-1，w_t-1，h_t-1) (wherein (x)_t-1，y_t-1) To estimate the center coordinates of the target, (w)_t-1，h_t-1) For estimating the width and height of the target), the center position of the target of the previous frame is the center of the target search area of the current frame (the t frame), the target search area is expanded to be wide and high according to a specified proportion k, and the search area (x) of the current frame is generated_t-1，y_t-1，k*w_t-1，k*h_t-1) (ii) a Then, the feature f of the search area is extracted using the feature extraction network_tAfter two full connection layers, a prediction Gaussian response graph consistent with the size of the search area is generated

(

A mapping function representing a fully connected layer; weight₁,weight₂A weight coefficient matrix representing a full connection layer), and the maximum response position is the target center coordinate (x) estimated by the current frame_t，y_t) (ii) a The online training classifier fully considers the tracked target and the background area, and the classifier is continuously updated to estimate the position of the target;

step 4, dynamically constructing a classifier training sample library

1) Setting a reference frame updating interval as T, calling a target detection unit to update a reference frame when the current frame number T can be completely divided by T, removing all outdated samples in a sample library, using the updated reference frame to reinitialize a classifier, and simultaneously sequentially adding newly generated samples into the sample library along with the tracking process, so that the similarity between the characteristics of a target in the sample library and the currently tracked sample is higher, and the central position of the target can be accurately estimated;

2) when the confidence coefficient of the predicted position of the tracking frame is smaller than a set threshold, the current frame target tracking fails, the decision control unit sends information of a reference frame needing to be reinitialized to the detection and correction unit, and after the detection and correction unit receives the information of the detection frame of the target detection unit, the information is sent to the reference frame initialization unit, data enhancement and other operations are carried out, and finally the information is sent to a dynamically constructed sample library;

step 5, fine trimming of the target tracking position;

the method comprises two parts of a feature extraction network and a similarity evaluation network:

1) feature extraction network

The feature extraction uses a ResNet18 network, balance and reserve the information of a previous template and update the information of a current reference frame, provide the features combining the current and historical states of a target for a neural network, improve the tracking stability, extract the features of a search area for three parts of an image frame at the middle time of the reference frame, the current frame, the reference frame and the current frame, respectively send the features into an accurate region-of-interest pooling layer, and are used for calculating the confidence coefficient of a predicted position by a similarity evaluation network;

2) similarity evaluation network

The core of the similarity evaluation network is a precise region-of-interest pooling layer, the input of which comprises two parts, the first part is bilinear interpolation with interpolation coefficient of IC for an image feature map extracted by the network

IC (x, y, i, j) ═ max (0, 1- | x-i |) × max (0, 1- | y-j |) (2) maps the discrete feature map to a continuous space and obtains the feature map f (x, y)

In the formulas (2) and (3), (x, y) are characteristic diagram center seatsThe index (i, j) is the coordinate index on the feature map, w_i，jThe weight value corresponding to the position (i, j) on the feature map; the second part of the input is the coordinates (x) of the upper left corner of the rectangular box₂，x₁) And the coordinates of the lower right corner (y)₂，y₁) (ii) a Performing accurate region-of-interest pooling operation according to the obtained continuous spatial feature map and the coordinates of the rectangular frame, and reserving target features on the image to the maximum extent to prepare for further comparing the similarity of the reference target and the target of the historical frame; finally, the feature map f (x, y) is doubly integrated

And dividing by the area of the rectangular frame to obtain a precise region of interest Pooling (PrROI Pooling)

After the characteristics of the accurate region-of-interest pooling layer are obtained, the three characteristics of the reference frame, the intermediate frame and the current frame are spliced and input into the full-connection layer to output the final position confidence; and comparing the similarity degree of the candidate target and the historical target, and finding the maximum similar target as a tracking result.

Images