CN111815677A - Target tracking method, device, terminal device and readable storage medium - Google Patents

Abstract

Embodiments of the present invention disclose a target tracking method, device, terminal device, and readable storage medium. The method includes using a feature pyramid network to extract n template feature maps of different scales from a template frame; Extracting n detection feature maps of different scales from the The preset number of candidate images, and the scores and positions corresponding to each candidate image; when the n region candidate sub-networks complete the processing of the n template feature maps and the corresponding detection feature maps, according to each The score of the selected image determines the top m candidate images with the highest score among all the candidate images as the tracking target, and the position corresponding to the tracking target is used as the target position. This scheme achieves accurate tracking of small targets.

Description

Target tracking method, device, terminal device and readable storage medium

technical field

The present invention relates to the field of target tracking, and in particular, to a target tracking method, device, terminal device and readable storage medium.

Background technique

In recent years, deep learning has begun to enter the field of target tracking, attracting the attention of more and more scholars. The low-level features of the tracking image have higher resolution, which is convenient for accurate target positioning, while the high-level features contain more semantic information, which can handle large target changes and prevent the tracker from drifting, which is more convenient for tracking. The target is located in the range, so using deep learning can better extract the features of the target and express the target better. However, due to the problems of large training samples and long online update time in deep learning, there are still many challenges in the timeliness of tracking.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention provides a target tracking method, apparatus, terminal device and readable storage medium.

The first embodiment of the present invention proposes a target tracking method, which includes:

Extract n template feature maps of different scales from template frames using feature pyramid network;

Using the feature pyramid network to extract from the detection frame n detection feature maps of different scales corresponding to the scales of the n template feature maps one-to-one;

Using the ith region candidate sub-network to determine a preset number of candidate images on the detection feature map according to the ith template feature map and the detection feature map with the same scale as the ith template feature map, and The score and position corresponding to each candidate image;

After the n region candidate sub-networks finish processing the n template feature maps and the corresponding detection feature maps, determine the top m candidate images with the highest scores among all the candidate images according to the score of each candidate image is used as the tracking target, and the position corresponding to the tracking target is used as the target position.

In the target tracking method proposed by the second embodiment of the present invention, the i-th region candidate sub-network is used in the i-th template feature map and the detection feature map with the same scale as the i-th template feature map. Determine the preset number of candidate images on the detection feature map, as well as the scores and positions corresponding to each candidate image, including:

The i-th template feature map is subjected to a convolution operation to obtain a template sample set of a first preset size and a template sample set of a second preset size;

Performing a convolution operation on the detection feature map with the same scale as the i-th template feature map to obtain a detection sample set of a third preset size and a detection sample set of a fourth preset size;

The classification branch of the ith region candidate sub-network calculates each device in the detection sample set of the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size. The score of the selected image;

The regression branch of the i-th region candidate sub-network determines each sample in the detection sample set of the fourth preset size according to the template sample set of the second preset size and the detection sample set of the fourth preset size s position;

The position corresponding to each candidate image is determined according to the position of each sample in the detection sample set of the fourth preset size, and the detection sample set of the third preset size is equivalent to the detection sample set of the fourth preset size.

The target tracking method proposed by the third embodiment of the present invention further includes:

determining the position response value of the target position corresponding to the first m candidate images;

When the maximum position response value is greater than a preset response threshold, the candidate image corresponding to the maximum position response value is used as the template frame.

In the above-mentioned target tracking method, the response value is calculated according to the following formula:

其中，

表示所述位置响应值，t^*表示所述目标位置，y(t^*)表示所述目标位置t^*的响应结果，t表示距离所述目标位置最近的干扰位置，y(t)表示干扰位置t的响应结果，Δ是二次连续可微函数。

in,

represents the position response value, t ^* represents the target position, y(t ^* ) represents the response result of the target position t ^* , t represents the closest interference position to the target position, and y(t) represents the interference position The response result of t, Δ is a quadratic continuously differentiable function.

In the target tracking method described in the above embodiment, the target tracking model corresponding to the target tracking method is pre-trained by using the training sample set, until the error loss of the target tracking model is less than a preset error threshold;

The error loss is calculated using the following loss function:

其中，

表示误差损失，β_i表示第i个区域候选子网络的加权系数，μ表示衰减参数，

表示n个区域候选子网络的加权后的加权响应值；

in,

represents the error loss, β _i represents the weighting coefficient of the ith region candidate sub-network, μ represents the attenuation parameter,

Represents the weighted weighted response value of the n regional candidate subnetworks;

The weighted weighted response value calculation formula is as follows:

其中，y_β(t^*)表示所述目标位置t^*的加权后的响应结果，t表示距离所述目标位置最近的干扰位置，y_β(t)表示干扰位置t的加权后响应结果，Δ是二次连续可微函数；

Among them, y _β (t ^* ) represents the weighted response result of the target position t ^* , t represents the interference position closest to the target position, y _β (t) represents the weighted response result of the interference position t, Δ is a quadratic continuously differentiable function;

The calculation formula of the weighted response result is as follows:

s_i(t)表示第i个区域候选子网络响应结果。

s _i (t) represents the response result of the ith region candidate sub-network.

In the target tracking method described in the above embodiment, the n different scales include 32×32 pixel scales, 64×64 pixel scales, 128×128 pixel scales, and 256×256 pixel scales. at least one.

A fourth embodiment of the present invention provides a target tracking device, which includes:

The template feature map acquisition module is used to extract n template feature maps of different scales from the template frame by using the feature pyramid network;

The detection feature map acquisition module is used to extract from the detection frame n detection feature maps of different scales corresponding to the scales of the n template feature maps one-to-one by using the feature pyramid network;

The candidate image determination module is used to determine the pre-equipment on the detection feature map according to the ith template feature map and the detection feature map with the same scale as the ith template feature map using the ith region candidate sub-network Select a number of candidate images, and the scores and positions corresponding to each candidate image;

The tracking target determination module is used to determine the highest score among all the candidate images according to the score of each candidate image after the n region candidate sub-networks have completed the processing of the n template feature maps and the corresponding detection feature maps The first m candidate images are used as the tracking target, and the position corresponding to the tracking target is used as the target position.

The above-mentioned alternative image determination module includes:

a template sample set obtaining unit, configured to perform a convolution operation on the i-th template feature map to obtain a template sample set of a first preset size and a template sample set of a second preset size;

a detection sample set acquisition unit, configured to perform a convolution operation on the detection feature map with the same scale as the i-th template feature map to obtain a detection sample set of a third preset size and a detection sample set of a fourth preset size;

The candidate image score calculation unit is used for the classification branch of the ith region candidate sub-network to calculate the ith sample set according to the template sample set of the first preset size and the detection sample set of the third preset size. The scores of each candidate image in the detection sample set of three preset sizes;

A sample position determination unit, used for the regression branch of the ith region candidate sub-network to determine the fourth preset according to the template sample set of the second preset size and the detection sample set of the fourth preset size The position of each sample in the detection sample set of the size;

A candidate image position determination unit, configured to determine the position corresponding to each candidate image according to the position of each sample in the detection sample set of the fourth preset size, the detection sample set of the third preset size and the fourth preset size. The detection sample sets of the preset size are equivalent.

The above-mentioned embodiment relates to a terminal device, including a memory and a processor, where the memory is used for storing a computer program, and the processor runs the computer program to enable the terminal device to execute the above-mentioned target tracking method.

The above-mentioned embodiments relate to a readable storage medium storing a computer program, the computer program executing the above-mentioned target tracking method when running on a processor.

The present invention uses the feature pyramid network to extract n template feature maps of different scales from the template frame; uses the feature pyramid network to extract from the detection frame n detection features of different scales corresponding to the scales of the n template feature maps one-to-one Figure; Utilize the ith regional candidate sub-network to determine the preset number of candidate images on the detection feature map according to the ith template feature map and the detection feature map with the same scale as the ith template feature map , and the corresponding scores and positions of each candidate image; when the n regional candidate sub-networks process the n described template feature maps and the corresponding detection feature maps, according to the score of each candidate image, in all candidate images The top m candidate images with the highest scores are determined as the tracking target, and the position corresponding to the tracking target is used as the target position. On the one hand, the technical solution of the present invention uses the feature pyramid network as the feature extraction layer of the tracking framework, which effectively integrates low-level high-resolution information and high-level high-semantic information, which can more accurately locate the target position and track objects in small targets. The tracking performance is particularly outstanding; on the other hand, the tracking targets are screened through the improved region candidate sub-network to achieve accurate tracking of small targets.

Description of drawings

In order to illustrate the technical solutions of the present invention more clearly, the accompanying drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be It is regarded as the limitation of the protection scope of the present invention. In the various figures, similar components are numbered similarly.

1 shows a schematic flowchart of a target tracking method according to an embodiment of the present invention;

2 shows a schematic structural diagram of a target tracking model according to an embodiment of the present invention;

FIG. 3 shows a schematic flowchart of determining candidate image scores and positions according to an embodiment of the present invention;

4 shows a schematic structural diagram of an area candidate sub-network model according to an embodiment of the present invention;

5 shows a schematic flowchart of another target tracking method according to an embodiment of the present invention;

FIG. 6 shows a schematic structural diagram of a target tracking device according to an embodiment of the present invention;

FIG. 7 shows a schematic structural diagram of determining the score and position of an alternative image according to an embodiment of the present invention.

Description of main component symbols:

1-target tracking device; 100-template feature map acquisition module; 200-detection feature map acquisition module; 300-candidate image determination module; 400-tracking target determination module; 500-response value calculation module; 600-template frame update module 310-template sample set acquisition unit; 320-detection sample set acquisition unit; 330-candidate image score calculation unit; 340-sample position determination unit; 350-candidate image position determination unit.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments.

The components of the embodiments of the invention generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Thus, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present invention.

Hereinafter, the terms "comprising", "having" and their cognates, which may be used in various embodiments of the present invention, are only intended to denote particular features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be construed as first excluding the presence of or adding one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing or the possibility of a combination of the foregoing.

Furthermore, the terms "first", "second", "third", etc. are only used to differentiate the description and should not be construed as indicating or implying relative importance.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of this invention belong. The terms (such as those defined in commonly used dictionaries) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized or overly formal meaning, unless explicitly defined in the various embodiments of the present invention.

The invention is a target tracking method proposed by improving the existing three technologies, including a twin neural network structure, a feature pyramid network and a region candidate network.

Siamese network structure means that the main structure of the network consists of upper and lower branches, and the two branches share the ownership value of the same network. Like the twins, it is often used to solve many categories or uncertainties, but the number of samples under each category is relatively small. fewer classification problems. In the field of visual target tracking, the upper branch of the Siamese network is the template branch used to extract the appearance features of the template frame, and the lower branch is the detection branch, whose input is based on the tracking result of the previous frame. The candidate regions intercepted on the frame for search, after passing through the same network, use the feature map of the template branch and the feature maps of multiple candidate regions of the current frame to calculate the similarity, and use the candidate region with the highest score as the tracking of the current frame result.

The feature pyramid usually refers to obtaining multiple images of different scales for a single input image, and connecting the four vertices of these images of different scales to construct an image pyramid similar to the real pyramid. The Feature Pyramid Network (FPN) used in the field of target tracking can add a scale dimension (or depth) to the two-dimensional image. Unlike traditional detection algorithms that only use top-level features for prediction, the Feature Pyramid Network can The features of different levels are fused and predicted independently at different feature layers to obtain more robust semantic information. Through its own bottom-up line and top-down line, the feature pyramid network makes full use of the low-level high-resolution information and high-level high-semantic information, especially for small targets, the feature pyramid network also increases the resolution of feature maps. rate for more useful information about small goals.

The Region Proposal Network (RPN) is a network for extracting candidate boxes, which first appeared in the Faster RCNN structure. It uses a candidate box (also called anchor box (Anchor) technique), which is often used in computer vision to represent a fixed reference box. In the target tracking task, the tracked target has the characteristics of uncertain category, position uncertainty, scale uncertainty, etc. The improved anchor frame technology can cover about all positions by pre-establishing a set of fixed reference frames of different scales and different positions. and scale, each fixed reference frame is responsible for detecting the target whose intersection ratio is greater than the preset threshold. The regional candidate network not only has a good recognition effect, but also has a fast recognition speed.

Example 1

This embodiment, referring to FIG. 1 , shows a target tracking method including the following steps:

Step S100 : extracting n template feature maps of different scales from the template frame by using the feature pyramid network.

In the initial stage of target tracking, the first frame in the video frame can be used as the template frame, and the feature pyramid network (FPN) can be used to extract n template feature maps of different scales from the template frame. Exemplarily, referring to Figure 2, the Feature Pyramid Network (FPN) extracts 4 template feature maps of different scales from the template frame, including 32×32 pixel scale template feature maps, 64×64 pixel scale Template feature map, template feature map with 128×128 pixel scale, and template feature map with 256×256 pixel scale.

It should be understood that n is a positive integer, which can be set to 3, 4, 5, 6, etc., which can be adjusted according to the actual effect of target tracking, and the scale of the template feature map can also be flexibly set according to specific needs.

Step S200 : extracting n detection feature maps of different scales corresponding to the scales of the n template feature maps one-to-one from the detection frame by using a feature pyramid network.

The detection frame is the current frame where the tracking target needs to be acquired, and the feature pyramid network is used to extract from the detection frame n detection feature maps of different scales corresponding to the scales of the n template feature maps one-to-one. Exemplarily, see Fig. 2, the Feature Pyramid Network (FPN) extracts 4 detection feature maps of different scales from the detection frame, including 32×32 pixel scale detection feature maps, 64×64 Detection feature maps, 128×128 pixel-scale detection feature maps, and 256×256 pixel-scale detection feature maps.

Step S300: Determine a preset number of candidates on the detection feature map according to the ith template feature map and the detection feature map with the same scale as the ith template feature map using the ith region candidate sub-network. image, and the corresponding score and position of each candidate image.

Exemplarily, see Figure 2, when n is 4, corresponding to 4 region candidate sub-networks (RPN), the template feature map of 32×32 pixel point scale and the detection feature map of 32×32 pixel point scale are used as The input of the first region candidate sub-network (RPN), the template feature map of 64 × 64 pixel scale and the detection feature map of 64 × 64 pixel scale are used as the input of the second region candidate sub-network (RPN), The template feature map of 128×128 pixel point scale and the detection feature map of 128×128 pixel point scale are used as the input of the third region candidate sub-network (RPN), the template feature map of 256×256 pixel point scale and 256 The detection feature map of ×256 pixel scale is used as the input of the fourth region candidate sub-network (RPN).

The four region candidate sub-networks (RPNs) respectively determine a preset number of candidate images on the corresponding detection feature maps, as well as the scores and positions corresponding to each candidate image. Three kinds of anchor boxes with aspect ratios can be preset on the detection feature map of each scale to cover the possible tracking targets in the detection feature map. The three aspect ratios include 1:2, 1:1 and 2:1 . Correspondingly, the detection feature maps of 4 scales include 12 preset anchor boxes.

The classification score of each anchor frame can be calculated by the classification branch of the regional candidate sub-network (RPN), and the regression position of each anchor frame can be determined by the regression branch of the regional candidate sub-network (RPN). It should be understood that each anchor frame contains a Select an image, the regression position of each anchor frame is the position of the corresponding candidate image, and the classification score of each anchor frame is the score of the corresponding candidate image.

Step S400: After the n region candidate sub-networks complete the processing of the n template feature maps and the corresponding detection feature maps, determine the top m with the highest score among all the candidate images according to the score of each candidate image. The candidate image is used as the tracking target, and the position corresponding to the tracking target is used as the target position.

According to the score corresponding to each candidate image, the candidate images are sorted in descending order according to the score, and the top m candidate images with the highest score are determined as the tracking target. Among them, m is a preset positive integer.

The above i is less than or equal to n.

In this embodiment, the feature pyramid network is used to extract n template feature maps of different scales from the template frame; Feature map; use the ith region candidate sub-network to determine a preset number of candidates on the detection feature map according to the ith template feature map and the detection feature map with the same scale as the ith template feature map image, and the scores and positions corresponding to each candidate image; when the n region candidate sub-networks process the n template feature maps and the corresponding detection feature maps, the score of each candidate image is The top m candidate images with the highest determined scores in the images are selected as the tracking target, and the position corresponding to the tracking target is used as the target position. On the one hand, the technical solution of this embodiment uses the feature pyramid network as the feature extraction layer of the tracking framework, which effectively integrates the low-level high-resolution information and the high-level high-semantic information, which can locate the target position more accurately, and can track small objects more accurately. The tracking performance on objects is particularly outstanding; on the other hand, the tracking targets are screened through the improved region candidate sub-network to achieve accurate tracking of small targets.

Example 2

The region candidate sub-network, shown in Figure 4, contains a classification branch for distinguishing objects from background and a regression branch for bounding box regression.

Further, referring to FIG. 3 , the step S300 of the above-mentioned Embodiment 1 includes the following steps:

Step S310: Perform a convolution operation on the i-th template feature map to obtain a template sample set of a first preset size and a template sample set of a second preset size.

The first convolution layer of the ith region candidate sub-network performs a convolution operation on the input ith template feature map to obtain a template sample set of a first preset size and a template sample set of a second preset size.

表示大小为4×4的模板样本的特征在k种不同的锚框上有2k种变化。应当理解，2k种变化表示每一锚框中的图像可能存在两种情况，是背景或者是目标，即0或1两种状态。Further, a template sample set with a size of 4×4×(2k×256) of the first preset size is obtained on the classification branch of the ith region candidate sub-network

The features representing template samples of size 4×4 have 2k variations on k different anchor boxes. It should be understood that the 2k changes indicate that the image in each anchor box may have two situations, that is, the background or the target, that is, two states of 0 or 1.

表示为4×4个像素点尺度的模板样本的特征在k种不同的锚框上有4k种变化。应当理解，4k种变化对应位置的宽、高、横坐标和纵坐标，每一锚框通过对应的宽、高、横坐标和纵坐标进行表示。Further, a template sample set of a second preset size of 4 × 4 × (4k × 256) is obtained on the regression branch of the ith region candidate sub-network

The features of the template samples represented as 4 × 4 pixel scales have 4k variations on k different anchor boxes. It should be understood that the 4k changes correspond to the width, height, abscissa, and ordinate of the position, and each anchor frame is represented by the corresponding width, height, abscissa, and ordinate.

Step S320: Perform a convolution operation on the detection feature map with the same scale as the i-th template feature map to obtain a detection sample set of a third preset size and a detection sample set of a fourth preset size.

The first convolution layer of the i-th region candidate sub-network performs a convolution operation on the input detection feature map with the same scale as the i-th template feature map to obtain a third preset size detection sample set and a fourth preset size. Set the size of the detection sample set.

在第i个区域候选子网络的回归分支上得到大小为20×20×256第四预设尺寸的检测样本集

Further, a detection sample set of a third preset size of 20×20×256 is obtained on the classification branch of the ith region candidate sub-network

A detection sample set with a size of 20×20×256 fourth preset size is obtained on the regression branch of the i-th region candidate sub-network

256 in the above scale represents the number of channels of template samples, and the dimension of the feature is expanded to 256 dimensions through the network training process of the feature pyramid.

Step S330: The classification branch of the ith region candidate sub-network calculates the detection sample of the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size Set the scores for each candidate image.

★表示相关操作。The classification branch will give the classification score corresponding to each input detection sample, that is, the detailed score predicted as the target or background, and the corresponding score can be expressed as

★ indicates related operations.

Step S340: The regression branch of the ith region candidate sub-network determines the detection sample of the fourth preset size according to the template sample set of the second preset size and the detection sample set of the fourth preset size The location of the individual samples in the set.

这个位置回归值包含横坐标、纵坐标、宽和高，分别对应d_x，d_y，d_w和d_h四个值，

★表示相关操作。The regression branch gives the position regression value of each detection sample

This position regression value includes abscissa, ordinate, width and height, corresponding to four values of d _x , d _y , d _w and d _h respectively,

★ indicates related operations.

Step S350: Determine the position corresponding to each candidate image according to the position of each sample in the detection sample set of the fourth preset size, the detection sample set of the third preset size and the detection sample of the fourth preset size set equivalent.

The detection sample set of the third preset size and the detection sample of the fourth preset size are sample sets of the same size, and the position corresponding to each candidate image may be determined according to the position of each sample in the detection sample set of the four preset sizes.

和回归输出信息

可以得到得分最高的m个备选位置的位置信息

具体计算公式如下：The top m alternative classification output information

and regression output information

The position information of the m candidate positions with the highest score can be obtained

The specific calculation formula is as follows:

其中

为第i个备选位置对应的备选框的原始中心坐标和长宽，cls表示分类分支和reg表示回归分支，对于每个下标：i∈[0,w)，j∈[0,h)，l∈[0,2k)，p∈[0,k)，每一个A为一个输出信息的向量集合。

in

is the original center coordinate and length and width of the candidate box corresponding to the ith candidate position, cls represents the classification branch and reg represents the regression branch, for each subscript: i∈[0,w), j∈[0,h ), l∈[0,2k), p∈[0,k), each A is a vector set of output information.

Example 3

This embodiment, referring to FIG. 5 , shows that after the above steps S100-S400, the target tracking method further includes the following steps:

Step S500: Determine the position response value of the target position corresponding to the first m candidate images.

The position response values of the target positions corresponding to the first m candidate images can be calculated respectively according to the following formula:

其中，

表示所述位置响应值，t^*表示m个备选图像中任一个备选图像对应的目标位置，y(t^*)表示该目标位置t^*的响应结果，t表示距离该目标位置最近的干扰位置，y(t)表示干扰位置t的响应结果，Δ是二次连续可微函数，t与t^*越接近，Δ(t-t^*)趋近于0，t与t^*距离越远，Δ(t-t^*)趋近于1。

in,

represents the position response value, t ^* represents the target position corresponding to any candidate image among the m candidate images, y(t ^* ) represents the response result of the target position t ^* , and t represents the interference closest to the target position Position, y(t) represents the response result of the disturbance position t, Δ is a quadratic continuous differentiable function, the closer t is to t ^* , Δ(tt ^* ) approaches 0, and the farther the distance between t and t ^* is, Δ( tt ^* ) approaches 1.

It should be understood that m position response values corresponding to the m candidate positions can be determined according to the above formula.

Step S600: When the maximum position response value is greater than a preset response threshold, use the candidate image corresponding to the maximum position response value as the template frame.

A candidate image whose corresponding position response value is greater than a preset response threshold is selected from the m candidate positions as a new template frame, so as to continue to perform the above step S100.

The above is based on the online update method of high-scoring sample feedback, and the high-scoring candidate samples in the tracking process are used as new template frames for subsequent detection tasks. Effectively improve the accuracy and robustness of target tracking.

Further, use the training sample set to pre-train the target tracking model corresponding to the target tracking method, until the error loss of the target tracking model is less than the preset error threshold; the error loss is calculated using the following loss function:

其中，

表示误差损失，β_i表示第i个区域候选子网络的加权系数，μ表示衰减参数，

表示n个区域候选子网络的加权后的加权响应值，所述加权后的加权响应值计算公式如下：

in,

represents the error loss, β _i represents the weighting coefficient of the ith region candidate sub-network, μ represents the attenuation parameter,

Represents the weighted weighted response value of the n regional candidate sub-networks, and the calculation formula of the weighted weighted response value is as follows:

其中，y_β(t^*)表示所述目标位置t^*的加权后的响应结果，t表示距离所述目标位置最近的干扰位置，y_β(t)表示干扰位置t的加权后响应结果，Δ是二次连续可微函数；所述加权后的响应结果计算公式如下：

Among them, y _β (t ^* ) represents the weighted response result of the target position t ^* , t represents the interference position closest to the target position, y _β (t) represents the weighted response result of the interference position t, Δ is a quadratic continuously differentiable function; the calculation formula of the weighted response result is as follows:

s_i(t)表示第i个区域候选子网络响应结果，其中，加权系数之和为1，即

s _i (t) represents the response result of the ith region candidate sub-network, where the sum of the weighting coefficients is 1, that is,

The above loss function is used to calculate the error loss of the target tracking model corresponding to the target tracking method. When the error loss of the target tracking model is less than the preset error threshold, it indicates that the tracking quality of the target tracking model corresponding to the target tracking method meets the standard.

Example 4

This embodiment, referring to FIG. 6 , shows a target tracking apparatus 1 including: a template feature map acquisition module 100 , a detection feature map acquisition module 200 , a candidate image determination module 300 and a tracking target determination module 400 .

The template feature map acquisition module 100 is used to extract n template feature maps of different scales from the template frame by using the feature pyramid network; the detection feature map acquisition module 200 is used to extract from the detection frame by using the feature pyramid network. The scale of the template feature map corresponds to the detection feature maps of n different scales; the candidate image determination module 300 is used to utilize the ith region candidate sub-network according to the ith template feature map and the ith template The detection feature maps with the same feature map scale determine a preset number of candidate images on the detection feature map, as well as the scores and positions corresponding to each candidate image; the tracking target determination module 400 is used for when n regions After the candidate sub-network completes the processing of the n template feature maps and the corresponding detection feature maps, according to the scores of each candidate image, the top m candidate images with the highest scores are determined as the tracking targets among all the candidate images. , and the position corresponding to the tracking target is taken as the target position.

Further, referring to FIG. 7 , the candidate image determination module 300 includes: a template sample set acquisition unit 310, a detection sample set acquisition unit 320, an candidate image score calculation unit 330, a sample position determination unit 340, and a candidate image position determination unit 350.

The template sample set obtaining unit 310 is configured to perform a convolution operation on the i-th template feature map to obtain a template sample set of a first preset size and a template sample set of a second preset size; the detection sample set obtaining unit 320 , for performing a convolution operation on the detection feature map with the same scale as the i-th template feature map to obtain a detection sample set of the third preset size and a detection sample set of the fourth preset size; the candidate image score A calculation unit 330, used for the classification branch of the ith region candidate sub-network to calculate the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size The score of each candidate image in the detection sample set of Suppose the detection sample set of the size determines the position of each sample in the detection sample set of the fourth preset size; the candidate image position determination unit 350 is configured to determine the position of each sample in the detection sample set of the fourth preset size according to the position of each sample. For the positions corresponding to the candidate images, the detection sample set of the third preset size is equivalent to the detection sample set of the fourth preset size.

A target tracking device 1 further includes: a response value calculation module 500 for determining the position response values of the target positions corresponding to the first m candidate images; a template frame update module 600 for when the maximum position response value is greater than When the preset response threshold is used, the candidate image corresponding to the largest position response value is used as the template frame.

The target tracking apparatus 1 of this embodiment is used to execute the target tracking method described in the above embodiments through the cooperative use of the template feature map acquisition module 100 , the detection feature map acquisition module 200 , the candidate image determination module 300 and the tracking target determination module 400 . , the implementations and beneficial effects involved in the above embodiments are also applicable in this embodiment, and are not repeated here.

The foregoing embodiment relates to a terminal device, including a memory and a processor, where the memory is used to store a computer program, and the processor runs the computer program to enable the terminal device to execute the target tracking method described in the foregoing embodiment. .

The above embodiments relate to a readable storage medium, which stores a computer program, and when the computer program runs on a processor, executes the target tracking method described in the above embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may also be implemented in other manners. The apparatus embodiments described above are only schematic, for example, the flowcharts and structural diagrams in the accompanying drawings show the possible implementation architectures and functions of the apparatuses, methods and computer program products according to various embodiments of the present invention and operation. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart diagrams, can be implemented using dedicated hardware-based systems that perform the specified functions or actions. be implemented, or may be implemented in a combination of special purpose hardware and computer instructions.

In addition, each functional module or unit in each embodiment of the present invention may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention. should be included within the protection scope of the present invention.

Claims (10)

1. A method of object tracking, the method comprising:

extracting n template feature graphs with different scales from the template frame by using a feature pyramid network;

extracting n detection feature maps with different scales, which are in one-to-one correspondence with the scales of the n template feature maps, from a detection frame by using a feature pyramid network;

determining preset candidate number of candidate images and corresponding scores and positions of the candidate images on a detection feature map by using an ith area candidate sub-network according to the ith template feature map and the detection feature map with the same scale as the ith template feature map;

and when the n regional candidate sub-networks finish processing the n template feature maps and the corresponding detection feature maps, determining m front candidate images with the highest scores in all the candidate images as tracking targets according to the scores of all the candidate images, and taking the positions corresponding to the tracking targets as target positions.

2. The method according to claim 1, wherein the determining, by using the ith area candidate sub-network, a preset candidate number of candidate images and scores and positions corresponding to the candidate images on the detection feature map according to the ith template feature map and the detection feature map with the same scale as the ith template feature map comprises:

performing convolution operation on the ith template characteristic diagram to obtain a template sample set with a first preset size and a template sample set with a second preset size;

performing convolution operation on the detection characteristic graph with the same scale as the ith template characteristic graph to obtain a detection sample set with a third preset size and a detection sample set with a fourth preset size;

the classification branch of the ith regional candidate sub-network calculates the score of each candidate image in the detection sample set of the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size;

determining the position of each sample in the detection sample set with the fourth preset size according to the template sample set with the second preset size and the detection sample set with the fourth preset size by the regression branch of the ith area candidate sub-network;

and determining the position corresponding to each alternative image according to the position of each sample in the detection sample set with the fourth preset size, wherein the detection sample set with the third preset size is equal to the detection sample set with the fourth preset size.

3. The target tracking method of claim 1, further comprising:

determining position response values of target positions corresponding to the first m candidate images;

and when the maximum position response value is larger than a preset position response threshold value, taking the alternative image corresponding to the maximum position response value as the template frame.

4. The object tracking method of claim 3, wherein the position response value is calculated according to the formula:

wherein,

representing said position response value, t^*Representing said target position, y (t)^*) Representing said target position t^*T represents the interference position closest to the target position, y (t) represents the response result of the interference position t, and Δ is a quadratic continuous differentiable function.

5. The target tracking method according to claim 1, wherein a target tracking model corresponding to the target tracking method is trained in advance by using a training sample set until an error loss of the target tracking model is smaller than a preset error threshold;

the error loss is calculated using the following loss function:

wherein,

represents the error loss, beta_iRepresents the weighting coefficients of the i-th region candidate sub-network, mu represents the attenuation parameter,

weighted response values representing the n regional candidate subnets;

the weighted response value calculation formula is as follows:

wherein, y_β(t^*) Representing said target position t^*T represents the interference bit closest to the target positionPosition y_β(t) represents the weighted response result of the interference location t, Δ being a quadratic continuous differentiable function;

the weighted response result calculation formula is as follows:

s_i(t) represents the ith area candidate sub-network response result.

6. The method of any of claims 1-5, wherein the n different scales comprise at least one of a 32 x 32 pixel scale, a 64 x 64 pixel scale, a 128 x 128 pixel scale, and a 256 x 256 pixel scale.

7. An object tracking device, the device comprising:

the template characteristic image acquisition module is used for extracting n template characteristic images with different scales from the template frame by utilizing the characteristic pyramid network;

the detection characteristic graph acquisition module is used for extracting n detection characteristic graphs with different scales, which are in one-to-one correspondence with the scales of the n template characteristic graphs, from a detection frame by utilizing a characteristic pyramid network;

the candidate image determining module is used for determining a preset candidate number of candidate images and corresponding scores and positions of the candidate images on the detection feature map by utilizing an ith regional candidate sub-network according to the ith template feature map and the detection feature map with the same scale as the ith template feature map;

and the tracking target determining module is used for determining the first m candidate images with the highest scores in all the candidate images as the tracking targets according to the scores of all the candidate images after the n regional candidate sub-networks process the n template feature images and the corresponding detection feature images, and the positions corresponding to the tracking targets are used as target positions.

8. The target tracking device of claim 7, wherein the alternative image determination module comprises:

a template sample set obtaining unit, configured to perform convolution operation on the ith template feature map to obtain a template sample set of a first preset size and a template sample set of a second preset size;

a detection sample set obtaining unit, configured to perform convolution operation on the detection feature map with the same scale as the ith template feature map to obtain a detection sample set of a third preset size and a detection sample set of a fourth preset size;

a candidate image score calculation unit, configured to calculate, by the classification branch of the ith regional candidate subnetwork, a score of each candidate image in the detection sample set of the third preset size according to the template sample set of the first preset size and the detection sample set of the third preset size;

a sample position determining unit, configured to determine, by the regression branch of the i-th area candidate subnetwork, a position of each sample in the fourth preset-sized detection sample set according to the second preset-sized template sample set and the fourth preset-sized detection sample set;

and the candidate image position determining unit is used for determining the position corresponding to each candidate image according to the position of each sample in the detection sample set with the fourth preset size, wherein the detection sample set with the third preset size is equal to the detection sample set with the fourth preset size.

9. A terminal device, comprising a memory for storing a computer program and a processor for executing the computer program to enable the terminal device to perform the object tracking method of any one of claims 1 to 6.

10. A readable storage medium, characterized in that it stores a computer program which, when run on a processor, performs the object tracking method of any one of claims 1 to 6.

Images