CN115619815A - Object tracking method and object tracking device - Google Patents

Abstract

Embodiments of the present invention provide an object tracking method and an object tracking device, which are suitable for low-latency applications. In the method, object detection is performed on one of consecutive image frames. Object detection is used to identify objects. Temporarily store consecutive image frames. Object tracking is performed on the temporarily stored continuous image frames according to the object detection results. Object tracking is used to associate objects in one of successive image frames with another. In this way, the accuracy of tracking can be improved and the requirement of low latency can be met.

Description

Object tracking method and object tracking device

technical field

The present invention relates to an image processing technology, and in particular, relates to an object tracking method and an object tracking device.

Background technique

Object detection (detection) and object tracking (tracking) are important researches in computer vision technology, and have been widely used in fields such as video communication, medical treatment, driving assistance, and security.

The main function of object detection is to identify object types in a Region of Interest (ROI). There are many algorithms for object detection. For example, YOLO (You Only Look Once) is a neural network algorithm with the characteristics of light data and high efficiency. It is worth noting that in the YOLO third-generation (version 3, V3) architecture, the upsampling layer can learn more subtle features, which in turn helps to detect objects of smaller size. Another example, RetinaFace is mainly for face detection. RetinaFace can provide single-stage dense face positioning in natural scenes, using Feature Pyramid Network (FPN) to be responsible for faces of different sizes (for example, smaller faces), and using multi-task loss (multi-task loss), thus providing higher accuracy for face detection. For another example, Adaptive Boosting (AdaBoost) uses the wrong samples of the previous classifier to train the next classifier, and adds weak classifiers to improve the classification results, which is more sensitive to abnormal data or noise data. Spend.

On the other hand, the main function of object tracking is to track the same object framed by the front and rear image frames. There are also many algorithms for object tracking. For example, the optical flow method can infer the moving speed and direction of the object by detecting the change of the intensity of image pixels over time. However, the optical flow method is easily misjudged by light changes and the influence of other objects. For another example, the Minimum Output Sum of Squared Error (MOSSE) filter utilizes the correlation between the area to be detected and the tracking target to determine the area to be detected as the tracking target. It is worth noting that the MOSSE filter can update the filter parameters for the covered tracking target, so that it can be re-tracked when the tracking target reappears. For another example, the Scale Invariant Feature Transform (SIFT) algorithm determines the position, scale and rotation invariance of feature points and generates corresponding feature vectors, and determines the position and orientation of the target by matching feature vectors.

Generally speaking, object detection is time-consuming compared to object tracking, but the results of object tracking may have inaccurate problems. In some application scenarios, both technologies may affect the user experience. For example, the application scenario of real-time video conferencing has low-latency requirements. If the detection takes too long, moving objects cannot be accurately framed. For example, the frame selection result in the first frame is obtained after four frames of object detection, but the position of the object has changed between the four frames, making the frame in the fourth frame displayed in real time The selection result is inaccurate. Or, the wrong target is being tracked. It can be seen that, for the requirement of low delay and high accuracy, the existing technology still needs to be improved.

Contents of the invention

Embodiments of the present invention are directed to an object tracking method and an object tracking device, which perform continuous tracking based on object detection results, thereby meeting the requirement of low latency and providing high accuracy.

According to an embodiment of the present invention, the object tracking method is suitable for low-latency applications and includes (but is not limited to) the following steps: performing object detection on one of one or more consecutive image frames. Object detection is used to identify objects. Temporarily store consecutive image frames. Object tracking is performed on the temporarily stored continuous image frames according to the object detection results. Object tracking is used to associate one of the successive image frames with an object in the other.

According to an embodiment of the present invention, an object tracking device is suitable for low-latency applications and includes, but is not limited to, a memory and a processor. The memory is used to store program codes. The processor is coupled to the memory. The processor is configured to load and execute program code to perform the following steps: perform object detection on one or more consecutive image frames, temporarily store the consecutive image frames, and perform object detection based on the result of the object detection. The stored continuous image frames are used for object tracking. Object detection is used to identify objects. Object tracking is used to associate objects in one of the successive image frames with the other.

Based on the above, according to the object tracking method and the object tracking device according to the embodiment of the present invention, the continuous image frames in the process of object detection are temporarily stored, and the result of the object detection is obtained, and those temporarily stored continuous image frames are tracked based on this result target in . In this way, the high accuracy of object detection and the high efficiency of object tracking can be combined, and the requirements of low-latency applications can be met.

Description of drawings

The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain principles of the invention.

FIG. 1 is a component block diagram of an object tracking device according to an embodiment of the present invention;

FIG. 2 is a flowchart of an object tracking method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram describing tracking of consecutive image frames according to an embodiment of the present invention;

FIG. 4 is a flowchart of a target update mechanism according to an embodiment of the present invention;

FIG. 5 is a sequence diagram of object detection and tracking according to an embodiment of the present invention;

FIG. 6 is a sequence diagram of object detection and tracking according to another embodiment of the present invention;

FIG. 7 is a sequence diagram of a target update mechanism according to an embodiment of the present invention;

FIG. 8 is a timing diagram of a target updating mechanism according to another embodiment of the present invention.

Explanation of reference numbers

100: object tracking device;

110: memory;

111: buffer;

130: processor;

131: detect tracker;

132: detector;

133: main tracker;

135: secondary tracker;

S210～S250, S410～S460, S510～S530: steps;

F1～F5: continuous image frame;

ROI～ROI6: region of interest;

QF: queue frame;

501: object detection;

503: object tracking;

D1: period;

D2, D3, D4: period;

C1～C4: Confidence degree;

t1: time point.

detailed description

Reference will now be made in detail to the exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference symbols are used in the drawings and descriptions to refer to the same or like parts.

FIG. 1 is a block diagram of components of an object tracking device 100 according to an embodiment of the invention. Referring to FIG. 1 , the object tracking device 100 includes (but not limited to) a memory 110 and a processor 130 . The object tracking device 100 may be a desktop computer, a notebook computer, a smart phone, a tablet computer, a server, a monitoring device, a medical testing instrument, an optical testing instrument or other computing devices.

The memory 110 can be any type of fixed or removable random access memory (Radom Access Memory, RAM), read only memory (Read Only Memory, ROM), flash memory (flash memory), traditional hard disk (HardDisk Drive, HDD), solid state Hard disk (Solid-State Drive, SSD) or similar components. In one embodiment, the memory 110 is used to record program codes, software modules, configuration configurations, data (such as image frames, detection/tracking results, confidence levels, etc.) or other files, and its implementation will be described in detail later. example.

In one embodiment, the memory 110 includes a buffer 111 . The buffer 111 may be one of the one or more memories 110 , and may also represent one or more memory blocks in the memories 110 . The buffer 111 is used for temporarily storing image frames, and its function will be described in detail in the following embodiments. One or more image frames can be wired or wirelessly connected to an image capture device (such as a camera, video camera, or monitor), a server (such as an image streaming server, or a cloud server), or a storage medium (such as , flash disk, hard disk or database server).

The processor 130 is coupled to the memory 110, and the processor 130 may be a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphic Processing unit, GPU), or other programmable general-purpose or special-purpose microprocessors Microprocessor, Digital Signal Processor (DSP), Programmable Controller, Field Programmable Gate Array (Field Programmable Gate Array, FPGA), Application-Specific Integrated Circuit (ASIC), A neural network accelerator or other similar component or combination of the above. In one embodiment, the processor 130 is used to execute all or part of the operations of the object tracking device 100 , and can load and execute program codes, software modules, files and data recorded in the memory 110 . In some embodiments, the functions of the processor 130 can be realized by software.

The processor 130 includes a detection tracker 131 and a secondary tracker 135 . Either or both of the detection tracker 131 and the sub-tracker 135 can be realized by independent digital circuits, chips, neural network accelerators or other processors, or their functions can be realized by software.

In one embodiment, the detection tracker 131 includes a detector 132 and a main tracker 133 . The detector 132 is used for object detection. Object detection is, for example, determining a region of interest (Region of Interest, ROI) (or a bounding box, Rectangular box (bounding rectangle)), and then identify the type of the target (for example, male or female, dog or cat, table or chair, car or traffic lights, etc.). The detector 132, for example, can apply an algorithm based on a neural network (for example, YOLO, a region-based convolutional neural network (Region Based Convolutional Neural Networks, R-CNN), or a fast R-CNN (Fast CNN)) or based on feature matching Object detection can be achieved using algorithms such as Histogram of Oriented Gradient (HOG), Harr, or Feature Alignment of Speeded Up Robust Features (SURF). It should be noted that the embodiment of the present invention does not limit the algorithm used by the detector 132 .

In one embodiment, the primary tracker 133 and the secondary tracker 135 are used for object tracking. Object tracking is used to associate one of the successive image frames with an object in the other. Sequential image frames represent those consecutive image frames of a video or video stream. Object tracking, for example, is to determine the position, movement, direction and correlation of other motions of the same target (the corresponding position can be selected by the region of interest) in adjacent image frames, and then locate the moving target. The main tracker 133 and the secondary tracker 135 can apply optical flow method, sorting method (Simple Online And Realtime Tracking, SORT), depth sorting method (Deep SORT), joint detection and embedding vector (Joint Detection and Embedding, JDE) model, for example Or other tracking algorithms to achieve object tracking. It should be noted that the embodiments of the present invention do not limit the algorithms used by the primary tracker 133 and the secondary tracker 135 , and the primary tracker 133 and the secondary tracker 135 can use the same or different algorithms.

In some embodiments, the object tracking device 100 may further include a display (not shown). The display is coupled to the processor 130 . The display can be a liquid crystal display (Liquid-Crystal Display, LCD), a light-emitting diode (Light-Emitting Diode, LED) display, an organic light-emitting diode (Organic Light-Emitting Diode, OLED), a quantum dot display (Quantum dot display) or other types monitor. In one embodiment, the display is used for displaying image frames or object detected/tracked image frames.

Hereinafter, the methods described in the embodiments of the present invention will be described in conjunction with various devices, components and/or modules in the object tracking device 100 . Each process of the method can be adjusted according to the implementation situation, and is not limited thereto.

FIG. 2 is a flowchart of an object tracking method according to an embodiment of the invention. Referring to FIG. 2 , the detector 132 of the detection tracker 131 performs object detection on one of one or more consecutive image frames (step S210 ). Specifically, in some application scenarios, such as video calling, image streaming, image monitoring or games, the processor 130 can obtain one or more continuous image frames (herein referred to as continuous image frames). A continuous image frame refers to a collection of adjacent image frames based on the image capture device or the frame rate of the recorded image (eg, measured in Frames Per Second (FPS) or frequency). For example, if the frame rate is 60FPS, 60 image frames within one second can be called continuous image frames. However, the continuous image frame is not limited to the image frame within one second. For example, the continuous image frame can also be an image frame within one and a half seconds, two seconds or two and one-third seconds.

In response to the input of consecutive image frames (eg, originating from the image capture device, server or storage medium, and stored in the memory 110 ), the detector 132 accesses an input consecutive image frame from the memory 110 . In one embodiment, in order to achieve the function of real-time processing, the detector 132 may perform object detection on the first continuous image frame currently input. In another embodiment, the detector 132 may perform object detection on other input continuous image frames. That is, ignore the first continuous image frame or ignore multiple consecutive image frames. It should be noted that the first frame here means the first frame input at a certain time point or the first frame frame accessed to the memory 110 at this point in time, and is not limited to images or the starting frame of the video stream.

On the other hand, the description of the object detection can refer to the above description for the detector 132 , and will not be repeated here.

For example, FIG. 3 is a schematic diagram describing the tracking of continuous image frames by the detection tracker 131 according to an embodiment of the present invention. Please refer to FIG. 3 , the detector 132 determines the ROI corresponding to the location of the target in the first consecutive image frame F1 among the consecutive image frames F1-F4, and identifies the target in the ROI accordingly. . It should be noted that the second to fourth consecutive image frames F2-F4 shown in FIG. 3 represent subsequent image frames of the first continuous image frame F1.

The processor 130 may temporarily store one or more consecutive image frames (step S230 ) into the buffer 111 . Specifically, some low-latency applications require real-time processing of incoming, accessed, or captured images. The low-latency application relates to a video application in which the time delay between the input time point of a continuous frame image and the output time point of the same continuous frame image is within a certain allowable time. For example, video calling/meeting, or live streaming. According to different needs, these video applications may additionally provide such as face detection, brightness adjustment, special effect processing or other image processing. However, if the image processing period is too long, it will affect the experience result of the application. For example, in a real-time video conference, if the face detection period is too long, the movement of the head may cause the face position obtained by the detection result to deviate from the face position in the current output image, and the displayed image cannot accurately frame the person. Face. Therefore, in the embodiment of the present invention, the continuous image frames received during the object detection process can be retained, so that the object detection result can update the tracking target in the retained image frame, and make the image frame of this image frame The output time point can be later than the end time point of its object detection.

In one embodiment, during all or part of the object detection in step S210, the processor 130 may temporarily store one or more continuous image frames input (system, for example, the object tracking device 100) during this period in buffer 111. Taking FIG. 3 as an example, the detector 132 performs object detection on the continuous image frames F1. While the detector 132 receives the consecutive image frames F1 and obtains the ROI in the consecutive image frames F1, the memory 110 sequentially stores the consecutive image frames F1-F4. The processor 130 can store these consecutive image frames F1 - F4 as a queue frame QF in the buffer 111 .

In another embodiment, the processor 130 may further temporarily store other consecutive image frames accessed outside the object detection period. For example, the processor 130 frames the last image before the object detection period or the next continuous image after the period.

In yet another embodiment, the processor 130 may temporarily store one or more consecutive image frames input to the system during all or part of the period before the object tracking is completed.

It should be noted that in the example shown in FIG. 3 , all consecutive image frames are temporarily stored in the buffer 111 during the object detection period, but it is not limited thereto.

In one embodiment, the processor 130 can compare those consecutive image frames stored temporarily with the upper limit. The upper limit of the number is related to the space size of the buffer 111 , the detection speed of the detector 132 or the requirement of processing efficiency. For example, the maximum quantity is 8, 10 or 20 sheets. The processor 130 may delete at least one of the temporarily stored consecutive image frames according to the comparison result of the consecutive image frames and the upper limit. In response to those consecutive image frames temporarily stored being equal to or greater than the upper limit, the processor 130 may delete some of the consecutive image frames in the buffer 111 . For example, the processor 130 may delete consecutive image frames sorted in even order or odd order, or randomly delete a specific number of consecutive frames in the buffer 111 . On the other hand, the processor 130 may reserve all or part of the consecutive image frames in the buffer 111 in response to the fact that the number of consecutive image frames temporarily stored does not reach the upper limit.

In another embodiment, if the space of the buffer 111 allows all consecutive image frames received during object detection, the processor 130 may retain those consecutive image frames.

It should be noted that the upper limit of the number may be fixed, or may change in response to factors such as the real-time processing speed of the detector 132 , the complexity of the system operation, and subsequent application requirements.

The main tracker 133 may perform object tracking on one or more temporarily stored consecutive image frames according to the object detection result (step S250 ). In one embodiment, the result of the object detection includes the ROI of the object. As shown in FIG. 3 , the ROI corresponds to the position of the object in the continuous image frame subject to object detection. It should be noted that the region of interest ROI may completely or partially frame the target, and this embodiment of the present invention does not limit it. In some embodiments, the result of the object detection also includes the type of the target.

On the other hand, the description of the object tracking can refer to the above description for the main tracker 133 , and will not be repeated here.

In addition, in response to the completion of the object detection of one of the consecutive image frames (i.e., obtaining the result of object detection, for example, detecting the ROI of the consecutive image frame F1 shown in FIG. 3 ), the main tracker 133 to perform object tracking on one or more consecutive image frames in the buffer 111. In other words, before the object detection on the first consecutive image frame is completed, the main tracker 133 disables or does not track the first consecutive image frame or other subsequent input consecutive image frames.

In one embodiment, the main tracker 133 can determine the correlation between the ROIs in the object detection result and those consecutive image frames temporarily stored, and determine another ROI according to the correlation. The correlation relates to the position, orientation and/or velocity of one or more objects in the one or more regions of interest between adjacent consecutive image frames.

Taking FIG. 3 as an example, the main tracker 133 continuously tracks the object in the region of interest ROI obtained by the detector 132 in consecutive image frames F1 - F4 , and updates the region of interest ROI2 as the object moves.

In one embodiment, it is assumed that the result of the object detection includes a detection ROI corresponding to the object (ie, corresponding to the position of the object in the consecutive image frames subject to object detection). In addition, it is assumed that tracking a region of interest means that the object tracks a previously tracked area. In other words, the tracking ROI is the ROI that the object is tracked in one or more consecutive image frames at or before the current time point as a tracking basis. The main tracker 133 can update the tracking ROI targeted by the object tracking to the detection ROI obtained from the object detection. In other words, tracking ROIs is directly replaced by detecting ROIs.

FIG. 4 is a flowchart of an object update mechanism according to an embodiment of the invention. Referring to FIG. 4 , the processor 130 accesses the input continuous image frame from the memory 110 (step S410 ), and detects the object in the accessed continuous image frame through the detection tracker 131 . At this time, the sub-tracker 135 may complete the tracking of the previous continuous image frames, and further determine whether the detection tracker 131 is busy (step S420 ). However, no matter whether the detection tracker 131 is busy or not, the secondary tracker 135 still uses the ROI obtained from the previous consecutive image frames to track the target (step S430 ). On the other hand, if the detection tracker 131 is not busy, it means that a detection and tracking region of interest has been obtained (that is, the 132 detector has completed detection, and 133 the main tracker has completed tracking all temporarily stored continuous images) (step S440 ), then the main tracker 133 can use the new ROI output by the detector 132 to update the currently tracked ROI (that is, update the tracking target, step S450), and after the continuous tracking is completed/finished and temporarily stored A detection and tracking region of interest is obtained after the continuous image frame, compared or calculated with a tracking region of interest obtained by the secondary tracker 135, and one of them is selected or mixed and calculated to obtain a final region of interest for updating The ROI that the secondary tracker 135 is currently tracking (step S460).

In one embodiment, the processor 130 may disable the tracking of the object that previously tracked the ROI according to the time generated by the object detection tracking result. Assuming that the detection tracker 131 produces a detection tracking result when the secondary tracker 135 starts a round of tracking but has not completed it, the secondary tracker 135 can disable or not perform object tracking before starting the next round of detection tracking. In the next cycle of object tracking, the secondary tracker 135 directly starts tracking based on the detection and tracking results.

For example, FIG. 5 is a sequence diagram of object detection and tracking according to an embodiment of the present invention, which is used to explain the decision-making mechanism of step S460 in FIG. 4 in detail. Referring to FIG. 5 , during the period D1 when the detection tracker 131 performs object detection and tracking 501 , the object tracking 503 of the secondary tracker 135 completes the tracking of two consecutive image frames. While the secondary tracker 135 is performing the object tracking 503 of the third continuous image frame, the detection tracker 131 has completed or almost completed the object detection and tracking 501 . That is, during the third object tracking 503 period D2, the detection tracker 131 executes the object detection tracking 501 and obtains a new ROI accordingly (step S510). And within a certain period before and after the next time the detection tracker 131 starts the object detection and tracking 501 , the object tracking 503 can perform tracking based on the new ROI obtained by the object detection and tracking 501 (step S530 ). In another embodiment, the secondary tracker 135 that restarts the tracking may perform object tracking 503 based on the newly obtained detection tracking ROI obtained from the object detection tracking 501 and the tracking ROI obtained from the previous object tracking 503 . For example, the secondary tracker 135 may use a weighted average of both the detection tracking ROI and the tracking ROI. Wherein, the weight used in the weighted average can be changed according to the requirement of the user, and is not limited by the embodiment of the present invention. Alternatively, the secondary tracker 135 may select one of detecting and tracking the ROI and tracking the ROI.

In one embodiment, the processor 130 may determine the time difference between the time point when the latest object detection tracking 501 is completed and the time point when the last object tracking 503 is completed. The time difference represents whether the time point of the latest result obtained by the secondary tracker 135 is close to the time point of the latest result obtained by the detection tracker 131 . The secondary tracker 135 and the detection tracker 131 can decide whether to use both the detection and tracking ROI and the tracking ROI for object tracking and object detection according to the time difference.

For example, FIG. 6 is a timing diagram of object detection and tracking according to another embodiment of the present invention. Please refer to FIG. 6 , the object tracking 503 is continuously executed regardless of whether the object tracking 503 obtains a result. However, secondary tracker 135 may determine the time difference between the end of period D1 and the result of period D4 and compare this time difference to a difference threshold. If the time difference is less than the difference threshold, the object tracking 503 can use the weighted average of the ROI obtained during the period D4 and the ROI obtained during the period D1 by the object detection tracking 501 . On the other hand, if the time difference is not less than the difference threshold, then the object tracking 503 and the object detection tracking 501 only use the ROI obtained by the object detection tracking 501 during the period D1.

In one embodiment, assuming that the period of object detection has not been recorded, the secondary tracker 135 may decide to update the tracking ROI according to the confidence level of the tracking ROI (that is, the area previously tracked by the secondary tracker 135 ) in object tracking. A region of interest is tracked for detection (ie, the result of object detection tracking). In some application scenarios, the target of object tracking may be occluded suddenly, so that the confidence of the object tracking result may be low (for example, less than a confidence threshold). At this time, when the object tracking of the secondary tracker 135 is completed, the secondary tracker 135 can update the result of object detection and tracking or use the weighted average of the detection tracking ROI and the tracking ROI as the final ROI .

For example, FIG. 7 is a sequence diagram of a target updating mechanism according to an embodiment of the present invention. Referring to FIG. 7 , it is assumed that the confidence level C4 of the ROI3 among the confidence levels C1 - C4 of the results of the secondary tracker 135 for the consecutive image frames F1 - F4 is less than the confidence level threshold. At this time, the secondary tracker 135 can update the ROI3 to the ROI4 obtained by the detection tracker 131 . For another example, if the number of the confidence levels C1 - C4 is smaller than the confidence level threshold is greater than the number threshold, the secondary tracker 135 may also update the ROI3 to the ROI4 obtained by the detection tracker 131 . For another example, the secondary tracker 135 may also use the weighted average of ROI3 and ROI4, and the weight of ROI3 may be lower.

In one embodiment, the secondary tracker 135 may decide to update the tracking ROI (that is, the area previously tracked by the secondary tracker 135) to the detection and tracking ROI (ie, the object detection tracking result) according to the detection result of the scene transition. ). The scene transition is different from the scenes of two adjacent consecutive image frames. The processor 130 can determine the change degree of the background color, contrast or specific pattern, and obtain the detection result of the scene transition (for example, the scene is different/converted or the same/unconverted). For example, if the change degree is greater than the change threshold, the detection result is that the scene has changed, and the secondary tracker 135 can update the ROI. For another example, if the degree of change is not greater than the change threshold, the detection result is that the scene has not changed, and the secondary tracker 135 can keep tracking the ROI or use detection to track the ROI and track the ROI.

For example, FIG. 8 is a sequence diagram of a target updating mechanism according to another embodiment of the present invention. Referring to FIG. 8 , assume that the processor 130 detects that the scene transition has been converted at time point t1. For example, the content of the continuous image frame F2 is daytime, but the content of the continuous image frame F3 is night. In addition, for the consecutive image frames F3 , the sub-tracker 135 can update the ROI5 obtained from the consecutive image frames F2 to the ROI6 recently output by the detection tracker 131 .

In one embodiment, in response to completion of object tracking in one or more consecutive image frames, processor 130 may request display of the results of the object tracking. For example, the processor 130 can display continuous image frames and the ROI framed by the object tracking through the display.

Taking Figure 3 as an example, table (1) is a time relationship table:

Table 1)

During the period when the detector 132 detects the continuous image frames F1 , the processor 130 inputs the continuous image frames F1 ˜ F4 to the buffer 111 . At this time, the continuous image frames F1-F3 displayed on the display have no object detection or object tracking results yet. When the continuous image frame F4 is displayed on the display, the main tracker 133 can use the region of interest output by the detector 132 to track the targets in those consecutive image frames F1-F4 temporarily stored, and can display the object tracking results accordingly (the region of interest ROI2 in the continuous image frame F4 shown in FIG. 3 ). In other embodiments, the region of interest ROI2 is used to compare or calculate with the tracking region of interest obtained by the secondary tracker 135, select one or mix and calculate the final region of interest, when the display shows continuous image information box F4, and at the same time display the final ROI.

In one embodiment, the detector 132 may perform object detection on the image frames after those consecutive image frames temporarily stored in the buffer 111, and disable or disable the object detection on other of those consecutive image frames previously buffered. Perform object detection. That is, the detector 132 disables or does not perform object detection on all input consecutive image frames. The detection period of the detector 132 for a single frame may be much longer than the tracking period of the main tracker 133 for a single frame, and the detection period cannot even meet the low-latency requirement of the application scenario. When the detector 132 outputs a result, other continuous image frames during the detection period may have been requested for output or other processing multiple times. As shown in Table (1), the display outputs continuous image frames F1-F3, but the detector 132 is still performing object detection on the continuous image frame F1. In response to the output of the object detection result, the detector 132 can directly perform object detection on the newly input continuous image frames, while disabling or not continuing to perform object detection on other previously temporarily stored continuous image frames. Taking FIG. 3 as an example, the detector 132 detects the input image frame following the continuous image frame F4.

In another embodiment, the detection tracker 131 starts object detection on newly input continuous image frames according to a fixed time interval, a fixed number of image frame intervals, or detection results of scene transitions, and any object detection As an independent event, it does not ask whether there are currently outstanding object detections. The result output of any object detection and tracking is used to update the result output of the previous object detection and tracking. Since the time spent for each detection and tracking is variable, the previous object detection and tracking here is determined by the time point of the output result. In another embodiment, the detection tracker 131 selects which frame in the continuous image frames to perform object detection according to a fixed time interval, a fixed number of image frame intervals, or detection results of scene transitions. The start time point of the detection tracker 131 can be slightly earlier or slightly later than the previous embodiment based on a fixed time interval, a fixed image frame number interval, or the detection result of scene transition, but after the start, it will be based on a fixed time interval, fixed image The frame number interval or the detection result of the scene transition is selected to perform object detection on a specific frame in the continuous image frame, and selectively stop the previous object detection or object tracking, so as to increase the startup time of the detection tracker 131 point of elasticity.

To sum up, in the object tracking method and the object tracking device according to the embodiments of the present invention, the objects in the previously temporarily stored consecutive image frames can be tracked based on the object detection results. In this way, the accuracy of object tracking can be improved regardless of the type of the object (eg, human, animal, or non-biological). In addition, due to the high processing efficiency of the tracker, embodiments of the present invention can meet the requirements of real-time video or other low-latency applications.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (22)

1. An object tracking method suitable for low latency applications, the object tracking method comprising:

performing object detection on one of at least one continuous image frame, wherein the object detection is used for identifying a target;

temporarily storing the continuous image frames; and

performing object tracking on the temporarily stored continuous image frames according to the object detection result, wherein the object tracking is used for associating one of the continuous image frames with the target in another one of the continuous image frames.

2. The object tracking method of claim 1, wherein the step of temporarily storing the successive image frames comprises:

the successive image frames input to the system during all or part of the period before the object detection and object tracking are completed are temporarily stored.

3. The object tracking method of claim 1, wherein the continuous image frame comprises a plurality of continuous image frames, and the step of temporarily storing the continuous image frames comprises:

comparing the temporarily stored continuous image frames with the upper limit of the number; and

deleting at least one of the temporarily stored continuous image frames according to the comparison result of the continuous image frames and the number upper limit.

4. The object tracking method according to claim 1, wherein the result of the object detection includes a first region of interest of the target, the first region of interest corresponding to a position of the target in the continuous image frame subject to the object detection, and the step of performing the object tracking on the temporarily stored continuous image frame according to the result of the object detection includes:

and determining the relevance of the first interested region among a plurality of temporary stored continuous image frames, and determining a second interested region according to the relevance.

5. The object tracking method of claim 4, further comprising:

another object tracking is performed on said one of said successive image frames to determine a third region of interest.

6. The object tracking method of claim 5, further comprising:

updating the third interested region determined by the other object tracking to the second interested region, and performing the other object tracking again according to the second interested region.

7. The object tracking method of claim 6, wherein the step of updating the third region of interest determined by the other object tracking to the second region of interest comprises:

determining to update the third interested region to the second interested region or the weighted average of the second interested region and the third interested region according to the confidence of the third interested region in the tracking of the other object.

8. The object tracking method of claim 7, further comprising:

when the other object tracking is finished, updating the output final region of interest with the second region of interest or a weighted average of the second region of interest and the third region of interest.

9. The object tracking method of claim 1, wherein the step of performing the object detection on one of the successive image frames comprises:

determining the one for the object detection according to a time interval, an interval of frame numbers of the image, or a detection result of scene transition.

10. The object tracking method according to claim 1, further comprising:

when the object tracking is finished, updating the output final region of interest with the result of the object tracking.

11. The object tracking method according to claim 1, further comprising:

initiating another object detection on another of the consecutive image frames before the object tracking is completed according to a detection result of a time interval, an image frame number interval, or a scene transition, wherein the another object detection is also used for identifying the target; and

selectively stopping the object detection or the object tracking.

12. An object tracking device suitable for low latency applications, the object tracking device comprising:

a memory to store program code; and

a processor, coupled to the memory, configured to load and execute the program code to perform:

performing object detection on one of at least one continuous image frame, wherein the object detection is used for identifying a target;

temporarily storing the continuous image frames; and

performing object tracking on the temporarily stored continuous image frames according to the object detection result, wherein the object tracking is used for associating the target in one of the continuous image frames with the target in the other one of the continuous image frames.

13. The object tracking device of claim 12, wherein the processor is also configured to:

the successive image frames input to the system during all or part of the period before the object detection and object tracking are complete are temporarily stored.

14. The object tracking device of claim 12, wherein the continuous image frame comprises a plurality of continuous image frames, and the processor is further configured to:

comparing the temporarily stored continuous image frames with the upper limit of the number; and

deleting at least one of the temporarily stored consecutive image frames according to a comparison result of the consecutive image frames and the number upper limit.

15. The object tracking device of claim 12, wherein a result of the object detection comprises a first region of interest of the target, the first region of interest corresponding to a location of the target in the successive image frames detected by the object, and the processor is further configured to:

and determining the relevance of the first interested region among a plurality of temporary stored continuous image frames, and determining a second interested region according to the relevance.

16. The object tracking device of claim 15, wherein the processor is also configured to:

another object tracking is performed on said one of said successive image frames to determine a third region of interest.

17. The object tracking device of claim 16, wherein the processor is also configured to:

updating the third region of interest determined by the other object tracking to the second region of interest, and performing the other object tracking again according to the second region of interest.

18. The object tracking device of claim 17, wherein the processor is also configured to:

and determining to update the third interest region to the second interest region or a weighted average of the second interest region and the third interest region according to the confidence of the third interest region in the tracking of the other object.

19. The object tracking device of claim 18, wherein the processor is also configured to:

when the other object tracking is finished, updating the output final region of interest with the second region of interest or a weighted average of the second region of interest and the third region of interest.

20. The object tracking device of claim 12, wherein the processor is also configured to:

determining the one for performing the object detection according to a time interval, an interval of frame numbers of the image, or a detection result of the scene transition.

21. The object tracking device of claim 12, wherein the processor is also configured to:

when the object tracking is finished, updating the output final interested area with the result of the object tracking.

22. The object tracking device of claim 12, wherein the processor is also configured to:

initiating another object detection on another one of the consecutive image frames before the object tracking is completed according to a detection result of a time interval, an image frame number interval, or a scene transition, wherein the another object detection is also used for identifying the target; and

selectively stopping the object detection or the object tracking.

Images