TWI788758B - Target image tracking system and method - Google Patents

Abstract

The present disclosure provides a target image tracking method, which includes steps as follows. The image of the target object is obtained through the camera. The particle filter of multiple importance re-sampling is performed on the image of the target object. When calculating the particle similarity, the calculation is divided into multiple stages according to the computational complexity of the various target features, so as to track the target in real-time. The semantic segmentation is performed on the image of the target object, and the analysis result of semantic segmentation is utilized to predict the location of the target at the next time instant.

Description

Target image tracking system and method

The present invention relates to a tracking system and method, and in particular to a target image tracking system and method.

In recent years, due to the improvement of computer performance and the popularization of parallel computing, the application of image processing has become more and more extensive. The visual tracking of objects is one of them. It is mainly used in the monitoring and tracking of pedestrians and the detection of vehicles on the road. , military operations and UAV rescue and so on.

Traditional image tracking is widely used, however, it is prone to tracking failure.

The present invention proposes a target image tracking system and method to improve the problems of the prior art.

In an embodiment of the present invention, the object image tracking system proposed by the present invention includes a photographing device, a storage device, and a processor, and the processor is electrically connected to the photographing device and the storage device. The photographing device obtains the image of the target object, the storage device stores at least one instruction, and the processor is used to access and execute at least one instruction to: perform multiple importance resampling particle filters on the image of the target object, and calculate particle similarity according to The computational complexity of various target features is divided into multiple stages of calculation to track the target in real time; perform semantic segmentation on the image of the target, and predict the next location of the target based on the analysis results of the semantic segmentation.

In one embodiment of the present invention, the processor is used for accessing and executing at least one instruction to: execute the sparse representation to determine the occlusion rate of the target object; use the occlusion rate to determine whether to update the template of the target object.

In one embodiment of the present invention, the processor is used for accessing and executing at least one instruction: based on the optical flow information in the image, using the sparse optical flow method to determine the offset of the camera device, so as to correct the image displacement.

In an embodiment of the present invention, the processor is used to access and execute at least one instruction to: extract the region of the target object from the image through semantic segmentation, and execute the dense optical flow method to determine the regional movement trend of the target object, thereby predicting movement of the target.

In an embodiment of the present invention, the processor is used to access and execute at least one instruction to: determine the occlusion status of the target area; when the target area is occluded by other areas, mask multiple importance resampling particle filters Updates to templates for objects in .

In one embodiment of the present invention, the object image tracking method proposed by the present invention includes the following steps: (a) obtaining an image of the object through a camera device; (b) performing multiple importance resampling on the image of the object Particle filter, when calculating particle similarity, is divided into multiple stages according to the computational complexity of various target features, so as to track the target in real time; (c) perform semantic segmentation on the image of the target, according to the semantic segmentation Analyze the results and predict where the target will appear next.

In an embodiment of the present invention, the step (b) includes: performing a sparse representation method to determine the occlusion rate of the target object; using the occlusion rate to determine whether to update the template of the target object.

In an embodiment of the present invention, the step (b) includes: based on the optical flow information in the image, using the sparse optical flow method to determine the offset of the photographing device, so as to correct the image displacement.

In one embodiment of the present invention, step (c) includes: extracting the region of the target object from the image through semantic segmentation, and performing a dense optical flow method to determine the regional movement trend of the target object, thereby predicting the movement of the target object.

In an embodiment of the present invention, step (c) includes: judging the masking status of the target area; when the target area is covered by other areas, masking multiple importances and resampling the target object template in the particle filter update.

To sum up, the technical solution of the present invention has obvious advantages and beneficial effects compared with the prior art. The object image tracking system and object image tracking method of the present invention first implement a multi-importance re-sampling technique to solve the problems of low efficiency and inability to process images in real time when using multiple object features for image tracking. When calculating the similarity of various target features, the calculation is divided into multiple stages according to the complexity of the calculation, so as to save calculation time and achieve instant and robust tracking results. Furthermore, the present invention adds a novel and accurate semantic segmentation method under the particle filter tracking of the target object, which can obtain better tracking effect.

The above-mentioned description will be described in detail in the following implementation manners, and further explanations will be provided for the technical solution of the present invention.

In order to make the description of the present invention more detailed and complete, reference may be made to the accompanying drawings and various embodiments described below, and the same numbers in the drawings represent the same or similar elements. On the other hand, well-known elements and steps have not been described in the embodiments in order to avoid unnecessarily limiting the invention.

Please refer to FIG. 1, the technical aspect of the present invention is a target image tracking system 100, which can be applied in monitoring and detection systems, or widely used in various technical links. It is worth mentioning that the object image tracking system 100 of this technical aspect can achieve considerable technical progress and has extensive industrial application value. The specific implementation of the target image tracking system 100 will be described below with reference to FIG. 1 .

It should be understood that various implementations of the object image tracking system 100 are described in conjunction with FIG. 1 . In the following description, for ease of explanation, numerous specific details are further set forth in order to provide a comprehensive illustration of one or more embodiments. However, the technology may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to effectively describe the embodiments. The term "for example" used here means "as an example, instance or illustration". Any embodiment described herein as "by way of example" is not necessarily to be construed as preferred or advantageous over other embodiments.

FIG. 1 is a block diagram of an object image tracking system 100 according to an embodiment of the present invention. As shown in FIG. 1 , the object image tracking system 100 may include a storage device 110 , a processor 120 and a camera device 130 . For example, the storage device 110 may generally refer to a hard disk, memory, temporary register or other storage media, the processor 120 may be a central processing unit, and the photographing device 130 may be a color camera, a thermal imager and/or other image sensors detector. In an embodiment of the present invention, the photographing device 130 is only a single camera, which reduces the space and cost occupied by multiple cameras, and eliminates the operation of lens image matching.

Structurally, the processor 120 is electrically connected to the storage device 110 and the camera device 130 . It should be understood that, in the descriptions of the embodiments and the scope of the patent application, the description of "electrical connection" may generally refer to the indirect electrical coupling of one element to another element through other elements, or the direct electrical coupling of an element without passing through other elements. electrically connected to another component. For example, the storage device 110 may be an internal storage device directly electrically connected to the processor 120, or the storage device 110 may be an external storage device indirectly coupled to the processor 120 through an external electronic circuit.

When in use, the photographing device 130 acquires an image of a target object (such as a specific person), the storage device 110 stores at least one instruction, and the processor 120 is used to access and execute at least one instruction to: perform multiple importance on the image of the target object Resampling particle filter, when calculating the particle similarity, it is divided into multiple stages according to the computational complexity of various target features, so as to track the target in real time; perform semantic segmentation on the image of the target, based on the analysis of semantic segmentation As a result, the position where the target object will appear next is predicted. In this way, the object image tracking system 100 uses semantic segmentation to enhance the tracking effect of the particle filter, and more accurately predict the movement trend of the object.

Regarding the above-mentioned multi-importance resampling particle filter, for example, first, the color image captured by the photographing device 130 uses a multi-importance resampling particle filter to obtain multiple sampling points near the target object (considered as particle). Firstly, the weight score is obtained from the similarity analysis, and the sparse representation is used to judge the mask and update the template. And the offset of the photographing device 130 is obtained from the sparse optical flow to correct the image displacement. All particles are sorted according to their weight, and the top 10% are reserved. Gaussian random numbers are taken from these particles to make up about 90% of the particles that have been replaced. According to this rule, it loops repeatedly to achieve the tracking of the particle filter. It should be understood that the use of "about", "approximately" or "approximately" herein is used to modify any quantity that may vary slightly, but such slight changes will not change its essence. Unless otherwise specified in the embodiments, it means that the error range of the numerical value modified by "about", "approximately" or "approximately" is generally allowed within 20%, preferably within 10%. Within, and more preferably within five percent.

When calculating multiple importance resamplings, for example, the processor 120 first performs partial similarity calculations on all predicted state particles, and performs preliminary evaluation by feature comparison with a very fast calculation speed. First, evaluate the two faster similarities of color and thermal imager temperature based on the results predicted at the previous moment, and take out these two particles with higher similarity from the similarity comparison results. The second stage of calculation is more complicated Edge profile similarity verification. Then, a small number of relatively similar particles are selected from the similarity comparison results of the second stage, and the similarity comparison of the next stage is carried out, and so on, extending to the subsequent levels. In this process, the faster method will be selected first in the first stage, the purpose is to quickly screen out the more likely areas of the target and eliminate unnecessary calculations in the next stage. When performing the similarity verification in the subsequent stage, you can choose a verification method that is more complicated and has a slower calculation time. The similarity verification method that is the most complicated and takes the longest time will be in the final stage. In this way, because of the screening in the previous stage, it is possible to reduce the amount of calculations in areas with low target candidate probability, and at the same time, to accurately find the target through the comparison in the subsequent stage, and to maintain the immediacy of the target image tracking system 100 .

In practice, when the target is blocked by other objects, the particle filter may cause an update error, misjudging the position of the target at the next moment, resulting in tracking failure, and then it is more difficult to track back to the original target. Therefore, in an embodiment of the present invention, the processor 120 is used to access and execute at least one instruction to: execute the sparse representation method to determine the occlusion rate of the target object; use the occlusion rate to determine whether to update the template of the target object .

The storage device 110 can store the sparse appearance model, which includes the target template set and the broken template. The similarity calculation of the sparse representation is the result of linear combination of multiple templates (such as the template set of the target) and the broken template, which can track the target stably, especially when the target is obscured, it can still be correct It can accurately find out the position of the target object, and at the same time, it can estimate the degree of occlusion of the target object. Sparse representation In order to increase the robustness of tracking when the target is occluded, the occluded part of the image is pieced back into the original image with a fragmented template. The correlation coefficient of the broken template can be used as the basis for shading, and then estimate the shading degree of the target object, and determine whether the template about the target object is updated.

When the target object gradually changes, it will slowly start to differ from the original features. At this time, in order to keep tracking effectively, the template of the target object can be updated. For example, a moving target will have different color and texture characteristics due to changes in the viewing angle. Changes in sunlight or lighting will also cause changes in the appearance and color of the target. Timely and effective updating of target object templates can establish a complete set of templates to help strengthen the entire target object image tracking system 100 and reduce the chance of tracking failure. However, updating the template at the wrong time (such as when the target is occluded) will do the opposite, which will easily cause the tracking to fail, and the tracking template set will also record distortions, so that the tracking action will stay on the background or other objects in the environment. superior.

In addition to the previously calculated occlusion rate, the color image similarity between the object and the background can also be used to evaluate its uniqueness in the background environment. If the similarity between the target and the background is too high, it is not suitable to update the template, which will reduce the uniqueness of the target and easily lead to errors in the tracking process. In an embodiment of the present invention, when the similarity between the object and the background is less than a preset ratio and the occlusion rate is within a preset range, the processor 120 updates the template and stores it in the storage device 110 .

Regarding the above preset ratio and preset range, for example, the similarity between the target object and the background is less than 40%, and the occlusion rate is between 10% and 30%. When the occlusion rate exceeds 30%, it is considered that the target has been occluded, and it is not suitable to update the template at this time. And when the occlusion rate is lower than 10%, it will not be updated, because the target object may be in a motionless state. At this time, the update template will easily lose the previous change characteristics, and the template set will have highly correlated characteristics, which will not be conducive to future tracking.

In the field of computer image vision, optical flow represents the moving speed of objects in the image, and each point of the image on each frame (frame) contains the direction of the momentary movement, and moves to the point of the next frame, so Optical flow is a representation in vector form. The entire image captured by the photographic device 130 is analyzed by the optical flow method, and the motion field of all pixels in the entire image can be obtained, which is dense optical flow (Dense Optical Flow). However, if you only take a small number of corner points in the image, usually at intervals of several pixels, and then calculate the moving speed, you will get sparse optical flow (Sparse Optical Flow). In one embodiment of the present invention, the processor 120 is configured to access and execute at least one instruction: based on the optical flow information in the image, use the sparse optical flow method to determine the offset of the photographing device 130, so as to correct the displacement of the image . In this way, the particle filter tracking part will use sparse optical flow to correct the movement of the camera 130 , and the follow-up semantic segmentation tracking analysis will use dense optical flow to strengthen the judgment of the movement trend of the target object.

For example, when the source image is taken by a human hand, or on a moving object (such as: a moving vehicle), or a monitor fixed on a street corner, it may also be blown by strong winds shaking. In such situations, in order to track the target more accurately and eliminate external interference factors, the present invention uses sparse optical flow to feed back the action of the camera 130 to the target image tracking system 100 for correction, which is more effective. Help stabilize target tracking.

Semantic Segmentation of images is one of the most important links under the development of artificial intelligence. It is a visual recognition technology under Deep Learning. It can deeply develop the entire visual image, targeting each pixel With point recognition technology, individual classifications are converted into segmented images. In an embodiment of the present invention, the processor 120 is used to access and execute at least one instruction to: extract the region of the target object from the image through semantic segmentation, execute the dense optical flow method to determine the regional movement trend of the target object, To predict the movement of the target object. In this way, in order to strengthen the tracking, in addition to relying on the objects separated by semantic segmentation, the present invention can also use dense optical flow to strengthen the judgment of prediction, and assist the tracking effect of the particle filter according to the movement trend of the object.

In one embodiment of the present invention, the processor 120 is configured to access and execute at least one instruction to: determine the occlusion status of the target area; when the target area is occluded by other areas, block multiple importance resampling particles The update of the template in the filter with respect to the object. Specifically, in the occluded state, the template update of the particle filter is stopped to prevent the multi-importance resampling particle filter from being updated to a wrong template, causing tracking failure or tracking to a wrong target. In the unoccluded state, it is normal to notify the multi-importance resampling particle filter to update the template normally, and it can continue to track after maintaining the shape of the target. In this way, semantic segmentation can strengthen the multiple importance resampling particle filter and assist the tracking effect of the multiple importance resampling particle filter.

Based on the above, multiple importance resampling particle filters and semantic segmentation can simultaneously process the images captured by the camera 130 in parallel, and after being processed by semantic segmentation, more accurate features of the target object and its neighboring objects can be obtained. Then use the dense optical flow method to predict the dynamics of the target and its neighbors, predict the position at the next moment, and feed back to the particle filter to filter out the wrong particles. In addition, at the same time, the above dynamic information is combined to make a masking judgment, and the judged masking status is also sent back to the multi-importance resampling particle filter as a basis for updating the template.

In order to further explain the method performed by the above object image tracking system 100 , please refer to FIGS. 1 and 2 . FIG. 2 is a flow chart of an object image tracking method 200 according to an embodiment of the present invention. As shown in FIG. 3, the object image tracking method 200 includes steps S201-S209 (it should be understood that the steps mentioned in this embodiment can be adjusted according to actual needs unless the order is specifically stated. sequentially, even concurrently or partially concurrently).

The object image tracking method 200 may take the form of a computer program product on a non-transitory computer-readable recording medium having a plurality of computer-readable instructions embodied in the medium. Suitable recording media may include any of the following: Non-volatile memory such as: Read Only Memory (ROM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM); Volatile Memory, such as: Static Access Memory (SRAM), Dynamic Access Memory (SRAM), Double Data Rate Random Access Memory (DDR -RAM); optical storage devices, such as: CD-ROM, DVD-ROM; magnetic storage devices, such as: hard disk drive, floppy disk drive.

In one embodiment of the present invention, in the object image tracking method 200, the image of the object is obtained through the photographing device 130; multiple importance resampling particle filters are performed on the image of the object, and when calculating the particle similarity, According to the computational complexity of various target features, it is divided into multiple stages of calculation to track the target in real time; perform semantic segmentation on the image of the target, and predict the next position of the target based on the analysis results of the semantic segmentation.

In the object image tracking method 200, steps S201-S205 are mainly related to multiple importance resampling particle filters, and steps S206-S209 are mainly related to semantic segmentation.

Specifically, in step S201 , an image of the target object is obtained through the photographing device 130 . In step S202, similarity analysis is performed, that is, multiple importance resampling particle filters are performed on the image of the target object. When calculating the particle similarity, the calculation is divided into multiple stages according to the computational complexity of various target object features. Track objects in real time. In step S203, the sparse representation masking process is performed, that is, the sparse representation method is executed to determine the masking rate of the target object; and whether the template of the target object is updated or not is determined by using the masking rate. In step S204, optical flow correction and dynamic correction are performed, that is, based on the optical flow information in the image, the sparse optical flow method is used to determine the offset of the photographing device, so as to correct the displacement of the image.

On the other hand, in step S206, image segmentation is performed through semantic segmentation. In step S207, object classification region extraction is performed, that is, the region of the object is extracted from the image through semantic segmentation. In step S208, the dynamic comparison of dense optical flow is performed, that is, the dense optical flow method is used to judge the regional movement trend of the target object, and based on this, the target object is estimated in step S209, so that in step S205, the target object is predicted, and the target object is predicted trend.

Furthermore, in step S209 , an occlusion determination is performed to determine the occlusion state of the area of the target. Particle update is performed, that is, after semantic segmentation processing, the position at the next moment is predicted, and fed back to multiple importance resampling particle filters to filter out particles with wrong positions. In addition, when it is determined in step S209 that the area of the object is covered by other areas, in step S205 , the updating of the templates related to the object in the multi-importance resampling particle filter is shielded.

To sum up, the technical solution of the present invention has obvious advantages and beneficial effects compared with the prior art. The object image tracking system 100 and the object image tracking method 200 of the present invention first implement a multi-importance resampling technique to solve the problem of low efficiency and inability to process in real time when using multiple object features for image tracking. When calculating the similarity of various target features, the calculation is divided into multiple stages according to the complexity of the calculation, so as to save calculation time and achieve instant and robust tracking results. Furthermore, the present invention adds a novel and accurate semantic segmentation method under the particle filter tracking of the target object, which can obtain better tracking effect.

Although the present invention has been disclosed above in terms of implementation, it is not intended to limit the present invention. Anyone skilled in this art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection of the present invention The scope shall be defined by the appended patent application scope.

In order to make the above and other objects, features, advantages and embodiments of the present invention more obvious and understandable, the accompanying symbols are explained as follows: 100: Target image tracking system 110: storage device 120: Processor 130: Photographic device 200: Target Image Tracking Method S201～S209: steps

In order to make the above and other objects, features, advantages and embodiments of the present invention more clearly understood, the accompanying drawings are described as follows: FIG. 1 is a block diagram of an object image tracking system according to an embodiment of the present invention; and FIG. 2 is a flowchart of a target image tracking method according to an embodiment of the present invention.

200: Target Image Tracking Method

S201~S209: steps

Claims (10)

A target image tracking system, comprising: a photographing device for obtaining an image of a target; a storage device for storing at least one instruction; and a processor electrically connected to the photographing device and the storage device, the processor for storing Fetching and executing the at least one instruction to: perform a multiple importance resampling particle filter on the image of the target object, and divide the calculation into multiple stages according to the computational complexity of various target object characteristics when calculating the particle similarity , to track the target in real time, wherein the multiple importance resampling particle filter obtains multiple sampling points of the various target features by Gaussian random numbers near the target, and the lower the computational complexity of the multiple target features The features of the target object are processed at a stage with higher priority; and a semantic segmentation is performed on the image of the target object, and the location where the target object appears next is predicted according to the analysis result of the semantic segmentation. The object image tracking system as described in claim 1, wherein the processor is used to access and execute the at least one instruction to: execute a sparse representation to determine an occlusion rate of the target object; and use the occlusion rate to determine Whether the template for this object is updated or not. The object image tracking system as claimed in claim 1, wherein the processor is configured to access and execute the at least one instruction to: Based on an optical flow information in the image, a sparse optical flow method is used to judge the offset of the photographing device, so as to correct the image displacement. The object image tracking system as described in Claim 1, wherein the processor is used to access and execute the at least one instruction to: extract a region of the object from the image through the semantic segmentation, perform a dense light The flow method judges the movement trend of the target object in this area, so as to predict the movement of the target object. The object image tracking system as described in claim 4, wherein the processor is used to access and execute the at least one instruction to: determine the occlusion state of the area of the object; and when the area of the object is covered by other areas When masking, mask updates of a template in the multi-importance resampling particle filter with respect to the object. A target object image tracking method, comprising the following steps: (a) obtaining an image of a target object through a photographic device; (b) performing a plurality of importance resampling particle filters on the image of the target object, and calculating When particle similarity is calculated, it is divided into multiple stages according to the computational complexity of various target features to track the target in real time, wherein the multiple importance resampling particle filter obtains multiple The various target features of the sampling points, the target features with lower computational complexity among the various target features, rank Processing at a stage with higher priority; and (c) performing a semantic segmentation on the image of the target object, and predicting the next appearance position of the target object according to the analysis result of the semantic segmentation. The target object image tracking method as described in claim 6, wherein step (b) includes: performing a sparse representation method to determine an occlusion rate of the target object; and using the occlusion rate to determine the template update and the target object no. The object image tracking method as described in claim 6, wherein step (b) includes: based on an optical flow information in the image, using a sparse optical flow method to judge the offset of the photographing device, and correct the image accordingly displacement. The target object image tracking method as described in claim 6, wherein step (c) includes: extracting a region of the target object from the image through the semantic segmentation, and performing a dense optical flow method to determine the target object Regional movement trend, so as to predict the movement of the target. The object image tracking method as described in Claim 9, wherein step (c) includes: judging the occlusion state of the area of the object; and when the area of the object is occluded by other areas, shielding the various Importance Resamples the update of a template for the object in the particle filter.

Images