JP7630935B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to a technology for tracking a subject.

Technologies for tracking a specific subject within an image include those that use brightness or color information, and template matching. In recent years, technology that uses Deep Neural Networks (hereafter abbreviated as DNN) has been attracting attention as a highly accurate tracking technology.

Non-Patent Document 1 is one method for tracking a specific subject in an image. An image containing the target to be tracked and an image that is the search range are input to a Convolutional Neural Network (hereafter abbreviated as CNN) with the same weights. The position of the target to be tracked in the image in the search range is identified by calculating the cross-correlation between the features obtained from the CNN. While this type of tracking method can accurately identify the position of the target to be tracked, it is prone to failing to track the wrong target when an object similar to the target to be tracked overlaps on the screen.

To avoid this, there are methods such as that described in Patent Document 1, which create a histogram from the color features and depth information of the area of the detected object, and then check the changes in this to determine whether the object is occluded or not.

U.S. Patent Application No. 10185877(B2)

Bertinetto et al. , Fully-Convolutional Siamese Networks for Object Tracking, arXiv 2016

However, the method shown in Patent Document 1 has the problem that when objects with similar postures or external characteristics overlap on the screen, it is difficult to determine whether they are the same or not because differences in the histograms of features such as color and texture are difficult to detect. For example, in group sports competitions, multiple people in a small area often wear the same clothes and postures, and different people may fail to be tracked as the same person. The present invention has been made in consideration of such problems, and aims to continue stable tracking even when objects with similar external characteristics and postures are close to each other.

The information processing device according to the present invention that solves the above problem is an information processing device that detects objects from an image, and includes: a feature extraction means that extracts image features of objects detected from the image; an estimation means that estimates, for each object detected from the image, occlusion information that indicates an occlusion relationship with other objects detected from the image based on the output of a trained model that outputs a likelihood that an object detected from the image is occluded by another object; and an identification means that identifies, for each object detected from the image, a correspondence relationship with an object detected in an image captured at a time different from that of the image, based on at least the image features and the occlusion information.

The present invention allows for stable and continuous tracking even when objects with similar external features and poses are in close proximity.

FIG. 1 is a schematic diagram illustrating an example of object detection; FIG. 1 is a diagram showing an example of a hardware configuration of an information processing device; FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; 1 is a flowchart showing a processing procedure executed by an information processing device; FIG. 13 is a diagram showing an example of a result of processing by an information processing device; FIG. 13 is a diagram showing an example of a result of processing by an information processing device; 1 is a flowchart showing a processing procedure executed by an information processing device; Example of deriving occlusion information FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; FIG. 13 is a diagram showing an example of a result of processing by an information processing device; 1 is a flowchart showing a processing procedure executed by an information processing device; FIG. 13 is a diagram showing an example of a learning process of an information processing device; FIG. 13 is a diagram showing an example of a result of processing by an information processing device; FIG. 13 is a diagram showing an example of a result of processing by an information processing device; FIG. 13 is a diagram showing an example of a result of processing by an information processing device; FIG. 13 is a diagram showing an example of a learning process of an information processing device;

<Embodiment 1>
The information processing device according to the embodiment will be described with reference to the drawings. Note that the same reference numerals in the drawings perform the same operation, and therefore the description will be omitted. Also, the components shown in the embodiment are merely examples, and are not intended to limit the scope of the present invention to those only.

In this embodiment, a function for detecting and tracking a person from a video or continuously captured still image frames will be described. The scope of application is not limited to the category of object to be detected and tracked, but in this embodiment 1, the target is limited to a person. In this embodiment, a person is detected in each temporally consecutive image, and a person is associated with a person in each consecutive image, thereby tracking the person. In particular, this embodiment assumes the shooting of a sporting event, etc., in which the people's clothing, direction of movement, etc. are similar, and they frequently approach and cross paths. In such cases, simply associating people with similar external characteristics, such as the position of the person in each image or the color of their clothing, is likely to result in incorrect association. Such a failure is referred to here as incorrect matching.

In this embodiment, we focus on the information regarding occlusion output by a trained model that has learned the patterns of occlusion relationships when objects overlap as seen by the photographer. By using the occlusion relationships output by the trained model in conjunction with identifying the correspondence between objects, we can reduce mistakes in matching people in the foreground with people in the background, improving tracking accuracy.

This is shown in Figure 1. Images 2100, 2120, and 2140 in Figure 1(A) show three still frames of a video showing two playing cards of the same design crossing each other on a table from above (arranged from left to right in chronological order). It is not possible to determine how the cards in each image moved by observing only images 2100, 2120, and 2140. On the other hand, Figure 1(B) is an example of the same scene captured at a higher frame rate than (A). In other words, it is a group of images captured at a shorter time interval. If you observe the images in Figure 1(B) in chronological order, you can associate which card moved to where in the next image. Overall, it is relatively easy to estimate that the card 2201 on the left passed over the card 2202 on the right and moved to the right. As shown in images 2210 and 2230, by observing the appearance that occurs transiently at the moment when objects cross each other, it is possible to determine which card passes in front and which card is in the back in image 2220. This determination does not necessarily require a special sensor such as a 2.5-dimensional depth image or information with high generation costs such as optical flow. It can be solved as a pattern recognition problem between the occlusion relationship and the appearance feature, which is what appearance is likely to occur when objects overlap in front and back. This is also true for scenes such as the intersection of people shown in Figures 1(C) and 1(D). In these Figures 1(C) and 1(D), the appearance of the people's clothes, posture, etc., and the direction of movement are the same. In such a case, if observing while paying attention to the appearance state before and after the objects cross, it is determined that the person 2401 is in the front and the person 2402 is in the back in image 2420 of Figure 1(D). If this occlusion relationship is maintained in subsequent images, when person 2401 occludes person 2402 once, the person 2401 in the foreground is tracked, and the occlusion relationship and image features of person 2401 in the background are maintained. Then, when the occlusion is removed, it is possible to continue tracking person 2401 and detect person 2402 in the background again. This is an explanation showing an overview of the principle of this embodiment. Detailed processing will be described later.

2 is a hardware configuration diagram of an information processing device 1 that tracks a tracking target by image recognition in this embodiment. The CPU H101 executes a control program stored in the ROM H102 to control the entire device. The RAM H103 temporarily stores various data from each component. It also expands the program and makes it executable by the CPU H101. The storage unit H104 stores data to be processed in this embodiment and stores data to be tracked. As a medium for the storage unit H104, a HDD, a flash memory, various optical media, etc. can be used. The input unit H105 is composed of a keyboard, a touch panel, a dial, etc., and accepts input from the user, and is used when setting a tracking target, etc. The display unit H106 is composed of a liquid crystal display, etc., and displays the subject and tracking results to the user. In addition, this device can communicate with other devices such as a shooting device via the communication unit H107.

Figure 3 is a block diagram showing an example of the functional configuration of an information processing device. In Figure 3, the processes executed in the CPU H101 are shown as functional blocks. The information processing device 1 has an image acquisition unit 201, an object detection unit 202, an occlusion information generation unit 203, an extraction unit 204, and a matching unit 205, and is connected to an external storage unit 206. The storage unit 206 may be inside the information processing device 1. The functions of each unit will be briefly described. The image acquisition unit 201 acquires images of a video or continuous still images in which a specific object (a person in this embodiment) is captured by an imaging device. The object detection unit 202 detects image features indicating a predetermined object from the images acquired by the image acquisition unit 201. For example, the area of a person in an image is detected based on a learned model that has previously learned image features indicating a human body (head or moving body) using images of people in various postures. The occlusion information generating unit 203 estimates occlusion information indicating an occlusion relationship between each object detected from an image and other objects detected from the image based on a trained model that has learned image features indicating an occlusion relationship between an occluding object and an occluded object. The occlusion information is a likelihood (occlusion/occlusion score) that indicates the possibility that the target object is occluded by other objects. For example, if an object is highly likely to be occluded by other objects, the occlusion/occlusion score is made closer to 1. If an object is highly likely to occlude other objects, the occlusion/occlusion score is made closer to 0. The occlusion/occlusion score indicating such an occlusion relationship is estimated using the trained model. The extracting unit 204 stores occlusion information for each object detected in an image in the storage unit 206. The matching unit 205 matches objects detected between multiple images. That is, based on the occlusion information, a correspondence relationship between each object detected from an image and an object detected in an image captured at a different time from the image is identified. Objects can be tracked by correctly associating objects detected from images captured at different times. In addition, by assuming that the occlusion relationship between objects is maintained for a certain period of time, objects can be associated with each other using the occlusion relationship. The storage unit 206 stores the occlusion score of each detected object. Details of the processing of each functional unit will be explained using the flowchart in FIG. 4.

Figure 4 is a flowchart showing the processing flow of this embodiment. In the following explanation, each process (step) is represented by adding an S to the beginning, and the notation of the process (step) is omitted. However, the information processing device does not necessarily have to perform all of the processes described in this flowchart. The processing shown in the flowchart in Figure 4 is executed by the CPU H101 of Figure 2, which is a computer, in accordance with a computer program stored in the memory unit H104.

In S301, the information processing device 1 starts a loop process that is repeated for each video frame. In S302, the image acquisition unit 201 sequentially acquires image frames of a video or continuous still images capturing a person. Subsequent processing from S301 to S311 is performed sequentially for each image. Note that the image acquisition unit 201 may acquire images captured by an imaging device connected to the information processing device, or may acquire images stored in the storage unit H104. Video frames 3100, 3110, 3120, 3130, and 3140 in FIG. 5(A) are examples of acquired image frames.

Next, in S303, the object detection unit 20 detects at least one or more predetermined objects from the acquired image based on the image features of the predetermined objects (people in this case). A known technique for detecting objects from within an image is the method by Liu (Liu, SSD: Single Shot Multibox Detector. In: ECCV2016). The result of detecting candidate objects from within an image is shown in FIG. 5(A). The rectangular frames 3101, 3102, 3103, 3111, 3112, 3113, 3121, 3122, 3131, 3132, 3141, and 3142 in FIG. 5(A) are bounding boxes (hereinafter BB) that indicate the detected object regions.

In S304, the occlusion map generating unit 203 generates an occlusion map for each image that indicates occlusion information for each region as to whether it is occluded or not (occlusion relationship). The occlusion map generating unit 203 estimates the visible region (occluded object region) of the occluded object for each image. Here, it is determined whether each person overlaps with another person, and if so, whether the person is in the back or the front, and the result is output as an occlusion state score (likelihood) for each region. This is a type of semantic region segmentation recognition task, and can be realized using known methods such as the method of Chen et al. (Chen, DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, 2016).

Figure 6 (A) shows a schematic diagram explaining the generation process of the occlusion map and an example of the result. The neural network 402 is a neural network that judges the occlusion state of each pixel of the input image from the input image. When an RGB image 401 is input, the neural network 402 outputs the result of estimating whether or not a person is present in the image and whether or not the person is occluded as an occlusion map 404. The map outputs an occlusion score of 0 for an area estimated to be an unoccluded person or a non-person area, and 1 for an area of an occluded person. The darker the area in the occlusion map 404, the higher the occlusion score. In other words, the black area indicates that it is estimated to be an area of an occluded person. The neural network 402 has been trained in advance so that it can output such an output for the input image (learning will be described later). The occlusion map 404 shown in the figure shows an example of an ideal output state as an estimation result.

In addition to the RGB image 401, a derivative form may be considered in which a 2.5-dimensional depth image 405 is separately acquired using a dedicated sensor or the like. The depth image 405 is linked to the 3-channel RGB image 401 to generate 4-channel information, which is learned and recognized as an image input instead of an RGB image. This makes it possible to obtain information on occluded areas with higher accuracy.

Next, in S305, the information processing device 1 executes the loop process from S306 to S307 for each object detected in S303. In S305 to S308, the extraction unit 204 extracts information indicating the occlusion relationship for each detected object from the generated occlusion map and stores it in the storage unit 206. In S306, the extraction unit 204 extracts information indicating the occlusion relationship for each detected object from the occlusion map. Specifically, the occlusion scores of the occlusion map 404 in the person detection frame 407 in FIG. 6A are averaged. The closer this occlusion score is to 1, the more likely the object is occluded, and the closer the occlusion score is to 0, the more likely the object is not occluded. Note that, assuming that the position of the detection frame is shifted or that the occlusion map 404 contains noise, the weighted average is obtained with emphasis on the center of the frame as shown in FIG. 6B. The calculation 412 in the figure means the element product (Hadamard product) for each partial region of each image. Map 413 has a peak in the center and is a two-dimensional Gaussian function whose image block sum is 1 (the vertical and horizontal sizes have been transformed to fit the person detection frame). An example of the obtained results is shown in FIG. 6(A) with the symbol occ as occlusion score values 409 and 410. The detection frame on the left is determined to have a high occlusion score because the person is at the back, and the detection frame on the right is determined to have a low occlusion score because the person is at the front. Examples of the results of processing the above-mentioned process on the input image in FIG. 5 are shown in FIG. 5(B) (occlusion map) and FIG. 5(C) (estimated occlusion score for each frame). This shows that the occlusion score corresponding to person 3101 located at the back is relatively higher than that of person 3102 between the start and end of the intersection.

In S307, the memory unit 206 stores the occlusion score of each detected object acquired by the extraction unit 204. At the same time, information on the position and size of each detected object, and feature quantities related to the object's appearance, such as color and texture histograms, are also stored. Here, these multiple types of quantities are collectively referred to as the feature quantities of the detected object. Note that, in addition to this, intermediate layer information of a neural network may also be used as the feature quantities related to the appearance. (For example, "Hariharan, et.al, Hypercolumns for Object Segmentation and Fine-grained Localization, in CVPR2015").

In S308, the information processing device 1 ends the loop that is repeated for each image and the loop that is repeated for the people detected in each image. This loop ends for each image when occlusion information has been acquired for all people detected in that image. Next, in S309 to S310, the matching unit 205 matches objects between previous and next images (however, this is not performed in the case of the first video frame because there are no previous frames). First, in S309, the matching unit 205 acquires the occlusion score, position/size, and appearance features, which are features of past objects stored in the memory unit 206. Next, in S310, the matching unit 205 matches objects detected in past video frames with objects detected in the frame currently being processed.

A detailed process flow of the matching unit 205 in S310 is shown in FIG. 7. In S501, the matching unit 205 first creates pairs of all combinations between the detected object in the current frame and the object detected in the previous frame. If n people and m people are detected in the previous and next frames, respectively, a total of n×m pairs are generated. Next, in S502, the matching unit 205 calculates the similarity between all object pairs. As the similarity, an index based on the difference in the feature amounts between the detected objects can be used. As an example, the similarity between the past detected object _c1 and the current detected object _c2 is calculated as follows:
(Formula 1)
L(c ₁ , c ₂ )= -W ₁ | | BB ₁ -BB ₂ | |
-W ₂ | | f ₁ - f ₂ | | -W ₃ | | occ ₁ - occ ₂ | |
Here, BB is a vector summarizing four variables (center coordinate value x, center coordinate value y, width, height) of each object, and f indicates the characteristics of each object. ||x|| is the ^Lp norm of x. occ is the occlusion score of each object. W1, W2, and W3 are balance coefficients equal to or greater than 0 that are set by empirical or machine learning adjustment. Here, the variation of each feature may be statistically obtained in advance and each feature may be normalized. Even when objects intersect with each other, by tracking a person on the side that occludes other objects, when the person on the occluded side is confirmed again in the image, the value of the third term of Equation 1 becomes small, and the similarity is calculated to be low. In other words, this process reduces the possibility of matching people with different occlusion relationships, and erroneous matching of tracking can be suppressed.

Next, in S503, the correspondence unit 205 performs correspondence (matching) to identify the correspondence between objects based on the similarity between past objects and current objects. There are several matching methods. For example, there is a method of prioritizing matching among candidates with high similarity, and a method using the Hungarian algorithm. Here, the former is used.

In S503, if the matching unit 205 has not yet completed matching for all objects in the current frame, it proceeds to S506, where it matches the pair with the greatest similarity as the same person. Once matching has been completed, the objects of the pair are removed from the matching candidates. During the above process, if the magnitude of the greatest similarity among the pairs remaining at that point falls below a predetermined threshold, this means that there are no more similar object pairs remaining. In that case, it does not attempt to match any further (S505) and ends the matching process.

The above steps S301 to S311 are performed for each video frame. As a result, as shown in the example result in FIG. 5(D), people are detected from the video, and a series of tracking results are obtained showing where each object has moved (the same people between frames are marked with symbols A, B, and C to show the tracking results).

<Modification>
In this embodiment, the distance of each index such as the occlusion score and the appearance is weighted and summed based on the difference as the similarity of matching between object pairs. Here, for example, KL divergence can be used. It is also possible to perform metric learning to obtain a more accurate distance index. In addition, instead of using a single similarity only once, a rule-based method or a stepwise judgment method can be used in which the similarity is first judged based on the appearance features, and if the condition is satisfied, it is judged based on the similarity of the occlusion score. Furthermore, it is also possible to use a known classifier method such as a neural network or a support vector machine to learn and classify the explanatory variables as features and the objective variables as the result of whether or not they are the same object, and to judge matching based on this value. As described above, the correspondence between objects between frames is not limited to a specific form.

As yet another derivative, in order to stabilize the estimated value of the occlusion score, a moving average of past scores may be used as shown in the following formula.
(Formula 2)
occ ^EMA _(t) = (1-α)×occ ^EMA _(t-1) + α×occ _(t)
The above formula is a value called the exponential moving average, occ ^EMA _(t) is the exponential moving average of the occlusion score at time t, occ _(t) is the occlusion score at time t, and α is a coefficient of 0<α≦1. For objects that have been tracked in multiple past frames, the exponential moving average is calculated using the above formula, and when comparing similarities, the exponential moving average occlusion score is used instead of the original occlusion score. This allows matching based on the average value of the occlusion scores for multiple frames in cases where an overlapping state gradually occurs across multiple frames at the time of intersection, making the correspondence between objects more stable.

As yet another derived form, matching can be performed using not only the similarity between previous and next frames, but also the feature values and positions of multiple past frames n steps back. By using this method, even if an object is occluded and cannot be tracked in a frame, it can be tracked again if the occlusion is removed in a subsequent frame. In this form, for example, the average feature value of the object is calculated going back n frames, and the similarity with the object detected in the current frame is calculated based on this. Alternatively, the similarity between the object in the past n frames and the object in the current frame is calculated, and the object with the highest average value of the n similarities obtained is associated with the object. It is also possible to perform bidirectional judgment using not only the past but also the results of future frames n steps. This form is inferior in real-time processing because the results are not known until future frames are processed, but it is more accurate than the method of looking only at the past.

Yet another possible variant is one that deals with detection failures. In object detection and tracking, temporary failures in object detection can occur due to reasons such as an object's posture changing to an unusual shape. If such a failure to detect occurs, when matching frames, an object that existed in the previous frame is determined to not match in the current frame. This causes tracking to be interrupted. To prevent such failures, the following can be devised. That is, if an unmatched person is encountered in matching, the information about that person is stored in a list, and the person is added to the list of candidates for matching when matching the next frame (if a certain amount of time has passed and the object is still unmatched, it is determined that the object itself no longer exists and is removed from the list. Here, this is called timeout processing).

There are various ways to match objects across video frames in this way, and we are not limited to any particular form.

<Variations in the form of occlusion information and learning methods>
In this embodiment, the visible area (occluded object area) of the occluded object is estimated as the occlusion map. Various derived forms are also conceivable. An example is shown in FIG. 8. Here, as shown in FIG. 8B, it is possible to estimate other than the visible area of the object on the back side as in image 801. For example, the entire area of the object on the back side is estimated as in image 802. Also, as in image 803, the area of the occluding object on the front side is estimated (an object that does not overlap with other objects as in area 440 in the figure is also estimated as the front side area. However, as another form, it is conceivable not to include such a single object in the front side area). Also, instead of estimating the foreground area as in images 801 to 803, it is conceivable to estimate the position of the center or center of gravity of the object as in image 804. In the case of image 804, a large positive value is estimated in the area near the center of the occluded object, and a small negative value is estimated in the area near the center of the occluded object (the central area of the object here can be estimated as a Gaussian function area as shown in the figure).

Here, the learning method of the information of the occlusion state will be explained with reference to FIG. 6(D). The neural network 402 shown in the above-mentioned Chen et al. method etc. outputs the occlusion score map 403 of the occluded object for the RGB image 401, which is the input image. An example of the result of 403 is shown in 430. The CHEN et al. method etc. is a method for estimating the foreground region of a specific category object, but here, a teacher value 431 of the occlusion information is given, and the neural network 402 is trained so that a map similar to the teacher value 431 can be obtained by estimation. Specifically, the output result map 403 is compared with the teacher value 431, and a loss value calculation 432 is performed by a known method such as cross entropy or square error. The weight parameters of the neural network 402 are adjusted by the error backpropagation method etc. so that the loss value gradually decreases (this process is the same as the Chen et al. method, so details are omitted). It is necessary to give a sufficient amount of the input image and the teacher value. Creating training values for overlapping object regions is costly, so it is possible to create training data using CG or an image synthesis method that cuts out and overlaps object images. This concludes the learning method.

Furthermore, in this embodiment, an index called an occlusion score is obtained within the object frame obtained above. Various examples of the form of obtaining occlusion information are shown in FIG. 8 (C). FIG. 8 (C1) shows the form of this embodiment. In addition, various other methods are possible, such as (C2) obtaining the difference value between the occlusion score on the back side and the occlusion score on the front side within the object frame, or (C3) referencing only one point of the score at the center of the object. Also, when obtaining within a frame, a method is conceivable in which areas that overlap with other object frames are omitted from the obtaining because it is unclear which object's area it is.

Furthermore, a method can be considered in which the above-mentioned <estimation of the occlusion state> and <obtainment of the occlusion score for each object> are performed simultaneously. One example is a method called anchor, which is a well-known method used in Liu's method and others. This method obtains a set of candidate frames for objects, and it is possible to use this to estimate and associate each candidate frame with an occlusion score indicating whether it is an occluding object or an occluded object (the details of this form will be shown in embodiment 3, so the explanation will be omitted here).

Furthermore, multiple forms of occlusion information as shown above may be obtained and used as multidimensional features related to occlusion in subsequent object matching. Alternatively, the proportion of the object's occluded area may be estimated by machine learning from the multidimensional features related to occlusion. In this case, the multidimensional features related to occlusion may be used as explanatory variables, the proportion of the object's occluded area may be used as a target variable, and regression estimation may be performed using a publicly known technique such as logistic regression.

<Embodiment 2>
In this embodiment, detection and tracking of a person are performed in the same manner as in the first embodiment. The hardware configuration is the same as in FIG. 2 of the first embodiment. A block diagram showing an example of the functional configuration in this embodiment is shown in FIG. 9A. A shielding state determination unit 301 is newly added to the configuration of the first embodiment. In the first embodiment, during tracking, the frame of a person may not be able to detect a person due to overlapping of the people. For example, as shown in the video frame 4120 in FIG. 10A, when the overlapping area between people is large, it is often the case that the person in the back cannot be detected. In such a case, the shielding state determination unit 301 determines that a person is present but is in a shielded state.

Unlike non-detection caused by a temporary failure of the object detection unit as described in the first embodiment, when a group of people are moving in the same direction at the same speed, the non-detection state continues for a long time. Furthermore, the position on the screen where the object leaves the occluded state may be far from the position where the occluded state began. For this reason, it is desirable to increase the success rate of tracking by proactively determining that the object is in an occluded state and performing processing according to the estimated state.

In this embodiment, the overall processing flow is the same as in FIG. 4 of the first embodiment, but the details of the processing in S310 are different as follows. Here, only the processing in S310 that differs from the first embodiment will be described. A detailed flow of the processing in S310 performed by the occlusion state determination unit 301 will be described with reference to FIG. 11. First, in S601, objects are associated with each other in the current frame and the previous frame as in the previous embodiment. If there is an object in the previous frame that was not associated with each other in S602, it is possible that the object has entered an occluded state. Therefore, in S603, it is checked whether the occlusion score of the object up to that point is equal to or higher than a threshold value. This is because if the frame rate of the video frame is sufficiently high, the occlusion score often becomes high before and after the object becomes undetected due to occlusion. Furthermore, in S604, it is checked whether the number of detections of the object in the current frame has decreased in the surrounding area of the object, and if the above two results are true, it is estimated that the object has entered an occluded state and is stored in the list of occluded states (S605). For the objects stored in the list of occluded states, the feature amount and position at the time of the previous detection are also stored. This improves the chances of tracking when the object is unobstructed and detected again.

S606 to S610 are processes for judging whether an object in an occluded state has reappeared. If there is an object in the current frame that was not associated in S603, there is a possibility that it has escaped from the occluded state and can be detected again. Therefore, in S607, it is checked whether the occluded score of the object is equal to or higher than a threshold. Furthermore, in S608, it is checked whether the number of detections of the object in the current frame has increased in the surrounding area of the object. If both results are true and the object has a similarity to any of the objects stored in the occluded state list that is higher than a predetermined threshold (S608), it is estimated that the object has escaped from the occluded state and reappeared. At that time, the associated object is removed from the occluded state list (S609). For the object removed from the occluded state list, the feature amount and position detected from the current input image are obtained.

Here, some ideas for matching processing could be, for example, reducing the distance-based similarity penalty when matching with an occluded person when matching objects between frames; lengthening the timeout period for waiting for a person to reappear; setting a lower threshold for matching with occluded objects than for matching between normal objects; etc.

Furthermore, although the explanation here assumes two people overlapping, overlaps can also occur between three or more people. In this case, if it is determined that a person has entered an occluded state, they are added to a list of occluded states, and when they reappear, they are associated with the previous frame and removed from the list of occluded states each time. This makes it possible to deal with cases of three or more people to a certain extent.

<Embodiment 3>
In this embodiment, a form of tracking a single object designated by a user will be described. Here, the tracking target is not limited to a specific category such as a human body, but a form of tracking an unspecified object designated by a user will be dealt with. For example, it may be an animal such as a dog, or a vehicle such as a car.

The functional block diagram is shown in FIG. 9(B). A new tracking object designation unit 302 has been added to the previous configuration. The functions of the tracking object designation unit 302 and the object detection unit 202 can be easily realized by using the method of Non-Patent Document 1. The tracking object designation unit 302 is a functional unit in which the user designates the frame position of the tracking target object in the video frame. This initializes the characteristics of the object to be tracked. The object detection unit 202 identifies the image area in each video that is most similar to the target object. An example of the identification result is shown in FIG. 12(A). The frame 5111 on the video frame 5110 in FIG. 12(A) is the frame of the tracking object designated by the user. In the video frame 5120, this object moves to the right side in the screen and is detected as a frame 5121 by the object detection unit 202. The method of Non-Patent Document 1 is an excellent object tracking method, but erroneous switching easily occurs between similar objects. Therefore, in this embodiment, as in the previous embodiments, information regarding the occlusion state of the tracking object is estimated, and it is determined whether or not an erroneous switch has occurred.

For this purpose, a neural network 6300 as shown in FIG. 13B is used as the occlusion information generating unit 203. This is a classifier that, when an image 6301 of a detected tracking object (here, the aspect ratio is normalized to a square image for ease of processing) is input, looks at the image pattern and outputs a classification result 6302 of whether it is occluded (Yes) or not (No). The presence or absence of occlusion is defined as the percentage of the object's area that is occluded or not. The values of these two classes are given as teacher values to train the neural network 6300. This technology is a widely known method similar to normal image classification tasks, so details are omitted. In addition, if the teacher value (target variable) is given as the percentage of the occluded area rather than the binary value of occlusion or not and regression learning is performed, the occlusion percentage can be estimated as in the estimation result 6303. In this regression learning, square error or the like is used as a loss value given during learning.

Figures 14(A) and (B) show the results of estimating the occlusion degree of a candidate object to be tracked by the occlusion information generation unit 203. In comparison with the object detection result shown in Figure 14(A), the estimated occluded area by the object detection unit 202 is shown in Figure 14(B) with the symbol occ. In this figure, the fluctuation range of the occlusion score is smaller than a predetermined value (e.g., a value such as 0.3), and it can be determined that tracking has not failed (here, the success or failure of tracking may be determined using not only the occlusion score but also the similarity of the feature amounts of position and appearance as used in embodiment 1).

On the other hand, in FIG. 14(C), object 7201 passes behind object 7202 from video frame 7220 to 7230, and as a result, the object detection unit 202 erroneously estimates the position of the object in video frame 7230 as frame 7231. In this case, the occlusion score fluctuates significantly from 0.4 to 0.0 as shown in FIG. 14(D), so it can be determined that erroneous tracking has occurred due to the intersection. If it is determined that erroneous tracking has occurred, it is possible to take measures such as stopping detection at that point or correcting it at a later stage.

This concludes the explanation of this embodiment.

It is desirable that the occlusion information generating unit 203 be trained to be able to estimate the occlusion state of various objects so that the occlusion state of unspecified objects can be determined, as shown in 5110 to 5150 in FIG. 12(A).

As another derivative form, in FIG. 13(B), an image 6301 cut by an object frame is shown as the input image. However, since it is important to observe not only the object itself but also its surroundings in order to determine whether it is in an occluded state, it is also possible to input a wider range as the input image (in that case, a similar range is also cut out and input during estimation).

As another derived form, as shown in FIG. 13(C), a form in which object detection and occlusion degree estimation are performed simultaneously using candidate frames called anchors, as in the Liu method described above, is also considered. The anchor frames are a set of candidate frames of multiple sizes and aspect ratios as shown in FIG. 13(D) (three types of anchor frames are illustrated here). The anchor frames are placed in each block region in the image, as shown in the result image 6450 in FIG. 13(C). When an image is input, the neural network 6400 generates an occlusion score map 6430 indicating whether or not the object is present in each anchor in each block region. The occlusion score map 6430 is a set of three maps corresponding to the three types of anchor frames. An example of the estimation result is shown in FIG. 13(C) 6450 (the above method is widely known, so please refer to the Liu method described above for details).

As a derived form of this embodiment, an object occlusion score map 6440 is generated at the same time as estimating whether an object exists. This is a map that estimates the occlusion ratio for each anchor frame if an object is present there. This map also consists of three maps corresponding to the number of anchor types (during learning, the neural network 6400 can be trained by providing a teacher value of the occlusion score for each anchor frame in each block of the image). An example result is shown in FIG. 13(C) 6460. An example of the final integration of the two estimated maps is shown as an example integration result 6470.

The above explanation is an example of object detection, but since the method in Non-Patent Document 1 is also an anchor candidate frame-based method, it is possible to construct a derivative form that tracks an object while simultaneously estimating its occlusion score.

<Embodiment 4>
In this embodiment, a form in which a single object designated by a user is tracked will be described. The diagram of the functional blocks is the same as that of the third embodiment, and is shown in FIG. 9B. In the previous embodiments, when comparing similarities, object correspondence between frames was performed on a rule basis, such as by comparing features between the immediately preceding and following frames, or by comparing using n frames before and after. In this embodiment, this part is replaced with machine learning to perform more accurate correspondence.

A recurrent neural network is a technology that can process time-series data to perform identification, classification, etc., and a representative method is the long short term memory network (hereinafter referred to as LSTM), which is known from the method of Byeon et al. (Byeon et al., Scene labeling with LSTM recurrent neural networks, CVPR 2015). This method can determine changes in the characteristics of objects over time and associate them with each other. A schematic diagram of the configuration of this embodiment and an example of the results is shown in Figure 15. Here, one object 9102 is specified as the tracking target and is tracked using the method of Bertinetto et al. (a false switch occurs in the video frame at t=2). The LSTM units 9501-9504 in FIG. 15(C) receive features 9401-9404 of the object being tracked at each time, determine whether tracking is successful or unsuccessful, and output the results as outputs 9701-9704. Although multiple LSTM units are shown in the figure, this does not indicate the existence of multiple units, but rather the state of the same unit at each time. The LSTM unit at each time sends a recurrent input 9802 to the LSTM unit at the next time. The LSTM unit changes its internal state as necessary based on the recurrent input 9802, which includes the features of the object at that time and past information up to that point. This makes it possible to determine whether tracking at the current time is successful or not, taking into account how the object's pattern is changing over time.

The features input to the LSTM unit can be the features of the neural network described in embodiment 3. For example, the input value to the final layer 6320 of the neural network 6310 that determines the occlusion score of an object in FIG. 13(B) is used. Here, the input value (multidimensional vector) is used instead of the output value of the previous layer (a single-valued scalar). This is because by using the same multidimensional features as those used to determine the occlusion state, various types of information regarding occlusion can be incorporated into the LSTM. This is expected to enable various occlusion patterns to be determined.

When the LSTM unit is trained, whether tracking at each moment is successful or unsuccessful is given as a teaching value, and each weight parameter of the LSTM is adjusted. As another form, as shown in FIG. 16(D), if the teaching value is given as to whether or not the object is occluded, rather than the success or failure of tracking, it is also possible to train the unit to determine whether or not it is occluded.

As another embodiment, similar to the derived embodiment described in the third embodiment, a tracked object may be detected based on an anchor frame, and feature quantity 6420 of neural network 6400 in FIG. 13(C) that simultaneously determines the object's position and occlusion score may be used as 9401. With this embodiment, it is possible to simultaneously and quickly determine the tracking and detection of an object and the occlusion score of the object.

<Embodiment 5>
In this embodiment, a form in which a single object designated by a user is tracked will be described. The basic functional configuration is the same as in embodiment 1. In this embodiment, relative perspective information between objects is used as occlusion information of the object.

An example is shown in FIG. 16 (A1). Here, an RGB image 801 is prepared as a learning image. Furthermore, a distance image 833 corresponding to the RGB image 801 is obtained by a laser rangefinder device, stereo measurement, or the like. The distance image 833 is a grayscale representation of the absolute value of the distance from the camera, and the whiter the color, the closer the object is. In this embodiment, the RGB image 801 is used as an input image, and the distance image is used as a teaching value 831 to learn the weights of the neural network 402. However, obtaining an output result 830 that is exactly the same as the distance image 831 as an absolute value is a relatively difficult problem in terms of pattern recognition, and the occlusion information used in this embodiment does not need to be so highly accurate. Therefore, in this embodiment, learning is performed to estimate the relative perspective relationship between nearby objects.

For example, as shown in output result 830 in the figure, the distance estimation results 8301 and 8302 for persons 8011 and 8012 are incorrect as absolute values. Estimation results 8301 and 8303 corresponding to person 8013, who is distant from person 8011, also have incorrect perspective relationships. However, if we limit the perspective order relationship to two nearby persons 8011 and 8012, the results are correct. In this way, the system learns to be able to correctly estimate the <perspective order relationship> between <local objects>, and these are compiled and used as object occlusion information.

The above is achieved by applying the following innovation to the calculation of loss values during learning. Figure 16 (A2) shows a teacher value area 831a, which is an enlarged area near the symbol * on the teacher value 831 in Figure 16 (A1). The corresponding output result area 830a is also shown. Here, attention is paid to each pixel i and pixel j on the area 831a, and the loss of the pixel pair is calculated based on whether the perspective relationship is correct or not. Here, the perspective relationship between pixel i and pixel j on the area 830a matches the teacher value, so no loss occurs. On the other hand, if the estimated result is as in area 830b, the perspective relationship is incorrect, so a loss is recorded. This judgment is made for all pixel pairs within a specified distance. Finally, the total loss value is calculated by dividing the number of pairs with incorrect perspective relationships by the total number of pairs. The neural network 802 that has been trained in this way finishes learning, and the output result 834 of the estimated distance is shown in Figure 16 (B).

Next, the relative distance output results 834 are tallied to determine the occluded likelihood of the object. Here, the results are tallied for each detection frame using separately detected person detection frames 8351 and 8352. The distance values within each frame are averaged to d ^ave ₁ and d ^ave _2. Next, the distance values are compared between adjacent object frames, normalized, and converted into an occluded likelihood score value occ. For example, the conversion is performed using the following formula.
(Formula 3)
occ _i =Sigmoid(Log(d ^ave _i /d ^ave _j ))
=1/(1+d ^ave _j /d ^ave _i ),
occ _j =1/(1+d ^ave _i /d ^ave _j ),
where Sigmaid(x) = 1/(1 + exp(-x)).
Here, i and j are two adjacent detection object frames with overlapping parts. When three or more objects overlap, the occlusion score _occi may be calculated by the above formula for each object, and the maximum value among them may be used as the occlusion score of the object.

The above is the process of performing relative distance estimation and tallying up the occlusion score. The tracking process using the occlusion score is the same as in embodiment 1, so it will not be described here.

The following are some of the derivative learning ideas that can be considered. (1) Losses are tallied only for pairs where the difference in distance teacher values is equal to or greater than a certain threshold Θ. This allows for robust learning against noise when observing distance images. (2) (1) is performed, and a margin area is set. For example, losses are generated not only when the perspective relationship of the pair is correct or not, but also when the perspective relationship is correct and the values are not relatively far apart by more than a certain threshold Θ. (3) Losses are also generated as errors when the output value for a pixel pair where the difference in distance teacher values is less than the threshold Θ is greater than the threshold Θ. This suppresses noisy output.

As mentioned above, there are various possible forms, but any form that can learn distances relatively and locally is acceptable, and is not limited to one form. The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a data communication network or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. The program may also be provided by recording it on a computer-readable recording medium.

REFERENCE SIGNS LIST 1 Information processing device 201 Image acquisition unit 202 Object detection unit 203 Occlusion information generation unit 204 Feature amount acquisition unit 205 Corresponding unit 206 Storage unit

Claims (16)

An information processing device for detecting an object from an image,
A feature extraction means for extracting image features of an object detected from the image;
an estimation means for estimating, for each object detected from the image, occlusion information indicating an occlusion relationship between the object detected from the image and another object detected from the image, based on an output of a trained model that outputs a likelihood indicating that the object detected from the image is occluded by another object;
and an identification means for identifying, for each object detected from the image, a correspondence relationship with an object detected in an image captured at a different time from the image based on at least the image features and the occlusion information. The information processing device according to claim 1, characterized in that the estimation means estimates the occlusion information indicating a partial area of an object that is occluded for each object detected from the image based on the output of the trained model. The information processing device according to claim 1 or 2, characterized in that the identification means identifies a correspondence between the image and an object detected in an image captured at a different time from the image, based on the image features of the object and the occlusion information estimated by the estimation means. The information processing device according to any one of claims 1 to 3, further comprising a storage means for storing the likelihood in association with the image features of the object. The information processing device according to any one of claims 1 to 4, characterized in that the occlusion information is information that indicates, for each region of the image, a larger likelihood in the region of the object that is occluded and a smaller likelihood in the other regions. The information processing device according to any one of claims 1 to 5, characterized in that the identification means identifies, for each object detected from the image, a correspondence relationship between the object detected from an image captured at a different time from the image, based on the position in the image, the image features detected from the image, and the occlusion relationship in the image. The method further includes acquiring means for acquiring a region for each of the objects from the image,
The information processing apparatus according to claim 1 , wherein the estimation means estimates the occlusion information indicating the presence or absence of an occluded object for each of the object regions acquired by the acquisition means. a determination means for determining whether or not an object is occluded based on a correspondence between each object detected from the image identified by the identification means and an object detected in an image captured at a different time from the image; and
8. The information processing apparatus according to claim 1, further comprising: a storage unit for storing the object that is obstructed. the determining means determines, as a first object, an object that does not correspond to any object detected in an image captured prior to the image;
9. The information processing apparatus according to claim 8, wherein said storage means stores information indicating that said first object was occluded at the time when said image was captured. the determining means determines, among the objects detected from the image, an object that does not correspond to an object detected in an image captured prior to the image as a second object;
The information processing device according to claim 9, characterized in that the storage means stores information indicating that the first object was not occluded at the time the image was captured when a similarity between the second object detected from the image and the first object determined by the storage means to be occluded in an image captured prior to the image is greater than a predetermined threshold value. If two objects are detected in a first image and one object is detected in a second image captured after the first image,
The estimation means estimates the occlusion information indicating that an object detected in the second image occludes another object,
The information processing device according to any one of claims 1 to 10, characterized in that the identification means is configured to identify, based on the occlusion information estimated by the estimation means, an object detected in the first image that is occluding other objects, as being the same object as the object detected in the second image. When two objects are detected from a third image captured after the second image,
the estimation means estimates the occlusion information indicating that an object detected from the third image, which is associated with an image feature of the object detected from the second image, is occluding another object; and
The information processing device according to claim 11, characterized in that the identification means identifies, for an object detected from the first image that is occluded by another object, an object detected from the third image that is different from an object associated with an image feature of an object detected from the second image, as being the same object as the object occluded in the second image. The information processing device according to any one of claims 1 to 12, characterized in that the trained model is a neural network. The information processing device according to any one of claims 1 to 13, characterized in that the trained model is a model that has been trained on image features that indicate an occlusion relationship between an occluding object and an occluded object. A program for causing a computer to function as each of the means possessed by an information processing device according to any one of claims 1 to 14. An information processing method performed by an information processing device that detects an object from an image, comprising:
a feature extraction step of extracting image features of the object detected from the image;
an estimation step of estimating, for each object detected from the image, occlusion information indicating an occlusion relationship between the object detected from the image and another object detected from the image, based on an output of a trained model that outputs a likelihood indicating that the object detected from the image is occluded by another object;
and an identification step of identifying, for each object detected from the image, a correspondence relationship between the object detected in the image and an object detected in an image captured at a different time from the image based on at least the image features and the occlusion information.

Images