JP2024173641A - Image processing device and image processing method - Google Patents

Abstract

To provide a technique that, even when a subject to be tracked is no longer detected from a picked-up image, enables continuation of tracking of the subject to be tracked.SOLUTION: An image processing apparatus detects a subject from a frame, and in a case where a subject having identification information same as identification information of a subject to be tracked is not detected from a first frame, when identification information of a subject detected from the first frame includes new identification information different from identification information of a subject detected from a second frame before the timing at which the subject to be tracked is not detected, updates the identification information of the subject to be tracked to the new identification information.SELECTED DRAWING: Figure 2

Description

The present invention relates to subject tracking technology.

Generally, there is an automatic tracking technology in which a camera capable of adjusting pan, tilt, and zoom, known as a PTZ camera, automatically tracks a tracking target detected from a captured image in response to a user request. In the automatic tracking technology, the pan, tilt, and zoom are automatically controlled so that the tracking target is located at a desired position within the camera's angle of view. Patent Document 1 discloses a method for calculating the control parameters of a PTZ camera required to display a tracking target at the center of the screen based on the coordinates of the tracking target displayed on the screen.

Japanese Patent Application Publication No. 9-181961

Automatic tracking technology has an issue known as "losing," in which the target to be tracked becomes confused with another object and is lost. When this occurs, automatic tracking cannot continue normally. This invention provides technology that makes it possible to continue tracking the target subject even if the target subject is no longer detected in the captured image.

One aspect of the present invention is characterized by comprising a detection means for detecting a subject from a frame, and an update means for updating the identification information of the subject to be tracked to new identification information if a subject with identification information identical to the identification information of the subject to be tracked is not detected in a first frame and the identification information of the subject detected in the first frame includes new identification information different from the identification information of the subject detected in a second frame prior to the timing at which the subject to be tracked was not detected.

According to the present invention, it is possible to continue tracking a subject to be tracked even if the subject to be tracked is no longer detected in a captured image.

FIG. 1 is a diagram showing an example of a system configuration. FIG. 2 is a block diagram showing an example of the hardware configuration of each of the camera 100 and the controller 200. 4A is a flowchart showing the operation of the camera 100, and FIG. 4B is a flowchart showing the operation of the controller 200. FIG. 11 is a diagram showing an example of an inference result. 11A and 11B are diagrams showing examples of displaying captured images and inference results. 4A is a flowchart showing the operation of the camera 100, and FIG. 4B is a flowchart showing the operation of the controller 200.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

[First embodiment]
First, a configuration example of a system that tracks and captures a subject designated by a user as a tracking target will be described with reference to Fig. 1. As shown in Fig. 1, the system according to this embodiment has a camera 100 having a function of tracking and capturing an image of a subject, and a controller 200 having a function of controlling the camera 100 and processing information from the camera 100. The camera 100 and the controller 200 are connected to a LAN (Local Area Network) 400, and are configured to be able to communicate data with each other via the LAN 400. The LAN 400 may be wired or wireless. The network between the camera 100 and the controller 200 is not limited to a specific network.

First, we will explain the camera 100. The camera 100 is an image processing device that has an imaging unit 111 and a drive unit 109 that controls the imaging direction (pan direction, tilt direction) and zoom of the imaging unit 111.

When capturing a moving image, the imaging unit 111 outputs an image of each frame as a captured image, and when capturing still images periodically or irregularly, the imaging unit 111 outputs the still images (frames) as captured images. The imaging unit 111 transmits the captured images and various information to the controller 200 via the LAN 400. The driving unit 109 controls the imaging direction (pan direction and tilt direction) and zoom of the imaging unit 111 so as to track and capture an image of a subject that is set as a subject to be tracked.

Next, the controller 200 will be described. The controller 200 processes various information transmitted from the camera 100 and notifies the camera 100 of the subject to be tracked that has been selected in response to a user operation.

Next, an example of the hardware configuration of the camera 100 and the controller 200 will be described using the block diagram in FIG. 2. The configuration shown in FIG. 2 is merely one example of a configuration capable of executing the various processes described below, and can be modified/altered as appropriate.

First, an example of the hardware configuration of the camera 100 will be described. The CPU 101 executes various processes using computer programs and data stored in the RAM 102. In this way, the CPU 101 controls the operation of the entire camera 100, and also executes or controls various processes that will be described as processes performed by the camera 100.

RAM 102 is an example of a high-speed storage device such as a DRAM. RAM 102 has an area for storing computer programs and data loaded from storage device 103, and an area for storing captured images output from image processing unit 106. RAM 102 also has a work area used by CPU 101, inference unit 110, and image processing unit 106 when executing various processes. In this way, RAM 102 can provide various areas as appropriate.

The storage device 103 includes volatile and non-volatile storage devices such as ROM, flash memory, and HDD. The storage device 103 stores an operating system (OS), computer programs and data for causing the CPU 101, the inference unit 110, the image processing unit 106, and the like to execute or control various processes described as processes performed by the camera 100.

The inference unit 110 detects a subject from the captured image stored in the RAM 102 and infers the position and presence of the subject. The inference unit 110 is a computing device specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). Although a GPU is generally effective for use in learning processing, a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array) may achieve the same function. The processing of the inference unit 110 may also be performed by the CPU 101. Note that although the inference unit 110 is used in this embodiment, it is not used in the second embodiment described later, and therefore in the second embodiment, the camera 100 does not need to have the inference unit 110.

The drive unit 109 is connected to the drive I/F 108. The drive unit 109 has a rotation mechanism for changing the imaging direction and zoom of the camera 100, and has a mechanical drive system and a motor as a drive source. The drive unit 109 performs rotation such as pan and tilt operations and zoom operations for optically changing the angle of view based on instructions received from the CPU 101 via the drive I/F 108.

The image sensor 107 has an optical system and an image sensor such as a CCD or CMOS. The light entering through the optical system is received by the image sensor, which outputs an image corresponding to the received light.

The image processing unit 106 converts the image output from the image sensor 107 into an image in a specified image format, and outputs the converted image as a captured image. Note that the image processing unit 106 may use the image output from the image sensor 107 to perform image quality adjustments such as color correction, exposure control, and sharpness correction, and cropping to cut out a specified area from the image.

The camera 100 is connected to the LAN 400 via the network I/F 105, and the camera 100 communicates with external devices such as the controller 200 via a communication medium such as Ethernet (registered trademark). Note that communication may also be performed via another I/F, such as a serial communication I/F (not shown).

The CPU 101, RAM 102, storage device 103, inference unit 110, drive I/F 108, image processing unit 106, and network I/F 105 are all connected to a system bus 111.

Next, the controller 200 will be described. The controller 200 is a computer device such as a PC (personal computer), a tablet terminal device, or a smartphone.

The CPU 201 executes various processes using computer programs and data stored in the RAM 202. As a result, the CPU 201 controls the operation of the entire controller 200, and executes or controls various processes that will be described as processes performed by the controller 200.

The RAM 202 has an area for storing computer programs and data loaded from the storage device 203, and a work area used by the CPU 201 and the inference unit 210 when executing various processes. The RAM 202 also has an area for storing various information received from the camera 100 via the network I/F 204. In this way, the RAM 202 can provide various areas as needed.

The storage device 203 includes volatile and non-volatile storage devices such as ROM, flash memory, and HDD. The storage device 203 stores an operating system (OS), computer programs and data for causing the CPU 201 and the inference unit 210 to execute or control various processes described as processes performed by the controller 200, and the like.

The inference unit 210 has the same functions as the inference unit 110 described above. Note that the inference unit 210 is not used in this embodiment, but is used in the second embodiment described below, so in this embodiment, the controller 200 does not need to have the inference unit 210.

The user input I/F 206 is a user interface such as a keyboard, mouse, touch panel screen, joystick, buttons, and dials, and can be operated by the user to input various information and instructions to the controller 200.

The display unit 205 has a liquid crystal screen or a touch panel screen, and can display the results of processing by the CPU 201 and the inference unit 210 as images, text, and the like. The display unit 205 is not limited to being a device that the controller 200 has, and may be an external device that is connected to the controller 200 by wire or wirelessly. The display unit 205 may also be a projection device such as a projector that projects images and text.

The controller 200 is connected to the LAN 400 via the network I/F 204, and the controller 200 is responsible for communication with external devices such as the camera 100 via a communication medium such as Ethernet (registered trademark). Note that communication may also be performed via another I/F, such as a serial communication I/F (not shown).

The CPU 201, RAM 202, storage device 203, inference unit 210, user input I/F 206, display unit 205, and network I/F 204 are all connected to a system bus 207.

Next, the operation of the camera 100 will be described with reference to the flowchart in FIG. 3(a). The process according to the flowchart in FIG. 3(a) is for one frame of captured image, and in reality, the camera 100 performs the process according to the flowchart in FIG. 3(a) for each frame of captured image.

In step S101, the image processing unit 106 generates a captured image as described above in accordance with instructions from the CPU 101, and stores the generated captured image in the RAM 102. The CPU 101 also inputs the captured image to the inference unit 110, which generates an inference result for the captured image in accordance with instructions from the CPU 101, and stores the generated inference result in the RAM 102.

The inference unit 110 reads out a trained model created using a machine learning method such as deep learning from the storage device 103. The inference unit 110 then inputs the captured image into the trained model and performs calculations (inference) on the trained model to obtain the position of the subject in the captured image as an inference result for the subject. If the captured image contains multiple subjects, the inference unit 110 obtains an inference result for each subject. Note that, in order to improve the processing speed of the inference, the image processing unit 106 may reduce the captured image and store it in the RAM 102, and the reduced captured image may be input to the trained model.

Then, the CPU 101 assigns an ID, which is identification information unique to the subject, to the subject inference result. If the captured image contains multiple subjects, the CPU 101 assigns an ID unique to each subject to the inference result for that subject.

Now, the inference results will be described with reference to Figs. 4(a) to (e). In Fig. 4, the subject is a person. Fig. 4(a) is a diagram showing the inference results when two people (person 701 and person 702) are present in the captured image 700. When the captured image 700 in Fig. 4(a) is input to the inference unit 110, the inference unit 110 outputs "the image coordinates of the upper left vertex and the image coordinates of the lower right vertex of a rectangle 710 that includes the entire body of the person 701" as the position of the person 701 (the inference result of the person 701), and outputs "the image coordinates of the upper left vertex and the image coordinates of the lower right vertex of a rectangle 711 that includes the entire body of the person 702" as the position of the person 702 (the inference result of the person 702).

The CPU 101 also assigns "1" to the inference result for person 701 as ID 720, which is identification information unique to that person 701, and assigns "2" to the inference result for person 702 as ID 721, which is identification information unique to that person 702.

The position of a person output by the inference unit 110 is not limited to the image coordinates of the upper left vertex and the lower right vertex of a rectangle including the entire body of the person. For example, as shown in FIG. 4B, the image coordinates of the upper left vertex and the lower right vertex of a rectangle 730 including the head of the person 701 may be output as the position of the person 701, and the image coordinates of the upper left vertex and the lower right vertex of a rectangle 731 including the head of the person 702 may be output as the position of the person. Furthermore, the image coordinates of the upper left vertex and the lower right vertex of a rectangle including other parts of the person may be output as the position of the person. Furthermore, instead of the image coordinates of the upper left vertex and the lower right vertex of the rectangle, the image coordinates of the center of the rectangle and the width and height of the rectangle may be output.

In addition, to detect different objects, such as heads and faces, rather than people, you can change the trained model to one that has been trained based on the training data that corresponds to the desired output.

In addition, although the case where the subject is detected using a trained model created using a machine learning method has been described, the method of detecting the subject is not limited to a specific method. For example, a method such as the SIFT method, which detects the subject by matching local feature points in a captured image, or a template matching method, which detects the subject by calculating the similarity with a template image, may be used.

Furthermore, the CPU 101 assigns IDs to the inference results for each frame to the current inference result. At this time, the CPU 101 compares the inference results of past frames with the inference results of the current frame to assign IDs so that the same ID is assigned to the same subject even if the subject has moved. Specifically, a method may be used in which the position and size of the detected rectangle is compared between past and current frames, and the same ID is assigned to the closest subject.

Here, the assignment of IDs to inference results will be described with reference to Figures 4(c) to (e). Figures 4(c) to (e) show information necessary for the following description, namely, a rectangle 710 including the entire body of person 701 detected in the previous frame, a center 740 of rectangle 710, a rectangle 710a including the entire body of person 701 detected in the current frame, a center 740a of rectangle 710a, and a circle 750 of radius L1 centered on center 740; other information is omitted.

L1 is stored in advance as a parameter in storage device 103. If, in the current frame, the center of rectangle B of subject B is within distance L1 from the center position of rectangle A of subject A in the previous frame, and the ratio (area ratio) of the area (number of pixels) of rectangle A to the area (number of pixels) of rectangle B is within specified range X (for example, 0.9 to 1.1), CPU 101 determines that subject B is the same subject as subject A, and assigns to the inference result of subject B the same ID as the ID assigned to the inference result of subject A. The specified range X is stored in advance as a parameter in storage device 103.

In the example of FIG. 4(c), the center 740a of rectangle 710a is within distance L1 (i.e., within circle 750) from the center 740 of rectangle 710 in the previous frame, and the ratio (area ratio) of the area (number of pixels) of rectangle 710 to the area (number of pixels) of rectangle 710a is within specified range X. Therefore, CPU 101 assigns the same ID to the inference result of the person corresponding to rectangle 710, to the inference result of the person corresponding to rectangle 710a.

In the example of FIG. 4(d), the center 740a of the rectangle 710a is not within the distance L1 (i.e., within the circle 750) from the center 740 of the rectangle 710 in the previous frame. Therefore, the CPU 101 assigns a new ID to the inference result of the person corresponding to the rectangle 710a, which is different from the ID assigned to the inference result of the person corresponding to the rectangle 710.

In the example of FIG. 4(e), the center 740a of rectangle 710a is within distance L1 (i.e., within circle 750) from the center 740 of rectangle 710 in the previous frame, but the ratio (area ratio) of the area (number of pixels) of rectangle 710 to the area (number of pixels) of rectangle 710a is not within the specified range X. Therefore, CPU 101 assigns a new ID to the inference result of the person corresponding to rectangle 710a that is different from the ID assigned to the inference result of the person corresponding to rectangle 710. In addition, the same ID may be assigned to objects whose aspect ratio changes between the previous frame and the current frame within threshold Y.

Alternatively, a method may be used in which the position of a rectangle in the current frame is predicted using a Kalman filter or the like based on the change in the position of a rectangle corresponding to the same ID in the past few frames, and an ID equal to the predicted rectangle is assigned to the inference result of the person corresponding to the rectangle closest to the predicted rectangle position.

Alternatively, a method may be used in which feature amount A of an image within a rectangle is obtained for an object detected in the current frame, feature amount A is compared with feature amount B of an image within each rectangle corresponding to each object detected in a previous frame, and the inference result for person A is assigned the same ID as the ID assigned to the inference result for the person in the rectangle with feature amount B that is closest to feature amount A.

The following describes the case where the feature vector of an image is used as a feature. First, a machine learning model is prepared that is trained to output similar feature vectors for images of a specific person viewed from different angles and output different feature vectors for images of different people. Specifically, images of various subjects taken from various angles are prepared in advance as learning images, and images of the same subject are labeled with the same ID and used as input for the machine learning model. The output is a feature vector output from the image. A feature vector space in which the feature vectors output from each learning image are mapped is learned by Deep Metric Learning. In deep metric learning, the distance between two feature amounts reflects the similarity of the data, and learning is performed so that the distance between feature amounts output from input images with similar meanings is close, and the distance between feature vectors output from input images with distant meanings is far. Loss functions used in deep metric learning to calculate distance include triplet loss, contrastive loss, and class classification error. If images of the same subject taken from different angles are input to a machine learning model trained in this way, the feature vectors output as inference results will be similar. Any loss function may be used as long as it is defined so that the loss decreases according to the similarity in meaning of the input images. Note that although an example using deep metric learning has been described, this is not limited to this, and any learning method that can separate feature vector data groups into feature vector groups of images of the same subject and feature vector groups of images of different subjects can be used.

Also, the feature amount used for ID assignment is not limited to this, and a method of obtaining color information of the image within the rectangle of the current frame, comparing this with color information within each rectangle of past frames, and assigning the same ID to the person who is closest to the person may be used. Specifically, color information can be obtained by calculating the average of pixel values at a specific percentage of the position within the rectangle, or the average of pixel values in a specific region. Alternatively, the average of the entire rectangle may be taken, or, for example, if the upper body is wearing the same uniform, the average of pixel values in the lower region of the rectangle may be taken, so long as it is possible to obtain the color of the region where the difference in the subject's skin or clothing is clearly visible. Also, a histogram of the entire rectangle or a specific region may be obtained as color information, and compared with the histograms within each rectangle of past frames using methods such as Pearson's correlation function or chi-square test, and the same ID may be assigned to the person with the highest similarity.

A threshold value may also be set for the difference in features for assigning the same ID (the difference between feature A and feature B in the above example). In this case, if the difference between feature A and feature B closest to feature A is equal to or less than the threshold value, the CPU 101 assigns to the inference result of person A the same ID as the ID assigned to the inference result of the rectangular person with feature B.

Functions may also be acquired using a machine learning model that is trained to output similar features for images of a specific person seen from different angles, and output different features for images of different people. IDs may also be assigned by combining these methods.

In step S102, the CPU 101 reads out the captured image and the inference result (including the ID of the inference result) stored in the RAM 102 as transmission information, and transmits the read transmission information to the controller 200 via the network I/F 105.

In step S103, the CPU 101 determines whether or not the "ID of the subject to be tracked (tracked subject ID)" has been received from the controller 200 via the network I/F 105. If the result of this determination is that the tracked subject ID has been received from the controller 200, the process proceeds to step S105, and if the tracked subject ID has not been received from the controller 200, the process proceeds to step S104.

In step S104, the CPU 101 determines whether or not a determined tracking subject ID is stored in the RAM 102 (whether or not the tracking subject ID has been determined). If the result of this determination is that a determined tracking subject ID is stored in the RAM 102 (the tracking subject ID has been determined), the process proceeds to step S106. On the other hand, if a determined tracking subject ID is not stored in the RAM 102 (the tracking subject ID has not been determined), the process proceeds to step S101.

For example, if the tracking subject ID stored in RAM 102 is set to an initial value (e.g., NULL), CPU 101 may determine that a determined tracking subject ID is not stored in RAM 102 (the tracking subject ID has not been determined). In this case, if the tracking subject ID stored in RAM 102 is set to a value other than the initial value, CPU 101 determines that a determined tracking subject ID is stored in RAM 102 (the tracking subject ID has been determined).

In step S105, the CPU 101 stores the tracking subject ID received from the controller 200 via the network I/F 105 in the RAM 102. In step S106, the CPU 101 determines whether or not there is an inference result to which the same ID as the tracking subject ID is assigned among the inference results (inference results of the subject detected from the captured image of the current frame) stored in the RAM 102 in step S101. In other words, the CPU 101 determines whether or not a subject corresponding to the same ID as the tracking subject ID has been detected from the captured image of the current frame.

If this determination shows that the inference results for the subject detected from the captured image of the current frame include an inference result that has been assigned the same ID as the tracked subject ID (a subject corresponding to the same ID as the tracked subject ID has been detected), processing proceeds to step S109.

On the other hand, if there is no inference result for the subject detected from the captured image of the current frame that has been assigned the same ID as the tracking subject ID (a subject corresponding to the same ID as the tracking subject ID has not been detected), the CPU 101 determines that the tracking target subject has been lost, stores the current time T in the RAM 102, and the process proceeds to step S107. The current time is obtained by a timer function of the CPU 101. Note that the CPU 101 may store information that specifies the timing when the tracking target subject was lost, such as the frame number of the current frame or a timestamp, in addition to or instead of the current time.

An example of the state of the subject in the captured image at this time will be described using Figures 4(a) and (f). Here, we will explain the case where the tracking subject ID = 1 (i.e., the subject to be tracked is person 701).

After the state of FIG. 4(a), for example, if person 701 and person 702 move and person 702 passes in front of person 701 (on the camera 100 side) and crosses over, person 701 is blocked by person 702 in the captured image 700 and becomes invisible (a state in which person 701 has been lost) as shown in FIG. 4(f). As a result, there is no tracking subject ID among the IDs obtained from the captured image 700 of FIG. 4(f). In this case, the process proceeds from step S106 to step S107.

In step S107, the CPU 101 acquires the ID of the inference result of the subject detected from the current frame and the ID of the inference result of the subject detected from a frame prior to time T (prior to the above timing) from the "inference results of the subject detected from the frame for each frame" stored in the RAM 102. The CPU 101 then determines whether the IDs of the inference result of the subject detected from the current frame include a new ID that is different from any of the IDs of the inference result of the subject detected from a frame prior to time T.

If the result of this determination is that the IDs of the inference results of the subject detected from the current frame include a new ID that is different from any of the IDs of the inference results of the subject detected from frames prior to time T, the process proceeds to step S108. On the other hand, if the IDs of the inference results of the subject detected from the current frame do not include a new ID that is different from any of the IDs of the inference results of the subject detected from frames prior to time T, the process proceeds to step S101.

An example of the state of the subject in the captured image at this time will be described using Figures 4(a), (f), and (g). For example, the captured image 700 in Figure 4(f) is the current frame, and the captured image 700 in Figure 4(a) is the frame before time T. At this time, the ID of the inference result obtained for the captured image 700 in Figure 4(f) is only ID=2, and the IDs of the inference results obtained for the captured image 700 in Figure 4(a) are ID=1 and 2, so there is no new ID. In this case, the process proceeds from step S107 to step S101.

On the other hand, when the person 701 and the person 702 move further from the state of FIG. 4(f) and the intersection ends, the person 701 appears again as shown in FIG. 4(g). The inference unit 110, which has received the captured image 700 of FIG. 4(g), outputs the inference results of the person 701 and the person 702. Here, the inference result of the person after the intersection is not assigned the same ID as the ID of the inference result of the person before the intersection, but a new ID may be assigned, which may cause the tracking target subject to be lost. In the example of FIG. 4(g), the inference result of the captured image 700, the inference result of the person after the intersection is not assigned the same ID as the ID (ID=1) of the inference result of the person 701 before the intersection, but is assigned ID=3 as a new ID 722.

The IDs of the inference results obtained for the captured image 700 in FIG. 4(g) are ID=2, 3, and the IDs of the inference results obtained for the captured image 700 in FIG. 4(a) are ID=1, 2, so ID=3 becomes the new ID. In this case, the process proceeds from step S107 to step S108.

Examples of when a new ID is assigned include when the subject's position immediately before it was lost is far from the position of the subject when it reappears, or when the subject's orientation or posture changes after crossing, causing a large change in the size of the subject's rectangle before and after it is lost. Another example of when a new ID is assigned is when the subject's orientation or the way the external light hits it changes after crossing, causing a large change in features such as color information before and after it is lost.

The method of judgment in step S107 is not limited to a specific judgment method. For example, CPU 101 stores in RAM 102 the ID of the inference result of the subject detected in the frame in which the loss occurred. CPU 101 may then determine whether the ID of the inference result of the subject detected in the current frame includes a new ID that is different from any of the IDs stored in RAM 102. For example, CPU 101 may store in RAM 102 the ID of the inference result of the subject detected in captured image 700 in FIG. 4(f) (the frame in which the loss occurred) and determine whether the ID of the inference result of the subject detected in captured image 700 in FIG. 4(g) (the current frame) includes a new ID that is different from any of the IDs stored in RAM 102.

If the result of this determination is that the IDs of the inference results for the subject detected in the current frame include a new ID that is different from any of the IDs stored in RAM 102, processing proceeds to step S108.

On the other hand, if the inference result IDs of the subject detected in the current frame do not include a new ID that is different from any of the IDs stored in RAM 102, processing proceeds to step S101.

In step S108, the CPU 101 updates the tracking subject ID stored in the RAM 102 to a new ID. Since a new ID of 3 exists for the captured image 700 in FIG. 4(g), in this case the CPU 101 updates the tracking subject ID to "3."

If a subject is lost due to cross-over, the new ID that appears after the time the subject is lost is likely to be the ID of the subject being tracked before it was lost. Therefore, by updating the ID of the subject being tracked to a new ID, tracking of the subject present in the captured image can be resumed.

Before the process of step S108, the CPU 101 may determine whether or not to update the tracking subject ID to a new ID based on the position of the subject corresponding to the new ID. An example will be described below with reference to FIG. 4.

As shown in FIG. 4(h), the CPU 101 sets a circle 770 of radius L2 centered on the position of the lost subject (position 760 in FIG. 4(f)). The method of determining the position of the lost subject is not limited to a specific method. For example, the position of the lost subject may be determined by adding the movement vector of the tracked subject in the frame immediately before the subject was lost to the position of the tracked subject in the frame immediately before the subject was lost. The position of the subject that concealed the lost subject may also be determined as the position of the lost subject. L2 is stored in advance as a parameter in the storage device 103. If the position of the subject corresponding to the new ID is within the circle 770, the CPU 101 determines that the tracked subject ID should be updated to the new ID, and performs the process of step S108.

For example, as shown in FIG. 4(h), when person 701 reappears after the intersection and another person 703 enters from outside the angle of view, a new ID is assigned not only to the inference result of person 701 but also to the inference result of person 703. In FIG. 4(h), ID=4 is assigned as a new ID 723 to the inference result of person 703. In this case, CPU 101 calculates the distance D1 between position 761 and position 760 of person 701, and the distance D2 between position 762 and position 760 of person 703. As a result, position 761 of person 701 is inside circle 770 (D1<L2), and position 762 of person 703 is outside circle 770 (D2>L2). Therefore, CPU 101 updates the tracking subject ID to the new ID=3 assigned to the inference result of person 701, rather than the new ID=4 assigned to the inference result of person 703. In this way, the ID of the tracked subject is not updated to the ID of a subject that appears at a distance from the subject's position when it was lost, allowing for proper tracking.

Note that, although it has been stated that the tracking subject ID is updated with the ID of a subject that appears within a range (threshold range) of a circle with a radius of L2 from the position of the subject when it was lost, the threshold range is not limited to this. For example, the shape of the threshold range is not limited to a circle, and it may be a rectangle or an asymmetric shape.

The value of L2 may not be constant, but may be changed dynamically each time a subject becomes lost. For example, it may be determined based on the size of the rectangle in the inference result before the subject to be tracked was lost. The larger the size of the rectangle, the larger the value of L2 may be set accordingly, so that the tracking subject ID can be updated appropriately even when the subject is lost.

In addition, for example, the moving speed of the rectangle of the tracking target subject may be calculated from the moving distance of the rectangle of the tracking target subject before the tracking target subject is lost, and the value of L2 may be determined according to that speed. Specifically, the faster the speed, the larger L2 may be set, so that the tracking target subject ID can be appropriately updated even when the tracking target subject is lost.

In addition, for example, the direction of movement of the rectangle of the inference result before the subject to be tracked was lost may be calculated, and the shape of the threshold range may be changed. Specifically, if the subject to be tracked moved to the right in the captured image before it was lost, a threshold range may be set that is larger on the right side of the position where the subject was lost, and larger on the left side.

In this way, the ID of the tracking subject is updated appropriately while preventing erroneous updates to the ID of the tracking subject. In addition, for example, the value of L2 may be determined according to the time that has elapsed since the tracking subject was lost. The longer the time that has elapsed since the tracking subject was lost, the larger the value of L2 may be set, so that the ID of the tracking subject can be updated appropriately even when the tracking subject is lost due to collision with a large object. In this way, even if the tracking subject is lost, the tracking subject ID can be updated with a new ID, allowing appropriate tracking of the tracking subject to continue.

In addition, if the feature amount of the newly appeared ID in step S108 is close to the feature amount of the tracking subject before it was lost, the tracking subject ID may be updated to the newly appeared ID as a new tracking subject ID.

When feature vectors are used as features, the feature vectors of all subjects are stored in RAM 102 in step S101. The feature vectors are calculated using the method described in step S101. When a tracking subject ID is detected in step S106, the feature vector of the subject to which the tracking subject ID is assigned is stored in RAM 102. For a newly appearing ID in step S108, if the difference between the feature vector of the subject to which the ID is assigned and the feature vector of the tracking subject read from RAM 102 is within a threshold, the tracking subject ID is updated with the newly appearing ID.

The orientation of the face or body may also be used as a feature. For an object with a newly-appearing ID assigned in step S108, if the orientation of the face or body is close to the orientation of the face or body of the tracking subject before the loss, the newly-appearing ID may be used as a new tracking subject ID to update the tracking subject ID. A trained model is prepared that has learned the image of a person's face or body and the orientation of the face or body in the image (dividing the entire direction into four, either forward, rightward, leftward, or backward) as teacher data. In addition to the above, the orientation of the face or body may be divided into eight directions and given in more detail, such as right-diagonal backward. The orientation of the face or body may also be given as 0°-359°, with the front being 0°. The more detailed the orientation, the more detailed the inference result. When an image of a person's face or body is input to this trained model, the orientation of the subject can be obtained as an inference result. Specifically, the orientation of the body and face of all subjects is acquired in advance in step S101, and if a tracking subject ID is detected in step S106, the orientation of the subject to which the tracking subject ID is assigned is stored in RAM 102. For a newly appearing ID in step S108, if the orientation of the subject to which the ID is assigned is the same as the orientation of the tracking subject read from RAM 102, the tracking subject ID is updated with the newly appearing ID. Also, a trained model that outputs the positions of body parts such as the nose, mouth, eyes, ears, and arms from the image of the subject may be prepared, and the orientation of the face and body may be determined from the number and positional relationship of the parts obtained as a result of inference. For example, if only one eye and one ear are recognized in the recognized rectangle of the subject, and the nose is on the left side, it can be determined that the orientation of the subject is left sideways. The method of determination is not limited to this.

By this processing, when a subject crosses the path, it often moves in one direction and passes by the subject, so even if the subject is lost, the tracking subject ID is updated appropriately depending on the direction of the face or body. As described above, when using a trained model that outputs more precise directions and angles for the direction of the face or body, if the direction of the lost tracking subject and the direction of the subject with the newly appeared ID are within a threshold, the tracking subject ID may be updated with the newly appeared ID. For example, when the direction is output in eight divisions in all directions, if the direction of the lost tracking subject was facing diagonally forward to the right, and the direction of the subject with the newly appeared ID is facing right, diagonally forward to the right, or downward, the tracking subject ID may be updated with the newly appeared ID. For example, if the orientation is output as 0°-359° in all directions, and the orientation of the lost tracked subject was 35°, if the orientation of a subject with a newly-appearing ID is between 25° and 45°, i.e., within the range of ±10° of the subject's orientation, the tracked subject ID may be updated with the newly-appearing ID. By allowing a certain degree of flexibility in the conditions for updating the ID in this way, the system becomes robust to even slight changes in the orientation of the tracked subject after it is lost.

In addition, in addition to the orientation of the face and body, attributes such as the posture and body shape of the subject may be used as feature quantities. A trained model that has learned attributes such as the posture and body shape of the subject may be prepared, and if the attributes of the tracking subject before the loss occurrence are similar to the attributes of the object that has been assigned a new ID in step S108, the tracking subject ID may be updated with the newly appeared ID as a new tracking subject ID. Specifically, in the case of posture, a trained model is prepared that trains the captured image and the tilt angle of the torso part as teacher data and outputs the tilt angle of the torso from the captured image. In the case of body shape, a trained model is prepared that trains the captured image and a numerical value that quantifies the degree of fullness as teacher data and outputs the degree of fullness from the captured image. By using such a trained model, it is possible to determine objects with similar postures and body shapes before and after the loss. By performing such processing, the characteristics of the tracking target subject before and after the loss occurrence are taken into account, and the tracking subject ID is appropriately updated even when the loss occurs.

The size and aspect ratio of the detected rectangle may also be used as a feature. If the rectangular area and aspect ratio of the subject recognized in step S101 are close to those of the tracking subject before it was lost, and the rectangular area and aspect ratio of the object newly assigned an ID in step S108, the tracking subject ID may be updated with the newly appeared ID as the new tracking subject ID. By performing such processing, the tracking subject ID is updated appropriately even when it is lost, taking into account the features related to the size of the tracking target subject before and after it was lost.

The color of the subject may also be used as a feature. For the ID newly appearing in step S108, if the body color is close to the body color of the tracking subject before the loss, the tracking subject ID may be updated with the newly appearing ID as a new tracking subject ID. The body color is obtained by calculating the average of pixel values at a specific percentage of positions in the rectangle output by the inference unit 110 or pixel values in a specific area. Specifically, the body colors of all subjects are acquired in advance in step S101, and if a tracking subject ID is detected in step S106, the color of the subject to which the tracking subject ID is assigned is stored in RAM 102. For the ID newly appearing in step S108, if the difference between the color of the subject to which the ID is assigned and the color of the tracking subject read from RAM 102 is within a threshold, the tracking subject ID is updated with the newly appearing ID. The body color may be obtained by taking the average of the entire rectangle output by the inference unit 110, or, for example, if the upper body is wearing the same uniform, the pixel values of the area below the rectangle may be taken as the average, so long as it is possible to obtain the color of the area where the difference in the subject's skin or clothing is prominent. In addition, a histogram of the entire rectangle or a specific area may be obtained as body color information. In this case, the histogram of the subject with the newly appeared ID and the histogram of the tracked subject before it was lost are compared using a method such as Pearson's correlation function or chi-square test, and if the similarity is high, the ID is updated. With this processing, the characteristics of the body and clothing colors of the tracked subject are taken into account, and the tracked subject ID is updated appropriately even when it is lost. The ID may also be updated by combining these features.

In step S109, the CPU 101 determines whether the "tracking subject ID inference result (position of the subject to be tracked)" stored in the RAM 102 in step S101 matches the "tracking subject ID inference result (position of the subject to be tracked)" stored in the RAM 102 in the previous step S101. The determination that the two positions match is not limited to a determination that the two positions are the same position, but may also include a determination that the difference between the two positions is within a threshold value.

If it is determined that there is a match, the process for the current frame according to the flowchart in FIG. 3(a) ends, and if there is no match, the process proceeds to step S110.

In step S110, CPU 101 calculates the difference between the position of the subject to be tracked stored in RAM 102 in step S101 and the position of the subject to be tracked stored in RAM 102 in the previous step S101. CPU 101 then uses the calculated difference to calculate angular velocities (vectors) in the pan and tilt directions that allow imaging unit 111 to track and capture the subject to be tracked. For example, CPU 101 may calculate the angular velocity in the pan direction to be greater the greater the difference in the horizontal direction, and may calculate the angular velocity in the tilt direction to be greater the greater the difference in the vertical direction.

In step S111, CPU 101 uses the vector obtained in step S110 to obtain drive parameters for panning and tilting at a desired speed in the direction toward the subject to be tracked. Specifically, CPU 101 obtains drive parameters for controlling motors (not shown) included in drive unit 109 for the pan direction and tilt direction.

In step S112, the CPU 101 controls the drive of the drive unit 109 via the drive I/F 108 based on the drive parameters determined in step S111, and the drive unit 109 rotates through the drive control. This allows the imaging direction of the imaging unit 111 to be changed, i.e., a pan/tilt operation to be performed.

In addition, the CPU 101 may determine parameters for controlling the zoom position of the imaging unit 111 so that the subject to be tracked is included as large as possible within the angle of view range of the imaging unit 111, and may control the driving of the driving unit 109 based on the parameters.

Next, the operation of the controller 200 will be described with reference to the flowchart in FIG. 3(b). In step S201, the CPU 201 receives transmission information sent from the camera 100 via the network I/F 204, and stores the received transmission information in the RAM 202.

In step S202, the CPU 201 causes the display unit 205 to display the captured image and the inference result contained in the transmission information stored in the RAM 202 in step S201. An example of the captured image and the inference result displayed on the display unit 205 is shown in FIG. 5(a). Hereinafter, the CPU 201 will be responsible for all display control of the display screen displayed on the display unit 205 and processing performed in response to user operations on the display screen.

As shown in FIG. 5(a), a selection screen 800 displayed on the display screen of the display unit 205 displays a captured image 900 included in the transmission information. A rectangle 910 is drawn on the captured image 900 based on the inference result corresponding to person 701 included in the transmission information, and a rectangle 911 is drawn on the captured image 900 based on the inference result corresponding to person 702 included in the transmission information. Also drawn on the captured image 900 are "1", which is the ID 920 of the inference result corresponding to person 701, and "2", which is the ID 921 of the inference result corresponding to person 702.

The values of ID920 and ID921 are displayed as the IDs currently being detected at the bottom of the selection screen 800. By looking at the selection screen 800 displayed on the display unit 205, the user of the controller 200 can check the image captured by the camera 100 and the results of inference on the captured image.

Next, in step S203, the CPU 201 acquires the ID of the subject to be tracked, selected by the user through the user input I/F 206, as a tracking subject ID, and stores the acquired tracking subject ID in the RAM 202. The configuration for acquiring the tracking subject ID is not limited to a specific configuration. An example of a configuration for acquiring the tracking subject ID is described below.

The user operates the user input I/F 206 to specify the rectangle of the subject that the user wishes to select as the subject to be tracked from among rectangles 910 and 911 displayed on the selection screen 800 in FIG. 5(a). For example, the user can select the desired rectangle by tilting the joystick left or right, pressing the left or right button, or touching the desired rectangle on the touch panel screen. The method of selecting the rectangle is not limited to a specific method of specification.

When the user operates the user input I/F 206 to select rectangle 910, the selection screen 800 in FIG. 5(b) is displayed on the display screen 800 of the display unit 205. On the selection screen 800 in FIG. 5(b), rectangles 910 and 911 are displayed in different display forms so that the user can visually recognize that rectangle 910 has been selected. At this time, since rectangle 910 has been selected, "1", which is the ID 920 of the person 701 corresponding to rectangle 910, is displayed at the bottom of the selection screen 800 as the selected ID.

When the user operates the user input I/F 206 to determine the person 701 as the subject to be tracked, the selection screen 800 of FIG. 5(c) is displayed on the display screen 800 of the display unit 205. On the selection screen 800 of FIG. 5(c), the rectangles 910 and 911 are displayed in different display forms so that the user can visually recognize that the rectangle 910 has been determined.

For example, the user can determine the desired rectangle by pressing the joystick, pressing the confirm button, or touching the desired rectangle twice on the touch panel screen. The method of determining the rectangle is not limited to a specific designation method.

The screen shown in FIG. 5 is merely one example of a screen for allowing the user to visually confirm the selection state of the subject to be tracked, and the above operations on the screen are also merely one example. For example, instead of a rectangle, ID910 or ID921 may be selected, or the user may directly input the tracking subject ID by operating the user input I/F 206.

Next, in step S204, the CPU 201 transmits the tracking subject ID stored in the RAM 202 in step S203 to the camera 100 via the network I/F 204.

As described above, according to this embodiment, even if a subject with the same ID as the tracking subject ID is not detected in the captured image, if the ID of the subject detected in the captured image contains a new ID that is different from the ID of the subject detected in the captured image before the subject was lost, the new ID is updated as the tracking subject ID. This makes it possible to track and capture the image of the tracking target subject even after the subject is lost.

Second Embodiment
The following describes the differences from the first embodiment, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. In this embodiment, the camera 100 transmits a captured image to the controller 200, and the controller 200 generates a control command for causing the camera 100 to track a subject to be tracked, and transmits the generated control command to the camera 100. The camera 100 then tracks the subject to be tracked according to the control command, and captures an image of the subject to be tracked.

In this embodiment, the drive unit 109 operates according to a control command transmitted from the controller 200. The inference unit 210 performs the same operation as the inference unit 110 using the captured image transmitted from the camera 100 to obtain an inference result for the subject included in the captured image.

Next, the operation of the camera 100 operating as an image processing device will be described with reference to the flowchart in FIG. 6(a). In step S301, similar to step S101 above, the image processing unit 106 generates a captured image in accordance with instructions from the CPU 101, and stores the generated captured image in the RAM 102. The CPU 101 then transmits the captured image stored in the RAM 102 to the controller 200 via the network I/F 105.

In step S302, the CPU 101 receives the control command sent from the controller 200 via the network I/F 105 and stores the received control command in the RAM 102.

In step S303, the CPU 101 reads values indicating the operation direction and control amount for each of the pan direction and tilt direction from the control command stored in the RAM 102 in step S302.

In step S304, the CPU 101 determines drive parameters for panning and tilting in the desired direction at the desired speed based on the values read in step S303. The CPU 101 may convert the drive parameters into drive parameters by referring to a conversion table stored in advance in the storage device 103 based on the operation direction and control amount included in the control command.

In step S305, the CPU 101 controls the drive of the drive unit 109 via the drive I/F 108 based on the drive parameters determined in step S304, and the drive unit 109 rotates through this drive control. This allows the imaging direction of the imaging unit 111 to be changed, i.e., a pan/tilt operation to be performed.

Next, the operation of the controller 200 will be described with reference to the flowchart in FIG. 6(b). The process according to the flowchart in FIG. 6(b) is for one frame of captured image, and in reality, the controller 200 performs the process according to the flowchart in FIG. 6(b) for each frame of captured image.

In step S401, the CPU 201 receives the captured image sent from the camera 100 via the network I/F 204 and stores the received captured image in the RAM 202.

In step S402, the CPU 201 inputs the captured image stored in the RAM 202 in step S401 to the inference unit 210. The inference unit 210 performs the same processing as the inference unit 110 described above according to instructions from the CPU 201 to generate an inference result for the subject included in the captured image, and stores the generated inference result in the RAM 202. Furthermore, the CPU 201 assigns a unique ID to the subject for the inference result for the subject, as in the first embodiment.

In step S403, the CPU 201 causes the display unit 205 to display the captured image stored in the RAM 202 in step S401 and the inference result (including the ID of the inference result) stored in the RAM 202 in step S402. The display contents are the same as those in the first embodiment.

In step S404, the CPU 201 accepts a user's selection operation of a subject to be tracked. The user's selection operation of a subject to be tracked is the same as in the first embodiment.

In step S405, the CPU 201 determines whether a user operation has been performed to change the tracking subject ID stored in the RAM 202. For example, the CPU 201 determines whether a user operation has been performed to select a tracking subject for the first time through a selection operation, or to select a tracking subject different from the previous tracking subject.

If the result of this determination is that a user operation has been performed to change the tracking subject ID stored in RAM 202, the process proceeds to step S407. On the other hand, if a user operation has not been performed to change the tracking subject ID stored in RAM 202, the process proceeds to step S406.

In step S406, the CPU 201 determines whether or not a determined tracking subject ID is stored in the RAM 202 (whether or not the tracking subject ID has been determined) in the same manner as in step S104 above. If the result of this determination is that a determined tracking subject ID is stored in the RAM 202 (the tracking subject ID has been determined), the process proceeds to step S408. On the other hand, if a determined tracking subject ID is not stored in the RAM 202 (the tracking subject ID has not been determined), the process proceeds to step S401. In step S407, the CPU 201 stores the ID of the subject selected as the subject to be tracked in response to a user operation in the RAM 202 as the tracking subject ID.

In step S408, the CPU 201 determines whether a subject corresponding to the same ID as the tracking subject ID has been detected in the captured image of the current frame, in the same manner as in step S106 above.

If the result of this determination is that a subject corresponding to the same ID as the tracking subject ID has been detected, the process proceeds to step S411. On the other hand, if a subject corresponding to the same ID as the tracking subject ID has not been detected, the CPU 201 determines that the tracking target subject has been lost, stores the current time T in the RAM 202, and the process proceeds to step S409, as in the first embodiment.

In step S409, similar to step S107, the CPU 201 determines whether the IDs of the inference results of the subject detected from the current frame include a new ID that is different from any of the IDs of the inference results of the subject detected from frames prior to time T.

If the result of this determination is that a new ID is included in the IDs of the inference results of the subject detected from the current frame, the process proceeds to step S410. On the other hand, if a new ID is not included in the IDs of the inference results of the subject detected from the current frame, the process proceeds to step S401.

In step S410, the CPU 201 updates the tracking subject ID stored in the RAM 202 to a new ID. In step S411, similar to step S109 above, the CPU 201 determines whether the "tracking subject ID inference result (position of the subject to be tracked)" stored in the RAM 202 in step S402 matches the "tracking subject ID inference result (position of the subject to be tracked)" stored in the RAM 202 in the previous step S402.

If it is determined that there is a match, the process for the current frame according to the flowchart in FIG. 6(a) ends; if there is no match, the process proceeds to step S412.

In step S412, similar to step S110 above, the CPU 201 calculates the difference between the position of the subject to be tracked stored in the RAM 202 in step S402 and the position of the subject to be tracked stored in the RAM 202 in the previous step S402. The CPU 201 then uses the calculated difference to calculate angular velocities (vectors) in the pan and tilt directions that allow the imaging unit 111 to track and image the subject to be tracked.

In step S413, the CPU 201 converts the vector data calculated in step S412 into a control command in accordance with a protocol previously determined as a method for controlling the camera 100, and stores the converted control command in the RAM 202.

In step S414, the CPU 201 transmits the control command stored in the RAM 202 in step S413 to the camera 100 via the network I/F 204.

In this way, in this embodiment, even if the controller 200 is provided with a function for generating a control command for causing the camera 100 to track the subject to be tracked and transmitting the generated control command to the camera 100, the same effect as in the first embodiment can be obtained. The numerical values, processing timing, processing order, processing subject, data (information) acquisition method/destination/source/storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

Furthermore, any or all of the embodiments described above may be used in appropriate combination.Furthermore, any or all of the embodiments described above may be used selectively.

Other Embodiments
The present disclosure can also be realized by a process in which a program for realizing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. Also, the present disclosure can be realized by a circuit (e.g., ASIC) for realizing one or more functions.

The invention of this specification includes the following image processing device, image processing method, and computer program.
(Item 1)
A detection means for detecting a subject from a frame;
and if a subject having identification information identical to the identification information of the subject to be tracked is not detected in the first frame, and the identification information of the subject detected in the first frame includes new identification information that is different from the identification information of the subject detected in the second frame prior to the timing when the subject to be tracked was not detected, updating the identification information of the subject to be tracked to the new identification information.
(Item 2)
The image processing device described in item 1, characterized in that the update means further determines whether to update the identification information of the subject to be tracked to the new identification information based on whether the position of the subject corresponding to the new identification information is located within a range.
(Item 3)
3. The image processing device according to claim 2, wherein the update means changes the size of the range according to the size of the subject to be tracked before the timing.
(Item 4)
3. The image processing device according to claim 2, wherein the update means changes the size of the range in accordance with a moving speed of the subject to be tracked before the timing.
(Item 5)
3. The image processing device according to claim 2, wherein the update means changes a shape of the range based on a moving direction of the tracking target before the timing.
(Item 6)
3. The image processing device according to claim 2, wherein the update means changes the size of the range in accordance with the elapsed time from the timing.
(Item 7)
moreover,
An imaging means is provided,
7. The image processing device according to claim 1, wherein the detection means detects a subject from a frame captured by the imaging means.
(Item 8)
moreover,
8. The image processing device according to item 7, further comprising a means for controlling the imaging means so as to track and image a subject having the same identification information as the identification information of the subject to be tracked, among the subjects detected from the frames captured by the imaging means.
(Item 9)
7. The image processing device according to any one of items 1 to 6, wherein the detection means detects a subject from a frame received from an external imaging means capable of communicating with the image processing device.
(Item 10)
moreover,
10. The image processing device according to item 9, further comprising: a means for generating a control command for controlling the imaging means to track and image a subject having the same identification information as the identification information of the subject to be tracked, and a means for transmitting the control command to the imaging means.
(Item 11)
moreover,
After the timing when the tracking target subject is no longer detected, if the feature amount of the detected subject to which new identification information has been added is close to the feature amount of the tracking target subject before the tracking target subject is no longer detected,
2. The image processing device according to item 1, further comprising a means for selecting, as a tracking target, a subject to which new identification information has been added after the timing at which the tracking target subject is no longer detected.
(Item 12)
12. The image processing device according to item 11, wherein the feature amount includes at least one of the subject's orientation, body shape, posture, size, and aspect ratio.
(Item 13)
12. The image processing device according to item 11, wherein the feature amount includes at least one of a color of the subject and a histogram.
(Item 14)
An image processing method performed by an image processing device, comprising:
a detection step in which a detection means of the image processing device detects a subject from a frame;
an updating step in which, when a subject having identification information identical to the identification information of the subject to be tracked is not detected in the first frame, the updating means of the image processing device updates the identification information of the subject to be tracked to the new identification information if the identification information of the subject detected in the first frame includes new identification information that is different from the identification information of the subject detected in the second frame prior to the time when the subject to be tracked was not detected.
(Item 15)
14. A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 13.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100: Camera 101: CPU 102: RAM 103: Storage device 105: Network I/F 106: Image processing unit 107: Image sensor 108: Drive I/F 109: Drive unit 110: Inference unit 111: System bus 200: Controller 201: CPU 202: RAM 203: Storage device 204: Network I/F 205: Display unit 206: User input I/F 207: System bus 210: Inference unit

Claims (15)

A detection means for detecting a subject from a frame;
and if a subject having identification information identical to the identification information of the subject to be tracked is not detected in the first frame, and the identification information of the subject detected in the first frame includes new identification information that is different from the identification information of the subject detected in the second frame prior to the timing when the subject to be tracked was not detected, updating the identification information of the subject to be tracked to the new identification information. The image processing device according to claim 1, characterized in that the update means further determines whether to update the identification information of the subject to be tracked to the new identification information based on whether the position of the subject corresponding to the new identification information is located within a range. The image processing device according to claim 2, characterized in that the update means changes the size of the range depending on the size of the subject to be tracked before the timing. The image processing device according to claim 2, characterized in that the update means changes the size of the range depending on the moving speed of the subject to be tracked before the timing. The image processing device according to claim 2, characterized in that the update means changes the shape of the range based on the direction of movement of the tracking target before the timing. The image processing device according to claim 2, characterized in that the update means changes the size of the range depending on the elapsed time from the timing. moreover,
An imaging means is provided,
2. The image processing apparatus according to claim 1, wherein the detection means detects a subject from a frame captured by the imaging means. moreover,
8. The image processing device according to claim 7, further comprising: a means for controlling the imaging means so as to track and image a subject having the same identification information as a tracking target subject among subjects detected from a frame captured by the imaging means. The image processing device according to claim 1, characterized in that the detection means detects a subject from a frame received from an external imaging means capable of communicating with the image processing device. moreover,
10. The image processing device according to claim 9, further comprising: a means for generating a control command for controlling the imaging means to track and image a subject having the same identification information as the identification information of the subject to be tracked, and transmitting the control command to the imaging means. moreover,
After the timing when the tracking target subject is no longer detected, if the feature amount of the detected subject to which new identification information has been added is close to the feature amount of the tracking target subject before the tracking target subject is no longer detected,
2 . The image processing device according to claim 1 , further comprising: a means for selecting, as a tracking target, a subject to which new identification information has been added after the timing at which the tracking target subject is no longer detected. The image processing device according to claim 11, characterized in that the feature amount includes at least one of the subject's orientation, body shape, posture, size, and aspect ratio. The image processing device according to claim 11, characterized in that the feature amount includes at least one of the color and histogram of the subject. An image processing method performed by an image processing device, comprising:
a detection step in which a detection means of the image processing device detects a subject from a frame;
an updating step in which, when a subject having identification information identical to the identification information of the subject to be tracked is not detected in the first frame, the updating means of the image processing device updates the identification information of the subject to be tracked to the new identification information if the identification information of the subject detected in the first frame includes new identification information that is different from the identification information of the subject detected in the second frame prior to the time when the subject to be tracked was not detected. A computer program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 13.

Images