JP2018112890A - Object tracking program, object tracking method, and object tracking device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately detect the same object even when the objects overlap each other on an image. An object tracking program detects an object from each of a first image and a second image captured at a different time from the first image, and positions the object in front of the first image. It is determined whether or not there is an overlap between the first object and the second object located on the back side, and if the overlap occurs, an image of the area where the overlap area is removed. The range used for determining the presence or absence of the second object is determined based on the above, and the range is moved based on the tracking result of the first object, whereby the second in the second image. By setting a search area for tracking the object of the above and searching the search area for the second image, a computer is made to execute a process of determining the presence or absence of the second object. [Selection diagram] Fig. 5

Description

  The present invention relates to an object tracking program, an object tracking method, and an object tracking apparatus.

  In recent years, surveillance cameras have been installed in various places such as stores and roads. In connection with this, the image captured by the surveillance camera is tried not only for the purpose of conventional monitoring but also for other purposes. As an example, there is a usage mode in which a person's movement route is obtained from an image captured by a surveillance camera and used for marketing, traffic volume survey, or the like.

  Here, in order to calculate the movement path of a person, it is required to accurately detect the same person in each image and perform person tracking even in a situation where the persons overlap on the image.

  In contrast, for example, the same person can be searched by using a characteristic of a surveillance camera installed at an overhead position and searching for a predetermined area using a part (head area, etc.) that is relatively unlikely to overlap. There has been proposed a method of detecting a person and tracking a person.

JP 2014-186524 A JP 2012-108785 A JP 2006-031153 A

  However, in the case of an image captured by a surveillance camera, it is difficult to accurately capture parts such as eyes and nose, and the image feature amount that can be extracted from the head region or the like is limited. Further, in a situation where persons overlap on the image, a plurality of persons may be included in a predetermined area to be searched. For this reason, even if a predetermined region is searched using image feature amounts extracted from the head region or the like, it is difficult to detect the same person in each image, and there is a problem that high tracking accuracy cannot be obtained.

  In one aspect, the same object can be detected with high accuracy even when the objects overlap each other on the image.

According to one aspect, the object tracking program is
An object is detected from each of the second images captured at different times of the first image and the first image;
Determining whether or not there is an overlap between the first object located in front of the first image and the second object located on the back side in the first image;
When overlapping has occurred, based on the image of the area from which the overlapping area has been removed, a range to be used for determining the presence or absence of the second object is determined,
A search area for tracking the second object in the second image is set by moving the range based on the tracking result of the first object, and the second image By searching the search area, the computer is caused to execute processing for determining the presence or absence of the second object.

  Even when the objects overlap on the image, the same object can be detected with high accuracy.

It is a figure which shows an example of the system configuration | structure of the whole person tracking system. It is a figure which shows the example of installation of each apparatus of a person tracking system. It is a figure which shows an example of moving image data and an example of tracking object information. It is a figure which shows an example of the hardware constitutions of a person tracking device. It is a figure which shows an example of a function structure of a person tracking device. It is a flowchart of a person tracking process. It is a flowchart of the person detection process performed by a person detection part. It is the figure which showed typically the person detection process performed by a person detection part. It is a flowchart of the overlap determination process performed by an overlap determination part. It is the figure which showed typically the overlap determination process performed by an overlap determination part. It is a flowchart of the overlap removal process performed by an overlap removal part. It is the figure which showed typically the overlap removal process performed by the overlap removal part. It is a flowchart of the existence range determination process performed by an existence range determination part. It is the figure which showed typically the existence range determination process performed by an existence range determination part. It is a flowchart of the movement amount calculation process performed by the movement amount calculation unit. It is the figure which showed typically the movement amount calculation process performed by the movement amount calculation part. It is a flowchart of a feature area calculation process executed by a feature area calculation unit. It is the 1st figure showing typically the feature region calculation processing performed by the feature region calculation part. It is the 2nd figure showing typically the feature region calculation processing performed by the feature region calculation part. It is a flowchart of the search process performed by a search part. It is the figure which showed typically the search process performed by the search part. It is a figure which shows an example of the tracking object information after a search process.

  Hereinafter, embodiments will be described with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, the duplicate description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[Embodiment]
<System configuration of the entire person tracking system>
First, a system configuration of the entire person tracking system including the person tracking apparatus according to an embodiment will be described. FIG. 1 is a diagram illustrating an example of a system configuration of the entire person tracking system.

  As shown in FIG. 1, the person tracking system 100 includes an imaging device 110, a moving image data storage device 120, a person tracking device 130, and an analysis device 140. Each device included in the person tracking system 100 is connected to each other via a network 150.

  The imaging device 110 is a so-called surveillance camera and is installed in a store, a road, or the like. The imaging device 110 continuously captures images for a predetermined period, and transmits the captured moving image data to the moving image data storage device 120 via the network 150.

  The moving image data storage device 120 stores moving image data transmitted from the imaging device 110. The moving image data stored by the moving image data storage device 120 is transmitted to the person tracking device 130 in response to a request from the person tracking device 130.

  The person tracking device 130 is installed with a person tracking program and a tracking target information output program. The person tracking device 130 functions as the person tracking unit 131 and the tracking target information output unit 132 by executing the program.

  The person tracking unit 131 reads moving image data from the moving image data storage device 120, detects a person in each frame of the moving image data, and detects a person (same person) that matches the detected person in the next frame. In this way, person tracking is performed. The person tracking unit 131 stores information about the person detected in each frame in the tracking target information storage unit 133 as tracking target information.

  The tracking target information output unit 132 transmits the tracking target information stored in the tracking target information storage unit 133 to the analysis device 140 in response to a request from the analysis device 140.

  The analysis device 140 analyzes the movement path of the person based on the tracking target information in consideration of the arrangement of each structure included in the imaging range of the imaging device 110. Thereby, the user of the analysis apparatus 140 can plan a marketing strategy according to the analysis result, for example.

<Installation example of person tracking system>
Next, an installation example of each device (the imaging device 110, the moving image data storage device 120, the person tracking device 130, and the analysis device 140) of the person tracking system 100 will be described. FIG. 2 is a diagram illustrating an installation example of each device of the person tracking system.

  The example of FIG. 2 shows a case where the person tracking system 100 is installed for a store. As shown in FIG. 2, the imaging device 110 is installed near the ceiling of the store 200 in the store. The moving image data storage device 120 and the analysis device 140 are installed in the employee waiting room 210. Furthermore, the person tracking device 130 is set in a processing center or the like that processes moving image data of each store and transmits tracking target information to the analysis device 140 of each store.

  The moving image data picked up by the image pickup device 110 is temporarily stored in the moving image data storage device 120 and processed by the person tracking device 130 to generate tracking target information. The analysis device 140 reads the tracking target information from the person tracking device 130, analyzes the movement path of each person on the sales floor 200, and displays the analysis result. The display screen 220 shows an example of the analysis result displayed on the analysis device 140. As shown in the display screen 220, according to the analysis device 140, by analyzing the tracking target information, each movement route from the time when each person in the sales floor 200 enters from the entrance of the sales floor 200 to the present time. Can be displayed as a curve.

<Moving image data and tracking target information>
Next, moving image data processed by the person tracking device 130 and tracking target information stored in the person tracking device 130 will be described. FIG. 3 is a diagram illustrating an example of moving image data and an example of tracking target information.

  Among these, Fig.3 (a) is a figure which shows an example of moving image data. The moving image data processed by the person tracking device 130 includes a plurality of frames, and the person tracking device 130 processes the moving image data in units of frames. In FIG. 3A, image data 300 indicates image data for one frame processed by the person tracking device 130. As shown in FIG. 3A, the image data of each frame has an x-axis in the horizontal direction and a y-axis in the vertical direction. Note that the example in FIG. 3A shows a state in which a person 301 is detected and a person area 302 is calculated in the image data 300 to be processed.

  FIG. 3B is a diagram illustrating an example of tracking target information. In the tracking target information 310, information about a person detected in the image data to be processed is recorded. Information items recorded in the tracking target information 310 include “person ID”, “time [frame]”, “person area”, “movement amount (x, y)”, “overlap area”, and “overlap flag”. , “Overlapping relationship”, and “tracking flag”.

  In “person ID”, an identifier for identifying the person when a new person is detected in the image data to be processed is recorded.

  In “time [frame]”, the frame number of the image data including the person identified by the person ID is recorded.

  The “person area” includes “x coordinate [pixel]”, “y coordinate [pixel]”, “width [pixel]”, and “height [pixel]”. In “x coordinate [pixel]”, the x coordinate of the upper left vertex of the person area including the person identified by the person ID (for example, in the case of the person area 302 in FIG. 3A, the x of the upper left vertex 311). Coordinates) is recorded. In “y coordinate [pixel]”, the y coordinate of the upper left vertex of the person area including the person identified by the person ID (for example, y in the upper left vertex 311 in the case of the person area 302 in FIG. 3A). Coordinates) is recorded. “Width [pixel]” includes the length in the x direction of the person area including the person identified by the person ID (for example, the length indicated by reference numeral 312 in the case of the person area 302 in FIG. 3A). ) Is recorded. “Height [pixel]” includes the length in the y direction of the person area including the person identified by the person ID (for example, the length indicated by reference numeral 313 in the case of the person area 302 in FIG. 3A). Is recorded.

  In “movement amount (x, y)”, the movement amount of the person when compared with the image data of the frame number immediately before the frame number of the image data to be processed is recorded. In “movement amount (x, y)”, when a person detected in the image data of the previous frame number can also be detected in the image data to be processed (that is, when the person can be tracked) B), the movement amount of the person is recorded.

  The “overlap area” includes “x coordinate [pixel]”, “y coordinate [pixel]”, “width [pixel]”, and “height [pixel]”. In the “x coordinate [pixel]”, when two person areas overlap in the image data, the x coordinate of the upper left vertex of the overlapping area (overlapping area) is recorded. In “y coordinate [pixel]”, the y coordinate of the upper left vertex of the overlapping area is recorded. In “width [pixel]”, the length of the overlapping region in the x direction is recorded. In “height [pixel]”, the length of the overlapping region in the y direction is recorded.

  In the “overlap flag”, “1” is recorded when the person area of the person identified by the person ID overlaps with the person area of another person in front, and the person area of the other person in front is recorded. "0" is recorded when it does not overlap.

  In the “overlapping relationship”, when the person area of the person identified by the person ID overlaps with the person area of another person in the foreground (when there is an overlapping relationship), A person ID is recorded.

  In the “tracking flag”, “0” is recorded in the overlap flag, and the same person as the person identified by the person ID has a frame number one after the frame number of the image data to be processed. When it is detected in the image data, “1” is recorded.

<Hardware configuration of person tracking device>
Next, the hardware configuration of the person tracking device 130 will be described. FIG. 4 is a diagram illustrating an example of a hardware configuration of the person tracking device. As shown in FIG. 4, the person tracking device 130 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, and a RAM (Random Access Memory) 403. The CPU 401, ROM 402, and RAM 403 form a so-called computer.

  The person tracking device 130 includes an auxiliary storage device 404, a display device 405, an operation device 406, an I / F (Interface) device 407, and a drive device 408. Each unit of the person tracking device 130 is connected to each other via a bus 409.

  The CPU 401 is a device that executes various programs (for example, a person tracking program, a tracking target information output program, etc.) installed in the auxiliary storage device 404. The ROM 402 is a nonvolatile memory. The ROM 402 functions as a main storage device that stores various programs, data, and the like necessary for the CPU 401 to execute various programs installed in the auxiliary storage device 404. Specifically, the ROM 402 stores a boot program such as a basic input / output system (BIOS) or an extensible firmware interface (EFI).

  The RAM 403 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). The RAM 403 functions as a main storage device that provides a work area that is expanded when various programs installed in the auxiliary storage device 404 are executed by the CPU 401.

  The auxiliary storage device 404 stores various programs, information generated by executing the various programs (for example, tracking target information 310), and information used when the various programs are executed (for example, similarities described later). This is an auxiliary storage device for storing a degree map or the like. The tracking target information storage unit 133 is realized in the auxiliary storage device 404.

  The display device 405 is a display device that displays an internal state or the like of the person tracking device 130. The operation device 406 is an input device for an administrator of the person tracking device 130 to input various instructions to the person tracking device 130.

  The I / F device 407 is a communication device that connects to the network 150 and communicates with the moving image data storage device 120 and the analysis device 140.

  The drive device 408 is a device for setting the recording medium 410. The recording medium 410 here includes a medium for recording information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, or the like. The recording medium 410 may include a semiconductor memory that electrically records information, such as a ROM and a flash memory.

  The various programs stored in the auxiliary storage device 404 are installed by, for example, setting the distributed recording medium 410 in the drive device 408 and reading the various programs recorded in the recording medium 410 by the drive device 408. Is done.

<Functional configuration of person tracking device>
Next, a functional configuration of the person tracking unit 131 of the person tracking device 130 will be described. FIG. 5 is a diagram illustrating an example of a functional configuration of the person tracking apparatus.

  As shown in FIG. 5, the person tracking unit 131 of the person tracking device 130 includes a person detection unit 501, an overlap determination unit 502, an overlap removal unit 503, an existence range determination unit 504, a movement amount calculation unit 505, and a feature region calculation unit 506. And a search unit 507.

  The person detection unit 501 is an example of a detection unit. When a new person appears in the image data to be processed, the person detection unit 501 detects the person and calculates the person area of the detected person.

  The overlap determination unit 502 is an example of a determination unit, and for the image data to be processed, whether or not there is an overlap of person areas for a person newly detected by the person detection unit 501 or a person already detected and tracked Determine. The image data to be processed includes a person whose person area overlaps the person area of another person (a person on the back side) and a person who does not overlap (a person on the near side). Therefore, the overlap determination unit 502 determines a person whose person areas overlap (a person on the back side) and a person who does not overlap (a person on the near side).

  Among these, for the person on the near side, the same person as the person can be easily detected in the image data to be processed next (that is, the person can be easily tracked). On the other hand, for the person on the back side, it is difficult to detect the same person as the person in the image data to be processed next (that is, person tracking is difficult).

  In view of this, in the person tracking unit 131, first, the overlap removing unit 503 removes the overlapping regions of the person regions from the image data to be processed for the person on the back side that is difficult to track. As a result, it is possible to identify a range in which a person on the back side exists.

  Subsequently, in the person tracking unit 131, the existence range determination unit 504, which is an example of a determination unit, tracks a person on the back side using an image in a range where the person on the back side exists (image data to be processed next). The effective range for determining the presence or absence of the same person as the person on the back side is determined. The existence range determination unit 504 determines the range by generating a similarity map. The similarity map is a map that calculates the similarity with an image in a range in which a person on the back side exists at each position of the image data to be processed, and expresses the similarity at each position as a gray image. In the similarity map, the high-density range is a range in which the features of the person on the back side are noticeable, and is an effective range for tracking the person on the back side.

  On the other hand, the movement amount calculation unit 505 detects and detects the same person as the person from the next processing target image data for the person determined to be non-overlapped by the overlap determination unit 502 (the person on the near side). The person area of the person is calculated. Further, the movement amount calculation unit 505 calculates the movement amount of the person on the near side between the processing target image data and the next processing target image data.

  The feature area calculation unit 506 is an example of a calculation unit, and the amount of movement from the processing target image data immediately before the processing target image data and the interval between the processing target image data and the next processing target image data. The movement amount change of the person on the near side is calculated by comparing the movement amount of the person. In this way, by calculating the change in the amount of movement of the person on the near side, how the overlapping area of the person on the back side calculated in the image data to be processed changes in the next image data to be processed Can be estimated.

  Therefore, the feature area calculation unit 506 estimates the overlap area after the change based on the change in the movement amount of the person on the near side, and estimates the range where the person on the back side exists in the next image data to be processed. To do.

  Furthermore, the feature region calculation unit 506 calculates the region of the image data to be processed according to the estimated range as a feature region used when searching for a person on the back side.

  The search unit 507 is generated by the existence range determination unit 504 according to the movement amount calculated by the movement amount calculation unit 505 (movement amount between the processing target image data and the next processing target image data). Move the similarity map. Further, the search unit 507 identifies the position of the similarity map in the next image data to be processed, and sets an area based on the identified position similarity map (similarity map after movement) as a search area.

  Further, the search unit 507 performs a search on the search area to determine whether or not the same person as the back person exists in the next image data to be processed. Specifically, the search unit 507 searches for the search area using the image of the feature area calculated by the feature area calculation unit 506. If the image of the search area is similar to the image of the feature area calculated by the feature area calculation unit 506, the search unit 507 determines that the same person as the back person is present in the search area.

  As described above, the person tracking unit 131 uses the tracking result of the person who is easy to track the person (the person on the near side) to calculate a feature region used when searching for a person who is difficult to track the person and Set the area. Thus, according to the person tracking unit 131, it is possible to perform a search based on more image feature amounts than in the case where the head region or the like is a feature region. Further, according to the person tracking unit 131, a more effective search area (a search area that is more likely to exist) is compared with a case where a predetermined range is searched as a search area regardless of the degree of overlap with the person on the near side. ) Can be searched.

  As a result, the person tracking unit 131 can accurately detect the same person even when the persons overlap in the image.

<Flow of person tracking process>
FIG. 6 is a flowchart of the person tracking process. Hereinafter, the flow of the person tracking process performed by each unit of the person tracking unit 131 will be described with reference to the flowchart shown in FIG. In addition, when each part of the person tracking part 131 performs the person tracking process shown in FIG. 6, it is assumed that the person tracking part 131 has acquired moving image data from the moving image data storage device 120 in advance.

  In step S601, the person detection unit 501 reads the moving image data acquired by the person tracking unit 131 in units of frames.

  In step S <b> 602, the person detection unit 501 performs person detection processing on the read frame-unit image data to detect a person newly included in the processing target image data, and records it in the tracking target information 310. To do.

  In step S <b> 603, the overlap determination unit 502 executes overlap determination processing for each person including the person detected in step S <b> 602, thereby determining whether or not person regions overlap, and records the determination result in the tracking target information 310. To do.

  In step S604, the overlap removal unit 503 removes the overlap region by executing an overlap removal process on the person region determined to have overlap in step S603.

  In step S605, the existence range determination unit 504 identifies the existence range of persons who are determined to have overlap by executing the existence range determination process when the overlapping region is removed in step S604. In addition, the existence range determination unit 504 performs an existence range determination process, thereby generating a similarity map based on the identified existence range, and determining an effective range for tracking a person who is determined to have an overlap. To do.

  In step S606, the movement amount calculation unit 505 executes the movement amount calculation process to detect the same person as the person determined not to overlap in step S603 from the image data to be processed next. Further, the movement amount calculation unit 505 calculates the movement amount based on the person area of the person detected from the next processing target image data by performing the movement amount calculation process, and records the movement amount in the tracking target information 310.

  In step S607, the feature region calculation unit 506 determines whether or not feature region calculation processing is necessary. In the case where there is a person who has been determined that the overlap determination unit 502 has overlapped person areas in step S603 and has not performed the feature area calculation process, the feature area calculation unit 506 needs the feature area calculation process. judge. In this case, the feature area calculation unit 506 proceeds to step S608 and performs processing for the person who is determined to have the overlap.

  Specifically, in step S608, the feature area calculation unit 506 executes a feature area calculation process, so that the change in the movement amount of the person determined to have no overlap based on the movement amount calculated in step S606. Is calculated. Further, the feature region calculation unit 506 calculates a feature region based on the calculated change in the movement amount.

  In step S609, the search unit 507 executes a search process. Specifically, the search unit 507 sets a search area in the next image data to be processed based on the similarity map generated in step S605 and the movement amount calculated in step S606. In addition, the search unit 507 performs a search using the feature region image calculated in step S608 in the set search region. Thereby, the search unit 507 determines whether or not the same person as the person who is determined to have an overlap in the processing target image data exists in the next processing target image data. In addition, the search unit 507 records the search result in the tracking target information 310. Note that the processing in step S608 and step S609 is executed for all persons determined to have overlapping human areas.

  When the processes in step S608 and step S609 are executed for all persons determined to have overlapping human areas, the characteristic area calculation unit 506 determines in step S607 that the characteristic area calculation processing is unnecessary, Proceed to step S610.

  In step S610, the person detection unit 501 determines whether or not to continue the same processing for the image data of the next frame. If it is determined in step S610 that the process is to be continued, the process returns to step S601. On the other hand, if it is determined in step S610 that the process is not continued, the person tracking process is terminated.

<Details of processing by each part of the person tracking part>
Next, details of each process (steps S602 to S606 and S608 to S609) performed by each unit of the person tracking unit 131 will be described.

(1) Details of Person Detection Processing by Person Detection Unit First, details of the person detection processing (step S602) by the person detection unit 501 will be described. FIG. 7 is a flowchart of the person detection process executed by the person detection unit.

  When the image data to be processed is read, the person detection process shown in FIG. 7 is executed. In step S <b> 701, the person detection unit 501 calculates a difference from the image data having a frame number one before the frame number of the image data to be processed. Thereby, the person detection unit 501 calculates background difference image data.

  In step S <b> 702, the person detection unit 501 identifies an area (person area candidate) that may be a person area from the background difference image data, and determines each pixel of the identified person area candidate from the image data to be processed. Extract pixel values. In addition, the person detection unit 501 calculates a HOG (Histograms of Oriented Gradients) feature amount based on the extracted pixel value.

  In step S <b> 703, the person detection unit 501 determines whether the identified person area candidate is a person area by inputting the calculated HOG feature amount to a classifier generated in advance.

  In step S704, when the person detection unit 501 determines that the identified person area candidate is a person area, the person detection unit 501 calculates the x coordinate and y coordinate of the upper left vertex of the area, and the width and height of the area. In addition, the person detection unit 501 records the calculated x coordinate, y coordinate, width, and height in the tracking target information 310 as a person area of a new person in the processing target image data.

  FIG. 8 is a diagram schematically illustrating a person detection process executed by the person detection unit. In FIG. 8, image data 801 is image data having a frame number one before the frame number of the image data 811 to be processed.

  The person detection unit 501 calculates the background image data 802 by detecting the difference between the image data 811 and the image data 801, and detects a person area candidate 803.

Furthermore, the person detection unit 501 extracts the pixel value of each pixel of the person region candidate 803 from the image data 811 and calculates the HOG feature amount. In addition, the person detection unit 501 determines whether the person area candidate 803 is a person area by inputting the calculated HOG feature amount to the classifier 820 generated in advance. Example of FIG. 8, the result of the determination, it is determined that a person region, x-coordinate _{= x 812,} y-coordinate _{= y 812,} width _{= W 812,} how the person area 812 is calculated height _{= H 812} Show.

(2) Details of Overlap Determination Processing by Overlap Determination Unit Next, details of the overlap determination processing (step S603) by the overlap determination unit 502 will be described. FIG. 9 is a flowchart of the overlap determination process executed by the overlap determination unit. It is assumed that the person area for each person included in the image data to be processed has already been calculated when starting the overlap determination process shown in FIG.

  In step S <b> 901, the overlap determination unit 502 reads, from the tracking target information 310, the “person area” of the person to be determined in the overlap determination process among the persons included in the image data to be processed.

  In step S <b> 902, the overlap determination unit 502 reads “person area” of another person included in the image data to be processed from the tracking target information 310.

  In step S <b> 903, the overlap determination unit 502 determines whether there is an overlap of person areas for the person to be determined. If it is determined in step S903 that there are overlapping human areas (Yes in step S903), the process proceeds to step S904.

  In step S <b> 904, the overlap determination unit 502 determines whether the “person area” of the person to be determined has a smaller y coordinate than the “person area” of another person. If it is determined that the y coordinate is small, the overlap determination unit 502 sets “1” in the “overlap flag” associated with the person area of the determination target person in the tracking target information 310.

  In step S <b> 905, the overlap determination unit 502 identifies the person ID of another person (a person in a person area whose y coordinate is larger than the person area of the person to be determined). Further, the overlap determination unit 502 records the identified person ID in the “overlap relationship” corresponding to the “person ID” of the person to be determined in the tracking target information 310. Thereafter, the overlap determination unit 502 proceeds to step S907.

  On the other hand, if it is determined in step S903 that there is no overlapping of person areas (in the case of No in step S903), the process proceeds to step S906. In step S <b> 906, the overlap determination unit 502 sets the “overlap flag” associated with the “person area” of the determination target person in the tracking target information 310 to “0”. Thereafter, the overlap determination unit 502 proceeds to step S907.

  In step S907, the overlap determination unit 502 determines whether or not processing has been performed for all human areas included in the processing target image data. If it is determined in step S907 that there is a person region of a person who has not been processed (Yes in step S907), the process returns to step S901.

  On the other hand, when it is determined that the process has been performed for the person areas of all persons (No in step S907), the overlap determination process ends.

  FIG. 10 is a diagram schematically illustrating overlap determination processing executed by the overlap determination unit. 10A shows the image data 1000 to be processed, and FIG. 10B shows the tracking target information 310 when the image data 1000 to be processed is processed.

  The image data 1000 to be processed is image data with a frame number “239”. In addition, it is assumed that “3” is assigned as the “person ID” to the person 1010 included in the image data 1000 and “2” is assigned as the “person ID” to the person 1020. For simplification of description, the person with “1” as “person ID” in the image data 1000 is omitted.

  Assuming that the person 1020 is a person to be determined in the overlap determination process, the overlap determination unit 502 reads the person area 1021 of the person 1020 and the person area 1011 of another person 1010 from the tracking target information 310. In addition, the overlap determination unit 502 determines whether there is an overlap between the person area 1021 and the person area 1011.

Specifically, the overlap determination unit 502 determines that there is an overlap when the following condition is satisfied.
X1 _s <x2 _s <x1 _e or x1 _s <x2 _e <x1 _e is satisfied, and y1 _s <y2 _e <y1 _e is satisfied.
Or
X2 _s <x1 _s <x2 _e or x2 _s <x1 _e <x2 _e is satisfied, and y2 _s <y1 _s <y2 _e is satisfied.

However, x1 _s , x1 _e , y1 _s , and y1 _e indicate the minimum value and maximum value of the x coordinate of the person region 1011, and the minimum value and maximum value of the y coordinate, respectively. Further, x2 _s , x2 _e , y2 _s , and y2 _e indicate the minimum value and maximum value of the x coordinate of the person region 1021, and the minimum value and maximum value of the y coordinate, respectively.

  In the case of FIG. 10A, the overlap determination unit 502 determines that the person area 1011 overlaps the person area 1021. For this reason, the overlap determination unit 502 records the determination result in association with the image data to be processed (frame number = “239”) in the tracking target information 310. Specifically, the overlap determination unit 502 sets the “overlap flag” corresponding to the person ID = “2” of the person region 1021 having a small y coordinate value to “1” (see FIG. 10B). Further, the overlap determination unit 502 records the person ID = “3” of the person region 1011 having a large y coordinate value in the “overlap relationship” corresponding to the person ID = “2” of the determination target person 1020 (see FIG. 10 (b)).

  If the person 1010 is a person to be determined in the overlap determination process, the overlap determination unit 502 determines that the person 1010 does not overlap with another person 1020. For this reason, the overlap determination unit 502 records the determination result in association with the image data to be processed (frame number = “239”) in the tracking target information 310. Specifically, the overlap determination unit 502 sets “overlap flag” corresponding to the person ID = “3” of the person 1010 to “0” (see FIG. 10B).

(3) Details of Overlap Removal Processing by Overlap Removal Unit Next, details of the overlap removal processing (step S604) by the overlap removal unit 503 will be described. FIG. 11 is a flowchart of the overlap removal process executed by the overlap removal unit.

  In step S <b> 1101, the overlap removing unit 503 reads the “overlap flag” of the person to be processed included in the image data to be processed from the tracking target information 310 of the tracking target information storage unit 133.

  In step S1102, the overlap removing unit 503 determines whether or not the “overlap flag” is “1”. If it is determined in step S1102 that the “overlap flag” is “1” (Yes in step S1102), the process proceeds to step S1103.

  In step S <b> 1103, the overlap removal unit 503 reads the “overlap relationship” of the person to be processed from the tracking target information 310 of the tracking target information storage unit 133.

  In step S <b> 1104, the overlap removing unit 503 reads the “person area” of the person corresponding to the read “overlap relationship” from the tracking target information 310 of the tracking target information storage unit 133.

  In step S 1105, the overlap removing unit 503 reads the “person area” of the person to be processed from the tracking target information 310 of the tracking target information storage unit 133. Further, the overlap removing unit 503 calculates an overlap area with the “person area” read in step S1104, records the overlap area in the tracking target information 310, and then proceeds to step S1106.

  Accordingly, the overlap removal unit 503 can identify a range in which the “overlap region” is removed from the “person region” of the person to be processed as a range in which the person to be processed exists.

  On the other hand, if it is determined in step S1102 that the “overlap flag” is not “1” (in the case of No in step S1102), the process directly proceeds to step S1106.

  In step S1106, the overlap removing unit 503 determines whether or not processing has been performed on the person regions of all persons included in the image data to be processed. If it is determined in step S1106 that there is a person area of a person who has not performed processing (in the case of Yes in step S1106), the process returns to step S1101.

  On the other hand, if it is determined in step S1106 that the process has been performed for all the person areas (in the case of No in step S1106), the overlap removal process ends.

  FIG. 12 is a diagram schematically illustrating the overlap removal process executed by the overlap removal unit. Among these, FIG. 12A shows the image data 1000 to be processed, and FIG. 12B shows the calculated overlapping area 1210. FIG. 12C shows the tracking target information 310 when the overlap removal process is executed for the image data 1000.

  The image data 1000 to be processed is image data with a frame number “239”. The person 1010 included in the image data 1000 is assigned “3” as the “person ID”, and the person 1020 is assigned “2” as the “person ID”. For simplification of explanation, the person whose “person ID” is “1” in the image data 1000 is omitted.

  Assuming that the person 1020 is the person to be processed, the overlap removing unit 503 reads “1” from the tracking target information 310 as the “overlap flag” of the person 1020 (person ID = “2”). “3” is read out as “Relation”.

  Further, the overlap removal unit 503 reads “person area” (x coordinate, y coordinate, width, height) having “person ID” “3” from the tracking target information 310, and displays “person” of the person 1020 to be processed. An overlapping region 1210 is calculated between the region "(x coordinate, y coordinate, width, height). For example, the overlap removing unit 503 calculates the x coordinate and the y coordinate of the overlap region 1210 based on the following expression.

ただし、ｍａｘ（ｘ１_ｓ，ｘ２_ｓ）は、ｘ座標のうち大きい方の値を選択することを表す。図１２（ｃ）の場合、ｘ１_ｓ＝４０、ｘ２_ｓ＝４５であるため、ｍａｘ（ｘ１_ｓ，ｘ２_ｓ）＝４５となる。

However, max (x1 _s , x2 _s ) represents selecting the larger value of the x coordinates. In the case of FIG. 12C, since x1 _s = 40 and x2 _s = 45, max (x1 _s , x2 _s ) = 45.

Further, min (x1 _e , x2 _e ) represents that the smaller value of the x coordinates is selected. In the case of FIG. 12C, since x1 _e = 50 and x2 _e = 54, min (x1 _e , x2 _e ) = 50.

Similarly, max (y1 _s , y2 _s ) represents selecting the larger value of the y coordinates. In the case of FIG. 12C, since y1 _s = 46 and y2 _s = 35, max (y1 _s , y2 _s ) = 46.

Further, min (y1 _e , y2 _e ) represents that the smaller value of the y coordinates is selected. In the case of FIG. 12C, since y1 _e = 78 and y2 _e = 65, min (y1 _e , y2 _e ) = 65.

In the example of FIG. 12C, the overlap removing unit 503 uses the x coordinate, y coordinate, width, and height of the overlap region 1210 as follows:
X coordinate = max (x1 _s , x2 _s ) = 45,
Y coordinate = max (y1 _s , y2 _s ) = 46,
Width = min (x1 _e , x2 _e ) −max (x1 _s , x2 _s ) = 5,
Height = min (y1 _e , y2 _e ) −max (y1 _s , y2 _s ) = 19
And “45”, “46”, “5”, and “19” are recorded in the “overlap area” of the tracking target information 310, respectively.

(4) Details of Existence Range Determination Processing by Existence Range Determination Unit Next, details of the existence range determination processing (step S605) by the existence range determination unit 504 will be described. FIG. 13 is a flowchart of the existence range determination process executed by the existence range determination unit.

  In step S <b> 1301, the existence range determination unit 504 is a “person area” of a person included in the image data to be processed and a “person” of a person whose “overlap flag” is “1” (that is, a person on the back side). The “region” and the “overlapping region” are read from the tracking target information 310.

  In step S <b> 1302, the existence range determination unit 504 identifies a range in which a person on the back side exists based on the read “person area” and “overlap area”.

  In step S1303, the existence range determination unit 504 scans the range of the person on the back side and the surrounding area using the identified existence range image, and calculates the similarity of the images at each position.

  In step S1304, the existence range determination unit 504 generates a similarity map by generating a grayscale image based on the image similarity at each position calculated in step S1303. Thereby, the effective range for tracking the person on the back side is determined. The existence range determination unit 504 stores the generated similarity map.

  In step S <b> 1305, the existence range determination unit 504 determines whether or not processing has been performed for the person regions of all persons whose “overlap flag” is “1”. If it is determined in step S1305 that there is a person region of a person who has not been processed (Yes in step S1305), the process returns to step S1301.

  On the other hand, if it is determined in step S1305 that processing has been performed for all the person areas of which the “overlap flag” is “1” (in the case of No in step S1305), the existence range determination processing ends.

  FIG. 14 is a diagram schematically showing existence range determination processing executed by the existence range determination unit. In FIG. 14A, the existence range determination unit 504 identifies a range where the person 1020 exists, which is obtained by removing the overlapping area from the person area of the person 1020. In addition, the existence range determination unit 504 extracts an image of the existence range 1401 of the person 1020. Further, the existence range determination unit 504 scans the range of the person 1020 and the surrounding area using the extracted image of the existence range 1401, and calculates the similarity of the images at each position. Note that the existence range determination unit 504 calculates the similarity based on the following equation, for example.

ただし、Ｔは、存在範囲１４０１の画像（各画素の画素値）を表し、Ｉは、処理対象の画像データ（ただし、人物１０２０の範囲及びその周辺の範囲の画像データ）の各位置（ｘ，ｙ）での画像（各画素の画素値）を表す。また、ｈは、存在範囲１４０１の高さを表し、ｗは、存在範囲１４０１の幅を表す。

However, T represents an image (pixel value of each pixel) in the existence range 1401, and I represents each position (x, X) of the image data to be processed (however, the image data in the range of the person 1020 and its surrounding range). y) represents an image (pixel value of each pixel). H represents the height of the existence range 1401, and w represents the width of the existence range 1401.

  FIG. 14B shows the calculation result of the similarity at each position as a gray image 1411. A region with high density in the grayscale image 1411 indicates that the degree of similarity is high, and the feature of the person 1020 appears remarkably. On the other hand, the low density area indicates that the degree of similarity is low and the characteristics of the person 1020 do not appear so much. Therefore, when searching for the same person as the person 1020 in the image data to be processed next, it can be said that it is effective to use a high-density area in the grayscale image 1411.

  The existence range determination unit 504 stores the grayscale image 1411 obtained by calculating the similarity as a similarity map.

(5) Details of Movement Amount Calculation Processing by Movement Amount Calculation Unit Next, details of the movement amount calculation processing (step S606) by the movement amount calculation unit 505 will be described. FIG. 15 is a flowchart of the movement amount calculation process executed by the movement amount calculation unit.

  In step S <b> 1501, the movement amount calculation unit 505 reads the “overlap flag” of the processing target person from the tracking target information 310 of the tracking target information storage unit 133 for the processing target image data.

  In step S1502, the movement amount calculation unit 505 determines whether or not the read “overlap flag” is “1”. If it is determined in step S1502 that the “overlap flag” is not “1” (in the case of No in step S1502), it is determined that the person to be processed is the front person, and the process proceeds to step S1505.

  On the other hand, if it is determined in step S1502 that the “overlap flag” is “1” (in the case of Yes in step S1502), it is determined that the person to be processed is a person on the back side, and step S1503. Proceed to

  If the person to be processed is a person on the back side, in step S1503, the movement amount calculation unit 505 reads the “overlapping relationship” of the persons to be processed from the tracking target information 310 of the tracking target information storage unit 133.

  In step S1504, the movement amount calculation unit 505 determines whether or not the “tracking flag” in the image data to be processed next of the person corresponding to the read “overlap relationship” is “1”.

  If it is determined in step S1504 that the “tracking flag” is “1” (Yes in step S1504), the process proceeds to step S1511.

  On the other hand, if it is determined in step S1504 that the “tracking flag” is not “1” (in the case of No in step S1504), the process proceeds to step S1505.

  In step S1505, the movement amount calculation unit 505 scans the image data of the next processing target using the image of the person region of the person corresponding to the person area of the person to be processed or the “overlapping relationship” of the person to be processed. Then, the similarity is calculated.

  In step S1506, the movement amount calculation unit 505 determines whether the calculated similarity is equal to or greater than a predetermined threshold. If it is determined in step S1506 that the threshold value is equal to or greater than the predetermined threshold (in the case of Yes in step S1506), the process proceeds to step S1507.

  In step S <b> 1507, the movement amount calculation unit 505 determines that the person corresponding to the “overlapping relationship” between the person to be processed or the person to be processed and the person region at the highest similarity in the next image data to be processed. Are related as the same person.

  In step S1508, the movement amount calculation unit 505 sets the “tracking flag” in the next processing target image data of the person corresponding to the processing target person or the “overlapping relationship” of the processing target person to “1”.

  In step S <b> 1509, the movement amount calculation unit 505 calculates the movement amount (movement amount between the processing target image data and the next processing target image data) of the person area associated as the same person. Further, the movement amount calculation unit 505 records the calculated movement amount in the tracking target information 310 in association with the person to be processed or the person corresponding to the “overlapping relationship” of the person to be processed. At this time, the movement amount calculation unit 505 records the calculated movement amount in association with the frame number of the next image data to be processed.

  On the other hand, if it is determined in step S1506 that it is not equal to or greater than the predetermined threshold (in the case of No in step S1506), the process proceeds to step S1510.

  In step S1510, the movement amount calculation unit 505 sets “tracking flag” in the next processing target image data of the person corresponding to the processing target person or the “overlapping relationship” of the processing target person to “0”.

  In step S1511, the movement amount calculation unit 505 determines whether or not processing has been performed for all persons included in the image data to be processed. If it is determined in step S1511 that there is a person who has not performed the process (Yes in step S1511), the process returns to step S1501.

  On the other hand, if it is determined in step S1511 that the process has been performed for all persons (No in step S1511), the movement amount calculation process ends.

  FIG. 16 is a diagram schematically illustrating movement amount calculation processing executed by the movement amount calculation unit. Image data 1610 in FIG. 16A is image data to be processed (frame number = “239”), and image data 1620 in FIG. 16B is image data to be processed next (frame number = “239”). 240 ").

  In the image data 1610 to be processed, a person area 1611 is a person area of a person whose “person ID” is “3”, and a person area 1612 is a person area of a person whose “person ID” is “1”. is there. Furthermore, the person 1613 is a person whose “person ID” is “2”.

  Here, it is assumed that the movement amount calculation unit 505 performs the movement amount calculation process for the person (person ID = “3”) in the person area 1611 in the image data 1610 to be processed. In the case of a person in the person area 1611 (person ID = “3”), since the “overlap flag” is “0”, the movement amount calculation unit 505 uses the person area 1611 to perform image data 1620 to be processed next. , And the similarity of the images at each position is calculated. The movement amount calculation unit 505 uses the following similarity, for example, when scanning the next processing target image data 1620.

ただし、Ｔは、画像データ１６１０の人物領域１６１１の画像（各画素の画素値）を表し、Ｉは、次の処理対象の画像データ１６２０の各位置（ｘ，ｙ）での画像（各画素の画素値）を表す。また、ｈは、画像データ１６１０の人物領域１６１１の高さを表し、ｗは人物領域１６１１の幅を表す。

However, T represents the image (pixel value of each pixel) of the person region 1611 of the image data 1610, and I represents the image (each pixel of each pixel) at each position (x, y) of the image data 1620 to be processed next. Pixel value). Further, h represents the height of the person area 1611 of the image data 1610, and w represents the width of the person area 1611.

  FIG. 16B shows that the highest similarity is obtained at the position of the area 1621 as a result of scanning the image data 1620 (frame number = “240”) to be processed next using the person area 1611. Show. The movement amount calculation unit 505 determines that the person included in the region 1621 is the same person as the person (person ID = “3”) detected in the processing target image data 1610 (frame number = “239”). .

  Then, the movement amount calculation unit 505 records the x-coordinate, y-coordinate, width, and height of the area 1621 in the “person area” whose “person ID” is “3” and “frame number” is “240”. (See FIG. 16D). Further, the movement amount calculation unit 505 records “1” in the “tracking flag” in which “person ID” is “3” and “frame number” is “240” (see FIG. 16D). Further, the movement amount calculation unit 505 sets the difference in position between the region 1621 and the person region 1611 (difference in the x direction) to the “movement amount” where the “person ID” is “3” and the “frame number” is “240”. , Y direction difference) is recorded (see FIG. 16D). The “movement amount” of the person whose “person ID” is “3” can be represented by an arrow 1631 in FIG.

  Subsequently, in the image data 1610 to be processed, it is assumed that the movement amount calculation unit 505 performs the movement amount calculation processing for the person in the person area 1612 (“person ID” is “1”). The movement amount calculation unit 505 performs the same processing for the person in the person area 1612 as in the case of the person in the person area 1611.

  That is, since the “overlap flag” is “0” even in the case of the person in the person area 1612 (“person ID” is “1”), the movement amount calculation unit 505 uses the person area 1612 to perform the next process. The target image data 1620 is scanned, and the similarity of images at each position is calculated.

  In FIG. 16B, the image data 1620 to be processed next (“frame number” is “240”) is scanned using the person area 1612, and the highest similarity is obtained at the position of the area 1622. It shows that. The movement amount calculation unit 505 is the same person as the person ("person ID" is "1") detected in the processing target image data 1610 ("frame number" is "239"). Judge that there is.

  Then, the movement amount calculation unit 505 records the x-coordinate, y-coordinate, width, and height of the area 1622 in the “person area” whose “person ID” is “1” and “frame number” is “240”. (See FIG. 16D). Further, the movement amount calculation unit 505 records “1” in the “tracking flag” in which “person ID” is “1” and “frame number” is “240” (see FIG. 16D). Further, the movement amount calculation unit 505 sets the difference between the positions of the region 1622 and the person region 1612 (difference in the x direction) to the “movement amount” where the “person ID” is “1” and the “frame number” is “240”. , Y direction difference) is recorded (see FIG. 16D). The “movement amount” of the person whose “person ID” is “1” can be represented by an arrow 1632 in FIG.

  Subsequently, it is assumed that the movement amount calculation unit 505 performs a movement amount calculation process on the person 1613 in the image data 1610 to be processed. In the case of the person 1613 (“person ID” is “2”), since the “overlap flag” is “1”, the movement amount calculation unit 505 reads “overlap relationship”.

  Here, the “overlapping relationship” of the person 1613 (“person ID” is “2”) is “3”, and the “person ID” in the next processing target image data 1620 (“frame number” is “240”). The “tracking flag” of “3” is already “1”.

  For this reason, the movement amount calculation unit 505 determines the person corresponding to the “overlapping relationship” of the person 1613 to be processed (the person whose “person ID” is “3”) in the image data 1620 to be processed next. Do not scan based on region.

  Also, the movement amount calculation unit 505 does not perform scanning based on the person area in the image data 1620 to be processed next for the person 1613 to be processed (for the person 1613, instead of the characteristic area calculation process and search). Process).

(6) Details of Feature Area Calculation Process by Feature Area Calculation Unit Next, details of the feature area calculation process (step S608) by the feature area calculation unit 506 will be described. FIG. 17 is a flowchart of the feature area calculation process executed by the feature area calculation unit.

  In step S <b> 1701, the feature area calculation unit 506 reads the “overlap flag” and “tracking flag” of the person to be processed from the tracking target information 310 of the tracking target information storage unit 133 in the processing target image data.

  In step S1702, the feature region calculation unit 506 determines whether the read “overlap flag” is “1” and the “tracking flag” is “0”. If it is determined in step S1702 that the read “overlap flag” is not “1”, or if it is determined that the “tracking flag” is not “0” (in the case of No in step S1702), step S1708 is performed. Proceed to

  On the other hand, if it is determined in step S1702 that the read “overlap flag” is “1” and the “tracking flag” is “0” (Yes in step S1702), the process proceeds to step S1703. .

  In step S <b> 1703, the feature region calculation unit 506 reads the “overlapping relationship” of the person to be processed from the tracking target information 310 of the tracking target information storage unit 133.

  In step S 1704, the feature area calculation unit 506 reads the “movement amount” of the person corresponding to the read “overlap relationship” from the tracking target information 310 of the tracking target information storage unit 133. At this time, the feature area calculation unit 506 records “movement amount” recorded in association with the frame number of the image data to be processed and “frame amount” recorded in association with the frame number of the next image data to be processed. Read “Movement amount”.

  In step S1705, the feature area calculation unit 506 calculates a change in the movement amount of the person corresponding to the “overlapping relationship” of the person to be processed. Specifically, the feature area calculation unit 506 calculates a change in the movement amount by calculating a difference between the two movement amounts read in step S1704.

  In step S1706, the feature area calculation unit 506 estimates a change in the overlapping area in the next image data to be processed based on the change in the movement amount. Also, the feature area calculation unit 506 estimates the existence range of the person to be processed in the next image data to be processed based on the estimated overlapping area after the change.

  In step S1707, the feature area calculation unit 506 calculates the area of the image data to be processed according to the estimated range as a feature area used when searching for a person on the back side.

  In step S1708, the feature area calculation unit 506 determines whether or not processing has been performed for all persons included in the image data to be processed. If it is determined in step S1708 that there is a person who has not performed processing (Yes in step S1708), the process returns to step S1701.

  On the other hand, if it is determined in step S1708 that processing has been performed for all persons (No in step S1708), the feature area calculation processing ends.

  FIG. 18 is a first diagram schematically showing the feature region calculation process executed by the feature region calculation unit. FIG. 18A shows the relationship between the person 1802 and the person 1801 that accompanies a change in the amount of movement in the y direction of the person 1801 (the person on the near side) that is in an overlapping relationship with the person 1802 (the person on the back). The change in the positional relationship in the y direction is shown. FIG. 18B shows a change in the overlapping area 1805 accompanying a change in the positional relationship in the y direction between the person 1802 and the person 1801.

  The feature region calculation unit 506 estimates a change in the overlapping region of the far-side person 1802 in the image data to be processed based on the change in the movement amount of the near-side person 1801 in the y direction. Thereby, the feature region calculation unit 506 estimates the existence range in the next processing target image data of the person 1802 determined to have an overlap. Also, the feature area calculation unit 506 calculates the area of the image data to be processed according to the estimated range as a feature area used when searching for a person on the back side.

  In the image data to be processed, it is assumed that the person 1801 and the person 1802 have a positional relationship indicated by reference numeral 1800 in FIG. Further, it is assumed that the overlapping area of the person 1802 at that time with respect to the person 1801 has the size indicated by the overlapping area 1805.

  In such a case, in the image data to be processed next, the positional relationship between the person 1801 and the person 1802 in the y direction may change to any of the positional relationships indicated by reference numerals 1820, 1830, and 1840.

The positional relationship indicated by reference numeral 1820 indicates that the movement amount (Δy _t ) of the person 1801 calculated in the processing target image data is calculated in the y direction of the person 1801 calculated in the next processing target image data. The positional relationship is shown when the amount of movement is smaller than (Δy _{t + 1} ).

Note that Δy _t represents the amount of movement in the y direction of the person 1801 calculated in the image data to be processed. Specifically, Δy _t is the difference between the y coordinate of the person area of the person 1801 in the image data to be processed and the y coordinate of the person area of the person 1801 in the image data immediately before the image data to be processed. Calculated.

Δy _{t + 1} represents the amount of movement of the person 1801 in the y direction calculated in the next processing target image data. Specifically, Δy _{t + 1} is calculated from the difference between the y coordinate of the person region of the person 1801 in the next processing target image data and the y coordinate of the person region of the person 1801 in the processing target image data.

When Δy _{t + 1} is smaller than Δy _t , the overlapping region 1825 in the person 1802 is smaller in the y direction than the overlapping region 1805.

Incidentally, the overlap 'When _s, oy' the minimum value of y coordinate of the area 1825 oy _s = oy _s - can be represented as _{(Δy t -Δy t + 1)} ( however, oy _s is, y coordinates of the overlap region 1805 Represents the minimum value).

The positional relationship indicated by reference numeral 1830 indicates that the movement amount (Δy _t ) of the person 1801 calculated in the image data to be processed is calculated in the y direction of the person 1801 calculated in the next image data to be processed. The positional relationship in the same case as the movement amount (Δy _{t + 1} ) is shown. When Δy _{t + 1} is the same as Δy _t , the overlapping area 1835 in the person 1802 is equal to the overlapping area 1805. That is, oy ' _s = oy _s .

The positional relationship indicated by reference numeral 1840 indicates that the movement amount (Δy _t ) of the person 1801 calculated in the image data to be processed is calculated in the y direction of the person 1801 calculated in the next image data to be processed. The positional relationship in the case where it is larger than the movement amount (Δy _{t + 1} ) is shown. When Δy _{t + 1} is larger than Δy _t , the overlapping area 1845 in the person 1802 is larger in the y direction than the overlapping area 1805.

Incidentally, the overlap 'When _s, oy' the minimum value of y coordinate of the area 1845 oy _s = oy _s - can be represented as _{(Δy t -Δy t + 1)} .

  In this way, the feature area calculation unit 506 estimates the change in the overlapping area in the y direction of the back person in the next image data to be processed based on the change in the movement amount in the y direction of the near person. . Thereby, the feature area calculation unit 506 can estimate the existence range (y direction) of the person on the back side in the image data to be processed next. Further, the feature area calculation unit 506 can calculate an area in the image data to be processed corresponding to the estimated existence range (y direction) as the feature area.

FIG. 19 is a second diagram schematically illustrating the feature area calculation process executed by the feature area calculation unit. FIG. 19A shows the relationship between the person 1902 and the person 1902 accompanying the change in the amount of movement in the x direction of the person 1901 (the person on the near side) that is in an overlapping relationship with the person 1902 to be processed (the person on the back side). The change in the positional relationship in the x direction is shown. However, FIG. 19 (a) shows a change in positional relationship when the pxf _t ≧ pxb _t. Note that the pxf _t, x coordinate of the person area of the person 1901 in the image data to be processed, the pxb _t, represents the x-coordinate of the person area of the person 1902 in the image data to be processed.

FIG. 19 (b), due to a change in the x-direction positional relationship between the person 1902 and the person 1901, a change in the overlap area, indicating a change in the overlap region when the pxf _t ≧ pxb _t .

  In the image data to be processed, it is assumed that the person 1901 and the person 1902 are in a positional relationship indicated by reference numeral 1900 in FIG. Further, it is assumed that the overlapping area of the person 1902 with respect to the person 1901 at that time is the size indicated by the overlapping area 1905.

  In such a case, in the image data to be processed next, the positional relationship between the person 1901 and the person 1902 may change to any of the positional relationships indicated by reference numerals 1920, 1930, and 1940.

The positional relationship indicated by reference numeral 1920 indicates that the movement amount (Δx _t ) of the person 1901 in the x direction calculated in the processing target image data is calculated in the next processing target image data in the x direction of the person 1901. The positional relationship in the case where it is larger than the movement amount (Δx _{t + 1} ) is shown.

The amount of movement (Δx _t ) of the person 1901 in the x direction is the x coordinate of the person area of the person 1901 in the image data to be processed and the x of the person area of the person 1901 in the image data immediately before the image data to be processed. Calculated based on the difference from the coordinates.

In addition, the movement amount (Δx _{t + 1} ) of the person 1901 in the x direction is the difference between the x coordinate of the person area of the person 1901 in the next image data to be processed and the x coordinate of the person area of the person 1901 in the image data to be processed. Calculated by the difference.

When Δx _{t + 1} is larger than Δx _t , the overlapping area 1925 in the person 1902 is smaller in the x direction than the overlapping area 1905.

Note that if the minimum value of the x coordinate of the overlapping area 1925 is ox ′ _s , it can be expressed as ox ′ _s = ox _s − (Δx _t −Δx _{t + 1} ) (where ox _s is the x coordinate of the overlapping area 1905). Represents the minimum value).

The positional relationship indicated by reference numeral 1930 indicates that the movement amount (Δx _t ) of the person 1901 in the x-direction calculated in the processing target image data is calculated in the next processing target image data in the x-direction of the person 1902. The positional relationship in the case where it is the same as the movement amount (Δx _{t + 1} ) is shown. When Δx _{t + 1} is the same as Δx _t , the overlapping area 1935 in the person 1902 is equal to the overlapping area 1905. That is, ox ′ _s = ox _s .

The positional relationship indicated by reference numeral 1940 indicates that the movement amount (Δx _t ) of the person 1901 in the x direction calculated in the processing target image data is calculated in the next processing target image data in the x direction of the person 1902. The positional relationship is shown when it is less than the movement amount (Δx _{t + 1} ). When Δx _{t + 1} is smaller than Δx _t , the overlapping area 1945 in the person 1902 becomes larger in the x direction than the overlapping area 1905.

Incidentally, the overlap 'When _s, ox' the minimum value of x coordinate of the area 1945 ox _s = ox _s - can be represented as _{(Δx t -Δx t + 1)} .

Similarly, FIG. 19C shows the change in the positional relationship in the x direction between the person 1902 and the person 1901 accompanying the change in the movement amount in the x direction of the person 1901 that is in an overlapping relationship with the person 1902 to be processed. The change in the positional relationship in the case of pxf _t <pxb _t is shown. FIG. 19D shows a change in the overlap region in accordance with a change in the positional relationship in the x direction between the person 1902 and the person 1901, and shows a change in the overlap region when pxf _t <pxb _t . .

  In the image data to be processed, it is assumed that the person 1901 and the person 1902 are in a positional relationship indicated by reference numeral 1950 in FIG. In addition, it is assumed that the overlapping area of the person 1902 with respect to the person 1901 at that time is the size indicated by the overlapping area 1955.

  In such a case, in the image data to be processed next, the positional relationship between the person 1901 and the person 1902 may change to any of the positional relationships indicated by reference numerals 1970, 1980, and 1990.

The positional relationship indicated by reference numeral 1970 indicates that the movement amount (Δx _t ) of the person 1901 in the x-direction calculated in the processing target image data is calculated in the x-direction of the person 1901 calculated in the next processing target image data. The positional relationship is shown when it is less than the movement amount (Δx _{t + 1} ).

The amount of movement (Δx _t ) of the person 1901 in the x direction is the x coordinate of the person area of the person 1901 in the image data to be processed and the x of the person area of the person 1901 in the image data immediately before the image data to be processed. Calculated based on the difference from the coordinates.

In addition, the movement amount (Δx _{t + 1} ) of the person 1901 in the x direction is the difference between the x coordinate of the person area of the person 1901 in the next image data to be processed and the x coordinate of the person area of the person 1901 in the image data to be processed. Calculated by the difference.

When Δx _{t + 1} is smaller than Δx _t , the overlapping region 1975 in the person 1902 is smaller in the x direction than the overlapping region 1955.

If the maximum value of the x coordinate of the overlapping region 1975 is ox ′ _e , it can be expressed as ox ′ _e = ox _e − (Δx _t −Δx _{t + 1} ) (where ox _e is the x coordinate of the overlapping region 1955). Represents the maximum value of

The positional relationship indicated by reference numeral 1980 indicates that the movement amount (Δx _t ) of the person 1901 in the x-direction calculated in the image data to be processed is calculated in the x-direction of the person 1902 in the image data to be processed next. The positional relationship in the case where it is the same as the movement amount (Δx _{t + 1} ) is shown. When Δx _{t + 1} is the same as Δx _t , the overlapping area 1985 in the person 1902 is equal to the overlapping area 1955. That is, ox ′ _e = ox _e .

The positional relationship indicated by reference numeral 1990 indicates that the movement amount (Δx _t ) of the person 1901 in the x direction calculated in the processing target image data is calculated in the next processing target image data in the x direction of the person 1902. The positional relationship in the case where it is larger than the movement amount (Δx _{t + 1} ) is shown. When Δx _{t + 1} is larger than Δx _t , the overlapping area 1985 in the person 1902 is larger in the x direction than the overlapping area 1955.

If the maximum value of the x coordinate of the overlapping area 1995 is ox ′ _e , it can be expressed as ox ′ _e = ox _e − (Δx _t −Δx _{t + 1} ).

  In this way, the feature area calculation unit 506 estimates the change in the overlapping area in the x direction of the far-side person in the image data to be processed next, based on the change in the movement amount in the x direction of the person on the near side. . Thereby, the feature region calculation unit 506 can estimate the existence range (x direction) of the person on the back side in the image data to be processed next. In addition, the feature region calculation unit 506 can calculate a region of image data to be processed according to the estimated existence range (x direction) as a feature region.

(7) Details of Search Process by Search Unit Next, details of the search process (step S609) by the search unit 507 will be described. FIG. 20 is a flowchart of search processing executed by the search unit.

  In step S2001, the search unit 507 reads “overlap flag” and “tracking flag” of the person to be processed from the tracking target information 310 in the image data to be processed.

  In step S2002, the search unit 507 determines whether the read “overlap flag” is “1” and the “tracking flag” is “0”. If it is determined in step S2002 that the read “overlap flag” is not “1”, or if it is determined that the “tracking flag” is not “0” (in the case of No in step S2002), step S2008 is performed. Proceed to

  On the other hand, if it is determined in step S2002 that the read “overlap flag” is “1” and the “tracking flag” is “0” (Yes in step S2002), the process proceeds to step S2003. .

  In step S2003, the search unit 507 reads the similarity map for the processing target person in the processing target image data stored by the existence range determination unit 504.

  In step S2004, the search unit 507 reads the “overlapping relationship” of the person to be processed from the tracking target information 310, and reads the “person area” of the person corresponding to the read “overlapping relationship” from the tracking target information 310. .

  In step S <b> 2005, the search unit 507 reads “movement amount” in the image data of the next processing target of the person corresponding to the read “overlap relationship” from the tracking target information 310.

  In step S2006, the search unit 507 moves the similarity map read in step S2003 based on the movement amount read in step S2005, and sets a search area in the next processing target image data.

  In step S2007, the search unit 507 searches the set search area for the person to be processed using the image of the feature area calculated by the feature area calculation unit 506, and determines whether or not the same person as the person to be processed exists. judge. If the search unit 507 detects the same person as the person to be processed as a result of the search, the search unit 507 sets the person area of the detected person as the person area in the next process target image data in the tracking target information 310. Record. Further, the search unit 507 records “1” in the “tracking flag” corresponding to the “person area” in the image data to be processed next in the tracking target information 310.

  In step S2008, the search unit 507 determines whether or not processing has been performed for all persons included in the image data to be processed. If it is determined in step S2008 that there is a person who has not performed processing (Yes in step S2008), the process returns to step S2001.

  On the other hand, if it is determined in step S2008 that the process has been performed for all persons (No in step S2008), the search process ends.

  FIG. 21 is a diagram schematically illustrating search processing executed by the search unit. In FIG. 21, the similarity map 2110 indicates a similarity map of the processing target person 2101 (back person) in the processing target image data 2100 read by the search unit 507. Further, the image data 2120 indicates the amount of movement of the person 2121 (the person on the near side) in the next image data to be processed.

  The image data 2130 shows a state in which the search unit 507 moves the similarity map 2110 to a position corresponding to the amount of movement of the front person 2121 in the next image data to be processed.

  Further, the image data 2140 shows a state in which the search unit 507 sets the search area 2142 in the next image data to be processed based on the similarity map after movement.

  The search unit 507 sets an area (x, y) that satisfies the following conditions as the search area 2142.

ただし、ｓ_ｘｍｉｎ、ｓ_ｘｍａｘ、ｓ_ｙｍｉｎ、ｓ_ｙｍａｘは、処理対象の画像データ２１００における類似度マップのｘ座標の最小値、最大値、ｙ座標の最小値、最大値をそれぞれを表している。また、Δｘ、Δｙは、次の処理対象の画像データにおける、手前側の人物２１２１の移動量を表している。また、ｓｉｍ［ｘ，ｙ］は、座標（ｘ，ｙ）における類似度を表し、ｔｈは閾値を表している。更に、ｐｙは、次の処理対象の画像データにおける、手前側の人物２１２１の人物領域のｙ座標を表している。

However, s _xmin , s _xmax , s _ymin , and s _ymax represent the minimum value and maximum value of the x coordinate, the minimum value and maximum value of the y coordinate of the similarity map in the image data 2100 to be processed, respectively. Δx and Δy represent the amount of movement of the person 2121 on the near side in the next image data to be processed. Further, sim [x, y] represents the similarity at the coordinates (x, y), and th represents the threshold value. Further, py represents the y coordinate of the person area of the front person 2121 in the next image data to be processed.

  As a result, for the person 2101 to be processed, the search area 2142 corresponding to the similarity map 2110 after movement can be set in the image data 2140 to be processed next. When the search area 2142 is set by the search unit 507, the person detection unit 501 searches the set search area 2142 using the image of the characteristic area, and is the same person as the person 2101 to be processed. The presence or absence of is determined. In the image data 2140, a person 2141 indicates a person who is determined to be the same person as the person 2101 in the set search area 2142.

  FIG. 22 is a diagram illustrating an example of tracking target information after the search processing. As shown in FIG. 22, by performing the search process, the person area (person ID “2”) of the person 2141 (“person ID” is “2”) is included in the “person area” of frame number = “240” of the tracking target information 310. x coordinate, y coordinate, width, height) are recorded.

  As is clear from the above description, the person tracking device 130 detects a person in a plurality of images, and whether or not there is an overlap between the person on the near side and the person on the far side in each image. Determine. Further, when an overlap occurs, the person tracking device 130 generates a similarity map based on an image of a non-overlapping area (existing range) among the person areas of the person on the back side. Determine the effective range for tracking people. Further, the person tracking device 130 calculates a change in an area (existing range) where there is no overlap based on the tracking result obtained by tracking the person on the near side, and calculates the changed range as a feature area. In addition, the person tracking device 130 tracks the back person by moving a range (similarity map) effective for tracking the back person based on the tracking result of the front person. Set the search area. Furthermore, the person tracking device 130 determines the presence or absence of a person on the back side by searching the set search area using the image of the feature area.

  As described above, the person tracking device 130 sets an appropriate search area and calculates an appropriate feature area using the tracking result of the person on the near side even when an overlap occurs, and calculates the calculated feature. A person is searched using the image of the area.

  Thereby, according to person tracking device 130, even when overlap has occurred, the same person can be detected accurately. That is, according to the person tracking device 130, high tracking accuracy can be obtained in person tracking.

[Other Embodiments]
In the above embodiment, the moving image data captured by the imaging device 110 is described as being stored in the moving image data storage device 120. However, the moving image data captured by the imaging device 110 is stored in the person tracking device 130. May be.

  In the above embodiment, the person tracking device 130 transmits the tracking target information to the analysis device 140 and the analysis device 140 analyzes the tracking target information. However, the tracking target information is analyzed by the person tracking device 130. May be.

  In the above embodiment, no particular reference has been made to the method of calculating the similarity when searching for the search region using the image of the feature region of the person on the back side. For example, the similarity when generating the similarity map is described. A calculation method similar to that of the degree is used.

  In the above-described embodiment, the case where the image of the existing range (pixel value of each pixel) is used to generate the similarity map has been described. However, the image feature amount calculated based on the image of the existing range is used. Also good. Similarly, in the above-described embodiment, the case where the image of the person area of the person on the front side (pixel value of each pixel) is used to search for the same person as the person on the front side is described. An image feature amount calculated based on the image of the region may be used.

  In the above embodiment, the object to be tracked is described as a person, but the object to be tracked is not limited to a person, and may be other than a person (for example, a vehicle). That is, the person tracking device, the person tracking program, and the person tracking system described in the above embodiments are examples of the object tracking device, the object tracking program, and the object tracking system, respectively.

In addition, in the disclosed technology, forms such as the following supplementary notes are conceivable.
(Appendix 1)
An object is detected from each of the second images captured at different times of the first image and the first image;
Determining whether or not there is an overlap between the first object located in front of the first image and the second object located on the back side in the first image;
When overlapping has occurred, based on the image of the area from which the overlapping area has been removed, a range to be used for determining the presence or absence of the second object is determined,
A search area for tracking the second object in the second image is set by moving the range based on the tracking result of the first object, and the second image Determining the presence or absence of the second object by searching the search area;
An object tracking program for causing a computer to execute processing.
(Appendix 2)
Based on the tracking result obtained by tracking the first object, calculate the area after the change when the area where the overlapping area is removed,
The object according to claim 1, wherein the presence or absence of the second object is determined by searching the search area for the second image using an image of the area after the change. Tracking program.
(Appendix 3)
By scanning the range of the second object and the peripheral area of the second object, and calculating the similarity with the image of the area from which the overlapping area is removed at each position, the second object 3. The object tracking program according to appendix 2, wherein a range used for determining the presence or absence of the object is determined.
(Appendix 4)
Based on the tracking result obtained by tracking the first object, a change in the amount of movement of the first object is calculated,
The supplementary note is characterized in that a change in the area from which the overlap area is removed is estimated from a change in the overlap area according to a change in the calculated movement amount of the first object, and the area after the change is calculated. 3. The object tracking program according to 3.
(Appendix 5)
Computer
An object is detected from each of the second images captured at different times of the first image and the first image;
Determining whether or not there is an overlap between the first object located in front of the first image and the second object located on the back side in the first image;
When overlapping has occurred, based on the image of the area from which the overlapping area has been removed, a range to be used for determining the presence or absence of the second object is determined,
A search area for tracking the second object in the second image is set by moving the range based on the tracking result of the first object, and the second image Determining the presence or absence of the second object by searching the search area;
An object tracking method characterized by executing processing.
(Appendix 6)
A detection unit that detects an object from each of the second images captured at different times of the first image and the first image;
A determination unit that determines whether or not there is an overlap between the first object located in front of the first image and the second object located on the far side in the first image;
A determination unit that determines a range to be used for the determination of the presence or absence of the second object based on an image of a region from which the overlap region is removed when an overlap occurs;
A search area for tracking the second object in the second image is set by moving the range based on the tracking result of the first object, and the second image An object tracking apparatus comprising: a search unit that determines the presence or absence of the second object by searching the search area.

  Note that the present invention is not limited to the configurations shown here, such as combinations with other elements, etc., in the configurations described in the above embodiments. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form.

100: person tracking system 110: imaging device 120: moving image data storage device 130: person tracking device 131: person tracking unit 132: tracking target information output unit 133: tracking target information storage unit 140: analysis device 310: tracking target information 501 : Person detection unit 502: overlap determination unit 503: overlap removal unit 504: existence range determination unit 505: movement amount calculation unit 506: feature region calculation unit 507: search unit 1210: overlap region 1401: existence range 1411: gray image 2110: Similarity map 2142: search area

Claims (6)

An object is detected from each of the second images captured at different times of the first image and the first image;
Determining whether or not there is an overlap between the first object located in front of the first image and the second object located on the back side in the first image;
When overlapping has occurred, based on the image of the area from which the overlapping area has been removed, a range to be used for determining the presence or absence of the second object is determined,
A search area for tracking the second object in the second image is set by moving the range based on the tracking result of the first object, and the second image Determining the presence or absence of the second object by searching the search area;
An object tracking program for causing a computer to execute processing. Based on the tracking result obtained by tracking the first object, calculate the area after the change when the area where the overlapping area is removed,
2. The object according to claim 1, wherein the presence or absence of the second object is determined by searching the search area for the second image using an image of the area after the change. Object tracking program.   By scanning the range of the second object and the peripheral area of the second object, and calculating the similarity with the image of the area from which the overlapping area is removed at each position, the second object The object tracking program according to claim 2, wherein a range to be used for determining the presence or absence of the object is determined. Based on the tracking result obtained by tracking the first object, a change in the amount of movement of the first object is calculated,
The area after removing the overlap area is estimated from the change in the overlap area according to the change in the calculated movement amount of the first object, and the area after the change is calculated. Item tracking program according to Item 3. Computer
An object is detected from each of the second images captured at different times of the first image and the first image;
Determining whether or not there is an overlap between the first object located in front of the first image and the second object located on the back side in the first image;
When overlapping has occurred, based on the image of the area from which the overlapping area has been removed, a range to be used for determining the presence or absence of the second object is determined,
A search area for tracking the second object in the second image is set by moving the range based on the tracking result of the first object, and the second image Determining the presence or absence of the second object by searching the search area;
An object tracking method characterized by executing processing. A detection unit that detects an object from each of the second images captured at different times of the first image and the first image;
A determination unit that determines whether or not there is an overlap between the first object located in front of the first image and the second object located on the far side in the first image;
A determination unit that determines a range to be used for the determination of the presence or absence of the second object based on an image of a region from which the overlap region is removed when an overlap occurs;
A search area for tracking the second object in the second image is set by moving the range based on the tracking result of the first object, and the second image An object tracking apparatus comprising: a search unit that determines the presence or absence of the second object by searching the search area.

Images