JP2015069452A - Mobile object information management method, apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a device for associating customer attribute information with customer flow information.SOLUTION: A method of associating attribute information on a person who analyzes photographic information with flow information on a person who analyzes measuring information of a position measuring device, includes: a first step of calculating an estimation point e1 at which a person is present in a real space from a y coordinate value y1 of a center position of a face in an image at a time t1 included in an attribute having a specific attribute identifier; a second step of calculating an estimation point e2 at which a person is present in the real space from a y coordinate value y2 of a center position of a face of an image at a time t2 included in an attribute having a specific attribute identifier; and a third step calculating, as a score, a sum of a distance between a position and the estimation point e1 in the real space at the nearest time to the time t1 for every flow identifier in the flow information and a distance between a position and the estimation point e2 in the real space at the nearest time to the time t2 for every flow identifier in the flow information, and associating the flow identifier having the lowest score with a specific attribute identifier.

Description

  The present invention relates to a method, apparatus, and program for associating customer movement information (flow lines) with customer attributes (age / gender).

  In recent years, with the lower prices of various sensors and the evolution of networks, we installed cameras and position measurement devices in stores to analyze information about how customers traveled in stores and store sales. Some measures are being taken to improve it. In particular, it is important to understand and analyze how customers with different attributes (age / gender) are moving in the store (flow line) in order to improve the store.

For example, the following Non-Patent Document 1 discloses a method of detecting a face reflected in a camera image and determining a person's attributes (age / gender). By photographing the entrance of the store with a camera and using the technique of Non-Patent Document 1, it becomes possible to analyze the attributes of the customers who entered the store.
Non-Patent Document 2 discloses an apparatus for measuring the distance to an object by scanning the periphery with a laser beam. By installing the device in the store, information on how customers are traveling around the store can be obtained. Based on this information, it is possible to analyze whether the customer is traveling along the route intended by the store (flow line) or the like.
Patent Document 1 below discloses a system that reads attribute information recorded on a card (for example, a point card) owned by a customer when entering a store and associates the read attribute information with a flow line. Has been.

Site Sensing Co., Face Grapher, http://site-sensing.com Hokuyo Electric Co., Ltd., scanner range sensor (range sensor), http://www.hokuyo-aut.co.jp/02senser/#scanner

JP 2012-73915 A

  However, Patent Document 1 is based on the premise that a customer owns a card, and cannot be applied to a customer who does not have a point card or a store without a point card. Further, Non-Patent Document 1 and Non-Patent Document 2 are apparatuses for acquiring customer attributes and flow lines, respectively, and do not disclose specific methods for associating attributes and flow lines.

  The present invention has been made in view of such a situation, and an object thereof is to provide a method and an apparatus for associating customer attribute information acquired by a camera with flow line information acquired by a range sensor. is there.

In order to solve the above-mentioned problems, a mobile object information management method according to the present invention is obtained by analyzing person attribute information obtained by analyzing information photographed by a camera and information measured by a position measuring device. The moving body information management device associates the flow line information of a person to be obtained,
The mobile information management device is
A first step of calculating an estimated point e1 where a person exists in real space from the y coordinate value y1 of the center position of the face in the camera image at time t1 included in the attribute having the specific attribute identifier in the attribute information. When,
A second estimation point e2 where a person exists in real space is calculated from the y coordinate value y2 of the center position of the face in the camera image at time t2 included in the attribute having the specific attribute identifier in the attribute information. Steps,
For each flow line identifier in the flow line information, the distance between the position in the real space at the time closest to the time t1 and the estimated point e1, and for each flow line identifier in the flow line information, A sum of the distance between the position in the real space at the time closest to time t2 and the estimated point e2 is used as a score, and a flow line identifier that minimizes the score is associated with the specific attribute identifier. And a step.
In addition, a range sensor is used as the position measuring device.
Further, in the first step and the second step, in addition to the y coordinate value y1 and the y coordinate value y2, information on the average height estimated from the gender and age of the person is added to estimate the estimated point e1. The point e2 is calculated.

The mobile body information management device according to the present invention includes person attribute information obtained by analyzing information photographed by a camera, and person flow information obtained by analyzing information measured by a position measurement device. A matching device,
First means for calculating an estimated point e1 where a person exists in real space from the y coordinate value y1 of the center position of the face in the camera image at time t1 included in an attribute having a specific attribute identifier in the attribute information When,
A second estimation point e2 where a person exists in real space is calculated from the y coordinate value y2 of the center position of the face in the camera image at time t2 included in the attribute having the specific attribute identifier in the attribute information. Means,
For each flow line identifier in the flow line information, the distance between the position in the real space at the time closest to the time t1 and the estimated point e1, and for each flow line identifier in the flow line information, A sum of the distance between the position in the real space at the time closest to time t2 and the estimated point e2 is used as a score, and a flow line identifier that minimizes the score is associated with the specific attribute identifier. And means.
In addition, a range sensor is used as the position measuring device.
Further, in the first means and the second means, in addition to the y coordinate value y1 and the y coordinate value y2, information on the average height estimated from the gender and age of the person is added to estimate the estimated point e1. The point e2 is calculated.
In addition, a range sensor is used as the position measuring device.
Further, in the first means and the second means, in addition to the y coordinate value y1 and the y coordinate value y2, information on the average height estimated from the gender and age of the person is added to estimate the estimated point e1. The point e2 is calculated.

A program according to the present invention relates to a moving object that associates attribute information of a person obtained by analyzing information photographed by a camera with flow line information of a person obtained by analyzing information measured by a position measuring device. A program used in information management,
The mobile information management
First means for calculating an estimated point e1 where a person exists in real space from the y coordinate value y1 of the center position of the face in the camera image at time t1 included in an attribute having a specific attribute identifier in the attribute information When,
A second estimation point e2 where a person exists in real space is calculated from the y coordinate value y2 of the center position of the face in the camera image at time t2 included in the attribute having the specific attribute identifier in the attribute information. Means,
For each flow line identifier in the flow line information, the distance between the position in the real space at the time closest to the time t1 and the estimated point e1, and for each flow line identifier in the flow line information, A sum of the distance between the position in the real space at the time closest to time t2 and the estimated point e2 is used as a score, and a flow line identifier that minimizes the score is associated with the specific attribute identifier. A program characterized by functioning as a means.

  According to the present invention, it becomes possible to associate customer attribute information acquired by a camera with flow line information acquired by a range sensor, and what kind of attribute a customer has and how to travel around a store I can grasp. Furthermore, measures such as store sales improvement such as store layout change can be implemented based on this information.

It is a block block diagram which shows embodiment of the mobile body information management system which concerns on this invention. It is a block block diagram of a mobile body information management server. It is a block diagram of attribute information. It is a block diagram of flow line information. It is a block diagram of flow line attribute information. It is the schematic of a camera image. It is a figure explaining an estimated point. It is a figure explaining the calculation method of a score. It is a flowchart which shows a score calculation process. It is a figure explaining the score which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment of the present invention is not limited to the embodiments described later, and various modifications can be made within the scope of the technical idea. In all the drawings for explaining the embodiments, members having the same function are denoted by the same or related reference numerals, and repeated description thereof is omitted.

FIG. 1 is a block diagram of a mobile information management system configuration according to an embodiment of the present invention.
The movement information management system of this embodiment includes a camera 102, a position measurement device 103, and a moving body information management server 106 installed in a store 101.

  The camera 102 captures the direction of the entrance / exit 105 of the store 101 and transmits a video to the mobile information management server 106 via the network 107. Here, the camera 102 is photographing the direction of the entrance / exit 105, but the area to be photographed may be arbitrary. Further, not only one camera but also a plurality of cameras may be provided. The camera corresponds to a security camera or a web camera installed in the store.

  The position measuring device 103 is a device that records information on how the customer moved in the store from moment to moment (flow line). Examples of the position measuring device 103 include a range sensor, an indoor GPS (Global Positioning System), and a position measuring device based on Wi-Fi radio wave intensity. For example, when the position measuring device 103 is a range sensor, the distance between the range sensor and the object in the store at a certain time is measured, and the object is analyzed and extracted from the person portion. You can know the flow line. Although four position measuring apparatuses 103 are shown in FIG. 1, the number is not limited.

  The network 107 is used to enable the camera 102 and the position measurement device 103 to communicate with the mobile information management server 106. For example, Ethernet (registered trademark), Wi-Fi (registered trademark), USB (Universal Serial Bus) and the like correspond thereto, but are not limited thereto.

  Note that the camera 102 and the position measurement device 103 are not necessarily connected to the network 107 during measurement. For example, an image captured by the camera 102 may be recorded as information in the camera, and information in the camera 102 may be transmitted to the mobile information management server 106 after the store is closed.

FIG. 2 shows the configuration of the mobile information management server 106.
The moving body information management server 106 acquires the measurement information from the position measurement device 103 and creates the flow line information 205, and acquires and analyzes the video from the camera 102, thereby reflecting the information on the camera. The attribute information management unit 202 that creates the attribute information 206 by estimating the customer attributes (age, gender, etc.) and the flow line attribute correspondence information 207 by associating the flow line and the attribute of the customer passing in front of the camera 102 A flow line attribute information management unit 203, a storage unit 204,
And a network connection unit 208.

  The storage unit 204 stores flow line information 205, attribute information 206, and flow line attribute correspondence information 207. The storage unit 204 corresponds to a RAM, an HDD, or the like.

  In this embodiment, the mobile information management server 106 is assumed to be a server, a PC (Personal Computer), a virtual machine, or the like. In addition, although not shown in figure, the mobile body information management server 106 shall be equipped with CPU (Central Processing Unit), memory, an input instruction device (a keyboard, a mouse), etc. as needed.

  Further, the place where the mobile information management server 106 is installed is not limited. It may be installed in the store or on the cloud. Although the configuration of the mobile information management server 106 is shown in FIG. 2, individual components may be in separate mobile information management servers 106.

FIG. 3 is a diagram showing a configuration of the attribute information 206.
The attribute information 206 in FIG. 3 will be described by taking the second line item as an example.
The attribute information management unit 202 analyzes the video taken by the camera 102 and detects a human face. A unique identifier “1” is assigned to each detected face as field_id 301, the detected time “2013-05-01T10: 00: 00.000 + 09: 00” is set as start_time 302, and the horizontal component “200” of the center coordinate of the face in the video "Start_cx304" and the vertical component "300" is start_cy305.
The attribute information management unit 202 tracks the person after detecting the face, and ends the tracking when the person is out of the shooting range of the camera or the face cannot be identified.
The end time “2013-05-01T10: 00: 05.123 + 09: 00” at that time is set as end_time 303, the horizontal component “500” of the center coordinates of the face in the video is set as end_cx306, and the vertical component “400” is set as end_cy307.
Further, the attribute information management unit 202 estimates the age and gender of the person, and sets the age “20” as age 308 and the gender “male” as gender 309.

Here, for example, “2013-05-01T10: 00: 00.000 + 09: 00” means May 1, 2013 10:00:00 000 milliseconds (9 hours time difference). Is not limited.
The attribute information 206 is an example and may include attribute information other than age and gender.

FIG. 4 is a diagram showing the configuration of the flow line information 205.
The flow line information 205 in FIG. 4 will be described by taking the second line item as an example.
The flow line information management unit 201 analyzes the information measured by the position measurement device 103 and detects the position of the customer in the store. A unique identifier “1” is assigned to each detected customer as human_id 401, the detected time “2013-05-01T10: 00: 00.000 + 09: 00” is set as time 402, and the horizontal component “100” of the customer's coordinates in the store is assigned. Is x403, and the vertical component “200” is y404.
The flow line information management unit 201 records position information of customers in the store as flow line information 205 regularly or irregularly. However, for human_id 401, if it is estimated that the same person as the customer detected previously is used, the identifier assigned at that time is used.
Therefore, for example, if only the human_id 401 having the value “1” is extracted from the flow line information 205, a series of information indicating where the customer assigned with the human_id 401 “1” existed in the store at which time can be obtained. .

  So far, the flow line information 205 and the attribute information 206 have been described as having no relation to each other. However, when information about what kind of attribute (gender / age) in the store is used to buy the customer is obtained, when establishing measures such as improving store sales, Can be used as more valuable information.

Next, a method for associating the attribute assigned to the customer who entered the store from the store entrance 105 with the flow line will be described.
First, reference lines necessary for associating attributes with flow lines will be described with reference to FIG. Here, the camera image 601 is an example when the entrance / exit 105 is photographed by the camera 102.
A start point 605 and an end point 607 represent the center of the customer's face. Here, the start point 605 and the end point 607 when the customer moves from the start point 605 to the end point 607 in front of the camera 102 are illustrated in the same camera image 601.

A reference line 602 is set in order to determine which area in the real space in the store 101 corresponds to the area captured by the camera image 601. Here, the left end of the reference line 602 is a point P603, and the right end is a point Q604. The reference line 602 is a straight line, for example, an automatic door of a store or a floor grounding portion of a security gate, and points P603 and Q604 are in the camera image such as both ends of the automatic door and both ends of the security gate. And the actual position in the store are preferably associated with each other.

<Estimation of distance from reference line>
Comparing the start point 605 and the end point 607 in FIG. 6, it can be seen that as the customer approaches the camera 102, the center position of the face moves to the lower part of the camera image 601. This will be specifically described with reference to FIGS. The intersection point of the line drawn perpendicularly from the end point 607 to the reference line 602 is the intersection point E608, the corresponding point in the real space of the intersection point E608 is the intersection point E'704 in FIG. 7, and the installation position of the camera 102 in the real space is Assuming that the point is O ′ 706, the person having the end point 607 exists on the straight line connecting the intersection E ′ 704 and the point O ′ 706 in the real space. As a correspondence method between the intersection point E608 and the intersection point E′704, if the intersection point E608 internally divides the reference line 602 into m to n, a point that internally divides the reference line 701 into m to n is the intersection point E′704. And
Here, the reference line 602, the point P603, and the point Q604 in the camera image 601 correspond to the reference line 701, the point P′702, and the point Q′703 in the real space, respectively.

Next, a method for estimating the position of the person in the real space will be described.
Assuming that the estimated position is an estimated point E ′ 705, the estimated point E ′ 705 includes a straight line L ′ 707 drawn parallel to the reference line 701 at a distance away from the reference line 701 by a distance RE ′ 706 and an intersection point E ′ 704. It becomes an intersection with point O'708. Therefore, if the distance RE ′ 706 can be estimated from the end point 607 in the camera image 601, the estimated point E ′ 705 can be calculated.

An example of a method for estimating the distance RE ′ 706 will be described.
For example, the vertical width of the camera image 601 is 600 pixels, and the y-coordinate value of the center position of the face when a person stands on the reference line 602 is 200 pixels when measured from the top of the camera image 601, and before the camera 102 Assume that the y-coordinate value of the center position of the face is 600 pixels when standing. In this case, by linearly interpolating the y coordinate value and the distance from the reference line 602, if the y coordinate value is 400, it can be estimated that a person is present at an intermediate point between the reference line 602 and the camera 102. .

  In the above description, only the y-coordinate value is used. However, since the attribute information 206 includes information on age and gender, for example, the distance RE'706 is estimated by taking into account information on the national average height for each gender. Also good. For example, since children and adults are generally shorter in height, when the adult and child stand at the entrance, the y-coordinate value in the camera image becomes larger for the child. Based on this, the interpolation method can be changed.

<Introduction of score>
Below, the concept of the score which becomes the characteristic of this invention is described. First, consider the case where only one person can enter the store at the same time because the store entrance is narrow, or the case where there are few people entering and exiting even though the entrance is wide. In this case, the flow line and the attribute can be easily associated by associating the time when the person is reflected on the camera 102 and the time when the position measuring device 103 detects the person at the entrance / exit.

However, for example, when a plurality of people such as family customers enter the store from the entrance at the same time, the position measuring device 103 detects a plurality of people at the time when the person is reflected on the camera 102. Cannot be determined.
Therefore, in the present invention, a flow line that minimizes the score is extracted using the distance between the estimated point at the time when the face is detected and the position of the person at the time as a score.
The meaning of score minimization will be described with reference to FIG. In FIG. 10, it is assumed that two people enter the store through the entrance 105 and the flow line A 1001 and the flow line B 1002 are obtained by the position measuring device 103, respectively. Further, it is assumed that the camera 102 detects a store visitor and creates corresponding attribute information 206. The value of start_time 302 at this time is temporarily “time T”.

Based on the above assumption, an estimated point 1007 at “time T” is obtained according to the method described above. Next, the positions measured by the position measurement device 103 at the time closest to “time T” for each of the flow line A1001 and the flow line B1002 are set as a position A1003 and a position B1004, respectively. At this time, the distance A1005 is obtained as the distance between the position A1003 and the estimated point 1007, and the distance B1006 is obtained as the distance between the position B1004 and the estimated point 1007. Comparing the distance A1005 and the distance B1006, the distance A1005 is smaller. That is, it can be determined that the person corresponding to the flow line A is present at the place closest to the estimated point 1007 estimated from the position of the face reflected on the camera 102 at “time T”.
In this way, by introducing the score, even when a plurality of people enter the store from the entrance / exit 105, it is possible to appropriately select a person to be associated with the attribute information 206.

  In addition, in order to simplify the description above, the score is the distance between the estimated point corresponding to start_time 302 and the position of the person at the time closest to the time, but the estimated point corresponding to end_time 303 is added to the left. The accuracy is more improved when the total value of the distances from the person's position at the time closest to the time is used as the score.

<Correspondence method of flow line and attribute>
Based on the above, a description will be given of a method in which the flow line attribute information management unit 203 creates the flow line attribute correspondence information 207 when the flow line information 205 and the attribute information 206 are given.
As shown in FIG. 5, the flow line attribute correspondence information 207 includes a table that associates face_id 301 with human_id 401.

The procedure for creating the flow line attribute correspondence information 207 will be described below with reference to the flowcharts of FIGS. 8 and 9.
First, one line is extracted from the attribute information 206 in order from the top (step 901).
Next, for each human_id 401 in the flow line information 205, a line having a time 402 closest to each of the attributes start_time 302 and end_time 303 extracted in step 901 is searched (step 902). For example, if a human_id 401 of “1” is included, as indicated by a flow line T ′ 809 in FIG. 8, the time when the customer is moving from time to time at the time 402 closest to the start_time 302. It can be seen that the position E ′ 808 exists at time S ′ 804, similarly at time 402 closest to the end_time 303.
Here, assuming that the start_time 302 is the start point and the end_time 303 is the end point, the center of the person's face in the camera image 601 corresponds to the start point 605 and the end point 607, respectively.

Next, as described above, the intersection point S606 and the intersection point E608 corresponding to the start point 605 and the end point 607 are obtained in the camera image, and the intersection point S′801 and the intersection point E′805 corresponding to the real space are obtained. Using these pieces of information, the estimated point S ′ 802 and the estimated point E ′ 806 are estimated from the y coordinate values start_cy 305 and end_cy 307 of the face center position acquired in step 901 (step 903).
Further, a distance RS′803 between the estimated point S′802 and the position S′804 and a distance RE′807 between the estimated point E′806 and the position E′808 are obtained for each human_id, and the distance RS′803 and the distance RE are calculated. The total value of '807 is used as a score (step 904).

As described above, for each face_id 301, information on a set of each human_id 401 and score is obtained. The human_id 401 that minimizes the score is selected from these (step 905). Thereby, one human_id 401 corresponding to face_id 301 can be given, and a set of face_id 301 and human_id 301 is recorded as flow line attribute correspondence information 207 (step 906).
The above procedure is repeated until there is no more item of attribute information 205 (step 907), and the creation of the flow line attribute correspondence information 207 is completed.

  As described above, the customer attribute and the flow line obtained from the measurement information of the camera 102 and the position measuring device 103 in the store are associated with each other by the above method, thereby obtaining a flow line with the attribute information. Based on this, it is possible to analyze, for example, how customers with different attributes travel around the store, and it can be used for planning sales improvement measures in the store.

DESCRIPTION OF SYMBOLS 101 ... Store 102 ... Camera 103 ... Position measuring device 106 ... Mobile body information management server 201 ... Flow line information management part 202 ... Attribute information management part 203 ... Flow line attribute information management part 205 ... Flow line information 206 ... Attribute information 207 ... Flow line attribute correspondence information

Claims (7)

The attribute information of the person obtained by analyzing the information shot with the camera,
A method of associating the flow information of a person obtained by analyzing information measured by a position measuring device with a moving body information management device,
The mobile information management device is
A first step of calculating an estimated point e1 where a person exists in real space from the y coordinate value y1 of the center position of the face in the camera image at time t1 included in the attribute having the specific attribute identifier in the attribute information. When,
A second estimation point e2 where a person exists in real space is calculated from the y coordinate value y2 of the center position of the face in the camera image at time t2 included in the attribute having the specific attribute identifier in the attribute information. Steps,
For each flow line identifier in the flow line information, the distance between the position in the real space at the time closest to the time t1 and the estimated point e1, and for each flow line identifier in the flow line information, A sum of the distance between the position in the real space at the time closest to time t2 and the estimated point e2 is used as a score, and a flow line identifier that minimizes the score is associated with the specific attribute identifier. A moving body information management method comprising the steps of:   The mobile object information management method according to claim 1, wherein a range sensor is used as the position measuring device.   In the first step and the second step, in addition to the y-coordinate value y1 and the y-coordinate value y2, information on the average height estimated from the gender and age of the person is further added to estimate points e1 and e2 The mobile body information management method according to claim 1 or 2, characterized in that: A device that associates attribute information of a person obtained by analyzing information photographed by a camera with flow line information of a person obtained by analyzing information measured by a position measuring device,
First means for calculating an estimated point e1 where a person exists in real space from the y coordinate value y1 of the center position of the face in the camera image at time t1 included in an attribute having a specific attribute identifier in the attribute information When,
A second estimation point e2 where a person exists in real space is calculated from the y coordinate value y2 of the center position of the face in the camera image at time t2 included in the attribute having the specific attribute identifier in the attribute information. Means,
For each flow line identifier in the flow line information, the distance between the position in the real space at the time closest to the time t1 and the estimated point e1, and for each flow line identifier in the flow line information, A sum of the distance between the position in the real space at the time closest to time t2 and the estimated point e2 is used as a score, and a flow line identifier that minimizes the score is associated with the specific attribute identifier. A mobile information management apparatus comprising: means.   The mobile object information management apparatus according to claim 4, wherein a range sensor is used as the position measurement apparatus.   In the first means and the second means, in addition to the y-coordinate value y1 and the y-coordinate value y2, information on the average height estimated from the gender and age of the person is added to estimate points e1 and e2 The mobile body information management device according to claim 4 or 5, characterized in that: A program used in mobile body information management that associates attribute information of a person obtained by analyzing information photographed by a camera with flow line information of a person obtained by analyzing information measured by a position measurement device. There,
The mobile information management
First means for calculating an estimated point e1 where a person exists in real space from the y coordinate value y1 of the center position of the face in the camera image at time t1 included in an attribute having a specific attribute identifier in the attribute information When,
A second estimation point e2 where a person exists in real space is calculated from the y coordinate value y2 of the center position of the face in the camera image at time t2 included in the attribute having the specific attribute identifier in the attribute information. Means,
For each flow line identifier in the flow line information, the distance between the position in the real space at the time closest to the time t1 and the estimated point e1, and for each flow line identifier in the flow line information, A sum of the distance between the position in the real space at the time closest to time t2 and the estimated point e2 is used as a score, and a flow line identifier that minimizes the score is associated with the specific attribute identifier. A program characterized by functioning as a means.

Images