JP2017083980A - Behavior automatic analyzer and system and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently acquire a flow line of a person's behavior without omission in a person's behavior analysis device. SOLUTION: A person behavior analysis device 1 has an area information setting unit 6 for setting a detection area for recognizing a person's position and action from a measurement device 2, and a position information measurement for recognizing a person's position and action based on the detection area. It has a unit 4 and an action recognition unit 5, an action log output unit 7 that outputs the measurement result as log data, and a movement line complementing unit 8 that complements the movement line of a person based on the log data. [Selection diagram] Fig. 1

Description

  The present invention relates to a behavior analysis device, system, and method, and more particularly, to a behavior analysis device, system, and method for analyzing information about a person's behavior and trends.

By grasping the purchasing process of customers in a store such as a supermarket, the store can take efficient sales promotion measures such as improvement of the product shelf layout and sales promotion campaign. Therefore, there is a high need for a person (customer) behavior analysis device that acquires a customer's purchase process and analyzes information such as purchase behavior and purchase trends. As a means of analyzing customer behavior, business data such as POS has been used in the past, but this analysis can only handle information on products purchased by customers and products not purchased, and proposes appropriate sales measures. It may be difficult to do. Therefore, by using a measuring device so that more effective sales measures can be taken, the customer's movement route up to the purchase of the product and the behavior of the customer when the purchase is not achieved are recognized. Attention has been focused on the technology to grasp the detailed purchasing process.
For example, in Patent Document 1, a customer's travel route and purchase behavior are linked by acquiring customer location information and purchased product information using a mobile device. Patent Document 2 captures not only the customer's travel route but also the behavior of the customer by photographing how a customer purchases a product using a surveillance camera and applying image recognition technology to the camera image. Yes.

JP 2014-052759 A International Publication No. 2015/033577

In Patent Document 1, a mobile device in which a special application is installed is used as a measurement sensor. Therefore, although it is possible to grasp a movement route by always acquiring customer position information, there is a problem that it is impossible to recognize customer behavior such as picking up a product. As preparation for measurement, it is necessary to have a customer install an application directly on his / her device, or to prepare a device on which the application is already installed at the store side. For this reason, the former method imposes a burden on customers and may limit the age group of customers to be analyzed. The latter method limits the number of customers that can be analyzed for reasons such as cost. There is a problem that it is assumed that the
In Patent Document 2, since the purchase process of the customer is recognized by using a camera, not only the customer can be tracked but also the action can be recognized by analyzing the acquired image information. However, in order to track the customer from the camera video and extract the flow line indicating the travel route and purchase behavior, the target customer is always detected for each frame to determine whether or not they are the same person. Need to keep saving. For this reason, there is a problem that the flow line may be interrupted when the analysis target is a wide range or when the human image detection rate is low, and even if the flow line can be acquired without being interrupted, the amount of data becomes enormous and efficient. There is a problem that a general analysis may be difficult.

  In view of the above points, an object of the present invention is to efficiently acquire a flow line of behavior of a person (target) such as a customer without omission.

According to the first solution of the present invention,
A behavior analysis device,
A position information measuring unit that measures the position of the object from the three-dimensional information acquired from the measurement result of the measuring device;
From the three-dimensional information acquired from the measurement result of the measurement device, an action recognition unit that recognizes one of the movement of the target and a predetermined action;
An area information setting unit for setting one or a plurality of detection areas for recognizing the position and action of the target in the measurement range of the measurement device;
An action log output unit that creates an action log representing the target action for each detection area;
A flow line complement unit for combining a plurality of flow line logs including a plurality of action logs of the same target according to predetermined action process information representing a series of actions performed by the target based on the plurality of action logs; A behavior analysis apparatus comprising:

According to the second solution of the present invention,
A behavior analysis system,
A measuring device;
A three-dimensional information acquisition unit that acquires target three-dimensional information from the measurement device;
A behavior analysis device,
The behavior analysis device includes:
A position information measuring unit that measures the position of the object from the three-dimensional information acquired from the measurement result of the measuring device;
From the three-dimensional information acquired from the measurement result of the measurement device, an action recognition unit that recognizes one of the movement of the target and a predetermined action;
An area information setting unit for setting one or a plurality of detection areas for recognizing the position and action of the target in the measurement range of the measurement device;
An action log output unit that creates an action log representing the target action for each detection area;
A flow line complement unit for combining a plurality of flow line logs including a plurality of action logs of the same target according to predetermined action process information representing a series of actions performed by the target based on the plurality of action logs; Is provided.

According to the third solution of the present invention,
A behavior analysis method,
Measure the position of the target from the three-dimensional information obtained from the measurement results of the measuring device,
From the three-dimensional information acquired from the measurement result of the measurement device, recognize any one of the movement of the target and a predetermined action,
Set one or more detection areas to recognize the position and action of the target in the measurement range of the measurement device,
Create an action log that represents the target action for each detection area,
Based on multiple behavior logs, a behavior analysis method is provided that combines multiple flow-line logs containing multiple behavior logs for the same target according to predetermined behavior process information representing a series of actions performed by the target. Is done.

According to the fourth solution of the present invention,
A behavior analysis device,
A position information measurement unit that measures the position of the target from the measurement result of the measurement device;
An action recognition unit for recognizing one of the movement of the target and a predetermined action from the measurement result of the measurement device;
An area information setting unit for setting one or a plurality of detection areas for recognizing the position and action of the target in the measurement range of the measurement device;
A behavior analysis apparatus comprising:

  According to the present invention, it is possible to efficiently acquire a flow line of behavior of a person (target) such as a customer without omission.

1 is a functional block diagram of a human behavior analysis apparatus according to a first embodiment of the present invention. The figure explaining the flow of the three-dimensional information acquisition part 3. FIG. The figure explaining the flow of the positional information measurement part 4. FIG. Explanatory drawing of an example of the customer action recognized by the action recognition part 5. FIG. Explanatory drawing of an example of the detection area set by the area information setting part 6. FIG. Explanatory drawing of an example of the action log output by the action log output part 7. FIG. The figure explaining the flow of the flow line complementation part 8. FIG. Explanatory drawing of an example of the action log output by process S1 and process S2 of the flow line complement part 8, and a flow line log. Explanatory drawing of an example of the action process information used by the flow line complement part 8. FIG. Explanatory drawing of an example of the complemented flow line log output by the flow line complement part 8. FIG. For example, an explanatory diagram of an example of an action log reflecting gender attributes is shown. Explanatory drawing of an example of the result output by the analysis result output part 9. FIG. The functional block diagram of the person action analysis apparatus of Example 2 of this invention. Explanatory drawing of an example of the corresponding | compatible table of the measurement item used by the measurement information selection part 93, and the measurement item to select. The figure explaining the flow of the common map preparation part 94. FIG. Explanatory drawing of an example of the detection area set by the area information setting part 6 at the time of use of a some measuring device. FIG. 17 is an explanatory diagram of an example in which area information is added to the detection area setting example shown in FIG. 16.

  Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram of the human behavior analysis apparatus according to the first embodiment of the present invention. In this embodiment, an example in which a stereo camera is used as a measurement device to acquire position information of a person (target) and recognition of an action will be described. However, this embodiment is limited to these measurement devices. Not. In addition, as a measuring device, various measuring devices, such as a single camera, a three-dimensional sensor, a laser sensor (laser radar), an infrared tag, can be used, for example. The human behavior analysis apparatus 1 shown in FIG. 1 uses a stereo camera in which two cameras 2 are adjacent to each other to extract a flow line of a target based on a behavior log of a specific target in the camera 2 (imaging device). It is a device that analyzes. In FIG. 1, each function of blocks 3, 4, 5, 6, 7, 8, and 9 is, for example, a computing device, a main storage device, a camera having an external storage device, a computer prepared separately from the camera, or the like. It is realized by. First, the outline of each function shown in FIG. 1 will be described. The three-dimensional information acquisition unit 3 acquires the target three-dimensional information from the stereo camera, and the position information measurement unit 4 measures the target position from the acquired three-dimensional information. The function and action recognition unit 5 recognizes the target position and the action related to the movement and purchase of the target from the acquired three-dimensional information, and the area information setting unit 6 sets the target position and action in the measurement range of the stereo camera. The action log output unit 7 functions to output the target position and action for each detected detection area as log data, and the flow line complementation unit 8 organizes the output action log. The function of complementing the target flow line, the analysis result output unit 9 is a function of outputting the result of analyzing the acquired target flow line information. Details of each function will be described below.

FIG. 2 shows a flow in which the 3D information acquisition unit 3 acquires 3D information including image and distance information from the stereo camera. In FIG. 2, the image acquisition unit 10 functions to acquire an image from the camera, and the distance calculation unit 11 calculates the distance between the target in the image and the camera from the image acquired from the camera 2 by the image acquisition unit 10 and the camera parameters. It is a function.
The image acquisition unit 10 is means for converting visible light acquired from at least one camera 2 at a predetermined time period into an electrical signal via a CCD or a CMOS element. In the case of a digital system, digital conversion is also performed in addition to this. Hereinafter, the image output from the image acquisition unit 10 is referred to as a captured image.
The distance calculation unit 11 calculates the distance between the object and the camera in the captured image from the captured images obtained by the two cameras and the camera parameters estimated in advance by a known calibration technique in each camera. As a distance calculation method, for example, a general method called stereo matching that calculates parallax using a basic matrix F obtained from camera parameters can be used. The basic matrix F is a parameter matrix indicating the relative positional relationship between the two cameras, and is expressed by the internal parameters K ₀ and K ₁ and the external parameters D ₀ and D ₁ of the two cameras as shown in the following formula. It can be calculated. In the following equation, f in the internal parameter K is the focal length, a is the aspect ratio, s is the skew, (v _c , u _c ) is the center coordinate of the image coordinates, and (r ₁₁ , R ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ , r ₃₁ , r ₃₂ , r ₃₃ ) indicate the direction of the camera, and (t _x , t _y , t _z ) indicate the camera installation position.

  FIG. 3 shows an example of a flow in which the position information measurement unit 4 uses the distance information derived from the three-dimensional information acquisition unit 3 to detect a target and measure the position. In FIG. 3A, a person 20a, 20b to be measured, a product shelf 21 not to be measured, and a parallax image 22 showing the distance value from the camera to the object for each pixel are shown. The position information measuring unit 4 first extracts a person region by background difference processing shown in FIG. In FIG. 3B, a background parallax image 23 acquired in advance is shown, and only the product shelf 21 exists in the image. Note that such a background parallax image 23 can be captured and stored in advance by a camera. Therefore, the position information measurement unit 4 can create a parallax image 24 indicating a person region where only a person exists by taking the difference between the input parallax image 22 where the person exists and the background parallax image 23. Next, the position information measurement unit 4 distinguishes the person from the person area by the viewpoint conversion process shown in FIG. 3C, and acquires the three-dimensional information of each person. In FIG. 3C, a person's three-dimensional information 25 estimated from the image coordinates and distance information of the parallax image 24, and a bird's-eye view image 26 of the three-dimensional information 25 viewed from the viewpoint directly above are shown. Note that the three-dimensional information indicates information of a point group of three-dimensional coordinates plotted on a specific coordinate space. Moreover, as a method for creating a bird's-eye view image, a general method such as perspective projection using camera parameters is used. Then, as shown in FIG. 3C, the position information measuring unit 4 calculates the degree of coincidence between the 3D information template 27 of the person in the overhead image and the 3D information of each person in the overhead image 26. The person is detected by determining whether or not the person exists according to the degree. Next, the position information measuring unit 4 measures the position information of the person from the detected three-dimensional information of the person. As a measuring method, for example, as shown in FIG. 3D, there is a method of determining the barycentric coordinates 28a and 28b of the three-dimensional information of each person in the overhead image 26 as the position information of the person. In addition to this method, any method can be used as long as it can detect a person using three-dimensional information or image information to measure a position.

FIG. 4 shows an example of behavior related to movement and purchase of a person (customer) to be recognized by the behavior recognition unit 5. 4A, FIG. 4A is an action in which the customer 30 enters the measurement range, FIG. 4B is an action in which the customer 30 leaves the measurement range, and FIG. 4C is an action in which the customer 30 stays. 4 (d) is an action in which the customer 30 picks up a product from the merchandise shelf 31, FIG. 4 (e) is an action in which the customer 30 puts the product into the cart 32, and FIG. 4 (f) is an action in which the customer 30 is the store staff 33. It is an action to receive customer service.
For “entrance” in FIG. 4A, “exit” in FIG. 4B, and “stay” in FIG. 4C, the action recognition unit 5 determines the position of the person acquired by the position information measurement unit 4. For example, when a person's position moves from outside to inside of a certain area, enter, when a person's position moves out of a certain area, leave, or when the person's position does not change for a certain period of time Is recognized as staying. As for FIG. 4 (d) “get the product”, FIG. 4 (e) “put the product into the cart”, and FIG. 4 (f) “customer service”, the action recognition unit 5 starts from the 3D information acquisition unit 3. Using the acquired three-dimensional information, for example, it is recognized by a general three-dimensional point cloud analysis or image analysis technique. For example, FIG. 4 (d) “Get a product” detects whether a customer's hand is detected and whether or not the customer's hand and the three-dimensional point cloud information of the product shelf or the product are connected to each other. e) “Put product in cart” means that the customer, the product, and the cart are recognized separately and the 3D point cloud of the product has moved from the customer to the cart. The customer and the store staff can be recognized separately, and whether or not the 3D point cloud information of the store staff and the customer are close to each other can be determined. In addition, as a method for identifying a customer and a store staff, for example, a method of detecting a person who continues a staying behavior for the longest time in a measurement range or a time longer than a predetermined threshold as a store staff, or in advance There is no particular limitation, such as a method of identifying the store staff from the image information by having the store staff wear a uniform of a special color or shape. In addition to the behaviors shown in FIG. 4, the customer behaviors to be recognized are behavior processes related to customer purchases such as “view products”, “return products to the shelf”, “roam in front of product shelves”, etc. The recognition method is not particularly limited as long as it uses three-dimensional information or image information.

  FIG. 5 shows an example of setting a detection area in which the area information setting unit 6 recognizes customer behavior in the measurement range. FIG. 5A shows a detection area setting value 40 in the measurement range. In FIG. 5B, an overhead view 41 of the measurement range, the product shelf 42, detection areas 43 a, 43 b, 43 c, 43 d for recognizing the “entry” and “exit” behavior of the customer, and the “residence” behavior of the customer Detection area 44 for recognizing the customer, detection area 45 for recognizing the actions of the customer “take the product”, “return the product to the shelf” and “put the product into the cart”; Area 46 is shown. In the area information setting unit 6, the user of the human behavior analysis apparatus selects a customer's behavior that the user wants to recognize in the measurement range, and sets the position and area ID of the detection area for each behavior. As a method for selecting the customer behavior and setting the position of the detection area, for example, as shown in FIG. 5, a measurement view of the measurement range is displayed on a PC display or the like, and manually using an interface such as a GUI. In addition to the setting method, the camera area is automatically set after estimating the shooting range environment from the captured image or parallax image by scene recognition, etc., or the camera image is displayed as it is and the area is directly drawn and set in the GUI etc. There are no particular restrictions on the method used. As a method for setting the position of the detection area, the area shape is not limited to a rectangle, and various shapes such as an ellipse, a circle, a triangle, a polygon, and a rhombus may be used. Also good. Further, the customer behavior set in each detection area is not necessarily one, and a plurality of behaviors may be set.

  FIG. 6 shows an example of an action log output by the action log output unit 7 and a list of area IDs and action IDs described in the action log. FIG. 6 shows an action log 50, an area ID list 51, and an action ID list 52. In the action log output unit 7, when a specific action is recognized by the action recognition unit 5 in the detection area set by the area information setting unit 6, the recognition time, area ID, action ID, and person ID are described. Create a behavior log and output it. The details of the action log will be described. The recognition time is the time when the action is recognized, the area ID is the ID number of the detection area set by the area information setting unit 6, and the action ID is recognized by the action recognition unit 5. The customer action ID number and the person ID are the ID numbers of the target person (customer) recognized by the action recognition unit 5. As a person ID assignment method, for example, in the person detected for each frame by the position information measuring unit 4, it is determined whether or not the same person exists in the previous frame and the current frame. There is a method of assigning a person ID, and assigning a new person ID if it does not exist. In addition, as a method for determining whether or not the same person exists, for example, a method for determining the same person if the difference in position distance measured by the position information measuring unit 4 is equal to or less than a certain threshold, or image information A method of determining whether or not they are the same person by a general face recognition method using the method is used. Further, in this embodiment, since the flow line of the same customer can be acquired even when the assignment of the person ID is not successful, it is not always necessary to assign the same person ID to the same person. There is no particular limitation on the method other than the method described above. The action log output method is not particularly limited, for example, using a general readable / writable format such as a text file or an Excel file.

  FIG. 7 shows a processing flow of the flow line complementation unit 8. The flow line complementing unit 8 first arranges all (or processing target) action logs in time series according to the recognition time (S1), and then extracts action logs having the same person ID from the arranged action logs. A flow line log is created (S2). Next, the flow line complementation unit 8 compares each created flow line log with predetermined action process information representing a series of customer actions related to the purchase / non-purchase of goods, and the movement that is considered to be missing. The line logs or the flow line logs estimated as a series of actions are connected and complemented (S3). Next, the flow line complementing unit 8 calculates the flow line reliability of the complemented flow line log (S4), and if the reliability is less than the specified threshold value, the flow line logs are combined again with different flow line logs. If the reliability is equal to or higher than the threshold value, it is output as the final flow line log (S5). Hereinafter, each process of the flow line complementation unit 8 will be described.

  FIG. 8 shows an example of the action log arranged in time series by the process S1 of the flow line complementation unit 8 and the flow log of the same person ID extracted by the process S2. FIG. 8 is an overhead view 60 of the measurement range in which the product shelf 42 exists, a list 61 of action IDs, an example 62 of action logs sorted in time series by the process S1, and actions sorted in time series by the process S2. It is an example of a flow line log in which only action logs indicating the same person ID are extracted from the log, and in order, flow line logs 63a, 63b, 63c, 63d of personal ID 10, personal ID 11, personal ID 12, and personal ID 13; Is shown. Since the flow logs 63a, 63b, 63c, and 63d are obtained by arranging the action logs of the persons determined to be the same, in the situation where there is no missing action log, the flow of the customer can be traced by following these flow line log information. It becomes possible to grasp the line. However, since the action log may actually be missing, such as when position measurement or action recognition fails or is impossible, the flow line complementation unit 8 performs the action log in the processes S3 and S4 described below. Compensate for the lack of and create a flow line.

  FIG. 9 shows an example of behavior process information corresponding to the behavior ID list 61 used in the process S3 of the flow line complementation unit 8. The behavior process information may be held only in a plurality of matrix data as shown in FIG. 9A, or may be held in combination with a matrix data group and a state transition diagram as shown in FIG. 9B. FIG. 9 shows an action process 70 indicating “not interested in product shelf”, an action process 71 indicating “interested in product shelf”, an action process 72 indicating “purchase product”, and “not shown but purchased product”. Is a matrix data for an action process 73 indicating In addition to the above, the action ID list and the action process information may be selected by the user of the present system for analysis, and are not particularly limited. For example, by adding “customer service without receiving customer service”, “purchase product without customer service”, “purchase product with customer service”, etc. to the behavior ID list, etc. Analysis of product sales by presence or absence and quality comparison of customer service is possible.

FIG. 10 shows flow line logs 80 and 81 obtained by combining and complementing the flow line logs 63a, 63b, 63c, and 63d using the action process information by the process S3 of the flow line complementing unit 8. In this example, the action log of the last flow line log has a flow line log 63a whose action ID is “take a product” and the action line log 63c has an action ID of the first flow line log “return a product to the shelf”. And the flow line log 63b whose action ID of the last flow line log is “entry” and the flow line log 63d whose action ID of the first flow line log is “exit” The log is complemented. In addition, in process S3, when the flow line logs complement each other, and there is no flow line log of the connection destination along the action process, for example, the flow line complementation unit 8 sets the flow line with the shortest recognition time interval. The log is searched, and if it is shorter than a certain threshold time T _s , the log combined with the flow log can be output, and if it is long, the log can be output as it is without combining. For example, when the first action of the flow line log 63c is “end of stay” due to failure to recognize the customer action, it is assumed that the flow line log 63a cannot find a flow line log suitable for the action process. . For this reason, the flow line complementing unit 8 searches for a flow line log having the shortest recognition time within the threshold time T _s among the incomplete flow line logs and combines them, and if not found, combines them. Can be output.

  The flow line complementing unit 8 calculates the flow line reliability of each flow line supplemented in the process S4 in order to determine whether or not all these flow line logs output in the process S3 are accurate. As a method of calculating the flow line reliability, for example, there is a method of obtaining from the ratio between the total number of flow lines output from the process S3 and the number of passers-by in the measurement range as in the following formula.

ここで、計測範囲における通行者数とは計測範囲の近くを通過した人物の人数であり、例えば図８の俯瞰図６０の場合では、エリア１００とエリア１０１を合わせた領域の少し外側に更に大きな矩形のエリアを設定してそのエリアに進入する人数のみ別途計測することで取得できる。本手法では、動線補完部８は、上述の数式によって出力された動線信頼度が１に近い程、処理３によって出力された動線ログが正確であると判定し、動線信頼度が１よりも特定の閾値以上離れている場合は行動プロセスに沿わない動線ログ同士の結合を解除する、または結合の組み合わせパターンを変更する、または閾値時間Ｔ_ｓの値を変更するなどの方法により、再度動線ログを作成して処理Ｓ４により動線信頼度を算出することができる。

Here, the number of passers by in the measurement range is the number of persons who have passed near the measurement range. For example, in the case of the overhead view 60 in FIG. 8, the area 100 and area 101 are slightly larger outside the combined area. This can be obtained by setting a rectangular area and separately measuring only the number of people entering the area. In this method, the flow line complementation unit 8 determines that the flow line log output by the processing 3 is more accurate as the flow line reliability output by the above-described mathematical formula is closer to 1, and the flow line reliability is higher. When it is more than a specific threshold value apart from 1, the connection between the flow line logs that do not follow the behavior process is canceled, the combination pattern of the connection is changed, or the value of the threshold time T _s is changed. The flow line log can be created again, and the flow line reliability can be calculated by the process S4.

As another method for calculating the flow line reliability, there is a method of determining that the number of incomplete flow line logs output in step S3 is the flow line reliability, and that the smaller the value is, the higher the flow line reliability is. As another method, the movement time between the calculated customer's areas is compared by comparing the area ID with the recognition time of the last flow line log of the combined flow lines and the first flow line log of the combined destination. There is a method of determining that the reliability is low when the speed is set as the flow line reliability and the speed is obviously faster or slower than the normal walking speed. In addition to this, there is a method that uses face recognition technology and attribute estimation technology together. In the method using the face recognition technology, when the behavior recognition unit 5 recognizes the customer's behavior, the customer's face information is quantified by a general face recognition technology using image features and the value is recorded in the behavior log. Or may be written in advance in the flow line log. Then, when calculating the flow line reliability of the combined flow line log, the reliability of the flow line log can be determined by using the digitized difference of the face information. In addition, as a method using the attribute estimation technique in combination, the customer flow recognized by the behavior recognition unit 5 is estimated in advance, and the attribute information is reflected in the behavior log or the flow log, so that the combined flow log There is a method of determining whether or not there is a flow line reliability depending on whether or not the attribute information is the same. As attribute information, in addition to age, gender, nationality, etc., information such as an individual or group, a cart user or an unused user may be used.
FIG. 11 illustrates an example of an action log that reflects, for example, gender attributes.
In addition to the method of calculating the flow line reliability of all the flow line logs output in step S3, the flow line log that completely matches the action process is excluded from the target, or the incomplete flow line log is used. You may choose the method of not doing.

  FIG. 12 is an output example of the result of analyzing the information of the customer flow line log output from the flow line complementation unit 8 by the analysis result output unit 9. FIG. 12 shows a comparison of product purchase rates depending on the presence or absence of customer service for customers who are interested in the product, and it can be seen that the product purchase rate has improved as a result of customer service. From this analysis result, the store side can quantitatively grasp the effect of customer service, and can take efficient sales promotion measures such as business improvement, advertisement, and review of advertising methods. Note that this example specializes in a method for acquiring the customer's flow line without omission, and the analysis method using the acquired flow line and the output method of the analysis result may be a system that uses the flow line as input. There is no particular limitation.

In the first embodiment of the present invention, with the functional configuration described above, one or a plurality of detection areas corresponding to the customer behavior are set in the measurement range, and the customer behavior for each recognized detection area is output as an action log. By organizing the action log output using the information and the action process information, it is possible to efficiently acquire the flow line of the customer without omission.
In addition, in Example 1, in addition to using a stereo camera with two cameras adjacent to each other as a measuring device, it was acquired from image information acquired using one camera or using three or more cameras. The series of flows shown in FIG. 1 may be performed by performing position measurement and action recognition of a customer from image information and distance information, or from image information or 3D information acquired using a 3D sensor.

  FIG. 13 is a block diagram of the human behavior analysis apparatus according to the second embodiment of the present invention. In FIG. 13, the functions of blocks 2, 3, 4, 5, 6, 7, 8, and 9 are the same as those in the first embodiment. A wide-range human behavior analysis device 91 using a plurality of measuring devices acquires a wide range of human flow lines by using another measuring device 92 in addition to the measuring device of the stereo camera 2. Explaining each function, the measurement information selection unit 93 has a function of selecting an item to be measured based on the type of measurement device to be used. The measurement information selection unit 93 selects measurement items according to the stereo camera 2 and the measurement device 92, and performs position measurement by the position information measurement unit 4, action recognition by the action recognition unit 5, or both according to the selection contents. To do. The common map creation unit 94 creates a common map in advance in order to obtain consistency of position information between the measurement result of the measurement device 92 and the measurement result by the stereo camera, and obtains conversion information that can reflect each measurement result on the common map. It has the function to do. In addition, the measurement result conversion unit 95 measures the item selected by the measurement information selection unit 93 for each measurement device (in this example, the stereo camera 2 and the measurement device 92), and the measurement result is displayed by the common map creation unit 94. It has a function to convert according to the acquired information. Details of the measurement information selection unit 93, the common map creation unit 94, and the person measurement unit 95 will be described below.

  FIG. 14 shows an example of a correspondence table between the types of measurement devices and the measurement items selected by the measurement information selection unit 93. The measuring device is not particularly limited as long as it is a device capable of measuring the position of a person and / or recognizing a person's action, or both. In addition, in a measurement apparatus that can perform both position measurement and action recognition, a method of selecting only items that are considered to have high measurement accuracy may be used.

FIG. 15 is a diagram for explaining the flow of the common map creation unit 94. The process of the common map creation unit 94 is a process that is performed in advance before measurement, for example. The common map creation unit 94 first acquires common map information in the measurement range as shown in FIG. Then, on the common map, a reference of a person or an object is arranged in advance at four points 100a, 100b, 100c, and 100d, and the position of the person is measured by each measuring device. FIG. 15B is a diagram in which the measurement result of the position of the person in the measurement device 92 is plotted on the two-dimensional coordinate axes used in the calculation processing of each measurement device such as the stereo camera 2 and the measurement device 92. . In FIG. 15B, plots 101a, 101b, 101c, and 101d correspond to the points 100a, 100b, 100c, and 100d in FIG. 15A, respectively. Therefore, the perspective transformation matrix between different two-dimensional coordinate axes can be calculated by using the four corresponding points, and the plot position measured by each measuring device using the calculated matrix is associated with the point on the common map. Is possible. By obtaining this perspective transformation matrix in each measurement device such as the stereo camera 2 and the measurement device 2, consistency of measurement results of different measurement devices can be obtained on a common map. When consistency is obtained using a common map as in this method, the area information setting unit 6 sets the detection area on the common map to ensure consistency between the detection area and the measurement result. Becomes easy. In addition, as a method for creating a common map, a product that is used as a mark in the measurement range, such as a product shelf 42 shown in FIG. You may create automatically using real information. In addition, as a method of corresponding the coordinate axes of the measurement apparatus and the coordinate axes of the common map, a transformation matrix using a technique other than the general perspective transformation described above (for example, an appropriate transformation technique such as projective transformation or affine transformation). May be requested. In addition to using a common map, the method for obtaining consistency between the measurement results of different measurement devices is a method capable of matching the measurement results of each measurement device or between the measurement results and the detection area. There is no particular limitation.
In accordance with the measurement item selected by the measurement information selection unit 93, after the position information measurement unit 4 performs the position measurement of the person and / or the action recognition unit 5 performs the action recognition of the person, the measurement result conversion unit 95 A result obtained by converting the coordinate information of the measurement result using a transformation matrix such as a perspective transformation matrix calculated by the common map creation unit 94 is output. By passing the converted result to the action log output unit 7, the action logs of the stereo camera 2 and the measuring device 92 on the same coordinate axis can be output.

  In the second embodiment of the present invention, by integrating the action logs output by two different measuring devices with the functional configuration described above, it is possible to efficiently acquire a wide range of customer flow lines without omission.

  FIG. 16 illustrates an example of a detection area set by the area information setting unit 6 when a plurality of measuring devices are used. In the second embodiment, not only two measuring apparatuses but also three or more measuring apparatuses may be used. For example, two stereo cameras and one laser sensor are used as the measuring apparatus, as shown in FIG. By setting the detection area, it is possible to acquire a wider range of customer flow lines. FIG. 16 shows a common map 110 in a measurement range in which three product shelves 42a and 42b exist, an enlarged view 111 in the vicinity of the product shelf 42a, and detection areas 100, 101, 102, and 103 of the stereo camera installed at the point 112a. 3 shows a detection area 104 of a stereo camera installed in FIG. 1, detection areas 200, 201, 202, and 203 of a laser sensor installed at a point 113, and a list 114 of customer actions recognized in each detection area. With the setting method shown in FIG. 16, not only the purchase behavior of the customer in the product shelf 42a can be acquired in detail, but also the flow of movement of the customer in the entire measurement range shown in the common map 110 can be grasped. Services such as planning a shelf layout improvement plan that makes the flow of movement smoother. In the area information setting unit 6, in the example of FIG. 16, although there is no overlap in the detection areas for different measurement devices, the detection areas may be set in duplicate. Further, when setting the detection area in the area information setting unit 6, the area information may be reflected in the action log by adding area information such as “product shelf” or “passage” to the area ID. .

  FIG. 17 shows an explanatory diagram of an example in which area information is added to the detection area setting example shown in FIG. In FIG. 17, “product shelf A” indicates an area set near the product shelf 42 a, “product shelf B” indicates an area set near the product shelf 42 b, and “passage” indicates that a passerby passes. Area information. By adding the area information as shown in FIG. 17 to the action log, the flow line complementing unit 8 calculates the flow line reliability when the flow line logs are combined, and the analysis result output unit 9 outputs the analysis result. Area information may be used. For example, as a method for calculating the flow line reliability using area information, the area information is compared with the recognition time of the last flow line log of the combined flow lines and the first flow line log of the combined destination. Then, by setting the calculated movement speed between customer areas as the flow line reliability, there is a method of determining that the reliability is low when it is clearly faster or slower than the normal walking speed. In addition, as a method of using area information as an analysis result, there is a method of performing analysis for each product shelf by outputting an analysis result for each area information of “product shelf”.

  In the second embodiment, as a method for calculating the flow line reliability used when the flow line logs are combined by the flow line complementation unit 8, the reliability may be determined based on the characteristics of the measuring device. For example, when using a stereo camera and a laser sensor as a measurement device, and in the case of a flow log of customer behavior related only to position measurement such as `` entrance '', `` exit '', `` stay '', etc. Using the characteristics that the position measurement accuracy of the laser camera is high and the position measurement accuracy of the stereo camera is high in situations where there are many people overlapping, if the density of people at the time of measurement is low, the reliability of the flow log of the laser sensor If the density is high, there is a method of increasing the reliability of the flow log of the stereo camera. As another example, using a stereo camera and a three-dimensional sensor as a measuring device, a flow log of customer height related to detailed behavior recognition of people such as “holding a product” and “viewing a product” In this case, there is a method of making the reliability of the flow line log of the three-dimensional sensor with high action recognition accuracy higher than the flow line log of the stereo camera.

[Appendix]

In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.
Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files for realizing each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.
Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.
In addition, the application location of the present invention is not limited to the commercial facilities such as the stores shown in the above-described embodiments, but is used to analyze information on the behavior and trends of persons such as factories, offices, logistics, medical care facilities, etc. It is possible to apply.

DESCRIPTION OF SYMBOLS 1 Person action analysis apparatus 2 Camera 3 Three-dimensional information acquisition part 4 Position information measurement part 5 Action recognition part 6 Area information setting part 7 Action log output part 8 Flow line complement part 9 Analysis result output part

Claims (16)

A behavior analysis device,
A position information measuring unit that measures the position of the object from the three-dimensional information acquired from the measurement result of the measuring device;
From the three-dimensional information acquired from the measurement result of the measurement device, an action recognition unit that recognizes one of the movement of the target and a predetermined action;
An area information setting unit for setting one or a plurality of detection areas for recognizing the position and action of the target in the measurement range of the measurement device;
An action log output unit that creates an action log representing the target action for each detection area;
A flow line complement unit for combining a plurality of flow line logs including a plurality of action logs of the same target according to predetermined action process information representing a series of actions performed by the target based on the plurality of action logs; A behavior analysis device equipped with
The behavior analysis apparatus according to claim 1,
The behavior analysis apparatus characterized in that the flow line complementing unit supplements a flow line log using at least one of time-series information, behavior pattern information, and target attribute information.
The behavior analysis apparatus according to claim 2,
The flow line complement part is:
Arrange multiple action logs in time series according to recognition time,
Extract action logs of the same target from multiple arranged action logs to create each flow log,
Comparing each flow line log with the action process information and complementing the target flow line by combining the flow line logs estimated to be missing or by combining the flow line logs estimated to be a series of actions,
A behavior analysis device characterized by that.
The behavior analysis apparatus according to claim 3,
The flow line complementing unit calculates a flow line reliability that indicates whether or not the combined flow line logs are accurate, and selects a plurality of flow line logs to be combined based on the flow line reliability. A behavior analysis device characterized by:
The behavior analysis apparatus according to claim 4,
The flow line complementing unit calculates a flow line reliability based on at least one of time-series information, target position information, and target attribute information.
The behavior analysis apparatus according to claim 4,
The flow line complement part is:
Furthermore, the flow line reliability of the complemented flow line log is calculated,
If the flow line reliability is less than a predetermined threshold, the flow line logs are complemented by combining different flow line logs again. If the flow line reliability is greater than a predetermined threshold, the flow line log is complemented. A behavior analysis device characterized by outputting.
The behavior analysis apparatus according to claim 1,
The area information setting unit presets one or a plurality of detection areas for recognizing a target action in a measurement range, and one or a plurality of actions for recognition in each detection area.
The behavior analysis apparatus according to claim 7,
The area information setting unit includes an interface for setting or changing a target action to be recognized as setting of a detection area.
The behavior analysis apparatus according to claim 1,
Recognize target behavior using multiple measuring devices,
A behavior analysis device comprising a measurement result conversion unit that outputs measurement results of a plurality of the measurement devices as operation logs that can be used in common.
The behavior analysis apparatus according to claim 9, wherein
A common map creating unit for creating conversion information of a common map for taking consistency of recognition results of the plurality of measuring devices;
The measurement result conversion unit integrates recognition results of the plurality of measurement devices using conversion information of a common map.
The behavior analysis apparatus according to claim 1,
A plurality of the measuring devices are provided,
A measurement information selection unit for selecting an item to be measured based on the type of each measurement device;
A common map creation unit for obtaining conversion information to be reflected on a common map for taking consistency of position information of measurement results of the measurement devices;
According to the item selected by the measurement information selection unit, the conversion information obtained by the common map creation unit for the target position measured by the position information measurement unit and / or the target behavior recognized by the behavior recognition unit And a measurement result conversion unit that converts the measurement results of the plurality of measurement devices, and
A behavior analysis apparatus comprising:
The behavior analysis apparatus according to claim 1,
The action log includes the action of the recognized target, the recognition time, the detection area, and the target ID.
The behavior analysis apparatus characterized in that the behavior log output unit outputs a behavior log.
The behavior analysis apparatus according to claim 1,
An action analysis apparatus further comprising an analysis result output unit that analyzes the acquired flow log of the target and outputs a product purchase rate or other analysis results.
A behavior analysis system,
A measuring device;
A three-dimensional information acquisition unit that acquires target three-dimensional information from the measurement device;
A behavior analysis device,
The behavior analysis device includes:
A position information measuring unit that measures the position of the object from the three-dimensional information acquired from the measurement result of the measuring device;
From the three-dimensional information acquired from the measurement result of the measurement device, an action recognition unit that recognizes one of the movement of the target and a predetermined action;
An area information setting unit for setting one or a plurality of detection areas for recognizing the position and action of the target in the measurement range of the measurement device;
An action log output unit that creates an action log representing the target action for each detection area;
A flow line complement unit for combining a plurality of flow line logs including a plurality of action logs of the same target according to predetermined action process information representing a series of actions performed by the target based on the plurality of action logs; Behavior analysis system with
A behavior analysis method,
Measure the position of the target from the three-dimensional information obtained from the measurement results of the measuring device,
From the three-dimensional information acquired from the measurement result of the measurement device, recognize any one of the movement of the target and a predetermined action,
Set one or more detection areas to recognize the position and action of the target in the measurement range of the measurement device,
Create an action log that represents the target action for each detection area,
A behavior analysis method for combining a plurality of flow line logs including a plurality of behavior logs of the same target in accordance with predetermined behavior process information representing a series of behaviors performed by the target based on the plurality of behavior logs.
A behavior analysis device,
A position information measurement unit that measures the position of the target from the measurement result of the measurement device;
An action recognition unit for recognizing one of the movement of the target and a predetermined action from the measurement result of the measurement device;
An area information setting unit for setting one or a plurality of detection areas for recognizing the position and action of the target in the measurement range of the measurement device;
A behavior analysis device equipped with

Images