JP7750598B2 - Behavior recognition device, behavior recognition method, and program - Google Patents

Description

The technical field relates to behavior recognition devices, behavior recognition methods, and programs that recognize behavior.

Devices that use the results of behavior recognition to execute appropriate processing can execute more appropriate processing as the accuracy of human behavior recognition improves. However, because human behavior varies from person to person, it is difficult to recognize behavior through simple matching alone.

As a related technology, Patent Document 1 proposes a system that can detect human behavior. The system in Patent Document 1 includes multiple RFID (Radio Frequency Identification) tags attached to multiple objects present in a space, a portable device carried by a user moving within the space, and a processing unit that receives and processes information from the RFID tags from the portable device.

The processing unit first pre-stores behavior patterns corresponding to specific user behaviors and counts the number of times ordered pairs (features) included in the behavior patterns are included in the user's behavior log. Next, the processing unit calculates feature points based on the number of times and compares the feature points with a threshold to detect specific behaviors corresponding to the behavior patterns.

Japanese Patent Application Laid-Open No. 2008-097472

However, in Patent Document 1, an RFID reader carried by a user moving through a space is used to receive information from RFID tags attached to objects such as switches for electrical appliances or lights, bags, mobile phones, and wallets present in spaces such as offices, schools, stores, and homes, but images acquired in a time series are not used.

Patent Document 1 also uses ordered pairs that represent combinations of two or more objects touched by a user when performing a behavior, and the order in which they were touched. Patent Document 1 then counts the number of times an ordered pair included in a behavior pattern appears in a behavior log, calculates feature points based on the number of times the ordered pair appears in the behavior log, and compares the feature points with a predetermined threshold to determine the behavior corresponding to the behavior pattern.

Therefore, if two objects are not available, for example, behaviors such as a person standing up from a chair or a person stopping to talk on a phone call cannot be recognized even using the technology in Patent Document 1.

One aspect of this is to provide a behavior recognition device, a behavior recognition method, and a program that accurately recognize the behavior of an object.

In order to achieve the above object, a behavior recognition device in one aspect includes:
an estimation unit that estimates a motion of a target object for each target object image by using target object images corresponding to the target object included in images acquired in time series;
a behavior determination unit that, when the same estimated behavior occurs consecutively a preset number of times, determines the behavior of the target object to be the estimated behavior, and, when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object, determines the behavior with the matching order as the behavior of the target object;
The present invention is characterized by having the following.

In order to achieve the above object, a behavior recognition method in one aspect includes:
an estimation step of estimating a motion of a target object for each target object image using target object images corresponding to the target object included in images acquired in time series;
a behavior determination step of determining the behavior of the target object as the estimated behavior when the same estimated behavior occurs consecutively a predetermined number of times, and determining the behavior with the matching order as the behavior of the target object when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object;
The present invention is characterized by having the following.

Furthermore, in order to achieve the above object, the program in one aspect comprises:
an estimation step of estimating a motion of a target object for each target object image using target object images corresponding to the target object included in images acquired in time series;
a behavior determination step of determining the behavior of the target object as the estimated behavior when the same estimated behavior occurs consecutively a predetermined number of times, and determining the behavior with the matching order as the behavior of the target object when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object;
The present invention is characterized in that the following is executed.

One aspect is that it can accurately recognize the behavior of objects.

FIG. 1 is a diagram illustrating an example of a behavior recognition device. FIG. 2A is a diagram for explaining the behavior of a person standing up from a chair. FIG. 2B is a diagram for explaining behavior of making a call while standing still. FIG. 3 is a diagram illustrating an example of a system having a behavior recognition device. FIG. 4 is a diagram illustrating an example of the operation information. FIG. 5 is a diagram illustrating an example of behavior determination information. FIG. 6 is a diagram for explaining the behavior determination unit in the first modification. FIG. 7 is a diagram for explaining the transition of the action determination of the behavior of the first modified example. FIG. 8 is a diagram for explaining the reduction of the influence of noise in the first modification. FIG. 9 is a diagram for explaining the transition of the action determination of the behavior of the second modification. FIG. 10 is a diagram for explaining the reduction of the influence of noise in the second modification. FIG. 11 is a diagram for explaining the transition of the action determination of the behavior of the third modification. FIG. 12 is a diagram for explaining a configuration in which processes for determining a plurality of behaviors are executed in parallel. FIG. 13 is a diagram illustrating an example of the operation of the behavior recognition device. FIG. 14 is a block diagram illustrating an example of a computer that realizes the behavior recognition device according to the embodiment and the first to fourth modifications.

First, an overview will be provided to facilitate understanding of the embodiments described below. A device that uses the results of recognizing human behavior to perform appropriate processing will be able to perform appropriate processing the more accurately it can recognize human behavior. In other words, being able to accurately recognize human behavior will enable a variety of applications.

Specific applications include devices that recognize and provide assistance to people's behavior, devices that recognize people's gestures and allow them to perform command operations, and devices that recognize and point out the form of people playing sports (for example, pitching form in baseball, swing form in golf, etc.).

Furthermore, it can also be applied to devices that recognize the behavior of non-human objects, such as animals and robots, and perform processing using the recognition results.

Behavior refers to a series of actions performed by an object such as a person, animal, or robot. Examples of human behavior include standing up from a chair, stopping to talk on the phone, picking up a cup to drink, putting down a bag and walking away, and getting up from bed.

As mentioned above, a behavior is a series of movements of an object, so it can be considered as one or more movements performed in a specific order. For example, when standing up from a chair, a person can be considered to be performing the movements of sitting, squatting, and standing, in that order.

When a person is standing still and talking on a phone, the actions of the person can be considered to be walking, operating a communication device, talking on the communication device, and operating another communication device, in that order. The communication device could be, for example, a smartphone.

However, the time required for each action that makes up a behavior varies depending on the person, the surrounding environment, etc. Therefore, it cannot be recognized by simple matching processing alone.

Through this process, the inventors have identified the challenges involved in accurately recognizing the behavior of objects such as people, animals, and robots, and have also derived means to solve these challenges.

In other words, the inventors have devised a method for recognizing behavior regardless of the time required for an action by considering the actions and their order as a sequence of multiple, predetermined actions performed in a predetermined order. As a result, the behavior of objects such as people, animals, and robots can be recognized with high accuracy.

Embodiments will be described below with reference to the drawings. Note that in the drawings described below, elements having the same or corresponding functions will be given the same reference numerals, and repeated explanations may be omitted.

(Embodiment)
The configuration of a behavior recognition device 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram illustrating an example of a behavior recognition device.

[Device configuration]
1 is a device for accurately recognizing the behavior of an object. As shown in FIG. 1, the behavior recognition device 10 includes an estimation unit 11 and a behavior determination unit 12.

The estimation unit 11 estimates the movement of a target object for each target object image using target object images corresponding to the target object contained in images acquired in time series. All frames of the target object images may be used, or frames selected at predetermined intervals may be used.

When the same estimated behavior occurs a predetermined number of times in succession, the behavior determination unit 12 determines the behavior of the target object as the estimated behavior. Furthermore, when the order in which the target object's behaviors are determined matches the order in which the predetermined behaviors transition for each behavior representing a series of behaviors of the target object, the behavior determination unit 12 determines the behavior with the matching order as the behavior of the target object.

The number of times (motion confirmation threshold) is determined based on the rate of motion error estimation. The number of times is determined, for example, through experiments or simulations. For example, even when a person makes a fast motion, the same motion continues for several frames, so setting a motion confirmation threshold is effective.

As such, according to this embodiment, behavior can be recognized with high accuracy even if the time required for the behavior varies. A specific explanation will be given using Figures 2A and 2B. Figure 2A is a diagram illustrating the behavior of a person standing up from a chair. Figure 2B is a diagram illustrating the behavior of a person standing still and talking on a phone.

In the example of Figure 2A, the estimation unit 11 first acquires images in chronological order and estimates the person's movements in the order they were acquired. That is, it estimates sitting, crouching, and standing movements.

Next, the behavior determination unit 12 determines the estimated movement if the same estimated movement occurs a preset number of times (movement determination threshold), for example, twice in a row. In the example of Figure 2A, the sitting movement is estimated three times, the squatting movement is estimated two times, and the standing movement is estimated three times, so the movements are determined to be sitting, squatting, and standing.

Next, the behavior determination unit 12 determines that the behavior is a person standing up from a chair because the determined behavior and the order of the determined behaviors match the order of transition of predetermined actions for the behavior of a person standing up from a chair, i.e., sitting, squatting, and standing.

In the example of Figure 2B, to recognize the behavior of a person standing still while talking on a phone, the behavior determination unit 12 first acquires multiple images in chronological order and estimates the person's actions in the order they are acquired. That is, it estimates the actions of walking, operating a communication device, talking on a communication device, and operating a communication device.

Next, the behavior determination unit 12 determines the estimated behavior when the same estimated behavior is repeated a preset number of times (behavior determination threshold), for example, twice in succession in the example of Figure 2B. In the example of Figure 2B, the behavior of walking is estimated twice, the behavior of operating a communication device is estimated twice, and the behavior of talking on a communication device is estimated three times, so the behaviors are determined to be walking, operating a communication device, and talking on a communication device, respectively.

Next, the behavior determination unit 12 determines that the behavior is that of a person stopping and talking on a phone because the determined actions and the order of the determined actions match the order of transition of pre-set actions for the behavior of a person stopping and talking on a phone, i.e., the action of walking, the action of operating a communication device, and the action of talking on the communication device.

As can be seen from the above, even if the time required to perform an action varies from person to person, the action can be determined regardless of the length of time, and behavior can be recognized based on the determined actions and the order in which they were performed, allowing for accurate behavior recognition.

[System configuration]
The configuration of the behavior recognition device 10 in the embodiment will be described in more detail with reference to Fig. 3. Fig. 3 is a diagram illustrating an example of a system including a behavior recognition device.

As shown in FIG. 3, the system 100 in this embodiment includes a behavior recognition device 10, an imaging device 20, and an information processing device 30.

The behavior recognition device 10 is, for example, a CPU (Central Processing Unit), a programmable device such as an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or a circuit equipped with one or more of these, or an information processing device such as a server computer, personal computer, or mobile terminal.

The imaging device 20 outputs two-dimensional or three-dimensional images captured in time series to the behavior recognition device 10. The imaging device 20 is, for example, a camera, or a device equipped with a camera and LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging). Possible cameras include, for example, monocular cameras (e.g., wide-angle cameras, fisheye cameras, omnidirectional cameras), compound-eye cameras (e.g., stereo cameras, multi-cameras), and RGB-D cameras (e.g., depth cameras, ToF cameras).

The information processing device 30 acquires the behavior recognition results (behavior information) from the behavior recognition device 10 and executes predetermined processing using the acquired behavior information. The information processing device 30 is, for example, a device that recognizes human behavior and provides assistance, a device that recognizes human gestures and allows commands to be operated, or a device that recognizes and indicates the form of a person playing a sport (for example, pitching form in baseball, swing form in golf, etc.).

The information processing device 30 is, for example, a CPU, a programmable device such as an FPGA, a GPU, or a circuit equipped with one or more of these, or an information processing device such as a server computer, a personal computer, or a mobile terminal.

In the example of Figure 3, the behavior recognition device 10, imaging device 20, and information processing device 30 are shown separately, but the behavior recognition device 10, imaging device 20, and information processing device 30 may be combined into a single device. Also, the behavior recognition device 10 and imaging device 20 may be combined into a single device.

The behavior recognition device 10 will now be described. The behavior recognition device 10 includes a preprocessing unit 13, an estimation unit 11, a behavior determination unit 12, and a notification unit 16.

The pre-processing unit 13 processes the images captured by the imaging device 20 so that movements and behaviors can be efficiently recognized in subsequent processing. Specifically, the pre-processing unit 13 first acquires images captured in real time by the imaging device 20 in chronological order, or images captured in the past in chronological order from a storage device (not shown).

Next, the pre-processing unit 13 uses existing technology to convert the acquired image into an image that is easier to recognize behavior. Existing technology includes processes such as contrast adjustment and noise removal.

The estimation unit 11 uses target object images corresponding to target objects, such as people, animals, and robots, whose behaviors are the subject of the target object, to estimate the movement of the target object for each target object image. The estimation unit 11 includes an object information extraction unit 14 and a movement estimation unit 15.

The object information extraction unit 14 extracts information about the target object from the image acquired from the pre-processing unit 13. Specifically, the object information extraction unit 14 first acquires the image from the pre-processing unit 13. The object information extraction unit 14 then uses existing technology to extract information about the target object and information about things other than the target object from the target object image. Examples of existing technology include deep learning-based object recognition and pattern matching.

For example, if the target object is a person, information about the person includes the person's skeletal coordinates, facial information, etc. Information about non-human objects is information indicating the name, position, etc. of objects other than humans captured in the target object image.

Note that the process of extracting information about people (information about target objects) and the process of extracting information about non-people (information other than target objects) may be performed in parallel, or one of the processes may be performed first.

The motion estimation unit 15 estimates the motion of the target object for each target object image using information about the target object, or information about the target object and information about things other than the target object. Specifically, the motion estimation unit 15 first obtains information about the target object, or information about the target object and information about things other than the target object, from the object information extraction unit 14.

Next, the movement estimation unit 15 estimates the movement of the target object for each target object image using information about the target object, or information about the target object and information about things other than the target object. The movement estimation unit 15 then outputs movement information representing the estimated movement of the target object to the behavior determination unit 12.

For example, when the target object is a person, movement estimation is performed for each target object image (frame) using existing technologies such as matching processing using human skeletal information and movement classification using time-series learning. Note that when estimating the target object's movement using movement classification using time-series learning, the results of person recognition and object recognition from the past few frames may also be used.

Matching processes using human skeletal information use multiple pieces of skeletal information (for example, combinations of the angles of three skeletal coordinate points) that have been registered in advance. For example, the angles of the right wrist, right elbow, and right shoulder, and the angles of the right wrist, neck, and right elbow are used to determine whether the arm is outstretched or bent, and similarity is calculated for matching.

Movement classification using time series learning extracts features that represent the characteristics of movement from skeletal information and trains them using an RNN (Recurrent Neural Network) machine learning model to classify very short periods of movement (short periods of movement of a few frames, rather than movements that span seconds like behavior).

Motion information is information that identifies the movements that make up a behavior, such as sitting, squatting, walking, standing, operating a smartphone, talking on the phone, holding a cup, drinking from a cup, walking with a bag, putting down the bag, sleeping in bed, sitting on a bed, etc.

Figure 4 is a diagram illustrating an example of action information. In the example of Figure 4, each action is associated with action information "1" to "12."

When the estimated movement occurs consecutively a preset number of times (movement confirmation threshold), the behavior determination unit 12 determines that the movement of the target object is the estimated movement.

Specifically, first, the behavior determination unit 12 acquires movement information from the estimation unit 11. Next, if the same movement information appears consecutively a predetermined number of times (movement determination threshold), the behavior determination unit 12 determines the movement of the target object to be the estimated movement.

Then, if the order in which the actions are determined matches the order in which the actions transition, which is preset for each behavior representing a series of actions of the target object, the behavior determination unit 12 determines that the behavior with the matching order is the behavior of the target object.

Specifically, first, the behavior determination unit 12 acquires confirmed movement information from the movement estimation unit 15. Next, the behavior determination unit 12 refers to the behavior determination information using the confirmed movement information and the order in which the confirmed movement information was acquired, and detects a matching behavior.

Figure 5 is a diagram illustrating an example of behavior determination information. In the example of Figure 5, the behavior determination information associates a behavior, behavior information for identifying the behavior, and information indicating the order of actions (action order). In the example of Figure 5, the behavior information associates "A" to "F" with each behavior. The action order associates each behavior with information indicating the order of actions used to recognize the behavior (the order in which the action information was determined).

The notification unit 16 outputs information representing the detected behavior to the information processing device 30. Specifically, the notification unit 16 first obtains information representing the detected behavior from the behavior determination unit 12.

Next, the notification unit 16 determines whether or not to notify the user of the behavior, based on the predetermined priority associated with each behavior. For example, the notification unit 16 outputs only behaviors with high priority to the information processing device 30. The priority is set to an arbitrary, predetermined priority. The priority is determined, for example, through experiments, simulations, etc.

(Variation 1)
Fig. 6 is a diagram for explaining the behavior determination unit in Modification 1. In the example of Fig. 6, the behavior determination unit 12 has action determination units 61 (61A, 61B, 61C). The number of action determination units 61 is not limited to three. It is determined by the number of actions for which the behavior is determined.

Each operation determination unit 61 (61A, 61B, 61C) has a status management unit 62 (62A, 62B, 62C) and a counter 63 (63A, 63B, 63C).

The status management unit 62 (62A, 62B, 62C) increments the counter's count value when it acquires preset motion information for each motion, and decrements the count value according to the noise tolerance when it acquires motion information other than the preset motion information. When the count value reaches a preset motion confirmation threshold, it confirms the motion of the target object as the estimated motion.

In variant 1, a motion confirmation threshold and a noise tolerance value are set. In this variant, the motion confirmation threshold is a value used to confirm the motion of the target object. The noise tolerance value is a value that indicates the maximum number of times noise acquisition is permitted.

Figure 7 is a diagram illustrating the transition of behavioral confirmation in Variation 1. The example in Figure 7 shows the initial state and operation states 1, 2, and 4 corresponding to operation information "1," "2," and "4." Note that the count value of each counter is 0 in the initial state.

Operation state 1 has state 1T_1 (state where count value = 1) and state 1C (state where count value = 2). In state management unit 62A (operation state 1), the operation confirmation threshold (corresponding to the number of states (number of stages)) is set to 2. The noise tolerance value is also set to 1.

Operation state 2 has state 2T_1 (count value = 1) and state 2C (count value = 2). In state management unit 62B (operation state 2), the operation determination threshold is set to 2. The noise tolerance is also set to 1.

Operation state 4 has state 4T_1 (count value = 1) and state 4C (count value = 2). In state management unit 62C (operation state 4), the operation determination threshold is set to 4. The noise tolerance is also set to 1.

In the example in Figure 7, first, when operation information "1" is acquired once, the system transitions from the initial state to state 1T_1. Next, when operation information "1" is acquired twice, the system transitions to state 1C, and operation 1 is confirmed. In other words, when the count value corresponding to operation state 1 becomes 2, operation 1 is confirmed.

Next, after action 1 is confirmed, if action information "2" is acquired once, the system transitions from action state 1 to state 2T_1. Next, if action information "2" is acquired twice, the system transitions to state 2C and action 2 is confirmed. In other words, when the count value corresponding to action state 2 reaches 2, action 2 is confirmed.

Next, after actions 1 and 2 are confirmed, if action information "4" is acquired once, the system transitions from action state 2 to state 4T_1. Next, if action information "4" is acquired twice, the system transitions to state 4C and action 4 is confirmed. In other words, when the count value corresponding to action state 4 reaches 2, action 4 is confirmed.

Then, if the order in which the actions are determined matches the order in which the actions transition, which is preset for each action representing a series of actions of the target object, the behavior determination unit 12 determines the behavior with the matching order as the behavior of the target object. Note that the count value of each counter is initially 0.

Figure 8 is a diagram illustrating the reduction of the effects of noise in variant 1. Assume that the motion information is acquired in the order shown in Figure 8. In this case, in the example of Figure 8, once motion 1 is confirmed and motion information "2" is acquired once, the motion state transitions from state 1 to state 2T_1.

Next, if motion information "2" is acquired twice, the system transitions to state 2C and confirms motion 2. However, in the example of Figure 8, motion information "6", which is noise, is then acquired. In this case, since the motion information is not "2" or "4", the system transitions to state 2T_1.

Next, in the example of Figure 8, operation information "2" is acquired again, causing a transition to state 4C. After that, operation information "4" is acquired, causing a transition from operation state 2 to state 4T_1.

According to variant 1, behavior can be recognized with high accuracy even when erroneously estimated motion information is acquired (when noise is acquired). Therefore, according to variant 1, behavior can be recognized with high accuracy even when erroneously estimated motion information is acquired (when noise is acquired).

However, in the example in Figure 8, the noise tolerance value is 1, so if noise is detected twice in a row, the system will transition to the initial state and the count value will be set to 0.

(Variation 2)
Fig. 9 is a diagram for explaining the transition of the action determination of the behavior in Modification 2. In the example of Fig. 9, an initial state and action states 1, 2, and 4 corresponding to action information "1", "2", and "4" are shown.

Operation state 1 has state 1T_1 (state where count value = 1), state 1T_2 (state where count value = 2), and state 1C (state where count value = 3). In state management unit 62A (operation state 1), the operation confirmation threshold (corresponding to the number of states (number of stages)) is set to 3. The noise tolerance value is also set to 2.

Operation state 2 has state 2T_1 (count value = 1), state 2T_2 (count value = 2), and state 2C (count value = 3). In state management unit 62B (operation state 2), the operation determination threshold is set to 3. The noise tolerance is also set to 2.

Operation state 4 has state 4T_1 (count value = 1), state 4T_2 (count value = 2), and state 4C (count value = 3). In state management unit 62C (operation state 4), the operation determination threshold is set to 3. The noise tolerance is also set to 2.

In the example of Figure 9, first, when operation information "1" is acquired once, the system transitions from the initial state to state 1T_1. Next, when operation information "1" is acquired twice, the system transitions from state 1T_1 to state 1T_2. Next, when operation information "1" is acquired three times, the system transitions to state 1C, and operation 1 is confirmed.

Next, after action 1 is confirmed, if action information "2" is acquired once, the system will transition from action state 1 to state 2T_1. Next, if action information "2" is acquired twice, the system will transition from state 2T_1 to state 2T_2. Next, if action information "2" is acquired three times, the system will transition to state 2C and action 2 will be confirmed.

Next, after actions 1 and 2 are confirmed, if action information "4" is acquired once, the state changes from action state 2 to state 4T_1. Next, if action information "4" is acquired twice, the state transitions from state 4T_1 to state 4T_2. Next, if action information "4" is acquired three times, the state transitions to state 4C, and action 4 is confirmed.

Figure 10 is a diagram illustrating the reduction of the effects of noise in variant 2. Assume that the motion information is acquired in the order shown in Figure 10. In this case, once motion 1 is confirmed and motion information "2" is acquired once, the motion state transitions from state 1 to state 2T_1.

Next, if motion information "2" is acquired twice, the system transitions to state 2T_1, and if motion information "2" is acquired twice, the system transitions to state 2C and confirms motion 2. However, in the example of Figure 10, motion information "5", which is noise, is then acquired twice in a row. In this case, since the motion information is not "2" or "4", the system transitions to state 2T_2 and then state 2T_1.

However, in the example of Figure 10, after acquiring motion information "2" twice in succession, motion information "4" is acquired, resulting in a transition from motion state 2 to state 4T_1. Therefore, according to variant example 2, even when erroneously estimated motion information is acquired (when noise is acquired), behavior can be recognized with high accuracy.

However, in the example in Figure 10, the noise tolerance value is 2, so if noise is detected three times in a row, the system will transition to the initial state and the count value will be set to 0.

(Variation 3)
Fig. 11 is a diagram for explaining the transition of the action determination of the behavior in Modification 3. In the example of Fig. 11, an initial state and action states 1, 2, and 4 corresponding to action information "1", "2", and "4" are shown.

Operating state 1 has state 1T_1 (count value = 1), state 1T_2 (count value = 2), and state 1C (count value = 3). In state management unit 62A (operating state 1), the operation determination threshold (corresponding to the number of states (number of stages)) is set to 3. The noise tolerance is also set to 1. Therefore, regardless of the state of the counter, it will transition to state 1T_1 (count value = 1).

Operating state 2 has state 2T_1 (count value = 1), state 2T_2 (count value = 2), and state 2C (count value = 3). In state management unit 62B (operating state 2), the operation determination threshold is set to 3. The noise tolerance is also set to 1. Therefore, regardless of the state of the counter, it will transition to state 2T_1 (count value = 1).

Operating state 4 has state 4T_1 (count value = 1), state 4T_2 (count value = 2), and state 4C (count value = 3). In state management unit 62C (operating state 4), the operation determination threshold is set to 3. The noise tolerance is also set to 1. Therefore, regardless of the state of the counter, it will transition to state 4T_1 (count value = 1).

In the example of Figure 11, first, when operation information "1" is acquired once, the system transitions from the initial state to state 1T_1. Next, when operation information "1" is acquired twice, the system transitions from state 1T_1 to state 1T_2. Next, when operation information "1" is acquired three times, the system transitions to state 1C, and operation 1 is confirmed.

Next, after action 1 is confirmed, if action information "2" is acquired once, the system will transition from action state 1 to state 2T_1. Next, if action information "2" is acquired twice, the system will transition from state 2T_1 to state 2T_2. Next, if action information "2" is acquired three times, the system will transition to state 2C and action 2 will be confirmed.

Next, after actions 1 and 2 are confirmed, if action information "4" is acquired once, the state changes from action state 2 to state 4T_1. Next, if action information "4" is acquired twice, the state transitions from state 4T_1 to state 4T_2. Next, if action information "4" is acquired three times, the state transitions to state 4C, and action 4 is confirmed.

Therefore, according to variant example 3, behavior can be recognized with high accuracy even when erroneously estimated motion information is acquired (when noise is acquired).

(Variation 4)
12 is a diagram illustrating a configuration for executing processes for determining multiple behaviors in parallel. In the example of FIG. 12, behaviors A, B, and C are executed in parallel to reduce the time required to determine behaviors A, B, and C.

In the example of Figure 12, the behavior determination unit 12A for behavior A has operation determination units 61A, 61B, and 61C as described above. The behavior determination unit 12B for behavior B has operation determination units 61D, 61E, and 61F. The behavior determination unit 12C for behavior C has operation determination units 61D, 61E, 61F, and 61D.

The movement determination unit 61A determines the sitting movement (movement 1). The movement determination unit 61B determines the crouching movement (movement 2). The movement determination unit 61C determines the standing movement (movement 4).

The action confirmation unit 61D confirms the action of walking (action 3). The action confirmation unit 61E confirms the action of operating a smartphone (action 5). The action confirmation unit 61F confirms the action of talking on a phone (action 6).

Therefore, according to variant example 4, authentication can be completed in a short time by performing multiple behavior recognitions in parallel. However, the number of behavior determination units 12 and the types of behaviors that are performed in parallel are not limited to the example in Figure 12.

[Device operation]
Next, the operation of the behavior recognition device in the embodiment and modified examples 1 to 4 will be described with reference to FIG. 13 . FIG. 13 is a diagram for explaining an example of the operation of the behavior recognition device. In the following description, the diagram will be referred to as appropriate. Furthermore, in the embodiment and modified examples 1 to 4, the behavior recognition method is implemented by operating the behavior recognition device. Therefore, the description of the behavior recognition method in the embodiment will be replaced by the following description of the operation of the behavior recognition device.

As shown in FIG. 13, first, the pre-processing unit 13 acquires images captured in real time by the imaging device 20, or images captured in past time series from a storage device (not shown) (step A1).

Next, the pre-processing unit 13 processes the image captured by the imaging device 20 so that movements and behaviors can be efficiently recognized in subsequent processing (step A2).

Next, the object information extraction unit 14 extracts information about the target object from the image acquired from the preprocessing unit 13 (step A3). Specifically, in step A3, the object information extraction unit 14 uses existing technology to extract information about the target object and information about things other than the target object from the target object image.

Next, the movement estimation unit 15 estimates the movement of the target object for each target object image using information about the target object, or information about the target object and information about things other than the target object (step A4).

Specifically, in step A4, the movement estimation unit 15 estimates the movement of the target object for each target object image using information about the target object, or information about the target object and information about something other than the target object.

Next, if the estimated movement occurs consecutively a preset number of times (movement confirmation threshold), the behavior determination unit 12 determines the movement of the target object to be the estimated movement (step A5).

Specifically, in step A5, the behavior determination unit 12 uses the determined operation information and the order in which the determined operation information was acquired to refer to the behavior determination information and detect a matching behavior. Note that in step A5, the behavior may be determined by executing the processes shown in Variations 1 to 4 described above.

Next, the notification unit 16 outputs information representing the detected behavior to the information processing device 30 (step A6). Specifically, in step A6, the notification unit 16 determines whether to notify the user of the behavior, depending on the pre-set priority associated with each behavior. The notification unit 16 outputs, for example, only behaviors with high priority to the information processing device 30.

[Effects of the embodiment and modifications 1 to 4]
According to the embodiment, a behavior can be considered as a plurality of different predetermined actions performed in a predetermined order, and by using the actions and their order, the behavior can be recognized regardless of the time required for the actions.

Furthermore, according to variants 1 to 4, behavior can be recognized with high accuracy even when erroneously estimated motion information is acquired (when noise is acquired).

[program]
The program in the embodiment and modifications 1 to 4 may be a program that causes a computer to execute steps A1 to A6 shown in Fig. 13. By installing and executing this program in a computer, the behavior recognition device and behavior recognition method in the embodiment and modifications 1 to 4 can be realized. In this case, the processor of the computer functions as a preprocessing unit 13, an estimation unit 11 (object information extraction unit 14, motion estimation unit 15), a behavior determination unit 12, and a notification unit 16, and performs processing.

Furthermore, the programs in the embodiment and variants 1 to 4 may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the preprocessing unit 13, estimation unit 11 (object information extraction unit 14, motion estimation unit 15), behavior determination unit 12, and notification unit 16.

[Physical configuration]
Here, a computer that realizes the behavior recognition device by executing the program in the embodiment and modifications 1 to 4 will be described with reference to Fig. 14. Fig. 14 is a block diagram showing an example of a computer that realizes the behavior recognition device in the embodiment and modifications 1 to 4.

As shown in FIG. 14, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader/writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so that they can communicate data with each other. Note that the computer 110 may include a GPU or FPGA in addition to or instead of the CPU 111.

The CPU 111 performs various calculations by expanding the programs (codes) in the embodiments and variants 1 to 4 stored in the storage device 113 into the main memory 112 and executing them in a predetermined order. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). The programs in the embodiments and variants 1 to 4 are provided in a state stored on a computer-readable recording medium 120. The programs in the embodiments and variants 1 to 4 may also be distributed over the Internet connected via the communication interface 117. The recording medium 120 is a non-volatile recording medium.

Specific examples of the storage device 113 include a hard disk drive and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

The data reader/writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes the processing results of the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, and optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

Note that the behavior recognition device in the embodiment and variants 1 to 4 can also be realized using hardware corresponding to each part, rather than a computer on which a program is installed. Furthermore, the behavior recognition device may be partially realized by a program and the remaining part by hardware.

[Note]
The following supplementary notes are further disclosed regarding the above-described embodiments: Part or all of the above-described embodiments and Modifications 1 to 4 can be expressed by (Supplementary Note 1) to (Supplementary Note 9) described below, but are not limited to the following descriptions.

(Appendix 1)
an estimation unit that estimates a motion of a target object for each target object image by using target object images corresponding to the target object included in images acquired in time series;
a behavior determination unit that, when the same estimated behavior occurs consecutively a preset number of times, determines the behavior of the target object to be the estimated behavior, and, when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object, determines the behavior with the matching order as the behavior of the target object;
A behavior recognition device having the above.

(Appendix 2)
2. The behavior recognition device according to claim 1,
The behavior determination unit increments a count value of a counter when it acquires predetermined motion information for each motion, subtracts the count value according to a noise tolerance value when it acquires motion information other than the predetermined motion information, and determines the motion of the target object as an estimated motion when the count value reaches a predetermined motion determination threshold.

(Appendix 3)
3. The behavior recognition device according to claim 1,
A behavior recognition device having a notification unit that determines whether or not to notify a user of the behavior according to a preset priority.

(Appendix 4)
The computer
an estimation step of estimating a motion of a target object for each target object image using target object images corresponding to the target object included in images acquired in time series;
a behavior determination step of determining the behavior of the target object as the estimated behavior when the same estimated behavior occurs consecutively a predetermined number of times, and determining the behavior with the matching order as the behavior of the target object when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object;
A behavior recognition method that performs

(Appendix 5)
5. The behavior recognition method according to claim 4, further comprising:
The behavior determination step increments a count value of a counter when predetermined motion information for each motion is acquired, decrements the count value according to a noise tolerance when motion information other than the predetermined motion information is acquired, and determines the motion of the target object as an estimated motion when the count value reaches a predetermined motion determination threshold.

(Appendix 6)
6. The behavior recognition method according to claim 4 or 5,
A behavior recognition method comprising: a notification unit that determines whether or not to notify a user of the behavior according to a preset priority.

(Appendix 7)
On the computer,
an estimation step of estimating a motion of a target object for each target object image using target object images corresponding to the target object included in images acquired in time series;
a behavior determination step of determining the behavior of the target object as the estimated behavior when the same estimated behavior occurs consecutively a predetermined number of times, and determining the behavior with the matching order as the behavior of the target object when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object;
A program that executes the following.

(Appendix 8)
8. The program of claim 7,
The behavior determination step is a program that adds a count value of a counter when predetermined motion information is acquired for each motion, subtracts the count value according to a noise tolerance value when motion information other than the predetermined motion information is acquired, and determines the motion of the target object as an estimated motion when the count value reaches a predetermined motion determination threshold.

(Appendix 9)
9. The program according to claim 7 or 8,
a notification step for determining whether or not to notify the user of the behavior according to a preset priority.

The invention has been described above with reference to embodiments, but the invention is not limited to the above-described embodiments. Various modifications that would be understandable to a person skilled in the art can be made to the configuration and details of the invention within the scope of the invention.

The above description makes it possible to accurately recognize the behavior of an object. It is also useful in fields where it is necessary to recognize the behavior of an object.

REFERENCE SIGNS LIST 10 Behavior recognition device 11 Estimation unit 12, 12A, 12B, 12C Behavior determination unit 13 Preprocessing unit 14 Object information extraction unit 15 Action estimation unit 16 Notification unit 20 Imaging device 30 Information processing device 61, 61A, 61B, 61C, 61D, 61E, 61F Action determination unit 62, 62A, 62B, 62C Status management unit 63, 63A, 63B, 63C Counter 110 Computer 111 CPU
112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader/writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims (6)

an estimation means for estimating a motion of a target object for each target object image by using target object images corresponding to the target object included in images acquired in time series;
a behavior determination means for determining the behavior of the target object as the estimated behavior when the same estimated behavior occurs a predetermined number of times in succession, and determining the behavior with the matching order as the behavior of the target object when the order in which the target object's behaviors are determined matches the order in which predetermined behaviors transition for each behavior representing a series of behaviors of the target object ;
The behavior determination means increments a count value of a counter when it acquires predetermined motion information for each motion, decrements the count value according to a noise tolerance value when it acquires motion information other than the predetermined motion information, and determines the motion of the target object as an estimated motion when the count value reaches a predetermined motion determination threshold.
Behavior recognizer. The behavior recognition device according to claim 1 ,
A behavior recognition device having a notification means for determining whether or not to notify a user of the behavior according to a preset priority. using target object images corresponding to the target object included in the images acquired in time series, estimating a movement of the target object for each of the target object images;
When the same estimated action is performed a predetermined number of times in succession, the action of the target object is determined to be the estimated action, and when the order in which the actions of the target object are determined matches the order in which actions that are preset for each behavior representing a series of actions of the target object transition, the behavior with the matching order is determined to be the behavior of the target object;
Furthermore, when predetermined motion information for each motion is acquired, the count value of the counter is incremented, and when motion information other than the predetermined motion information is acquired, the count value is decremented according to a noise tolerance value, and when the count value reaches a predetermined motion determination threshold, the motion of the target object is determined to be the estimated motion.
Behavioral recognition methods. The behavior recognition method according to claim 3 ,
A behavior recognition method that determines whether or not to notify a user of the behavior according to a preset priority. On the computer,
using target object images corresponding to the target object included in the images acquired in time series, estimating the motion of the target object for each of the target object images;
When the same estimated action is performed a predetermined number of times in succession, the action of the target object is determined to be the estimated action; when the order in which the actions of the target object are determined matches the order in which actions preset for each behavior representing a series of actions of the target object transition, the behavior with the matching order is determined to be the behavior of the target object ;
Furthermore, when predetermined motion information for each motion is acquired, the count value of the counter is incremented, and when motion information other than the predetermined motion information is acquired, the count value is decremented according to a noise tolerance value. When the count value reaches a predetermined motion determination threshold, the motion of the target object is determined to be the estimated motion.
program. 6. The program according to claim 5 ,
A program that determines whether or not to notify the user of the behavior according to a preset priority.