JP2019091260A - Apparatus, method and program for time-series analysis - Google Patents

Abstract

To provide an apparatus for time-series analysis capable of performing automatic analysis in accordance with a feature of the data by using time-series data of actions of a plurality of persons.SOLUTION: An apparatus for time-series analysis 1 in one embodiment of the present invention has an information acquisition part 111 that acquires time-series information of a first-user's feature and second-user's feature, a feature selection part 113 that selects a feature of a learning subject based on duration time occurring at the same time and different time for the first-user's feature and the second-user's feature, in the time-series information the information acquisition part 111 has acquired, and a learning part 114 that performs learning using the time-series information of the learning subject feature the feature selection part 113 has selected.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a time-series analysis device, a time-series analysis method, and a time-series analysis program for analyzing human time-series behavior.

  In recent years, research has been conducted to predict the outcome of behavior by analyzing human behavior. Non-Patent Document 1 describes a technology for estimating the mental state of a customer by analyzing an indicator combining a plurality of non-verbal behaviors such as the customer's gaze and attitude in communication between a store clerk and the customer. It is done.

Hidehito Honda and 3 others, "Career-customer interaction in face-to-face sales situations: preference estimation from non-verbal behavior", [online], June 7, 2016, Japan Society for Artificial Intelligence, [October 6, 2017 Day Search], Internet <URL: https://kaigi.org/jsai/webprogram/2016/pdf/47.pdf>

  However, the technology described in Non-Patent Document 1 focuses only on the behavior of the person who is the customer, and does not consider the behavior of the person who is the store clerk. In addition, Non-Patent Document 1 shows a method of performing analysis manually, and does not show a method of performing analysis automatically by an apparatus.

  The present invention has been made in view of the above-mentioned point, and a time-series analyzer capable of automatically analyzing according to the characteristics of data using time-series data of a plurality of human actions, a time-series analyzer An object of the present invention is to provide an analysis method and a time series analysis program.

  In the time-series analysis apparatus according to the first aspect of the present invention, an acquisition unit for acquiring time-series information of a feature of a first user and a characteristic of a second user, and the time-series information acquired by the acquisition unit A selection unit for selecting a feature to be learned based on a duration when the feature of one user and the feature of the second user occur at the same time and different times; and the learning selected by the selection unit And a learning unit that performs learning using time-series information of the target feature.

  The time series analysis device predicts a result based on the time series information acquired by the acquisition unit by inputting the time series information acquired by the acquisition unit to a model generated by the learning unit by the learning. You may further have an analysis part.

  The selection unit may calculate, as the duration, an average value of a time range in which the feature of the first user and the feature of the second user occur at the same time and different times.

  The selection unit may calculate the duration by Cross-Recurrence Quantification Analysis.

  The selection unit may select the feature of the learning target based on the number of combinations of the feature of the first user and the feature of the second user whose duration is equal to or greater than a predetermined threshold. .

  The acquisition unit may acquire body parts of the first user and the second user as the features of the first user and the second user.

  The time series analysis apparatus converts the position of the body part of the first user into coordinates based on the center of the body of the first user, and the position of the body part of the second user is the second user The image processing apparatus may further include a conversion unit that converts coordinates of the body center of the image.

  The learning unit may perform learning as to whether the result of the interaction between the first user and the second user is positive.

  The learning unit may perform learning as to whether or not the action pattern of the first user and the action pattern of the second user are the same.

  The learning unit may repeat learning using time-series information of the feature to be learned within a time window of a predetermined length while shifting the time window.

  The acquisition unit may acquire the language spoken by the first user and the second user as the feature of the first user and the second user.

  The acquisition unit extracts a word from a language spoken by the first user and the second user, and clusters the generated word by clustering the features of the first user and the second user. It may be acquired as

  In the time-series analysis method of the second aspect of the present invention, the computer executes the steps of acquiring time-series information of the features of the first user and the features of the second user, and the time acquired in the acquiring step Selecting, on the basis of sequence information, a feature to be learned on the basis of a duration when the feature of the first user and the feature of the second user occur at the same time and at different times; Performing learning using time-series information of the feature to be learned selected in the step.

  The time-series analysis program according to the third aspect of the present invention is an acquisition unit that acquires time-series information of a feature of a first user and a feature of a second user in a processor, and the time-series information acquired by the acquisition unit A selecting unit that selects a feature to be learned based on a duration when the feature of the first user and the feature of the second user occur at the same time and different times; and the selecting unit selects It functions as a learning unit that performs learning using time-series information of the feature to be learned.

  According to the present invention, analysis can be performed automatically using time series data of a plurality of people's actions.

It is a schematic diagram of the time-series analysis system which concerns on 1st Embodiment. It is a schematic diagram of the co-occurrence relation of the movement between users. It is a block diagram of a time-series analysis device concerning a 1st embodiment. It is a schematic diagram which shows the position of the user's body site | part which the information acquisition part which concerns on 1st Embodiment acquires. It is a figure which shows the repetitive plot which the feature selection part which concerns on 1st Embodiment produces | generates. It is a figure which shows TT (Trapping Time) index which the feature selection part which concerns on 1st Embodiment calculates. It is a figure which shows the flowchart of the time-series analysis method by the time-series analyzer which concerns on 1st Embodiment. It is a figure which shows the flowchart of the learning process which concerns on 1st Embodiment. It is a figure which shows the flowchart of the analysis process which concerns on 1st Embodiment. It is a figure which shows the flowchart of the learning process which concerns on 2nd Embodiment. It is a figure which shows the flowchart of the learning process which concerns on 3rd Embodiment. It is a figure which shows the flowchart of the learning process which concerns on 4th Embodiment.

First Embodiment
FIG. 1 is a schematic view of a time-series analysis system S according to the present embodiment. The time series analysis system S includes a time series analysis device 1, a display unit 14, an imaging unit 15, and an audio input unit 16. The time series analysis system S may include devices such as other servers and terminals.

  The display unit 14 is a display device capable of displaying information received from the time series analysis device 1 such as a liquid crystal display or a CRT (Cathode Ray Tube) display. The display unit 14 displays the result of analysis by the time series analyzer 1 and the like.

  The imaging unit 15 is an imaging device that captures a user and acquires an image. The imaging unit 15 includes an imaging element such as a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. The imaging unit 15 transmits information indicating images acquired in time series to the time series analysis device 1. The images acquired in time series may be moving images or a group of still images captured at predetermined time intervals. The imaging unit 15 is provided at a position at which the whole body or a part of the body of one or more users to be analyzed can be imaged. A plurality of imaging units 15 may be provided.

  The voice input unit 16 is a microphone for acquiring voices uttered by the user. The voice input unit 16 transmits, to the time series analysis device 1, information indicating the voices acquired in time series. The voice input unit 16 is provided at a position where voices of one or more users to be analyzed can be obtained. A plurality of voice input units 16 may be provided.

  The time-series analysis device 1 is a device configured by a single computer or a cloud that is a collection of computer resources. The time-series analysis device 1 performs learning using at least one of time-series images acquired by the imaging unit 15 and time-series speech acquired by the voice input unit 16. Furthermore, the time-series analysis device 1 analyzes the result such as prediction by applying a model obtained by learning to at least one of the image acquired by the imaging unit 15 and the sound acquired by the voice input unit 16.

[Description of co-occurrence relationship of movement]
FIG. 2 is a schematic view of a co-occurrence relation of movement between users. FIG. 2 illustrates the interaction between the seller U1 (first user) and the buyer U2 (second user). The purpose of the seller U1 is to persuade the buyer U2 to purchase a product.

  The reaction of the buyer U2 who has heard the story of the seller U1 differs depending on whether or not it shows interest. The buyer U2 responds to the description of the seller U1 if he is interested, the reaction of the body movement tends to be long and the movement time tends to be long, and if he is not interested, the reaction of the body movement is small. The movement time tends to be short. The tendency of reaction changes depending on whether or not you show interest in interactions such as meetings and conversations as well as sales of products.

  For example, as shown in FIG. 2, after a particular body part of seller U1 giving an explanation moves, if a buyer U2 is interested in the explanation, a specific body part of buyer U2 will move, the buyer The particular body part of the buyer U2 does not move if U2 is not interested in the explanation. Since the reaction of the buyer U2 to the description of the seller U1 occurs after the buyer U2 thinks, a delay (that is, a time delay) occurs from the description of the seller U1.

  Although there are a large number of combinations of body parts of two users, it is a combination of some of them that changes the tendency of the reaction depending on whether or not they show interest. The time-series analysis device 1 according to the present embodiment selects a body part (feature) important for analysis of the result based on the co-occurrence relation in which the time delay of the movement of two users is considered, and the selected body part Learn and analyze using

  As described above, since the time-series analysis apparatus 1 selectively uses body parts (features) important for analysis of the results for learning and analysis, the accuracy of learning and analysis can be enhanced, and the time taken for calculation is shortened. it can.

  Here, although the time-series analysis device 1 analyzes sales of goods between users as an example, the present invention is not limited to this, and it is possible to determine whether the result is positive or not, such as contract, agreement, and persuasion, among arbitrary users Interaction can be analyzed.

[Configuration of Time Series Analyzer 1]
FIG. 3 is a block diagram of the time series analysis device 1 according to the present embodiment. In FIG. 3, arrows indicate the flow of main data, and there may be flows of data other than those shown in FIG. In FIG. 3, each block is not a hardware (apparatus) -based configuration but a function-based configuration. Thus, the blocks shown in FIG. 3 may be implemented in a single device, or may be implemented separately in multiple devices. Transfer of data between the blocks may be performed via any means such as a data bus, a network, a portable storage medium, and the like.

  The time series analyzer 1 includes a control unit 11, an interface 12, and a storage unit 13. The control unit 11 includes an information acquisition unit 111, an information conversion unit 112, a feature selection unit 113, a learning unit 114, and an analysis unit 115.

  The interface 12 is an interface for exchanging signals with the display unit 14, the imaging unit 15, and the voice input unit 16. The interface 12 performs predetermined processing on the signals received from the display unit 14, the imaging unit 15, and the voice input unit 16 to acquire data, and inputs the acquired data to the control unit 11. The interface 12 also performs predetermined processing on the data input from the control unit 11 to generate a signal, and transmits the generated signal to the display unit 14, the imaging unit 15, and the voice input unit 16.

  The storage unit 13 is a storage medium including a read only memory (ROM), a random access memory (RAM), a hard disk drive, and the like. The storage unit 13 stores in advance a program to be executed by the control unit 11.

  The control unit 11 is a processor such as a central processing unit (CPU), for example, and executes the program stored in the storage unit 13 to obtain an information acquisition unit 111, an information conversion unit 112, a feature selection unit 113, and a learning unit 114. And functions as an analysis unit 115. At least a part of the functions of the control unit 11 may be performed by an electric circuit. In addition, at least a part of the functions of the control unit 11 may be executed by a program executed via a network.

  The time-series analyzer 1 according to the present embodiment is not limited to the specific configuration shown in FIG. For example, the time-series analyzer 1 is not limited to one device, and may be configured by connecting two or more physically separated devices in a wired or wireless manner.

  The information acquisition unit 111 acquires time-series image information acquired by the imaging unit 15. The information acquisition unit 111 may acquire image information captured from the imaging unit 15 each time, or may collectively acquire image information of a predetermined time range from the imaging unit 15. Further, the information acquisition unit 111 may read and acquire the image information stored in the storage unit 13. When using speech for learning and analysis, the information acquisition unit 111 may acquire time-series speech information acquired by the speech input unit 16.

  The information acquisition unit 111 calculates the time-series position of each body part of each user by applying the human body detection technology to the acquired time-series image information. Known techniques can be used as human body detection techniques.

  FIG. 4 is a schematic view showing the position of the body part P of the user acquired by the information acquisition unit 111 according to the present embodiment. The position of each body part P and the position of the center C of the human body are shown in FIG. The information conversion unit 112 uses the main joints of the human body as the body part P. Not limited to this, as the body part P, a predetermined part that can be detected from an image obtained by imaging a human body can be used.

  The information acquisition unit 111 calculates the position of each body part P at a certain point in time by human body detection technology. The information acquisition unit 111 calculates the position of the center C (for example, the center of gravity) of the human body from the calculated position of the body part P. The information acquisition unit 111 causes the storage unit 13 to store the position of the body part P and the position of the center C calculated for each time point of the time-series image as time-series information.

  The information conversion unit 112 performs feature space conversion on the position of each body part P calculated by the information acquisition unit 111. In the human body detection technology, there are various cases where coordinates in a space common to all users or in different spaces are calculated as the position of the body part P due to the influence of the environment of data acquisition and acquisition equipment. The feature space conversion means converting the position of the body part P from coordinates in a common space or different spaces to coordinates in a space for each user. Thereby, the similarity of the movement of the body part between the users can be determined.

  Specifically, the information conversion unit 112 performs feature space conversion of three-dimensional coordinates of the body part P using Expression (1).

  For each of the x coordinate, y coordinate, and z coordinate in equation (1), the appended Norm indicates the transformed coordinates of the body part P, and the appended θ indicates the coordinates before conversion of the body part P. , BodyCenter attached indicates the coordinates of the center C. The three-dimensional coordinates of the body part P are converted into coordinates based on the center C of each user according to Equation (1). In the following, processing is performed using the coordinates after conversion as the position of each body part P.

  The feature selection unit 113 selects a body part P (feature) that is important for the analysis of the result based on the movement relationship of the body part P between the users in consideration of the time delay. In order to evaluate the relationship of movement of body part P between users, first, feature selection unit 113 repeats CRQA (Cross Recurrence Quantification Analysis) for each combination of body parts P between users. Generate a plot (Recurrence Plot).

  Here, the x coordinate, y coordinate and z coordinate of the body part P are treated as independent features. That is, the feature selection unit 113 combines the x-coordinate of the seller U1's right hand with the x-coordinate of the buyer U2's right hand, the combination of the x-coordinate of the seller U1's right hand and the y-coordinate of the buyer U2's right hand, Each body part P, such as the combination of the x coordinate of the right hand of the person U1 and the z coordinate of the right hand of the buyer U2, the combination of the x coordinate of the right hand of the seller U1 and the x coordinate of the right elbow of the buyer U2, etc. Generate repeated plots of all combinations of each coordinate of.

  FIG. 5 is a diagram showing a repeated plot A generated by the feature selection unit 113 according to the present embodiment for a combination of one body part P. As shown in FIG. The repeated plot A in FIG. 5 is, for example, directed to the combination of the x-coordinate of the right hand of the seller U1 and the x-coordinate of the right hand of the buyer U2. The horizontal axis of repeated plot A represents the time for seller U1, and the vertical axis represents the time for buyer U2. On the repeated plot A, a co-occurrence region A1 is generated which indicates that the positions of the time series of both the body part P of the seller U1 and the body part P of the purchaser U2 are changed (that is, moved). . For example, the repeated plot A represents that the x-coordinate of the right hand of the buyer U2 has changed under the influence of the change of the x-coordinate of the right hand of the seller U1. To generate the repeated plot A, use known optimization techniques. In addition, relevant parameters may be randomly selected at a plurality of times, and optimal parameters may be used by experiments.

  In the repeated plot A, both the body part P of the seller U1 and the body part P of the buyer U2 are moving while shifting the time of the seller U1 and the time of the buyer U2 (for example, the amount of change in position is predetermined) Time) or more) is plotted. For example, it is assumed that the body part P of the seller U1 moves from t1 seconds to t2 seconds, and the body part P of the buyer U2 moves from t3 seconds to t4 seconds and t5 seconds to t6 seconds. In this case, as shown in FIG. 5, the feature selection unit 113 shares portions where t1 seconds to t2 seconds of the seller U1 and t3 seconds to t4 seconds and t5 seconds to t6 seconds of the buyer U2 are common. It is generated on the repeated plot A as the origin area A1. Although only a part of the co-occurrence area A1 is shown in FIG. 5 for visibility, more co-occurrence areas A1 may be generated.

  Such repeated plot A extracts a co-occurrence relation in consideration of temporal delays in combinations of body parts P between users. Therefore, based on the repeated plot A, the feature selection unit 113 generates the relation that the movement of the specific body part P of the buyer U2 occurs at the same time and a different time as the movement of the specific body part P of the seller U1. The presence can be determined.

  Then, the feature selection unit 113 calculates a TT (Trapping Time) index of CRQA using the equation (2) with respect to the repeated plot A generated for each combination of body parts P between users.

  In equation (2), TT represents a TT index, v represents the vertical time length of the co-occurrence region A1 of the repeated plot A, and P (v) represents a frequency distribution of v. The TT index is the average time length in the vertical direction of the co-occurrence area A1 of the repeated plot A. That is, the TT index is an index indicating the duration of the occurrence (co-occurrence) of the movement of the buyer U2 with respect to the movement of the seller U1.

  The TT index is an index indicating that the movement of the body part P between users occurs simultaneously or delayed. The larger the TT index calculated for the combination of body parts P between certain users, the higher the frequency or the longer the occurrence of simultaneous or delayed movement of the body parts P between users. Therefore, the size of the TT index corresponds to the strength of the relationship of the body part P between the users.

  FIG. 6 is a view showing a distribution map B of the TT index calculated by the feature selection unit 113 according to the present embodiment for each combination of body parts P between users. The distribution diagram B is used herein to explain the TT index, and the feature selection unit 113 does not have to generate it.

  The horizontal axis of the distribution diagram B represents the number identifying each coordinate of each body part P of the seller U1, and the vertical axis represents the number identifying each coordinate of each body part P of the buyer U2. Although FIG. 6 shows 20 body parts P as the horizontal axis and the vertical axis for visibility, in fact, the horizontal axis and the vertical axis correspond to the number of body parts P used for the calculation. The distribution diagram B represents a combination of body parts P in which the TT index is equal to or greater than a predetermined threshold with a black cell B1, and a combination of body parts P with a TT index less than the predetermined threshold in a white cell B2.

  The distribution diagram B shows the reaction response of the movement of each body part P of the buyer U2 to the movement of each body part P of the seller U1. For example, with respect to the coordinates of the body part P of the number 10 of the seller U1, the TT index is equal to or more than the predetermined threshold with respect to the numbers 2, 7, 10, 13, 16 (total five) of the buyer U2. The body part P thus associated with the movement of the many other body parts P is important for the analysis of the results. Therefore, the feature selection unit 113 selects a body part P (feature) to be used for analysis of the result based on the TT index.

  Specifically, the feature selection unit 113 selects a body part P to be used for analysis of the result based on the number of combinations of body parts P between users whose TT index is equal to or greater than a predetermined threshold value. More specifically, the feature selection unit 113 uses the body part P of the seller U1 whose cell B1 whose TT index is equal to or more than a predetermined threshold corresponds to a predetermined number or more of columns for analysis of the result. Select as site P. Alternatively, the feature selection unit 113 selects a predetermined number of body parts P of the seller U1 as body parts P to be used for analysis of results in descending order of the number of cells B1 whose TT index is equal to or more than a predetermined threshold.

  With such a configuration, the time-series analyzer 1 can selectively use body parts P (features) that cause many movements with respect to the other party for learning and analysis. The selection condition of the body part P based on the TT index shown here is an example, and the feature selection unit 113 may use other selection conditions.

  In order to improve the accuracy of selection, the feature selection unit 113 collects the time-series positions of the body part P for a plurality of seller U1 and buyer U2 pairs, averages them, and obtains an iterative plot A. It may be generated and the body part P may be selected.

  In order to improve selection accuracy, the feature selection unit 113 collects time series positions of the body part P for a plurality of seller U1 and buyer U2 pairs, and in each pair, the buyer U2 for the seller U1 The distribution map B of the TT index of B and the distribution map B of the TT index of the seller U1 for the buyer U2 are calculated. The feature selection unit 113 may select a common important body part P on the horizontal axis of each distribution map B.

  In order to improve selection accuracy, the feature selection unit 113 collects time series positions of the body part P for a plurality of seller U1 and buyer U2 pairs, and in each pair, the buyer U2 for the seller U1 The distribution map B of the TT index of B and the distribution map B of the TT index of the seller U1 for the buyer U2 are calculated. The distribution map B of each TT index follows the gamma distribution, and therefore, in the TT index according to the gamma distribution, a plurality of sales corresponding to TT index values distributed in a range greater than or equal to a predetermined threshold away from the origin A common important body part P in the person U1 and the buyer U2 may be selected.

  When learning time-series information, the learning unit 114 applies the learning technology to the time-series position of the body part P (feature) of the seller U1 and the buyer U2 selected by the feature selection unit 113. To carry out learning. The learning unit 114 determines whether sales (a contract, an agreement, a persuasion, etc. may be successful) from the seller U1 to the buyer U2 in the input data as teacher data for learning (that is, whether the result is positive or not). Use the information in The learning unit 114 performs learning to generate a model for predicting the result and store the model in the storage unit 13. As learning techniques, known learning techniques such as neural networks and support vector machines can be used.

  When analyzing time series information, the analysis unit 115 reads the model generated by the learning unit 114 from the storage unit 13. Then, the analysis unit 115 performs analysis by inputting the time-series positions of the body part P (feature) of the seller U1 and the purchaser U2 selected by the feature selection unit 113 into the read out model. The model predicts whether the result of the interaction between the seller U1 and the buyer U2 is positive or not from the position of the time series of the body part P of the seller U1 and the buyer U2 inputted. Then, the analysis unit 115 acquires the result predicted by the model, causes the display unit 14 to display the result, or causes the storage unit 13 to store the result.

  Thus, the time-series analysis apparatus 1 according to the present embodiment is an important body part P (characteristic) for analysis of the result based on the relationship of movement of the body part P between users in consideration of time delay. And perform learning and analysis using the selected body part P. Therefore, the accuracy of learning and analysis can be improved, and the time required for calculation can be shortened.

[Flowchart of time series analysis method]
FIG. 7 is a view showing a flowchart of a time series analysis method by the time series analysis device 1 according to the present embodiment. The flowchart of FIG. 7 is started, for example, when the user performs a predetermined start operation on the time-series analyzer 1.

  The time series analyzer 1 first performs a learning process described later with reference to FIG. 8 (S1). Next, the time series analysis device 1 performs analysis processing described later with reference to FIG. 9 using the model generated by the learning processing in step S1 (S2).

  The time series analysis device 1 does not necessarily have to perform the analysis process of step S2 following the learning process of step S1, but the analysis process of step S2 on condition that the start instruction from the manager of the time series analysis device 1 is received. You may In addition, the time-series analysis device 1 may perform the analysis process of step S2 multiple times after the learning process of step S1.

[Flowchart of learning process]
FIG. 8 is a diagram showing a flowchart of learning processing according to the present embodiment. The flowchart of FIG. 8 is executed in step S1 of the time series analysis method of FIG.

  First, the information acquisition unit 111 acquires time-series image information of two users who are learning objects captured by the imaging unit 15 (S101). Then, the information acquisition unit 111 calculates the position of each body part of each user from the time-series image information acquired in step S101.

  The information conversion unit 112 performs feature space conversion on the position of each body part calculated by the information acquisition unit 111 using the above-mentioned equation (1) (S102). In the following, processing is performed using the coordinates at which feature space conversion has been performed in step S102 as the position of each body part.

  The feature selection unit 113 generates a repeated plot of the time-series position (i.e., movement) of the body part for each combination of body parts (features) between users, and calculates the TT index using the above equation (2). (S103). Then, the feature selection unit 113 selects a body part (feature) to be used for analysis of the result based on the TT index calculated in step S103 (S104).

  The learning unit 114 executes learning by applying a learning technique to the time-series position of the body part (feature) selected by the feature selection unit 113 in step S104 (S105). The learning unit 114 causes the storage unit 13 to store a model (model parameter) generated by learning.

Flowchart of Analysis Process
FIG. 9 is a view showing a flowchart of analysis processing according to the present embodiment. The flowchart of FIG. 9 is executed in step S2 of the time series analysis method of FIG.

  First, the information acquisition unit 111 acquires time-series image information of two users to be analyzed, which are captured by the imaging unit 15 (S201). Then, the information acquisition unit 111 calculates the position of each body part of each user from the time-series image information acquired by the information acquisition unit 111 in step S201.

  The information conversion unit 112 performs feature space conversion on the position of each body part calculated by the information acquisition unit 111 using the above-mentioned equation (1) (S202). In the following, processing is performed using the coordinates subjected to the feature space conversion in step S202 as the positions of the respective body parts.

  The feature selection unit 113 selects the body part (feature) that the learning unit 114 used to generate the model in step S105 of the learning process (S203).

  The analysis unit 115 reads the model generated by the learning unit 114 in step S105 of the learning process from the storage unit 13 (S204). Then, the analysis unit 115 performs analysis by inputting the time-series position of the body part (feature) selected by the feature selection unit 113 in step S203 to the model read out in step S204 (S205). The model predicts whether the result is positive or not from the position of the time series of the body parts of the two users inputted. Then, the analysis unit 115 acquires the result predicted by the model, causes the display unit 14 to display the result, or causes the storage unit 13 to store the result.

  Although the above-mentioned time-series analysis device 1 performs analysis to predict whether the result of sales, contract, agreement, persuasion, etc. is positive, it may perform analysis to predict the effect of education or group learning. In this case, the time-series analysis device 1 selects a body part to be used for analysis based on the co-occurrence relation of the body part between the teacher and the students or between the students, and performs learning and analysis. As teacher data for learning, the time-series analysis device 1 can use the evaluation of the teacher from the student, the score of the student's examination, or the like.

[Effect of First Embodiment]
When learning and analyzing the interaction between users using movement of body parts such as joints of a human body, the larger the number of body parts to be calculated, the longer the calculation time. In addition, since the movement of the body part includes natural movement unrelated to the interaction between the users, the accuracy of learning and analysis may be reduced if all the body parts are to be calculated.

  Therefore, the time-series analysis device 1 according to the present embodiment selects and selects an important body part (feature) for the analysis of the result based on the relationship in which the time delay of the movement of the body part between the users is considered. The body part is used for learning and analysis. As a result, since the time series analysis device 1 can reduce the number of body parts to be calculated, the calculation time can be shortened. Moreover, since the time-series analysis apparatus 1 can narrow down and use the important body part in connection with the interaction between users, it can raise the precision of learning and analysis.

[Modification of First Embodiment]
In the above-described time series analysis device 1, the number and type of body parts P selected for learning and analysis may be adjusted by feeding back the learning result. Specifically, the feature selecting unit 113 randomly selects a random number of body parts P using the body parts P selected according to the condition of the above-mentioned TT index as a population.

  Then, the learning unit 114 performs the above-described learning using the number and types of body parts P randomly selected by the feature selection unit 113. The feature selecting unit 113 and the learning unit 114 repeat selection and learning of the body part P a predetermined number of times to generate a plurality of models. The learning unit 114 inputs test data to each of the plurality of generated models to calculate the accuracy. Then, the learning unit 114 selects a model with the highest calculated accuracy and causes the storage unit 13 to store the selected model. The analysis unit 115 executes the analysis by reading out the selected model with the highest accuracy and inputting time-series information of the body part P selected in generation of the model into the model.

  With such a configuration, since the time-series analysis apparatus 1 can selectively use more suitable number and types of body parts P, the accuracy of learning and analysis can be improved.

Second Embodiment
While the first embodiment predicts the result of interaction between users by learning the relationship of features between users, the present embodiment analyzes the user's behavior pattern by learning the relationship of features between users. Make predictions of The device configuration of the time series analysis system S is the same as that of the first embodiment.

  In the present embodiment, the behavior pattern refers to the type of behavior in daily life such as cleaning, resting and chewing, and the type of behavior on work such as desk work, typing and thinking. Focusing on the same action pattern, even if the order and frequency of actions in the action pattern differ depending on the user, common features appear when time is shifted (that is, given a time delay). On the other hand, in different behavior patterns, common features do not appear much even if time is shifted.

  Therefore, the feature selection unit 113 of the time series analysis device 1 according to the present embodiment predicts the action pattern based on the movement relationship of the body part (feature) between different users in consideration of the time delay in each action pattern. Choose important body parts for The selection conditions of the body part are the same as in the first embodiment.

  The learning unit 114 uses the feature selection unit 113 using time series information pairs of different users corresponding to a certain action pattern, and time series information pairs of a user corresponding to a certain action pattern and a user not corresponding to the action pattern. The learning is performed by applying a learning technique to the time series position of the selected body part. The learning unit 114 uses, as teacher data for learning, information as to whether or not the input data corresponds to a specific same behavior pattern. By repeating this learning for each behavior pattern, the learning unit 114 generates a model for each behavior pattern and causes the storage unit 13 to store the model.

  When analyzing the time-series information, the analysis unit 115 inputs the time-series position corresponding to the body part selected in the generation of each model to the model for each behavior pattern generated by the learning unit 114. Perform analysis by Then, the analysis unit 115 outputs an action pattern whose degree of adaptation to the movement of the body part input to the model for each action pattern exceeds a predetermined threshold as a prediction result of the action pattern.

[Flowchart of learning process]
FIG. 10 is a diagram showing a flowchart of learning processing according to the present embodiment. The flowchart of FIG. 10 is executed in step S1 of the time series analysis method of FIG.

  First, the information acquisition unit 111 acquires time-series image information for each of the action patterns of the two users who are learning objects captured by the imaging unit 15 (S111). Then, the information acquisition unit 111 calculates the position of each body part of each user from the time-series image information acquired in step S111.

  The information conversion unit 112 performs feature space conversion on the position of each body part calculated by the information acquisition unit 111 using the above-mentioned equation (1) (S112). In the following, processing is performed using the coordinates at which feature space conversion has been performed in step S112 as the position of each body part.

  The feature selection unit 113 generates, for each of the behavior patterns, repetitive plots of time-series positions (i.e., movements) of body parts for each combination of body parts (features) between users, using the above-mentioned equation (2). The TT index is calculated (S113). Then, based on the TT index calculated in step S113, the feature selection unit 113 selects body parts (features) to be used for predicting the action pattern for each action pattern (S114).

  The learning unit 114 executes learning by applying a learning technique to the time-series position of each action pattern of the body part (feature) selected by the feature selection unit 113 in step S114 (S115). The learning unit 114 stores the model (parameter of the model) generated by learning in the storage unit 13 for each behavior pattern.

  The analysis processing according to the present embodiment is the same as the flowchart of FIG. 9 except that the model for each behavior pattern is used and the data of the pseudo partner for each behavior pattern is assumed to be exemplar data prepared in advance. is there.

  The time-series analysis device 1 may perform acquisition of information for each behavior pattern, feature space conversion, feature selection, and execution of learning and analysis in parallel with each other in the learning process and the analysis process. Thereby, processing can be performed at high speed.

  Also in this embodiment, as in the modification of the first embodiment, a plurality of models are generated using randomly selected numbers and types of body parts, and a model with high accuracy is selected from them. It is also good.

[Effect of Second Embodiment]
The time-series analysis device 1 according to the present embodiment selects and selects an important body part (feature) for the prediction of the action pattern, based on the relationship taking into consideration the time delay of the movement of the body part between the users. The body part is used for learning and analysis. As a result, it is possible to shorten the calculation time of learning and prediction of behavior patterns and to improve the accuracy.

Third Embodiment
While the first embodiment focuses on the interaction of two users, the present embodiment performs learning and analysis focusing on interactions in groups of three or more. The device configuration of the time series analysis system S is the same as that of the first embodiment.

  In interaction such as a meeting performed in a group of three or more, there is a pair of users who perform active interaction and a pair of users who perform inactive interaction, and the topic changes with time. Therefore, the time-series analysis device 1 according to the present embodiment classifies whether interaction between users is active or not by performing learning and analysis for each pair of users while shifting a time window of a predetermined length.

  Specifically, the feature selection unit 113 sets a time window of a predetermined length, and selects a pair of two users among a plurality of users to be learned. Then, the feature selection unit 113 uses the time-series information in the set time window and is important for analysis of the result based on the relationship in which the time delay of the movement of the body part (feature) between the selected users is considered. Select the appropriate body part. The selection condition of the body part is the same as that of the first embodiment except that it is performed in units of time windows. The time window is a range that limits the start time and the end time of time series information to be processed. The length of the time window is set in advance in the storage unit 13 by the manager of the time series analysis device 1. The feature selecting unit 113 repeatedly selects a body part while shifting the time window by a predetermined number of times and selecting another pair of users to be learned.

  The learning unit 114 repeatedly performs learning while shifting the time window by applying a learning technique to the time-series position of the body part selected by the feature selection unit 113 within the time window set by the feature selection unit 113. Run. The learning unit 114 also repeatedly performs learning similarly for another user pair selected by the feature selection unit 113. The learning unit 114 uses, as teaching data for learning, classification of whether or not the interaction between users is active in the input data.

  When analyzing time-series information, the analysis unit 115 performs analysis by inputting the time-series position of the body part selected in the model to the model generated by the learning unit 114. The model classifies whether or not the interaction is active while shifting the time window from the inputted time-series positions of the body parts of the two users. Then, the analysis unit 115 acquires the result classified by the model, causes the display unit 14 to display the result, or stores the result in the storage unit 13.

  If the analysis unit 115 determines that active communication is to be performed based on the model generated by the learning unit 114, the analysis unit 115 may predict a positive or negative result by applying the first embodiment within each time window. .

[Flowchart of learning process]
FIG. 11 is a diagram showing a flowchart of learning processing according to the present embodiment. The flowchart of FIG. 11 is executed in step S1 of the time series analysis method of FIG.

  First, the information acquisition unit 111 acquires time-series image information of a plurality of users who are learning objects captured by the imaging unit 15 (S121). Then, the information acquisition unit 111 calculates the position of each body part of each user from the time-series image information acquired in step S121.

  The information conversion unit 112 performs feature space conversion on the position of each body part calculated by the information acquisition unit 111 using the above-mentioned equation (1) (S122). In the following, processing is performed using the coordinates subjected to the feature space conversion in step S122 as the positions of the respective body parts.

  The feature selecting unit 113 sets a time window of a predetermined length as an initial position (S123). The initial position of the time window is, for example, the first time of the input time-series image information. The feature selection unit 113 shifts the position of the time window for each predetermined time. In the following, processing is performed using the time-series position of each body part in the time window set in step S123.

  The feature selection unit 113 selects a pair of two users among the plurality of users to be learned (S124). The feature selection unit 113 selects another pair of two users for each repetition. In the following, processing is performed on the user selected in step S124.

  The feature selection unit 113 generates a repeated plot of the time-series position (i.e., movement) of the body part for each combination of body parts (features) between users, and calculates the TT index using the above equation (2). (S125). Then, the feature selection unit 113 selects a body part (feature) to be used for analysis of the result based on the TT index calculated in step S125 (S126).

  The learning unit 114 executes learning by applying a learning technique to the time-series position of the body part (feature) selected by the feature selection unit 113 in step S126 (S127). The learning unit 114 causes the storage unit 13 to store a model (model parameter) generated by learning.

  If the process has not been completed for all pairs of users (NO in S128), the time-series analyzer 1 returns to step S124, selects another pair of users, and repeats the process. If the process has not been completed for the entire time of the input data (NO in S129), the time series analyzer 1 returns to step S123, shifts the position of the time window by a predetermined time, and repeats the process.

  When the process is completed for all pairs of users and the process is completed for all the time of the input data (YES in S128, YES in S129), the time series analyzer 1 ends the learning process.

  The analysis processing according to the present embodiment is the same as the flowchart of FIG. 9 except that data is input to the model for each time window and each pair of users.

  The time series analysis device 1 may perform acquisition of information for each pair of users, feature space conversion, feature selection, execution of learning and execution of analysis in parallel with each other in the learning process and the analysis process. Thereby, processing can be performed at high speed.

  Also in this embodiment, as in the modification of the first embodiment, a plurality of models are generated using randomly selected numbers and types of body parts, and a model with high accuracy is selected from them. It is also good.

[Effect of the third embodiment]
The time-series analysis device 1 according to the present embodiment can perform learning and analysis for each pair of users in an interaction such as a meeting performed by a group of three or more people. Further, since the time-series analysis device 1 performs processing while shifting the time window, it is possible to perform high-accuracy learning and analysis even in a situation where the topic changes with time.

Fourth Embodiment
The first embodiment performs learning and analysis using the motion of the user's body part, whereas the present embodiment performs learning and analysis using the language spoken by the user. The device configuration of the time series analysis system S is the same as that of the first embodiment.

  In the present embodiment, the language spoken by the user may be text information input by the user by e-mail, chat or the like, or voice information from the user acquired by the voice input unit 16. The reaction of the second user who saw or heard the language of the first user differs depending on whether or not he / she shows interest. Therefore, the time-series analysis device 1 according to the present embodiment selects a language feature important for analysis of the result instead of the movement of the body part of the user, and performs learning and analysis using the selected language feature.

  Since the words used in daily conversation are innumerable, it is not practical to extract the co-occurrence relation of all the word combinations. The time-series analysis device 1 according to the present embodiment clusters words included in a language issued by a user, and uses clusters as features. The time series analyzer 1 stores the correspondence relationship between the word and the cluster in the storage unit 13 in advance. The time-series analysis apparatus 1 uses, for example, a cluster defined for each user emotion as listed below.

  Reassurance / gratitude / happiness / relaxation / respect / expectation / love / fear / regret / aversion / jealousy / anger

  The cluster shown here is an example, and the administrator of the time-series analysis apparatus 1 can define any cluster. In this embodiment, a group of words belonging to one cluster is treated as one feature.

  The information acquisition unit 111 acquires time-series voice information acquired by the voice input unit 16 or time-series character information stored in the storage unit 13. Text information such as e-mails and chats, etc., to be learned and analyzed is obtained from another system and stored in advance in the storage unit 13. Then, the information acquisition unit 111 applies the natural language processing technology to the acquired voice information or text information to extract words included in the language issued by each user. Known natural language processing techniques can be used. It is desirable that the information acquisition unit 111 apply preprocessing such as unnecessary word removal to voice information or text information in order to improve the accuracy of analysis.

  The information conversion unit 112 clusters the words extracted by the information acquisition unit 111 according to the definition of the cluster stored in advance in the storage unit 13. The information conversion unit 112 determines each of the generated clusters as a feature to be used for learning and analysis.

  The feature selection unit 113 uses the time-series information of each cluster (that is, the generation time of the word belonging to each cluster) based on the relationship in which the time delay of the generation time of the cluster (feature) between two users is considered. Choose clusters that are important for analysis of the results. The method of selecting clusters by the feature selection unit 113 is the same as the method of selecting a body part of the user in the first embodiment.

  The learning unit 114 executes learning by applying a learning technique to the generation time of the cluster selected by the feature selection unit 113. The learning unit 114 uses information on whether the result of sale, contract, agreement, persuasion or the like is positive in the input data as teacher data of learning.

  When analyzing time-series information, the analysis unit 115 performs the analysis by inputting, to the model generated by the learning unit 114, the generation time of the cluster selected in the generation of the model. The model predicts whether the result of sales, contract, agreement, persuasion etc. is positive or not from the generation time of the cluster of two users inputted. Then, the analysis unit 115 acquires the result predicted by the model, causes the display unit 14 to display the result, or stores the result in the storage unit 13.

[Flowchart of learning process]
FIG. 12 is a diagram showing a flowchart of learning processing according to the present embodiment. The flowchart of FIG. 12 is executed in step S1 of the time series analysis method of FIG.

  First, the information acquisition unit 111 acquires time-series voice information acquired by the voice input unit 16 or time-series character information (e-mail, chat, etc.) stored in the storage unit 13 (S131). Then, the information acquisition unit 111 applies the natural language processing technology to the voice information or the character information acquired in step S131, thereby extracting a word included in the language issued by each user (S132).

  The information conversion unit 112 clusters the words extracted by the information acquisition unit 111 in step S132 in accordance with the cluster definition stored in advance in the storage unit 13 (S133). Then, the information conversion unit 112 determines each cluster generated in step S133 as a feature to be used for learning and analysis (S134).

  The feature selection unit 113 generates an iterative plot of cluster occurrence time for each combination of clusters (features) between users, and calculates a TT index using the above-mentioned equation (2) (S135). Then, the feature selection unit 113 selects clusters (features) used for learning and analysis based on the TT index calculated in step S135 (S136).

  The learning unit 114 executes learning by applying a learning technique to the generation time of the cluster (feature) selected by the feature selection unit 113 in step S136 (S137). The learning unit 114 causes the storage unit 13 to store a model (model parameter) generated by learning.

  The analysis process according to the present embodiment is the same as the flowchart of FIG. 9 except that a word cluster is used as a feature instead of a body part.

  Also in this embodiment, as in the modification of the first embodiment, a plurality of models are generated using randomly selected numbers and types of clusters, and a model with high accuracy is selected from among the plurality of models. Good.

  By combining the present embodiment with the first to third embodiments, the time-series analysis apparatus 1 can detect the movement of the body part of the user (i.e. non-verbal behavior) and the language spoken by the user (i.e. verbal behavior) Both may be used as features to perform learning and analysis. With such a configuration, analysis can be performed in various situations, such as a situation where one user is explaining in language and the other user silently listens to the explanation.

[Effect of Fourth Embodiment]
The time series analysis device 1 according to the present embodiment clusters the words extracted from the language issued by the user, and uses the generated cluster as a feature. Then, the time-series analysis device 1 selects a cluster important for analysis of the result based on the relationship taking into consideration the time delay of the generation time of clusters (features) among users, and performs learning using the selected cluster And analyze. As a result, calculation time can be shortened and accuracy can be improved also in learning and prediction of verbal behavior.

  As mentioned above, although the present invention was explained using an embodiment, the technical scope of the present invention is not limited to the range given in the above-mentioned embodiment, and various modification and change are possible within the range of the gist. is there. For example, a specific embodiment of device distribution and integration is not limited to the above embodiment, and all or a part thereof may be functionally or physically distributed and integrated in any unit. Can. In addition, new embodiments produced by any combination of a plurality of embodiments are also included in the embodiments of the present invention. The effects of the new embodiment generated by the combination combine the effects of the original embodiment.

  The control unit 11 (processor) of the time series analysis device 1 is a main body of each step (process) included in the time series analysis method shown in FIGS. That is, the control unit 11 reads a time series analysis program for executing the time series analysis method shown in FIGS. 7 to 12 from the storage unit 13, executes the time series analysis program, and executes each unit of the time series analysis device 1. By controlling, the time-series analysis method shown in FIGS. The steps included in the time-series analysis method illustrated in FIGS. 7 to 12 may be partially omitted, the order between the steps may be changed, and a plurality of steps may be performed in parallel.

S Time Series Analysis System 1 Time Series Analysis Device 11 Control Unit 111 Information Acquisition Unit 112 Information Conversion Unit 113 Feature Selection Unit 114 Learning Unit 115 Analysis Unit

Claims (14)

An acquisition unit configured to acquire time-series information of the first user feature and the second user feature;
In the time-series information acquired by the acquisition unit, the feature to be learned is determined based on the duration when the feature of the first user and the feature of the second user occur at the same time and at different times. A selection unit to select,
A learning unit that performs learning using time-series information of the feature to be learned selected by the selection unit;
A time series analyzer having a.   The system further includes an analysis unit that predicts a result based on the time-series information acquired by the acquisition unit by inputting the time-series information acquired by the acquisition unit into a model generated by the learning by the learning unit. The time-series analyzer according to Item 1.   The selection unit calculates an average value of a time range in which the feature of the first user and the feature of the second user occur at the same time and at different times as the duration. The time-series analyzer according to 2.   The time series analysis device according to claim 3, wherein the selection unit calculates the duration by Cross-Recurrence Quantification Analysis.   The selection unit selects a feature of the learning target based on the number of combinations of the feature of the first user and the feature of the second user whose duration is equal to or more than a predetermined threshold. The time series analyzer according to any one of 1 to 4.   The time series according to any one of claims 1 to 5, wherein the acquisition unit acquires body parts of the first user and the second user as the features of the first user and the second user. Analysis equipment.   The position of the body part of the first user is converted into coordinates based on the center of the body of the first user, and the position of the body part of the second user is referred to the center of the body of the second user The time series analysis device according to claim 6, further comprising: a conversion unit configured to convert into the coordinates.   The time series analysis according to any one of claims 1 to 7, wherein the learning unit performs learning whether or not the result of the interaction between the first user and the second user is positive. apparatus.   The time series analysis device according to any one of claims 1 to 7, wherein the learning unit performs learning whether the action pattern of the first user and the action pattern of the second user are the same. .   The learning unit according to any one of claims 1 to 9, wherein the learning unit repeatedly performs learning using time series information of the feature to be learned within a time window of a predetermined length while shifting the time window. Time series analyzer.   11. The time according to any one of claims 1 to 10, wherein the acquisition unit acquires the language spoken by the first user and the second user as the feature of the first user and the second user. Sequence analyzer.   The acquisition unit extracts a word from a language spoken by the first user and the second user, and clusters the generated word by clustering the features of the first user and the second user. The time series analysis device according to claim 11, which is acquired as Computer runs,
Acquiring time series information of the first user feature and the second user feature;
In the time-series information acquired in the acquiring step, the learning target is based on a duration when the feature of the first user and the feature of the second user occur at the same time and different times. Selecting a feature;
Performing learning using time-series information of the feature to be learned selected in the selecting step;
Time series analysis method having. Processor,
An acquisition unit configured to acquire time-series information of the features of the first user and the features of the second user;
In the time-series information acquired by the acquisition unit, the feature to be learned is determined based on the duration when the feature of the first user and the feature of the second user occur at the same time and at different times. A selecting unit to select, and a learning unit to perform learning using time-series information of the feature to be learned selected by the selecting unit;
Time series analysis program to function as.

Images