JP2023008679A - Information providing method, information providing program, and marking support system - Google Patents

Abstract

[Problem] To improve the quality of information provision regarding the results of technique recognition. [Solution] In the information provision method, a computer executes a process of calculating a feature amount corresponding to a technique output by a machine learning model for technique recognition from skeletal information input to the machine learning model for technique recognition, calculating a difference between the calculated feature amount and a feature amount obtained from training data of skeletal information used to train the machine learning model for technique recognition that is labeled with the same technique as the technique, and outputting the calculated feature amount difference. [Selected Figure] Figure 6

Description

The present invention relates to information provision technology.

Human posture recognition and action recognition are used in various fields. Taking gymnastics as an example, the current scoring method is based on visual observation by multiple referees. The number of situations where it is difficult for referees to recognize techniques is increasing. As a result, there are concerns about maintaining the fairness and accuracy of scoring, such as the scoring results of athletes differing for each referee.

Against this background, we acquire 3D point cloud data of athletes using a 3D laser sensor, detect the athlete's skeletal information, recognize the techniques performed by the athlete, and provide information on the recognition results of the techniques. A scoring support system has been proposed that assists referees in scoring.

For example, in a scoring support system, the likelihood output by a machine learning model used for skill recognition is provided as part of providing information on skill recognition results. As an aspect, this draws attention to mistakes in automatic scoring in situations where there is a high possibility that the machine learning model is outputting a label for an incorrect technique, such as an over-detection or a false detection.

Japanese Patent Application Laid-Open No. 2017-228100 Special Table 2014-509011 JP 2019-166311 A U.S. Patent Application Publication No. 2012/0214594

However, as will be described below, the above likelihood is not always sufficient as the quality of providing information on the recognition result of the trick.

In other words, the above likelihood simply corresponds to the difference between the skeletal information used as training data when training the machine learning model and the skeletal information input in situations such as scoring. It does not match. In particular, the greater the number of types of classes classified by the machine learning model, and the greater the impact of individual differences on the classification accuracy of the machine learning model, the more difficult it is for the likelihood and true likelihood to match. For example, the label may be correct even if the likelihood output by the machine learning model is low, and the label may be incorrect even if the likelihood output by the machine learning model is high. In addition, the decision process by machine learning models is different from that of humans and may be a black box, so even if the likelihood output by machine learning models is provided, there is also an aspect that is not suitable for explanation to humans. .

An object of one aspect is to provide an information providing method, an information providing program, and a scoring support system capable of improving the quality of providing information on skill recognition results.

In an information providing method according to one aspect, a feature amount corresponding to a technique output by the machine learning model for technique recognition is calculated from skeleton information input to the machine learning model for technique recognition, and the calculated feature amount and , out of the training data of the skeleton information used for training the machine learning model for the technique recognition, the difference between the feature amount obtained from the training data labeled with the same technique as the technique is calculated. A computer executes a process of outputting the difference of the feature amounts.

According to one embodiment, it is possible to improve the quality of providing information on the recognition results of tricks.

FIG. 1 is a diagram showing a configuration example of a scoring support system. FIG. 2 is a schematic diagram showing the skeleton recognition function. FIG. 3 is a schematic diagram showing the technique recognition function. FIG. 4 is a diagram showing an example of a multi-angle view. FIG. 5 is a diagram showing an example of an automatic scoring view. FIG. 6 is a block diagram of a functional configuration example of a server device according to the first embodiment; FIG. 7 is a diagram showing an example of the scoring support view. FIG. 8 is a diagram showing an example of a training assistance view. FIG. 9 is a flowchart illustrating the procedure of information provision processing according to the first embodiment. FIG. 10 is a block diagram illustrating a functional configuration example of a server device according to the second embodiment; FIG. 11 is a flowchart illustrating the procedure of information provision processing according to the second embodiment. FIG. 12 is a diagram illustrating a hardware configuration example.

Hereinafter, embodiments of an information providing method, an information providing program, and a scoring support system according to the present application will be described with reference to the accompanying drawings. Each embodiment merely shows one example or one aspect, and such examples do not limit the numerical values, the range of functions, the usage scene, and the like. Further, each embodiment can be appropriately combined within a range that does not contradict the processing contents.

<System configuration>
FIG. 1 is a diagram showing a configuration example of a scoring support system. A scoring support system 1 shown in FIG. 1 acquires three-dimensional data of an actor 3, who is a subject, and supports scoring of techniques performed by the actor 3 through skeleton recognition, technique recognition, and the like.

As an example of such a technique, a performance in gymnastics will be given, but in addition to performance in figure skating, movements in various sports, such as a baseball swing and a soccer shoot, may also be included in the category of techniques.

As shown in FIG. 1 , the scoring support system 1 may include a 3D laser sensor 5 , a server device 10 and a client terminal 30 .

The 3D (Three-Dimensional) laser sensor 5 is an example of a sensor device that uses an infrared laser or the like to measure the distance to an object, the so-called depth, for each pixel corresponding to a scanning point. For example, the 3D laser sensor 5 may be a laser sensor using a depth image camera or LIDAR (Light Detection and Ranging) technology, such as a MEMS (Micro-Electro-Mechanical Systems) mirror-type laser sensor.

The server device 10 is an example of a computer that provides various services. As an example only, the server device 10 provides a scoring support service in which a skeleton recognition function 7, a technique recognition function 8, and a scoring support function 9 are packaged as shown in FIG. For example, the skeleton recognition function 7 uses the depth image measured by the 3D laser sensor 5 to provide a function of detecting skeleton information such as the positions of the skeleton parts of the performer 3, such as joint positions. The technique recognition function 8 provides a function of recognizing techniques performed by the performer 3 using time-series data of skeleton information obtained by skeleton recognition by the skeleton recognition function 7 . The scoring support function 9 provides a function for supporting scoring based on the skill recognition result by the skill recognition function 8 .

By causing an arbitrary computer to execute package software that realizes the skeleton recognition function 7, the technique recognition function 8, and the scoring support function 9, the above-described scoring support service can be provided. For example, the server device 10 can provide the skeleton recognition function 7, the technique recognition function 8, and the scoring support function 9 as a cloud service by being implemented as a SaaS (Software as a Service) type application. In addition, the server device 10 can be implemented as a server that provides the skeleton recognition function 7, the technique recognition function 8, and the scoring support function 9 on-premises.

The client terminal 30 is an example of a computer that receives the above services. Such a client terminal 30 is used by people involved in gymnastics, such as referees and trainers of the performers 3, as just one example of users of the above services. For example, the client terminal 30 may be a desktop computer such as a personal computer. In addition, the client terminal 30 is not limited to a general-purpose information processing device such as a laptop computer, a mobile terminal device, or a wearable terminal. good too.

Note that FIG. 1 shows an example in which the skeleton recognition function 7, the technique recognition function 8, and the scoring support function 9 are provided as a package, but each function may be provided as a module. In this case, the skeleton recognition function 7, the technique recognition function 8, and the scoring support function 9 may each be realized by different computers, and may be provided by the same vendor or different vendors.

<Skeleton recognition function>
FIG. 2 is a schematic diagram showing the skeleton recognition function 7. As shown in FIG. As shown in FIG. 2, the skeleton recognition function 7 can be realized by a hybrid system that combines skeleton recognition using a machine learning model and fitting, just as an example.

For example, for skeletal recognition, a machine learning model 7m, such as a CNN (Convolutional Neural Network) neural network, which inputs depth images and outputs estimated values of 3D skeletal coordinates, can be used. The "3D skeletal coordinates" referred to here refer to the coordinates at which joints such as the head, shoulders, spine, elbows, wrists, hips, knees, and ankles are positioned in a three-dimensional space, and correspond to an example of "skeletal information." obtain.

For training such a machine learning model 7m, a data set 7TR including training data in which depth images and 3D skeletal coordinates of correct labels are associated can be used. For example, the training data can be prepared by generating depth images using computer graphics or the like from 3D skeletal coordinates for gymnastics. Under such a data set, in the learning phase, the depth image is used as an explanatory variable for the machine learning model 7m, the label is used as the objective variable for the machine learning model 7m, and any machine learning algorithm, such as deep learning, is used for machine learning. Model 7m can be trained. For example, the parameters of the machine learning model 7m are updated based on the loss between the estimated value of the 3D skeleton coordinates output by the machine learning model 7m to which the depth image is input and the 3D skeleton coordinates of the correct labels. This yields a trained machine learning model 7M. In the inference phase, multi-viewpoint depth images output from multi-viewpoint 3D laser sensors 5A to 5N installed so as to overcome occlusion by the gymnastics equipment and the performer 3 himself are input to the machine learning model 7M. The machine learning model 7M to which the multi-viewpoint depth images are input in this way outputs the 3D skeleton coordinates of the performer 3 .

In the fitting, the output of the 3D skeleton coordinates of the machine learning model 7M, the fitting result in the previous frame, etc. are used as initial values, and the human body model is applied to the 3D point cloud in which the multi-viewpoint depth images are integrated. For example, by defining an evaluation function (likelihood) that indicates the degree of matching between the coordinates of the 3D point cloud and the surface coordinates of the human body model, and optimizing the joint angle with the highest likelihood, the 3D skeleton coordinates are determined. do.

The 3D laser sensor 5 and the skeleton recognition function 7 realize markerless 3D sensing that performs three-dimensional measurement of the movement of the performer 3 .

<Technique recognition function>
FIG. 3 is a schematic diagram showing the technique recognition function 8. As shown in FIG. As shown in FIG. 3, the technique recognition function 8 can be realized by a machine learning model 8m, for example, a CNN-based neural network, which receives time-series data of 3D skeletal coordinates and outputs gymnastics technique names.

For training such a machine learning model 8m, a data set 8TR including training data associated with time-series data of 3D skeleton coordinates and technique names of correct labels can be used. For example, the training data can be time-series data of 3D skeletal coordinates in which performances have been demonstrated in gymnastics. Under such a data set, in the learning phase, the 3D skeleton coordinates are used as explanatory variables of the machine learning model 8m, the labels are used as the objective variables of the machine learning model 8m, and any machine learning algorithm, such as deep learning, is used to machine A learning model 8m can be trained. For example, the parameters of the machine learning model 8m are updated based on the loss between the estimated skill name output by the machine learning model 8m to which the time series data of the 3D skeletal coordinates is input and the skill name of the correct label. This yields a trained machine learning model 8M. In the inference phase, time-series data of 3D skeleton coordinates obtained by skeleton recognition by the skeleton recognition function 7 is input to the machine learning model 8M. The machine learning model 8M to which the time-series data of the 3D skeletal coordinates is input in this way outputs the name of the technique performed by the performer 3 as the recognition result of the technique. At this time, information that can be identified by matching the technique name with rules and scoring rules, such as the technique group name, the difficulty level of the technique, the difficulty evaluation score, etc., should be output as information related to the recognition result of the technique. can also

<Scoring support function>
The scoring support function 9 can provide information to gymnastics participants based on 3D skeleton coordinates by the skeleton recognition function 7, technique recognition results by the technique recognition function 8, or a combination thereof. For example, the scoring support function 9 can provide a multi-angle view shown in FIG. 4, an automatic scoring view shown in FIG. 5, and the like as examples of user interfaces for referees corresponding to an example of a person involved in gymnastics. By realizing such a scoring support function 9 as a Web application, it is possible to provide the multi-angle view shown in FIG. 4 and the automatic scoring view shown in FIG.

FIG. 4 is a diagram showing an example of a multi-angle view. As shown in FIG. 4, in the multi-angle view 400, the 3D skeletal coordinates of the performer 3 are modeled for each frame in which the 3D laser sensor 5 acquires the depth image. , Free viewpoint, etc. are displayed. Whether or not to display the skeletal model from a plurality of viewpoints can be switched by operating the tabs 410 and 420 . For example, when the "Multi" tab 410 is selected, the skeletal model of actor 3 is displayed for each of a plurality of viewpoints, as shown in FIG. On the other hand, when the "Single" tab 420 is selected, the skeletal model of actor 3 corresponding to the specified viewpoint is displayed. Furthermore, in the multi-angle view 400, by sliding the slider 430A on the seek bar 430, it is possible to specify the frame in which the skeleton model of the actor 3 is to be displayed. According to such a multi-angle view 400, it is possible to confirm in detail the joint angles of the performer 3, which can be one of the scoring criteria.

FIG. 5 is a diagram showing an example of an automatic scoring view. As shown in FIG. 5, autoscoring view 500 includes skeletal model view 510 and autoscoring table 520 . For example, the skeletal model view 510 displays a skeletal model in which the 3D skeletal coordinates of actor 3 at a user-specified angle are modeled. The frame in which the skeleton model of actor 3 is displayed in skeleton model view 510 can be specified by sliding slider 530 A on seek bar 530 . The automatic scoring table 520 displays the recognition results of tricks in chronological order. In the example shown in the figure, the name of the technique, the group number of the technique, the difficulty level of the technique, and the difficulty level value are displayed in chronological order in the vertical direction of the window of the automatic scoring view 500, that is, from the top to the bottom. be done. In addition to the individual display of information on each trick, the D-score scoring results for all performances are also displayed. Here, FIG. 5 shows an example in which the D-score tab 540 is being selected, but when the E-score tab 550 is selected, the display is switched to the display of the E-score scoring results for all performances. can be done. Such an automatic scoring view 500 can reduce the burden of referee work.

In addition to the multi-angle view and the automatic scoring view, the likelihood of the technique output by the machine learning model 8M used for technique recognition can be provided as part of providing information on the technique recognition result. This has the aspect of calling attention to mistakes in automatic scoring in situations where there is a high possibility that the machine learning model 8M is outputting a label for an incorrect technique, for example, in situations such as over-detection or mis-detection.

In FIGS. 4 and 5, content for referees is used as an example of providing information on the recognition results of tricks, but the target of information provision is not limited to referees. For example, content for trainers or entertainment may be provided.

<One aspect of the challenge>
As will be described below, the likelihood output by the machine learning model 8M is not always of sufficient quality as the quality of providing information on the recognition result of the technique.

In other words, the likelihood described above merely corresponds to the difference between the skeletal information used as training data when training the machine learning model 8M and the skeletal information input in situations such as scoring. do not match. In particular, the greater the number of types of classes classified by the machine learning model 8M, and the greater the impact of individual differences on the classification accuracy of the machine learning model 8M, the more difficult it is for the likelihood and true likelihood to match. . For example, the label may be correct even if the likelihood output by the machine learning model 8M is low, and the label may be incorrect even if the likelihood output by the machine learning model 8M is high. In addition, the judgment process by the machine learning model 8M is different from humans and may be a black box, so even if the likelihood output by the machine learning model 8M is provided, it is not suitable for explanation to humans. There is also

<Configuration of server device 10>
Therefore, in this embodiment, a functional configuration of the server device 10 having an information providing function capable of improving the quality of providing information on the recognition results of tricks will be described. FIG. 6 is a block diagram illustrating a functional configuration example of the server device 10 according to the first embodiment. FIG. 6 schematically shows blocks corresponding to the functions of the server device 10 .

As shown in FIG. 6 , the server device 10 has a communication interface section 11 , a storage section 13 and a control section 15 . FIG. 6 shows functional units related to the scoring support service including the above information providing function. Absent. In addition, the server device 10 may be provided with functions that existing computers are provided with by default or as an option, without being limited to the functional units shown in FIG. 6 .

The communication interface unit 11 corresponds to an example of a communication control unit that controls communication with other devices such as the 3D laser sensor 5 and the client terminal 30 . As an example only, the communication interface unit 11 can be realized by a network interface card such as a LAN (Local Area Network) card. As one aspect, the communication interface unit 11 receives depth images from the 3D laser sensor 5 on a frame-by-frame basis, and outputs information on skill recognition results to the client terminal 30 .

The storage unit 13 is a functional unit that stores various data. By way of example only, the storage unit 13 is implemented by storage, such as internal, external or auxiliary storage. For example, the storage unit 13 stores a data set 8TR of training data used to generate the machine learning model 8M. In addition to this data set 8TR, the storage unit 13 stores various data such as a data set 7TR of training data used to generate the machine learning model 7M, 3D skeleton coordinates, technique recognition results, and automatic scoring results. be able to.

The control unit 15 is a processing unit that performs overall control of the server device 10 . For example, the control unit 15 is realized by a hardware processor. As shown in FIG. 6, the control unit 15 includes an acquisition unit 15A, a skeleton recognition unit 15B, a technique recognition unit 15C, a feature amount calculation unit 15D, a feature amount difference calculation unit 15E, and a scoring support unit 15F. have.

The acquisition unit 15A is a processing unit that acquires a depth image. As an example only, the acquiring unit 15A can acquire multi-viewpoint depth images output from the multi-viewpoint 3D laser sensors 5A to 5N on a frame-by-frame basis. Here, the information source from which the acquisition unit 15A acquires the depth image may be any information source, and is not limited to communication via the network NW. For example, the acquisition unit 15A may acquire the depth image from a storage included in the server device 10 or removable media that can be attached to and detached from the server device 10, such as a memory card or USB (Universal Serial Bus) memory.

The skeleton recognition unit 15B is a processing unit that executes skeleton recognition. The skeleton recognition unit 15B can correspond to the skeleton recognition function 7 shown in FIG. As one embodiment, the skeleton recognition unit 15B can use the trained machine learning model 7M shown in FIG. For example, when the acquisition unit 15A acquires a new frame of the depth image, the skeleton recognition unit 15B inputs the multi-viewpoint depth image corresponding to the new frame to the machine learning model 7M. Then, the skeleton recognition unit 15B outputs the estimated values of the 3D skeleton coordinates of the performer 3 output from the machine learning model 7M to which the multi-viewpoint depth image is input, to the technique recognition unit 15C.

The technique recognition unit 15C is a processing unit that executes technique recognition. The technique recognition unit 15C can correspond to the technique recognition function 8 shown in FIG. As one embodiment, the technique recognition unit 15C can use the trained machine learning model 8M shown in FIG. For example, when the skeleton recognition unit 15B outputs the 3D skeleton coordinates corresponding to a new frame, the technique recognition unit 15C acquires the time-series data of the 3D skeleton coordinates for a specific number of frames from the new frame. Input to machine learning model 8M. Then, the technique recognition unit 15C outputs the technique recognition result of the performer 3, which is output from the machine learning model 8M to which the time-series data of the 3D skeleton coordinates is input, to the feature amount calculation unit 15D.

The feature amount calculation unit 15D is a processing unit that calculates feature amounts related to techniques. As an embodiment, the feature amount calculation unit 15D can start processing when a technique is recognized by the technique recognition unit 15C, for example, when the label output by the machine learning model 8M is not detected, that is, when there is no technique. For example, the feature amount calculation unit 15D calculates a feature amount corresponding to each scoring item of the technique recognized by the technique recognizing unit 15C. That is, since the type and number of scoring items defined by the scoring rule may differ depending on the type of technique, different types and different numbers of feature values are calculated for each technique. For example, in the case of “forward-backward split leg jump”, feature amounts such as back-and-forth leg spread, front knee angle, and jump height are calculated. In addition, taking the example of "front and back split wheel jump", in addition to the front and rear split legs, the front knee angle and the jump height, the feature values such as the height of the back legs and the warp angle of the upper body are further calculated. be. For example, the feature amount calculation unit 15D specifies a frame to be used for calculating the feature amount from the time-series data of the 3D skeleton coordinates input to the machine learning model 8M for each feature amount corresponding to the scoring item of the technique. Then, the feature amount is calculated from the 3D skeleton coordinates in the frame.

The feature amount difference calculation unit 15E is a processing unit that calculates a feature amount difference between the input data used for technique recognition and the training data corresponding to the technique recognition result. For example, the feature amount calculated by the feature amount calculation unit 15D is used as the feature amount of the input data. On the other hand, as a feature quantity of the training data, a 3D skeleton corresponding to the training data labeled with the same technique as the technique recognized by the technique recognition unit 15C in the data set 8TR used to generate the machine learning model 8M. A feature amount calculated from time-series data of coordinates is used. Note that the feature amount of the training data can be calculated by the feature amount calculation unit 15D on demand, but the feature amount calculated in advance can be used as a preset.

More specifically, the feature amount difference calculation unit 15E calculates the number of L pieces of training data labeled with the same technique as the technique recognized by the technique recognition unit 15C and the technique recognized by the technique recognition unit 15C. A feature amount difference is calculated for each of K kinds of feature amounts corresponding to the number of scoring items. The difference between the feature amounts calculated in this way means that the closer the value is to zero, the smaller the distance of the feature amount corresponding to the scoring item between the input data and the training data, or the higher the similarity of the feature amounts. can. That is, the smaller the value of the difference between the feature amounts, the higher the possibility that the technique is performed according to the scoring item corresponding to the technique labeled with the training data. Therefore, it is possible to quantify the likelihood that the technique is performed according to the scoring rules in gymnastics. As a result, it is possible to provide an index that expresses the true likelihood that the technique is being performed and that is suitable for the human judgment process.

Further, the feature quantity difference calculation unit 15E calculates the likelihood of the feature quantity from the difference in the feature quantity. Hereinafter, the likelihood of a feature amount may be referred to as a "feature amount likelihood". As just one example, the feature quantity likelihood can be calculated by normalizing the feature quantity difference to a specific numerical range, for example, a range of 0-1. At this time, the reciprocal of the difference of the feature amounts can be obtained from the aspect of normalization to the feature amount likelihood that the closer to 1 the more likely the feature amount is and the closer to 0 the less likely the feature amount is.

After the K types of feature quantity likelihoods are calculated in this way, the feature quantity difference calculating unit 15E calculates one representative value representing the K types of feature quantity likelihoods, for example, a statistical value, as the skill likelihood. be able to. Hereinafter, the likelihood of a technique may be referred to as "technique likelihood". As an example, the minimum value of K types of feature quantity likelihoods can be derived as the technique likelihood. In addition, an average value such as an arithmetic average or a weighted average of K types of feature quantity likelihoods, or a median value of K types of feature quantity likelihoods may be derived as skill likelihoods. By calculating such a skill likelihood for each of L pieces of training data, the feature quantity difference calculating unit 15E calculates L pieces of skill likelihood.

The scoring support unit 15F is a processing unit that supports scoring. The scoring support unit 15F can correspond to the scoring support function 9 shown in FIG. As one aspect, the scoring support unit 15F can cause the client terminal 30 to display the multi-angle view 400 shown in FIG. 4 and the automatic scoring view 500 shown in FIG. As another aspect, the scoring support unit 15F can provide the client terminal 30 with the feature amount likelihood and the technique likelihood calculated by the feature amount difference calculation unit 15E as part of providing information on the technique recognition result. For example, the scoring support unit 15F extracts the minimum skill likelihood from among the L number of skill likelihoods calculated by the feature amount difference calculation unit 15E, and K types of feature quantity likelihoods are extracted. After that, the scoring support unit 15F causes the client terminal 30 to display the minimum skill likelihood or the K types of feature amount likelihood calculated from the training data corresponding to the minimum skill likelihood.

FIG. 7 is a diagram showing an example of the scoring support view. As shown in FIG. 7, the scoring support view 700 includes a skeletal model view 710 that displays a skeletal model in which the 3D skeletal coordinates of actor 3 are modeled. The frame for displaying the skeleton model of the actor 3 in the skeleton model view 710 can be interlocked according to the position of the slider 720A on the seek bar 720. FIG. In the example of FIG. 7, the skeletal model view 710 displays the skeletal model of the technique "back and forth leg jump with front leg bent and stretched" being performed in the frame corresponding to the position of the slider 720A on the seek bar 720. .

Furthermore, according to the position of the slider 720A on the seek bar 720, the display of the symbol bar 730 that displays the performance performed by the performer 3 in chronological order is interlocked. For example, the trick being performed in the frame corresponding to the position of the slider 720A on the seek bar 720 and the tricks performed before and after that are displayed side by side in chronological order. In the example of FIG. 7, each technique of "Ron dart", "Backward stretch somersault, two foot landing", "Single turn with one leg standing", "Back and forth leg jump with front leg bent and straightened", and "Sisonne". Symbols are displayed side by side from left to right, for example.

Furthermore, symbols included in the symbol bar 730 are displayed in association with skill likelihoods corresponding to the symbols. In the example of FIG. 7, a skill likelihood of "0.91" is displayed below the symbol "Rondart", and a skill likelihood of "0. .99” is displayed, and the skill likelihood of “0.87” is displayed below the symbol “1 turn on one leg”. Furthermore, a technique likelihood of "0.25" is displayed below the symbol "forward and backward split jump with front leg bent and extended", and a technique likelihood of "0.85" is displayed below the symbol "sissonne". It is

Here, among the symbols and technique likelihoods displayed on the symbol bar 730, the display form of symbols or technique likelihoods satisfying a specific condition and the display form of other symbols or other technique likelihoods are distinguished. . For example, if the skill likelihood is equal to or less than a threshold value, or if the skill likelihood is the lowest in the symbol bar 730, the symbol or skill likelihood corresponding to either condition can be highlighted. In the example of FIG. 7, the symbol "forward-backward split-leg jump with front leg bent and stretched" is displayed with hatching, and the skill likelihood font is displayed in bold.

According to the symbol bar 730 as described above, it is possible to increase the degree of attention of the "back and forth leg jump with the front leg bent and extended" which is more likely to be over-detected or erroneously detected among the technique recognition results than other techniques. As a result, it is possible to draw attention to the automatic scoring error of "forward and backward open leg jump with front leg bent and extended". As a result, for example, it is possible to estimate whether or not a "front-rear split leg jump" similar in motion to a "front-rear split leg jump" is being erroneously detected, and to confirm the same.

FIG. 8 is a diagram showing an example of a training assistance view. As shown in FIG. 8, the training support view 800 includes GUI components similar to the skeletal model view 710, seek bar 720, slider 720A and symbol bar 730 shown in FIG. Further included.

The feature quantity likelihood list 810 lists the feature quantity likelihoods of the types of feature quantities corresponding to the scoring items of the technique being executed in the frame corresponding to the position of the slider 720A on the seek bar 720. Is displayed.

In the example of FIG. 8, from the side where the technique being performed is "front-back split leg jump with front leg bent and straightened", the feature value likelihood list 810 includes front-back leg spread "0.88", front knee angle Three feature quantity likelihoods 1 to 3 of "0.25" and jump height "0.98" are displayed.

Furthermore, among the three feature quantity likelihoods 1 to 3, the front knee angle "0.25" is relatively lower than the other feature quantity likelihoods, or a threshold value that can be set in a numerical range of 0 to 1, for example, 0 It is highlighted because it satisfies the condition that it is lower than .4.

According to the feature value likelihood list 810, the front knee angle is a relatively and absolutely low value of "0.25" in combination with the technique recognition result "back-and-forth leg jump with the front leg bent and extended". Something can be presented to actor 3, his trainer, and so on. As a result, it is possible to make suggestions for improvements such as insufficient bending and stretching of the front leg as stipulated in the scoring rules for "front-back leg jump with the front leg bent and extended".

<Process flow>
FIG. 9 is a flowchart illustrating the procedure of information provision processing according to the first embodiment. The process illustrated in FIG. 9 may begin, by way of example only, when a new frame of depth images is acquired. As shown in FIG. 9, the skeleton recognition unit 15B inputs the depth image acquired by the acquisition unit 15A to the machine learning model 7M, thereby outputting the estimated values of the 3D skeleton coordinates of the performer 3 to the machine learning model 7M. Skeleton recognition is executed by causing the robot to perform skeleton recognition (step S101).

Subsequently, the technique recognition unit 15C inputs the time-series data of the 3D skeleton coordinates for a specific number of frames retroactively from the new frame to the machine learning model 8M, thereby recognizing the technique recognition result of the performer 3 as the machine learning model. By causing 8M to output, technique recognition is executed (step S102).

At this time, if the technique is recognized in the technique recognition of step S102, for example, if the label output by the machine learning model 8M is not detected, that is, if there is no technique (Yes in step S103), the feature amount calculation unit 15D performs the following Execute the process. That is, the feature amount calculation unit 15D calculates K kinds of feature amounts corresponding to the scoring items of the technique recognized in the technique recognition of step S102 (step S104).

Then, the feature amount difference calculation unit 15E repeats the processing from step S105 to step S107 the number of times corresponding to the number L of training data to which the label of the same technique as the technique recognized in the technique recognition in step S102 is assigned. Repeat loop processing 1 is started. Although an example of loop processing is given here, the processing from step S105 to step S107 may be performed in parallel for each number L of training data.

Further, the feature amount difference calculation unit 15E repeats the processing of steps S105 and S106 for the number of times corresponding to the number of feature amount types K corresponding to the number of scoring items of the technique recognized in the technique recognition of step S102. Loop processing 2 is started. Also here, the processing of steps S105 and S106 may be performed in parallel for each type number K of feature amounts.

That is, the feature amount difference calculating unit 15E calculates the feature amount difference between the input data used for the technique recognition and the training data corresponding to the technique recognition result (step S105). For example, the feature amount calculated in step S104 is used as the feature amount of the input data. On the other hand, as a feature value of the training data, a 3D model corresponding to the training data labeled with the same technique as the technique recognized in the technique recognition in step S102 in the data set 8TR used to generate the machine learning model 8M. A feature amount calculated from the time-series data of the skeletal coordinates is used.

Subsequently, the feature quantity difference calculator 15E calculates a feature quantity likelihood from the feature quantity difference calculated in step S105 (step S106).

By repeating such loop processing 2, the feature quantity likelihood can be obtained for each of the number K of types of feature quantities. When the loop process 2 ends, the feature amount difference calculation unit 15E calculates one representative value representing the K types of feature amount likelihoods, for example, the minimum value, as the skill likelihood (step S107). Furthermore, by repeating the loop process 1, the skill likelihood can be obtained for each number L of training data.

When the loop process 1 ends, the scoring support unit 15F calculates the minimum skill likelihood among the L skill likelihoods obtained in the loop process 1 and the training data corresponding to the minimum skill likelihood. The client terminal 30 displays the feature amount likelihood of the type (step S108), and the process ends.

<One aspect of the effect>
As described above, the server device 10 according to the present embodiment is configured to match the feature amount obtained from the skeleton information input to the machine learning model for technique recognition with the same technique as the technique output by the machine learning model for technique recognition. provides the difference from the feature value obtained from the skeleton information of the training data. For this reason, it is possible to express the true likelihood that the technique is being performed and to provide information on an index suitable for the human judgment process. Therefore, according to the server device 10 of the present embodiment, it is possible to improve the quality of providing information on the recognition results of tricks.

Now, in the above-described first embodiment, there is an example in which the feature value difference, and thus the feature value likelihood and the skill likelihood are calculated for each number L of training data to which the label of the same skill as the skill recognized in the skill recognition is given. However, it is not always necessary to calculate the difference of the feature amount for each of the L pieces of training data.

In other words, when calculating the feature amount, the scoring items differ depending on the technique recognized by the technique recognition. A process such as specifying a frame to be used for calculating a feature amount is performed from the frame.

From the aspect of reducing such processing load, in this embodiment, the difference in 3D skeleton coordinates is calculated between the input data used for technique recognition and the training data corresponding to the technique recognition result. An application example will be described in which certain training data is narrowed down as an object for calculating the difference of the feature quantity.

FIG. 10 is a diagram illustrating a functional configuration example of the server device 20 according to the second embodiment. In FIG. 10, functional units having the same functions as those of the server device 10 shown in FIG. As shown in FIG. 10, server device 20 differs from server device 10 shown in FIG. 6 in that control unit 25 further includes skeleton difference calculation unit 25A.

The skeleton difference calculation unit 25A is a processing unit that calculates a difference between 3D skeleton coordinates. Hereinafter, the difference in 3D skeleton coordinates may be referred to as "skeletal difference".

By way of example only, the skeleton difference calculation unit 25A can start processing when a technique is recognized by the technique recognition unit 15C, for example, when the label output by the machine learning model 8M is not detected, ie, no technique.

For example, the skeleton difference calculation unit 25A calculates the difference in 3D skeleton coordinates between the input data used for technique recognition and the training data corresponding to the technique recognition result. For example, as the input data, time-series data of 3D skeleton coordinates obtained by skeleton recognition by the skeleton recognition unit 15B, and from another point of view, time-series data of 3D skeleton coordinates used for technique recognition by the technique recognition unit 15C are used. be done. On the other hand, the 3D skeleton coordinates corresponding to the training data labeled with the same technique as the technique recognized by the technique recognition unit 15C in the data set 8TR used to generate the machine learning model 8M as the training data. Series data are used.

More specifically, the skeleton difference calculation unit 25A performs the following processing for each of L pieces of training data labeled with the same technique as the technique recognized by the technique recognition unit 15C. That is, when the input data and the training data have different numbers of frames of time-series data of 3D skeleton coordinates, the skeleton difference calculation unit 25A unifies the numbers of frames of both. When unifying the number of frames between the input data and the training data in this way, the larger number of frames can be matched with the smaller number of frames, or the smaller number of frames can be matched with the larger number of frames. can. For example, when matching a smaller number of frames to a larger number of frames, resampling is required to interpolate the missing frames. On the other hand, if you want to match the larger number of frames to the smaller number of frames, you only need to thin out the excess frames. From this aspect, the skeleton difference calculation unit 25A thins out the 3D skeleton coordinates corresponding to the excess number of frames from the time-series data of the 3D skeleton coordinates that have the larger number of frames in the input data and the training data. , to unify the number of frames of both.

After unifying the number of frames, the skeletal difference calculation unit 25A calculates the difference in 3D skeletal coordinates between the input data and the training data for each of the unified M frames and for each N of joints defined in the skeletal model. , that is, the skeleton difference is calculated. The "difference" obtained here may be an absolute value or a square value. After calculating the skeleton difference for each of the N joints, the skeleton difference calculator 25A calculates the total value of the N skeleton differences. After that, when the total value of N skeleton differences is calculated for each of M frames, the skeleton difference calculator 25A calculates an average value from the total values of N skeleton differences for M frames. By calculating the average value of N skeletal differences as a representative value in this way, even if the number of frames M to be unified between training data and input data is different for each L training data, L It eliminates the need to adjust the number of frames between training data. Then, when the average value of the N skeleton differences is calculated for each of the L pieces of training data, the skeleton difference calculation unit 25A determines that the average value of the N skeleton differences among the L pieces of training data is the smallest. Select some training data.

As a result, among the L pieces of training data, the training data having the shortest distance from the input data or having the highest degree of similarity to the input data can be narrowed down.

FIG. 11 is a flowchart illustrating the procedure of information provision processing according to the second embodiment. The process illustrated in FIG. 11 can also be started, by way of example only, when a new frame of the depth image is acquired. As shown in FIG. 11, the skeleton recognition unit 15B inputs the depth image acquired by the acquisition unit 15A to the machine learning model 7M, thereby outputting the estimated values of the 3D skeleton coordinates of the performer 3 to the machine learning model 7M. Skeleton recognition is executed by causing the robot to perform skeleton recognition (step S101).

Subsequently, the technique recognition unit 15C inputs the time-series data of the 3D skeleton coordinates for a specific number of frames retroactively from the new frame to the machine learning model 8M, thereby recognizing the technique recognition result of the performer 3 as the machine learning model. By causing 8M to output, technique recognition is executed (step S102).

At this time, if the technique is recognized in the technique recognition of step S102, for example, if the label output by the machine learning model 8M is not detected, that is, if the technique is not absent (Yes in step S103), the skeleton difference calculation unit 25A performs the following Execute the process.

That is, the skeleton difference calculation unit 25A repeats the processing from step S201 to step S204 the number of times corresponding to the number L of training data to which the label of the same technique as the technique recognized in the technique recognition in step S102 is assigned. A loop process 11 is started. Although an example in which loop processing is performed is given here, the processing from step S201 to step S204 may be performed in parallel for each number L of training data.

The skeleton difference calculation unit 25A thins out 3D skeleton coordinates corresponding to the excess number of frames from the time-series data of the 3D skeleton coordinates of whichever of the input data and the training data has the larger number of frames, thereby calculating the numbers of both frames. Unify (step S201). Note that if the number of frames is the same between the input data and the training data, the process of step S201 can be skipped.

Thereafter, the skeleton difference calculation unit 25A starts loop processing 12 that repeats the processing of steps S202 and S203 the number of times corresponding to the number M of frames unified in step S201. Although an example in which loop processing is performed is given here, the processing of steps S202 and S203 may be performed in parallel for each number M of frames.

Further, the skeleton difference calculation unit 25A starts loop processing 13 for repeating the processing of step S202 a number of times corresponding to the number N of joints defined in the skeleton model. Although an example in which loop processing is performed is given here, the processing of step S202 may be performed in parallel for each number N of joints.

That is, the skeleton difference calculation unit 25A calculates the difference in 3D skeleton coordinates, that is, the skeleton difference, between the input data and the training data (step S202). By repeating such loop processing 13, a skeleton difference can be obtained for each of the N joints.

When the loop process 13 ends, the skeleton difference calculation unit 25A calculates the total value of N skeleton differences (step S203). By repeating such loop processing 12, the total value of N skeleton differences can be obtained for every M frames.

When loop processing 12 ends, the skeleton difference calculation unit 25A calculates an average value from the total values of N skeleton differences for M frames (step S204). By repeating such a loop process 11, it is possible to obtain an average value of N skeletal differences for every L training data.

Then, when the loop process 11 ends, the skeleton difference calculation unit 25A selects the training data having the smallest average value of N skeleton differences among the L pieces of training data (step S205).

After that, a loop process 14 is started in which the processes from step S206 to step S208 are repeated a number of times corresponding to the number K of feature amount types corresponding to the scoring items of the technique recognized in the technique recognition in step S102. Note that although an example of loop processing is described here, the processing from step S206 to step S208 may be performed in parallel for each type number K of feature amounts.

That is, the feature amount calculation unit 15D calculates the feature amount corresponding to the scoring item of the technique recognized in the technique recognition of step S102 (step S206).

Then, the feature amount difference calculating unit 15E calculates the feature amount difference between the input data used for technique recognition and the training data selected in step S205 (step S207). For example, the feature amount calculated in step S206 is used as the feature amount of the input data. On the other hand, as the feature amount of the training data, the feature amount calculated from the time-series data of the 3D skeletal coordinates corresponding to the training data selected in step S205 in the data set 8TR used to generate the machine learning model 8M is Used.

Subsequently, the feature quantity difference calculation unit 15E calculates a feature quantity likelihood from the feature quantity difference calculated in step S207 (step S208).

By repeating such a loop process 14, it is possible to obtain the feature amount likelihood for each of the number K of types of feature amounts. When the loop process 14 ends, the feature amount difference calculation unit 15E calculates one representative value representing the K types of feature amount likelihoods, for example, the minimum value, as the technique likelihood (step S209).

Finally, the scoring support unit 15F causes the client terminal 30 to display the skill likelihood calculated in step S209 and the K types of feature quantity likelihood obtained in the loop processing 14 (step S210), and ends the process. .

As described above, in the server device 20 according to the present embodiment, similar to the server device 10 according to the first embodiment, input data used for technique recognition and training data corresponding to the recognition result of the technique A difference in feature amount corresponding to the scoring item of the technique is calculated and output. For this reason, it is possible to provide an index that expresses the true likelihood that the technique is being performed and that is suitable for the human judgment process. Therefore, also in the server device 20 according to the present embodiment, it is possible to improve the quality of providing information on the recognition results of tricks.

Further, the server device 20 according to the present embodiment calculates the difference in 3D skeleton coordinates between the input data used for technique recognition and the training data corresponding to the technique recognition result, and selects the training data with the smallest difference. Narrow down the calculation target of the difference of the feature amount. For this reason, according to the server device 20 according to the present embodiment, it is possible to identify the type of feature amount to be calculated, and use the time-series data of the 3D skeleton coordinates for each type of feature amount to calculate the feature amount. It is possible to reduce the load of processing such as frame identification.

Although embodiments of the disclosed apparatus have been described so far, the present invention may be embodied in various forms other than the embodiments described above. Therefore, other embodiments included in the present invention will be described below.

<Training data>
In the above-described first embodiment, an example of calculating the feature quantity likelihood and the technique likelihood for each of L pieces of training data labeled with the same technique as the technique recognized in the technique recognition in step S102 shown in FIG. mentioned. Further, in the above-described second embodiment, an example of calculating a skeleton difference for each of L pieces of training data labeled with the same technique as the technique recognized in the technique recognition in step S102 shown in FIG. 11 was given. These calculation targets do not necessarily have to be the L pieces of training data.

As an example only, one training data representative of the L training data is generated by calculating a statistical value of the 3D skeleton coordinates, such as an arithmetic average or a weighted average, among the L training data. It is possible to use the training data obtained for calculation. As another example, the difference in 3D skeleton coordinates is calculated for each pair of training data that can be extracted from L training data, and the pair of training data having the largest difference in 3D skeleton coordinates among all pairs is used as the calculation target. can also be used.

<Skeleton difference>
In the above-described second embodiment, an example of using skeleton differences from the aspect of narrowing down L training data to one training data was given, but the skeleton differences selected in step S205 shown in FIG. can be output together with the technique likelihood and feature likelihood.

By providing such skeleton difference information, as described below, it is possible to cope with situations in which the skill likelihood and the feature quantity likelihood alone are insufficient. For example, take a technique called ``forward roll'', which rotates forward from a sitting position, stands upside down, and then rotates further forward to a sitting position. The feature values of this technique are: "starts with feet on", "rotates 180 degrees", "stands upside down with hands on", "rotates 180 degrees", and "ends with feet on". is a feature quantity. On the other hand, there may be a situation in which the player suddenly grabs the balance beam to avoid falling headfirst from the balance beam due to a failure in some trick, and then makes a full turn and falls. In this case, depending on the situation, the technique likelihood and the feature quantity likelihood may be calculated by falling from the state where the feet are on the ground, grabbing the balance beam in the vicinity of 180° rotation, rotating 180° again, and landing on the feet. When compared, it may become a movement similar to a forward roll. Even in such a case, since a difference is likely to appear in the skeletal difference between when the technique fails and when the forward roll is performed, it can be used to distinguish between when the technique fails and when the forward roll is performed.

<Decentralization and integration>
Also, each component of each illustrated device may not necessarily be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured. For example, the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the feature amount calculation unit 15D, the feature amount difference calculation unit 15E, or the scoring support unit 15F may be connected as external devices of the server device 10 via a network. good. In addition, the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the skeleton difference calculation unit 25A, the feature amount calculation unit 15D, the feature amount difference calculation unit 15E, or the scoring support unit 15F are external devices of the server device 20 via the network. You may make it connect. In addition, the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the feature amount calculation unit 15D, the feature amount difference calculation unit 15E, or the scoring support unit 15F may be provided by separate devices, connected to a network, and cooperated. , the functions of the server device 10 may be realized. Further, another device has the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the skeleton difference calculation unit 25A, the feature amount calculation unit 15D, the feature amount difference calculation unit 15E, or the scoring support unit 15F. The functions of the server device 20 may be realized by being and cooperating.

<Hardware configuration>
Moreover, various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a work station. Therefore, an example of a computer that executes an information providing program having functions similar to those of the first and second embodiments will be described below with reference to FIG.

FIG. 12 is a diagram illustrating a hardware configuration example. As shown in FIG. 12, the computer 100 has an operation section 110a, a speaker 110b, a camera 110c, a display 120, and a communication section . Furthermore, this computer 100 has a CPU 150 , a ROM 160 , an HDD 170 and a RAM 180 . Each part of these 110 to 180 is connected via a bus 140 .

As shown in FIG. 12, the HDD 170 includes the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the feature amount calculation unit 15D, the feature amount difference calculation unit 15E, and the scoring support unit shown in the first embodiment. An information providing program 170a having the same function as 15F can be stored. The HDD 170 also includes the acquisition unit 15A, the skeleton recognition unit 15B, the technique recognition unit 15C, the skeleton difference calculation unit 25A, the feature amount calculation unit 15D, the feature amount difference calculation unit 15E, and the scoring support unit 15F described in the second embodiment. An information providing program 170a that exhibits the same function as that can be stored. This information providing program 170a may be integrated or separated like each component shown in FIG. 6 or FIG. In other words, the HDD 170 does not necessarily store all the data shown in the first and second embodiments, and the HDD 170 only needs to store data used for processing.

Under such an environment, the CPU 150 reads out the information providing program 170 a from the HDD 170 and develops it in the RAM 180 . As a result, the information providing program 170a functions as an information providing process 180a as shown in FIG. The information providing process 180a deploys various data read from the HDD 170 in an area assigned to the information providing process 180a among the storage areas of the RAM 180, and executes various processes using the deployed various data. For example, examples of processing executed by the information providing process 180a may include the processing shown in FIGS. 9 and 11, and the like. Note that the CPU 150 does not necessarily have to operate all the processing units described in the first embodiment, as long as the processing units corresponding to the processes to be executed are virtually realized.

The information providing program 170a does not necessarily have to be stored in the HDD 170 or the ROM 160 from the beginning. For example, the information providing program 170a is stored in a “portable physical medium” such as a flexible disk inserted into the computer 100, a so-called FD, CD-ROM, DVD disk, magneto-optical disk, IC card, or the like. Then, the computer 100 may acquire and execute the information providing program 170a from these portable physical media. Also, the information providing program 170a is stored in another computer or server device connected to the computer 100 via a public line, Internet, LAN, WAN, or the like. The information providing program 170a stored in this manner may be downloaded to the computer 100 and executed.

The following notes are further disclosed with respect to the embodiments including the above examples.

(Appendix 1) calculating a feature value corresponding to a technique output by the machine learning model for technique recognition from the skeleton information input to the machine learning model for technique recognition;
The difference between the calculated feature amount and the feature amount obtained from the training data labeled with the same technique as the technique among the training data of the skeleton information used for training the machine learning model for the technique recognition is calculated. calculate,
Output the difference of the calculated feature amount,
A method of providing information characterized in that processing is executed by a computer.

(Appendix 2) Between the skeleton information input to the machine learning model for technique recognition and a plurality of training data with the same technique label as the technique output by the machine learning model for technique recognition The computer further executes a process of calculating a skeleton difference,
The process of calculating the feature amount difference is performed by comparing the feature amount obtained from one of the plurality of training data having the smallest skeleton difference and the feature amount calculated in the feature amount calculation process. Including processing to calculate the difference,
The information providing method according to Supplementary Note 1, characterized by:

(Appendix 3) The process of calculating the skeleton difference is performed by comparing the skeleton information input to the machine learning model for technique recognition and the statistical value of the skeleton information calculated from the plurality of training data. Including the process of calculating
The information providing method according to appendix 2, characterized by:

(Additional remark 4) In the process of calculating the skeletal difference, the skeletal difference is calculated for each pair of training data extracted from the plurality of training data, and the pair of training data having the largest skeletal difference among the pairs is calculated. , a process of calculating the skeleton difference between the skeleton information input to the machine learning model for technique recognition,
The information providing method according to appendix 2, characterized by:

(Appendix 5) The outputting process includes a process of outputting the likelihood of the feature amount in which the difference of the feature amount is normalized.
The information providing method according to Supplementary Note 1, characterized by:

(Appendix 6) The process of calculating the feature amount includes a process of calculating the feature amount for each type of feature amount corresponding to the technique output by the machine learning model for technique recognition,
The process of calculating the difference in the feature amount includes a process of calculating the difference in the feature amount for each type of the feature amount,
The outputting process includes outputting a statistical value representing the likelihood of a plurality of feature amounts obtained by normalizing the feature amount difference for each type of the feature amount,
The information providing method according to Supplementary Note 1, characterized by:

(Appendix 7) calculating a feature value corresponding to the technique output by the machine learning model for technique recognition from the skeleton information input to the machine learning model for technique recognition;
The difference between the calculated feature amount and the feature amount obtained from the training data labeled with the same technique as the technique among the training data of the skeleton information used for training the machine learning model for the technique recognition is calculated. calculate,
Output the difference of the calculated feature amount,
An information providing program characterized by causing a computer to execute processing.

(Appendix 8) Between the skeleton information input to the machine learning model for technique recognition and a plurality of training data labeled with the same technique as the technique output by the machine learning model for technique recognition causing the computer to further execute a process of calculating a skeleton difference;
The process of calculating the feature amount difference is performed by comparing the feature amount obtained from one of the plurality of training data having the smallest skeleton difference and the feature amount calculated in the feature amount calculation process. Including processing to calculate the difference,
The information providing program according to Supplementary Note 7, characterized by:

(Appendix 9) The process of calculating the skeleton difference is performed by comparing the skeleton information input to the machine learning model for technique recognition and the statistical value of the skeleton information calculated from the plurality of training data. Including the process of calculating
The information providing program according to appendix 8, characterized by:

(Supplementary Note 10) The process of calculating the skeletal difference calculates the skeletal difference for each pair of training data extracted from the plurality of training data. , a process of calculating the skeleton difference between the skeleton information input to the machine learning model for technique recognition,
The information providing program according to appendix 8, characterized by:

(Appendix 11) The process of outputting includes a process of outputting the likelihood of the feature amount obtained by normalizing the difference of the feature amount.
The information providing program according to Supplementary Note 7, characterized by:

(Appendix 12) The process of calculating the feature amount includes a process of calculating the feature amount for each type of feature amount corresponding to the technique output by the machine learning model for technique recognition,
The process of calculating the difference in the feature amount includes a process of calculating the difference in the feature amount for each type of the feature amount,
The outputting process includes outputting a statistical value representing the likelihood of a plurality of feature amounts obtained by normalizing the feature amount difference for each type of the feature amount,
The information providing program according to Supplementary Note 7, characterized by:

(Appendix 13) A sensor device that acquires a depth image;
a skeleton recognition unit that performs skeleton recognition on the depth image; a technique recognition unit that recognizes a technique by inputting the skeleton information obtained by the skeleton recognition into a machine learning model for technique recognition; and the technique recognition machine. A feature amount calculation unit that calculates a feature amount corresponding to a technique output by the machine learning model for technique recognition from skeleton information input to the learning model, and the calculated feature amount and the machine learning model for technique recognition. A feature amount difference calculation unit that calculates the difference between the feature amount obtained from the training data labeled with the same technique as the above technique among the training data of the skeleton information used in the training of , and the calculated feature amount of the a server device having an information output unit that outputs the difference to a client terminal;
A scoring support system characterized by having:

(Appendix 14) Between the skeleton information input to the machine learning model for technique recognition and a plurality of training data having the same technique label as the technique output by the machine learning model for technique recognition The server device further has a skeleton difference calculation unit that calculates a skeleton difference,
The feature amount difference calculation unit calculates a difference between a feature amount obtained from one of the plurality of training data having the smallest skeleton difference and the feature amount calculated in the process of calculating the feature amount. The scoring support system according to Supplementary Note 13, characterized by:

(Appendix 15) The skeleton difference calculation unit calculates the skeleton difference between skeleton information input to the machine learning model for technique recognition and statistical values of skeleton information calculated from the plurality of training data. 15. The scoring support system according to Supplementary Note 14, characterized by:

(Supplementary Note 16) The skeleton difference calculation unit calculates a skeleton difference for each pair of training data extracted from the plurality of training data, and the pair of training data having the largest skeleton difference among the pairs, and the 15. The scoring support system according to Supplementary note 14, wherein the skeleton difference is calculated between skeleton information input to a machine learning model for technique recognition.

(Supplementary note 17) The scoring support system according to Supplementary note 13, wherein the information output unit outputs the likelihood of the feature amount obtained by normalizing the difference of the feature amount.

(Appendix 18) The feature amount calculation unit calculates the feature amount for each type of feature amount corresponding to the technique output by the machine learning model for technique recognition,
The feature amount difference calculation unit calculates the difference between the feature amounts for each type of the feature amount,
The scoring according to Supplementary Note 13, wherein the information output unit outputs a statistical value representing the likelihood of a plurality of feature amounts obtained by normalizing the difference between the feature amounts for each type of the feature amount. support system.

1 Scoring support system 3 Performer 5 3D laser sensor 10 Server device 11 Communication interface unit 13 Storage unit 15 Control unit 15A Acquisition unit 15B Skeleton recognition unit 15C Technique recognition unit 15D Feature amount calculation unit 15E Feature amount difference calculation unit 15F Scoring support Department

Claims (8)

calculating a feature amount corresponding to a technique output by the machine learning model for technique recognition from skeleton information input to the machine learning model for technique recognition;
The difference between the calculated feature amount and the feature amount obtained from the training data labeled with the same technique as the technique among the training data of the skeleton information used for training the machine learning model for the technique recognition is calculated. calculate,
Output the difference of the calculated feature amount,
A method of providing information characterized in that processing is executed by a computer. Calculate a skeleton difference between the skeleton information input to the machine learning model for technique recognition and a plurality of training data labeled with the same technique as the technique output by the machine learning model for technique recognition. The computer further executes a process to
The process of calculating the feature amount difference is performed by comparing the feature amount obtained from one of the plurality of training data having the smallest skeleton difference and the feature amount calculated in the feature amount calculation process. Including processing to calculate the difference,
2. The information providing method according to claim 1, characterized by: The process of calculating the skeleton difference is a process of calculating the skeleton difference between the skeleton information input to the machine learning model for technique recognition and the statistical value of the skeleton information calculated from the plurality of training data. including,
3. The information providing method according to claim 2, characterized by: The process of calculating the skeleton difference calculates the skeleton difference for each pair of training data extracted from the plurality of training data, and calculates the training data of the pair having the largest skeleton difference among the pairs, and the technique recognition data. including a process of calculating the skeleton difference between the skeleton information input to the machine learning model for
3. The information providing method according to claim 2, characterized by: The process of outputting includes a process of outputting the likelihood of the feature amount obtained by normalizing the difference of the feature amount,
The information providing method according to any one of claims 1 to 4, characterized in that: The process of calculating the feature amount includes a process of calculating the feature amount for each type of feature amount corresponding to the technique output by the machine learning model for technique recognition,
The process of calculating the difference in the feature amount includes a process of calculating the difference in the feature amount for each type of the feature amount,
The outputting process includes outputting a statistical value representing the likelihood of a plurality of feature amounts obtained by normalizing the feature amount difference for each type of the feature amount,
The information providing method according to any one of claims 1 to 4, characterized in that: calculating a feature amount corresponding to a technique output by the machine learning model for technique recognition from skeleton information input to the machine learning model for technique recognition;
The difference between the calculated feature amount and the feature amount obtained from the training data labeled with the same technique as the technique among the training data of the skeleton information used for training the machine learning model for the technique recognition is calculated. calculate,
Output the difference of the calculated feature amount,
An information providing program characterized by causing a computer to execute processing. a sensor device that acquires a depth image;
a skeleton recognition unit that performs skeleton recognition on the depth image; a technique recognition unit that recognizes a technique by inputting the skeleton information obtained by the skeleton recognition into a machine learning model for technique recognition; and the technique recognition machine. A feature amount calculation unit that calculates a feature amount corresponding to a technique output by the machine learning model for technique recognition from skeleton information input to the learning model, and the calculated feature amount and the machine learning model for technique recognition. A feature amount difference calculation unit that calculates the difference between the feature amount obtained from the training data labeled with the same technique as the above technique among the training data of the skeleton information used in the training of , and the calculated feature amount of the a server device having an information output unit that outputs the difference to a client terminal;
A scoring support system characterized by having:

Images