CN114578367B - Real-time motion trail monitoring method, device, equipment and readable storage medium - Google Patents

Abstract

The application relates to a real-time motion trail monitoring method, a device, equipment and a readable storage medium, which relate to the technical field of motion monitoring, wherein point cloud data is obtained in real time from a point cloud database through a rotation vector sent by a wearable device, the distance between an intelligent terminal and the wearable device is measured by utilizing frequency modulation continuous wave ultrasonic dynamic ranging, and real-time real position coordinates of a joint are obtained through prediction through a hidden Markov model based on the point cloud data, the distance between the intelligent terminal and the wearable device and instantaneous acceleration information sent by the wearable device, so that the monitoring and tracking of the motion trail of the joint are realized. According to the application, the real-time monitoring and tracking of the joint movement are realized through the intelligent terminal and the wearable equipment, the use cost can be effectively reduced, the movement track is monitored through the rotation vector information and the acceleration information acquired by the wearable equipment, and the accuracy of the movement track monitoring can be effectively ensured.

Description

Real-time motion trajectory monitoring method, device, equipment and readable storage medium

Technical Field

The present application relates to the field of motion monitoring technology, and in particular to a real-time motion trajectory monitoring method, device, equipment and readable storage medium.

Background technique

As the economy develops and people's living standards improve, their awareness of fitness is also increasing. For example, more and more people are gradually moving from indoors to outdoors and choosing appropriate fitness exercises to improve their physical fitness. However, during exercise, people's improper exercise postures often lead to a variety of muscle and bone-related health problems. In more serious cases, incorrect fitness movements can damage internal organs. Therefore, it is very important to correct people's improper exercise postures through reasonable exercise monitoring methods.

In the related art, the monitoring of motion posture and trajectory is often achieved by establishing a human skeleton model. However, most of the high-precision real-time modeling methods in the prior art need to be based on images or videos to be realized, which means that the monitoring of human posture and trajectory depends on the quality of the image or video, in other words, it depends on the quality of the shooting equipment. Therefore, although the image or video-based method can have sufficient accuracy and real-time performance under the premise of high-quality images or high-definition videos, it has the problem of high cost, and for outdoor athletes, it is obviously unrealistic to arrange high-cost tracking equipment to track them in real time during exercise; in addition, high-cost shooting equipment still cannot obtain high-definition pictures under poor light conditions. It can be seen that the high-precision real-time modeling method based on images or videos is still greatly affected by environmental factors (such as light, temperature and humidity, etc.), which may lead to errors in the modeling results.

Summary of the invention

The present application provides a real-time motion trajectory monitoring method, device, equipment and readable storage medium to solve the problems of high cost and great influence of environmental factors existing in the related technology.

In a first aspect, a real-time motion trajectory monitoring method is provided, comprising the following steps:

Receiving a first rotation vector and a first instantaneous acceleration information of a first joint at a time T sent by a wearable device;

Based on the first rotation vector, a corresponding first position set is found from a preset point cloud database, where the preset point cloud database includes a mapping relationship between the rotation vector and the position set, and the position set includes a plurality of preset position coordinates corresponding to the joint;

Acquire a second position set of the first joint at time T-1, and filter out a first point cloud state space from a preset point cloud database according to the second position set and the first position set, wherein the first point cloud state space includes a plurality of point cloud states, each point cloud state being composed of preset position coordinates in the first position set and preset position coordinates in the second position set;

Calculate the distance between the smart terminal and the wearable device at time T based on frequency modulated continuous wave ranging to obtain a first distance;

Respectively obtain the second distance between the smart terminal and the wearable device at the T-1th moment, the third distance between the smart terminal and the wearable device at the T-2th moment, the first actual position coordinates of the first joint at the T-1th moment, and the second actual position coordinates of the first joint at the T-2th moment;

Calculate the location coordinates of the smart terminal at time T based on the first distance, the second distance, the third distance, the first actual location coordinates, and the second actual location coordinates;

The first instantaneous acceleration information, the first point cloud state space and the position coordinates of the intelligent terminal at the Tth moment are input into the hidden Markov model to obtain the third actual position coordinates of the first joint at the Tth moment.

In some embodiments, before the step of receiving the first rotation vector and the first instantaneous acceleration information of the first joint at the T moment sent by the wearable device, the method further includes:

The arm motion is modeled based on the D-H model to obtain the linkage model;

Inputting multiple joint angles on the arm into the link model to obtain multiple rotation vectors;

A mapping relationship between each rotation vector and its corresponding position set including a plurality of preset position coordinates is created to obtain a point cloud database.

In some embodiments, the model parameters in the hidden Markov model include an initial probability distribution, a state transition probability distribution, and an output probability distribution, the probability expression of the state transition probability distribution includes instantaneous acceleration information, the location coordinates of the smart terminal, and the distance between the smart terminal and the wearable device, and the output probability distribution is 1.

In some embodiments, the wearable device includes an inertial sensor and an acceleration sensor, the inertial sensor is used to collect the first rotation vector of the first joint at the Tth moment, and the acceleration sensor is used to collect the first instantaneous acceleration information of the first joint at the Tth moment.

In a second aspect, a real-time motion trajectory monitoring device is provided, comprising:

A receiving unit, configured to receive a first rotation vector and a first instantaneous acceleration information of a first joint at a time T sent by a wearable device;

A search unit, which is used to search for a corresponding first position set from a preset point cloud database based on the first rotation vector, the preset point cloud database includes a mapping relationship between the rotation vector and the position set, and the position set includes a plurality of preset position coordinates corresponding to the joint;

a generating unit, which is used to obtain a second position set of the first joint at the time T-1, and filter out a first point cloud state space from a preset point cloud database according to the second position set and the first position set, wherein the first point cloud state space includes a plurality of point cloud states, each point cloud state is composed of preset position coordinates in the first position set and preset position coordinates in the second position set;

A distance measuring unit, which is used to calculate the distance between the smart terminal and the wearable device at the Tth moment based on the frequency modulated continuous wave ranging to obtain a first distance;

an acquisition unit, which is used to respectively acquire a second distance between the smart terminal and the wearable device at the T-1th moment, a third distance between the smart terminal and the wearable device at the T-2th moment, a first actual position coordinate of the first joint at the T-1th moment, and a second actual position coordinate of the first joint at the T-2th moment;

A calculation unit, which is used to calculate the position coordinates of the smart terminal at the Tth moment based on the first distance, the second distance, the third distance, the first actual position coordinates and the second actual position coordinates;

The prediction unit is used to input the first instantaneous acceleration information, the first point cloud state space and the position coordinates of the intelligent terminal at the Tth moment into the hidden Markov model to obtain the third actual position coordinates of the first joint at the Tth moment.

In some embodiments, the apparatus further comprises a creating unit, which is configured to:

The arm motion is modeled based on the D-H model to obtain the linkage model;

Inputting multiple joint angles on the arm into the link model to obtain multiple rotation vectors;

A mapping relationship between each rotation vector and its corresponding position set including a plurality of preset position coordinates is created to obtain a point cloud database.

In some embodiments, the model parameters in the hidden Markov model include an initial probability distribution, a state transition probability distribution, and an output probability distribution, the probability expression of the state transition probability distribution includes instantaneous acceleration information, the location coordinates of the smart terminal, and the distance between the smart terminal and the wearable device, and the output probability distribution is 1.

In some embodiments, the wearable device includes an inertial sensor and an acceleration sensor, the inertial sensor is used to collect the first rotation vector of the first joint at the Tth moment, and the acceleration sensor is used to collect the first instantaneous acceleration information of the first joint at the Tth moment.

In a third aspect, a real-time motion trajectory monitoring device is provided, comprising: a memory and a processor, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the aforementioned real-time motion trajectory monitoring method.

In a fourth aspect, a computer-readable storage medium is provided, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned real-time motion trajectory monitoring method is implemented.

The beneficial effects brought about by the technical solution provided in this application include: it can realize real-time monitoring and tracking of joint movement, effectively reduce the cost of use, and reduce the impact of external environmental factors on the accuracy of rotation vector information and acceleration information collection, so as to effectively ensure the accuracy of motion trajectory monitoring.

The present application provides a real-time motion trajectory monitoring method, device, equipment and readable storage medium, which obtains point cloud data from a point cloud database in real time through the rotation vector sent by the wearable device, and uses frequency-modulated continuous wave ultrasonic dynamic ranging to measure the distance between the smart terminal and the wearable device, and then based on the point cloud data, the distance between the smart terminal and the wearable device, and the instantaneous acceleration information sent by the wearable device, the real-time true position coordinates of the joint are obtained through hidden Markov model prediction, thereby realizing the monitoring and tracking of the joint motion trajectory. It can be seen that the present application only needs to use a smart terminal and a wearable device to realize real-time monitoring and tracking of joint motion, and it does not need to arrange high-cost tracking shooting equipment, which effectively reduces the cost of use, and the rotation vector information and acceleration information collected by the wearable device are used to realize the monitoring of the motion trajectory, which can effectively reduce the influence of external environmental factors on the accuracy of the rotation vector information and acceleration information collection, thereby effectively ensuring the accuracy of motion trajectory monitoring.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of a flow chart of a real-time motion trajectory monitoring method provided in an embodiment of the present application;

FIG. 2 is a schematic diagram of the structure of a real-time motion trajectory monitoring device provided in an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

The embodiments of the present application provide a real-time motion trajectory monitoring method, device, equipment and readable storage medium, which can solve the problems of high cost and great influence of environmental factors existing in the related technology.

FIG1 is a real-time motion trajectory monitoring method provided by an embodiment of the present application, comprising the following steps:

Step S10: receiving the first rotation vector and the first instantaneous acceleration information of the first joint at the time T sent by the wearable device;

Furthermore, the wearable device includes an inertial sensor and an acceleration sensor, the inertial sensor is used to collect a first rotation vector of the first joint at the Tth moment, and the acceleration sensor is used to collect first instantaneous acceleration information of the first joint at the Tth moment.

Furthermore, before the step of receiving the first rotation vector and the first instantaneous acceleration information of the first joint at the T moment sent by the wearable device, the method further includes:

The arm motion is modeled based on the D-H model to obtain the linkage model;

Inputting multiple joint angles on the arm into the link model to obtain multiple rotation vectors;

A mapping relationship between each rotation vector and its corresponding position set including a plurality of preset position coordinates is created to obtain a point cloud database.

Step S20: searching a corresponding first position set from a preset point cloud database based on the first rotation vector, wherein the preset point cloud database includes a mapping relationship between the rotation vector and the position set, and the position set includes a plurality of preset position coordinates corresponding to the joint;

Step S30: obtaining a second position set of the first joint at the time T-1, and filtering out a first point cloud state space from a preset point cloud database according to the second position set and the first position set, wherein the first point cloud state space includes a plurality of point cloud states, each point cloud state being composed of preset position coordinates in the first position set and preset position coordinates in the second position set;

Step S40: Calculate the distance between the smart terminal and the wearable device at the Tth moment based on the frequency modulated continuous wave ranging to obtain a first distance;

Step S50: respectively obtaining the second distance between the smart terminal and the wearable device at the T-1th moment, the third distance between the smart terminal and the wearable device at the T-2th moment, the first actual position coordinates of the first joint at the T-1th moment, and the second actual position coordinates of the first joint at the T-2th moment;

Step S60: Calculate the location coordinates of the smart terminal at time T based on the first distance, the second distance, the third distance, the first actual location coordinates, and the second actual location coordinates;

Step S70: input the first instantaneous acceleration information, the first point cloud state space and the position coordinates of the intelligent terminal at the Tth moment into the hidden Markov model to obtain the third actual position coordinates of the first joint at the Tth moment.

Furthermore, the model parameters in the hidden Markov model include an initial probability distribution, a state transition probability distribution and an output probability distribution, the probability expression of the state transition probability distribution includes instantaneous acceleration information, the position coordinates of the smart terminal and the distance between the smart terminal and the wearable device, and the output probability distribution is 1.

The present application can realize real-time monitoring and tracking of joint movements through only a smart terminal and a wearable device. It does not need to deploy high-cost tracking shooting equipment, effectively reducing the cost of use. The rotation vector information and acceleration information collected by the wearable device can be used to monitor the motion trajectory, which can effectively reduce the impact of external environmental factors on the accuracy of the rotation vector information and acceleration information collection, thereby effectively ensuring the accuracy of motion trajectory monitoring.

Exemplarily, with the widespread popularity of mobile wearable devices, wearable devices have gradually become popular in fields such as medical care, entertainment, safety and fitness; and the key technology that can provide high-precision activity data and provide many conveniences for human life lies in the motion sensing technology of smart wearable devices, which can monitor a person's walking, running and riding motion data by relying on motion recognition technology, and monitor people's behavior and actions through artificial intelligence methods, and then remind them of irregular motion problems during exercise to avoid fitness safety hazards that accumulate over time. Currently, widely used wearable devices include smart wristbands, smart bracelets and smart belts, etc., which mainly monitor people's behavior and actions by obtaining motion data of different parts of a person's body, and using a single wearable device to monitor motion behavior is its further development trend. In this embodiment, the rotation vector information and instantaneous acceleration information for monitoring real-time motion trajectory can be obtained through a wearable device, wherein the wearable device can be a smart bracelet, a smart wristband, or a smart belt, or other wearable devices that can realize the collection of rotation vector information and instantaneous acceleration information, so the specific selection of wearable devices can be determined according to actual needs, and is not limited here.

In this embodiment, before obtaining the rotation vector information and instantaneous acceleration information through the wearable device, a point cloud database is first created. Taking the wearable device as a smart bracelet as an example, the upper limb and the forearm of the arm are used as the connecting rod model, and the standard DH model (Denavit-Hartenberg model is widely used in robot kinematics, which represents a very simple method for modeling robot connecting rods and joints, and can be used to represent transformations in any coordinates) is used for modeling: the connecting rod coordinate system is established through four parameters: connecting rod length _Li , connecting rod offset _di , torsion angle _αi , and joint angle _θi , and after two translations and two rotations, the homogeneous transformation matrix _Ai of adjacent connecting rods as shown in formula (1) is established.

Among them, the joint angles θ _i include five, which respectively represent shoulder flexion/extension, abduction/adduction and internal/external rotation, elbow flexion/extension and forearm internal/external rotation. This embodiment establishes a mapping between wrist joint rotation vector information and position information by extensively traversing all motion states, that is, each rotation vector will have multiple preset position coordinates corresponding to it, and multiple preset coordinates form a position set, so a rotation vector will have a position set with a mapping relationship, and it is saved to form a point cloud database for point cloud motion monitoring.

In addition, the position sets corresponding to the two rotation vectors can form a point cloud state space, and the point cloud state space will contain multiple point cloud states, and each point cloud state is composed of preset position coordinates in the two position sets. For example, the rotation vector F corresponds to the position set F′, position set F′＝{F′ ₁ ,F′ ₂ }, F′ ₁ and F′ ₂ both represent the position coordinates corresponding to the rotation vector F; the rotation vector M corresponds to the position set M′, position set M′＝{M′ ₁ ,M′ ₂ ,M′ ₃ }, M′ ₁ , M′ ₂ and M′ ₃ all represent the position coordinates corresponding to the rotation vector M; then the point cloud state space S composed of the rotation vector F and the rotation vector M is S＝{(F′ ₁ ,M′ ₁ ),(F _{′ 1} ,M′ ₂ ),(F′ ₁ ,M′ ₃ ),(F′ 2 ,M′ ₁ ),(F′ ₂ ,M′ ₂ ),(F′ ₂ ,M′ ₃ )}, and the point cloud state space includes six point cloud states: (F′ ₁ ,M′ ₁ ),(F′ ₁ ,M′ ₂ ),(F′ ₁ ,M′ 3 )}. ₃ ), (F′ ₂ ,M′ ₁ ), (F′ ₂ ,M′ ₂ ), (F′ ₂ ,M′ ₃ ).

Therefore, this embodiment can respectively collect the rotation vector information and instantaneous acceleration information of each joint through the built-in inertial sensor and acceleration sensor of the smart bracelet, and monitor the human body's motion trajectory based on the rotation vector information and instantaneous acceleration information. Among them, the acceleration information collected by the smart bracelet in real time can be used as the observation quantity of the hidden Markov model to predict the position of the movement, and then output the predicted position triplet. It does not require the deployment of high-cost shooting equipment, and can effectively solve a series of adverse effects caused by factors such as light, movement distance, and sports venue.

In addition, based on the point cloud data formed by the smart bracelet, this embodiment also uses a FMCW (Frequency Modulated Continuous Wave) ultrasonic dynamic ranging method mixed with a point cloud prediction algorithm to constrain the point cloud data space, so that the movement trajectory of the arm can be monitored and tracked in real time through only a smart terminal and a smart bracelet worn on the wrist without increasing the user's usage cost.

FMCW ultrasonic ranging is to transmit a continuous wave with a changing frequency within a frequency sweep period. The echo reflected by the object has a certain frequency difference with the transmitted signal. By measuring the frequency difference, the distance information between the target and the source of the sound wave signal can be obtained. Since the difference frequency signal has a low frequency, it can be processed with relatively simple hardware, which is suitable for data acquisition and digital signal processing. And since FMCW ultrasonic ranging transmits and receives at the same time, in theory, there is no ranging blind spot that exists in pulse radar, and the average power of the transmitted signal is equal to the peak power, so only low-power devices are needed to achieve it, thereby reducing the probability of being intercepted and interfered.

The principle of FMCW ultrasonic ranging will be further explained below.

Assume _fs (t) and _fe (t) represent the frequency variation functions of the transmitted and received signals respectively, and assume that the relative speed _vr is 0, then the following relationship exists at the rising and falling edges of the signal:

_fe (t)= _fs (t-τ) (3)

In the formula, _f0 represents the lowest frequency, _fc represents the frequency bandwidth of the FMCW wave, _tc represents the period of the FMCW wave, t represents the time, and τ represents the time difference from the departure to the reception of the sound;

There is a frequency difference function f _b (t) between f _s (t) and _fe (t):

f _b (t) = f _s (t) - _fe (t) (4)

Assuming the emitted signal is a cosine wave, its change in the time domain is as follows:

In formula (5), u _s (t) is the sending signal function, represents the amplitude of the transmitted signal, _fs (t) represents the frequency of the transmitted signal, and _φs represents the phase difference of the transmitted signal;

The received signal changes in the time domain as follows:

In formula (6), _ue (t) represents the received signal function, represents the amplitude of the received signal, _fe (t) represents the frequency of the received signal, and _φe represents the phase difference of the received signal.

Based on equations (4) to (6), we can obtain:

Then, multiply _ue (t) and _us (t) to get the mixed wave _um (t):

From formula (8), we can see that the frequency of the final mixed wave is It is only related to f _b (t), and the relative distance to be determined/> c represents the speed of the sound wave, f _c represents the frequency bandwidth of the FMCW wave, and t _c represents the period of the FMCW wave. Among them, f _b (t) can be obtained by low-pass filtering the mixed wave _um (t) and then using Fourier transform, and the relative distance R is proportional to f _b (t). It can be seen that the relative distance R can be solved by the mixed wave.

Therefore, this embodiment calculates the distance between the smart terminal and the wearable device based on frequency modulated continuous wave ranging, and constrains the point cloud data space through this distance. Among them, the smart terminal can be a smart phone or a tablet. The specific choice can be determined according to actual needs and is not limited here. This embodiment takes the smart terminal as a smart phone as an example, and realizes the position ranging between the two devices by sending ultrasonic waves between the smart phone and the smart bracelet.

Ultimately, the relevant information obtained through smart bracelets and smartphones needs to be input into the hidden Markov model to predict the trajectory of human movement.

The sequence of states randomly generated by the hidden Markov chain in the hidden Markov model is called the state sequence; each state has an output, which generates an observable random number sequence, called the observation sequence, and the observation sequence can also be used as a time series. The hidden Markov model is determined by the initial probability distribution, the state transition probability distribution, and the output probability distribution, that is, λ＝(A,B,π); where λ represents the hidden Markov model, A is the state transition probability matrix, B is the output probability matrix, and π is the initial state probability vector. A, B, and π are called the three elements of the hidden Markov model.

Hidden Markov models are mainly used to solve the following three basic problems:

1) Probability calculation problem: Given a model λ = (A, B, π) and an observed sequence O = (o ₁ , o ₂ , …, o _T ), calculate the probability P(O|λ) of the sequence O occurring under the model λ.

2) Learning problem: Given an observation sequence O = (o ₁ , o ₂ , …, o _T ), estimate the parameters of the model λ = (A, B, π) to maximize the probability of the observation sequence under the model. The maximum likelihood estimation method is usually used to estimate the parameters.

3) Prediction problem, also called decoding problem: Given the model λ = (A, B, π) and the observation sequence O = (o ₁ , o ₂ , …, o _T ), calculate the hidden state sequence with the highest probability of outputting the observation sequence.

This application mainly involves prediction problems related to hidden Markov, and the Viterbi algorithm is currently the more mainstream prediction algorithm. Since it uses the idea of dynamic programming, it can solve the shortcomings of the greedy algorithm. In fact, it can also effectively find the state sequence with the highest probability, and is currently the most frequently used prediction algorithm. Therefore, this embodiment will use the Viterbi algorithm extensively in motion monitoring to infer the position of the limbs during movement, thereby describing the movement trajectory and providing reliable data basis for the evaluation of movement norms.

Specifically, the positions of three consecutive time intervals can be used as a point cloud state of the joint. At this time, the acceleration of the joint when it is in this point cloud state can be calculated, and the acceleration value can be used as the observation value. By properly designing the state transition probability, the Viterbi algorithm can be used to infer the most likely position point.

Among them, the Viterbi algorithm has a time complexity of O(|S| ² T), S is the size of the state space, T is the size of the time series; assuming that the state is a triple of positions, then S = N ³ , then the time complexity of the Viterbi algorithm is O(N ⁶ T). Therefore, in order to reduce the time complexity of the Viterbi algorithm, it is necessary to restrict the continuity of the position in the state, put the output probability into the state transition probability calculation and reduce the complexity of S, so that the final time complexity can be reduced to O(N ³ T). The specific implementation is as follows:

According to the abstract function λ＝(A, B, π) of the hidden Markov model, it is necessary to know the initial probability distribution, transition probability distribution and output probability distribution of the hidden state, and it is known from the Viterbi algorithm that an observable time series needs to be obtained; this embodiment uses acceleration information as the observed output and the displacement over a period of time ΔT as the hidden state (that is, the point cloud state), that is Indicates the position from time T-1/> Position at time T/> The displacement tuple is , i represents any state, and the time interval between time T-1 and time T can be set to 0.05s.

Regarding the initial probability distribution: Since the initial position is unclear, we can simply use a uniform distribution for all states. When T = 1, the state space size is N ² , but the range of motion of the joint is limited to a very limited space due to speed, and the higher the sampling rate, the smaller the space. Therefore, the state space can be reduced to αN ² , where α is much smaller than 1, then:

Regarding the state transition probability distribution: the state transition probability of transferring from state i corresponding to time T to time T+1 to state j is expressed by Pr=(state _j |state _i ; T, T+1); this probability consists of three parts: the first part, first of all, the motion trajectory is continuous, that is, there must be continuity between state i and state j, that is, the end point of state i is the starting point of state j. This continuity property can reduce the space complexity from O( ^N4T ) to O( ^N3T ), so the following function can be used to express this relationship:

Where I represents the 0-1 probability distribution, Indicates the position corresponding to the end point of state i at time T.

In the second part, unlike the traditional Viterbi algorithm, this embodiment directly introduces the observed value into the state transition probability distribution, that is, the position information in the state is used to represent the acceleration, that is, the predicted acceleration value accel _i,j , and the difference between the predicted acceleration and the instantaneous acceleration actually measured by the smart bracelet is calculated. Assuming that the difference should satisfy the Gaussian distribution, the probability of the second part can be expressed as:

Among them, σ is a constant, and accel _observe represents the instantaneous acceleration actually measured by the smart bracelet.

Part 3: For the new state j ₂ , its position It must belong to the location set Set _T+1 that can be queried from the point cloud database, that is:

The final state transition probability distribution can be expressed as:

Regarding the output probability distribution: since the observed quantity is introduced into the state probability distribution part in this embodiment, the output probability can be set to 1.

Then, the predicted states state T ^-1 and state ^T corresponding to time T-1 and time T are obtained from the Viterbi algorithm of the hidden Markov model, where state ^T-1 represents the predicted state value at time T-1, and state ^T represents the predicted state value at time T. Therefore, a unique triplet can be obtained from two continuous state spaces, namely <loc _T-2 , loc _T-1 , loc _T >, where loc _T-2 , loc _T-1 and loc _T represent the actual position coordinates of the joint points at time T-2, T-1 and T respectively; at the same time, the distances dist T-2, dist T-1 and dist T between the smart bracelet and the smart phone at time _T-2 , _T-1 and _T can be obtained from the FMCW ranging algorithm; based on the above six known information, the ellipsoid equation can be solved to obtain the position coordinates of the smart phone in the motion monitoring coordinate system. And in the Viterbi prediction algorithm at time T+1, it can be calculated by With/> In/> The distance is used to constrain the point cloud space, that is, to add the fourth part to the state transition probability in the hidden Markov model:

in, Indicates the T+1th moment, /> Status/> To/> The distance ε represents the minimum allowed error.

Therefore, the state transition probability can be rewritten as follows:

It can be seen that according to the hidden Markov model and Viterbi algorithm, the position of each joint can be solved to characterize the motion trajectory.

This embodiment will be further explained below.

This embodiment collects raw data at a fixed frequency of 200Hz, such as collecting the rotation vector Rot of the wrist joint at time T on the smart bracelet and the instantaneous acceleration information Acc, and outputs the data stream Q = {Rot, Acc} after 50Hz preprocessing through the mean downsampling method during data preprocessing; in the preprocessing stage, a low-pass filter is used to reduce noise on the data. Secondly, the obtained rotation vector Rot is queried in the preset point cloud database to find the corresponding position set, and the position set of the wrist joint at time T-1 is obtained from the preset point cloud database; then, based on these two position sets, the point cloud state space matching the rotation vector Rot is obtained from the preset point cloud database, and it is input into the hidden Markov model.

Then, the distance between the smart phone and the smart bracelet at the Tth moment is calculated by using the dual-transmit and dual-receive FMCW ultrasonic frequency band sound waves between the smart bracelet and the smart phone. Among them, this embodiment calculates the time difference from the emission to the reception of the FMCW sounds of two different frequency bands by a single-peak monitoring method, thereby compensating for the clock error between the two different smart devices, and at the same time, by selecting a triangular waveform to compensate for the Doppler effect caused by movement, and finally solves the distance.

Finally, the point cloud states obtained at time T-1 and time T-2 are combined with the distance information of the ranging to calculate the position coordinates of the smartphone at time T. The acceleration information is introduced as the observation quantity when calculating the probability of each state in each hidden Markov model prediction iteration. Finally, the most likely state at time T is predicted, and the most likely position coordinates of the wrist joint at time T are obtained, so that the motion trajectory can be monitored and tracked in real time. The position coordinates of the smartphone at time T can be used as the point cloud space constraint for the next iteration.

Traditional motion monitoring methods mainly use visual data or sensor data worn at multiple locations on the human body. These methods are very strict and demanding on the site, and ordinary people cannot meet the rigid requirements of their motion monitoring when exercising, and their usability is low. However, this application only uses smart wearable devices and smart phones worn on the wrists of the human body. Without adding user usage costs, it can monitor the human motion trajectory in real time, allowing users to monitor anytime and anywhere without being restricted by the site, and its usability is high. In addition, since this application introduces two ultrasonic sound sources, it can achieve clock synchronization of two smart devices without adding any hardware costs and software optimization, ensuring that this application is ready to use, so as to achieve the purpose of high ease of use and strong usability; at the same time, by obtaining inertial sensor data and ultrasonic ranging, it is mixed with an improved hidden Markov model, and finally the accuracy can reach 7-10cm real-time trajectory tracking error.

The present application also provides a real-time motion trajectory monitoring device, including:

A receiving unit, configured to receive a first rotation vector and a first instantaneous acceleration information of a first joint at a time T sent by a wearable device;

A search unit, which is used to search for a corresponding first position set from a preset point cloud database based on the first rotation vector, the preset point cloud database includes a mapping relationship between the rotation vector and the position set, and the position set includes a plurality of preset position coordinates corresponding to the joint;

a generating unit, which is used to obtain a second position set of the first joint at the time T-1, and filter out a first point cloud state space from a preset point cloud database according to the second position set and the first position set, wherein the first point cloud state space includes a plurality of point cloud states, each point cloud state is composed of preset position coordinates in the first position set and preset position coordinates in the second position set;

A distance measuring unit, which is used to calculate the distance between the smart terminal and the wearable device at the Tth moment based on the frequency modulated continuous wave ranging to obtain a first distance;

an acquisition unit, which is used to respectively acquire a second distance between the smart terminal and the wearable device at the T-1th moment, a third distance between the smart terminal and the wearable device at the T-2th moment, a first actual position coordinate of the first joint at the T-1th moment, and a second actual position coordinate of the first joint at the T-2th moment;

A calculation unit, which is used to calculate the position coordinates of the smart terminal at the Tth moment based on the first distance, the second distance, the third distance, the first actual position coordinates and the second actual position coordinates;

The prediction unit is used to input the first instantaneous acceleration information, the first point cloud state space and the position coordinates of the intelligent terminal at the Tth moment into the hidden Markov model to obtain the third actual position coordinates of the first joint at the Tth moment.

It can be seen that the present application can realize real-time monitoring and tracking of joint movements through only a smart terminal and a wearable device. It does not need to deploy high-cost tracking shooting equipment, which effectively reduces the cost of use. The rotation vector information and acceleration information collected by the wearable device can be used to monitor the motion trajectory, which can effectively reduce the influence of external environmental factors on the accuracy of the rotation vector information and acceleration information collection, thereby effectively ensuring the accuracy of motion trajectory monitoring.

Furthermore, the device also includes a creating unit, which is used to:

The arm motion is modeled based on the D-H model to obtain the linkage model;

Inputting multiple joint angles on the arm into the link model to obtain multiple rotation vectors;

A mapping relationship between each rotation vector and its corresponding position set including a plurality of preset position coordinates is created to obtain a point cloud database.

Furthermore, the model parameters in the hidden Markov model include an initial probability distribution, a state transition probability distribution and an output probability distribution, the probability expression of the state transition probability distribution includes instantaneous acceleration information, the position coordinates of the smart terminal and the distance between the smart terminal and the wearable device, and the output probability distribution is 1.

Furthermore, the wearable device includes an inertial sensor and an acceleration sensor, the inertial sensor is used to collect a first rotation vector of the first joint at the Tth moment, and the acceleration sensor is used to collect first instantaneous acceleration information of the first joint at the Tth moment.

It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each unit can refer to the corresponding process in the aforementioned real-time motion trajectory monitoring method embodiment, and will not be repeated here.

The apparatus provided in the above embodiment may be implemented in the form of a computer program, and the computer program may be run on the real-time motion trajectory monitoring device as shown in FIG. 2 .

An embodiment of the present application also provides a real-time motion trajectory monitoring device, including: a memory, a processor and a network interface connected through a system bus, the memory storing at least one instruction, and the at least one instruction being loaded and executed by the processor to implement all or part of the steps of the aforementioned real-time motion trajectory monitoring method.

The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that the structure shown in FIG2 is only a block diagram of a portion of the structure related to the present application solution, and does not constitute a limitation on the computer device to which the present application solution is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

The memory can be used to store computer programs and/or modules. The processor realizes various functions of the computer device by running or executing the computer programs and/or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a video playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as video data, image data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device or other volatile solid-state storage device.

The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, all or part of the steps of the aforementioned real-time motion trajectory monitoring method are implemented.

The embodiment of the present application implements all or part of the aforementioned process, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each of the above methods when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, servers or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

The present application is described with reference to the flowchart and/or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, and the combination of the process and/or box in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims (10)

1. The real-time motion track monitoring method is characterized by comprising the following steps of:

Receiving a first rotation vector and first instantaneous acceleration information of a first joint at a T moment, which are sent by wearable equipment;

Searching a corresponding first position set from a preset point cloud database based on the first rotation vector, wherein the preset point cloud database comprises a mapping relation between the rotation vector and the position set, and the position set comprises a plurality of preset position coordinates corresponding to the joints;

Acquiring a second position set of a first joint at a T-1 moment, and screening a first point cloud state space from a preset point cloud database according to the second position set and the first position set, wherein the first point cloud state space comprises a plurality of point cloud states, and each point cloud state consists of preset position coordinates in the first position set and preset position coordinates in the second position set;

Calculating the distance between the intelligent terminal and the wearable equipment at the T moment based on the frequency modulation continuous wave ranging to obtain a first distance;

Respectively acquiring a second distance between the intelligent terminal and the wearable device at the T-1 moment, a third distance between the intelligent terminal and the wearable device at the T-2 moment, a first actual position coordinate of the first joint at the T-1 moment and a second actual position coordinate of the first joint at the T-2 moment;

Calculating the position coordinate of the intelligent terminal at the T moment based on the first distance, the second distance, the third distance, the first actual position coordinate and the second actual position coordinate;

And inputting the first instantaneous acceleration information, the first cloud state space and the position coordinate of the intelligent terminal at the T moment into a hidden Markov model to obtain a third actual position coordinate of the first joint at the T moment.

2. The method for monitoring a real-time motion trajectory according to claim 1, further comprising, before the step of receiving the first rotation vector and the first instantaneous acceleration information of the first joint at the T-th time transmitted by the wearable device:

Modeling arm movement based on the D-H model to obtain a connecting rod model;

Inputting a plurality of joint angles on an arm to the connecting rod model to obtain a plurality of rotation vectors;

And creating a mapping relation between each rotation vector and a corresponding position set comprising a plurality of preset position coordinates, and obtaining a point cloud database.

3. The method for monitoring a real-time motion trajectory according to claim 1, wherein: the model parameters in the hidden Markov model comprise initial probability distribution, state transition probability distribution and output probability distribution, wherein the probability expression of the state transition probability distribution comprises instantaneous acceleration information, position coordinates of the intelligent terminal and the distance between the intelligent terminal and the wearable device, and the output probability distribution is 1.

4. The method for monitoring a real-time motion trajectory according to claim 1, wherein: the wearable device comprises an inertial sensor and an acceleration sensor, wherein the inertial sensor is used for acquiring a first rotation vector of a first joint at a T moment, and the acceleration sensor is used for acquiring first instantaneous acceleration information of the first joint at the T moment.

5. A real-time motion trajectory monitoring device, comprising:

The receiving unit is used for receiving a first rotation vector and first instantaneous acceleration information of a first joint at a T moment, which are sent by the wearable equipment;

The searching unit is used for searching a corresponding first position set from a preset point cloud database based on the first rotation vector, the preset point cloud database comprises a mapping relation between the rotation vector and the position set, and the position set comprises a plurality of preset position coordinates corresponding to the joint;

The generating unit is used for acquiring a second position set of the first joint at the T-1 moment, and screening a first point cloud state space from a preset point cloud database according to the second position set and the first position set, wherein the first point cloud state space comprises a plurality of point cloud states, and each point cloud state consists of preset position coordinates in the first position set and preset position coordinates in the second position set;

The distance measuring unit is used for calculating the distance between the intelligent terminal and the wearable equipment at the T moment based on the frequency modulation continuous wave distance measurement to obtain a first distance;

The acquisition unit is used for respectively acquiring a second distance between the intelligent terminal and the wearable device at the T-1 moment, a third distance between the intelligent terminal and the wearable device at the T-2 moment, a first actual position coordinate of the first joint at the T-1 moment and a second actual position coordinate of the first joint at the T-2 moment;

The calculating unit is used for calculating the position coordinate of the intelligent terminal at the T moment based on the first distance, the second distance, the third distance, the first actual position coordinate and the second actual position coordinate;

The prediction unit is used for inputting the first instantaneous acceleration information, the first cloud state space and the position coordinate of the intelligent terminal at the T moment into the hidden Markov model to obtain the third actual position coordinate of the first joint at the T moment.

6. The real-time motion trajectory monitoring device of claim 5, further comprising a creation unit for:

Modeling arm movement based on the D-H model to obtain a connecting rod model;

Inputting a plurality of joint angles on an arm to the connecting rod model to obtain a plurality of rotation vectors;

And creating a mapping relation between each rotation vector and a corresponding position set comprising a plurality of preset position coordinates, and obtaining a point cloud database.

7. The real-time motion profile monitoring device of claim 5, wherein: the model parameters in the hidden Markov model comprise initial probability distribution, state transition probability distribution and output probability distribution, wherein the probability expression of the state transition probability distribution comprises instantaneous acceleration information, position coordinates of the intelligent terminal and the distance between the intelligent terminal and the wearable device, and the output probability distribution is 1.

8. The real-time motion profile monitoring device of claim 5, wherein: the wearable device comprises an inertial sensor and an acceleration sensor, wherein the inertial sensor is used for acquiring a first rotation vector of a first joint at a T moment, and the acceleration sensor is used for acquiring first instantaneous acceleration information of the first joint at the T moment.

9. A real-time motion trajectory monitoring device, comprising: a memory and a processor, the memory having stored therein at least one instruction that is loaded and executed by the processor to implement the real-time motion profile monitoring method of any one of claims 1 to 4.

10. A computer-readable storage medium, characterized by: the computer readable storage medium stores a computer program which, when executed by a processor, implements the real-time motion profile monitoring method of any one of claims 1 to 4.