CN112818898B - Model training method and device and electronic equipment - Google Patents

Abstract

The embodiments of the present disclosure disclose model training methods, devices and electronic devices. A specific implementation of the method includes: obtaining a training sample set; selecting a training sample from the training sample set, and performing the following training steps based on the selected training sample: inputting the sample human body image of the selected training sample into the initial neural network, obtaining Three-dimensional human posture information; determine the transformation matrix between the three-dimensional human posture information and the sample inertial motion capture data; use the transformation matrix to convert the posture key points and the inertial motion capture three-dimensional points into the same coordinate system to determine the posture key points and inertial actions Capture the differences between three-dimensional points; based on the determined differences, adjust the network parameters of the initial neural network; if the training end conditions are met, the adjusted initial neural network is determined as the trained three-dimensional human posture prediction network. This implementation can save calibration costs between different coordinate systems.

Description

Model training methods, devices and electronic equipment

Technical field

The embodiments of the present disclosure relate to the field of computer technology, and specifically to model training methods, devices and electronic devices.

Background technique

Human Pose Estimation is an important task in computer vision and an essential step for computers to understand human actions and behaviors. In recent years, methods using deep learning for human posture estimation have been proposed one after another, and have achieved performance that far exceeds traditional methods. In actual solution, the estimation of human posture is often transformed into the prediction problem of key points of the human body, that is, first predicting the position coordinates of each key point of the human body, and then determining the spatial position relationship between the key points based on prior knowledge, thus obtaining Predicted human skeleton.

Contents of the invention

This Disclosure is provided to introduce in simplified form the concepts that are later described in detail in the Detailed Description. This disclosure section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

Embodiments of the present disclosure provide a model training method, device and electronic equipment, which can save calibration costs between different coordinate systems and enable a three-dimensional human posture prediction network to achieve better accuracy.

In a first aspect, embodiments of the present disclosure provide a model training method, which method includes: obtaining a training sample set, wherein the training samples include sample human body images and sample inertial motion capture data corresponding to the sample human body images, and the sample inertial motion capture data The data is the inertial motion capture data of the human body presented in the sample human body image collected when shooting the sample human body image; select the training sample from the training sample set, and perform the following training steps based on the selected training sample: convert the sample of the selected training sample The human body image is input into the initial neural network to obtain the three-dimensional human posture information corresponding to the selected training sample; the transformation matrix between the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data is determined; using the transformation matrix, The posture key points indicated by the three-dimensional human posture information corresponding to the selected training samples and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data are converted to the same coordinate system, and the relationship between the posture key points and the inertial motion capture three-dimensional points is determined. based on the determined differences; adjust the network parameters of the initial neural network based on the determined differences; determine whether the preset training end conditions are met; if the training end conditions are met, the adjusted initial neural network is determined as the three-dimensional human body pose after training Prediction network.

In a second aspect, embodiments of the present disclosure provide a model training device. The device includes: a first acquisition unit configured to acquire a training sample set, wherein the training samples include sample human body images and sample inertial actions corresponding to the sample human body images. Capture data, the sample inertial motion capture data is the inertial motion capture data of the human body presented in the sample human body image collected when shooting the sample human body image; the training unit is used to select training samples from the training sample set, based on the selected training samples, Execute the following training steps: input the sample human body image of the selected training sample into the initial neural network to obtain the three-dimensional human body posture information corresponding to the selected training sample; determine the three-dimensional human body posture information corresponding to the selected training sample and the corresponding sample inertial motion capture Transformation matrix between data; use the transformation matrix to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data to the same coordinate system , determine the difference between the posture key points and the inertial motion capture three-dimensional points; based on the determined differences, adjust the network parameters of the initial neural network; determine whether the preset training end conditions are met; if the training end conditions are met, the adjusted The initial neural network is determined as the trained three-dimensional human posture prediction network.

In a third aspect, embodiments of the present disclosure provide an electronic device, including: one or more processors; a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, , causing one or more processors to implement the model training method of the first aspect.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the steps of the model training method of the first aspect are implemented.

The model training method, device and electronic device provided by the embodiments of the present disclosure obtain a training sample set; then, select training samples from the above-mentioned training sample set, and perform the following training steps based on the selected training samples: The sample human body image is input into the initial neural network to obtain the three-dimensional human posture information corresponding to the selected training sample; determine the transformation matrix between the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data; use the above transformation matrix , convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training samples and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into the same coordinate system, and determine the posture key points and the inertial motion capture three-dimensional points The difference between points; based on the determined difference, adjust the network parameters of the initial neural network; determine whether the preset training end conditions are met; if the above training end conditions are met, determine the adjusted initial neural network as the training is completed 3D human pose prediction network. By determining the transformation matrix between the three-dimensional human posture key points output by the network and the corresponding inertial motion capture three-dimensional points during the network training process, the three-dimensional human posture key points and the inertial motion capture three-dimensional points can be converted to the same coordinate system, respectively. Compared with this method, which requires calibrating the transformation relationship between the inertial capture coordinate system and the camera coordinate system when using inertial motion capture data as a data set for a three-dimensional human posture estimation algorithm, the method provided by this embodiment can save the cost of different coordinate systems. calibration cost, and can make the three-dimensional human posture prediction network achieve better accuracy.

Description of the drawings

The above and other features, advantages, and aspects of various embodiments of the present disclosure will become more apparent with reference to the following detailed description taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It is to be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

Figure 1 is an exemplary system architecture diagram in which various embodiments of the present disclosure may be applied;

Figure 2 is a flow chart of one embodiment of a model training method according to the present disclosure;

Figure 3 is a flow chart of one embodiment of predicting three-dimensional human posture information in the model training method according to the present disclosure;

Figure 4 is a schematic structural diagram of an embodiment of a model training device according to the present disclosure;

FIG. 5 is a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, which rather are provided for A more thorough and complete understanding of this disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of the present disclosure.

It should be understood that various steps described in the method implementations of the present disclosure may be executed in different orders and/or in parallel. Furthermore, method embodiments may include additional steps and/or omit performance of illustrated steps. The scope of the present disclosure is not limited in this regard.

As used herein, the term "include" and its variations are open-ended, ie, "including but not limited to." The term "based on" means "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as “first” and “second” mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units. Or interdependence.

It should be noted that the modifications of "one" and "plurality" mentioned in this disclosure are illustrative and not restrictive. Those skilled in the art will understand that unless the context clearly indicates otherwise, it should be understood as "one or Multiple”.

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are for illustrative purposes only and are not used to limit the scope of these messages or information.

Figure 1 illustrates an exemplary system architecture 100 to which embodiments of the model training methods of the present disclosure may be applied.

As shown in Figure 1, the system architecture 100 may include an inertial motion capture device 101, networks 1021, 1022, 1023, a terminal device 103 and a server 104. The network 1021 is a medium used to provide a communication link between the inertial motion capture device 101 and the terminal device 103 . Network 1022 is a medium used to provide a communication link between inertial motion capture device 101 and server 104 . The network 1023 is a medium used to provide a communication link between the terminal device 103 and the server 104 . Networks 1021, 1022, 1023 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The inertial motion capture device 101 may include, but is not limited to: inertial motion capture sensors installed on various nodes (eg, joints) of the human body and clothing for inertial motion capture. Install inertial motion capture sensors on the joints of the user's body, or the user can collect the posture and orientation of the body parts after wearing inertial motion capture clothing, and use the network 1021 to transmit the inertial motion capture data to the terminal device 103, or The inertial motion capture data is transmitted to the server 104 using the network 1022.

The user can use the terminal device 103 to interact with the server 104 through the network 1023 to send or receive messages, etc. For example, the user can use the terminal device 103 to collect human body images, and the server 104 can obtain the human body image from the terminal device 103 . Various communication client applications can be installed on the terminal device 103, such as inertial motion capture applications, image acquisition applications, instant messaging software, etc.

The terminal device 103 may first obtain a training sample set, wherein the sample inertial motion capture data in the training sample set may be obtained from the inertial motion capture device 101; then, a training sample may be selected from the above training sample set, based on the selected To train a sample, perform the following training steps: input the sample human body image of the selected training sample into the initial neural network to obtain the three-dimensional human posture information corresponding to the selected training sample; determine the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample Transformation matrix between inertial motion capture data; use the above transformation matrix to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data to Under the same coordinate system, determine the difference between the attitude key point and the inertial motion capture three-dimensional point; based on the determined difference, adjust the network parameters of the initial neural network; determine whether the preset training end conditions are met; if the above training end conditions are met , then the adjusted initial neural network is determined as the trained three-dimensional human posture prediction network.

The terminal device 103 may be hardware or software. When the terminal device 103 is hardware, it can be various electronic devices that have cameras and support information interaction, including but not limited to smart phones, tablet computers, laptop computers, etc. When the terminal device 103 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or as a single software or software module. There are no specific limitations here.

Server 104 may be a server that provides various services. For example, a training sample set may be obtained first, where the sample human body images in the training sample set may be obtained from the terminal device 103 , and the sample inertial motion capture data in the training sample set may be obtained from the inertial motion capture device 101 ; After that, a training sample can be selected from the above training sample set, and based on the selected training sample, the following training steps are performed: input the sample human body image of the selected training sample into the initial neural network, and obtain the three-dimensional human body posture corresponding to the selected training sample. information; determine the transformation matrix between the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data; use the above transformation matrix to combine the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and The inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data are converted to the same coordinate system, and the differences between the posture key points and the inertial motion capture three-dimensional points are determined; based on the determined differences, the network parameters of the initial neural network are adjusted. ; Determine whether the preset training end conditions are met; if the above training end conditions are met, the adjusted initial neural network is determined as the trained three-dimensional human posture prediction network.

It should be noted that the server 104 may be hardware or software. When the server 104 is hardware, it can be implemented as a distributed server cluster composed of multiple servers or as a single server. When the server 104 is software, it may be implemented as multiple software or software modules (for example, used to provide distributed services), or it may be implemented as a single software or software module. There are no specific limitations here.

It should also be noted that the model training method provided by the embodiment of the present disclosure can be executed by the server 104. In this case, the model training device is usually provided in the server 104. The model training method provided by the embodiment of the present disclosure can also be executed by the terminal device 103. In this case, the model training device is usually provided in the terminal device 103.

It should also be noted that when the model training method provided by the embodiment of the present disclosure is executed by the server 104, if the server 104 locally stores a training sample set, the exemplary system architecture 100 may not have an inertial motion capture device at this time. 101, networks 1021, 1022, 1023 and terminal device 103.

It should also be noted that when the model training method provided by the embodiment of the present disclosure is executed by the terminal device 103, if the terminal device 103 locally stores inertial motion capture data, initial neural network and other information, then the exemplary system Architecture 100 may be without inertial motion capture device 101, networks 1021, 1022, 1023 and server 104.

It should be understood that the numbers of inertial motion capture devices, networks, terminal devices and servers in Figure 1 are only illustrative. Depending on implementation needs, there can be any number of inertial motion capture devices, networks, end devices, and servers.

Continuing to refer to FIG. 2 , a process 200 of one embodiment of a model training method according to the present disclosure is shown. The model training method includes the following steps:

Step 201: Obtain a training sample set.

In this embodiment, the execution subject of the model training method (such as the terminal device or server shown in Figure 1) can obtain the training sample set. The training samples in the above training sample set may include sample human body images and sample inertial motion capture data corresponding to the sample human body images. The sample inertial motion capture data may be the inertial motion capture data of the human body presented in the sample human body image collected when shooting the sample human body image. Inertial motion capture is a new type of human motion capture technology. It uses wireless motion attitude sensors to collect the posture and orientation of body parts, uses the principles of human kinematics to restore the human body motion model, and uses wireless transmission to present the data in computer software. .

Here, while the user uses the inertial motion capture device to capture the inertial motion of the human body, the user can also use the camera device to photograph the user, thereby obtaining a human body image corresponding to the inertial motion capture data.

Step 202: Select a training sample from the training sample set, and perform the following training steps based on the selected training sample: input the sample human body image of the selected training sample into the initial neural network to obtain the three-dimensional human posture information corresponding to the selected training sample; Determine the transformation matrix between the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data; use the transformation matrix to combine the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample The inertial motion capture three-dimensional points indicated by the inertial motion capture data are converted to the same coordinate system, and the differences between the posture key points and the inertial motion capture three-dimensional points are determined; based on the determined differences, the network parameters of the initial neural network are adjusted; determine whether The preset training end conditions are met; if the training end conditions are met, the adjusted initial neural network is determined as the trained three-dimensional human posture prediction network.

In this embodiment, the above-mentioned execution subject may select training samples from the training sample set obtained in step 201, and perform the following training steps based on the selected training samples.

In this embodiment, the training step 202 may include sub-steps 2021, 2022, 2023, 2024, 2025 and 2026. in:

Step 2021: Input the sample human body image of the selected training sample into the initial neural network to obtain the three-dimensional human body posture information corresponding to the selected training sample.

In this embodiment, the above-mentioned execution subject can input the sample human body image of the selected training sample into the initial neural network to obtain the three-dimensional human body posture information corresponding to the selected training sample. The above-mentioned initial neural network can be various neural networks that can obtain three-dimensional human body posture information based on human body images, such as convolutional neural networks, deep neural networks, etc. Three-dimensional human body posture information can include the posture and orientation of various body parts, for example, the direction and position of human joints.

Step 2022: Determine the transformation matrix between the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data.

In this embodiment, the above-mentioned execution subject can determine the transformation matrix between the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data. Transformation matrix is a concept in mathematical linear algebra. In linear algebra, linear transformations can be represented by matrices. If T is a linear transformation that maps Rn to Rm, and x is a column vector with n elements, then we call the m×n matrix A the transformation matrix of T.

Here, the above-mentioned execution subject can use the least squares method to determine the transformation matrix between the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data. The least squares method, also known as the least squares method, is a mathematical optimization technique. It finds the best functional match of the data by minimizing the sum of squared errors. The least squares method can be used to easily obtain unknown data, and minimize the sum of square errors between the obtained data and the actual data.

Step 2023: Use the transformation matrix to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into the same coordinate system to determine the posture key points. Difference between point and inertial motion capture 3D points.

In this embodiment, the above-mentioned execution subject can use the above-mentioned transformation matrix to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into under the same coordinate system. Here, the above-mentioned execution subject can establish a coordinate system, can transform the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample into the established coordinate system, and can convert the sample inertial action corresponding to the selected training sample. The inertial motion capture three-dimensional points indicated by the capture data are transformed into the established coordinate system.

Afterwards, the above execution subject can determine the difference between the posture key points and the inertial motion capture 3D points. Specifically, the above execution subject can use a preset loss function to determine the difference between the posture key point and the inertial motion capture three-dimensional point. For example, the mean square error can be used as the loss function to determine the difference between the posture key point and the inertial motion capture three-dimensional point. The difference can also be determined by using the L2 norm as the loss function to determine the difference between the posture key points and the inertial motion capture three-dimensional points.

Step 2024: Adjust the network parameters of the initial neural network based on the determined difference.

In this embodiment, the above execution subject may adjust the network parameters of the initial neural network based on the difference determined in step 2023. Here, the above execution subject can use various implementation methods to adjust the network parameters of the initial neural network based on the differences between posture key points and inertial motion capture three-dimensional points. For example, the BP (BackPropagation) algorithm or the SGD (Stochastic Gradient Descent) algorithm can be used to adjust the network parameters of the initial neural network.

Step 2025: Determine whether preset training end conditions are met.

In this embodiment, the above execution subject can determine whether the preset training end conditions are met. The preset training end conditions here may include but are not limited to at least one of the following: the training time exceeds the preset duration; the number of training times exceeds the preset number; and the determined difference is less than the preset difference threshold.

If the above training end conditions are met, the above execution subject can execute step 2026.

Step 2026: If the training end conditions are met, the adjusted initial neural network is determined as the trained three-dimensional human posture prediction network.

In this embodiment, if it is determined in step 2025 that the above training end condition is met, the above execution subject may determine the adjusted initial neural network as the trained three-dimensional human posture prediction network.

The method provided by the above embodiments of the present disclosure can determine the transformation matrix between the three-dimensional human posture key points output by the network and the corresponding inertial motion capture three-dimensional points during the network training process, so that the three-dimensional human posture key points and the inertial motion capture three-dimensional points can be combined Points are converted to the same coordinate system. Compared with the method that requires calibrating the transformation relationship between the inertial capture coordinate system and the camera coordinate system when using inertial motion capture data as a data set for a three-dimensional human posture estimation algorithm, this embodiment provides This method can save the calibration cost between different coordinate systems, and can make the three-dimensional human posture prediction network achieve better accuracy.

In some optional implementations, if it is determined in step 2025 that the above training end conditions are not met, the above execution subject can use the adjusted initial neural network as the initial neural network and select unused data from the above training sample set. of training samples, and based on the reselected training samples, continue to perform the above training steps (sub-steps 2021-2026).

In some optional implementations, the training samples in the above training sample set may include sample human body videos and sample inertial motion capture data corresponding to the sample human body videos. The sample inertial motion capture data may be the inertial motion capture data of the human body presented in the sample human body video collected when shooting the sample human body video. Here, while the user uses the inertial motion capture device to capture the inertial motion of the human body, the user can also use the camera device to photograph the user, thereby obtaining a human body video corresponding to the inertial motion capture data. The above-mentioned execution subject can input the sample human body image of the selected training sample into the initial neural network in the following manner to obtain the three-dimensional human posture information corresponding to the selected training sample: the above-mentioned execution subject can input the sample human body video of the selected training sample into the initial neural network. In the network, the three-dimensional human posture information corresponding to the selected training samples is obtained. The above-mentioned initial neural network can be various neural networks that can obtain three-dimensional human posture information based on human body videos, such as convolutional neural networks, deep neural networks, etc. Three-dimensional human body posture information can include postures and orientations of various body parts, such as the directions and positions of human body joints. Instead of using the three-dimensional human posture information corresponding to the image and the corresponding inertial motion capture data to obtain the transformation matrix, here we use the three-dimensional human posture information corresponding to the entire video and the corresponding inertial motion capture data to obtain the transformation matrix, using video-level transformation Matrix, using a large amount of data to reduce errors, the random errors of the samples will offset part of each other, the average error of the network output will be smaller, and the error of the affine transformation obtained will be smaller accordingly.

In some optional implementations, the above execution subject can use the above transformation matrix in the following manner to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertia indicated by the corresponding sample inertial motion capture data. Convert motion capture three-dimensional points to the same coordinate system: the above execution subject can use the above transformation matrix to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample to the sample inertial motion capture data corresponding to the selected training sample The indicated inertial motion captures the coordinate system in which the 3D point is located. The coordinate system where the inertial motion capture three-dimensional point is located can be the human body coordinate system when collecting sample inertial motion capture data. For example, it can be the lower left corner of the human body area as the coordinate origin, and the two sides parallel to the ground and perpendicular to the ground as the coordinates. The coordinate system of the axis.

In some optional implementations, the above execution subject can use the above transformation matrix in the following manner to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertia indicated by the corresponding sample inertial motion capture data. Convert the motion capture three-dimensional points to the same coordinate system: Use the above transformation matrix to convert the inertial motion capture three-dimensional points indicated by the sample inertial motion capture data corresponding to the selected training samples to the three-dimensional human posture information corresponding to the selected training samples. In the coordinate system where the posture key points are located. The coordinate system where the posture key points are located may be the camera coordinate system corresponding to the sample human image of the selected training sample.

Referring further to FIG. 3 , FIG. 3 is a process 300 for predicting three-dimensional human posture information in the model training method according to this embodiment. The process 300 for predicting three-dimensional human posture information includes the following steps:

Step 301: Obtain the human body image to be predicted.

In this embodiment, the execution subject of the model training method (such as the terminal device or server shown in Figure 1) can directly or indirectly obtain the human body image to be predicted. For example, when the execution subject is a terminal device, the execution subject can directly obtain the human body image to be predicted input by the user; when the execution subject is a server, the execution subject can obtain the human body image from the terminal device through a wired connection or a wireless connection. Obtain the human body image to be predicted input by the user. Here, the above-mentioned human body image may include various parts of the human body, such as head, waist, arms, legs, etc.

Step 302: Input the human body image into the trained three-dimensional human posture prediction network to obtain the three-dimensional human posture information of the human body presented in the human body image.

In this embodiment, the above-mentioned execution subject can input the above-mentioned human body image into the trained three-dimensional human body posture prediction network to obtain the three-dimensional human body posture information of the human body presented in the above-mentioned human body image. The above-mentioned three-dimensional human posture prediction network is a three-dimensional human posture prediction network trained using the method described in Figure 2. The above-mentioned three-dimensional human posture prediction network can be used to characterize the correspondence between the image and the three-dimensional human posture information of the human body presented in the image. The above-mentioned three-dimensional human body posture information may include postures and orientations of various body parts, for example, the directions and positions of human body joints.

The method provided by the above embodiments of the present disclosure inputs the human body image into the three-dimensional human body posture prediction network trained by the method described in Figure 2, thereby predicting the three-dimensional human body posture information of the human body presented in the human body image. In this way, it can improve Accuracy of predicted three-dimensional human posture information.

With further reference to Figure 4, as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a model training device. The device embodiment corresponds to the method embodiment shown in Figure 2. The device can specifically Used in various electronic equipment.

As shown in Figure 4, the model training device 400 of this embodiment includes: a first acquisition unit 401 and a training unit 402. The first acquisition unit 401 is used to acquire a training sample set, where the training samples include sample human body images and sample inertial motion capture data corresponding to the sample human body images, and the sample inertial motion capture data are samples collected when shooting the sample human body images. Inertial motion capture data of the human body presented in the human body image; the training unit 402 is used to select training samples from the training sample set, and perform the following training steps based on the selected training samples: input the sample human body image of the selected training sample into the initial neural network , obtain the three-dimensional human posture information corresponding to the selected training sample; determine the transformation matrix between the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data; use the transformation matrix to convert the selected training sample corresponding to The posture key points indicated by the three-dimensional human posture information and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data are converted to the same coordinate system, and the difference between the posture key points and the inertial motion capture three-dimensional points is determined; based on the determination If the difference is found, adjust the network parameters of the initial neural network; determine whether the preset training end conditions are met; if the training end conditions are met, determine the adjusted initial neural network as the trained three-dimensional human posture prediction network.

In this embodiment, the specific processing of the first acquisition unit 401 and the training unit 402 of the model training device 400 can refer to step 201 and step 202 in the corresponding embodiment of FIG. 2 .

In some optional implementations, the above-mentioned model training device 400 may also include a feedback unit (not shown in the figure). The above feedback unit can be used to use the adjusted initial neural network as the initial neural network if the above training end conditions are not met, select unused training samples from the above training sample set, and continue to execute the above based on the selected training samples. training steps.

In some optional implementations, the above-mentioned model training device 400 may also include a second acquisition unit (not shown in the figure) and an input unit (not shown in the figure). The above-mentioned second acquisition unit can be used to acquire the human body image to be predicted; the above-mentioned input unit can be used to input the above-mentioned human body image into the above-mentioned trained three-dimensional human posture prediction network to obtain the three-dimensional human body posture information of the human body presented in the above-mentioned human body image. .

In some optional implementations, the training samples may include sample human body videos and sample inertial motion capture data corresponding to the sample human body videos. The sample inertial motion capture data may be presented in the sample human body videos collected when shooting the sample human body videos. Inertial motion capture data of the human body; and the above-mentioned training unit 402 can be further used to input the sample human body image of the selected training sample into the initial neural network in the following manner to obtain the three-dimensional human posture information corresponding to the selected training sample: the above-mentioned training unit 402 The sample human video of the selected training sample can be input into the initial neural network to obtain the three-dimensional human posture information corresponding to the selected training sample.

In some optional implementations, the above-mentioned training unit 402 may be further configured to use the above-mentioned transformation matrix in the following manner to capture the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data. The indicated inertial motion capture three-dimensional points are converted into the same coordinate system: the above-mentioned training unit 402 can use the above-mentioned transformation matrix to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample to the corresponding key points of the selected training sample. The coordinate system in which the inertial motion capture three-dimensional point indicated by the sample inertial motion capture data is located.

In some optional implementations, the above-mentioned training unit 402 may be further configured to use the above-mentioned transformation matrix in the following manner to capture the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data. The indicated inertial motion capture three-dimensional points are converted to the same coordinate system: the above-mentioned training unit 402 can use the above-mentioned transformation matrix to convert the inertial motion capture three-dimensional points indicated by the sample inertial motion capture data corresponding to the selected training sample to the selected training In the coordinate system where the posture key points indicated by the three-dimensional human posture information corresponding to the sample are located.

Referring now to FIG. 5 , a schematic structural diagram of an electronic device (such as the server or terminal device in FIG. 1 ) 500 suitable for implementing embodiments of the present disclosure is shown. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals ( Mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG. 5 is only an example and should not bring any limitations to the functions and usage scope of the embodiments of the present disclosure.

As shown in FIG. 5 , the electronic device 500 may include a processing device (eg, central processing unit, graphics processor, etc.) 501 that may be loaded into a random access device according to a program stored in a read-only memory (ROM) 502 or from a storage device 508 . The program in the memory (RAM) 503 executes various appropriate actions and processes. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502 and the RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

Generally, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 507 such as a computer; a storage device 508 including a magnetic tape, a hard disk, etc.; and a communication device 509. Communication device 509 may allow electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although FIG. 5 illustrates electronic device 500 with various means, it should be understood that implementation or availability of all illustrated means is not required. More or fewer means may alternatively be implemented or provided. Each block shown in Figure 5 may represent one device, or may represent multiple devices as needed.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such embodiments, the computer program may be downloaded and installed from the network via communication device 509, or from storage device 508, or from ROM 502. When the computer program is executed by the processing device 501, the above-described functions defined in the method of the embodiment of the present disclosure are performed. It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), removable Programmed read-only memory (EPROM or flash memory), fiber optics, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. In embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program code is carried. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device . Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the above.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a training sample set, where the training samples include sample human body images and sample human body images. Corresponding sample inertial motion capture data, the sample inertial motion capture data is the inertial motion capture data of the human body presented in the sample human body image collected when shooting the sample human body image; select the training sample from the training sample set, based on the selected training sample, Execute the following training steps: input the sample human body image of the selected training sample into the initial neural network to obtain the three-dimensional human body posture information corresponding to the selected training sample; determine the three-dimensional human body posture information corresponding to the selected training sample and the corresponding sample inertial motion capture Transformation matrix between data; use the transformation matrix to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data to the same coordinate system , determine the difference between the posture key points and the inertial motion capture three-dimensional points; based on the determined differences, adjust the network parameters of the initial neural network; determine whether the preset training end conditions are met; if the training end conditions are met, the adjusted The initial neural network is determined as the trained three-dimensional human posture prediction network.

Computer program code for performing operations of embodiments of the present disclosure may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, or a combination thereof, Also included are conventional procedural programming languages—such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In situations involving remote computers, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider). connected via the Internet).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more logic functions that implement the specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of specialized hardware and computer instructions.

According to one or more embodiments of the present disclosure, a model training method is provided, which method includes: obtaining a training sample set, wherein the training samples include sample human body images and sample inertial motion capture data corresponding to the sample human body images, and the sample The inertial motion capture data is the inertial motion capture data of the human body presented in the sample human body image collected when shooting the sample human body image; select the training sample from the training sample set, and perform the following training steps based on the selected training sample: The sample human body image of the sample is input into the initial neural network to obtain the three-dimensional human posture information corresponding to the selected training sample; determine the transformation matrix between the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data; use the transformation Matrix, convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into the same coordinate system to determine the posture key points and inertial motion capture The difference between three-dimensional points; based on the determined difference, adjust the network parameters of the initial neural network; determine whether the preset training end conditions are met; if the training end conditions are met, the adjusted initial neural network is determined to be trained. 3D human pose prediction network.

According to one or more embodiments of the present disclosure, the method further includes: if the training end condition is not met, using the adjusted initial neural network as the initial neural network, selecting unused training samples from the training sample set, and based on Select the training sample and continue the training steps.

According to one or more embodiments of the present disclosure, the method further includes: obtaining a human body image to be predicted; inputting the human body image into the trained three-dimensional human body posture prediction network to obtain the three-dimensional human body posture information of the human body presented in the human body image.

According to one or more embodiments of the present disclosure, the training sample includes a sample human body video and sample inertial motion capture data corresponding to the sample human body video. The sample inertial motion capture data is presented in the sample human body video collected when shooting the sample human body video. Inertial motion capture data of the human body; and inputting the sample human body image of the selected training sample into the initial neural network to obtain the three-dimensional human posture information corresponding to the selected training sample, including: inputting the sample human body video of the selected training sample into the initial neural network , the three-dimensional human posture information corresponding to the selected training sample is obtained.

According to one or more embodiments of the present disclosure, a transformation matrix is used to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into Under the same coordinate system, it includes: using the transformation matrix to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample to the inertial motion capture three-dimensional points indicated by the sample inertial motion capture data corresponding to the selected training sample. in the coordinate system.

According to one or more embodiments of the present disclosure, a transformation matrix is used to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into Under the same coordinate system, including: using the transformation matrix to convert the inertial motion capture three-dimensional points indicated by the sample inertial motion capture data corresponding to the selected training samples to the posture key points indicated by the three-dimensional human posture information corresponding to the selected training samples. in the coordinate system.

According to one or more embodiments of the present disclosure, a model training device is provided. The device includes: a first acquisition unit configured to acquire a training sample set, wherein the training samples include a sample human body image and a sample human body image corresponding to the sample human body image. Sample inertial motion capture data, the sample inertial motion capture data is the inertial motion capture data of the human body presented in the sample human body image collected when shooting the sample human body image; the training unit is used to select training samples from the training sample set, based on the selected To train a sample, perform the following training steps: input the sample human body image of the selected training sample into the initial neural network to obtain the three-dimensional human posture information corresponding to the selected training sample; determine the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample Transformation matrix between inertial motion capture data; use the transformation matrix to convert the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into the same Under the coordinate system, determine the difference between the attitude key point and the inertial motion capture three-dimensional point; based on the determined difference, adjust the network parameters of the initial neural network; determine whether the preset training end conditions are met; if the training end conditions are met, then The adjusted initial neural network is determined as the trained three-dimensional human posture prediction network.

According to one or more embodiments of the present disclosure, the device further includes: a feedback unit, configured to use the adjusted initial neural network as the initial neural network if the training end condition is not met, and select unused data from the training sample set. training samples, and continue to perform the training steps based on the selected training samples.

According to one or more embodiments of the present disclosure, the device further includes: a second acquisition unit for acquiring a human body image to be predicted; an input unit for inputting the human body image into the trained three-dimensional human posture prediction network to obtain Three-dimensional human posture information of the human body presented in the human body image.

According to one or more embodiments of the present disclosure, the training sample includes a sample human body video and sample inertial motion capture data corresponding to the sample human body video. The sample inertial motion capture data is presented in the sample human body video collected when shooting the sample human body video. Inertial motion capture data of the human body; and the training unit is further used to input the sample human body image of the selected training sample into the initial neural network in the following manner to obtain the three-dimensional human posture information corresponding to the selected training sample: The human body video is input into the initial neural network, and the three-dimensional human posture information corresponding to the selected training sample is obtained.

According to one or more embodiments of the present disclosure, the training unit is further configured to use the transformation matrix in the following manner to combine the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data. The inertial motion capture three-dimensional points are converted into the same coordinate system: using the transformation matrix, the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample are converted to the posture key points indicated by the sample inertial motion capture data corresponding to the selected training sample. Inertial motion captures the coordinate system in which the three-dimensional point is located.

According to one or more embodiments of the present disclosure, the training unit is further configured to use the transformation matrix in the following manner to combine the posture key points indicated by the three-dimensional human posture information corresponding to the selected training sample and the corresponding sample inertial motion capture data. Convert the inertial motion capture three-dimensional points to the same coordinate system: use the transformation matrix to convert the inertial motion capture three-dimensional points indicated by the sample inertial motion capture data corresponding to the selected training samples to the three-dimensional human posture information corresponding to the selected training samples. The coordinate system in which the indicated pose keypoint is located.

According to one or more embodiments of the present disclosure, an electronic device is provided, including: one or more processors; a storage device for storing one or more programs. When the one or more programs are processed by one or more The processor executes, causing one or more processors to implement the above model training method.

According to one or more embodiments of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the above model training method are implemented.

The units involved in the embodiments of the present disclosure may be implemented in software or hardware. The described unit may also be provided in a processor. For example, it may be described as follows: a processor includes a first acquisition unit and a training unit. The names of these units do not constitute a limitation on the unit itself under certain circumstances. For example, the first acquisition unit may also be described as "the unit that acquires the training sample set."

The above description is only a description of the preferred embodiments of the present disclosure and the technical principles applied. Persons skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to technical solutions composed of specific combinations of the above technical features, and should also cover the above-mentioned technical solutions without departing from the above-mentioned inventive concept. Other technical solutions formed by any combination of technical features or their equivalent features. For example, a technical solution is formed by replacing the above features with technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to).

Claims (14)

1. A method of model training, comprising:

acquiring a training sample set, wherein the training sample comprises a sample human body image and sample inertial motion capturing data corresponding to the sample human body image, and the sample inertial motion capturing data is inertial motion capturing data of a human body in the sample human body image acquired when the sample human body image is shot;

selecting a training sample from the training sample set, and executing the following training steps based on the selected training sample: inputting a sample human body image of the selected training sample into an initial neural network to obtain three-dimensional human body posture information corresponding to the selected training sample; determining a transformation matrix between three-dimensional human body posture information corresponding to the selected training sample and corresponding sample inertial motion capture data; converting the gesture key points indicated by the three-dimensional human gesture information corresponding to the selected training samples and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into the same coordinate system by utilizing the transformation matrix, and determining the difference between the gesture key points and the inertial motion capture three-dimensional points; based on the determined difference, adjusting network parameters of the initial neural network; determining whether a preset training ending condition is met; and if the training ending condition is met, determining the adjusted initial neural network as a three-dimensional human body posture prediction network after training is completed.

2. The method according to claim 1, wherein the method further comprises:

and if the training ending condition is not met, taking the adjusted initial neural network as the initial neural network, selecting unused training samples from the training sample set, and continuously executing the training step based on the selected training samples.

3. The method according to claim 1, wherein the method further comprises:

acquiring a human body image to be predicted;

inputting the human body image into the three-dimensional human body posture prediction network after training, and obtaining three-dimensional human body posture information of the human body displayed in the human body image.

4. The method of claim 1, wherein the training sample comprises a sample human body video and sample inertial motion capture data corresponding to the sample human body video, the sample inertial motion capture data being inertial motion capture data of a human body presented in the sample human body video acquired when the sample human body video was captured; and

the step of inputting the sample human body image of the selected training sample into the initial neural network to obtain three-dimensional human body posture information corresponding to the selected training sample comprises the following steps:

And inputting the sample human body video of the selected training sample into the initial neural network to obtain three-dimensional human body posture information corresponding to the selected training sample.

5. The method according to one of claims 1 to 4, wherein converting, using the transformation matrix, the gesture key points indicated by the three-dimensional human gesture information corresponding to the selected training samples and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into the same coordinate system includes:

and converting the gesture key points indicated by the three-dimensional human gesture information corresponding to the selected training samples into a coordinate system where the inertial motion capturing three-dimensional points indicated by the sample inertial motion capturing data corresponding to the selected training samples are located by utilizing the transformation matrix.

6. The method according to one of claims 1 to 4, wherein converting, using the transformation matrix, the gesture key points indicated by the three-dimensional human gesture information corresponding to the selected training samples and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into the same coordinate system includes:

and converting the inertial motion capturing three-dimensional points indicated by the sample inertial motion capturing data corresponding to the selected training sample into a coordinate system where the gesture key points indicated by the three-dimensional human gesture information corresponding to the selected training sample are located by utilizing the transformation matrix.

7. A model training device, comprising:

the first acquisition unit is used for acquiring a training sample set, wherein the training sample comprises a sample human body image and sample inertial motion capturing data corresponding to the sample human body image, and the sample inertial motion capturing data is inertial motion capturing data of a human body in the sample human body image acquired when the sample human body image is shot;

the training unit is used for selecting training samples from the training sample set, and based on the selected training samples, the following training steps are executed: inputting a sample human body image of the selected training sample into an initial neural network to obtain three-dimensional human body posture information corresponding to the selected training sample; determining a transformation matrix between three-dimensional human body posture information corresponding to the selected training sample and corresponding sample inertial motion capture data; converting the gesture key points indicated by the three-dimensional human gesture information corresponding to the selected training samples and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into the same coordinate system by utilizing the transformation matrix, and determining the difference between the gesture key points and the inertial motion capture three-dimensional points; based on the determined difference, adjusting network parameters of the initial neural network; determining whether a preset training ending condition is met; and if the training ending condition is met, determining the adjusted initial neural network as a three-dimensional human body posture prediction network after training is completed.

8. The apparatus of claim 7, wherein the apparatus further comprises:

and the feedback unit is used for taking the adjusted initial neural network as the initial neural network if the training ending condition is not met, selecting unused training samples from the training sample set, and continuously executing the training step based on the selected training samples.

9. The apparatus of claim 7, wherein the apparatus further comprises:

a second acquisition unit for acquiring a human body image to be predicted;

and the input unit is used for inputting the human body image into the three-dimensional human body posture prediction network after training to obtain three-dimensional human body posture information of the human body displayed in the human body image.

10. The apparatus of claim 7, wherein the training sample comprises a sample human body video and sample inertial motion capture data corresponding to the sample human body video, the sample inertial motion capture data being inertial motion capture data of a human body presented in the sample human body video acquired when the sample human body video was captured; and

the training unit is further used for inputting the sample human body image of the selected training sample into the initial neural network in the following manner to obtain three-dimensional human body posture information corresponding to the selected training sample:

And inputting the sample human body video of the selected training sample into the initial neural network to obtain three-dimensional human body posture information corresponding to the selected training sample.

11. The apparatus according to one of claims 7 to 10, wherein the training unit is further configured to convert the gesture key points indicated by the three-dimensional human gesture information corresponding to the selected training samples and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into the same coordinate system by using the transformation matrix:

and converting the gesture key points indicated by the three-dimensional human gesture information corresponding to the selected training samples into a coordinate system where the inertial motion capturing three-dimensional points indicated by the sample inertial motion capturing data corresponding to the selected training samples are located by utilizing the transformation matrix.

12. The apparatus according to one of claims 7 to 10, wherein the training unit is further configured to convert the gesture key points indicated by the three-dimensional human gesture information corresponding to the selected training samples and the inertial motion capture three-dimensional points indicated by the corresponding sample inertial motion capture data into the same coordinate system by using the transformation matrix:

And converting the inertial motion capturing three-dimensional points indicated by the sample inertial motion capturing data corresponding to the selected training sample into a coordinate system where the gesture key points indicated by the three-dimensional human gesture information corresponding to the selected training sample are located by utilizing the transformation matrix.

13. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon,

when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-6.

14. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-6.