CN111401318A - Action recognition method and device - Google Patents

Abstract

This specification provides an action recognition method and device, wherein the action recognition method includes: acquiring an image frame collected by an image acquisition device; segmenting the action area of a target object in the image frame to obtain an intermediate image; The image input action recognition model performs key point identification and coordinate mapping, and obtains the action group node corresponding to the identified action key point, and the three-dimensional coordinate information of the action key point mapping; according to the three-dimensional coordinate information, the action group The frame node and the virtual action to which it belongs are assembled to generate a 3D virtual object; the action recognition is performed on the 3D virtual object based on the action recognition data set, and the action type of the target object is determined.

Description

Action recognition method and device

technical field

The present specification relates to the technical field of gesture recognition, and in particular, to an action recognition method. This specification also relates to a motion recognition device, a computing device, and a computer-readable storage medium.

Background technique

With the development of Internet technology, AR interaction based on gesture recognition has been widely used in mobile terminals. Users can operate related applications on the mobile terminal through gestures without touching the mobile terminal. However, gesture recognition usually uses a large number of pre-collected gesture images as training samples, trains a gesture detector offline, and then deploys the detector online to realize gesture recognition, or recognizes through two-dimensional hand key points The two-dimensional gesture image is constructed by the method, and then the gesture recognition is realized based on the two-dimensional gesture image; whether the gesture recognition is realized by the detector or the gesture recognition by constructing the two-dimensional image, there are high cost, low versatility and low flexibility. Moreover, due to the relatively high degree of freedom of the human hand, there may be differences in the gesture types of the same gesture under different viewing angles, which further affects the accuracy of gesture recognition.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present specification provide an action recognition method. This specification also relates to a motion recognition device, a computing device, and a computer-readable storage medium, so as to solve the technical defects existing in the prior art.

According to a first aspect of the embodiments of the present specification, an action recognition method is provided, including:

Obtain the image frame collected by the image acquisition device;

Segmenting the action area of the target object in the image frame to obtain an intermediate image;

Inputting the intermediate image into the action recognition model to perform key point recognition and coordinate mapping, to obtain the action frame node corresponding to the identified action key point, and the three-dimensional coordinate information of the action key point mapping;

Generate a three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs;

Action recognition is performed on the three-dimensional virtual object based on the action recognition data set, and the action type of the target object is determined.

Optionally, the action recognition model performs key point recognition, including:

Identifying the action key point corresponding to the target object in the intermediate image, and determining the key point label of the action key point;

Determine the target node label corresponding to the key point label based on the correspondence between the key point label and the node label of the action frame node;

The action frame node corresponding to the action key point to which the key point label belongs is determined according to the target node label.

Optionally, the action recognition model performs coordinate mapping, including:

determining the position information of the action key point in the intermediate image;

The three-dimensional coordinate information corresponding to the action key point is mapped based on the position information.

Optionally, generating a three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs includes:

Determine the corresponding relationship between the action frame node and the three-dimensional coordinate information according to the action key point, and determine the connection relationship of the action frame node based on the virtual action frame;

The three-dimensional coordinate information is connected according to the connection relationship to generate the three-dimensional virtual object.

Optionally, after the step of generating a 3D virtual object according to the 3D coordinate information, the action frame node and the virtual action frame to which it belongs is performed, and the 3D virtual object is performed based on the motion recognition data set. Action recognition, before the step of determining the action type of the target object is performed, also includes:

detecting the number of objects of the generated three-dimensional virtual object;

In the case that the number of objects is greater than the preset number threshold, standardizing processing is performed on a plurality of three-dimensional virtual objects corresponding to the number of objects to obtain the virtual objects to be selected;

A target virtual object is selected from the virtual objects to be selected as the three-dimensional virtual object.

Optionally, before the step of acquiring the image frames acquired by the image acquisition device is performed, the method further includes:

Receive the click instruction submitted by the user through the action interaction page;

At least one action image frame is displayed to the user according to the click instruction; the action image frame includes a display area corresponding to the display action.

Optionally, performing motion recognition on the three-dimensional virtual object based on the motion recognition data set, and determining the motion type of the target object, including:

Receive the motion recognition data set issued by the server for the motion image frame; the motion recognition data carries the motion recognition rule matching the display action;

According to the action recognition rule, determine whether the action of the three-dimensional virtual object matches the display action;

If so, determine a presentation action type of the presentation action according to the action recognition data set, and use the presentation action type as the action type.

Optionally, after determining the display action type of the display action according to the action recognition data set, and performing the display action type as the action type sub-step, the method further includes:

Recommended information matching the action type is displayed to the user through the action interaction page.

Optionally, after the step of performing motion recognition on the three-dimensional virtual object based on the motion recognition data set and determining the motion type of the target object is performed, the method further includes:

matching the action type with the presentation action type of the presentation action;

If the matching is successful, displaying the successful matching information to the user through the action interaction page;

If the matching fails, reminding information is displayed to the user through the action interaction page, and the reminding information carries an action strategy.

Optionally, in the case where the target object is a hand, performing segmentation processing on the action area of the target object in the image frame to obtain an intermediate image, including:

detecting the characteristic area corresponding to the hand in the image frame;

Crop the image frame according to the feature area to obtain the intermediate image containing hand features;

Correspondingly, the action type is a gesture action type.

Optionally, performing motion recognition on the three-dimensional virtual object based on the motion recognition data set, and determining the motion type of the target object, including:

Analyzing the action recognition data set to obtain action elements;

determining an intermediate action type of the 3D virtual object based on the recognition result of the action recognition performed on the 3D virtual object by the action element;

The intermediate action type is determined as the action type of the target object.

Optionally, determining the intermediate action type of the 3D virtual object based on the recognition result of the action recognition performed on the 3D virtual object based on the action element includes:

calculating an action angle between each virtual sub-object in the three-dimensional virtual object;

The action angle is detected based on the action element, and the intermediate action type of the three-dimensional virtual object is determined according to the detection result.

Optionally, determining the intermediate action type of the 3D virtual object based on the recognition result of the action recognition performed on the 3D virtual object based on the action element includes:

Detecting the action position of each virtual sub-object in the three-dimensional virtual object according to the action element;

The intermediate action type of the three-dimensional virtual object is determined based on the detection result.

Optionally, the motion recognition data set is composed of an motion recognition package delivered by a server, wherein the motion recognition package contains recognition rules for recognizing the three-dimensional virtual object.

According to a second aspect of the embodiments of the present specification, a motion recognition device is provided, including:

an acquisition module, configured to acquire image frames collected by the image acquisition device;

a processing module, configured to perform segmentation processing on the action area of the target object in the image frame to obtain an intermediate image;

The recognition module is configured to input the intermediate image into the action recognition model to perform key point recognition and coordinate mapping, and obtain the action frame node corresponding to the identified action key point, and the three-dimensional coordinate information of the action key point mapping;

a generating module, configured to generate a three-dimensional virtual object according to the three-dimensional coordinate information, the action framing node and the virtual motion framing node to which it belongs;

The determining module is configured to perform motion recognition on the three-dimensional virtual object based on the motion recognition data set, and determine the motion type of the target object.

According to a third aspect of the embodiments of the present specification, a computing device is provided, including:

memory and processor;

The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions:

Obtain the image frame collected by the image acquisition device;

Segmenting the action area of the target object in the image frame to obtain an intermediate image;

Inputting the intermediate image into the action recognition model to perform key point recognition and coordinate mapping, to obtain the action frame node corresponding to the identified action key point, and the three-dimensional coordinate information of the action key point mapping;

Generate a three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs;

Action recognition is performed on the three-dimensional virtual object based on the action recognition data set, and the action type of the target object is determined.

According to a fourth aspect of the embodiments of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, implements the steps of the action recognition method.

In the action recognition method provided by an embodiment of this specification, in the process of recognizing the action placed by the user, image segmentation processing is performed on the acquired image frame to obtain an intermediate image including the target object, and then the intermediate image is input To the action recognition model, obtain the action frame node and three-dimensional coordinate information, and generate the three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs, and realize the The three-dimensional virtual object recognizes the action type, which effectively solves the problem that the image frame acquisition is not standard enough due to different image acquisition angles and affects the accuracy of the action type recognition. Determining the action type of the target object makes it more versatile, flexible and more scalable in the action recognition scene, and can realize that in the case of adding a new action type, there is no need to retrain the model, and by adding action recognition The data set method can realize the identification of new action types, making the application scenarios more extensive.

Description of drawings

FIG. 1 is a flowchart of an action recognition method provided by an embodiment of the present specification;

2 is a schematic diagram of an intermediate image in an action recognition method provided by an embodiment of the present specification;

3 is a schematic diagram of an action framing in an action recognition method provided by an embodiment of the present specification;

4 is a schematic diagram of a three-dimensional virtual object in an action recognition method provided by an embodiment of the present specification;

FIG. 5 is a process flow diagram of an action recognition method applied in a gesture recognition scene provided by an embodiment of the present specification;

6 is a schematic structural diagram of a motion recognition device provided by an embodiment of the present specification;

FIG. 7 is a structural block diagram of a computing device provided by an embodiment of the present specification.

Detailed ways

In the following description, numerous specific details are set forth in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar promotions without departing from the connotation of this specification. Therefore, this specification is not limited by the specific implementation disclosed below.

The terminology used in one or more embodiments of this specification is for the purpose of describing a particular embodiment only and is not intended to limit the one or more embodiments of this specification. As used in the specification or embodiments and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used in this specification in one or more embodiments refers to and includes any and all possible combinations of one or more of the associated listed items.

It will be understood that although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited by these terms. These terms are only used to distinguish the same type of information from each other. For example, a first could be termed a second, and similarly, a second could be termed a first, without departing from the scope of one or more embodiments of this specification. Depending on the context, the word "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determining."

In this specification, a motion recognition method is provided, and the specification also relates to a motion recognition apparatus, a computing device, and a computer-readable storage medium, which will be described in detail one by one in the following embodiments.

FIG. 1 shows a flowchart of an action recognition method provided according to an embodiment of the present specification, which specifically includes the following steps:

Step 102: Acquire an image frame collected by an image collection device.

In practical applications, by recognizing the gestures placed by users to provide users with corresponding services, the service platform can enable users to obtain corresponding services through convenient operations, which can effectively improve the user experience. Degree will be the basis to support this service; gesture recognition is usually done by training gesture detectors, and gesture recognition through two-dimensional hand key points. Although the purpose of recognition can be achieved, there are limitations in flexibility and accuracy. Due to the defect, when the collected gesture images are affected by the collection angle, the accuracy of gesture recognition will be greatly reduced, and when new gestures are added, the gesture detector needs to be retrained to be able to recognize new gestures. Gestures, so the limitations of the scene are relatively large.

In the action recognition method provided in this embodiment, in order to improve the accuracy of action recognition, and in the case of adding new actions, action recognition can still be performed. The image frame is subjected to image segmentation processing to obtain an intermediate image containing the target object, and then the intermediate image is input into the action recognition model to obtain the action frame node and three-dimensional coordinate information. According to the three-dimensional coordinate information, the action frame The nodes and their associated virtual actions are assembled to generate the 3D virtual object, which realizes the recognition of the action type through the generated 3D virtual object, and effectively solves the problem that the image frame collection is not standard enough due to different image capture angles, which affects the accuracy of action type recognition. At the same time, according to the action recognition data set, the action recognition of the three-dimensional virtual object is carried out, and the action type of the target object is determined, so that it has more generality, flexibility and better expansion in the action recognition scene, It can realize that in the case of adding a new action type, the model does not need to be retrained, and the recognition of the new action type can be realized by adding an action recognition data set, so that the application scene becomes more extensive.

In specific implementation, the action recognition method provided in this specification can recognize gesture actions. For example, the user can recognize the gesture of the user by placing the number "8" on the hand and recognizing the posture of the user's hand. The type is the number type "8"; or it can also identify body movements, for example, the user places the letter "Y" through body movements, and by recognizing the user's body movements, it can be recognized that the user's body movements are letters Type "Y"; in addition, leg movements can also be identified, for example, the user places the Chinese character "人" on his legs, and by recognizing the user's leg movements, it can be identified that the user's leg movements are the type of Chinese characters "people".

Based on this, action recognition is to recognize the action type of the user's hand, limb or leg placement action. Correspondingly, the target object is the user's hand, limb or leg; if the target object is the hand In the case of the user's hand, the action recognition method at this time is to recognize the action type placed by the user's hand, and the recognized action type is the type of the gesture placed by the user through the hand, such as the number "8". Or the number "1", etc.; when the target object is a limb, the action recognition method at this time is to identify the action type of the user's limb, and the identified action type is the limb where the user passes the limb. The type it belongs to, such as the letter "Y" or the Chinese character "ge", etc.

In this embodiment, the action recognition method will be described by taking the target object as the user's hand. In the case where the target object is a limb or a foot, reference may be made to the description related to this implementation, and details are not repeated here. , it should be noted that, regardless of whether the target object is a limb or a hand, the specific description content can refer to each other.

Specifically, the image capture device specifically refers to a device that captures images of the actions placed by the user, which may be mobile phones or cameras. In the obtained image, when the captured image frame contains the action placed by the user, subsequent action recognition can be performed, or a large number of image frames can be collected through the image acquisition device, and the image frame containing the action placed by the user can be selected. The image is used as an image frame for subsequent action recognition.

In addition, before performing action recognition, the user needs to be informed of the types of actions that can be recognized. For example, in a payment scenario, the gestures that can be recognized by the user are "1" and "2". The gesture "1" indicates the payment amount, and the gesture "2" Indicates giving up the payment amount. At this time, the gestures that can be recognized by the payment platform need to be notified to the user, and then the gestures placed by the user can be recognized, and the gestures "1" or "2" placed by the user can be recognized for subsequent operations. , based on this, the action that the user needs to place needs to be displayed to the user through the action interaction page. In one or more implementations of this embodiment, the specific implementation is as follows:

Receive the click instruction submitted by the user through the action interaction page;

At least one action image frame is displayed to the user according to the click instruction; the action image frame includes a display area corresponding to the display action.

In practical applications, the action interaction page specifically refers to a page displayed to the user through the terminal device, and action interaction can be performed on this page, that is, the action placed by the user is recognized, and based on this, the user is received through the action interaction page. The click instruction submitted, at this time, it can be determined that the user needs to obtain the corresponding service by placing the corresponding action. At this time, at least one action image frame is displayed to the user according to the click instruction, wherein the action image frame contains the display action. Corresponding display area, the display action is the action that informs the user to be placed, and the display action can be a gesture action or a physical action. Correspondingly, the display area is the action corresponding to the display action in the action image frame. area.

For example, in the applet selection scenario, after the user submits a click instruction on the action interaction page corresponding to the applet selection item, the gesture action image frame corresponding to each applet is displayed to the user through the action interaction page. If the display action in the gesture action image frame is the same gesture, you can jump from this page to the corresponding applet page. Based on this, after the user submits the click instruction, the action image corresponding to the applet a is displayed to the user according to the click instruction. Frame a, which includes the display action a in the action image frame; displays the action image frame b corresponding to the applet b to the user, includes the display action b in the action image frame, and shows the user the action image frame corresponding to the applet c c. The action image frame contains the display action c. After the user places the action by hand, the user's hand action will be recognized later, and the corresponding applet will be jumped according to the recognition result, or the recognition result will be the same as a. If the three actions of , b, and c are all inconsistent, no action will be taken.

Before acquiring the image frame, the action image frame containing the display area of the display action is displayed to the user, so that the user can be informed of the type of action that can be recognized, and it is convenient for the user to place the corresponding action, and accurate action recognition can be performed to provide corresponding services to effectively improve the user experience.

Step 104: Perform segmentation processing on the action area of the target object in the image frame to obtain an intermediate image.

Specifically, on the basis of the above-mentioned acquisition of image frames, further, it is necessary to perform image segmentation processing on the image frames to obtain an intermediate image containing the action area of the target object, and then perform the following steps according to the intermediate image. action recognition process.

Further, in the case where the target object is a hand, that is, the feature area corresponding to the hand needs to be segmented, cropped, or an intermediate image containing the hand, which is used for subsequent recognition of the action type of the hand. This embodiment In one or more embodiments, the specific implementation is as follows:

detecting the characteristic area corresponding to the hand in the image frame;

Crop the image frame according to the feature area to obtain the intermediate image containing hand features;

Correspondingly, the action type is a gesture action type.

In practical applications, after the image frame is acquired, the hand in the image frame needs to be detected. When it is detected that the image frame contains the characteristic area corresponding to the hand, the image frame will be added according to the characteristic area. After cropping, the intermediate image containing the hand features can be obtained, which is used for the subsequent recognition of the action type of the hand.

Step 106 : Input the intermediate image into an action recognition model to perform key point identification and coordinate mapping, and obtain an action frame node corresponding to the identified action key point and three-dimensional coordinate information of the action key point mapping.

Specifically, on the basis of obtaining the intermediate image through the image segmentation process, further, the key points of action corresponding to the target object in the intermediate image are identified and the coordinates are mapped, based on this, by inputting the intermediate image to the The action recognition model performs key point recognition and coordinate mapping, and obtains the action key points corresponding to the target object in the intermediate image output by the action recognition model, the action frame nodes corresponding to the action key points, and the 3D coordinate information for action key mapping.

In practical applications, the action key points specifically refer to the key points that can be identified in the actions placed by the user, and each action key point is used for subsequent action recognition of the target object, and the action grouping node is specifically: Refers to the nodes included in the virtual action frame. Correspondingly, the virtual action frame can be understood as a three-dimensional model. The three-dimensional model refers to the model structure in the three-dimensional space, and the model can be adjusted through the three-dimensional coordinate information. , so as to construct the three-dimensional virtual object, which can improve the accuracy of action recognition, and the three-dimensional virtual object is a three-dimensional three-dimensional model consistent with the action of the target object.

Referring to Fig. 2, in the case where the target object is a hand, the intermediate image shown in Fig. 2 (a) is obtained by performing image segmentation processing on the image frame at this time, which includes the characteristic area corresponding to the hand. Input the image into the action recognition model for key point recognition and coordinate mapping, and obtain the action key points corresponding to the identified hand. There are 21 action key points that can be identified by the hand. For the distribution of these 21 key points, see As shown in (b) of Figure 2, by performing key point recognition on the intermediate image shown in (a) of Figure 2, the 21 action key points are obtained, which are action key point 0, action key point 1..., and Action key point 20, at this time, according to the action grouping nodes included in the virtual action grouping, as shown in Figure 3, the action grouping nodes included in the action grouping are respectively the action grouping node a, the action grouping node b... , and the action group node u, a total of 21 action group nodes, the action recognition model can identify the action group node corresponding to each action key point, and determine that the action key point 0 corresponds to the action group node a, and the action key point 1 corresponds to Action framing node b..., and action key point 20 corresponds to action framing node u;

Based on this, in order to improve the accuracy of subsequent action recognition, it is necessary to construct a three-dimensional virtual object in three-dimensional space. At this time, the three-dimensional coordinate information mapped by the key points of each action will be determined. The three-dimensional virtual object is generated by the action framing node corresponding to the point.

In one or more implementations of this embodiment, the action recognition model performs key point recognition, including:

Identifying the action key point corresponding to the target object in the intermediate image, and determining the key point label of the action key point;

Determine the target node label corresponding to the key point label based on the correspondence between the key point label and the node label of the action frame node;

The action frame node corresponding to the action key point to which the key point label belongs is determined according to the target node label.

In one or more implementations of this embodiment, the motion recognition model performs coordinate mapping, including:

determining the position information of the action key point in the intermediate image;

The three-dimensional coordinate information corresponding to the action key point is mapped based on the position information.

In practical applications, in the process of key point recognition and coordinate mapping, the action recognition model actually identifies each action key point corresponding to the target object in the intermediate image, and at the same time determines the key point of the action key point. label, and then determine the target node label corresponding to the key point label according to the pre-established correspondence between the key point label and the node label of the action grouping node, and finally determine the key point label according to the target node label to which the key point label belongs. The action frame node corresponding to the action key point of ;

Based on this, after identifying each of the action key points, the position information of the action key point in the intermediate image is determined at the same time, and then the three-dimensional coordinate information corresponding to the action key point is mapped based on the position information, The three-dimensional coordinate information mapped by the action key point may be understood as configuring depth coordinates for each action key point according to the position information, so as to map the three-dimensional coordinate information.

Referring to Figure 2, after inputting the intermediate image shown in Figure 2 (a) into the action recognition model, 21 action key points corresponding to the hand are identified in the intermediate image, and the key points of each action key point are determined at the same time The labels are label that 0, label 1..., and label 20 respectively, and the node labels of the action building node are label a, label b..., and label u, respectively, based on the correspondence between the key point label and the node label of the action building node relationship, determine the node label corresponding to each key point label, according to the node label, you can determine the action frame node corresponding to each action key point, respectively, the action key point 0 corresponds to the action frame node a, and the action key point 1 corresponds to the action frame Node b..., and the action key point 20 corresponds to the action frame node u;

Based on this, it is also determined that the position information of each action key point in the intermediate image is that the position information corresponding to the action key point 0 is (x1, y1), the position information corresponding to the action key point 1 is (x2, y2)..., and The position information corresponding to the action key point 20 is (x20, y20), and then the three-dimensional coordinate information is mapped for each action key point based on the position information. Correspondingly, the three-dimensional coordinate information mapped by the action key point 0 is (x1, y1, z1) , the three-dimensional coordinate information mapped by the action key point 1 is (x2, y2, z2)..., and the three-dimensional coordinate information mapped by the action key point 20 is (x20, y20, z20), which is used for the subsequent construction of a three-dimensional virtual object, that is, a hand The corresponding three-dimensional hand model.

In practical applications, the process of mapping the three-dimensional coordinate information corresponding to the action key points according to the position information may be as follows: first, a two-dimensional coordinate system is constructed based on the intermediate image, and it is determined that each action key point is in The corresponding two-dimensional coordinate information in the two-dimensional coordinate system, and then determine the transformation relationship between the two-dimensional coordinate system and the three-dimensional coordinate system constructed based on the virtual action frame, and finally determine based on the transformation relationship. The two-dimensional coordinate information is mapped to the three-dimensional coordinate information in the three-dimensional coordinate system, and the three-dimensional coordinate information may be used as the three-dimensional coordinate information mapped by each action key point.

In addition, the action key points of the target object can be set according to the actual application scenario. After the action key points are set and completed, as long as each action key point is identified through the action recognition model, the action corresponding to the target can be subsequently constructed. 3D virtual objects for action recognition.

In summary, through the action recognition model, the action frame node corresponding to each action key point and the mapped three-dimensional coordinate information are identified, so that virtual objects can be constructed in the three-dimensional space to perform actions in the follow-up according to the three-dimensional coordinate information. Recognition, avoiding the occurrence of the situation that affects the recognition accuracy due to the angle problem, can effectively improve the accuracy of action recognition.

Step 108: Generate a three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs.

Specifically, on the basis of obtaining the action frame node corresponding to the action key point and the mapped three-dimensional coordinate information, further, according to the three-dimensional coordinate information, the action frame node and the virtual action to which it belongs The three-dimensional virtual object is generated by assembling the frame, and the three-dimensional virtual object specifically refers to a three-dimensional "model" that is consistent with the action placed by the target object. problem, and the situation that affects the accuracy occurs.

Further, in the process of generating the three-dimensional virtual object, it is necessary to determine the corresponding relationship between the three-dimensional coordinate information and the action frame node, so that the virtual frame node can be adjusted to generate the three-dimensional virtual object, In one or more implementations of this embodiment, the specific implementations are as follows:

Determine the corresponding relationship between the action frame node and the three-dimensional coordinate information according to the action key point, and determine the connection relationship of the action frame node based on the virtual action frame;

The three-dimensional coordinate information is connected according to the connection relationship to generate the three-dimensional virtual object.

In practical applications, after determining the action frame node corresponding to the action key point and the three-dimensional coordinate information mapped by the action key point, the corresponding relationship between the action frame node and the three-dimensional coordinate information can be determined, and at the same time. According to the virtual action grouping, the connection relationship of the action grouping nodes can be determined, that is, each action grouping node should be able to form a virtual action grouping according to the connection relationship;

Based on this, the three-dimensional coordinate information is connected according to the connection relationship, so as to generate a three-dimensional virtual object matching the action placed by the target object.

Following the above example, it is determined that the action key point 0 corresponds to the action frame node a, the action key point 1 corresponds to the action frame node b... The action key point 20 corresponds to the action frame node u, and the three-dimensional coordinate information mapped by the action key point 0 is (x1, y1, z1), the three-dimensional coordinate information mapped by the action key point 1 is (x2, y2, z2)..., and the three-dimensional coordinate information mapped by the action key point 20 is (x20, y20, z20) Based on, At this point, it can be determined that the three-dimensional coordinate information (x1, y1, z1) corresponding to the action group node a, the three-dimensional coordinate information (x2, y2, z2) corresponding to the action group node b, and the action group node u corresponding to the three-dimensional coordinate information (x20 , y20, z20);

Further, while determining the corresponding relationship between each action frame node and the three-dimensional coordinate information, it can be determined according to the virtual action frame that the connection relationship of the action frame nodes is (a, b, c, d, e), (a) , f, g, h, i), (a, j, k, l, m), (a, n, o, p, q) and (a, r, s, t, u), according to the connection The three-dimensional coordinate information mapped by the key points of each action is connected, and the skeleton corresponding to the hand as shown in (a) in Figure 4 can be constructed, and then the area corresponding to the center of the hand is constructed based on the connection relationship. Refer to As shown in (b) of Figure 4, the palm area is constructed by a, b, c, d, e and f. At this time, it can be determined that the skeleton corresponding to the gesture placed by the user's hand is preliminarily completed, and after rendering and generation processing The three-dimensional virtual object can be obtained, that is, the three-dimensional model corresponding to the gesture placed by the user is shown in (c) of FIG.

To sum up, while determining the corresponding relationship between the action frame node and the three-dimensional virtual object, the connection relationship of each action frame node in the virtual action frame is determined, and the three-dimensional coordinate information is connected according to the connection relationship. The three-dimensional virtual object can be generated by processing, and the three-dimensional "model" that is consistent with the action of the target object can be generated in the three-dimensional space, and then the action recognition of the three-dimensional virtual object can accurately identify the action type of the target object, effectively to improve the recognition accuracy.

In addition, when the 3D virtual object is generated, since it is based on the 3D coordinate information obtained from the intermediate image, there may be a situation where one action key point corresponds to two or more 3D coordinate information. There will be two or more possibilities for 3D coordinate information to generate 3D virtual objects. For example, the 3D coordinate information mapped by the action key point 20 of the hand is (x20, y20, z20) and (x21, y21, z21) , at this time, two 3D virtual objects will be generated, namely, the little finger of the hand is buckled to the palm, and the little finger is straightened. If you continue to perform gesture recognition, there will be two types of actions corresponding to the gestures. In order to avoid this from happening , the constructed 3D virtual object can be normalized. In one or more implementations of this embodiment, the specific implementation is as follows:

detecting the number of objects of the generated three-dimensional virtual object;

In the case that the number of objects is greater than the preset number threshold, standardizing processing is performed on a plurality of three-dimensional virtual objects corresponding to the number of objects to obtain the virtual objects to be selected;

A target virtual object is selected from the virtual objects to be selected as the three-dimensional virtual object.

In practical applications, on the basis of generating the three-dimensional virtual object, the number of objects of the three-dimensional virtual object is further detected, and in the case that the number of objects is greater than a preset number threshold, the generated three-dimensional virtual object There are multiple objects. First, it is necessary to standardize the multiple three-dimensional virtual objects to obtain the virtual objects to be selected corresponding to the number of objects, and then select the one to be selected with the highest degree of matching with the action of the target object among the multiple virtual objects to be selected. The virtual object is determined as the target virtual object, and the target virtual object is used as the three-dimensional virtual object for subsequent action recognition; in the case that the number of the objects is less than or equal to the number threshold, the generated three-dimensional virtual object is described at this time. If there is one object, the following step 110 can be executed. During specific implementation, the preset number threshold is 1.

For example, there are three generated 3D virtual objects that match the user's hand movements, namely 3D virtual object 1, 3D virtual object 2 and 3D virtual object 3, wherein the action of 3D virtual object 1 is to straighten the thumb and forefinger , the other three fingers are buckled to the palm, the action of the 3D virtual object 2 is to straighten the thumb, index finger and middle finger, the other two fingers are buckled to the palm, the action of the 3D virtual object 3 is to straighten the thumb, index finger and little finger, other When two fingers are clasped to the palm, it can be determined that there are three gestures;

Further, by calculating the similarity between the three gestures and the gestures in the intermediate images corresponding to the hands, the three-dimensional virtual object corresponding to the gesture with the highest similarity is selected and determined as the three-dimensional virtual object for subsequent identification, and the three-dimensional virtual object is determined by calculation. Object 1 has the highest similarity to the gesture in the intermediate image, then the 3D virtual object 1 is used as the target 3D virtual object to perform subsequent gesture recognition.

In practical applications, standardizing the three-dimensional virtual objects specifically refers to making multiple three-dimensional virtual objects face the same direction by means of rotation, scaling, translation, etc. In this case, the standardized three-dimensional virtual objects are more convenient to select the target virtual object. object as the three-dimensional virtual object.

To sum up, after the three-dimensional virtual object is generated, if there are multiple three-dimensional virtual objects, the virtual object to be selected can be obtained through standardization processing, and then the target virtual object is selected from the virtual objects to be selected as the three-dimensional virtual object. Subsequent object recognition can further improve the accuracy of action recognition.

Step 110: Perform motion recognition on the three-dimensional virtual object based on the motion recognition data set, and determine the motion type of the target object.

Specifically, on the basis of the above-mentioned generation of the 3D virtual object, motion recognition is performed on the 3D virtual object according to the motion recognition data set, so as to determine the motion type of the target object, wherein the motion recognition data The set is composed of an action identification package issued by the server, and the action identification package contains identification rules for identifying the three-dimensional virtual object; the server specifically refers to the server to which the platform related to action identification belongs.

In practical applications, the action recognition data set is composed of multiple action recognition packages, and each action recognition package contains at least one action recognition rule, and each action recognition rule will be able to recognize an action type, for example, an action The recognition rules are rules for judging whether each fingertip is in the palm area and whether each fingertip is straight; based on the action recognition rule, the action recognition of the three-dimensional virtual object is performed, and the thumb and index finger of the three-dimensional virtual object are determined to be straight, and the other three are straightened. The fingertips of each of the fingers are in the palm, and it is determined that the action type of the three-dimensional virtual object at this time is the number type "8", then it can be determined that the user's gesture is "8".

Further, in the process of identifying the action type, it needs to be realized according to the action identification data set. In one or more implementations of this embodiment, the specific implementation is as follows:

Analyzing the action recognition data set to obtain action elements;

determining an intermediate action type of the 3D virtual object based on the recognition result of the action recognition performed on the 3D virtual object by the action element;

The intermediate action type is determined as the action type of the target object.

In practical applications, it is first necessary to analyze the identification data to obtain action elements. The action elements specifically refer to the elements for performing action recognition on the three-dimensional virtual object, that is, to judge the action type of the three-dimensional virtual object. The condition of performing motion recognition on the 3D virtual object according to the motion element, at this time, the intermediate motion type of the 3D virtual object can be determined according to the motion recognition result, and then the intermediate motion type is determined as the target The action type of the object is sufficient.

Further, in the process of performing motion recognition on the 3D virtual object based on the action elements to determine the intermediate action type of the 3D virtual object, the first aspect can be achieved by calculating the action angle of each virtual sub-object of the 3D virtual object. , the second aspect can be implemented by calculating the position of each virtual sub-object in the three-dimensional virtual object. In one or more implementations of this embodiment, the process of determining the intermediate action type in the first aspect includes:

calculating an action angle between each virtual sub-object in the three-dimensional virtual object;

The action angle is detected based on the action element, and the intermediate action type of the three-dimensional virtual object is determined according to the detection result.

In practical applications, the virtual sub-object specifically refers to the sub-units that make up the three-dimensional virtual object. For example, when the target object is a hand, the generated three-dimensional virtual object is the three-dimensional hand object, and the virtual sub-object is the are each finger and palm in the 3D hand object; or when the target object is a limb, the generated 3D virtual object is the 3D limb object, and the virtual sub-objects are each limb and body in the 3D limb object.

Based on this, by calculating the action angle between each virtual sub-object in the three-dimensional virtual object, the posture of each virtual sub-object is determined, and then the action angle is detected based on the action element, which can be determined according to the detection result. the intermediate action type of the three-dimensional virtual object.

For example, the 3D virtual object is a three-dimensional model based on the hand. At this time, the virtual sub-objects are the fingers and the palm of the hand. By calculating the first action angle between the fingers and the palm, and the second action between the fingers Angle, determine that the angle between the little finger, the ring finger, the middle finger and the palm is 0, and the angle between the thumb and the index finger is 90 degrees. At this time, the action angle is detected according to the action elements, and it is determined that the angle between the little finger, the ring finger, and the middle finger and the palm is 0. When the angle between the thumb and the index finger is 90 degrees, and the gesture at this time has the same action element as the gesture type "8", the intermediate action type of the 3D virtual object is determined to be the gesture type "8" according to the detection result.

In one or more implementations of this embodiment, the process of determining the intermediate action type in the second aspect includes:

Detecting the action position of each virtual sub-object in the three-dimensional virtual object according to the action element;

The intermediate action type of the three-dimensional virtual object is determined based on the detection result.

In practical applications, the action position specifically refers to the position of each virtual sub-object. For example, when the three-dimensional virtual object is generated based on the hand, the virtual sub-object is each finger and the palm of the hand, and the little finger is determined. , The action position of the ring finger and the middle finger is in the palm, the action position of the thumb and the index finger is outside the palm, at this time, each action position is detected according to the action elements, and it is determined that the action position of the little finger, the ring finger and the middle finger is in the palm, the big If the action positions of the thumb and forefinger are outside the palm, the gesture at this time has the same action element as the gesture type "8", then the intermediate action type of the 3D virtual object is determined to be gesture type "8" according to the detection result.

To sum up, in the process of determining the intermediate action type of the three-dimensional virtual object, the intermediate action type of the three-dimensional virtual object can be detected by calculating the action angle or action position of each virtual sub-object in the three-dimensional virtual object, which is effective. The accuracy of the recognition of action types is improved.

In addition, on the basis of displaying the action image frame to the user through the action interaction page, to perform action recognition at this time, it is necessary to recognize the action of the target object according to the recognition rules related to the displayed action in the action image frame, so as to determine Check whether the action placed by the user is correct. If it is correct, it means that the action type is the same as the action type of the displayed action, then other processing can be performed. In one or more implementations of this embodiment, the specific implementation is as follows :

Receive the motion recognition data set issued by the server for the motion image frame; the motion recognition data carries the motion recognition rule matching the display action;

According to the action recognition rule, determine whether the action of the three-dimensional virtual object matches the display action;

If so, determine the presentation action type of the presentation action according to the action recognition data set, and use the presentation action type as the action type;

If not, no processing will be made;

Further, after the displayed action type is used as the action type, recommendation information matching the action type may be displayed to the user through the action interaction page.

In practical application, in the case of receiving the motion recognition data set issued by the server for the motion image frame, according to the motion recognition data that carries the motion recognition rule matching the display action, Determine whether the action of the three-dimensional virtual object matches the display action; if it matches, it means that the action placed by the user through the target object is the same as the display action. At this time, the display action type can be used as the action type. If it does not match, it means that the action placed by the user through the target object is different from the display action, and the user needs to re-place the action until the placement is correct, and the display action type is used as the action type;

Based on this, at this time, it can be determined that the action type placed by the user is correct, and the recommendation information matching the action type can be displayed to the user through the action interaction page.

For example, in the payment scenario, when the user places "1" by gesture, it means to make payment, and when the user places "2" by gesture, it means to give up the payment, and two gestures are displayed to the user through the mobile phone. and the meaning of the two gestures, the gestures placed by the user are collected through the camera on the mobile phone, and the corresponding three-dimensional virtual objects are generated. 1" corresponds to gesture "2"), to determine whether the action of the three-dimensional virtual object matches gesture "1" or gesture "2";

When it matches the gesture "1", it is determined that the user's gesture type is "1" to indicate payment, the gesture type "1" is determined as the user's gesture type, and a payment successful reminder is displayed to the user through the payment page. Information; when it matches the gesture "2", it is determined that the user's gesture type is "2" to indicate abandonment of payment, the gesture type "2" is determined as the user's gesture type, and the payment page is displayed to the user to abandon payment reminder information;

When it does not match with gesture "1" and gesture "2", it is determined that there is an error in the gesture placed by the user through the hand, and the gesture type cannot be correctly recognized, and the user will be shown the gesture recognition again through the payment page. reminder information.

To sum up, after the recognition of the action of the target object is completed, the corresponding recommendation information will be displayed to the user according to the type of the action, and if it cannot be recognized, the reminder information will be displayed to the user through the action interaction page. The user experience effect is effectively improved, and it is convenient for the user to obtain relevant recommendation information.

In addition, on the basis of displaying the action image frame to the user through the action interaction page, in the case of identifying the action type, the action type will be matched with the action type of the displayed action, and the corresponding reminder will be displayed according to the matching result. information, in one or more implementations of this embodiment, the specific implementation is as follows:

matching the action type with the presentation action type of the presentation action;

If the matching is successful, displaying the successful matching information to the user through the action interaction page;

If the matching fails, reminding information is displayed to the user through the action interaction page, and the reminding information carries an action strategy.

In practical applications, the action strategy specifically refers to reminding the user how to correctly place the action to be successful.

For example, in the applet selection scene, the gestures placed by the user are collected through the camera on the mobile phone, and the corresponding 3D virtual objects are generated. According to the action recognition data set, the action type of the 3D virtual object is identified as gesture "2", and the user is determined. The gesture placed is "2". When there is a display action corresponding to gesture "2" in the selection scene of the applet, the user will be shown a successful matching reminder through the action interaction page in the mobile phone, and jump to For the applet corresponding to gesture "2", when there is no display action corresponding to gesture "2" in the applet selection scene, a matching failure reminder will be displayed to the user through the action interaction page in the mobile phone, and the user will be notified to the user. Displays the types of gestures that exist in the applet, which is convenient for users to place actions.

In the action recognition method provided in this embodiment, in the process of recognizing the action placed by the user, image segmentation is performed on the acquired image frame to obtain an intermediate image including the target object, and then the intermediate image is input into the action Identify the model, obtain the action frame node and three-dimensional coordinate information, and generate the three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs, and realize the three-dimensional virtual object that can be generated by The object recognizes the action type, which effectively solves the problem that the image frame collection is not standard enough due to different image acquisition angles, which affects the accuracy of the action type recognition. The action type of the target object is described, which makes it more versatile, flexible and better expandable in the action recognition scene. It can realize that in the case of adding a new action type, there is no need to retrain the model. By adding action recognition data sets In this way, the recognition of new action types can be realized, making the application scenarios more extensive.

The action recognition method is further described below by taking the application of the action recognition method provided in this specification in a gesture recognition scene as an example with reference to FIG. 5 . 5 shows a processing flow chart of an action recognition method applied in a gesture recognition scene provided by an embodiment of the present specification, which specifically includes the following steps:

Step 502: Receive a click instruction submitted by the user through the action interaction page.

Specifically, in order to facilitate users to use the corresponding payment software and improve the convenience of users, the payment platform provides gesture recognition services for users. Users can perform corresponding payment operations by placing gesture actions on the action interaction page. When the user gestures In the case of "1", the payment account balance can be shown to the user, when the user's gesture is "2", the payment details can be displayed to the user, when the user's gesture is ..., that is, different gestures Corresponding to different payment information;

Based on this, when the user submits a click instruction, the user's gesture needs to be recognized at this time, so as to display the corresponding payment information to the user.

Step 504: Display the action image frame through the action interaction page according to the click instruction, and the action image frame includes a display area corresponding to the display action.

Specifically, the action image frame includes display areas corresponding to gesture "1" and gesture "2", and the user is informed of the gestures that can be placed through the action interaction page, so that the user can correctly place gestures and subsequent gesture recognition.

Step 506: Acquire image frames collected by the mobile phone.

Step 508: Perform segmentation processing on the feature area of the gesture in the image frame to obtain an intermediate image including the gesture.

Step 510: Input the intermediate image into the action recognition model to perform key point recognition and coordinate mapping, and obtain the gesture frame nodes corresponding to the identified gesture key points and the three-dimensional coordinate information of the gesture key point mapping.

Specifically, for key point identification and coordinate mapping, reference may be made to the relevant descriptions in the foregoing embodiments, and details are not described here in this embodiment.

Step 512 : Determine the correspondence between the gesture framing nodes and the three-dimensional coordinate information according to the gesture key points, and determine the connection relationship between the gesture framing nodes based on the virtual gesture framing.

Step 514: Perform connection processing on the three-dimensional coordinate information according to the connection relationship to obtain a plurality of three-dimensional virtual gestures.

Step 516: Standardize the multiple three-dimensional virtual gestures, and determine the target three-dimensional virtual gesture according to the standardization processing result.

Step 518: Receive the gesture recognition rules issued by the server for the action image frame.

Step 520: Perform gesture recognition on the target three-dimensional virtual gesture according to the gesture recognition rule, and determine the gesture type of the user.

Step 522 : determine whether the gesture type matches the gesture type of the display motion; if not, go to step 524 ; if yes, go to step 526 .

Step 524: Display the reminder information to the user.

Step 526: Show the user recommended information matching the gesture type.

Specifically, gesture recognition is performed on the three-dimensional virtual gesture according to the gesture recognition rules, and if it is determined that the user's gesture type is "1", the payment account balance is displayed to the user, and when it is determined that the user's gesture does not match the gesture type of the displayed action In the case of , indicating that there is a problem with the gesture placed by the user, a reminder message is displayed to the user to remind the user to place the gesture again.

The action recognition method provided in this embodiment realizes the recognition of action types through the generated three-dimensional virtual objects, effectively solves the problem that the image frame acquisition is not standard enough due to different image acquisition angles and affects the accuracy of action type recognition. The action recognition data set performs action recognition on the three-dimensional virtual object, and determines the action type of the target object, which makes it more versatile, flexible and better expansibility in the action recognition scene, and can realize the addition of new action types. In this case, there is no need to retrain the model, and the recognition of new action types can be realized by adding action recognition data sets, making the application scenarios more extensive.

Corresponding to the above method embodiments, the present specification also provides an embodiment of a motion recognition apparatus, and FIG. 6 shows a schematic structural diagram of a motion recognition apparatus provided by an embodiment of the present specification. As shown in Figure 6, the device includes:

an acquisition module 602, configured to acquire image frames acquired by the image acquisition device;

The processing module 604 is configured to perform segmentation processing on the action area of the target object in the image frame to obtain an intermediate image;

The identification module 606 is configured to input the intermediate image into the action recognition model to perform key point identification and coordinate mapping, and obtain the action frame node corresponding to the identified action key point, and the three-dimensional coordinate information of the action key point mapping;

The generating module 608 is configured to generate a three-dimensional virtual object according to the three-dimensional coordinate information, the action framing node and the virtual action framing node to which it belongs;

The determining module 610 is configured to perform motion recognition on the three-dimensional virtual object based on the motion recognition data set, and determine the motion type of the target object.

In an optional embodiment, the identifying module 606 includes:

determining a key point labeling unit, configured to identify the action key point corresponding to the target object in the intermediate image, and determine the key point label of the action key point;

determining a node label unit, configured to determine the target node label corresponding to the key point label based on the corresponding relationship between the key point label and the node label of the action grouping node;

A unit for determining an action building node is configured to determine, according to the target node label, the action building node corresponding to the action key point to which the key point label belongs.

In an optional embodiment, the identifying module 606 includes:

a location information determining unit configured to determine location information of the action key point in the intermediate image;

The mapping coordinate information unit is configured to map the three-dimensional coordinate information corresponding to the action key point based on the position information.

In an optional embodiment, the generating module 608 includes:

determining a connection relationship unit, configured to determine the corresponding relationship between the action frame node and the three-dimensional coordinate information according to the action key point, and determine the connection relationship of the action frame node based on the virtual action frame;

A three-dimensional virtual object generating unit is configured to perform connection processing on the three-dimensional coordinate information according to the connection relationship to generate the three-dimensional virtual object.

In an optional embodiment, the motion recognition device further includes:

a detection module, configured to detect the number of objects of the generated three-dimensional virtual object;

a standardization processing module, configured to perform standardization processing on a plurality of three-dimensional virtual objects corresponding to the number of objects when the number of objects is greater than a preset number threshold to obtain virtual objects to be selected;

The selection module is configured to select a target virtual object from the virtual objects to be selected as the three-dimensional virtual object.

In an optional embodiment, the motion recognition device further includes:

an instruction receiving module, configured to receive a click instruction submitted by the user through the action interaction page;

The display module is configured to display at least one action image frame to the user according to the click instruction; the action image frame includes a display area corresponding to the display action.

In an optional embodiment, the determining module 610 includes:

a receiving unit, configured to receive the motion recognition data set issued by the server for the motion image frame; the motion recognition data carries the motion recognition rule matching the display action;

a judgment unit, configured to judge whether the action of the three-dimensional virtual object matches the display action according to the action recognition rule;

If so, run the determine action type unit;

The determining action type unit is configured to determine a presentation action type of the presentation action according to the action recognition data set, and use the presentation action type as the action type.

In an optional embodiment, the motion recognition device further includes:

A display recommendation information module is configured to display recommendation information matching the action type to the user through the action interaction page.

In an optional embodiment, the motion recognition device further includes:

a matching module configured to match the action type with the presentation action type of the presentation action;

If the matching is successful, the first display information module is executed, and the first display information module is configured to display the successful matching information to the user through the action interaction page;

If the matching fails, a second display information module is executed, and the second display information module is configured to display reminder information to the user through the action interaction page, and the reminder information carries an action strategy.

In an optional embodiment, when the target object is a hand, the processing module 604 includes:

a feature region detection unit, configured to detect the feature region corresponding to the hand in the image frame;

an image cropping unit, configured to crop the image frame according to the feature area to obtain the intermediate image containing hand features;

Correspondingly, the action type is a gesture action type.

In an optional embodiment, the determining module 610 includes:

a parsing unit, configured to parse the action recognition data set to obtain action elements;

an identification unit, configured to determine an intermediate action type of the three-dimensional virtual object based on the identification result of the action recognition on the three-dimensional virtual object based on the action element;

A determination unit configured to determine the intermediate action type as the action type of the target object.

In an optional embodiment, the identifying unit includes:

a calculation submodule, configured to calculate the action angle between each virtual sub-object in the three-dimensional virtual object;

The first determining submodule is configured to detect the action angle based on the action element, and determine the intermediate action type of the three-dimensional virtual object according to the detection result.

In an optional embodiment, the identifying unit includes:

a detection sub-module, configured to detect the action position of each virtual sub-object in the three-dimensional virtual object according to the action element;

The second determination submodule is configured to determine the intermediate action type of the three-dimensional virtual object based on the detection result.

In an optional embodiment, the motion recognition data set is composed of an motion recognition package delivered by a server, wherein the motion recognition package contains recognition rules for recognizing the three-dimensional virtual object.

The motion recognition device provided in this specification performs image segmentation processing on the acquired image frame during the process of recognizing the motion placed by the user, obtains an intermediate image containing the target object, and then inputs the intermediate image into the motion recognition model, obtain the action frame node and three-dimensional coordinate information, and generate the three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs, and realize the three-dimensional virtual object that can be generated by Identifying the action type effectively solves the problem that the image frame collection is not standard enough due to different image acquisition angles and affects the accuracy of the action type recognition. The action type of the target object makes it more versatile, flexible and more scalable in the action recognition scene. It can realize that when a new action type is added, there is no need to retrain the model. In this way, the recognition of new action types can be realized, making the application scenarios more extensive.

The above is a schematic solution of a motion recognition apparatus according to this embodiment. It should be noted that the technical solution of the motion recognition device and the technical solution of the above-mentioned motion recognition method belong to the same concept, and the details that are not described in detail in the technical solution of the motion recognition device can be referred to the description of the technical solution of the above-mentioned motion recognition method. .

FIG. 7 shows a structural block diagram of a computing device 700 according to an embodiment of the present specification. Components of the computing device 700 include, but are not limited to, memory 710 and processor 720 . The processor 720 is connected with the memory 710 through the bus 730, and the database 750 is used for storing data.

Computing device 700 also includes access device 740 that enables computing device 700 to communicate via one or more networks 760 . Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. Access device 740 may include one or more of any type of network interface (eg, network interface card (NIC)), wired or wireless, such as IEEE 802.11 wireless local area network (WLAN) wireless interface, World Interoperability for Microwave Access ( Wi-MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and the like.

In one embodiment of the present specification, the above-described components of computing device 700 and other components not shown in FIG. 7 may also be connected to each other, such as through a bus. It should be understood that the structural block diagram of the computing device shown in FIG. 7 is only for the purpose of example, rather than limiting the scope of the present specification. Those skilled in the art can add or replace other components as required.

Computing device 700 may be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (eg, tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (eg, smart phones) ), wearable computing devices (eg, smart watches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. Computing device 700 may also be a mobile or stationary server.

The processor 720 is configured to execute the following computer-executable instructions:

Obtain the image frame collected by the image acquisition device;

Segmenting the action area of the target object in the image frame to obtain an intermediate image;

Inputting the intermediate image into the action recognition model to perform key point recognition and coordinate mapping, to obtain the action frame node corresponding to the identified action key point, and the three-dimensional coordinate information of the action key point mapping;

Generate a three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs;

Action recognition is performed on the three-dimensional virtual object based on the action recognition data set, and the action type of the target object is determined.

The above is a schematic solution of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above-mentioned motion recognition method belong to the same concept, and the details not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the above-mentioned motion recognition method.

An embodiment of the present specification further provides a computer-readable storage medium, which stores computer instructions, which, when executed by a processor, are used for:

Obtain the image frame collected by the image acquisition device;

Segmenting the action area of the target object in the image frame to obtain an intermediate image;

Inputting the intermediate image into the action recognition model to perform key point recognition and coordinate mapping, to obtain the action frame node corresponding to the identified action key point, and the three-dimensional coordinate information of the action key point mapping;

Generate a three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs;

Action recognition is performed on the three-dimensional virtual object based on the action recognition data set, and the action type of the target object is determined.

The above is a schematic solution of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the above-mentioned motion recognition method belong to the same concept, and the details not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the above-mentioned motion recognition method.

The foregoing describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

The computer instructions include computer program code, which may be in source code form, object code form, an executable file, some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable media may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable media Electric carrier signals and telecommunication signals are not included.

It should be noted that, for the convenience of description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that this specification is not limited by the described action sequence. Because in accordance with this specification, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily all necessary in the specification.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are provided only to aid in the elaboration of the present specification. Alternative embodiments are not intended to exhaust all details, nor do they limit the invention to only the described embodiments. Obviously, many modifications and variations are possible in light of the content of this specification. These embodiments are selected and described in this specification to better explain the principles and practical applications of this specification, so that those skilled in the art can well understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims (17)

1. An action recognition method, comprising: Obtain the image frame collected by the image acquisition device; Segmenting the action area of the target object in the image frame to obtain an intermediate image; Inputting the intermediate image into the action recognition model to perform key point recognition and coordinate mapping, to obtain the action frame node corresponding to the identified action key point, and the three-dimensional coordinate information of the action key point mapping; Generate a three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs; Action recognition is performed on the three-dimensional virtual object based on the action recognition data set, and the action type of the target object is determined. 2. The action recognition method according to claim 1, wherein the action recognition model carries out key point recognition, comprising: Identifying the action key point corresponding to the target object in the intermediate image, and determining the key point label of the action key point; Determine the target node label corresponding to the key point label based on the correspondence between the key point label and the node label of the action frame node; The action frame node corresponding to the action key point to which the key point label belongs is determined according to the target node label. 3. The action recognition method according to claim 2, wherein the action recognition model carries out coordinate mapping, comprising: determining the position information of the action key point in the intermediate image; The three-dimensional coordinate information corresponding to the action key point is mapped based on the position information. 4. The action recognition method according to claim 1, wherein generating a three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs, comprising: Determine the corresponding relationship between the action frame node and the three-dimensional coordinate information according to the action key point, and determine the connection relationship of the action frame node based on the virtual action frame; The three-dimensional coordinate information is connected according to the connection relationship to generate the three-dimensional virtual object. 5. The action recognition method according to claim 1 , after the step of generating a three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs is executed, and the action recognition-based The data set further includes: detecting the number of objects of the generated three-dimensional virtual object; In the case that the number of objects is greater than the preset number threshold, standardizing processing is performed on a plurality of three-dimensional virtual objects corresponding to the number of objects to obtain the virtual objects to be selected; A target virtual object is selected from the virtual objects to be selected as the three-dimensional virtual object. 6. The action recognition method according to claim 1, before the step of acquiring the image frames collected by the image capturing device is performed, further comprising: Receive the click instruction submitted by the user through the action interaction page; At least one action image frame is displayed to the user according to the click instruction; the action image frame includes a display area corresponding to the display action. 7. The motion recognition method according to claim 6, wherein the motion recognition is performed on the three-dimensional virtual object based on the motion recognition data set, and the motion type of the target object is determined, comprising: Receive the motion recognition data set issued by the server for the motion image frame; the motion recognition data carries the motion recognition rule matching the display action; According to the action recognition rule, determine whether the action of the three-dimensional virtual object matches the display action; If so, determine a presentation action type of the presentation action according to the action recognition data set, and use the presentation action type as the action type. 8 . The action recognition method according to claim 7 , wherein after determining the presentation action type of the presentation action according to the action recognition data set, and performing the presentation action type as the action type sub-step, further 8 . include: Recommended information matching the action type is displayed to the user through the action interaction page. 9. The motion recognition method according to claim 6, wherein after the step of performing motion recognition on the three-dimensional virtual object based on the motion recognition data set and determining the motion type of the target object is performed, the method further comprises: matching the action type with the presentation action type of the presentation action; If the matching is successful, displaying the successful matching information to the user through the action interaction page; If the matching fails, reminding information is displayed to the user through the action interaction page, and the reminding information carries an action strategy. 10. The action recognition method according to claim 1, wherein when the target object is a hand, performing segmentation processing on the action region of the target object in the image frame to obtain an intermediate image, comprising: detecting the characteristic area corresponding to the hand in the image frame; Crop the image frame according to the feature area to obtain the intermediate image containing hand features; Correspondingly, the action type is a gesture action type. 11. The action recognition method according to claim 1, wherein the action recognition is performed on the three-dimensional virtual object based on the action recognition data set, and the action type of the target object is determined, comprising: Analyzing the action recognition data set to obtain action elements; determining an intermediate action type of the 3D virtual object based on the recognition result of the action recognition performed on the 3D virtual object by the action element; The intermediate action type is determined as the action type of the target object. 12. The action recognition method according to claim 11, wherein the determination of the intermediate action type of the three-dimensional virtual object based on the recognition result of the action recognition on the three-dimensional virtual object based on the action elements comprises: calculating an action angle between each virtual sub-object in the three-dimensional virtual object; The action angle is detected based on the action element, and the intermediate action type of the three-dimensional virtual object is determined according to the detection result. 13. The motion recognition method according to claim 11, wherein the determination of the intermediate motion type of the 3D virtual object based on the recognition result of performing motion recognition on the 3D virtual object based on the motion elements, comprises: Detecting the action position of each virtual sub-object in the three-dimensional virtual object according to the action element; The intermediate action type of the three-dimensional virtual object is determined based on the detection result. 14. The motion recognition method according to claim 1, wherein the motion recognition data set is composed of an motion recognition package issued by a server, wherein the motion recognition package includes an identification for recognizing the three-dimensional virtual object rule. 15. An action recognition device, comprising: an acquisition module, configured to acquire image frames collected by the image acquisition device; a processing module, configured to perform segmentation processing on the action area of the target object in the image frame to obtain an intermediate image; The recognition module is configured to input the intermediate image into the action recognition model to perform key point recognition and coordinate mapping, and obtain the action frame node corresponding to the identified action key point, and the three-dimensional coordinate information of the action key point mapping; a generating module, configured to generate a three-dimensional virtual object according to the three-dimensional coordinate information, the action framing node and the virtual motion framing node to which it belongs; The determining module is configured to perform motion recognition on the three-dimensional virtual object based on the motion recognition data set, and determine the motion type of the target object. 16. A computing device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions: Obtain the image frame collected by the image acquisition device; Segmenting the action area of the target object in the image frame to obtain an intermediate image; Inputting the intermediate image into the action recognition model to perform key point recognition and coordinate mapping, to obtain the action frame node corresponding to the identified action key point, and the three-dimensional coordinate information of the action key point mapping; Generate a three-dimensional virtual object according to the three-dimensional coordinate information, the action frame node and the virtual action frame to which it belongs; Action recognition is performed on the three-dimensional virtual object based on the action recognition data set, and the action type of the target object is determined. 17. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the action recognition method according to any one of claims 1 to 14.

Images