WO2018033154A1 - Gesture control method, device, and electronic apparatus - Google Patents

Abstract

A gesture control method, device, and electronic apparatus. The method comprises: detecting a gesture in a video image being played (S110); upon detecting the gesture matching a predefined gesture, determining a display location of the service object to be displayed in the video image (S120); and drawing, using computer graphics, at the display location, the service object (S130). The technical solution is adopted to save network resources and/or system resources at a user terminal, adding entertaining values to video images, preventing distraction of a user from viewing a video normally, thereby reducing the negative response of a user to viewing of a service object in a video image, attracting viewer attention, and increasing impact of the service object.

Description

Gesture control method, device and electronic device

This application is required to be submitted to the China Patent Office on August 19, 2016, the application number is CN201610696340.1, and the invention name is “gesture detection network training, gesture detection, gesture control method and device”, and China patent is submitted on August 19, 2016. Bureau, application number is CN201610707579.4, the invention name is “gesture detection network training, gesture detection and control method, system and terminal” and submitted to China Patent Office on August 19, 2016, application number is CN 201610694510.2, and the invention name is “ The priority of the Chinese Patent Application for the Method, Apparatus, and Terminal Device for Gesture Control, the entire contents of which is incorporated herein by reference.

Technical field

The present application relates to information processing technologies, and in particular, to a gesture control method, apparatus, and electronic device.

Background technique

With the development of Internet technology, people are increasingly using the Internet to watch video, and Internet video offers business opportunities for many new businesses. Internet video has become an important business traffic portal and is considered a premium resource for ad placement.

Existing video advertisements are mainly inserted into a fixed-time advertisement before the video is played, or at a certain time of the video playback, or placed in a fixed position in the area where the video is played and its surrounding area.

Summary of the invention

The embodiment of the present application provides a solution for gesture control.

According to an aspect of the embodiments of the present application, a gesture control method includes: performing gesture detection on a currently played video image; and determining that a service object to be displayed is in the video image when detecting that the gesture matches a predetermined gesture a presentation location; the business object is drawn in a computer drawing manner at the presentation location.

According to another aspect of the embodiments of the present application, a gesture control apparatus includes: a gesture detection module configured to perform gesture detection on a currently played video image; and a presentation location determining module configured to match the gesture to the predetermined gesture Determining a presentation location of the business object to be displayed in the video image; a business object rendering module, configured to draw the business object by using a computer drawing manner at the presentation location.

According to still another aspect of the embodiments of the present application, an electronic device includes: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface complete each other through the communication bus The memory is for storing at least one executable instruction that causes the processor to perform the operations of the steps in the gesture control method of any of the above-described embodiments of the present application.

According to still another aspect of the embodiments of the present application, another electronic device is provided, including:

a processor and a gesture control device according to any of the above embodiments of the present application;

When the processor runs the gesture control device, the unit in the gesture control device described in any of the above embodiments of the present application is operated.

According to still another aspect of an embodiment of the present application, a computer program is provided, comprising computer readable code, when a computer readable code is run on a device, a processor in the device performs the above-described An instruction of each step in the gesture control method of an embodiment.

According to still another aspect of the embodiments of the present application, a computer readable storage medium is provided for storing computer readable instructions, and when the instructions are executed, each of the gesture control methods of any of the above embodiments of the present application is implemented. The operation of the steps.

According to the gesture control scheme provided by the embodiment of the present application, the human hand and gesture detection are performed on the currently played video image, and the presentation position corresponding to the detected gesture is determined, and then the computer display manner is used to draw the display position of the video image. The displayed business object, when the business object is used to display the advertisement, on the one hand, the business object to be displayed is drawn by using a computer drawing manner at the determined display position, and the business object is combined with the video playing, and does not need to transmit the video independent through the network. Additional advertising video data is conducive to saving network resources and/or system resources of the client; on the other hand, the business object is closely combined with the gestures in the video image to preserve the main image and motion of the video subject (such as the anchor) in the video image. It can also add fun to the video image, and can also avoid disturbing the user to watch the video normally, which is beneficial to reducing the user's dislike of the business object displayed in the video image, can attract the viewer's attention to a certain extent, and improve the influence of the business object. force.

The technical solutions of the present application are further described in detail below through the accompanying drawings and embodiments.

DRAWINGS

The accompanying drawings, which are incorporated in FIG.

The present application can be more clearly understood from the following detailed description, in which:

1 is a flow chart of an embodiment of a gesture control method of the present application;

2 is a flowchart of an embodiment of a method for acquiring a first convolutional network model and a second convolutional network model in the embodiment of the present application;

3 is a flowchart of another embodiment of a method for acquiring a first convolutional network model and a second convolutional network model in the embodiment of the present application;

4 is a flow chart of another embodiment of a gesture control method of the present application;

FIG. 5 is a flowchart of still another embodiment of the gesture control method of the present application; FIG.

6 is a structural block diagram of an embodiment of a gesture control apparatus of the present application;

7 is a structural block diagram of another embodiment of the gesture control apparatus of the present application;

8 is a schematic structural diagram of an embodiment of an electronic device of the present application;

FIG. 9 is a schematic structural diagram of another embodiment of an electronic device according to the present application.

detailed description

Various exemplary embodiments of the present application will now be described in detail with reference to the drawings. It should be noted that the relative arrangement of the components and steps, numerical expressions and numerical values set forth in the embodiments are not intended to limit the scope of the application.

In the meantime, it should be understood that the dimensions of the various parts shown in the drawings are not drawn in the actual scale relationship for the convenience of the description.

The following description of the at least one exemplary embodiment is merely illustrative and is in no way

Techniques, methods and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but the techniques, methods and apparatus should be considered as part of the specification, where appropriate.

It should be noted that similar reference numerals and letters indicate similar items in the following figures, and therefore, once an item is defined in one figure, it is not required to be further discussed in the subsequent figures.

Those skilled in the art can understand that the terms “first”, “second” and the like in the embodiments of the present application are only used to distinguish different steps, devices or modules, etc., and do not represent any specific technical meaning or between them. The inevitable logical order.

Embodiments of the present application can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and/or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, and the like include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients Machines, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above, and the like.

Electronic devices such as terminal devices, computer systems, servers, etc., can be described in the general context of computer system executable instructions (such as program modules) being executed by a computer system. Generally, program modules may include routines, programs, target programs, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types. The computer system/server can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communication network. In a distributed cloud computing environment, program modules may be located on a local or remote computing system storage medium including storage devices.

1 is a flow chart of an embodiment of a gesture control method of the present application. The gesture control method of various embodiments of the present application can be exemplarily executed by an electronic device such as a computer system, a terminal device, a server, or the like. Referring to FIG. 1, the gesture control method of this embodiment includes:

At step S110, gesture detection is performed on the currently played video image.

In various embodiments of the present application, the video image may be an image in a live broadcast video being broadcasted, a video image in a recorded video, or a video image being recorded. In various embodiments of the present application, the gestures may include, but are not limited to, one or any combination of the following: wave, scissors, fist, hand, palm closing or opening, heart hand, applause, thumbs up Pose the pistol posture, swing the V-hand and swing the OK hand.

In an optional example of the embodiments of the present application, a live video is taken as an example. Currently, the live video platform includes multiple, such as a pepper live broadcast platform, a YY live broadcast platform, etc., each live broadcast platform includes multiple live broadcast rooms. Each live room will include at least one anchor. The anchor can broadcast a video to a fan in the live room where the electronic device (such as a mobile phone, tablet, or PC) is located, and the live video includes multiple video images. . The subject in the above video image is usually a main character (ie, anchor) and a simple background, and the anchor often occupies a larger area in the video image. When a business object (such as an advertisement, etc.) needs to be inserted during the live video, the video image in the current live video can be obtained.

In addition, the video image in the embodiments of the present application may also be a video image in a short video that has been recorded. In this case, the user may use the electronic device to play the short video. In the process of playing, the electronic device may Each frame or each key frame or each sample frame video image is acquired as a video image to be processed.

In addition, in the embodiments of the present application, in the case of a video image being a video image being recorded, in the recording process The electronic device can acquire each frame or each key frame or each sample frame video image recorded as a video image to be processed.

In various embodiments of the present application, a mechanism for performing human hand detection on a video image and gesture detection in a human hand candidate region where a human hand is located may be set in an electronic device that plays a video image or an electronic device used by a user (eg, an anchor), through the above mechanism. The currently played video image (ie, the video image to be processed) may be detected to determine whether the user's hand information is included in the video image to be processed, and if so, the video image is acquired to pass the subsequent embodiment of the present application. The video image is processed; if not included, the video image may be discarded or not processed, and the next frame of video image may be acquired to continue the above processing. In various embodiments of the present application, the hand information may include, for example, but not limited to, a finger state and position, a state and position of the palm, a closing and opening of the hand, and the like.

For a video image including hand information (or a human hand), a human hand candidate region in which a human hand is located may be detected from the video image, wherein the human hand candidate region may be a minimum rectangular region in the video image that covers the entire human hand candidate region or other The area of a shape (such as an ellipse, etc.). An optional process may be: the electronic device acquires a video image currently being played as a video image to be processed, and adopts a preset mechanism to extract an image including a candidate region of the human hand from the video image, and then The image of the candidate region of the human hand can be analyzed and extracted by a preset mechanism, and the feature data of each 30 parts (including fingers, palms, etc.) in the candidate region of the human hand is obtained, and the analysis of the feature data is performed. The gesture in the candidate region of the human hand in the video image belongs to any of gestures such as waving, scissors, clenching, holding, palm closing or opening.

In addition, in order to accurately and quickly determine the presentation position of the business object to be displayed in the video image, the presentation position of the business object can be restricted by the hand position. The hand position may be a center position of the human hand candidate area, or may be a coordinate position determined by a plurality of edge positions such as a rectangular area or an elliptical area of the human hand candidate area. For example, after determining the area where the hand is located in the video image, the human hand candidate area is analyzed and calculated, and the center position of the human hand candidate area is determined as the hand position. For example, if the human hand candidate area is a rectangular area, the The diagonal length of the rectangular area can be selected as the hand position of the diagonal, so that the hand position determined based on the candidate area of the hand can be obtained. In addition to the central position of the human hand candidate area as the hand position, a plurality of edge positions such as a rectangular area or an elliptical area of the human hand candidate area may be used as the hand position, and the corresponding processing may be referred to as the center. The location is the content of the hand position and will not be described here.

In an alternative example, step S110 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by a gesture detection module 601 that is executed by the processor.

In step S120, when it is detected that the gesture matches the predetermined gesture, the presentation position of the business object to be displayed in the video image is determined.

In various embodiments of the present application, the business object to be displayed is an object created according to a certain business requirement, and may include, for example, but not limited to, advertisement, entertainment, weather forecast, traffic forecast, pet, and the like. In various embodiments of the present application, the service object may include any one or more of the following: a video, an image, an effect including semantic information, and the like. For example, the special effect including the semantic information may include, but is not limited to, including: At least one or any of the following special effects of the advertisement information: two-dimensional sticker effects, three-dimensional effects, particle effects, and the like. The presentation position may be a center position of a designated area in the video image, or may be a coordinate position or the like of a plurality of edge positions in the specified area. The predetermined gestures in the embodiments of the present application may include, but are not limited to, one or any combination of the following: wave, scissors, fist, hand, palm closing or opening, heart hand, applause, vertical Thumb, swing pistol posture, swing V-hand and swing OK hand.

In an optional example of the embodiments of the present application, feature data of a plurality of different gestures may be pre-stored, and different gestures are marked correspondingly to distinguish the meaning represented by each gesture. Through the processing of the above step S110, the human hand candidate area where the human hand and the human hand are located and the gesture in the human hand candidate area can be detected from the video image to be processed, and the detected hand gesture can be separately compared with the pre-stored gesture. If a plurality of different gestures stored in advance include the same gesture as the gesture of detecting the hand, it may be determined that the detected gesture matches the corresponding predetermined gesture.

In order to improve the accuracy of the matching, the matching result may be determined by calculation. For example, a matching algorithm may be set to calculate the matching degree between the feature data of any two gestures. For example, the matching data of the feature data of the detected gesture and the feature data of the pre-stored gesture may be used to obtain a matching degree value between the two. Calculating the matching degree value between the detected gesture and the feature data of each gesture stored in advance by using the above manner, and selecting the maximum matching degree value from the obtained matching degree value, if the maximum matching degree value exceeds The predetermined matching threshold may determine that the pre-stored gesture corresponding to the largest matching value matches the detected gesture of the hand. If the maximum matching degree value does not exceed the predetermined matching threshold, the matching fails, that is, the detected hand gesture is not a predetermined gesture, and at this time, the processing of step S110 described above may be continued on the subsequent video image.

In addition, when it is determined that the detected gesture matches the corresponding predetermined gesture, the meaning represented by the gesture of the matched hand may be determined first, and the meanings corresponding to or corresponding to the meaning may be selected among a plurality of preset display positions. The presentation location is the presentation location of the business object to be displayed in the video image. In addition, for the case of the hand position determined in the processing of the above step S110, the display position related to the meaning and the hand position or the corresponding display position may be selected among the plurality of preset display positions as the business object to be displayed. The presentation position in the video image. For example, in the case of video live broadcast, when the anchor of the anchor is detected, the hand can be The upper region of the candidate region is selected as the presentation location associated with or corresponding to it. As another example, when the gesture of the anchor wave is detected, the palm area or its background area may be selected as the presentation position associated with or corresponding to it.

In an alternative example, step S120 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a presentation location determining module 602 executed by the processor.

In step S130, the business object is drawn by computer drawing at the presentation position.

For example, in the case of a live video, when the gesture of the anchor is detected, the corresponding business object may be drawn by using a computer drawing method in the upper area of the palm of the candidate area of the anchor in the video image, for example, with a predetermined reservation. If the fan is interested in the business object, the fan can click on the area where the business object is located, and the electronic device of the fan can obtain the network link corresponding to the business object, and enter the business object through the network link. A related page in which to obtain resources related to the business object.

In an optional example of the embodiments of the present application, the business object may be drawn by using a computer drawing manner, and may be implemented by appropriate computer graphics image drawing or rendering, for example, but not limited to: based on Open Graphics Language (OpenGL). The graphics drawing engine draws and so on. OpenGL defines a professional graphical program interface for cross-programming language and cross-platform programming interface specifications. It is hardware-independent and can easily draw 2D or 3D graphics images. Through the OpenGL graphics rendering engine, not only can 2D effects be drawn, but also 3D stickers can be drawn, and 3D effects can be drawn and particle effects can be drawn. However, the application is not limited to the drawing method based on the OpenGL graphics rendering engine, and other methods may be adopted. For example, the drawing method based on the graphics engine (Unity) or the Open Computing Language (OpenCL) is also applicable to the present invention. Apply for each embodiment.

In an alternative example, step S130 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a business object rendering module 603 executed by the processor.

The gesture control method provided by the embodiment of the present application performs human hand and gesture detection on the currently played video image, and determines a presentation position corresponding to the detected gesture, and then draws a display to be displayed in a computer drawing manner at the above-mentioned display position of the video image. The business object, when the business object is used for displaying the advertisement, on the one hand, the computer object is used to draw the business object to be displayed at a certain display position, and the business object is combined with the video playing, and the network is not required to transmit the extra irrelevant to the video. Advertising video data is conducive to saving network resources and/or system resources of the client; on the other hand, the business object is closely combined with the gestures in the video image to preserve the main image and motion of the video subject (such as the anchor) in the video image. It can also add interest to the video image, and can also avoid disturbing the user to watch the video normally, which is beneficial to reducing the user's dislike of the business object displayed in the video image, can attract the viewer's attention to a certain extent, and improve the influence of the business object. .

In an optional example of the embodiments of the present application, the process of performing gesture detection on the currently played video image in step S110 in the foregoing embodiment shown in FIG. 1 may adopt a corresponding feature extraction algorithm or use a neural network model such as a volume. The product network model is implemented. 2 is a flow chart of an embodiment of a method for acquiring a first convolutional network model and a second convolutional network model in the embodiment of the present application. In this embodiment, the convolutional network model is taken as an example, and the human hand candidate region and the gesture detection of the video image are performed on the video image, and the first convolutional network model for detecting the human hand candidate region in the image may be pre-trained and used for the human hand. The second convolutional network model of the candidate region detection gesture. Referring to FIG. 2, the method for acquiring the first convolutional network model and the second convolutional network model of the embodiment includes:

In step S210, the first convolutional network model is trained according to the sample image containing the human hand annotation information, and the prediction information of the first convolutional network model for the human hand candidate region of the sample image is obtained.

In various embodiments of the present application, the human hand annotation information may include, for example, but not limited to, annotation information of a human hand area and/or annotation information of a gesture. The annotation information of the human hand area may include coordinate information of a location or a range of the human hand area, and the annotation information of the gesture may include specific type information of the gesture. In this embodiment, the labeling information of the human hand area and the labeling information of the gesture are not limited.

In various embodiments of the present application, the sample image containing the human hand annotation information may be a video image derived from an image acquisition device, and is composed of an image of one frame and one frame, or may be a single image or an image, and may also be a source. For other devices, the source and the access route of the sample image containing the hand-labeled information are not limited in this embodiment. An annotation operation can be performed in the sample image, for example, a plurality of human hand candidate regions can be marked in the sample image. In the embodiment of the present application, the human hand candidate area is the same as the human hand candidate area in the above embodiment of the present application.

In various embodiments of the present application, the prediction information of the candidate region of the human hand may include: location information of a region where the human hand is located in the sample image, for example, coordinate point information or pixel point information; integrity information of the human hand in the region where the human hand is located, for example, a human hand The area includes a complete human hand or only one finger; gesture information in the area where the human hand is located, for example, gesture type, and the like. The content of the prediction information of the candidate region of the human hand is not limited in this embodiment.

In each embodiment of the present application, the larger the resolution of the image, the larger the amount of data, and the more computing resources are required for the subsequent candidate region and gesture detection, the slower the detection speed, in view of this, In an optional implementation manner of the present application, the sample image may be an image that satisfies a preset resolution condition. For example, the above preset resolution condition may be: the longest side of the image does not exceed 640 pixels, the shortest side does not exceed 480 pixels, and the like.

After the sample image is obtained, for example, information of the human hand candidate region and the gesture may be marked in the sample image by manual labeling or machine labeling, and a plurality of sample images labeled with the human hand candidate region may be obtained. The labeled human hand candidate area may be a minimum rectangular area or an elliptical area in the image that can cover the whole hand.

In an optional example of various embodiments of the present application, the first convolutional network model may include: a first input layer, a first output layer, and a plurality of first convolution layers, wherein the first input layer is used to input an image, A plurality of first convolutional layers are used to detect an image to obtain a human hand candidate region, and the human hand candidate region is output through the first output layer. The network parameters of each layer in the first convolutional network model and the number of layers of the first convolutional layer may be set according to a preset rule, or may be randomly set, and which setting method may be determined according to actual needs.

Illustratively, when the first convolutional network model processes the sample image by using the plurality of first convolutional layers, that is, feature extraction is performed on the sample image, and the first convolutional network model obtains the candidate region of the human hand in the sample image, The first input layer obtains a sample image, and then extracts features of the sample image through the first convolutional layer, and determines a human hand candidate region in the sample image in combination with the extracted features and outputs through the first output layer.

Obtaining the annotation information of the region where the hand is located in the sample image, using the annotation information as a training basis, and inputting the sample image into the initial first convolutional network model model, and using the gradient descent method and the back propagation algorithm to the first The convolutional network model is trained to obtain the first convolutional network model. When training the first convolutional network model, the first input layer parameter, the first output layer parameter and the plurality of first convolution layer parameters may be trained, and then the first convolutional network model is constructed according to the obtained parameters. .

Illustratively, the first convolutional network model can be trained using a sample image containing human hand annotation information, so that the first convolutional network model obtained by the training is more accurate, and samples in various cases can be selected when selecting the sample image. The image, the sample image may include a sample image that labels the human hand information, and may also include a sample image that is not labeled with the human hand information.

In the embodiments of the present application, the first convolutional network model may be a Region Proposal Network (RPN). This embodiment is described by way of example. In the actual application, the first convolutional network model is not limited to this. For example, it may be another two-class or more classified convolutional neural network (CNN), a multi-box network (Multi-Box Network) or an end-to-end real-time target detection system (YOLO).

In an alternative example, step S210 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a human hand area determination module 604 that is executed by the processor.

In step S220, the prediction information of the human hand candidate region is corrected.

In this embodiment, the prediction information of the candidate region of the sample image obtained by training the first convolutional network model is a rough judgment result, and there may be a certain error rate. The prediction information of the candidate region of the hand is used as an input for training the second convolutional network model in the subsequent step, and the rough judgment result obtained by training the first convolutional network model can be corrected before training the second convolutional network model.

The optional correction process may be through manual correction, or introducing other convolutional network models to filter the error results, etc. The purpose of the correction is to improve the training second while ensuring that the input information of the second convolutional network model is accurate. The accuracy of the convolutional network model. This embodiment does not limit the correction process used.

In an alternative example, step S220 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a modification module 605 executed by the processor.

In step S230, the second convolutional network model is trained based on the predicted information and the sample image of the corrected human hand candidate region.

The second convolutional network model and the first convolutional network model share the feature extraction layer, and the parameters of the feature extraction layer are kept unchanged during the training process of the second convolutional network model.

In an optional example of various embodiments of the present application, the second convolutional network model may include: a second input layer, a second output layer, a plurality of second convolutional layers, and a plurality of fully connected layers. The second convolution layer is used for feature extraction, the fully connected layer is equivalent to the classifier, and the features extracted by the second convolution layer are classified, and the second convolutional network model is obtained when the gesture detection result in the sample image is obtained. The second input layer obtains the human hand candidate region, and then extracts the feature of the above-mentioned human hand candidate region through the second convolution layer, and the all-connected layer performs classification processing according to the feature of the human hand candidate region, determines whether the sample image includes the human hand, and includes the human hand. In the case, the gesture of the candidate region and the hand is determined, and finally the classification result is output through the second output layer. Since both the first convolutional network model and the second convolutional network model include a convolutional layer, in order to facilitate model training and reduce the amount of computation, the network parameters of the feature extraction layer in the above two convolutional network models can be set. The feature extraction layer is shared for the same network parameters, that is, the second convolutional network model and the first convolutional network model, and the parameters of the feature extraction layer are kept unchanged during the training process of the second convolutional network model.

Based on this, in the embodiment, when the second convolutional network model is obtained by training, the network parameters of the input layer and the network parameters of the classification layer may be trained first, and then the network parameters of the feature extraction layer of the first convolutional network model are obtained. The network parameter of the feature extraction layer of the second convolutional network model is determined, and then the second convolutional network model is constructed according to the network parameters of the input layer, the network parameters of the classification layer, and the network parameters of the feature extraction layer.

In the embodiment of the present application, the second convolutional network model may be trained by using the predicted information and the sample image of the corrected candidate region, so that the second convolutional network model obtained by the training is more accurate, and the sample image may be selected. A sample image in a plurality of cases may be selected, and the sample image may include a sample image marked with a gesture, and may also include a sample image not marked with a gesture.

In addition, the sample image in this embodiment may be a sample image that satisfies the above resolution conditions or other resolution conditions.

Through the gesture control method provided in this embodiment, two convolutional network models are respectively trained: the first convolutional network model is trained according to the sample image containing the human hand annotation information, and the first convolutional network model is obtained for the candidate region of the sample image. Predicting information; correcting prediction information of the candidate region of the human hand; training the second convolutional network model according to the predicted information of the corrected candidate region of the human hand and the sample image. Wherein, the first convolutional network model and the second convolutional network model have the following relationship: the first convolutional network model and the second convolutional network model share the feature extraction layer, and are maintained during the training of the second convolutional network model The parameters of the feature extraction layer are unchanged.

Before training the second convolutional network model, the rough judgment result obtained by training the first convolutional network model is corrected, and the corrected prediction information and the sample image of the candidate region are used as the input of the second convolutional network model. The accuracy of training the second convolutional network model can be improved while ensuring that the input information of the second convolutional network model is accurate.

In addition, the first convolutional network model and the second convolutional network model share the feature extraction layer, and the parameters of the feature extraction layer are kept unchanged during the training process of the second convolutional network model, and the feature extraction layer of the second convolutional network model The feature extraction layer of the first convolutional network model can be directly used to facilitate the training of the second convolutional network model, which is beneficial to reduce the computational amount of training the second convolutional network model.

In the embodiments of the present application, the second convolutional neural network may be, for example, a Fast Region with Convolutional Neural Network (FRCNN). This embodiment is only used as an example for description, and the second volume in practical application. The neural network is not limited to this. For example, it may be another two-class or multi-class convolutional neural network.

In an alternative example, step S230 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a convolutional model training module 606 executed by the processor.

In this embodiment, the first convolutional network model and the first convolutional network model obtained by the training can facilitate subsequent detection of the hand and gesture of the currently played video image, and determine the display position corresponding to the detected gesture. And then, in the above-mentioned display position of the video image, the business object to be displayed is drawn by using a computer drawing manner. When the business object is used for displaying the advertisement, on the one hand, the business object to be displayed is drawn by using a computer drawing manner at the determined display position, the business object. Combined with video playback, there is no need to transmit additional advertising video data that is not related to video through the network, which is conducive to saving network resources and/or system resources of the client; on the other hand, the business object is closely combined with the gestures in the video image, and can be retained. The main image and motion of the video subject (such as the anchor) in the video image can add interest to the video image, and can also avoid disturbing the user to watch the video normally, which is beneficial to reducing the user's dislike of the business object displayed in the video image. To some extent attract the attention of the audience and improve The influence of business objects.

In an optional example of the embodiments of the present application, the correcting the prediction information of the candidate region may include: inputting the plurality of supplementary negative sample images and the prediction information of the candidate region into the third convolutional neural network for classification, The negative samples in the candidate region are filtered to obtain prediction information of the corrected candidate region.

In various embodiments of the present application, the supplementary negative sample image may be, for example, a blank sample image without a human hand, or a sample image including information such as a human hand but marked with a human hand, or an image without a human hand, and the like. The supplemental negative sample image may be input only in the third convolutional neural network without inputting the first convolutional neural network and the second convolutional neural network, and the negative sample image may have only a negative sample image and no positive sample image.

In various embodiments of the present application, the specific number of the supplementary negative sample images input to the third convolutional neural network may fall within a predetermined allowable range from the difference in the number of the human hand candidate regions in the prediction information of the human hand candidate region, wherein the predetermined allowable range may be Set according to the actual situation, for example, set to a range of 3-5, including 3, 4, and 5. For example, if the number of candidate regions in the prediction information of the candidate region is 5, the number of complementary negative samples may be 8, 9, or 10. When the predetermined allowable range is set to 0, the specific number of the supplementary negative sample images input to the third convolutional neural network may be equal to the number of the human hand candidate regions in the prediction information of the human hand candidate region, for example, the prediction information of the human hand candidate region. The number of candidate regions in the middle is 5, and the number of supplementary negative samples is also 5.

In various embodiments of the present application, the third convolutional neural network is used to correct the prediction information of the candidate region of the human hand obtained by training the first convolutional neural network. The negative samples in the candidate region of the human hand can be filtered out, that is, the non-human hand region in the candidate region of the human hand is filtered out, and the predicted information of the corrected candidate region is obtained, so that the predicted information of the corrected candidate region is more accurate. The third convolutional neural network may be, for example, FRCNN, or other two-class or multi-class convolutional neural networks.

FIG. 3 is a flowchart of another embodiment of a method for acquiring a first convolutional network model and a second convolutional network model in the embodiment of the present application. As shown in FIG. 3, the method for obtaining the first convolutional network model and the second convolutional network model of the embodiment includes:

S310: Train the first convolutional neural network according to the sample image containing the human hand annotation information, and obtain prediction information of the first convolutional neural network for the human hand candidate region of the sample image.

In various embodiments of the present application, the sample image may be an image in a red, green, and blue (RGB) format, or may be an image in another format, for example, a color difference component (YUV) format image, etc., which is not limited in this application. .

In addition, the sample images in the embodiments of the present application may be obtained by using an image acquisition device. In actual applications, the acquired images may not satisfy the preset resolution conditions due to different hardware parameters, different settings, and the like of the image acquisition device. In an optional implementation manner of the present application, in the optional implementation manner of the present application, after the image acquisition device acquires the image, the collected image is scaled to obtain a sample image. .

In an implementation manner of the present application, the first convolutional neural network may include: a first input layer, a first feature extraction layer, and a first classification output layer, wherein the first classification output layer is used to predict a sample image. Whether the plurality of candidate regions are human candidate regions.

It should be noted that each layer included in the first convolutional neural network is functionally divided. For example, the first feature extraction layer may be composed of a convolution layer, or may be composed of convolution layers. The linear transformation layer is composed of a convolution layer, a nonlinear transformation layer and a pooling layer; the output result of the first classification output layer can be understood as a result of the two classification, which can be specifically implemented by a convolution layer. But it is not limited to being implemented by a convolutional layer.

In an optional example, when training the first convolutional neural network, the first input layer parameter, the first feature extraction layer parameter, and the first classification output layer parameter may be trained to be constructed, and then constructed according to the obtained parameters. The first convolutional neural network.

Illustratively, training the first convolutional neural network with the sample image can be understood as: training the initial model of the first convolutional neural network with the sample image to obtain a final first convolutional neural network. Among them, when the sample model is used to train the initial model of the first convolutional neural network, the gradient descent method and the backpropagation algorithm can be used for training.

The initial model of the first convolutional neural network may be determined according to factors such as the number of layers of the convolution layer manually set, the number of neurons in each convolution layer, and the like, and the number of convolution layers and neurons. The number and so on can be determined based on actual needs.

In an alternative example, step S310 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a human hand area determination module 604 that is executed by the processor.

S320: Replace the second feature extraction layer parameter of the second convolutional neural network for detecting the gesture with the first feature extraction layer parameter of the trained first convolutional neural network.

In an implementation manner of the present application, the second convolutional neural network may include: a second input layer, a second feature extraction layer, and a second classification output layer, wherein the second classification output layer is configured to output a sample image. Gesture detection results.

It should be noted that the second feature extraction layer is similar to the first feature extraction layer, and details are not described herein again. The output result of the second classification output layer may be understood as a result of multi-classification, which may be specifically implemented by a fully connected layer, but is not limited to being implemented by a fully connected layer.

In this embodiment, by using the first feature extraction layer parameter of the trained first convolutional neural network as the second feature extraction parameter, the training of the second feature extraction layer of the second neural network may be omitted, that is, The first convolutional neural network and the second convolutional neural network are jointly trained, and the two share the feature extraction layer, which is beneficial to improve the training speed of the convolutional neural network.

Exemplarily, the gesture detection result may include any one or more of the following predetermined gesture types: wave, scissors, fist, hand, thumbs up, pistol, OK hand, hand, open, closed, etc. In addition, it may optionally include: an unscheduled gesture type. The above-mentioned unscheduled gesture type can be understood as: a gesture type other than the predetermined gesture type described above or a situation indicating “no gesture”, which can further improve the gesture classification accuracy of the second convolutional neural network.

It should be noted that the present application is only described by taking the above gesture type as an example. In practice, the foregoing predetermined gesture type is not limited to the above.

In an alternative example, step S320 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a parameter replacement module executed by the processor.

S330: Training the second convolutional neural network parameter according to the prediction information and the sample image of the candidate region of the human hand, and keeping the second feature extraction layer parameter unchanged during the training process.

According to the prediction information and the sample image of the candidate region of the human hand, when training the second convolutional neural network, the human hand gesture in the sample image can be calibrated again, that is, the gesture of calibrating the human hand is open state, closed state, etc., and based on this time The calibration result and the above prediction information are trained on the initial model of the second convolutional neural network to obtain a final second convolutional neural network.

In an alternative example, step S330 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a second training module executed by the processor.

In an optional example of the embodiments of the present application, the second convolutional neural network parameter may be trained according to the prediction result of the human hand candidate region and the sample image, and the second feature extraction layer parameter is maintained during the training process. :

The prediction information of the candidate region of the human hand is corrected: the second convolutional neural network parameter is trained according to the predicted information and the sample image of the corrected candidate region of the human hand.

Illustratively, when the prediction information of the candidate region of the human hand is corrected, the plurality of supplementary negative sample images and the prediction information of the above-mentioned candidate region may be input into the third convolutional neural network for classification to filter the negative samples in the candidate region of the above-mentioned hand. , the predicted information of the corrected candidate region of the hand is obtained.

It should be noted that the supplementary negative sample image is only used as an input of the third convolutional neural network, and is not used as an input of the first convolutional neural network and the second convolutional neural network, and the supplementary negative sample image may be It is a blank image without a hand, or it can be an image containing a region similar to a hand (not a hand) but not labeled as containing a hand.

In addition, in the embodiment of the present application, the prediction information of the revised candidate area may also be performed by a labeler manually, which is not limited in this application.

Since the preset measurement result of the first convolutional neural network may have a large error, that is, the prediction information obtained by the first convolutional neural network is used to train the second convolutional neural network, the accuracy is poor, and the first volume is Compared with the prediction information obtained by the neural network, the accuracy of the prediction information of the corrected candidate region is much higher. Therefore, the prediction information of the corrected candidate region and the sample image are used to train the second convolutional neural network. The second convolutional neural network has a higher accuracy rate.

In this embodiment, the prediction information of the candidate region of the human hand is corrected, and then the second convolutional neural network parameter is trained according to the corrected prediction information of the candidate region and the sample image, thereby improving the accuracy of the second convolutional neural network obtained by the training. .

4 is a flow chart of another embodiment of the gesture control method of the present application. As shown in FIG. 4, the gesture control method of this embodiment includes: at step S410, acquiring a currently played video image.

For the content of the step S410, refer to the related content in step S110 of the embodiment shown in FIG. 1 , and details are not described herein again.

In this embodiment, the human hand candidate region corresponding to the hand information may be determined by the video image and the pre-trained convolutional network model, and the gesture of the hand is detected in the human hand candidate region. For the corresponding processing, refer to the following steps S420 to S440.

In step S420, the video image is detected by using the pre-trained first convolution network, and the first feature information of the video image and the prediction information of the candidate region of the human hand are obtained.

The first feature information includes hand feature information, and the first convolutional network model may be used to detect whether a plurality of candidate regions of the image segmentation are human hand candidate regions.

In this embodiment, the acquired video image containing the hand information may be input into the first convolutional network model, and the video image may be separately extracted and mapped by the network parameter in the first convolutional network model. And processing such as transforming to perform human hand candidate region detection on the video image to obtain a human hand candidate region included in the video image. For the prediction information of the candidate area, refer to the description and description in the foregoing embodiments, and details are not described herein again.

In step S430, the first feature information and the prediction information of the candidate region are used as the second feature information of the pre-trained second convolutional network model, and the second convolutional network model is used to perform the gesture of the video image according to the second feature information. Detecting, obtaining a gesture detection result of the video image.

The second convolutional network model and the first convolutional network model share a feature extraction layer. The gesture may include, but is not limited to, any one or more of the following: wave, scissors, fist, hand, clapping, palm open, palm closed, thumbs up, pistol pose, pendulum V and hand .

For the processing of the foregoing step S440, refer to related content in the foregoing embodiment of the present application, and details are not described herein again.

In an optional example, steps S410-S430 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by a gesture detection module 601 executed by the processor.

In step S440, when it is detected that the gesture matches the predetermined gesture, the feature points of the hand in the human hand candidate region corresponding to the detected gesture are extracted.

In various embodiments of the present application, for a video image including hand information, the hand includes certain feature points, such as finger points, palms, hand contours, and the like. The detection of the human hand in the video image and the determination of the feature point can be implemented in any suitable related art, which is not limited in the embodiment of the present application. For example, linear feature extraction methods such as principal component analysis (PCA), linear discriminant analysis (LDA), independent component analysis (ICA), etc.; for example, nonlinear feature extraction methods such as kernel principal component analysis (Kernel PCA), manifolds Learning, etc.; the trained neural network model can also be used to extract the feature points of the hand, such as the convolutional network model in the embodiment of the present application.

Taking a live video as an example, in the process of video live broadcast, the human hand is detected from the live video image and the feature points of the hand are determined; for example, during the playback of a recorded video, the playback is performed. In the video image, the human hand is detected and the feature points of the hand are determined; for example, during the recording of a certain video, the human hand is detected from the recorded video image and the feature points of the hand are determined.

In step S450, the presentation position of the business object to be displayed in the video image is determined according to the feature point of the hand.

In the implementation, after the feature points of the hand are determined, one or more presentation positions of the business object to be displayed in the video image may be determined based on the feature points of the hand.

In this embodiment, when determining the presentation position of the business object to be displayed in the video image according to the feature point of the hand, the optional implementation may include, for example:

In the first mode, according to the feature point of the hand, a pre-trained third convolution network model for detecting the presentation position of the business object from the video image is used, and the service to be displayed corresponding to the hand position is determined in the video image. The presentation position of the object; secondly, according to the feature point of the hand and the type of the business object to be displayed, the display position of the business object to be displayed corresponding to the detected gesture in the video image is determined in the video image.

Hereinafter, the above two modes will be exemplarily described.

method one

When the usage mode 1 determines the presentation position of the business object to be displayed in the video image, a convolutional network module can be trained in advance. Type, ie the third convolutional network model, the trained third convolutional network model has the function of determining the presentation position of the business object in the video image; or, the third party has been trained to have the determined business object A convolutional network model that exhibits the functionality of a location in a video image.

It should be noted that, in this embodiment, the training of the business object is taken as an example, but those skilled in the art should understand that the third convolutional network model can also train the opponent while training the business object. To achieve joint training of hands and business objects.

When training the third convolutional network model, an optional training method includes the following process:

(1) Obtaining a feature vector of a sample image of a business object to be trained.

The feature vector includes location information and/or confidence information of the business object in the business object sample image. The confidence information of the business object indicates the probability that the business object can achieve the effect (such as being focused or clicked or viewed) when the current location is displayed. The probability may be set according to the statistical analysis result of the historical data, or may be According to the results of the simulation experiment, it can also be set according to the artificial experience. In practical applications, only the location information of the business object may be trained according to actual needs, or only the confidence information of the business object may be trained, and both may be trained. When training both, the third convolutional network model after training can be used to determine the location information and confidence information of the business object more effectively and accurately, so as to provide a basis for the display of the business object.

The third convolutional network model is trained by using at least one sample image. In the embodiment of the present application, the third convolutional network model may be trained using the business object sample image including the business object, and those skilled in the art may understand that The sample image of the business object to be trained may include hand information in addition to the business object. In addition, the business object in the business object sample image in the embodiment of the present application may be pre-labeled with location information, or confidence information, or both location information and confidence information. Of course, in practical applications, this information can also be obtained through other means. By marking the corresponding information on the business object in advance, the data processing data and the number of interactions can be effectively saved, and the data processing efficiency is improved.

A business object sample image having location information and/or confidence information of the business object is used as a training sample, and feature vector extraction is performed to obtain a feature vector including location information and/or confidence information of the business object.

Alternatively, the third convolutional network model can be used to simultaneously train the counterpart and the business object. In this case, the feature vector of the business object sample image also includes the characteristics of the hand.

The extraction of the feature vector can be implemented in an appropriate manner in the related art, and details are not described herein again.

(2) Convolution processing of the feature vector to obtain the feature vector convolution result.

In this embodiment, the acquired feature vector convolution result includes location information and/or confidence information of the service object. In the case of joint training between the opponent and the business object, the feature vector convolution result also contains hand information.

The number of times of convolution processing on the feature vector can be set according to actual needs. That is, in the third convolutional network model, the number of layers of the convolution layer can be set according to actual needs, and details are not described herein again.

The eigenvector convolution result is the result of feature extraction of the feature vector, and the feature vector convolution result can effectively characterize the hand in the video image.

In the embodiment of the present application, when the feature vector includes both the location information of the service object and the confidence information of the service object, that is, when the location information and the confidence information of the service object are trained, The eigenvector convolution result is shared in the subsequent judgment of the convergence condition, and no need to perform repeated processing and calculation, which is beneficial to reduce resource loss caused by data processing and improve data processing speed and efficiency.

(3) determining whether the location information and/or the confidence information of the corresponding service object in the feature vector convolution result satisfies the convergence condition.

The convergence condition is appropriately set by a person skilled in the art according to actual needs. When the information satisfies the convergence condition, the network parameter in the third convolutional network model can be considered to be appropriate; when the information cannot satisfy the convergence condition, the network parameter in the third convolutional network model can be considered inappropriate, and it needs to be performed. Adjustment, the adjustment may be an iterative process until the result of convolution processing the feature vector using the adjusted network parameters satisfies the convergence condition.

In an optional manner, the convergence condition may be set according to a preset standard location and/or a preset standard confidence, for example, a location indicated by the location information of the service object in the feature vector convolution result and a preset The distance between the standard positions satisfies a certain threshold as a convergence condition of the location information of the service object; the difference between the confidence level indicated by the confidence information of the service object in the feature vector convolution result and the preset standard confidence satisfies a certain threshold The convergence condition of the confidence information as a business object, and the like.

Optionally, the preset standard location may be an average location obtained by averaging the location of the service object in the sample image of the business object to be trained; the preset standard confidence may be a sample image of the business object to be trained. The confidence level of the business objects in the average confidence obtained after averaging processing. Since the sample image is a sample to be trained and the amount of data is large, the standard position and/or the standard confidence can be set according to the position and/or confidence of the business object in the sample image of the business object to be trained, so that the standard position and the standard position are set. Standard confidence is also more objective and accurate.

Judging whether the position information and/or the confidence information of the corresponding business object in the feature vector convolution result satisfies the convergence condition An optional way to do this includes:

Obtaining the location information of the corresponding service object in the feature vector convolution result, and calculating the location indicated by the location information of the corresponding service object by calculating the Euclidean distance between the location indicated by the location information of the corresponding service object and the preset standard location Determining, according to the first distance, a first distance between the preset standard position and determining whether the location information of the corresponding service object satisfies the convergence condition;

and / or,

Obtaining the confidence information of the corresponding service object in the feature vector convolution result, calculating the Euclidean distance between the confidence level indicated by the confidence information of the corresponding service object and the preset standard confidence, and obtaining the confidence of the corresponding business object. The third distance between the confidence level of the information indication and the preset standard confidence level determines whether the confidence information of the corresponding service object satisfies the convergence condition according to the third distance. Among them, the Euclidean distance method is adopted, and the implementation is simple and can effectively indicate whether the convergence condition is satisfied. However, the embodiment of the present application is not limited thereto, and other methods such as a horse distance, a bar distance, and the like may also be adopted.

Optionally, as described above, the preset standard location is an average position obtained by averaging the positions of the business objects in the sample image of the business object to be trained; and/or, the preset standard confidence is to be trained. The confidence of the business object in the sample image of the business object is averaged after the average processing.

(4) If the convergence condition is satisfied, for example, the distance between the position indicated by the position information of the business object in the feature vector convolution result and the preset standard position satisfies a certain threshold, and the confidence of the business object in the feature vector convolution result If the difference between the confidence level of the information indication and the preset standard confidence satisfies a certain threshold, the training of the convolutional network model is completed; if the convergence condition is not satisfied, for example, the location information indication of the business object in the feature vector convolution result The distance between the position and the preset standard position does not satisfy a certain threshold, and/or the difference between the confidence level indicated by the confidence information of the business object and the preset standard confidence in the feature vector convolution result does not satisfy the certain The threshold value is adjusted according to the location information and/or the confidence information of the corresponding business object in the feature vector convolution result, and the network parameter of the third convolutional network model is adjusted according to the network parameter pair of the adjusted third convolutional network model. The third convolutional network model performs iterative training until the location information and/or confidence information of the service object after the iterative training is satisfied Convergence criteria.

By performing the above training on the third convolutional network model, the third convolutional network model can feature extracting and classifying the presentation position of the business object based on the hand presentation, thereby having the position of determining the presentation position of the business object in the video image. Features. Wherein, when the presentation location includes multiple, through the training of the above-mentioned business object confidence, the third convolutional network model may also determine the order of the presentation effects in the plurality of presentation locations, thereby determining the final based on the sequence of the advantages and disadvantages. Show position. When a business object is displayed in a subsequent application, a valid presentation location can be determined based on the current image in the video.

In addition, before performing the foregoing training on the third convolutional network model, the business object sample image may be pre-processed, including: acquiring a plurality of business object sample images, where each business object sample image includes a business object. The labeling information; determining the location of the business object according to the labeling information, determining whether the distance between the determined location of the business object and the preset location is less than or equal to a set threshold; and the business object sample image corresponding to the business object that is less than or equal to the set threshold , determined as the sample image of the business object to be trained. The preset position and the set threshold may be appropriately set by any suitable means by a person skilled in the art, for example, according to the statistical analysis result of the data or the related distance calculation formula or the artificial experience, etc., which is not limited by the embodiment of the present application.

By pre-processing the business object sample image, the sample image that does not meet the conditions can be filtered out to ensure the accuracy of the training result.

Through the above process, the training of the third convolutional network model is realized, and the trained third convolutional network model can be used to determine the presentation position of the business object in the video image. For example, in the live broadcast process, if the anchor clicks on the business object to indicate the display of the business object, after the third convolutional network model obtains the hand feature point of the anchor in the live video image, the location of the display service object may be indicated. For example, the forehead location of the anchor controls the live application to display the business object at the location; or, during the live broadcast of the video, if the anchor clicks on the business object to indicate the display of the business object, the third convolutional network model can be directly determined according to the live video image. The presentation location of the business object.

Way two

The presentation position of the business object to be displayed corresponding to the hand position is determined in the video image according to the feature point of the hand and the type of the business object to be displayed.

In the implementation, after the feature points of the hand are acquired, the presentation position of the business object to be displayed may be determined according to the set rules. Wherein, determining the presentation position of the business object to be displayed includes, for example, any one or more of the following: a palm area of the person in the video image, an upper area of the palm, a lower area of the palm, a background area of the palm, a body area other than the hand, The background area in the video image, the area within the setting range centering on the area where the hand is located in the video image, the area preset in the video image, and the like.

After the presentation location is determined, the presentation location of the business object to be displayed in the video image can be determined. For example, the center point of the presentation area corresponding to the presentation location is used to display the business object as the center point of the presentation location; for example, a certain coordinate position in the presentation area corresponding to the presentation location is determined as the center point of the presentation location, etc. This embodiment of the present application does not limit this.

In an optional implementation, when determining the presentation position of the business object to be displayed in the video image, the business object to be displayed is determined not only according to the feature point of the hand but also according to the type of the business object to be displayed. The position of the presentation in the video image. Ben Shen In various embodiments, the type of the business object includes, for example but not limited to, any one or more of the following: a forehead patch type, a cheek patch type, a chin patch type, a virtual hat type, a virtual clothing type, a virtual makeup type, The virtual headwear type, the virtual hair accessory type, and the virtual jewelry type may include, in addition to, a virtual cap type, a virtual cup type, a text type, and the like.

In addition, according to the type of the business object, the appropriate presentation position can be selected for the business object with reference to the feature point and the hand position of the hand.

Further, in a case where a plurality of presentation positions of the business object to be displayed in the video image are obtained according to the feature point of the hand and the type of the business object to be displayed, at least one presentation position may be selected from the plurality of presentation positions as The presentation location of the business object to be displayed in the video image. For example, for a text type business object, it can be displayed in the background area, or in the palm area of the character or the area above the hand.

In addition, the correspondence between the gesture and the presentation position may be stored in advance, and when it is determined that the detected gesture matches the corresponding predetermined gesture, the target presentation position corresponding to the predetermined gesture may be acquired from the correspondence between the pre-stored gesture and the presentation position. As the presentation location of the business object to be displayed in the video image. It should be noted that, although there is a correspondence between the above gesture and the presentation position, the gesture is not necessarily related to the presentation position. The gesture is only a way to trigger the presentation of the business object, and the presentation position and the human hand are not necessarily present. The relationship, that is, the business object can be displayed in a certain area of the hand, or can be displayed in other areas than the hand, such as the background area of the video image. In addition, the same gesture can also trigger the display of different business objects. For example, the anchor performs two wave gestures in succession, the first gesture can display two-dimensional sticker effects, the second gesture can display three-dimensional effects, etc. The content of the corresponding advertisement may be the same or different.

In an alternative example, steps S440-S450 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a presentation location determining module 602 executed by the processor.

In step S460, the business object to be displayed is drawn in a computer drawing manner at the presentation position.

When a business object is a two-dimensional sticker effect containing semantic information, the sticker can be used for advertising and display. Before drawing a business object, you can first obtain information about the business object, such as the identity and size of the business object. After the display position is determined, the business object may be scaled, rotated, etc. according to the coordinates of the presentation position, and then the business object to be displayed is displayed by a corresponding drawing method such as an open graphics language (OpenGL) graphics rendering engine. draw. In some cases, ads can also be displayed in 3D special effects, such as text or logos (LOGOs) that display ads through particle effects. For example, the virtual bottle cap type of two-dimensional sticker effects display the name of a product to attract viewers to watch, improve the efficiency of advertising and display.

In an alternative example, step S460 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a business object rendering module 603 executed by the processor.

The gesture control method provided by the embodiment of the present application triggers the display of the business object by using a gesture. When the business object is used for displaying the advertisement, on the one hand, the business object to be displayed is drawn by using a computer drawing manner at the determined presentation position, and the business object is The combination of video playback eliminates the need to transmit additional advertising video data unrelated to the video over the network, which is conducive to saving network resources and/or system resources of the client; on the other hand, the business object is closely combined with the gestures in the video image to retain the video. The main image and motion of the video main body (such as the anchor) in the image can add interest to the video image, and can also avoid disturbing the user to watch the video normally, which is beneficial to reducing the user's dislike of the business object displayed in the video image, and can be certain To the extent that it attracts the attention of the audience and enhances the influence of business objects.

FIG. 5 is a flow chart of still another embodiment of the gesture control method of the present application. As shown in FIG. 5, the gesture control method of this embodiment includes:

In step S501, the first convolutional network model is trained according to the sample image containing the human hand annotation information, and the prediction information of the first convolutional network model for the human hand candidate region of the sample image is obtained.

In an alternative example, step S501 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a human hand area determination module 604 that is executed by the processor.

In step S502, the prediction information of the human hand candidate region is corrected.

In an alternative example, step S502 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a modification module 605 executed by the processor.

In step S503, the second convolutional network model is trained based on the predicted information and the sample image of the corrected human hand candidate region.

The second convolutional network model and the first convolutional network model share the feature extraction layer, and the parameters of the feature extraction layer are kept unchanged during the training process of the second convolutional network model.

In an alternative example, step S503 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a convolutional model training module 606 executed by the processor.

For the content of the steps in the foregoing steps S501 to S503, refer to related content in the foregoing embodiment of the present application, and details are not described herein again.

In step S504, a feature vector of the business object sample image to be trained is acquired.

The feature vector includes location information and/or confidence information of the business object in the business object sample image, and a feature vector corresponding to the gesture. The business object sample image to be trained may be the sample image containing the human hand annotation information described above.

In this embodiment, there may be some sample images in the sample image of the business object that do not meet the training standard of the third convolutional network model, and the sample images may be filtered out by preprocessing the image of the business object sample.

In this embodiment, the business object sample image includes a business object, and the business object is marked with location information and confidence information. In an optional implementation, the location information of the central point of the business object is used as the location information of the business object. In this step, the sample image is filtered according to the location information of the business object. After obtaining the coordinates of the location indicated by the location information, the coordinates are compared with the preset location coordinates of the business object of the type, and the position variance of the two is calculated. If the location variance is less than or equal to the set threshold, the business object sample image may be used as the sample image to be trained; if the location variance is greater than the set threshold, the business object sample image is filtered out. The preset position coordinates and the set thresholds may be appropriately set by a person skilled in the art according to actual conditions. For example, since the images generally used for the third convolutional network model training have the same size, the threshold that can be set may be The length or width of the image is 1/20 to 1/5, and alternatively, it may be 1/10 of the length or width of the image.

In addition, the location and the confidence of the service object in the determined sample image of the business object to be trained may be averaged to obtain an average position and an average confidence, and the average position and the average confidence may be used as a basis for determining the convergence condition subsequently.

When the business object is a two-dimensional sticker special effect as an example, the business object sample image used for training in this embodiment is labeled with the coordinates of the advertisement location and the confidence of the advertisement space. Wherein, the advertisement position can be marked in the hand, the front background and the like, and the joint training of the advertisement points of the hand feature point and the front background can be realized, and the scheme is different from the scheme based on the manual training of one hand. Conducive to saving computing resources. The size of the confidence indicates the probability that this ad slot is a better ad slot, for example, if the ad slot is mostly occluded, the confidence is low.

At step S505, the feature vector is subjected to convolution processing to obtain a feature vector convolution result.

In step S506, it is determined whether the location information and/or the confidence information of the corresponding service object in the feature vector convolution result satisfies the convergence condition.

In step S507, if the convergence conditions in step S506 are all satisfied, the training on the third convolutional network model is completed; if the convergence conditions in step S506 are neither satisfied nor satisfied, the corresponding results are obtained according to the feature vector convolution result. The location information and/or confidence information of the business object, adjust the network parameters of the third convolutional network model, and iteratively train the third convolutional network model according to the adjusted network parameters of the third convolutional network model, The position information and/or confidence information of the business object after the iterative training corresponds to the convergence condition.

For the processing of the foregoing steps S504 to S507, reference may be made to related content in the foregoing embodiments of the present application, and details are not described herein again.

Through the processing of the above steps S504 to S507, the trained third convolutional network model can be obtained. For the structure of the third convolutional network model, reference may be made to the structure of the first convolutional network model or the second convolutional network model in the embodiment shown in FIG. 2 or FIG. 3 of the present application, and details are not described herein again.

The first convolutional network model, the second convolutional network model, and the third convolutional network model obtained by the above training may perform corresponding processing on the video image, and specifically may include the following steps S508 to S513.

In an optional example, steps S504-S507 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by a third training module in the gesture control device operated by the processor.

At step S508, the currently played video image is acquired.

In step S509, the video image is detected by using the pre-trained first convolution network, and the first feature information of the video image and the prediction information of the candidate region of the human hand are obtained.

In step S510, the first feature information and the prediction information of the candidate region are used as the second feature information of the pre-trained second convolutional network model, and the second convolutional network model is used to perform the video image according to the second feature information. Gesture detection, get the gesture detection result of the video image.

Wherein, in the case where it is determined that the human hand is included in the video image after the human hand candidate region detection, the gesture in the human hand candidate region may be determined in a probabilistic manner. For example, taking the palm open gesture and the palm closing gesture as an example, when the probability of the palm open gesture is high, the human hand with the palm open gesture in the video image can be considered, and when the probability of the palm closing gesture is high, the video image can be considered. The hand that contains the palm closing gesture.

Furthermore, in an optional implementation manner of the embodiment of the present application, the output result of the second convolutional network model model may include: a probability that the human hand candidate region does not include a human hand, and a probability that the human hand candidate region includes a palm open gesture. The candidate area of the human hand includes the probability of a human hand closing the gesture of the palm, and the like.

In order to improve the detection speed, in a case where the first convolutional layer parameter is consistent with the second convolutional layer parameter, the second convolutional network model model obtains a gesture for the video image according to the characteristics of the human hand candidate region and various predetermined gestures. When detecting the result, the second convolutional network model model may directly determine the first feature of the plurality of first convolutional layer extracted video images as the second feature of the plurality of second convolutional layer extracted human hand candidate regions, and then According to the second feature described above, the human hand candidate region is classified by a plurality of fully connected layers, and the gesture detection result for the video image is obtained, thereby saving the calculation amount and improving the detection speed.

In an optional example, steps S508-S510 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be The gesture detection module 601 running by the processor executes.

In step S511, when it is determined that the gesture of the detected hand matches the corresponding predetermined gesture, the feature point of the hand in the human hand candidate region corresponding to the detected gesture is extracted.

In step S512, a third convolutional network model for determining a presentation position of the business object in the video image is used according to the feature point of the hand, and the to-be-displayed corresponding to the hand position is determined in the video image. The location of the business object.

In an alternative example, steps S511-S512 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by a presentation location determining module 602 executed by the processor.

In step S513, the business object to be displayed is drawn in a computer drawing manner at the presentation position.

In an alternative example, step S460 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a business object rendering module 603 executed by the processor.

With the rise of Internet live broadcast and short video sharing, more and more videos appear as live or short video. Such videos are often dominated by characters (single characters or a small number of characters), with characters and simple backgrounds as the main scenes, and viewers mainly watch on mobile terminals such as mobile phones. In this case, for the delivery of certain business objects (such as advertisement delivery), the video image during the video playback process can be detected in real time through the solution provided in this embodiment, and the advertisement placement position with better effect is given. And does not affect the user's viewing experience, the delivery effect is better; the computer object is used to draw the business object to be displayed in the presentation position, and the business object is combined with the video playback, and no additional advertising video data irrelevant to the video is transmitted through the network. It is beneficial to save network resources and/or system resources of the client; in addition, the business object is closely combined with the gestures in the video image to preserve the main image and motion of the video subject (such as the anchor) in the video image, and can also add video images. The fun can also avoid disturbing the user to watch the video normally, which is beneficial to reducing the user's dislike of displaying the business object in the video image, and can attract the attention of the viewer to a certain extent and improve the influence of the business object. It can be understood that, in addition to advertising, business objects can be widely applied to other aspects, such as education, consulting, services, etc., by providing entertainment, appreciation and other business information to improve interaction and improve user experience.

Any of the image processing methods provided by the embodiments of the present application may be performed by any suitable device having data processing capabilities, including but not limited to: a terminal device, a server, and the like. Alternatively, any image processing method provided by the embodiment of the present application may be executed by a processor, such as the processor, by executing a corresponding instruction stored in the memory to execute any one of the image processing methods mentioned in the embodiments of the present application. This will not be repeated below.

One of ordinary skill in the art can understand that all or part of the steps of implementing the above image processing method may be completed by hardware related to program instructions, and the foregoing program may be stored in a computer readable storage medium. The steps of the foregoing method embodiments are performed; and the foregoing storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.

FIG. 6 is a structural block diagram of an embodiment of a gesture control apparatus of the present application. The gesture control apparatus of the embodiments of the present application can be used to implement the foregoing gesture control method embodiments of the present application. Referring to FIG. 6, the gesture control apparatus of this embodiment includes a gesture detection module 601, a presentation location determining module 602, and a business object rendering module 603.

The gesture detection module 601 is configured to perform gesture detection on the currently played video image.

The presentation location determining module 602 is configured to determine a presentation location of the business object to be displayed in the video image when the gesture is detected to match the predetermined gesture.

The business object drawing module 603 is configured to draw a business object by using a computer drawing manner at the presentation position.

The gesture control apparatus provided in this embodiment performs human hand candidate area and gesture detection on the currently played video image containing the hand information, and matches the detected gesture with the corresponding predetermined gesture, when the two match, Determining the presentation position of the business object to be displayed in the video image by using the location of the hand. When the business object is used for displaying the advertisement, on the one hand, the business object to be displayed is drawn by using a computer drawing manner at the determined presentation position, and the business object is The combination of video playback eliminates the need to transmit additional advertising video data unrelated to the video over the network, which is conducive to saving network resources and/or system resources of the client; on the other hand, the business object is closely combined with the gestures in the video image to retain the video. The main image and motion of the video main body (such as the anchor) in the image can add interest to the video image, and can also avoid disturbing the user to watch the video normally, which is beneficial to reducing the user's dislike of the business object displayed in the video image, and can be certain To attract the attention of the audience and improve the business objects Influence.

FIG. 7 is a structural block diagram of another embodiment of the gesture control apparatus of the present application. As shown in FIG. 7 , in comparison with the embodiment shown in FIG. 6 , in the gesture control apparatus of the present embodiment, the presentation position determining module 602 includes: a feature point extraction unit, configured to extract a human hand candidate region corresponding to the detected gesture. a feature point of the inner hand; a presentation position determining unit, configured to determine, according to the feature point of the hand, a display position of the business object to be displayed corresponding to the detected gesture in the video image.

Optionally, the presentation location determining module 602 is configured to determine, according to the feature point of the hand and the type of the business object to be displayed, a presentation location of the business object to be displayed corresponding to the detected gesture in the video image.

Optionally, the presentation location determining module 602 is configured to determine, according to the feature point of the hand and the type of the business object to be displayed, a plurality of presentation locations of the business object to be displayed corresponding to the detected gesture in the video image; Select at least one exhibition from multiple presentation locations Current location.

Optionally, the presentation location determining module 602 is configured to determine, when the determined gesture is matched with the corresponding predetermined gesture, a presentation position of the business object to be displayed corresponding to the predetermined gesture in the video image as the detected The position of the corresponding business object to be displayed in the video image.

Optionally, the presentation location determining module 602 is configured to obtain, from a correspondence between the pre-stored gesture and the presentation location, a target presentation location corresponding to the predetermined gesture as a business object to be displayed corresponding to the detected gesture in the video image. Show position.

Optionally, the business object may include: a video, an image, or an effect including semantic information, and the video image may include a still image or a live video image.

Optionally, the special effect including the semantic information includes any one or more of the following special effects including the advertisement information: two-dimensional sticker special effects, three-dimensional special effects, particle special effects, and the like.

Optionally, the presentation location may include any one or more of the following regions: a hair region of the person in the video image, a forehead region, a cheek region, a chin region, a body region other than the head, a background region in the video image, and a video image An area within the setting range centering on the area where the hand is located, a predetermined area in the video image, and the like.

Optionally, the type of the business object may include any one or more of the following types: forehead patch type, cheek patch type, chin patch type, virtual hat type, virtual clothing type, virtual makeup type, virtual headdress type, virtual Hair accessory type, virtual jewelry type.

Optionally, the gesture or the predetermined gesture may include any one or more of: wave, scissors, fist, hand, heart, applause, palm open, palm closed, thumbs up, pistol pose, pendulum V Hand and pendulum OK hand.

Optionally, the gesture detection module 601 is configured to detect a video image by using a pre-trained first convolution network, and obtain first feature information of the video image and prediction information of the candidate region of the human hand, where the first feature information includes hand feature information. Taking the first feature information and the prediction information of the candidate region as the second feature information of the pre-trained second convolutional network model, and using the second convolutional network model to perform the gesture detection of the video image according to the second feature information, Obtaining a gesture detection result of the video image; wherein the second convolutional network model and the first convolutional network model share the feature extraction layer.

Optionally, referring to FIG. 7, in another embodiment of the gesture control apparatus of the present application, the method further includes: a human hand area determining module 604, configured to train the first convolutional network model according to the sample image that includes the human hand annotation information, Obtaining prediction information of the first convolutional network model for the candidate region of the sample image; the correction module 605, for correcting the prediction information of the candidate region of the human hand; and the convolution model training module 606, for predicting the candidate region according to the revised candidate The information and the sample image train the second convolutional network model, wherein the second convolutional network model and the first convolutional network model share the feature extraction layer, and the parameters of the feature extraction layer are maintained during the training process of the second convolutional network model change.

Alternatively, optionally, in another embodiment of the embodiment shown in FIG. 6 of the present application, the method further includes: a human hand area determining module 604, configured to train the first convolution according to the sample image containing the human hand annotation information. a neural network, the prediction information of the first convolutional neural network for the human hand candidate region of the sample image is obtained; the parameter replacement module is configured to replace the second feature extraction layer parameter of the second convolutional neural network for detecting the gesture with the training a first feature extraction layer parameter of the first first convolutional neural network; a second training module, configured to train the second convolutional neural network parameter according to the prediction information of the candidate region of the human hand and the sample image, and maintain the second during the training process The feature extraction layer parameters are unchanged. Optionally, the second training module may include: a correction module 605, configured to correct prediction information of the candidate region of the human hand; and a convolution model training module 606, configured to train the second information according to the predicted information and the sample image of the corrected candidate region The neural network parameters are convolved and the second feature extraction layer parameters are kept unchanged during the training process.

Optionally, the human hand annotation information may include annotation information of the human hand area and/or annotation information of the gesture.

Optionally, the first convolutional neural network may include: a first input layer, a first feature extraction layer, and a first classification output layer, where the first classification output layer is used to predict whether multiple candidate regions of the sample image are candidate regions of the human hand. .

Optionally, the second convolutional neural network may include: a second input layer, a second feature extraction layer, and a second classification output layer, where the second classification output layer is configured to output a gesture detection result of the sample image.

Optionally, the correction module 605 is configured to input the plurality of supplementary negative sample images and the prediction information of the candidate region of the human hand into the third convolutional neural network for classification, to filter the negative samples in the candidate region of the human hand, and obtain the corrected human hand. Prediction information for candidate regions.

Optionally, the difference between the number of the human hand candidate regions and the number of supplementary negative sample images in the prediction information of the human hand candidate region is within a predetermined allowable range. For example, the number of human hand candidate regions in the prediction information of the human hand candidate region is equal to the number of supplementary negative sample images.

Alternatively, the first convolutional neural network may comprise an RPN; the second convolutional neural network may comprise an FRCNN.

Alternatively, the third convolutional neural network may comprise an FRCNN.

Optionally, the presentation location determining module 602 is configured to determine, by using a gesture and a pre-trained third convolutional network model for detecting a presentation location of the business object from the video image, to be displayed corresponding to the detected gesture. The presentation location of the business object.

FIG. 8 is a structural block diagram of still another embodiment of a processing apparatus for a video image of the present application. The embodiments of the present application do not limit the specific implementation of the electronic device. As shown in FIG. 8, the electronic device may include: a processor 802, a communication interface (Communications) Interface 804, memory 806, and communication bus 808. among them:

Processor 802, communication interface 804, and memory 806 complete communication with one another via communication bus 808.

The communication interface 804 is configured to communicate with network elements of other devices, such as other clients or servers.

The processor 802 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, or a graphics processor ( Graphics Processing Unit, GPU). The one or more processors included in the terminal device may be the same type of processor, such as one or more CPUs, or one or more GPUs; or may be different types of processors, such as one or more CPUs and One or more GPUs.

The memory 806 is for at least one executable instruction that causes the processor 802 to perform operations corresponding to a method of presenting a business object in a video image as in any of the above-described embodiments of the present application. The memory 806 may include a high speed random access memory (RAM), and may also include a non-volatile memory such as at least one disk memory.

FIG. 9 is a schematic structural diagram of an embodiment of an electronic device according to the present application. Referring to FIG. 9, there is shown a schematic structural diagram of an electronic device suitable for implementing a terminal device or a server of an embodiment of the present application. As shown in FIG. 9, the electronic device includes one or more processors, a communication unit, etc., such as one or more central processing units (CPUs) 901, and/or one or more A graphics processor (GPU) 913 or the like, the processor may execute various types according to executable instructions stored in read only memory (ROM) 902 or executable instructions loaded from random access memory (RAM) 903 from storage portion 908. Proper action and handling. Communication portion 912 can include, but is not limited to, a network card, which can include, but is not limited to, an IB (Infiniband) network card, and the processor can communicate with read only memory 902 and/or random access memory 903 to execute executable instructions over bus 904. The operation is performed by the communication unit 912, and communicates with other target devices via the communication unit 912, thereby performing operations corresponding to any of the gesture control methods provided by the embodiments of the present application, for example, performing gesture detection on the currently played video image; Determining a presentation location of the business object to be displayed in the video image when the predetermined gesture is matched; drawing the business object in a computer drawing manner at the presentation location.

Further, in the RAM 903, various programs and data required for the operation of the device can be stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other by a bus 904. In the case of RAM 903, ROM 902 is an optional module. The RAM 903 stores executable instructions or writes executable instructions to the ROM 902 at runtime, the executable instructions causing the processor 901 to perform operations corresponding to the gesture control method described above. An input/output (I/O) interface 905 is also coupled to bus 904. The communication unit 912 may be integrated or may be provided with a plurality of sub-modules (for example, a plurality of IB network cards) and on the bus link.

The following components are connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a storage portion 908 including a hard disk or the like. And a communication portion 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the Internet. The drive 911 is also connected to the I/O interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like is mounted on the drive 911 as needed so that a computer program read therefrom is installed into the storage portion 908 as needed.

It should be noted that the architecture shown in FIG. 9 is only an optional implementation manner. In a specific implementation process, the number and type of components in FIG. 9 may be selected, deleted, added, or replaced according to actual needs; Different function components can also be implemented in separate settings or integrated settings, such as GPU and CPU detachable settings or GPU can be integrated on the CPU, the communication part can be separated, or integrated on the CPU or GPU. and many more. These alternative embodiments are all within the scope of protection disclosed herein.

In particular, the processes described above with reference to the flowcharts may be implemented as a computer software program in accordance with an embodiment of the present disclosure. For example, an embodiment of the present disclosure includes a computer program product comprising a computer program tangibly embodied on a machine readable medium, the computer program comprising program code for executing the method illustrated in the flowchart, the program code comprising Executing an instruction corresponding to the method step provided by the embodiment of the present application, for example, an instruction for performing gesture detection on the currently played video image; and determining that the business object to be displayed is in the video image when detecting that the gesture matches the predetermined gesture An instruction to present a location; an instruction to draw the business object in a computer drawing manner at the presentation location.

In addition, the embodiment of the present application further provides a computer program, the computer program comprising computer readable code, the program code includes computer operating instructions, when the computer readable code is run on the device, the processor in the device executes An instruction for implementing each step in the gesture control method of any of the embodiments of the present application.

According to an embodiment of the present application, the process described above with reference to the flowcharts may be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product comprising a computer program tangibly embodied on a machine readable medium, the computer program comprising program code for executing the method illustrated in the flowchart, the program code comprising Performing the instruction corresponding to the method step provided by the embodiment of the present application, for example, performing gesture detection on the currently played video image; determining, when the gesture is matched with the predetermined gesture, determining the presentation position of the business object to be displayed in the video image Drawing the business object in a computer drawing manner at the presentation position.

In addition, the embodiment of the present application further provides a computer readable storage medium for storing computer readable instructions, which are executed to implement the operations of the steps in the gesture control method of any embodiment of the present application.

For the specific implementation of the steps of the computer program and the computer readable command, the corresponding steps in the above-mentioned embodiments and corresponding descriptions in the modules are not described herein. A person skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the device and the module described above may be referred to the corresponding process description in the foregoing method embodiment, and details are not described herein again.

The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same or similar parts between the various embodiments may be referred to each other. For the embodiments, such as the device, the device, the program, the storage medium, and the like, since the method basically corresponds to the method embodiment, the description is relatively simple. For related parts, refer to the description of the method embodiment.

The methods and apparatus of the present application may be implemented in a number of ways. For example, the methods and apparatus of the present application can be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The various components/steps described in the embodiments of the present application may be split into more components/steps according to the needs of the implementation, or two or more components/steps or partial operations of the components/steps may be combined into new components. /Steps to achieve the objectives of the embodiments of the present application. The above-described sequence of steps for the method is for illustrative purposes only, and the steps of the method of the present application are not limited to the order specifically described above unless otherwise specifically stated. Moreover, in some embodiments, the present application can also be implemented as a program recorded in a recording medium, the programs including machine readable instructions for implementing the method according to the present application. Thus, the present application also covers a recording medium storing a program for executing the method according to the present application.

It should be pointed out that the various steps/components described in the present application can be split into more steps/components according to the needs of the implementation, and two or more steps/components or partial operations of the steps/components can be combined into new ones. Steps/components to achieve the objectives of the present application.

The above method according to the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk, or a magneto-optical disk, or can be downloaded through a network. The computer code originally stored in a remote recording medium or non-transitory machine readable medium and to be stored in a local recording medium, whereby the methods described herein can be stored using a general purpose computer, a dedicated processor, or programmable or dedicated Such software processing on a recording medium of hardware such as an ASIC or an FPGA. It will be understood that a computer, processor, microprocessor controller or programmable hardware includes storage components (eg, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, when the software or computer code is The processing methods described herein are implemented when the processor or hardware is accessed and executed. Moreover, when a general purpose computer accesses code for implementing the processing shown herein, the execution of the code converts the general purpose computer into a special purpose computer for performing the processing shown herein.

The foregoing is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto, and any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application. It should be covered by the scope of protection of this application. Therefore, the scope of protection of the present application should be determined by the scope of the claims.

Claims (50)

A gesture control method, comprising: Performing gesture detection on the currently played video image; Determining a presentation position of the business object to be displayed in the video image when detecting that the gesture matches the predetermined gesture; The business object is drawn in a computer drawing manner at the presentation position. The method according to claim 1, wherein the determining a presentation location of the business object to be displayed in the video image comprises: Extracting feature points of the hand in the candidate region of the human hand corresponding to the detected gesture; Determining a presentation position of the business object to be displayed in the video image corresponding to the detected gesture according to the feature point of the hand. The method according to claim 2, wherein the determining, according to the feature point of the hand, the presentation position of the business object to be displayed corresponding to the detected gesture in the video image, including : And determining, according to the feature point of the hand and the type of the business object to be displayed, a display position of the business object to be displayed corresponding to the detected gesture in the video image. The method according to claim 3, wherein the determining the service to be displayed corresponding to the detected gesture according to the feature point of the hand and the type of the business object to be displayed The presentation position of the object in the video image includes: Determining, according to the feature point of the hand and the type of the business object to be displayed, a plurality of presentation positions of the business object to be displayed corresponding to the detected gesture in the video image; Selecting at least one presentation location from the plurality of presentation locations as a presentation location of the business object to be displayed in the video image. The method according to claim 1, wherein the determining a presentation location of the business object to be displayed in the video image comprises: And a target presentation position corresponding to the predetermined gesture is obtained as a presentation position of the business object to be displayed corresponding to the detected gesture in the video image, from a correspondence between the pre-stored gesture and the presentation position. The method according to claim 1, wherein the business object comprises: an effect comprising semantic information; and the video image comprises: a live video image. The method according to claim 6, wherein the special effect including the semantic information comprises at least one of the following special effects including the advertisement information: a two-dimensional sticker effect, a three-dimensional special effect, and a particle special effect. The method according to any one of claims 1-7, wherein the presentation position comprises any one or more of the following: a hair area of a person in a video image, a forehead area, a cheek area, a chin area, and a head. The body area, the background area in the video image, the area within the set range of the video image centered on the area where the hand is located, and the area preset in the video image. The method according to any one of claims 1-8, wherein the type of the business object comprises any one or more of the following: a forehead patch type, a cheek patch type, a chin patch type, and a virtual hat type. , virtual clothing type, virtual makeup type, virtual headwear type, virtual hair accessory type, virtual jewelry type. The method according to any one of claims 1-9, wherein the predetermined gesture comprises any one or more of the following: wave, scissors, fist, hand, heart, applause, palm open, palm Close, thumbs up, pistol pose, swing V-hand and swing OK hand. The method according to any one of claims 1-10, wherein the performing gesture detection on the currently played video image comprises: Detecting the video image by using a pre-trained first convolution network to obtain first feature information of the video image and prediction information of a candidate region of the hand, the first feature information including hand feature information; Using the first feature information and the prediction information of the human hand candidate region as a pre-trained second convolutional network model Second feature information, and performing gesture detection of the video image according to the second feature information by using the second convolutional network model to obtain a gesture detection result of the video image; wherein the second convolution The network model and the first convolutional network model share a feature extraction layer. The method according to claim 11, wherein before the performing gesture detection on the currently played video image, the method further comprises: Training the first convolutional network model according to the sample image containing the human hand annotation information, and obtaining prediction information of the first convolutional network model for the human hand candidate region of the sample image; Correcting prediction information of the candidate region of the human hand; Training a second convolutional network model according to the corrected prediction information of the human hand candidate region and the sample image, wherein the second convolutional network model and the first convolutional network model share a feature extraction layer, and The parameters of the feature extraction layer are kept unchanged during the training of the second convolutional network model. The method of claim 11 further comprising: Training the first convolutional neural network according to the sample image containing the human hand annotation information, and obtaining prediction information of the first hand convolutional neural network for the human hand candidate region of the sample image; Substituting the second feature extraction layer parameter of the second convolutional neural network for detecting the gesture with the first feature extraction layer parameter of the trained first convolutional neural network; And training the second convolutional neural network parameter according to the prediction information of the human hand candidate region and the sample image, and maintaining the second feature extraction layer parameter unchanged during training. The method according to claim 13, wherein the training the second convolutional neural network parameter according to the prediction result of the human hand candidate region and the sample image comprises: Correcting prediction information of the candidate region of the human hand; And clamping the second convolutional neural network parameter according to the corrected prediction information of the human hand candidate region and the sample image. The method according to any one of claims 12-14, wherein the human hand annotation information comprises annotation information of a human hand area and/or annotation information of a gesture. The method according to any one of claims 12-15, wherein the first convolutional neural network comprises: a first input layer, a first feature extraction layer and a first classification output layer, the first classification output The layer is configured to predict whether the plurality of candidate regions of the sample image are human hand candidate regions. The method according to any one of claims 12-16, wherein the second convolutional neural network comprises: a second input layer, a second feature extraction layer, a second classification output layer, and the second classification output The layer is used to output a gesture detection result of the sample image. The method according to any one of claims 12 and 14-17, wherein the correcting the prediction information of the candidate region of the human hand comprises: And inputting the plurality of supplementary negative sample images and the prediction information of the human hand candidate region into the third convolutional neural network to perform classification, so as to filter the negative samples in the human hand candidate region, and obtain the corrected prediction of the human hand candidate region. information. The method according to claim 18, wherein the difference between the number of human hand candidate regions and the number of the supplementary negative sample images in the prediction information of the human hand candidate region is within a predetermined allowable range. The method according to claim 19, wherein the number of human hand candidate regions in the prediction information of the human hand candidate region is equal to the number of the supplementary negative sample images. The method according to any one of claims 12-20, wherein the first convolutional neural network comprises: a regional solution network RPN, and/or the second convolutional neural network comprises: fast regional convolution Neural network FRCNN. The method of any of claims 18-21, wherein the third convolutional neural network comprises an FRCNN. The method according to any one of claims 1 to 22, wherein the determining a presentation location of the business object to be displayed in the video image comprises: Determining, by the detecting the detected gesture and the pre-trained third convolution network model for detecting a presentation position of the business object from the video image, determining a presentation position of the business object to be displayed corresponding to the detected gesture . A gesture control device, comprising: a gesture detection module, configured to perform gesture detection on a currently played video image; a presentation location determining module, configured to determine a presentation location of the business object to be displayed in the video image when the gesture is detected to match the predetermined gesture; A business object drawing module is configured to draw the business object by using a computer drawing manner at the presentation position. The device according to claim 24, wherein the presentation location determining module comprises: a feature point extracting unit, configured to extract a feature point of the hand in the candidate region of the human hand corresponding to the detected gesture; And a presentation position determining unit, configured to determine, according to the feature point of the hand, a presentation position of the business object to be displayed corresponding to the detected gesture in the video image. The device according to claim 25, wherein the presentation position determining unit is configured to determine, according to the feature point of the hand and the type of the business object to be displayed, corresponding to the detected gesture The presentation location of the business object to be displayed in the video image. The device according to claim 26, wherein the presentation position determining unit is configured to determine, according to the feature point of the hand and the type of the business object to be displayed, corresponding to the detected gesture The plurality of presentation locations of the business object to be displayed in the video image; selecting at least one presentation location from the plurality of presentation locations. The device according to claim 24, wherein the presentation location determining module is configured to acquire, from the correspondence between the pre-stored gesture and the presentation location, a target presentation location corresponding to the predetermined gesture as the detected The gesture corresponding to the presentation location of the business object to be displayed in the video image. The apparatus according to claim 24, wherein the business object comprises: an effect comprising semantic information; and the video image comprises: a live video type image. The apparatus according to claim 29, wherein the special effect including the semantic information comprises an effect of any one or more of the following forms of the advertisement information: a two-dimensional sticker effect, a three-dimensional effect, a particle effect. The device according to any one of claims 24 to 30, wherein the presentation position comprises any one or more of the following: a hair region of a person in the video image, a forehead region, a cheek region, a chin region, and a head portion. The body area, the background area in the video image, the area within the set range of the video image centered on the area where the hand is located, and the area preset in the video image. The device according to any one of claims 24 to 31, wherein the type of the business object comprises any one or more of the following: a forehead patch type, a cheek patch type, a chin patch type, a virtual hat type. , virtual clothing type, virtual makeup type, virtual headwear type, virtual hair accessory type, virtual jewelry type. The device according to any one of claims 24-32, wherein the predetermined gesture comprises any one or more of the following: wave, scissors, fist, hand, heart, applause, palm open, The palms are closed, the thumbs are vertical, the pistol poses, the V-hand and the OK hand are placed. The device according to any one of claims 24 to 33, wherein the gesture detecting module is configured to detect the video image by using a pre-trained first convolution network to obtain first feature information of the video image. And prediction information of the human hand candidate region, the first feature information including hand feature information; using the first feature information and the prediction information of the human hand candidate region as a second feature of the pre-trained second convolutional network model And performing, by the second convolutional network model, the gesture detection of the video image according to the second feature information, to obtain a gesture detection result of the video image; wherein the second convolution network model and the The first convolutional network model shares a feature extraction layer. The device according to claim 34, further comprising: a manual area determining module, configured to train the first convolutional network model according to the sample image containing the human hand annotation information, to obtain prediction information of the first convolutional network model for the human hand candidate area of the sample image; a correction module, configured to correct prediction information of the candidate region of the human hand; a convolution model training module, configured to train according to the corrected prediction information of the candidate region of the human hand and the sample image a second convolutional network model, wherein the second convolutional network model and the first convolutional network model share a feature extraction layer and maintain the feature extraction layer during training of the second convolutional network model The parameters are unchanged. The device according to claim 34, further comprising: a manual area determining module, configured to train the first convolutional neural network according to the sample image containing the human hand annotation information, to obtain prediction information of the first convolutional neural network for the human hand candidate area of the sample image; a parameter replacement module, configured to replace a second feature extraction layer parameter of the second convolutional neural network for detecting a gesture with a first feature extraction layer parameter of the first convolutional neural network after training; And a second training module, configured to train the second convolutional neural network parameter according to the prediction information of the human hand candidate region and the sample image, and keep the second feature extraction layer parameter unchanged during the training process. The device of claim 36, wherein the second training module comprises: a correction module, configured to correct prediction information of the candidate region of the human hand; a convolution model training module, configured to train the second convolutional neural network parameter according to the corrected prediction information of the human hand candidate region and the sample image, and maintain the second feature extraction layer parameter during training constant. The apparatus according to any one of claims 35-37, wherein said human hand annotation information comprises annotation information of a human hand area and/or annotation information of a gesture. The apparatus according to any one of claims 34 to 38, wherein said first convolutional neural network comprises: a first input layer, a first feature extraction layer and a first classification output layer, said first classification output The layer is configured to predict whether the plurality of candidate regions of the sample image are human hand candidate regions. The method according to any one of claims 34 to 39, wherein the second convolutional neural network comprises: a second input layer, a second feature extraction layer, a second classification output layer, and the second classification output The layer is used to output a gesture detection result of the sample image. The apparatus according to any one of claims 35 and 37 to 40, wherein the correction module is configured to input the plurality of supplementary negative sample images and the prediction information of the human hand candidate region into the third convolutional neural network. Classification is performed to filter negative samples in the human hand candidate region to obtain predicted information of the corrected human hand candidate region. The apparatus according to claim 41, wherein a difference between the number of human hand candidate regions and the number of the supplementary negative sample images in the prediction information of the human hand candidate region is within a predetermined allowable range. The apparatus according to claim 42, wherein the number of human hand candidate regions in the prediction information of the human hand candidate region is equal to the number of the supplementary negative sample images. The apparatus according to any one of claims 34-43, wherein said first convolutional neural network comprises: a regional solution network RPN, and/or said second convolutional neural network comprises: fast regional convolution Neural network FRCNN. Apparatus according to any of claims 34-44 wherein said third convolutional neural network comprises an FRCNN. The apparatus according to any one of claims 24 to 45, wherein said presentation position determining module is configured to pass the gesture and a pre-trained third volume for detecting a presentation position of a business object from a video image The product network model determines a presentation location of the business object to be displayed corresponding to the detected gesture. An electronic device comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is for storing at least one executable instruction that causes the processor to perform the operations of the steps of the gesture control method of any of claims 1-23. An electronic device, comprising: a processor and the gesture control device of any of claims 24-46; The unit in the gesture control device of any of claims 24-45 is operated when the processor runs the gesture control device. A computer program comprising computer readable code, wherein a processor in the device performs a gesture for implementing any of claims 1-23 when the computer readable code is run on a device Steps in the control method Command instructions. A computer readable storage medium for storing computer readable instructions, wherein the instructions are executed to perform the operations of the steps of the gesture control method of any of claims 1-23.

Images