WO2018033143A1 - Video image processing method, apparatus and electronic device - Google Patents

Abstract

Provided in the embodiments of the present application are a video image processing method, apparatus and electronic device. The method comprises: performing human face facial expression detection on a video image currently being played back which contains human face information; determining a presentation position of a service object to be presented in the video image when a detected facial expression matches a corresponding predetermined facial expression; and drawing the service object in the presentation position using a computer drawing mode. Using the embodiments of the present application may save network resources and/or system resources of a client, make a video image more interesting, and avoid bothering a user when normally watching a video, thereby reducing the user's feelings of opposition to a service object presented in the video image, while attracting the attention of an audience to a certain extent, and increasing the impact of the service object.

Description

Video image processing method, device and electronic device

This application claims the priority of the Chinese Patent Application filed on August 19, 2016, the Chinese Patent Application No. CN201610697472.6, entitled "Processing Method, Apparatus and Terminal Device for Video Image", the entire contents of which are incorporated by reference. Combined in this application.

Technical field

The present application relates to artificial intelligence technology, and in particular, to a video image processing method, apparatus, and electronic device.

Background technique

With the development of Internet technology, people are increasingly using the Internet to watch video, and Internet video offers business opportunities for many new businesses. Internet video is considered a premium resource for ad placement because it can be an important entry point for business traffic.

Existing video advertisements are mainly inserted into a fixed-time advertisement at a certain time of video playback, or placed in a fixed position in the area where the video is played and its surrounding area.

Summary of the invention

The embodiment of the present application provides a technical solution for video image processing.

According to an aspect of the embodiments of the present application, a method for processing a video image includes: performing facial expression detection of a face on a currently played video image including face information; and detecting a facial expression and a corresponding predetermined face When the expressions match, the presentation position of the business object to be presented in the video image is determined; and the business object is drawn by computer drawing at the presentation position.

According to another aspect of the embodiments of the present application, there is provided a video image detecting module, configured to perform facial expression detection of a face on a currently played video image including face information, and a display position determining module, configured to Determining, when the facial expression detected by the video image detecting module matches the corresponding predetermined facial expression, a presentation position of the business object to be presented in the video image; and a business object drawing module, configured to perform computer drawing at the display position The way to draw the business object.

According to still another aspect of the embodiments of the present application, another electronic device is provided, including:

A processor and a video image processing apparatus according to any of the above embodiments of the present application;

When the processor runs the structured text detection system, the units in the video image processing apparatus of any of the above embodiments of the present application are executed.

According to still another aspect of an embodiment of the present application, a computer program is provided, comprising computer readable code, when a computer readable code is run on a device, a processor in the device performs the above-described An instruction of each step in the method of processing a video image according to an embodiment.

According to still another aspect of the embodiments of the present application, a computer readable storage medium is provided for storing computer readable instructions, when executed, to implement the video image described in any of the above embodiments of the present application. The operation of each step in the processing method. The method, device, and terminal device for processing a video image according to an embodiment of the present application perform facial expression detection on a currently played video image including face information, and match the detected facial expression with a corresponding predetermined facial expression. When the two match, determine the presentation position of the business object to be presented in the video image, and then draw the business object by means of computer drawing at the presentation position, so that when the business object is used for displaying advertisements, on the one hand, The determined display position is drawn by using a computer drawing manner, and the business object is combined with the video playing, and the video data of the business object such as an advertisement, which is not related to the video, is transmitted through the network, thereby saving network resources and/or customers. On the other hand, the business object is closely combined with the facial expressions in the video image to preserve the main image and motion of the video subject (such as the anchor) in the video image, adding interest to the video image, and not disturbing. Users can watch videos normally, which can reduce user's view Disgusted with the business object image in the show, but also the audience's attention to a certain extent, increase the influence of business objects.

The technical solutions of the present application are further described in detail below through the accompanying drawings and embodiments.

DRAWINGS

The accompanying drawings, which are incorporated in FIG.

The present application can be more clearly understood from the following detailed description, in which:

1 is a flow chart showing an embodiment of a method for processing a video image of the present application;

2 is a flow chart showing an embodiment of an acquisition method of a first convolutional network model of the present application;

3 is a schematic structural diagram of an embodiment of a first convolutional network model of the present application;

4 is a flow chart showing another embodiment of a method for processing a video image of the present application;

FIG. 5 is a flow chart showing still another embodiment of a processing method of a video image of the present application; FIG.

6 is a block diagram showing the structure of an embodiment of a processing apparatus for video images of the present application;

7 is a structural block diagram showing another embodiment of a processing apparatus for a video image of the present application;

FIG. 8 is a schematic structural diagram of an embodiment of a terminal device according to the present application;

FIG. 9 is a schematic structural diagram of another embodiment of an electronic device according to the present application.

detailed description

Various exemplary embodiments of the present application will now be described in detail with reference to the drawings. It should be noted that the relative arrangement of the components and steps, numerical expressions and numerical values set forth in the embodiments are not intended to limit the scope of the application.

In the meantime, it should be understood that the dimensions of the various parts shown in the drawings are not drawn in the actual scale relationship for the convenience of the description.

The following description of the at least one exemplary embodiment is merely illustrative and is in no way

Techniques, methods and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but the techniques, methods and apparatus should be considered as part of the specification, where appropriate.

It should be noted that similar reference numerals and letters indicate similar items in the following figures, and therefore, once an item is defined in one figure, it is not required to be further discussed in the subsequent figures.

Embodiments of the present application can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and/or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, and the like include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients Machines, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above, and the like.

Electronic devices such as terminal devices, computer systems, servers, etc., can be described in the general context of computer system executable instructions (such as program modules) being executed by a computer system. Generally, program modules may include routines, programs, target programs, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types. The computer system/server can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communication network. In a distributed cloud computing environment, program modules may be located on a local or remote computing system storage medium including storage devices.

1 is a flow chart of an embodiment of a method for processing a video image of the present application. The method is performed by a computer system including a processing device of a video image. The method of processing a video image of various embodiments of the present application may be exemplarily performed by an electronic device such as a computer system, an electronic device, or a server. Referring to FIG. 1, a method for processing a video image of this embodiment includes:

In step S110, facial expression detection of the face is performed on the currently played video image containing the face information.

In various embodiments of the present application, the face information may include, for example, but is not limited to, information related to the face, eyes, nose, and/or hair. The video image may be an image of a live video that is being broadcast live, or a video image that has been recorded or is in the process of being recorded. Facial expressions include, but are not limited to, happiness, anger, pain, sadness, contemplation, daze, excitement, and the like.

In an optional example of the embodiments of the present application, a live video is taken as an example. Currently, the live video platform includes multiple, such as a pepper live broadcast platform, a YY live broadcast platform, etc., each live broadcast platform includes multiple live broadcast rooms. Each live room will include at least one anchor, and the anchor can broadcast a video to the fans in the live room where the electronic device is located, the live video includes multiple video images. The subject in the above video image is usually a main character (ie, anchor) and a simple background (such as the anchor's home or other video recording venue, etc.), and the anchor often occupies a larger area in the video image. When a business object (such as an advertisement) needs to be inserted in the process of the live video, the video image in the current live video can be obtained, and the video image can be detected by a preset face detection mechanism to determine the Whether the face information of the anchor is included in the video image, if the face information of the anchor is included, the video image is acquired or recorded for subsequent processing; if the face information of the anchor is not included, the video image of the next frame may be continued. The above related processing is performed to obtain a video image in which the video image includes the face information of the anchor.

In addition, the video image may also be a video image in a short video that has been recorded. In this case, the user can play the short video using the electronic device, and during the playing process, the electronic device can detect each frame of the video image. Whether to include the face letter of the anchor If the face information of the anchor is included, the video image is acquired for subsequent processing; if the face information of the anchor is not included, the video image may be discarded or not processed, and the next frame is obtained. The video image continues with the above processing.

In addition, in the case that the video image is a video image being recorded, during the recording process, the user can use his electronic device to detect whether the video image of each frame is included in the recorded video image of the anchor, if the person including the anchor For the face information, the video image is acquired for subsequent processing; if the face information of the anchor is not included, the video image may be discarded or not processed, and the next frame of the video image may be acquired to continue the above processing.

The electronic device that plays the video image or the electronic device used by the anchor is provided with a mechanism for performing facial expression detection on the video image, and the video image of each frame including the face information currently played by the facial expression detection mechanism can be Perform facial expression detection to obtain a facial expression of a face detected from the video image. An optional process may be that the electronic device acquires a video image currently being played, and is detected by a preset facial expression. The mechanism may extract an image including a face region from the video image, and then analyze and extract the image of the face region to obtain feature data of each part (including eyes, mouth, face, etc.) in the face region. By analyzing the feature data, it is determined which facial expression of the face in the video image belongs to happy, angry, painful, sad, contemplative, dazed, excited, and the like.

In an alternative example, step S110 may be performed by the processor invoking a corresponding instruction stored in the memory, or may be performed by video image detection module 601 being executed by the processor.

In step S120, when the detected facial expression matches the corresponding predetermined facial expression, the presentation position of the business object to be presented in the video image is determined.

In various embodiments of the present application, the business object is an object created according to a certain business requirement, such as advertisement, entertainment, weather forecast, traffic forecast, pet, education, consultation, service, and the like. The presentation position may be a center position of a designated area in the video image, or may be a coordinate of a plurality of edge positions in the specified area or the like. In various embodiments of the present application, the business object may be a special effect including semantic information, for example, may include at least one or any of the following special effects including the advertisement information: two-dimensional sticker special effects, three-dimensional special effects, particle special effects, and the like.

In various implementations of the present application, feature data of a plurality of different facial expressions may be stored in advance, and different facial expressions are marked correspondingly to distinguish the meaning represented by each facial expression. The facial expression of the face can be detected from the video image by the processing of the above step S110, and the feature data of the detected facial expression of the face can be compared with the feature data of each facial expression stored in advance, if The feature data of the plurality of different facial expressions stored in advance includes the same feature data as the feature data of the facial expression of the detected face, and then it can be determined that the detected facial expression matches the corresponding predetermined facial expression.

In order to improve the accuracy of the matching, the matching result may be determined by a calculation manner. For example, a matching algorithm may be set to calculate a matching degree between the feature data of any two facial expressions, for example, a facial expression that detects a human face may be used. The feature data is matched with the feature data of any of the pre-stored facial expressions to obtain a matching degree value between the two, and the detected facial expression of the face and each of the pre-stored faces are respectively calculated by the above manner. The matching degree value between the expressions is selected from the obtained matching degree value, and if the maximum matching degree value exceeds the predetermined matching threshold, the pre-stored facial expression corresponding to the largest matching degree value may be determined. Matches the detected facial expressions. If the maximum matching degree value does not exceed the predetermined matching threshold, the matching fails, that is, the detected facial expression is not a predetermined facial expression, and at this time, the processing of the above step S110 may be continued on the subsequent video image.

Optionally, when it is determined that the detected facial expression matches the corresponding predetermined facial expression, the meaning represented by the predetermined facial expression matched by the detected facial expression may be determined first, and may be in a plurality of preset display positions. The presentation position associated with the meaning of the matched predetermined facial expression or the corresponding presentation position is selected as the presentation position of the business object to be presented in the video image. For example, taking a live video as an example, when detecting a happy facial expression of the anchor, the face area or the background area may be selected as a presentation position related to or corresponding to the facial expression of the school.

In an alternative example, step S120 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a presentation location determining module 602 executed by the processor.

In step S130, the business object is drawn by computer drawing at the presentation position.

For example, in the case of a live video, when a happy facial expression is detected, a corresponding business object, such as an image advertisement with a predetermined product identifier, may be drawn by using a computer drawing in the area where the anchor of the video image is located. If the fan is interested in the business object, the area where the business object is located may be clicked on the electronic device, and the electronic device of the fan may obtain the network link corresponding to the business object, and enter the business object through the network link. A page on which you can get resources related to the business object.

In various embodiments of the present application, the business object can be drawn by computer drawing, and the appropriate computer graphics image can be adopted. Implementation by drawing or rendering, for example, may include, but is not limited to, drawing based on an Open Graphics Language (OpenGL) graphics rendering engine, and the like. OpenGL defines a professional graphical program interface for cross-programming language and cross-platform programming interface specifications. It is hardware-independent and can easily draw 2D or 3D graphics images. Through the OpenGL graphics rendering engine, not only can 2D effects be drawn, but also 3D stickers can be drawn, and 3D effects can be drawn and particle effects can be drawn. However, the application is not limited to the drawing method based on the OpenGL graphics rendering engine, and other methods may be adopted. For example, the drawing method based on the graphics engine (Unity) or the Open Computing Language (OpenCL) is also applicable to the present invention. Apply for each embodiment.

In an alternative example, step S130 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a business object rendering module 603 executed by the processor.

The method for processing a video image provided by the embodiment of the present application performs facial expression detection on a currently played video image including face information, and matches the detected facial expression with a corresponding predetermined facial expression, when the two match Determining the presentation position of the business object to be presented in the video image, and then drawing the business object by means of computer drawing at the presentation position, so that when the business object is used for displaying the advertisement, on the one hand, the computer is used at the determined display position. Drawing the business object, the business object is combined with video playing, and does not need to transmit video data of a business object such as an advertisement that is not related to the video through the network, thereby saving network resources and/or system resources of the client; Aspects, the business object is closely combined with the facial expressions in the video image, and can retain the main image and motion of the video subject (such as the anchor) in the video image, adding interest to the video image, and not disturbing the user to watch the video normally, Reduce the number of business pairs that users display on video images Resentment, but also to attract the audience's attention to a certain extent, increase the influence of business objects.

The process of performing face facial expression detection on the video image in the step S110 in the first embodiment may be implemented by using a corresponding feature extraction algorithm or using a neural network model such as a convolutional network model. 2 is a flow chart of an embodiment of a method for acquiring a first convolutional network model of the present application. In this embodiment, a convolutional network model is taken as an example to describe facial expression detection of a face on a video image. To this end, a first convolutional network model for detecting a face attribute in an image may be pre-trained. The method for acquiring the first convolutional network model of this embodiment may be performed by any electronic device having data acquisition, processing, and transmission functions, including, but not limited to, a mobile terminal. Personal computers (PCs), servers, etc., are not limited in this application. In order to train the first convolutional network model, the training sample may be obtained in a plurality of manners, and the training sample may be at least one sample image including face information, and the information of the face attribute is marked in the sample image.

Referring to FIG. 2, the method for obtaining the first convolutional network model of the embodiment includes:

In step S210, at least one sample image including face information is acquired. The sample image is labeled with information about the face attribute.

In various embodiments of the present application, the face attributes may include, for example, local attributes and global attributes, for example, but not limited to: hair color, hair length, eyebrow length, eyebrow thick or sparse, eye size, eyes open or The closure, the height of the bridge of the nose, the size of the mouth, the opening or closing of the mouth, whether or not to wear glasses, whether to wear a mask, etc., global attributes include, for example, but not limited to, race, gender, age, and expression. The sample image in this embodiment may be a plurality of images of video or continuous shooting, or may be any image including an image including a human face and/or an image not including a human face.

In various embodiments of the present application, the larger the resolution of the image, the larger the amount of data. When the face attribute detection is performed, the more computing resources are required, the slower the detection speed. In view of this, in the present application, In a specific implementation manner, the sample image may be an image that satisfies a preset resolution condition. For example, the above preset resolution condition may be: the longest side of the image does not exceed 640 pixels, the shortest side does not exceed 480 pixels, and the like.

The sample image in various embodiments of the present application may be obtained by an image acquisition device, which may be, for example, a dedicated camera or a camera integrated in other devices. However, in actual applications, due to different hardware parameters, different settings, and the like of the image acquisition device, the acquired image may not satisfy the above preset resolution condition, in order to obtain a sample image that satisfies the above preset resolution condition, in the present application In an optional implementation manner, after the image acquisition device acquires the image, the collected image may be scaled to obtain at least one sample image that meets the preset resolution condition.

After obtaining the sample image, the information of the face attribute, such as happiness, pain, sadness, anger, etc., may be marked in each sample image, and the information of the face attribute marked in each sample image and the sample image may be used as training data. storage.

In order to make the detection of the face attribute in the sample image more accurate, the face in the sample image can be positioned to obtain the exact position of the face in the sample image. For details, refer to the process of step S220 described below.

In step S220, for each sample image in the at least one sample image, a face and a face key point in the sample image are detected, and a face in the sample image is located through a face key point to obtain a face face. information.

In each embodiment of the present application, each face has a certain feature point, such as a corner of the eye, an end of the eyebrow, a corner of the mouth, a tip of the nose, and the like, and a boundary point of the face, etc., in which a key point of the face is obtained ( That is, the key feature points), you can calculate the sample through the key points of the face. a mapping of a face in the image to a preset standard face or a similar transformation, aligning the face in the sample image with the standard face, thereby positioning the face in the sample image to obtain a sample image. Positioning information of the face.

In step S230, a sample image containing face positioning information is taken as a training sample.

In order to make the detection result of the output of the first convolutional network model obtained by the training more accurate, the supervision information for training the first convolutional network model may be set in advance. For details, refer to the processing of step S240 described below.

In an alternative example, steps S210-S230 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a training sample acquisition module 604 executed by the processor.

In step S240, an attribute having a size order feature in the face attribute is encoded.

The attributes of the size order feature may be, for example, age, distance between two eyes, and the like.

In the embodiment, the age is taken as an example, and the standard age a is set, and the code may be one of the following forms or a combination thereof:

Form 1: Encoded as x ₁ , x ₂ , ... x _i ..., where x _i is a binary value, and the value is 0 or 1. If the age i is less than or equal to a, the value of x _i is 1, if the age If i is greater than a, the value of x _i is 0.

Form 2: coded as x ₁ , x ₂ ,...x _i ..., where x _i is a binary value, and the value is 0 or 1. If the age i is equal to a divided by k, then x _{i has} a value of 1, otherwise, the value of _i x is 0. Where k can be a positive integer of any value, and its value can be manually defined or randomly selected.

In an alternative example, step S240 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by an encoding module 605 executed by the processor.

In step S250, the coded attribute is used as the supervision information of the training first convolutional network model, and the first convolutional network model is trained using the training samples to obtain a first convolutional network model for detecting the face attribute in the image. .

In various embodiments of the present application, the front end of the first convolutional network model may include a combination of multiple convolutional layers, pooling layers, and non-linear layers, and the back end may be a loss layer, such as based on a cost function (softmax) and / or loss layer of an algorithm such as cross entropy.

As an optional example of various embodiments of the present application, an optional structure of the first convolutional network model is shown in FIG. 3, wherein:

A is an input layer for reading sample images, face attributes, and encoding of partial face attributes. The input layer may preprocess the sample image, and output a face image including positioning information, information of a face attribute, or a code of a partial face attribute. The input layer outputs the preprocessed face image to the convolution layer, and inputs the information of the preprocessed face attribute and/or the code of the partial face attribute to the loss layer.

The B layer is a convolution layer, and the input of the convolution layer is a pre-processed face image or image feature, and the feature of the face image is obtained by a predetermined linear transformation output.

The C layer is a nonlinear layer, and the nonlinear layer can nonlinearly transform the characteristics of the input of the convolution layer, so that the characteristics of the output have strong expression ability.

D is a pooling layer, which can map multiple values to a value. Therefore, the pooling layer can enhance the nonlinearity of the learned features, and can also make the spatial size of the output features smaller. Enhance the translation (ie, face translation) invariance of the learned features, keeping the extracted features unchanged. Wherein, the output feature of the pooling layer can be used again as the input data of the convolution layer or the input data of the fully connected layer.

As shown in FIG. 3, the outermost rectangular frame of the convolutional layer, the nonlinear layer, and the pooling layer indicates that the convolutional layer, the nonlinear layer, and the pooled layer may be repeated one or more times, that is, a convolutional layer or a nonlinear layer. The combination with the pooling layer may be repeated one or more times, wherein the output data of each pooling layer may be used as the re-input data of the convolution layer. Multiple combinations of the convolutional layer, the nonlinear layer and the pooled layer can better process the input sample image, so that the features in the sample image have better expression ability.

The E layer is a fully connected layer that linearly transforms the input data of the pooled layer and projects the learned features into a better subspace to facilitate property prediction.

The F layer is a nonlinear layer, and the nonlinear layer functions as a nonlinear layer, and the input characteristics of the fully connected layer are nonlinearly transformed. Its output characteristics can be used as input data for the loss layer or as input data for the fully connected layer again.

As shown in FIG. 3, the outermost rectangular frame of the fully connected layer and the nonlinear layer indicates that the fully connected layer and the nonlinear layer may be repeated one or more times.

The G layer is one or more loss layers that are primarily responsible for calculating the information of the predicted face attributes and/or the errors of the information and/or coding of the encoded face attributes.

The network parameters in the first convolutional network model can be trained by the gradient descent algorithm passed backwards, which can make the input layer input only the image, and can output the face attribute corresponding to the face in the input image. The information thus leads to the first convolutional network model.

Through the above process, the input layer is responsible for simply processing the input, and the combination of the convolutional layer, the nonlinear layer and the pooling layer is responsible for the feature extraction of the sample image, the information extracted from the fully connected layer and the nonlinear layer to the face attribute information and/or Or the mapped map, the loss layer is responsible for calculating the prediction error. Through the multi-layer design of the first convolutional network model described above, the extracted features can be richly expressed, thereby better predicting the face attributes. In addition, the information and coding of multiple face attributes are connected to the loss layer at the same time, which ensures that multiple tasks learn at the same time and share the features learned by the convolutional network.

In an alternative example, step S250 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by a first convolutional network model determination module 606 executed by the processor.

In this embodiment, the first convolutional network model obtained by the training can facilitate subsequent facial expression detection on the currently played video image containing the face information, and match the detected facial expression with the corresponding predetermined facial expression. When the two match, determine the presentation position of the business object to be presented in the video image, and then draw the business object by means of computer drawing at the presentation position, so that when the business object is used for displaying advertisements, on the one hand, The determined display position is drawn by using a computer drawing manner, and the business object is combined with the video playing, and the video data of the business object such as an advertisement, which is not related to the video, is transmitted through the network, thereby saving network resources and/or customers. On the other hand, the business object is closely combined with the facial expressions in the video image to preserve the main image and motion of the video subject (such as the anchor) in the video image, adding interest to the video image, and not disturbing. The user can watch the video normally, which can reduce the user's view on the video image. Now objectionable business objects, but also to attract the audience's attention to a certain extent, increase the influence of business objects.

4 is a flow chart of another embodiment of a method for processing a video image of the present application. The video image in the embodiment of the present application may be a video image in a live video, such as a video image when an anchor of a pepper live broadcast platform performs live video broadcast. Referring to FIG. 4, a method for processing a video image of this embodiment includes:

In step S410, the currently played video image containing the face information is acquired.

For the specific processing of the foregoing step S410, refer to the related content in step S110 in the foregoing embodiment shown in FIG. 1 , and details are not described herein again.

In step S420, based on the face information in the video image, the pre-trained first convolutional network model for detecting the face attribute in the image is used to perform facial expression detection of the face on the video image.

In this embodiment, the acquired video image including the face information may be input into the first convolutional network model trained in the foregoing embodiment shown in FIG. 2, and the video may be used by the first convolutional network model. The image performs processing such as pre-processing such as scaling, feature extraction, mapping, and transformation to perform facial expression detection on the face of the video image to obtain a facial expression of the face included in the video image.

In an alternative example, steps S410-S420 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by a video image detection module 601 that is executed by the processor.

In step S430, when the detected facial expression matches the corresponding predetermined facial expression, the feature point of the face attribute in the face region corresponding to the detected facial expression is extracted.

In various embodiments of the present application, for a video image including face information, a certain feature point such as an eye, a nose, a mouth, a facial contour, and the like may be included in the face. The detection of the face in the video image and the determination of the feature point can be implemented in any suitable related art, which is not limited in the embodiment of the present application. For example, linear feature extraction methods such as principal component analysis (PCA), linear discriminant analysis (LDA), independent component analysis (ICA), etc.; for example, nonlinear feature extraction methods such as kernel principal component analysis (Kernel PCA), manifolds The learning and the like can also be performed by using the trained neural network model, such as the convolutional network model in the embodiment of the present application, for the feature point extraction of the face attribute.

Taking a live video as an example, in the process of video live broadcast, the face is detected from the video image of the live video and the feature points of the face attribute are determined; for example, during the playback of a recorded video. , detecting a face from the played video image and determining a feature point of the face attribute; for example, detecting a face from the recorded video image and determining a feature point of the face attribute during recording of a certain video, etc. .

In step S440, the presentation position of the business object to be presented in the video image is determined according to the feature point of the face attribute.

In the implementation, after the feature points of the face attribute are determined, the one or more presentation positions of the business object to be displayed in the video image may be determined based on the basis.

In this embodiment, when determining the presentation position of the business object to be displayed in the video image according to the feature point of the target object, an optional implementation manner may include, but is not limited to:

In the first method, according to the feature point of the face attribute, the second convolutional network model for determining the presentation position of the business object in the video image is used to determine the presentation position of the business object to be presented in the video image. The second method determines the presentation position of the business object to be presented in the video image according to the feature point of the face attribute and the type of the business object to be presented.

Hereinafter, the above two modes will be exemplarily described.

method one

When the usage mode 1 determines the presentation position of the business object to be displayed in the video image, a convolutional network model is pre-trained, that is, the second convolutional network model, and the trained second convolutional network model has the determined service object in the The function of presenting the position in the video image; alternatively, a convolutional network model that has been trained by a third party to have the function of determining the presentation position of the business object in the video image can also be used directly.

It should be noted that, in this embodiment, the training of the business object is described, but those skilled in the art should understand that the second convolutional network model can also train the face when training the business object. Joint training of faces and business objects.

When pre-training the second convolutional network model, an optional training method includes the following process:

(1) Obtaining a feature vector of a sample image of the training sample.

The feature vector includes position information and/or confidence information of the business object in the sample image of the training sample, and a face feature vector corresponding to the face attribute in the sample image. The confidence information of the business object indicates the probability that the business object can achieve the effect (such as being focused or clicked or viewed) when the current location is displayed. The probability may be set according to the statistical analysis result of the historical data, or may be According to the results of the simulation experiment, it can also be set according to the artificial experience. In practical applications, only the location information of the business object may be trained according to actual needs, or only the confidence information of the business object may be trained, and the location information and or the confidence information of the business object may be trained. The location information and or the confidence information of the business object are trained, so that the trained second convolutional network model can more effectively and accurately determine the location information and the confidence information of the business object, so as to provide the display of the business object. in accordance with.

The second convolutional network model is trained by a large number of sample images. In the embodiment of the present application, the sample image of the training sample may be at least one sample image including the face information in the embodiment shown in FIG. 2, which may be used. The business object sample image with the business object is trained on the second convolutional network model. It should be understood by those skilled in the art that the business object sample image used for training may include face information in addition to the business object. In addition, the business object in the business object sample image in the embodiment of the present application may be marked with pre-labeled location information, or confidence information, or location information and or confidence information. Of course, in practical applications, this information can also be obtained through other means. By marking the corresponding information on the business object in advance, the data processing data and the number of interactions can be effectively saved, and the data processing efficiency is improved.

The location information and/or confidence information of the business object and the sample image of a certain face attribute are used as training samples, and the feature vector is extracted to obtain the feature information including the location information and/or the confidence information of the business object. The vector, as well as the face feature vector corresponding to the face attribute.

Optionally, the second convolutional network model can be used to simultaneously train the face and the business object. In this case, the feature vector of the sample image also includes the features of the face.

The extraction of the feature vector can be implemented in an appropriate manner in the related art, and details are not described herein again.

(2) Convolution processing of the feature vector to obtain the feature vector convolution result.

In this embodiment, the obtained feature vector convolution result includes the location information and/or the confidence information of the service object, and the feature vector convolution result corresponding to the face feature vector corresponding to the face attribute. In the case of joint training of faces and business objects, the feature vector convolution result may also include face information.

The number of times of convolution processing on the feature vector can be set according to actual needs, that is, in the second convolutional network model, the number of layers of the convolution layer can be set according to actual needs, and will not be described here.

The convolution result is the result of feature extraction of the feature vector, which can effectively represent the business object corresponding to the feature of the face in the video image.

In the embodiment of the present application, when the feature vector includes both the location information of the service object and the confidence information of the service object, that is, when the location information and the confidence information of the service object are trained, The eigenvector convolution result is shared in the subsequent judgment of the convergence condition, and no need to perform repeated processing and calculation, which is beneficial to reduce resource loss caused by data processing, and improve data processing speed and efficiency.

(3) determining whether the location information and/or the confidence information of the corresponding service object in the feature vector convolution result satisfies the convergence condition of the service object, and determining whether the corresponding face feature vector in the feature vector convolution result satisfies the face convergence condition .

The convergence condition can be appropriately set by a person skilled in the art according to actual needs. When the information satisfies the convergence condition, it can be considered that the network parameters in the second convolutional network model are properly set; when the information cannot satisfy the convergence condition, it can be considered that the network parameters in the second convolutional network model are not properly set and need to be performed. Adjustment, the adjustment may be an iterative process until the result of convolution processing the feature vector using the adjusted network parameters satisfies the convergence condition.

In an optional manner, the convergence condition may be set according to a preset standard location and/or a preset standard confidence, for example, a location indicated by the location information of the service object in the feature vector convolution result and a preset The distance between the standard positions satisfies a certain threshold as a convergence condition of the location information of the service object; the difference between the confidence level indicated by the confidence information of the service object in the feature vector convolution result and the preset standard confidence satisfies a certain threshold The convergence condition of the confidence information as a business object, and the like.

Optionally, the preset standard location may be an average location obtained by averaging the location of the service object in the sample image of the training sample; the preset standard confidence may be in the sample image of the training sample. The confidence level of the business object is averaged after the average processing. Since the sample image is a sample to be trained and the amount of data is large, the standard position and/or standard confidence can be set according to the position and/or confidence of the business object in the sample image of the training sample, so as to set the standard position and standard confidence. The degree is more objective and precise.

When determining whether the location information and/or the confidence information of the corresponding service object meets the convergence condition in the feature vector convolution result, an optional manner includes:

Obtaining the location information of the corresponding service object in the feature vector convolution result, and calculating the location indicated by the location information of the corresponding service object by calculating the Euclidean distance between the location indicated by the location information of the corresponding service object and the preset standard location Determining, according to the first distance, a first distance between the preset standard position and determining whether the location information of the corresponding service object satisfies the convergence condition;

and / or,

Obtaining the confidence information of the corresponding service object in the feature vector convolution result, calculating the Euclidean distance between the confidence level indicated by the confidence information of the corresponding service object and the preset standard confidence, and obtaining the confidence of the corresponding business object. A second distance between the confidence level of the information indication and the preset standard confidence, and determining, according to the second distance, whether the confidence information of the corresponding service object satisfies the convergence condition. Among them, the Euclidean distance method is adopted, and the implementation is simple and can effectively indicate whether the convergence condition is satisfied. However, the embodiment of the present application is not limited thereto, and other methods such as a horse distance, a bar distance, and the like may also be adopted.

Optionally, as described above, the preset standard position is an average position obtained by averaging the positions of the business objects in the sample image of the training sample; and/or, the preset standard confidence is the pair of training samples. The confidence level of the business object in the sample image is averaged after the average processing.

For determining whether the corresponding face feature vector in the feature vector convolution result satisfies the face convergence condition, it can be set by a person skilled in the art according to actual conditions, which is not limited by the embodiment of the present application.

(4) If the above convergence conditions are satisfied, that is, the position information and/or the confidence information of the corresponding business object in the feature vector convolution result satisfies the convergence condition of the business object, and the corresponding face feature vector in the feature vector convolution result If the face convergence condition is satisfied, the training of the second convolutional network model is completed; otherwise, as long as one convergence condition is not satisfied, for example, the location information and/or confidence information of the corresponding business object in the feature vector convolution result is not satisfied. Satisfying the convergence condition of the business object, and/or the corresponding face feature vector in the feature vector convolution result does not satisfy the face convergence condition, adjusting the network parameter of the second convolutional network model, and according to the adjusted second convolutional network model The network parameters are iteratively trained on the second convolutional network model until the position information and/or the confidence information and the face feature vector of the service object after the iterative training satisfy the corresponding convergence condition.

By performing the above training on the second convolutional network model, the second convolutional network model can feature extracting and classifying the presentation position of the business object based on the face presentation, thereby having the position of determining the presentation position of the business object in the video image. Features. Wherein, when the presentation location includes multiple, the second convolutional network model can also determine the order of the presentation effects in the plurality of presentation locations by the training of the confidence of the business object, and the plurality of presentations are based on the order of the pros and cons The final presentation position is determined in the presentation position. In subsequent applications, when a business object needs to be displayed, a valid presentation location can be determined based on the current image in the video.

In addition, before performing the foregoing training on the second convolutional network model, the sample image may be pre-processed, including: acquiring a plurality of sample images, where each sample image includes the annotation information of the business object; Determining the location of the business object, determining whether the distance between the determined location of the business object and the preset location is less than or equal to a set threshold; determining a sample image corresponding to the business object that is less than or equal to the set threshold as the sample image of the training sample. The preset position and the set threshold may be appropriately set by any suitable means by a person skilled in the art, for example, according to the statistical analysis result of the data or the related distance calculation formula or the artificial experience, etc., which is not limited by the embodiment of the present application.

By pre-processing the business object sample image, the sample image that does not meet the condition can be filtered out to improve the accuracy of the training result.

The training of the second convolutional network model is implemented by the above process, and the trained second convolutional network model can be used to determine the presentation position of the business object in the video image. For example, in the live broadcast process, if the anchor clicks on the business object to indicate the display of the business object, after the second convolutional network model obtains the facial feature point of the anchor in the live video image, the display location of the display business object may be indicated. Such as the anchor's forehead position, which in turn controls the live application to display the business object at that location; or, during the live broadcast of the video, If the anchor clicks on the business object to indicate the display of the business object, the second convolutional network model can directly determine the presentation location of the business object according to the live video image.

Way two

Determining the presentation position of the business object to be presented in the video image according to the feature point of the face attribute and the type of the business object to be presented.

In this implementation, after the feature points of the face attribute are acquired, the presentation position of the business object to be displayed may be determined according to the set rule. Wherein, determining the presentation position of the business object to be displayed includes, for example but not limited to, at least one or any of the following: a hair area of the character in the video image, a forehead area, a cheek area, a chin area, a body area other than the head, a video The background area in the image, the area within the setting range centering on the area where the hand is located in the video image, the area preset in the video image, and the like. The preset area may be appropriately set according to an actual situation, for example, an area within a setting range centering on a face area, or an area within a setting range other than a face area, or a background area, or the like. The embodiment of the present application does not limit this.

After the presentation location is determined, the presentation location of the business object to be displayed in the video image can be further determined. For example, the center point of the presentation location is used as the center point of the presentation location of the business object to display the business object; for example, determining a certain coordinate position in the presentation area corresponding to the presentation location as the center point of the presentation location, etc., This example does not limit this.

In an optional implementation, when determining the presentation location of the business object to be displayed in the video image, the service to be displayed is determined not only according to the feature point of the face attribute but also according to the type of the business object to be displayed. The position at which the object appears in the video image. The type of the business object may include, but is not limited to, at least one of the following or any of a plurality of types: a forehead patch type, a cheek patch type, a chin patch type, a virtual hat type, a virtual clothing type, a virtual makeup type, and a virtual type. Headwear type, virtual hair accessory type, virtual jewelry type, background type, virtual pet type, virtual container type, and the like. However, it is not limited thereto, and the type of the business object may be other suitable types, such as a virtual cap type, a virtual cup type, a text type, and the like.

Thus, according to the type of the business object, the feature location of the face attribute can be used as a reference to select an appropriate presentation location for the business object.

In addition, in a case where a plurality of presentation positions of the business object to be displayed in the video image are obtained according to the feature point of the face attribute and the type of the business object to be displayed, at least one presentation position may be selected from the plurality of presentation positions. As the final display location. For example, for a text type business object, it can be displayed in the background area, or it can be displayed on the person's forehead or body area.

In addition, in another example of the embodiments of the present application, the correspondence between the facial expression and the presentation position may be stored in advance, and when the detected facial expression matches the corresponding predetermined facial expression, the pre-stored facial expression may be In the correspondence with the presentation position, the target presentation position corresponding to the predetermined facial expression is acquired as the presentation position of the business object to be presented in the video image. It should be noted that, although there is a corresponding relationship between the facial expression and the display position, the facial expression is not necessarily related to the presentation position, and the facial expression is only a way to trigger the presentation of the business object, and the position and the face are displayed. There is also no necessary relationship, that is, a business object can be displayed in a certain area of the face, or can be displayed in other areas than the face, such as the background area of the video image.

In an alternative example, steps S430-S440 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a presentation location determining module 602 executed by the processor.

In step S450, the business object is drawn by computer drawing at the presentation position.

When the business object is a sticker containing semantic information, such as an advertisement sticker, before the business object is drawn, relevant information of the business object, such as the identifier and size of the business object, may be acquired first. After the display position is determined, the business object may be scaled, rotated, etc. according to the coordinates of the presentation position, and then the business object is drawn by a corresponding drawing method such as the drawing method of the OpenGL graphics rendering engine. In some cases, ads can also be displayed in 3D special effects, such as text or logos (LOGOs) that display ads through particle effects. For example, the virtual bottle cap type of advertising sticker displays the name of a product to attract viewers to watch, improving the efficiency of advertising and display.

In an alternative example, step S450 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a business object rendering module 603 executed by the processor.

The method for processing a video image provided by the embodiment of the present application, when the business object is used for displaying an advertisement, on the one hand, the business object is drawn by using a computer drawing manner at the determined display position, and the business object is combined with the video playing without passing through The network transmits video data of a business object such as an advertisement that is not related to the video, which is beneficial to save network resources and/or system resources of the client; on the other hand, the business object is closely combined with the facial expression in the video image, and can be retained in the video image. The main image and action of the video subject (such as the anchor) adds interest to the video image, and does not disturb the user to watch the video normally, which can reduce the user's dislike of the business object displayed in the video image, and can also be to some extent Attract the attention of the audience and increase the influence of business objects.

FIG. 5 is a flow chart of still another embodiment of a method for processing a video image of the present application. In this embodiment, the business object contains the advertisement information. The video image processing scheme of the embodiment of the present application is described by taking a two-dimensional sticker special effect, specifically an advertisement sticker as an example. Referring to FIG. 5, the processing method of the video image in this embodiment includes:

In step S501, at least one sample image including face information is acquired as a training sample. The sample image is labeled with information about the face attribute.

In an alternative example, step S501 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a training sample acquisition module 604 executed by the processor.

In step S502, an attribute having a size order feature in the face attribute is encoded.

In an alternative example, step S502 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by an encoding module 605 executed by the processor.

In step S503, the coded attribute is used as the supervision information of the training first convolutional network model, and the first convolutional network model is trained by using the training samples to obtain a first convolutional network model for detecting the face attribute in the image. .

In an alternative example, step S503 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a first convolutional network model determination module 606 executed by the processor.

The content of the steps in the above steps S501 to S503 is the same as that in the embodiment shown in FIG. 2, and details are not described herein again.

In step S504, a feature vector of the sample image of the above training sample is acquired.

The feature vector includes position information and/or confidence information of the business object in the sample image of the business object, and a face feature vector corresponding to the face attribute in the sample image.

Wherein, the face attribute (ie, the facial expression of the face) in the sample image can be determined when training the first convolutional network model.

In this embodiment, there are some sample images in the sample image that do not meet the training standard of the second convolutional network model, and the samples that do not conform to the training standard of the second convolutional network model may be preprocessed by the sample image. The image is filtered out.

In this embodiment, the sample image includes a business object, and each business object is labeled with location information and confidence information. In an optional implementation, the location information of the central point of the business object is used as the location information of the business object. In this step, the sample image is exemplarily filtered according to the location information of the business object. After obtaining the coordinates of the location indicated by the location information, the coordinates are compared with the preset location coordinates of the business object of the type, and the position variance of the two is calculated. If the position variance is less than or equal to the set threshold, the sample image may be used as a sample image of the training sample; if the position variance is greater than the set threshold, the business object sample image is filtered out. The preset position coordinates and the set thresholds may be appropriately set by a person skilled in the art according to actual conditions. For example, the images generally used for the second convolutional network model training have the same size, and the threshold that can be set may be The length or width of the image is 1/20 to 1/5, and alternatively, it may be 1/10 of the length or width of the image.

In addition, the position and confidence of the business object in the sample image of the determined training sample may be averaged to obtain an average position and an average confidence, and the average position and the average confidence may be used as a basis for determining the convergence condition subsequently.

When the business object is an advertisement sticker as an example, the sample image used for training in this embodiment needs to be marked with the coordinates of the advertisement location and the confidence of the advertisement slot. The size of the confidence indicates the probability that this ad slot is the best ad slot. For example, if this ad slot is mostly occluded, the confidence is low. Among them, the advertisement position can be marked in the face, the front background and the like, and the joint training of the advertisement points of the facial feature point and the front background can be realized, which is advantageous for the separate training scheme based on a certain technique such as facial expression. Save computing resources.

In an alternative example, step S504 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a feature vector acquisition module 607 executed by the processor.

In step S505, the feature vector is convoluted to obtain a feature vector convolution result.

It should be noted that, when performing convolution processing on the feature vector, convolution processing is performed on the feature information corresponding to the location information and/or the confidence information of the business object in the sample image, and also in each sample image. The face feature vector corresponding to the face attribute is convoluted, and the corresponding feature vector convolution result is obtained respectively.

In an alternative example, step S505 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a convolution module 608 executed by the processor.

In step S506, it is determined whether the location information and/or the confidence information of the corresponding service object in the feature vector convolution result satisfies the convergence condition of the service object, and determines whether the corresponding face feature vector in the feature vector convolution result satisfies the face convergence. condition.

In an alternative example, step S506 may be performed by the processor invoking a corresponding instruction stored in the memory, or may be performed by a convergence condition determination module 609 executed by the processor.

In step S507, if the convergence conditions in step S506 are satisfied, the training of the second convolutional network model is completed; otherwise, as long as one convergence condition is not satisfied, the network parameters of the second convolutional network model are adjusted and adjusted according to the adjustment Second convolutional network model network The parameters are iteratively trained on the second convolutional network model until the position information and/or confidence information of the service object after the iterative training and the face feature vector satisfy the corresponding convergence conditions.

In an alternative example, step S507 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a model training module 610 that is executed by the processor.

For the specific processing of the foregoing steps S505 to S507, refer to related content in the foregoing Embodiment 3, and details are not described herein again.

Through the processing of steps S504 to S507 described above, the trained second convolutional network model can be obtained. For the structure of the second convolutional network model, reference may be made to the structure of the first convolutional network model in the second embodiment, and details are not described herein again.

The first convolutional network model and the second convolutional network model obtained by the above training may perform corresponding processing on the video image, and may include the following steps S508 to S512.

In step S508, the currently played video image containing the face information is acquired.

In step S509, based on the face information in the video image, the pre-trained first convolutional network model for detecting the face attribute in the image is used to perform facial expression detection of the face on the video image.

In an alternative example, steps S508-S509 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by a video image detection module 601 that is executed by the processor.

In step S510, when it is determined that the detected facial expression matches the corresponding predetermined facial expression, the feature point of the face attribute in the face region corresponding to the detected facial expression is extracted.

In an alternative example, step S510 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a presentation location determining module 602 executed by the processor.

In step S511, according to the feature points of the face attribute, the second convolutional network model for determining the presentation position of the business object in the video image is used to determine the presentation of the business object to be presented in the video image. position.

In an alternative example, step S511 may be performed by a processor invoking a corresponding instruction stored in the memory, or may be performed by a presentation location determining module 602 executed by the processor or a presentation location determining unit therein.

In step S512, the business object is drawn by computer drawing at the presentation position.

In an alternative example, step S512 may be performed by a processor invoking a corresponding instruction stored in a memory, or may be performed by a business object rendering module 603 executed by the processor.

With the rise of Internet live broadcast and short video sharing, more and more videos appear as live or short video. Such videos are often dominated by characters (single characters or a small number of characters), with characters and simple backgrounds as the main scenes, and viewers mainly watch on mobile terminals such as mobile phones. In this case, for the delivery of certain business objects (such as advertisement delivery), the video image during the video playback process can be detected in real time through the solution provided in this embodiment, and the advertisement placement position with better effect is given. In order to improve the viewing experience without affecting the user's viewing experience, the business object is drawn by computer drawing in the determined presentation position, and the business object is combined with the video playing, and no additional business objects such as advertisements that are not related to the video are transmitted through the network. Video data is conducive to saving network resources and/or system resources of the client; in addition, the business object is closely combined with facial expressions in the video image, and retains the main image and motion of the video subject (such as the anchor) in the video image, and The video image adds interest, and does not disturb the user to watch the video normally, which can reduce the user's dislike of the business object displayed in the video image, and can attract the viewer's attention to a certain extent and improve the influence of the business object. It can be understood that, in addition to advertising, business objects can be widely applied to other aspects, such as education, consulting, services, etc., by providing entertainment, appreciation and other business information to improve interaction and improve user experience.

The processing method of any video image provided by the embodiment of the present application may be performed by any suitable device having data processing capability, including but not limited to: a terminal device, a server, and the like. Alternatively, the processing method of any video image provided by the embodiment of the present application may be executed by a processor, such as a processor, by executing a corresponding instruction stored in a memory to perform a processing method of any video image mentioned in the embodiment of the present application. This will not be repeated below.

A person skilled in the art can understand that all or part of the steps of implementing the foregoing video image processing method may be completed by using hardware related to the program instructions. The foregoing program may be stored in a computer readable storage medium. When executed, the steps including the above method embodiments are performed; and the foregoing storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.

FIG. 6 is a schematic structural diagram of an embodiment of a processing apparatus for video images of the present application. The video image processing apparatus of the embodiments of the present application can be used to implement the foregoing method for processing each video image of the present application. Referring to FIG. 6, the processing apparatus for the video image of this embodiment includes: a video image detecting module 601, a presentation position determining module 602, and a business object drawing module 603. among them:

The video image detecting module 601 is configured to perform facial expression detection of the face on the currently played video image including the face information.

a presentation location determining module 602, configured to: when the facial expression detected by the video image detecting module 602 is associated with a corresponding predetermined facial expression When matching, the presentation position of the business object to be presented in the video image is determined.

The business object drawing module 603 is configured to draw a business object by using a computer drawing manner at the presentation position.

The processing apparatus for the video image provided by the embodiment performs facial expression detection on the currently played video image including the face information, and matches the detected facial expression with the corresponding predetermined facial expression, when the two match Determining the presentation position of the business object to be presented in the video image, and then drawing the business object by means of computer drawing at the presentation position, so that when the business object is used for displaying the advertisement, on the one hand, the computer is used at the determined display position. Drawing the business object, the business object is combined with video playing, and does not need to transmit video data of a business object such as an advertisement that is not related to the video through the network, thereby saving network resources and/or system resources of the client; Aspects, the business object is closely combined with the facial expressions in the video image, and can retain the main image and motion of the video subject (such as the anchor) in the video image, adding interest to the video image, and not disturbing the user to watch the video normally, Reduce the number of business pairs that users display on video images Resentment, but also to attract the audience's attention to a certain extent, increase the influence of business objects.

In an optional example of the processing device embodiment of the video image of the present application, the video image detecting module 601 is configured to use the pre-trained and used for the face information in the video image that includes the face information that is currently played. A first convolutional network model of the face attribute in the image is detected, and a facial expression of the face is detected on the video image.

FIG. 7 is a schematic structural diagram of another embodiment of a processing apparatus for video images of the present application. Referring to FIG. 7, the processing apparatus of the video image further includes: a training sample obtaining module 604, configured to acquire at least one sample image including face information as a training sample, wherein the sample is compared with the embodiment shown in FIG. The image is labeled with the information of the face attribute; the encoding module 605 is configured to encode the attribute having the size order feature in the face attribute; the first convolutional network model determining module 606 is configured to use the coded attribute as the training first volume. The supervisory information of the product network model is trained on the first convolutional network model using training samples to obtain a first convolutional network model for detecting face attributes in the image.

Optionally, the training sample obtaining module 604 may include: a sample image acquiring unit, configured to acquire at least one sample image including face information; and a face positioning information determining unit, configured to detect the sample image in each sample image The face and the face key point, the face in the sample image is positioned by the face key point to obtain the face location information; the training sample determination unit is configured to use the sample image containing the face location information as the training sample.

Optionally, the presentation location determining module 602 may include: a feature point extraction unit, configured to acquire a feature point of a face attribute in a face region corresponding to the detected facial expression; a presentation position determining unit, configured to use the face attribute The feature point determines the presentation location of the business object to be presented in the video image.

Optionally, the presentation location determining unit is configured to determine, according to the feature point of the face attribute, a pre-trained second convolution network model for determining a presentation position of the service object in the video image, to determine the service to be presented. The position at which the object appears in the video image.

Optionally, the processing apparatus for the video image of the embodiment of the present application may further include: a feature vector obtaining module 607, configured to acquire a feature vector of the sample image of the training sample, where the feature vector includes location information of the business object in the sample image. And/or confidence information, and a face feature vector corresponding to the face attribute in the sample image; the convolution module 608 is configured to perform convolution processing on the feature vector to obtain a feature vector convolution result; the convergence condition determination module 609 uses Determining whether the location information and/or the confidence information of the corresponding business object in the feature vector convolution result satisfies the convergence condition of the business object, and determining whether the corresponding face feature vector in the feature vector convolution result satisfies the face convergence condition; The training module 610 is configured to: when the convergence conditions are satisfied, that is, the location information and/or the confidence information of the corresponding service object in the feature vector convolution result satisfies the convergence condition of the service object, and the corresponding result in the feature vector convolution result When the face feature vector satisfies the face convergence condition, the training of the second convolutional network model is completed. Otherwise, as long as one convergence condition is not satisfied, the network parameters of the second convolutional network model are adjusted, and the second convolutional network model is iteratively trained according to the adjusted network parameters of the second convolutional network model until iterative training The location information and/or confidence information of the subsequent business object and the face feature vector satisfy the corresponding convergence conditions.

Optionally, the presentation location determining module 602 is configured to determine a presentation location of the business object to be presented in the video image according to the feature point of the facial attribute and the type of the business object to be presented.

Optionally, the presentation location determining module 602 includes: a presentation location obtaining unit, configured to obtain, according to the feature point of the facial attribute and the type of the business object to be presented, a plurality of presentation locations of the business object to be presented in the video image; And a presentation location selection unit, configured to select at least one presentation location from the plurality of presentation locations as a presentation location of the business object to be presented in the video image.

Optionally, the presentation location determining module 602 is configured to obtain, from a correspondence between the pre-stored facial expression and the presentation location, a target presentation location corresponding to the predetermined facial expression as a presentation location of the business object to be presented in the video image.

Optionally, the business object may include: an effect containing semantic information; the video image may include a live video image or any other video image.

Optionally, the special effect including the semantic information may include at least one or any of the following special effects including the advertisement information: a two-dimensional sticker effect, a three-dimensional special effect, a particle special effect, and the like.

Optionally, the presentation location may include, but is not limited to, at least one or any of the following: a hair area of the person in the video image, a forehead area, a cheek area, a chin area, a body area other than the head, a background area in the video image In the video image, the area within the setting range centering on the area where the hand is located, and the area preset in the video image.

Optionally, the type of the business object includes at least one of the following or any of the following types: a forehead patch type, a cheek patch type, a chin patch type, a virtual hat type, a virtual clothing type, a virtual makeup type, a virtual headwear type, Virtual hair accessory type, virtual jewelry type, background type, virtual pet type, virtual container type.

Optionally, the facial expression includes at least one or any of the following: happy, angry, painful, sad, contemplative, exhausted, and the like.

Referring to FIG. 7, a schematic structural diagram of an electronic device according to Embodiment 7 of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the electronic device. As shown in FIG. 7, the electronic device can include a processor 802, a communications interface 804, a memory 806, and a communications bus 808. among them:

Processor 802, communication interface 804, and memory 806 complete communication with one another via communication bus 808.

The communication interface 804 is configured to communicate with network elements of other devices, such as other clients or servers.

The processor 802 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, or a graphics processor ( Graphics Processing Unit, GPU). The one or more processors included in the terminal device may be the same type of processor, such as one or more CPUs, or one or more GPUs; or may be different types of processors, such as one or more CPUs and One or more GPUs.

The memory 806 is for at least one executable instruction that causes the processor 802 to perform operations corresponding to a method of presenting a business object in a video image as in any of the above-described embodiments of the present application. The memory 806 may include a high speed random access memory (RAM), and may also include a non-volatile memory such as at least one disk memory.

FIG. 9 is a schematic structural diagram of another embodiment of an electronic device according to the present invention. Referring to FIG. 9, there is shown a schematic structural diagram of an electronic device suitable for implementing a terminal device or a server of an embodiment of the present application. As shown in FIG. 9, the electronic device includes one or more processors, a communication unit, etc., such as one or more central processing units (CPUs) 901, and/or one or more A graphics processor (GPU) 913 or the like, the processor may execute various types according to executable instructions stored in read only memory (ROM) 902 or executable instructions loaded from random access memory (RAM) 903 from storage portion 908. Proper action and handling. Communication portion 912 can include, but is not limited to, a network card, which can include, but is not limited to, an IB (Infiniband) network card, and the processor can communicate with read only memory 902 and/or random access memory 903 to execute executable instructions over bus 904. The operation corresponding to the processing method of any video image provided by the embodiment of the present application is performed, for example, the currently played video image including the face information is connected to the communication unit 912 and communicates with other target devices via the communication unit 912. Facial expression detection of a face; determining a presentation position of a business object to be presented in the video image when the detected facial expression matches a corresponding predetermined facial expression; drawing a computer drawing manner at the presentation position The business object.

Further, in the RAM 903, various programs and data required for the operation of the device can be stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other by a bus 904. In the case of RAM 903, ROM 902 is an optional module. The RAM 903 stores executable instructions or writes executable instructions to the ROM 902 at runtime, the executable instructions causing the processor 901 to perform operations corresponding to the processing methods of the video images described above. An input/output (I/O) interface 905 is also coupled to bus 904. The communication unit 912 may be integrated or may be provided with a plurality of sub-modules (for example, a plurality of IB network cards) and on the bus link.

The following components are connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a storage portion 908 including a hard disk or the like. And a communication portion 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the Internet. The drive 911 is also connected to the I/O interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like is mounted on the drive 911 as needed so that a computer program read therefrom is installed into the storage portion 908 as needed.

It should be noted that the architecture shown in FIG. 9 is only an optional implementation manner. In a specific implementation process, the number and type of components in FIG. 9 may be selected, deleted, added, or replaced according to actual needs; Different function components can also be implemented in separate settings or integrated settings, such as GPU and CPU detachable settings or GPU can be integrated on the CPU, the communication part can be separated, or integrated on the CPU or GPU. and many more. These alternative embodiments are all within the scope of the present disclosure.

In particular, the processes described above with reference to the flowcharts may be implemented as a computer software program in accordance with an embodiment of the present disclosure. For example, an embodiment of the present disclosure includes a computer program product comprising a computer program tangibly embodied on a machine readable medium, the computer program comprising program code for executing the method illustrated in the flowchart, the program code comprising Executing instructions corresponding to the method steps provided in the embodiments of the present application, for example, performing facial expression detection on a face of a currently played video image containing face information; and when the detected facial expression matches a corresponding predetermined facial expression, Determining a presentation location of the business object to be presented in the video image; drawing the business object in a computer drawing manner at the presentation location.

In addition, the embodiment of the present application further provides a computer program, the computer program comprising computer readable code, the program code includes computer operating instructions, when the computer readable code is run on the device, the processor in the device executes An instruction for implementing each step in the processing method of the video image of any of the embodiments of the present application.

In addition, the embodiment of the present application further provides a computer readable storage medium for storing computer readable instructions, which are executed to implement the operations of the steps in the video image processing method of any embodiment of the present application.

For the specific implementation of the steps of the computer program and the computer readable command, the corresponding steps in the above-mentioned embodiments and corresponding descriptions in the modules are not described herein. A person skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the device and the module described above may be referred to the corresponding process description in the foregoing method embodiment, and details are not described herein again.

The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same or similar parts between the various embodiments may be referred to each other. For a device, an electronic device, a program, a storage medium, and the like, the description is relatively simple because it basically corresponds to the method embodiment. For related parts, refer to the description of the method embodiment.

It should be pointed out that the various steps/components described in the present application can be split into more steps/components according to the needs of the implementation, and two or more steps/components or partial operations of the steps/components can be combined into new ones. Steps/components to achieve the objectives of the present application.

The above method according to the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk, or a magneto-optical disk, or can be downloaded through a network. The computer code originally stored in a remote recording medium or non-transitory machine readable medium and to be stored in a local recording medium, whereby the methods described herein can be stored using a general purpose computer, a dedicated processor, or programmable or dedicated Such software processing on a recording medium of hardware such as an ASIC or an FPGA. It will be understood that a computer, processor, microprocessor controller or programmable hardware includes storage components (eg, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, when the software or computer code is The processing methods described herein are implemented when the processor or hardware is accessed and executed. Moreover, when a general purpose computer accesses code for implementing the processing shown herein, the execution of the code converts the general purpose computer into a special purpose computer for performing the processing shown herein.

The foregoing is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto, and any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application. It should be covered by the scope of protection of this application. Therefore, the scope of protection of the present application should be determined by the scope of the claims.

Claims (34)

A method for processing a video image, comprising: Performing facial expression detection of a face on a currently played video image containing face information; Determining a presentation position of the business object to be presented in the video image when the detected facial expression matches the corresponding predetermined facial expression; The business object is drawn in a computer drawing manner at the presentation position. The method according to claim 1, wherein the performing facial expression detection of the face on the currently played video image containing the face information comprises: Based on the face information in the video image, a facial expression detection of the face is performed on the video image using a pre-trained first convolutional network model for detecting a face attribute in the image. The method of claim 2, wherein pre-training the first convolutional network model comprises: Obtaining at least one sample image including face information, wherein the sample image is labeled with information of a face attribute; Encoding an attribute having a size order feature in the face attribute; Using the coded attribute as the supervision information for training the first convolutional network model, training the first convolutional network model using the training sample, and obtaining a first convolution for detecting a face attribute in the image Network model. The method according to claim 3, wherein acquiring at least one sample image including face information as a training sample comprises: Obtaining at least one sample image including face information; Detecting a face and a face key point in the sample image for each of the at least one sample image, and positioning the face in the sample image by the face key point to obtain a face Positioning information; The sample image including the face location information is used as a training sample. The method according to any one of claims 1-4, wherein the determining a presentation location of the business object to be presented in the video image comprises: Acquiring feature points of the face attribute in the face region corresponding to the detected facial expression; Determining, according to a feature point of the face attribute, a presentation position of the business object to be presented in the video image. The method according to claim 5, wherein the determining, according to the feature point of the face attribute, the presentation position of the business object to be presented in the video image comprises: Determining, according to the feature point of the face attribute, a pre-trained second convolution network model for determining a presentation position of the business object in the video image, determining that the business object to be presented is in the video image Show position. The method of claim 6 wherein the pre-training of the second convolutional network model comprises: Obtaining a feature vector of the sample image of the training sample, where the feature vector includes: location information and/or confidence information of the business object in the sample image, and a face feature vector corresponding to the face attribute in the sample image ; Convoluting the feature vector to obtain a feature vector convolution result; Determining whether the location information and/or the confidence information of the corresponding service object in the feature vector convolution result satisfies the convergence condition of the service object, and determining whether the corresponding face feature vector in the feature vector convolution result satisfies the face convergence condition; If all are satisfied, completing training on the second convolutional network model; Otherwise, the network parameters of the second convolutional network model are adjusted, and the second convolutional network model is iteratively trained according to the adjusted network parameters of the second convolutional network model until the location information of the service object after the iterative training And/or the confidence information and the face feature vector both satisfy respective convergence conditions. The method according to any one of claims 5-7, wherein the determining, according to the feature point of the face attribute, the presentation position of the business object to be presented in the video image comprises: Determining a presentation location of the business object to be presented in the video image according to the feature point of the face attribute and the type of the business object to be presented. The method according to claim 8, wherein the display location of the business object to be presented in the video image is determined according to the feature point of the face attribute and the type of the business object to be presented, including : Obtaining a business object to be presented in the video according to the feature point of the face attribute and the type of the business object to be presented Multiple presentation locations in the image; Selecting at least one presentation location from the plurality of presentation locations as a presentation location of the business object to be presented in the video image. The method according to any one of claims 1-4, wherein the determining a presentation location of the business object to be presented in the video image comprises: The target presentation position corresponding to the predetermined facial expression is obtained as a presentation position of the business object to be presented in the video image from a correspondence relationship between the pre-stored facial expression and the presentation position. The method according to any one of claims 1 to 10, wherein the business object comprises: an effect comprising semantic information; the video image comprises: a live video type image. The method according to claim 11, wherein the special effect including the semantic information comprises one or any of a plurality of forms of special effects including the two-dimensional sticker effect, the three-dimensional special effect, and the particle special effect. The method according to any one of claims 1 to 12, wherein the presentation position comprises one or any of a plurality of regions: a hair region of a person in a video image, a forehead region, a cheek region, a chin region, and a head portion. The body area, the background area in the video image, the area within the set range of the video image centered on the area where the hand is located, and the area preset in the video image. The method according to any one of claims 1 to 13, wherein the type of the business object comprises one or any of the following types: a forehead patch type, a cheek patch type, a chin patch type, a virtual hat. Type, virtual clothing type, virtual makeup type, virtual headdress type, virtual hair accessory type, virtual jewelry type, background type, virtual pet type, virtual container type. A method according to any one of claims 1-14, wherein the facial expression comprises one or more of the following: happy, angry, painful, sad. A processing device for video images, characterized in that the device comprises: a video image detecting module, configured to perform facial expression detection on a face of a currently played video image including face information; a presentation location determining module, configured to determine a presentation location of the business object to be presented in the video image when the facial expression detected by the video image detection module matches a corresponding predetermined facial expression; A business object drawing module is configured to draw the business object by using a computer drawing manner at the presentation position. The device according to claim 16, wherein the video image detecting module is configured to use a pre-trained first for detecting a face attribute in the image based on the face information in the video image. A convolutional network model performs facial expression detection of a face on the video image. The device according to claim 17, further comprising: a training sample obtaining module, configured to acquire at least one sample image including face information, wherein the sample image is labeled with information of a face attribute; An encoding module, configured to encode an attribute having a size order feature in the face attribute; a first convolutional network model determining module, configured to use the encoded attribute as monitoring information for training the first convolutional network model, and use the training sample to train the first convolutional network model to obtain A first convolutional network model that detects the face attributes in the image. The apparatus according to claim 18, wherein the training sample acquisition module comprises: a sample image obtaining unit, configured to acquire at least one sample image including face information; a face positioning information determining unit, configured to detect a face and a face key point in the sample image for each of the sample images in the at least one sample image, by using the face key point in the sample image The face is positioned to obtain face location information; A training sample determining unit is configured to use the sample image including the face positioning information as a training sample. The device according to any one of claims 16 to 19, wherein the presentation location determining module comprises: a feature point extracting unit, configured to acquire a feature point of a face attribute in a face region corresponding to the detected facial expression; And a presentation location determining unit, configured to determine, according to the feature point of the face attribute, a presentation location of the business object to be presented in the video image. The device according to claim 20, wherein the presentation position determining unit is configured to use, according to the feature points of the face attribute, a pre-trained position for determining a presentation position of the business object in the video image. a second convolutional network model determining a presentation location of the business object to be presented in the video image. The device according to claim 21, further comprising: a feature vector obtaining module, configured to acquire a feature vector of a sample image of the training sample, where the feature vector includes: Position information and/or confidence information of the business object in the sample image, and a face feature vector corresponding to the face attribute in the sample image; a convolution module, configured to perform convolution processing on the feature vector to obtain a feature vector convolution result; a convergence condition determining module, configured to determine whether location information and/or confidence information of a corresponding service object in the feature vector convolution result satisfies a service object convergence condition, and determine a corresponding face in the feature vector convolution result Whether the feature vector satisfies the face convergence condition; a model training module, configured to perform training on the second convolutional network model if satisfied; otherwise, adjusting network parameters of the second convolutional network model and according to the adjusted network of the second convolutional network model The parameter performs iterative training on the second convolutional network model until the position information and/or the confidence information of the service object after the iterative training and the face feature vector satisfy the corresponding convergence condition. The device according to any one of claims 20 to 22, wherein the presentation location determining module is configured to determine, according to a feature point of the face attribute and a type of the business object to be presented, The location of the business object in the video image. The device according to claim 23, wherein the presentation location determining module comprises: a presentation location obtaining unit, configured to obtain, according to the feature point of the face attribute and the type of the business object to be presented, a plurality of presentation positions of the business object to be presented in the video image; And a presentation location selection unit, configured to select at least one presentation location from the plurality of presentation locations as a presentation location of the business object to be presented in the video image. The device according to any one of claims 16 to 19, wherein the presentation position determining module is configured to acquire a target corresponding to the predetermined facial expression from a correspondence between a pre-stored facial expression and a presentation position. The presentation location is used as a presentation location of the business object to be presented in the video image. The apparatus according to any one of claims 16 to 25, wherein the business object comprises an effect comprising semantic information; the video image comprises a live video type image. The apparatus according to claim 26, wherein the special effect including the semantic information comprises at least one or any of the following special effects including the advertisement information: a two-dimensional sticker effect, a three-dimensional special effect, and a particle special effect. The device according to any one of claims 16-27, wherein the presentation position comprises one or any of a plurality of regions: a hair region of a person in the video image, a forehead region, a cheek region, a chin region, and a head portion. The body area, the background area in the video image, the area within the set range of the video image centered on the area where the hand is located, and the area preset in the video image. The device according to any one of claims 16-28, wherein the type of the business object comprises one or any of the following types: a forehead patch type, a cheek patch type, a chin patch type, a virtual hat type. , virtual clothing type, virtual makeup type, virtual headwear type, virtual hair accessory type, virtual jewelry type, background type, virtual pet type, virtual container type. The device according to any one of claims 16-29, wherein the facial expression comprises one or more of the following: happy, angry, painful, sad. An electronic device comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is configured to store at least one executable instruction that causes the processor to perform an operation corresponding to the method of processing a video image according to any one of claims 1-15. An electronic device, comprising: A processor and a processing apparatus for a video image according to any of claims 16-30; The unit in the processing apparatus for the video image of any of claims 16-30 is operated while the processor is running the structured text detection system. A computer program comprising computer readable code, wherein a processor in the device executes a video for implementing any of claims 1-15 when the computer readable code is run on a device The instructions for each step in the image processing method. A computer readable storage medium for storing computer readable instructions, wherein the instructions are executed to perform the operations of the steps of the video image processing method of any of claims 1-15.

Images