TW202138992A - Method and apparatus for driving interactive object, device and storage medium - Google Patents

Abstract

The present disclosure relates to a method and an apparatus for driving interactive object, a device and a storage medium. The method comprises: obtaining a phoneme sequence corresponding to text data; obtaining a value of control parameter of at least one local area of an interactive object matched with the phoneme sequence; and controlling a posture of the interactive object according to the obtained value of control parameter.

Description

Driving method, device, equipment and storage medium of interactive object

[Cross reference of related applications]

This application is based on a Chinese patent application with an application number of 2020102458024 and an application date of March 31, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference into this application.

The present disclosure relates to the field of computer technology, and in particular to a driving method, device, equipment and storage medium of an interactive object.

Human-computer interaction is mostly based on keystrokes, touches, and voice input, and responds by presenting images, text, or virtual characters on the display. At present, virtual characters are mostly improved on the basis of voice assistants.

The embodiments of the present disclosure provide a driving solution for interactive objects.

According to an aspect of the present disclosure, a method for driving an interactive object is provided, the method includes: obtaining a phoneme sequence corresponding to text data; obtaining a control parameter value of at least one partial area of an interactive object matching the phoneme sequence; The control parameter value controls the posture of the interactive object.

With reference to any of the embodiments provided in the present disclosure, the method further includes: controlling the display device displaying the interactive object to display text according to the text data, and/or controlling the display device according to the phoneme sequence corresponding to the text data Output speech.

With reference to any one of the embodiments provided in the present disclosure, the control parameter of the local area of the interactive object includes the posture control vector of the local area, and the control parameter value of at least one local area of the interactive object matching the phoneme sequence is acquired, The method includes: performing feature coding on the phoneme sequence to obtain a first coding sequence corresponding to the phoneme sequence; obtaining a feature code corresponding to at least one phoneme according to the first coding sequence; obtaining the interaction corresponding to the feature code The attitude control vector of at least one local area of the object.

In conjunction with any one of the embodiments provided in the present disclosure, performing feature encoding on the phoneme sequence to obtain the first encoding sequence corresponding to the phoneme sequence includes: generating all phonemes for each of the multiple phonemes included in the phoneme sequence. The sub-coding sequence corresponding to the phoneme; and the first coding sequence corresponding to the phoneme sequence is obtained according to the sub-coding sequences respectively corresponding to the multiple phonemes.

With reference to any of the embodiments provided in the present disclosure, for each of the multiple phonemes included in the phoneme sequence, generating a sub-coding sequence corresponding to the phoneme includes: detecting whether the phoneme corresponds to each time point; The coding value at the time point with the phoneme is set to a first value, and the coding value at the time point without the phoneme is set to a second value to obtain the sub-coding sequence corresponding to the phoneme.

With reference to any of the embodiments provided in the present disclosure, the method further includes: for the sub-coding sequence corresponding to each phoneme of the multiple phonemes, using a Gaussian filter to perform Gaussian on the continuous values of the phonemes in time. Convolution operation.

With reference to any one of the embodiments provided in the present disclosure, controlling the posture of the interactive object according to the obtained control parameter value includes: obtaining a sequence of posture control vectors corresponding to the second coding sequence; and according to the posture control vector The sequence of controls the gesture of the interactive object.

With reference to any of the embodiments provided in the present disclosure, the method further includes: in the case that the time interval between the phonemes in the phoneme sequence is greater than a set threshold, controlling according to the set control parameter value of the local area The posture of the interactive object.

In conjunction with any one of the embodiments provided in the present disclosure, obtaining the attitude control vector of at least one local area of the interactive object corresponding to the feature code includes: inputting the feature code into a pre-trained loop neural network to obtain The attitude control vector of at least one partial area of the interactive object corresponding to the feature code.

With reference to any of the embodiments provided in the present disclosure, the loop neural network is obtained through feature coding sample training; the method further includes: obtaining a video segment of a character's voice, and obtaining a plurality of video segments containing all of them according to the video segment. The first image frame of the character; extract the corresponding speech segment from the film segment, obtain a sample phoneme sequence according to the speech segment, and perform feature encoding on the sample phoneme sequence; obtain the first image Feature encoding of at least one phoneme corresponding to the frame; transforming the first image frame into a second image frame containing the interactive object, and obtaining a posture control vector of at least one local area corresponding to the second image frame Value; according to the attitude control vector value, annotate the feature code corresponding to the first image frame to obtain the feature code sample.

With reference to any one of the embodiments provided in the present disclosure, the method further includes: training the initial loop neural network according to the characteristic coding samples, and training to obtain the loop neural network after the change of the network loss satisfies the convergence condition Wherein, the network loss includes the difference between the attitude control vector value of the at least one local area predicted by the loop neural network and the marked attitude control vector value.

According to an aspect of the present disclosure, there is provided a driving device for an interactive object. The device includes: a first acquiring unit for acquiring a phoneme sequence corresponding to a text data; a second acquiring unit for acquiring a phoneme sequence matching the phoneme sequence The control parameter value of at least one partial area of the interactive object; the driving unit is used to control the posture of the interactive object according to the acquired control parameter value.

According to one aspect of the present disclosure, an electronic device is provided, the device includes a memory and a processor, the memory is used to store computer instructions that can be run on the processor, and the processor is used to execute the computer instructions. The driving method of the interactive object described in any of the embodiments provided in the present disclosure is realized at a time.

According to one aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the method for driving an interactive object according to any one of the embodiments provided in the present disclosure.

The driving method, device, device, and computer-readable storage medium of an interactive object according to one or more embodiments of the present disclosure obtain a phoneme sequence corresponding to text data, and obtain at least one partial area of an interactive object matching the phoneme sequence The control parameter value of to control the posture of the interactive object, so that the interactive object can make a posture that matches the phoneme corresponding to the text data. The posture includes facial posture and body posture, so that the target object generates the interactive object is speaking The sense of text content enhances the interactive experience between the target object and the interactive object.

The exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the scope of the appended application.

The term "and/or" in this article is only an association relationship describing related objects, which means that there can be three relationships. For example, A and/or B can mean: A alone exists, A and B exist at the same time, and B exists alone. three conditions. In addition, the term "at least one" herein means any one or any combination of at least two of the multiple, for example, including at least one of A, B, and C, and may mean including those made from A, B, and C Any one or more elements selected in the set.

At least one embodiment of the present disclosure provides a method for driving an interactive object. The driving method may be executed by an electronic device such as a terminal device or a server. The terminal device may be a fixed terminal or a mobile terminal, such as a mobile phone, a tablet computer, or a game. Computer, desktop computer, advertising player, all-in-one, vehicle-mounted terminal, etc., the server includes a local server or a cloud server, etc. The method can also call a computer readable stored in the memory through the processor The way to implement instructions.

In the embodiment of the present disclosure, the interactive object may be any virtual image capable of interacting with the target object. In an embodiment, the interactive object may be a virtual character, or may also be a virtual animal, a virtual item, a cartoon image, or other virtual images capable of realizing interactive functions. The display form of the interactive object may be 2D or 3D, which is not limited in the present disclosure. The target object can be a user, a robot, or other smart devices. The dialogue mode between the interactive object and the target object may be an active dialogue mode or a passive dialogue mode. In one example, the target object can make a demand by making gestures or body movements, and trigger the interactive object to interact with it through active interaction. In another example, the interactive object may actively greet, prompt the target object to make an action, etc., so that the target object interacts with the interactive object in a passive manner.

The interactive object may be displayed through terminal equipment, which may be a television, an all-in-one machine with a display function, a projector, a virtual reality (Virtual Reality, VR) device, and an augmented reality (Augmented Reality, AR) device. ) Device, etc., the present disclosure does not limit the specific form of the terminal device.

FIG. 1 shows a display device proposed by at least one embodiment of the present disclosure. As shown in FIG. 1, the display device has a transparent display screen, and a three-dimensional picture can be displayed on the transparent display screen to present a virtual scene and interactive objects with a three-dimensional effect. For example, the interactive objects displayed on the transparent display screen in FIG. 1 include virtual cartoon characters. In some embodiments, the terminal device described in the present disclosure may also be the above-mentioned display device with a transparent display screen. The display device is configured with a memory and a processor, and the memory is used to store a computer that can run on the processor. Instructions, the processor is used to implement the method for driving an interactive object provided by the present disclosure when executing the computer instruction, so as to drive the interactive object displayed on the transparent display screen to communicate or respond to the target object.

In some embodiments, in response to the sound-driven data used to drive the interactive object to output a voice, the interactive object can emit a specified voice to the target object. The terminal device can generate sound-driven data according to the actions, expressions, identities, preferences, etc. of the target object around the terminal device to drive the interactive object to respond by sending a specified voice, thereby providing anthropomorphic services for the target object. It should be noted that the sound-driven data can also be generated in other ways, for example, generated by a server and sent to the terminal device.

During the interaction between the interactive object and the target object, when the interactive object is driven to emit a specified voice according to the sound-driven data, the interactive object may not be able to drive the interactive object to make facial movements synchronized with the specified voice, making the interactive object dull when the voice is emitted , Unnatural, affecting the interactive experience of the target object and the interactive object. Based on this, at least one embodiment of the present disclosure proposes a method for driving an interactive object, so as to improve the interactive experience between the target object and the interactive object.

FIG. 2 shows a flowchart of a method for driving an interactive object according to at least one embodiment of the present disclosure. As shown in FIG. 2, the method includes steps 201 to 203.

Step 201: Obtain the phoneme sequence corresponding to the text data.

The text data may be driving data used to drive the interactive object. The driving data can be the driving data generated by the server or the terminal device according to the action, expression, identity, preference, etc. of the target object interacting with the interactive object, or it can be the driving data called by the terminal device from the internal memory. This disclosure does not restrict the method of obtaining the textual information.

In the embodiment of the present disclosure, the phoneme corresponding to the morpheme can be obtained according to the morphemes contained in the text, so as to obtain the phoneme sequence corresponding to the text. Among them, the phoneme is the smallest phonetic unit divided according to the natural attributes of the speech, and a pronunciation action of a real person can form a phoneme.

In one embodiment, in response to the text being a Chinese text, the Chinese text can be converted into pinyin, using pinyin to generate a phoneme sequence, and generating a time stamp for each phoneme.

Step 202: Obtain a control parameter value of at least one partial region of an interactive object that matches the phoneme sequence.

The partial area is obtained by dividing the entire interactive object (including the face and/or body). The control of one or more partial areas of the face can correspond to a series of facial expressions or actions of the interactive object. For example, the control of the eye area can correspond to the facial actions of the interactive object such as opening, closing, blinking, and changing the perspective; another example is the mouth. The control of the part area can correspond to facial actions such as closing the mouth of the interactive object and opening the mouth to different degrees. The control of one or more local areas of the body may correspond to a series of physical actions of the interactive object. For example, the control of the leg area may correspond to the actions of the interactive object such as walking, jumping, and kicking.

The control parameter of the local area of the interactive object includes the attitude control vector of the local area. The attitude control vector of each local area is used to drive the local area of the interactive object to move. Different posture control vector values correspond to different motions or motion ranges. For example, for the posture control vector of the mouth area, one set of posture control vector values can make the mouth of the interactive object slightly open, and another set of posture control vector values can make the mouth of the interactive object open wider. By using different posture control vector values to drive the interactive object, the corresponding local area can perform different actions or actions with different amplitudes.

The local area can be selected according to the action of the interactive object that needs to be controlled. For example, when the face and limbs of the interactive object need to be controlled to move at the same time, the posture control vector value of all the local areas can be obtained; when the expression of the interactive object needs to be controlled At this time, the posture control vector value of the local area corresponding to the face can be obtained.

In the embodiment of the present disclosure, the control parameter value corresponding to the characteristic code can be determined by performing feature encoding on the phoneme sequence, thereby determining the control parameter value corresponding to the phoneme sequence. Different encoding methods can reflect different characteristics of the phoneme sequence. The present disclosure does not limit the specific encoding method.

In the embodiment of the present disclosure, the corresponding relationship between the feature code of the phoneme sequence corresponding to the text data and the control parameter value of the interactive object can be established in advance, so that the corresponding control parameter value can be obtained through the text data. The specific method for obtaining the control parameter value matching the feature code of the phoneme sequence of the text data will be described in detail later.

Step 203: Control the posture of the interactive object according to the acquired control parameter value.

Wherein, the control parameter value, such as the attitude control vector value, matches the phoneme sequence contained in the text data. For example, when the display device displaying the interactive object is controlled to display text according to the text data, and/or the display device is controlled to output speech according to the phoneme sequence corresponding to the text data, the gesture and the gesture made by the interactive object are The output voice and/or the displayed text are synchronized, thereby giving the target object a feeling that the interactive object is speaking.

In the embodiment of the present disclosure, by obtaining the phoneme sequence corresponding to the text data, and obtaining the control parameter value of at least one partial area of the interactive object matching the phoneme sequence, the posture of the interactive object can be controlled, so that the interactive object A gesture that matches the phoneme corresponding to the text data is made. The gesture includes facial gestures and body gestures, so that the target object has the feeling that the interactive object is speaking the text content, and the interactive experience of the target object is improved.

In some embodiments, the method is applied to a server, including a local server or a cloud server. The server processes text data to generate control parameter values of the interactive object, and according to the control The parameter value is rendered using a three-dimensional rendering engine to obtain an animation of the interactive object. The server may send the animation to the terminal for display to communicate or respond to the target object, and may also send the animation to the cloud, so that the terminal can obtain the animation from the cloud to communicate or communicate with the target object. Response. After the server generates the control parameter value of the interactive object, it can also send the control parameter value to the terminal, so that the terminal completes the process of rendering, generating animation, and displaying.

In some embodiments, the method is applied to a terminal, and the terminal processes text data, generates control parameter values of the interactive object, and renders the interactive object using a three-dimensional rendering engine according to the control parameter value to obtain the interactive The animation of the object, the terminal can display the animation to communicate or respond to the target object.

In some embodiments, the display device displaying the interactive object may be controlled to display text according to the text data, and/or the display device may be controlled to output speech according to the phoneme sequence corresponding to the text data.

In the embodiment of the present disclosure, since the control parameter value matches the phoneme sequence of the text data, the voice and/or text outputted according to the text data is different from controlling the gesture of the interactive object according to the control parameter value. In the case of synchronization, the gesture made by the interactive object is synchronized with the output voice and/or the displayed text, giving the target object the feeling that the interactive object is speaking.

In some embodiments, the control parameter of the at least one partial area of the interactive object includes an attitude control vector, and the attitude control vector can be obtained in the following manner.

First, feature encoding is performed on the phoneme sequence to obtain the encoding sequence corresponding to the phoneme sequence. In order to distinguish it from the coding sequence mentioned later, the coding sequence corresponding to the phoneme sequence of the text data is called the first coding sequence, that is, the first coding sequence is obtained by performing feature coding on the phoneme sequence.

For multiple phonemes included in the phoneme sequence, a sub-coding sequence corresponding to each phoneme is generated.

In an example, it is detected whether there is a first phoneme corresponding to each time point, and the first phoneme is any one of the multiple phonemes; the encoding value at the time point where the first phoneme is present is set as the first phoneme. Numerical value, the coding value at the time point without the first phoneme is set to the second numerical value, and the coding sequence corresponding to the first phoneme can be obtained after assigning the coding value at each time point. For example, the code value at the time point when the first phoneme is present may be set to 1, and the code value at the time point when the first phoneme is not present may be set to 0. That is, for each phoneme of the multiple phonemes included in the phoneme sequence, it is detected whether the phoneme corresponds to the phoneme at each time point; the encoding value at the time point where the phoneme is present is set to the first value, and the phoneme is not The encoding value at the time point of the phoneme is set to the second value, and the encoding sequence corresponding to the phoneme can be obtained after assigning the encoding value at each time point. Those with ordinary knowledge in the art should understand that the setting of the encoding value described above is only an example, and the encoding value may also be set to other values, which is not limited in the present disclosure.

The first coding sequence corresponding to the phoneme sequence is obtained according to the respective sub-coding sequences corresponding to the multiple phonemes.

In an example, for the sub-coding sequence corresponding to the first phoneme, a Gaussian filter may be used to perform a Gaussian convolution operation on the continuous values of the first phoneme in time, so as to filter and smooth the matrix corresponding to the feature encoding. The transition of the mouth area when each phoneme is converted.

FIG. 3 shows a schematic diagram of a driving method of an interactive object proposed by at least one embodiment of the present disclosure. As shown in Figure 3, the phoneme sequence 310 contains phonemes j, i1, j, and ie4 (for brevity, only some phonemes are shown), and corresponding sub-coding sequences 321, 322 are obtained for each phoneme j, i1, and ie4. , 323. In each sub-coding sequence, the corresponding code value at the time point with the phoneme is set to the first value (for example, 1), and the corresponding code value at the time point without the phoneme is set to the second value ( For example, 0). Taking the sub-coding sequence 321 as an example, at the time point when there is phoneme j in the phoneme sequence 310, the value of the sub-coding sequence 321 is the first value 1, and at the time point when there is no phoneme j, the value of the sub-coding sequence 321 is the first value. The two value is 0. All the sub-coding sequences constitute the first coding sequence 320.

Next, according to the first coding sequence, a feature code corresponding to at least one phoneme is obtained.

According to the encoding values of the sub-coding sequences 321, 322, and 323 corresponding to phonemes j, i1, and ie4, and the duration of the corresponding phonemes in the three sub-coding sequences, that is, the duration of j in the sub-coding sequence 321, From the duration of i1 in the sub-coding sequence 322 and the duration of ie4 in the sub-coding sequence 323, the characteristic information of the sub-coding sequences 321, 322, and 323 can be obtained.

In an example, a Gaussian filter may be used to perform Gaussian convolution operations on the consecutive values of phonemes j, i1, and ie4 in the sub-encoding sequences 321, 322, and 323, respectively, to smooth the feature encoding to obtain the smoothed的第一coding sequence 330. That is, the Gaussian convolution operation is performed on the continuous value of the phoneme in time through the Gaussian filter, so that the code value in each code sequence changes from the second value to the first value or from the first value to the second value. smooth. For example, in addition to 0 and 1, the values of the coding sequence also present intermediate state values, such as 0.2, 0.3, etc., and the posture control vector obtained according to the values of these intermediate states makes the interactive characters excessively move and change their expressions more smoothly , Naturally, improve the interactive experience of the target object.

In some embodiments, the feature code corresponding to at least one phoneme can be obtained by performing a sliding window on the first code sequence. Wherein, the first coding sequence may be a coding sequence after a Gaussian convolution operation.

A sliding window is performed on the coding sequence with a time window of a set length and a set step size, and the feature code in the time window is used as the feature code of the corresponding at least one phoneme. After the sliding window is completed, according to the obtained multiple A feature code can be used to obtain a second code sequence. Since the duration of each phoneme is different, and the duration of each phoneme is different in proportion to the length of the time window, the number of phonemes corresponding to the feature code in the time window may be 1, 2 or even more depending on the position of the time window. As shown in FIG. 3, by sliding a time window of a set length on the first coding sequence 320 or the smoothed first coding sequence 330, feature code 1, feature code 2, feature code 3 are obtained respectively, and so on, in After traversing the first coding sequence, feature code 1, feature code 2, feature code 3,..., Feature code M are obtained, thereby obtaining a second code sequence 340. Wherein, M is a positive integer, and its value is determined according to the length of the first coding sequence, the length of the time window, and the sliding step of the time window.

Finally, the attitude control vector of at least one partial area of the interactive object corresponding to the feature code is acquired.

According to feature code 1, feature code 2, feature code 3,..., feature code M, the corresponding attitude control vector 1, attitude control vector 2, attitude control vector 3,..., attitude control vector M can be obtained respectively, thereby obtaining attitude control 350 of the sequence of vectors.

The sequence 350 of the attitude control vector and the second coding sequence 340 are aligned in time. Since each feature code in the second coding sequence is obtained according to at least one phoneme in the phoneme sequence, the sequence of the attitude control vector Each control vector in 350 is also obtained based on at least one phoneme in the phoneme sequence. While playing the phoneme sequence corresponding to the text data, the interactive object is driven to make an action according to the sequence of the attitude control vector, that is, the interactive object can be driven to emit the sound corresponding to the text content while being synchronized with the sound The action of, gives the target object the feeling that the interactive object is speaking, and enhances the interactive experience of the target object.

Assuming that the feature code starts to be output at the set time of the first time window, the attitude control vector value before the set time can be set to a preset value, that is, when the phoneme sequence is just started to be played, the interactive object is made to do A preset action is generated, and after the set time, the sequence of the attitude control vector obtained according to the first coding sequence is used to drive the interactive object to make an action. Taking Figure 3 as an example, the feature code 1 starts to be output at time t0, and before time t0 corresponds to the default attitude control vector.

The length of the time window is related to the amount of information contained in the feature code. In the case where the amount of information contained in the time window is relatively large, the processing of the loop neural network will output a more uniform result. If the length of the time window is too large, the expression of the interactive object may not correspond to part of the text; if the length of the time window is too small, the expression of the interactive object may appear rigid when speaking. Therefore, the duration of the time window needs to be determined according to the minimum duration of the phoneme corresponding to the text data, so that the actions and sounds made by driving the interactive objects have a stronger correlation.

The sliding step of the time window is related to the time interval (frequency) of obtaining the attitude control vector, that is, it is related to the frequency of driving the interactive object to make an action. The length and step length of the time window can be set according to the actual interactive scene, so that the expressions and actions made by the interactive object are more closely related to the sound, and are more vivid and natural.

In some embodiments, when the time interval between phonemes in the phoneme sequence is greater than a set threshold, the interactive object is driven to perform an action according to the set attitude control vector of the local area. That is, when the interactive character pauses for a long time, the interactive object is driven to make a set action. For example, when the output voice pauses for a long time, the interactive object can be made to make a smiling expression or slightly swing the body to avoid the interactive object standing upright without expression during the long pause, thereby making the interactive object speak more Natural and smooth, it improves the interactive experience between the target object and the interactive object.

In some embodiments, the feature code can be input to a pre-trained loop neural network, and the loop neural network outputs the interaction corresponding to the feature code according to the first coding sequence. The attitude control vector of at least one local area of the object. Since the loop neural network is a time recurrent neural network, it can learn the historical information of the input feature code, and output the attitude control vector of the at least one local area according to the feature code sequence. Wherein, the characteristic coding sequence includes a first coding sequence and a second coding sequence. The loop neural network may be, for example, a long short-term memory network (Long Short-Term Memory, LSTM).

In the embodiment of the present disclosure, a pre-trained loop neural network is used to obtain the attitude control vector of at least one local area of the interactive object corresponding to the feature code, and the historical feature information of the feature code is merged with the current feature information , So that the historical attitude control vector has an impact on the change of the current attitude control vector, making the expression changes and body movements of the interactive characters more smooth and natural.

In some embodiments, the loop neural network can be trained in the following manner.

First, a feature code sample is obtained, the feature code sample is marked with a true value, and the true value is a posture control vector value of at least one partial area of the interactive object.

After obtaining the feature code samples, the initial loop neural network is trained according to the feature code samples, and the loop neural network is trained after the change of the network loss satisfies the convergence condition, wherein the network The loss includes the difference between the attitude control vector value of the at least one local area predicted by the loop neural network and the actual value.

In some embodiments, feature code samples can be obtained through the following methods.

First, obtain a film segment in which a character speaks, and obtain a plurality of first image frames containing the character according to the film segment. For example, a video segment in which a real person is speaking can be obtained.

Next, extract a corresponding speech segment from the film segment, obtain a sample phoneme sequence according to the speech segment, and perform feature encoding on the sample phoneme sequence. Wherein, the method of encoding the sample phoneme sequence is the same as the encoding method of the phoneme sequence corresponding to the text data described above.

According to the sample code sequence obtained by performing feature coding on the sample phoneme sequence, the feature code of at least one phoneme corresponding to the first image frame is obtained. Wherein, the at least one phoneme may be a phoneme within a set range of the appearance time of the first image frame.

Then, the first image frame is converted into a second image frame containing the interactive object, and the attitude control vector value of at least one local area corresponding to the second image frame is obtained. Wherein, the attitude control vector value may include the attitude control vector value of all the local areas, and may also include the attitude control vector value of some of the local areas.

Taking the first image frame as an image frame containing a real person as an example, the image frame of the real person can be converted into a second image frame containing the image represented by the interactive object, and the image of the real person The posture control vector of each local area corresponds to the posture control vector of each local area of the interactive object, so that the posture control vector of each local area of the interactive object in the second image frame can be obtained.

Finally, the feature code of at least one phoneme corresponding to the first image frame obtained above is annotated according to the attitude control vector value to obtain feature code samples.

In the embodiment of the present disclosure, the movie segment of a character is split into a plurality of corresponding first image frames and voice segments, and the first image frame containing the real person is transformed into the first image frame containing the interactive object. Two image frames are used to obtain the attitude control vector corresponding to the feature code of the phoneme, so that the feature code and the attitude control vector correspond better, so as to obtain high-quality feature code samples, making the action of the interactive object closer to the reality of the corresponding character action.

FIG. 4 shows a schematic structural diagram of a driving device for an interactive object according to at least one embodiment of the present disclosure. As shown in FIG. 4, the device may include: a first obtaining unit 401, configured to obtain a phoneme sequence corresponding to a text data; and second The obtaining unit 402 is configured to obtain the control parameter value of at least one partial area of the interactive object matching the phoneme sequence; the driving unit 403 is configured to control the posture of the interactive object according to the obtained control parameter value.

In some embodiments, the device further includes an output unit for controlling the display device displaying the interactive object to display text according to the text data, and/or controlling the display device according to the phoneme sequence corresponding to the text data Output speech.

In some embodiments, the second obtaining unit is specifically configured to: perform feature coding on the phoneme sequence to obtain a first coding sequence corresponding to the phoneme sequence; and obtain at least one phoneme corresponding to the phoneme sequence according to the first coding sequence. The feature code; obtaining the attitude control vector of at least one partial area of the interactive object corresponding to the feature code.

In some embodiments, when performing feature encoding on the phoneme sequence to obtain the first encoding sequence corresponding to the phoneme sequence, the second acquiring unit is specifically configured to: generate, for multiple phonemes included in the phoneme sequence, A sub-coding sequence corresponding to each phoneme; and obtaining the first coding sequence corresponding to the phoneme sequence according to the sub-coding sequences corresponding to the multiple phonemes respectively.

In some embodiments, when generating the sub-coding sequence corresponding to each phoneme for the multiple phonemes included in the phoneme sequence, the second acquiring unit is specifically configured to: detect whether there is a first phoneme corresponding to each time point, The first phoneme is any one of the multiple phonemes; by setting the code value at the time point when the first phoneme is present to the first value, the code value at the time point without the first phoneme is set Set to the second value to obtain the sub-coding sequence corresponding to the first phoneme.

In some embodiments, the device further includes a filtering unit for performing a Gaussian filter on the continuous value of the phoneme in time by using a Gaussian filter for the sub-coding sequence corresponding to each phoneme of the multiple phonemes. Convolution operation. In an embodiment, for the sub-coding sequence corresponding to the first phoneme, a Gaussian filter is used to perform a Gaussian convolution operation on the continuous values of the first phoneme in time, and the first phoneme is one of the multiple phonemes. Any kind.

In some embodiments, when acquiring the feature code corresponding to at least one phoneme according to the first encoding sequence, the second acquiring unit is specifically configured to: The sequence performs a sliding window, the feature code in the time window is used as the feature code of the corresponding at least one phoneme, and the second code sequence is obtained according to the multiple feature codes obtained by completing the sliding window.

In some embodiments, the driving unit is specifically configured to: obtain a sequence of a posture control vector corresponding to the second coding sequence; and control the posture of the interactive object according to the sequence of the posture control vector.

In some embodiments, the device further includes a pause driving unit, which is used to control the set control parameter value of the local area when the time interval between phonemes in the phoneme sequence is greater than a set threshold. The posture of the interactive object.

In some embodiments, when acquiring the attitude control vector of at least one partial region of the interactive object corresponding to the feature code, the second acquiring unit is specifically configured to: input the feature code into a pre-trained response. A loop neural network to obtain a posture control vector of at least one local area of the interactive object corresponding to the feature code.

In some embodiments, the neural network is obtained through phoneme sequence sample training; the device further includes a sample acquisition unit for: acquiring a video segment of a character's voice, and acquiring a plurality of video segments containing all of them according to the video segment. The first image frame of the character; extract the corresponding speech segment from the film segment, obtain a sample phoneme sequence according to the speech segment, and perform feature encoding on the sample phoneme sequence; obtain the first image Feature encoding of at least one phoneme corresponding to the frame; transforming the first image frame into a second image frame containing the interactive object, and obtaining a posture control vector of at least one local area corresponding to the second image frame Value; according to the attitude control vector value, annotate the feature code corresponding to the first image frame to obtain a feature code sample.

In some embodiments, the device further includes a training unit for training the initial loop neural network according to the characteristic coding samples, and trains to obtain the loop neural network after the change of the network loss satisfies the convergence condition Wherein, the network loss includes the difference between the attitude control vector value of the at least one local area and the labeled attitude control vector value predicted by the loop neural network.

At least one embodiment of this specification also provides an electronic device. As shown in FIG. 5, the device includes a memory and a processor. The memory is used to store computer instructions that can run on the processor. The computer instruction is used to implement the driving method of the interactive object described in any embodiment of the present disclosure.

At least one embodiment of this specification also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method for driving an interactive object according to any embodiment of the present disclosure is realized.

Those with ordinary knowledge in the art should understand that one or more embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, one or more embodiments of this specification may adopt the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification can be implemented on one or more computer-usable storage media (including but not limited to magnetic disk memory, CD-ROM, optical memory, etc.) containing computer-usable program codes. In the form of a computer program product.

The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, as for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment.

The foregoing describes specific embodiments of this specification. Other embodiments are within the scope of the attached patent application. In some cases, the actions or steps described in the scope of the patent application may be performed in a different order from the embodiment and still achieve desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown in order to achieve the desired results. In some embodiments, multiplexing and parallel processing are also possible or may be advantageous.

The embodiments of the subject and functional operations described in this specification can be implemented in the following: digital electronic circuits, tangible computer software or hardware, computer hardware including the structure disclosed in this specification and its structural equivalents, or A combination of one or more of them. The embodiments of the subject described in this specification can be implemented as one or more computer programs, that is, one or one of the computer program instructions encoded on a tangible non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device Multiple modules. Alternatively or additionally, the program instructions may be encoded on artificially generated propagated signals, such as machine-generated electrical, optical or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiver device for data transmission. The processing device executes. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The processing and logic flow described in this manual can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating based on input data and generating output. The processing and logic flow can also be executed by a dedicated logic circuit, such as FPGA (Field Programmable Gate Array) or ASIC (Dedicated Integrated Circuit), and the device can also be implemented as a dedicated logic circuit.

Computers suitable for executing computer programs include, for example, general-purpose and/or special-purpose microprocessors, or any other types of central processing units. Generally, the central processing unit will receive commands and data from read-only memory and/or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Usually, the computer will also include one or more large storage area devices for storing data, such as floppy disks, magneto-optical disks, or optical discs, or the computer will be operably coupled to this large storage area device to receive data or Send data to it, or both. However, the computer does not have to have such equipment. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a universal serial bus (USB) ) Portable storage devices for flash drives, to name a few.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (such as EPROM, EEPROM and flash memory devices), magnetic disks (Such as internal hard drives or flash drives), magneto-optical discs, and CD-ROM and DVD-ROM. The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

Although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claimed protection, but are mainly used to describe the features of specific embodiments of a particular invention. Certain features described in multiple embodiments in this specification can also be implemented in combination in a single embodiment. On the other hand, various features described in a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although features can function in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination can in some cases be removed from the combination, and the claimed The combination of protection can be directed to a sub-combination or a variant of the sub-combination.

Similarly, although operations are depicted in a specific order in the drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve desired results . In some cases, multiplexing and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can usually be integrated together in a single software product. Or packaged into multiple software products.

Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the attached patent application. In some cases, the actions described in the scope of the patent application can be executed in a different order and still achieve the desired result. In addition, the processes depicted in the drawings are not necessarily in the specific order or sequential order shown in order to achieve the desired result. In some implementations, multiplexing and parallel processing may be advantageous.

The above descriptions are only preferred embodiments of one or more embodiments of this specification, and are not intended to limit one or more embodiments of this specification. All within the spirit and principle of one or more embodiments of this specification, Any modification, equivalent replacement, improvement, etc. made should be included in the protection scope of one or more embodiments of this specification.

201, 202, 203: steps 401: The first acquisition unit 402: second acquisition unit 403: drive unit

In order to more clearly explain one or more embodiments of this specification or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the appendix in the following description The figures are only some of the embodiments described in one or more embodiments of this specification. For those with ordinary knowledge in the art, other figures can be obtained based on these figures without progressive labor. FIG. 1 is a schematic diagram of a display device in a driving method of an interactive object provided by at least one embodiment of the present disclosure. FIG. 2 is a flowchart of a method for driving an interactive object according to at least one embodiment of the present disclosure. FIG. 3 is a schematic diagram of the process of feature encoding for a phoneme sequence proposed by at least one embodiment of the present disclosure. FIG. 4 is a schematic structural diagram of a driving device for an interactive object provided by at least one embodiment of the present disclosure. FIG. 5 is a schematic structural diagram of an electronic device proposed in at least one embodiment of the present disclosure.

201, 202, 203: steps

Claims (14)

A driving method of interactive objects includes: Obtain the phoneme sequence corresponding to the text data; Acquiring a control parameter value of at least one partial area of an interactive object matching the phoneme sequence; Control the posture of the interactive object according to the acquired control parameter value. The method according to claim 1, further comprising: controlling the display device displaying the interactive object to display text according to the text data, and/or controlling the display device to output voice according to the phoneme sequence corresponding to the text data. The method according to claim 1 or 2, wherein the control parameter of the local area of the interactive object includes the attitude control vector of the local area; Obtaining the control parameter value of at least one partial area of the interactive object matching the phoneme sequence includes: Performing feature encoding on the phoneme sequence to obtain a first encoding sequence corresponding to the phoneme sequence; Obtaining a feature code corresponding to at least one phoneme according to the first coding sequence; Obtain a posture control vector of at least one partial area of the interactive object corresponding to the feature code. The method according to claim 3, wherein performing feature encoding on the phoneme sequence to obtain the first encoding sequence corresponding to the phoneme sequence includes: For each phoneme of the multiple phonemes included in the phoneme sequence, generating a sub-coding sequence corresponding to the phoneme; The first coding sequence corresponding to the phoneme sequence is obtained according to the respective sub-coding sequences corresponding to the multiple phonemes. The method according to claim 4, wherein for each of the multiple phonemes included in the phoneme sequence, generating a sub-coding sequence corresponding to the phoneme includes: Detecting whether the phoneme corresponds to each time point; By setting the coding value at the time point with the phoneme to the first value, and setting the coding value at the time point without the phoneme to the second value, the sub-coding sequence corresponding to the phoneme is obtained. The method described in claim 5 further includes: For the sub-coding sequence corresponding to each phoneme of the multiple phonemes, a Gaussian convolution operation is performed on the continuous values of the phoneme in time by using a Gaussian filter. The method according to any one of claim items 3 to 6, wherein obtaining a feature code corresponding to at least one phoneme according to the first coding sequence includes: A sliding window is performed on the first coding sequence with a time window of a set length and a set step size, the feature code in the time window is used as the feature code of the corresponding at least one phoneme, and the sliding is performed according to the completion of the sliding window. A plurality of the feature codes obtained by the window to obtain a second coding sequence; Controlling the posture of the interactive object according to the acquired control parameter value includes: Acquiring a sequence of attitude control vectors corresponding to the second coding sequence; Control the posture of the interactive object according to the sequence of the posture control vector. The method according to any one of claims 1 to 7, further comprising: In the case that the time interval between the phonemes in the phoneme sequence is greater than a set threshold, the posture of the interactive object is controlled according to the set control parameter value of the local area. The method according to claim 3, wherein obtaining a posture control vector of at least one partial area of the interactive object corresponding to the feature code includes: The feature code is input to a pre-trained loop neural network to obtain the attitude control vector of at least one local area of the interactive object corresponding to the feature code. The method according to claim 9, wherein the loop neural network is obtained through feature coding sample training, and the method further includes: Acquiring a film segment in which a character speaks, and acquiring a plurality of first image frames containing the character according to the film segment; Extracting a corresponding speech segment from the film segment, obtaining a sample phoneme sequence according to the speech segment, and performing feature encoding on the sample phoneme sequence; Acquiring a feature code of at least one phoneme corresponding to the first image frame; Transforming the first image frame into a second image frame containing the interactive object, and obtaining a posture control vector value of at least one local area corresponding to the second image frame; According to the attitude control vector value, annotate the feature code corresponding to the first image frame to obtain the feature code sample. The method described in claim 10 further includes: The initial loop neural network is trained according to the feature code samples, and the loop neural network is trained after the change of the network loss satisfies the convergence condition, wherein the network loss includes the loop neural network The difference between the attitude control vector value of the at least one local area obtained by road prediction and the marked attitude control vector value. A driving device for interactive objects includes: The first acquiring unit is used to acquire the phoneme sequence corresponding to the text data; The second acquiring unit is configured to acquire the control parameter value of at least one partial area of the interactive object matching the phoneme sequence; The driving unit is configured to control the posture of the interactive object according to the acquired control parameter value. An electronic device comprising a memory and a processor, the memory is used to store computer instructions that can be run on the processor, and the processor is used to implement any one of request items 1 to 11 when the computer instructions are executed. The method described. A computer-readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the method described in any one of request items 1 to 11 is realized.

Images