TWI760015B - Method and apparatus for driving interactive object, device and storage medium - Google Patents

Abstract

The present disclosure relates to a method and an apparatus for driving interactive object, a device and a storage medium. The interactive object is displayed in a display device, and the method comprises: obtaining driving data of the interactive object and determining a driving mode of the driving data; in response to the driving mode, obtaining a value of control parameter of the interactive object according to the driving data; and controlling a posture of the interactive object according to the value of control parameter.

Description

Driving method, device, device and storage medium for interactive objects

The present disclosure relates to the field of computer technology, and in particular, to a driving method, apparatus, device and storage medium for interactive objects.

Most of the human-computer interaction methods are based on keystrokes, touch, and voice input, and respond by presenting images, texts or virtual characters on the display screen. At present, virtual characters are mostly improved on the basis of voice assistants, which only output the voice of the device.

Embodiments of the present disclosure provide a driving solution for an interactive object.

According to an aspect of the present disclosure, there is provided a method for driving an interactive object, where the interactive object is displayed on a display device, the method comprising: acquiring driving data of the interactive object, and determining a driving mode of the driving data; responding to In the driving mode, the control parameter value of the interactive object is acquired according to the driving data; the posture of the interactive object is controlled according to the control parameter value.

With reference to any of the embodiments provided in the present disclosure, the method further includes: controlling the display device to output voice and/or display text according to the driving data.

With reference to any implementation manner provided in the present disclosure, the determining of the driving mode corresponding to the driving data includes: acquiring, according to the type of the driving data, a voice data sequence corresponding to the driving data, where the voice data sequence includes multiple a voice data unit; in response to detecting that the voice data unit includes target data, determine that the driving mode of the driving data is the first driving mode, and the target data is consistent with the preset control parameter value of the interactive object Corresponding; the acquiring, in response to the driving mode, the control parameter value of the interactive object according to the driving data includes: in response to the first driving mode, converting the preset control parameter corresponding to the target data to value, as the control parameter value of the interactive object.

With reference to any of the implementation manners provided in the present disclosure, the target data includes keywords or keywords, and the keywords or the keywords correspond to preset control parameter values of the setting action of the interactive object; or, the The target data includes syllables, and the syllables correspond to preset control parameter values of the interactive object for setting the mouth shape action.

With reference to any implementation manner provided in the present disclosure, the determining of the driving mode corresponding to the driving data includes: acquiring, according to the type of the driving data, a voice data sequence corresponding to the driving data, where the voice data sequence includes multiple a voice data unit; if it is not detected that the voice data unit includes target data, the driving mode of the driving data is determined to be the second driving mode, and the target data is consistent with the preset control parameter value of the interactive object. correspond. In response to the driving mode, acquiring the control parameter value of the interactive object according to the driving data includes: in response to the second driving mode, acquiring feature information of at least one voice data unit in the voice data sequence; Obtain the control parameter value of the interactive object corresponding to the feature information.

With reference to any of the embodiments provided in the present disclosure, the voice data sequence includes a phoneme sequence, and the acquiring feature information of at least one voice data unit in the voice data sequence includes: performing feature encoding on the phoneme sequence to obtain The first coding sequence corresponding to the phoneme sequence; according to the first coding sequence, a feature code corresponding to at least one phoneme is obtained; according to the feature code, feature information of the at least one phoneme is obtained.

With reference to any of the embodiments provided in the present disclosure, the voice data sequence includes a voice frame sequence, and the acquiring feature information of at least one voice data unit in the voice data sequence includes: acquiring the first voice data unit corresponding to the voice frame sequence. an acoustic feature sequence, the first acoustic feature sequence includes an acoustic feature vector corresponding to each speech frame in the speech frame sequence; according to the first acoustic feature sequence, acquire an acoustic feature vector corresponding to at least one speech frame ; According to the acoustic feature vector, obtain feature information corresponding to the at least one speech frame.

With reference to any of the embodiments provided in the present disclosure, the control parameter of the interactive object includes a facial posture parameter, the facial posture parameter includes a facial muscle control coefficient, and the facial muscle control coefficient is used to control the motion state of at least one facial muscle; The obtaining the control parameter value of the interactive object according to the driving data includes: obtaining the facial muscle control coefficient of the interactive object according to the driving data; controlling the posture of the interactive object according to the control parameter value , which includes: driving the interactive object to perform facial actions matching the driving data according to the acquired facial muscle control coefficients.

In conjunction with any of the embodiments provided in the present disclosure, the method further includes: acquiring driving data of a body posture associated with the facial posture parameter; driving the body posture according to the driving data of the body posture associated with the facial posture parameter value Interactive objects make body movements.

With reference to any of the implementation manners provided in the present disclosure, the control parameter value of the interactive object includes a control vector of at least one local area of the interactive object; and the acquiring the control parameter value of the interactive object according to the driving data includes: : obtaining a control vector of at least one partial area of the interactive object according to the driving data; the controlling the posture of the interactive object according to the control parameter value includes: controlling the at least one partial area according to the obtained A vector that controls the facial and/or body movements of the interactive object.

With reference to any of the embodiments provided in the present disclosure, the obtaining the control parameter value of the interactive object corresponding to the feature information includes: inputting the feature information into a pre-trained recurrent neural network, and obtaining the value of the control parameter corresponding to the feature information. The control parameter value of the interactive object corresponding to the feature information.

According to an aspect of the present disclosure, an apparatus for driving an interactive object is provided, the interactive object is displayed in a display device, the apparatus includes: a first acquiring unit, configured to acquire driving data of the interactive object, and determine the a driving mode of the driving data; a second acquiring unit, configured to respond to the driving mode and acquire a control parameter value of the interactive object according to the driving data; a driving unit, configured to control the interaction according to the control parameter value The pose of the object.

According to an aspect of the present disclosure, an electronic device is provided, the device includes a memory and a processor, the memory is used to store computer instructions that can be executed on the processor, and the processor is used to execute the computer instructions. At the same time, the driving method of the interactive object described in any one of the implementation manners provided by the present disclosure is implemented.

According to an aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the method for driving an interactive object according to any one of the implementation manners provided in the present disclosure.

The method, apparatus, device, and computer-readable storage medium for driving an interactive object according to one or more embodiments of the present disclosure acquire control parameter values of the interactive object according to the driving mode of the driving data of the interactive object, so as to control the The gesture of the interactive object, wherein, for different driving modes, the control parameter values of the corresponding interactive object can be obtained in different ways, so that the interactive object displays the content of the driving data and/or the corresponding voice matching. gesture, so that the target object feels that it is communicating with the interactive object, which improves the interactive experience between the target object and the interactive object.

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

The term "and/or" in this article is only a relationship to describe related objects, which means that there can be three relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. three situations. In addition, the term "at least one" herein refers to any combination of any one of a plurality or at least two of a plurality, for example, including at least one of A, B, and C, and may mean including those composed of A, B, and C. Any one or more elements selected in the collection.

At least one embodiment of the present disclosure provides a method for driving an interactive object. The driving method can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a fixed terminal or a mobile terminal, such as a mobile phone, a tablet computer, a game computer, desktop computer, advertising machine, all-in-one computer, vehicle-mounted terminal, etc., the server includes a local server or a cloud server, etc., the method can also call the computer-readable instructions stored in the memory through the processor. to fulfill.

In the embodiment of the present disclosure, the interactive object may be any virtual image capable of interacting with the target object. In one embodiment, the interactive object may be a virtual character, and may also be a virtual animal, a virtual item, a cartoon image, or other virtual images capable of realizing interactive functions. The presentation form of the interactive object can be either a 2D form or a 3D form, which is not limited in the present disclosure. The target object may be a user, a robot, or other smart devices. The interaction mode between the interactive object and the target object may be an active interaction mode or a passive interaction mode. In one example, the target object can issue a demand by making gestures or body movements, and trigger the interactive object to interact with it through active interaction. In another example, the interactive object may actively say hello, prompt the target object to make an action, etc., so that the target object interacts with the interactive object in a passive manner.

The interactive object can be displayed by a terminal device, and the terminal device can be a TV, an all-in-one machine with a display function, a projector, a virtual reality (Virtual Reality, VR) device, an augmented reality (Augmented Reality, AR) equipment, etc., the present disclosure does not limit the specific form of the terminal equipment.

FIG. 1 illustrates a display device according to at least one embodiment of the present disclosure. As shown in FIG. 1 , the display device has a transparent display screen, and a stereoscopic image can be displayed on the transparent display screen, so as to present a virtual scene with a stereoscopic effect and interactive objects. For example, the interactive objects displayed on the transparent display screen in FIG. 1 include virtual cartoon characters. In some embodiments, the terminal device described in the present disclosure may also be the above-mentioned display device with a transparent display screen. The display device is configured with a memory and a processor, and the memory is used to store a computer that can run on the processor. The processor is configured to implement the driving method of the interactive object provided by the present disclosure when executing the computer instruction, so as to drive the interactive object displayed on the transparent display screen to communicate or respond to the target object.

In some embodiments, in response to the driving data for driving the interactive object to output the voice, the interactive object can emit the specified voice to the target object. The terminal device can generate driving data according to the actions, expressions, identities, preferences, etc. of the target objects around the terminal device, so as to drive the interactive objects to communicate or respond by issuing specified voices, thereby providing anthropomorphic services for the target objects. It should be noted that the sound-driven data can also be generated in other ways, for example, generated by a server and sent to a terminal device.

During the interaction between the interactive object and the target object, when the interactive object is driven to emit a specified voice according to the driving data, the interactive object may not be driven to make facial movements synchronized with the specified voice, so that the interactive object is rigid and rigid when it emits a voice. Unnatural, affecting the interactive experience between the target object and the interactive object. Based on this, at least one embodiment of the present disclosure provides a method for driving an interactive object, so as to improve the experience of interacting between a target object and an interactive object.

FIG. 2 is a flow chart of a method for driving an interactive object according to at least one embodiment of the present disclosure. The interactive object is displayed on a display device. As shown in FIG. 2 , the method includes steps 201 to 203 .

In step 201, the driving data of the interactive object is acquired, and the driving mode of the driving data is determined.

In an embodiment of the present disclosure, the driving data may include audio data (voice data), text, and the like. The driving data may be the driving data generated by the server or the terminal device according to the action, expression, identity, preference, etc. of the target object interacting with the interactive object, or it may be obtained directly by the terminal device, such as calling from the internal memory. drive data, etc. The present disclosure does not limit the manner of acquiring the driving data.

According to the type of the driving data and the information contained in the driving data, the driving mode of the driving data can be determined.

In an example, a voice data sequence corresponding to the driving data may be acquired according to the type of the driving data, wherein the voice data sequence includes a plurality of voice data units. Wherein, the voice data unit may be composed of words or words, or may be composed of phonemes or syllables. Corresponding to the driving data of the text type, the word sequence, word sequence, etc. corresponding to the driving data can be obtained; corresponding to the driving data of the audio type, the phoneme sequence, syllable sequence, and speech frame corresponding to the driving data can be obtained. sequence and so on. In one embodiment, audio data and text data can be converted to each other. For example, the division of voice data units is performed after converting audio data into text data, or the division of voice data units is performed after converting text data into audio data, which is not limited in the present disclosure.

When it is detected that the voice data unit includes target data, it can be determined that the driving mode of the driving data is the first driving mode, and the target data corresponds to the preset control parameter value of the interactive object.

The target data may be a set keyword or a keyword, etc., and the keyword or the keyword corresponds to a preset control parameter value of the setting action of the interactive object.

In the embodiment of the present disclosure, each target data is pre-matched with a set action, and each set action is controlled by a corresponding control parameter value, so each target data matches the control parameter value of the set action. Taking the keyword "waving hand" as an example, in the case where the voice data unit includes "waving hand" in text form and/or "waving hand" in voice form, it can be determined that the driving data includes target data. .

Exemplarily, the target data includes syllables, and the syllables correspond to the preset control parameter values of the interactive object for setting the lip movement.

The syllable corresponding to the target data belongs to a pre-divided syllable type, and the one syllable type matches a preset mouth shape. Wherein, a syllable is a phonetic unit formed by a combination of at least one phoneme, and the syllable includes a syllable in a pinyin language and a syllable in a non-pinyin language (eg, Chinese). A syllable type refers to a syllable whose pronunciation action is consistent or basically consistent, and a syllable type can correspond to an action of an interactive object. In one embodiment, a syllable type may correspond to a set mouth shape when the interactive object speaks, that is, to a pronunciation action. In this way, different syllable types are matched with different control parameter values for setting the mouth shape. For example, the pinyin "ma", "man", "mang" type syllables, because the pronunciation actions of these syllables are basically the same, it can be regarded as are of the same type, and can correspond to the control parameter values of the mouth shape of the "mouth open" when the interactive object speaks.

If it is not detected that the voice data unit includes target data, it may be determined that the driving mode of the driving data is the second driving mode, and the target data corresponds to the preset control parameter value of the interactive object.

Those skilled in the art should understand that the above-mentioned first driving mode and second driving mode are only used as examples, and the embodiment of the present disclosure does not limit the specific driving mode.

In step 202, in response to the driving mode, a control parameter value of the interactive object is acquired according to the driving data.

For various driving modes of the driving data, the control parameter value of the interactive object may be obtained in a corresponding manner.

In an example, in response to the first driving mode determined in step 201, the preset control parameter value corresponding to the target data may be used as the control parameter value of the interactive object. For example, for the first driving mode, the preset control parameter value corresponding to the target data (for example, "waving hand") included in the voice data sequence may be used as the control parameter value of the interactive object.

In an example, in response to the second driving mode determined in step 201, feature information of at least one voice data unit in the voice data sequence may be acquired; control parameters of the interactive object corresponding to the feature information may be acquired value. That is, in the case where the target data is not detected in the speech data sequence, the corresponding control parameter value can be obtained according to the characteristic information of the speech data unit. The feature information may include feature information of the voice data unit obtained by feature encoding the voice data sequence, feature information of the voice data unit obtained according to the acoustic feature information of the voice data sequence, and the like.

In step 203, the posture of the interactive object is controlled according to the control parameter value.

In some embodiments, the control parameter of the interactive object includes a facial posture parameter, and the facial posture parameter includes a facial muscle control coefficient, and the facial muscle control coefficient is used to control the motion state of at least one facial muscle. In one embodiment, the facial muscle control coefficient of the interactive object may be acquired according to the driving data; and according to the acquired facial muscle control coefficient, the interactive object may be driven to perform a facial action matching the driving data.

In some embodiments, the control parameter value of the interactive object includes a control vector of at least one local area of the interactive object. In one embodiment, the control vector of at least one local area of the interactive object can be acquired according to the driving data; and the facial action and / or body movements.

The control parameter value of the interactive object is obtained according to the driving mode of the driving data of the interactive object, so as to control the posture of the interactive object, wherein for different driving modes, the corresponding interactive object can be obtained in different ways The control parameter value of , so that the interactive object displays a posture that matches the content of the driving data and/or the corresponding voice, so that the target object feels that it is communicating with the interactive object, and the interactive experience between the target object and the interactive object is improved. .

In some embodiments, the display device may also be controlled to output voice and/or display text according to the driving data. And while outputting voice and/or displaying text, the gesture of the interactive object can be controlled according to the control parameter value.

In the embodiment of the present disclosure, since the control parameter value matches the driving data, in the case where the output of the voice and/or the displayed text according to the driving data is synchronized with the gesture of controlling the interactive object according to the control parameter value, the interaction The gestures made by the object are also synchronized with the output speech and/or displayed text, thereby giving the target object a sense that the interactive object is communicating with it.

In some embodiments, the sequence of speech data includes a sequence of phonemes. In response to the drive data including audio data, the audio data can be split into a plurality of audio frames, and the audio frames can be combined according to the state of the audio frames to form phonemes; each phoneme formed according to the audio data forms a phoneme. phoneme sequence. Among them, the phoneme is the smallest phonetic unit divided according to the natural attributes of the voice, and one pronunciation action of a real person can form a phoneme. In response to the drive data being text, the phoneme corresponding to the morpheme may be obtained according to the morpheme contained in the text, thereby obtaining a corresponding phoneme sequence.

In some embodiments, the feature information of at least one voice data unit in the voice data sequence may be obtained by: performing feature coding on the phoneme sequence to obtain a first coding sequence corresponding to the phoneme sequence; The first coding sequence is used to obtain the feature code corresponding to at least one phoneme; and the feature information of the at least one phoneme is obtained according to the feature code.

FIG. 3 is a schematic diagram illustrating a process of feature encoding for a phoneme sequence. As shown in FIG. 3 , the phoneme sequence 310 includes phonemes j, i1, j, and ie4 (for the sake of brevity, only some phonemes are shown), and corresponding coding sequences 321, 322, 322, 322, 323. In each encoding sequence, the corresponding encoding value at the time point with the phoneme is set to a first value (for example, 1), and the corresponding encoding value at the time point without the phoneme is set to the second value (for example, is 0). Taking the coding sequence 321 as an example, at the time point when the phoneme sequence 310 has phoneme j, the value of the coding sequence 321 is set to the first value 1; at the time point when there is no phoneme j, the value of the coding sequence 321 is set to the second value. The value is 0. All coding sequences 321 , 322 , 323 constitute the total coding sequence 320 .

According to the coding values of the coding sequences 321, 322 and 323 corresponding to the phonemes j, i1 and ie4 respectively, and the durations of the corresponding phonemes in the three coding sequences, that is, the duration of j in the coding sequence 321, the duration of the coding sequence in the coding sequence The duration of i1 in 322 and the duration of ie4 in the coding sequence 323 can obtain the characteristic information of the coding sequences 321 , 322 and 323 .

For example, Gaussian convolution operation can be performed on successive temporal values of phonemes j, i1, ie4 in the encoded sequences 321, 322, 323 using a Gaussian filter to obtain the feature information of the encoded sequences. That is, the Gaussian convolution operation is performed on the continuous values of the phoneme in time through the Gaussian filter, so that the change stage of the coding value in each coding sequence from the second value to the first value or from the first value to the second value becomes smooth. The Gaussian convolution operation is performed on each coding sequence 321, 322, 323, respectively, so as to obtain the feature value of each coding sequence, wherein the feature value is a parameter constituting the feature information, and the set of feature information of each coding sequence is obtained. The feature information 330 corresponding to the phoneme sequence 310 . Those skilled in the art should understand that other operations may also be performed on each coding sequence to obtain characteristic information of the coding sequence, which is not limited in the present disclosure.

In the embodiment of the present disclosure, the characteristic information of the coding sequence is obtained according to the duration of each phoneme in the phoneme sequence, so that the change phase of the coding sequence is smooth. For example, the values of the coding sequence show an intermediate state except for 0 and 1. value, such as 0.2, 0.3, etc., and the attitude parameter values obtained according to the values of these intermediate states make the changes of the interactive characters' poses more gentle and natural, especially the changes of the interactive characters' expressions are more gentle and natural, and improve the The interactive experience of the target object.

In some embodiments, the facial pose parameters may include facial muscle control coefficients.

The movement of the human face, from an anatomical point of view, is the result of cooperative deformation of the muscles of various parts of the face. Therefore, the facial muscle model is obtained by dividing the facial muscles of the interactive object, and the movement of each muscle (region) obtained by the division is controlled by the corresponding facial muscle control coefficient, that is, the contraction/expansion control is performed on it, then It can make the faces of interactive characters make various expressions. For each muscle of the facial muscle model, motion states corresponding to different muscle control coefficients can be set according to the facial position where the muscle is located and the motion characteristics of the muscle itself. For example, for the upper lip muscle, the value range of its control coefficient is 0~1. Different values in this range correspond to different contraction/expansion states of the upper lip muscle. By changing the value, the longitudinal opening and closing of the mouth can be realized; For the muscle of the left corner of the mouth, the value of the control coefficient ranges from 0 to 1. Different values in this range correspond to the contraction/expansion state of the muscle at the left corner of the mouth. By changing the value, the lateral change of the mouth can be achieved.

While outputting the sound according to the phoneme sequence, the interactive object is driven to make a facial expression according to the facial muscle control coefficient corresponding to the phoneme sequence, so that when the display device outputs the sound, the interactive object can make the sound synchronously , so that the target object feels that the interactive object is talking, and the interactive experience of the target object is improved.

In some embodiments, the facial action of the interactive object may be associated with a body posture, that is, the facial posture parameter value corresponding to the facial action may be associated with the body posture, and the body posture may include body movements, Gestures, walking gestures, etc.

During the driving process of the interactive object, obtain the driving data of the body posture associated with the facial posture parameter value; while outputting the sound according to the phoneme sequence, according to the driving data of the body posture associated with the facial posture parameter value , and drive the interactive object to make body movements. That is, while driving the interactive object to make a facial action according to the driving data of the interactive object, the driving data of the associated body posture is also obtained according to the facial posture parameter value corresponding to the facial action, so that when the sound is output, , which can drive the interactive objects to make corresponding facial and body movements synchronously, so that the speaking state of the interactive objects is more vivid and natural, and the interactive experience of the target objects is improved.

Since the output of sound needs to maintain continuity, in one embodiment, the time window is moved on the phoneme sequence, and the phonemes in the time window during each movement are output, wherein the set duration is used as the time of each movement The step size of the window. For example, the length of the time window can be set to 1 second, and the set duration can be set to 0.1 seconds. While outputting the phonemes in the time window, obtain the gesture parameter value corresponding to the phoneme or the feature information of the phoneme at the set position of the time window, and use the gesture parameter value to control the gesture of the interactive object; the set position is the distance time The starting position of the window is the position where the duration is set. For example, when the length of the time window is set to 1s, the distance between the set position and the starting position of the time window can be 0.5s. With each movement of the time window, while outputting the phonemes in the time window, the posture of the interactive object is controlled by the corresponding attitude parameter value at the set position of the time window, so that the posture of the interactive object is synchronized with the output voice, giving the target The object feels like the interactive object is talking.

By changing the set duration, you can change the time interval (frequency) for obtaining the attitude parameter value, thereby changing the frequency of the interactive object making the gesture. The set duration can be set according to the actual interactive scene, so as to make the posture change of the interactive object more natural.

In some embodiments, the gesture of the interactive object can be controlled by obtaining a control vector of at least one local area of the interactive object.

The local area is obtained by dividing the whole (including face and/or body) of the interactive object. The control of one or more local areas of the face can correspond to a series of facial expressions or actions of the interactive object. For example, the control of the eye area can correspond to the facial actions of the interactive object such as opening, closing, blinking, and changing the perspective; another example is the mouth. The control of the facial area can correspond to the facial actions of the interactive object, such as closing the mouth and opening the mouth to different degrees. The control of one or more local regions of the body may correspond to a series of body actions of the interactive object, for example, the control of the leg region may correspond to the actions of the interactive object such as walking, jumping, and kicking.

The control parameters of the local area of the interactive object include the control vector of the local area. The gesture control vector of each local area is used to drive the local area of the interactive object to act. Different control vector values correspond to different motions or motion magnitudes. For example, for the control vector of the mouth region, one set of control vector values can make the mouth of the interactive object slightly open, and another set of control vector values can make the mouth of the interactive object open wide. By driving the interactive objects with different control vector values, the corresponding local regions can be made to perform different actions or actions of different magnitudes.

The local area can be selected according to the action of the interactive object that needs to be controlled. For example, when the face and limbs of the interactive object need to be controlled to move at the same time, the control vectors of all local areas can be obtained; when the expression of the interactive object needs to be controlled, Then the control vector of the local area corresponding to the face can be obtained.

In some embodiments, the feature code corresponding to at least one phoneme may be obtained by performing a sliding window on the first code sequence. Wherein, the first coding sequence may be a coding sequence after Gaussian convolution operation.

With a time window of a set length and a set step size, a sliding window is performed on the coding sequence, and the feature coding in the time window is used as the feature coding of the corresponding at least one phoneme, and after the sliding window is completed, according to the obtained multiple A feature code can be obtained to obtain the second code sequence. As shown in FIG. 4 , by sliding a time window of a set length on the first encoding sequence 420 or the smoothed first encoding sequence 430, feature encoding 1, feature encoding 2, and feature encoding 3 are obtained respectively, and so on. After traversing the first coding sequence, feature coding 1, feature coding 2, feature coding 3, . . . , feature coding M are obtained, thereby obtaining the second coding sequence 440. Wherein, M is a positive integer, and its value is determined according to the length of the first coding sequence, the length of the time window, and the step size of the time window sliding.

Corresponding control vector 1, control vector 2, control vector 3, .

The sequence 450 of control vectors and the second coding sequence 440 are aligned in time. Since each feature code in the second coding sequence is obtained from at least one phoneme in the phoneme sequence, the sequence 450 of control vectors is Each feature vector of is also obtained from at least one phoneme in the phoneme sequence. While playing the phoneme sequence corresponding to the text data, the interactive object is driven to act according to the sequence of the control vector, that is, the interactive object can be driven to emit the sound corresponding to the text content, and at the same time, it can make a sound synchronized with the sound. The action gives the target object the feeling that the interactive object is talking, which improves the interactive experience between the target object and the interactive object.

Assuming that the feature code is output at the set time of the first time window, the control vector before the set time can be set to the default value, that is, when the phoneme sequence is just started to be played, the interactive object can be set to the default value. Action, after the set time, the interactive object starts to use the sequence of control vectors obtained according to the first coding sequence to drive the interactive object to act. Taking Figure 4 as an example, the feature code 1 is output at time t0, and the default control vector is output before time t0.

The length of the time window is related to the amount of information contained in the feature code. When the amount of information contained in the time window is relatively large, the RNN process will output a more uniform result. If the length of the time window is too large, the expressions of the interactive objects when they speak may not correspond to part of the text; if the length of the time window is too small, the expressions of the interactive objects may appear stiff when they speak. Therefore, the duration of the time window needs to be determined according to the minimum duration of the phoneme corresponding to the text data, so that the action made by driving the interactive object has a stronger correlation with the sound.

The step size of the time window sliding is related to the time interval (frequency) of obtaining the control vector, which is related to the frequency of driving the interactive object to make an action. The length and step size of the time window can be set according to the actual interactive scene, so that the expressions and actions made by the interactive objects are more closely related to the sound, and more vivid and natural.

In some embodiments, when the time interval between phonemes in the phoneme sequence is greater than a set threshold value, the interactive object is driven to act according to the set control vector of the local area. That is, when the interactive character pauses for a long time, the interactive object is driven to perform the set action. For example, when the output speech pauses for a long time, the interactive object can be made to smile or slightly wiggle its body, so as to prevent the interactive object from standing upright with no expression during the long pause, so that the process of the interactive object speaking is more convenient. Natural and smooth, which improves the interactive experience of the target object.

In some embodiments, the voice data sequence includes a voice frame sequence, and acquiring feature information of at least one voice data unit in the voice data sequence includes: acquiring a first acoustic feature sequence corresponding to the voice frame sequence, where the The first acoustic feature sequence includes an acoustic feature vector corresponding to each speech frame in the speech frame sequence; according to the first acoustic feature sequence, an acoustic feature vector corresponding to at least one speech frame is obtained; according to the acoustic feature vector to obtain feature information corresponding to the at least one speech frame.

In the embodiment of the present disclosure, the control parameters of at least one local area of the interactive object may be determined according to the acoustic characteristics of the speech frame sequence, and the control parameters may also be determined according to other characteristics of the speech frame sequence.

First, acquire the acoustic feature sequence corresponding to the speech frame sequence. Here, in order to distinguish it from the acoustic feature sequence mentioned later, the acoustic feature sequence corresponding to the speech frame sequence is referred to as the first acoustic feature sequence.

In an embodiment of the present disclosure, the acoustic feature may be a feature related to speech emotion, such as a fundamental frequency feature, a common peak feature, a Mel Frequency Cepstral Cofficient (MFCC), and the like.

The first acoustic feature sequence is obtained by processing the entire sequence of speech frames. Taking the MFCC feature as an example, each speech frame in the sequence of speech frames can be subjected to windowing, fast Fourier transform, and filtering. , logarithmic processing and discrete cosine processing to obtain the MFCC coefficients corresponding to each speech frame.

The first acoustic feature sequence is obtained by processing the entire speech frame sequence, and reflects the overall acoustic feature of the speech data sequence.

In an embodiment of the present disclosure, the first acoustic feature sequence includes an acoustic feature vector corresponding to each speech frame in the speech frame sequence. Taking MFCC as an example, the first acoustic feature sequence includes MFCC coefficients of each speech frame. The first acoustic feature sequence obtained according to the speech frame sequence is shown in FIG. 5 .

Next, according to the first acoustic feature sequence, the acoustic feature corresponding to at least one speech frame is acquired.

When the first acoustic feature sequence includes an acoustic feature vector corresponding to each speech frame in the speech frame sequence, the same number of feature vectors corresponding to the at least one speech frame may be used as the speech frame acoustic characteristics. The above-mentioned same number of feature vectors may form a feature matrix, and the feature matrix is the acoustic feature of the at least one speech frame.

Taking FIG. 5 as an example, the N feature vectors in the first acoustic feature sequence form the acoustic features of the corresponding N speech frames; wherein, N is a positive integer. The first acoustic feature matrix may include a plurality of acoustic features, and speech frames corresponding to each of the acoustic features may partially overlap.

Finally, a control vector of at least one local area of the interactive object corresponding to the acoustic feature is obtained.

For the acquired acoustic feature corresponding to at least one speech frame, a control vector of at least one local area may be acquired. The local area can be selected according to the action of the interactive object that needs to be controlled. For example, when the face and limbs of the interactive object need to be controlled to move at the same time, the control vectors of all local areas can be obtained; when the expression of the interactive object needs to be controlled, Then the control vector of the local area corresponding to the face can be obtained.

While playing the voice data sequence, the interactive object is driven to act according to the control vector corresponding to each acoustic feature obtained through the first acoustic feature sequence, so that while the terminal device outputs sound, the interactive object can Actions that match the output sound include facial movements, expressions, and body movements, so that the target object feels that the interactive object is speaking. And because the control vector is related to the acoustic characteristics of the output sound, driving according to the control vector can make the expressions and body movements of the interactive objects have emotional factors, so that the speaking process of the interactive objects is more natural and vivid, thereby Improved the interactive experience between target objects and interactive objects.

In some embodiments, the acoustic feature corresponding to the at least one speech frame may be acquired by performing a sliding window on the first acoustic feature sequence.

These voices can be obtained by performing a sliding window on the first acoustic feature sequence with a time window of a set length and a set step, and taking the acoustic feature vectors in the time window as the acoustic features of the corresponding same number of speech frames. Acoustic features corresponding to the frames in common. After completing the sliding window, according to the obtained multiple acoustic features, a second acoustic feature sequence can be obtained.

Taking the driving method of the interactive object shown in FIG. 5 as an example, the speech frame sequence includes 100 speech frames per second, the length of the time window is 1s, and the step size is 0.04s. Since each feature vector in the first acoustic feature sequence corresponds to a speech frame, correspondingly, the first acoustic feature sequence also includes 100 feature vectors per second. During the sliding window process on the first acoustic feature sequence, 100 feature vectors in the time window are obtained each time as the acoustic features of the corresponding 100 speech frames. By moving the time window with a step size of 0.04s on the first acoustic feature sequence, the acoustic feature 1 corresponding to the 1st to 100th speech frames and the acoustic feature 2 corresponding to the 4th to 104th speech frames are obtained respectively, and By analogy, after traversing the first acoustic feature, acoustic feature 1, acoustic feature 2, ..., acoustic feature M are obtained, so as to obtain the second acoustic feature sequence, where M is a positive integer, and its value is based on the number of frames in the speech frame sequence (the number of feature vectors in the first acoustic feature sequence), the length of the time window and the step size are determined.

According to the acoustic feature 1, the acoustic feature 2, ..., the acoustic feature M, the corresponding control vector 1, the control vector 2, ..., and the control vector M can be obtained respectively, thereby obtaining a sequence of control vectors.

As shown in FIG. 5 , the sequence of control vectors and the second acoustic feature sequence are aligned in time, and the acoustic feature 1, acoustic feature 2, ..., acoustic feature M in the second acoustic feature sequence, are respectively obtained according to the N feature vectors in the first acoustic feature sequence. Therefore, while playing the speech frame, the interactive object can be driven to act according to the sequence of the control vectors.

Assuming that the output of acoustic features starts at the set time of the first time window, the control vector before the set time can be set to the default value, that is, when the audio frame sequence is just started to be played, the interactive object can be set to the default value. After the set time, the sequence of control vectors obtained according to the first acoustic feature sequence is used to drive the interactive object to act.

Taking Figure 5 as an example, the acoustic feature 1 is output at time t0, and the acoustic feature is output at an interval of 0.04s corresponding to the step size, the acoustic feature 2 is output at the time t1, and the acoustic feature 3 is output at the time t2. (M-1) The acoustic feature M is output at time. Correspondingly, the feature vector (i+1) corresponds to the time period t i~t(i+1), where i is an integer less than (M-1), and before time t0, the control vector is the default control vector.

In the embodiment of the present disclosure, while playing the voice data sequence, the interactive object is driven to act according to the sequence of control vectors, so that the action of the interactive object is synchronized with the output sound, and the target object is sent to the target object. With the feeling that the interactive object is talking, the interactive experience between the target object and the interactive object is improved.

The length of the time window is related to the amount of information contained in the acoustic feature. The longer the time window is, the more information it contains, and the stronger the correlation between the action and the sound that drives the interactive object. The step size of the time window sliding is related to the time interval (frequency) of obtaining the control vector, which is related to the frequency of driving the interactive object to make an action. The length and step size of the time window can be set according to the actual interactive scene, so that the expressions and actions made by the interactive objects are more closely related to the sound, and more vivid and natural.

In some embodiments, the acoustic features include L-dimension Mel-frequency cepstral coefficients MFCC, where L is a positive integer. MFCC represents the distribution of the energy of the speech signal in different frequency ranges. The L-dimension MFCC can be obtained by converting multiple speech frame data in the speech frame sequence to the frequency domain and using a mel filter including L subbands. . The control vector is obtained according to the MFCC of the speech data sequence, so as to drive the interactive object to perform facial movements and body movements according to the control vector, so that the expressions and body movements of the interactive object have emotional factors, and the speaking process of the interactive object is more Natural and vivid, thus improving the interactive experience between the target object and the interactive object.

In some embodiments, the feature information of the speech data unit may be input into a pre-trained recurrent neural network to obtain control parameter values of the interactive object corresponding to the feature information. Since the recurrent neural network is a time recursive neural network, it can learn the historical information of the input feature information, and output control parameters according to the sequence of speech units; for example, the control parameters can be facial gesture control parameters, or at least one local The control vector for the region.

In the embodiment of the present disclosure, a pre-trained recurrent neural network is used to obtain the control parameters corresponding to the feature information of the speech data unit, and the related historical feature information and the current feature information are fused, so that the historical control parameters are The changes of the current control parameters have an impact, making the expression changes and body movements of the interactive characters more gentle and natural.

In some embodiments, the recurrent neural network can be trained in the following manner.

First, obtain a sample of feature information. For example, the feature information sample can be obtained in the following manner.

Obtain a video segment of a character's voice, and extract the corresponding speech segment of the character from the video segment. For example, a video segment in which a real person is speaking can be obtained; the video segment is sampled to obtain a plurality of video segments containing the character. a first image frame; and sampling the speech segment to obtain a plurality of speech frames.

Obtain feature information corresponding to the voice frame according to the voice data unit included in the voice frame corresponding to the first image frame;

Converting the first image frame into a second image frame including the interactive object, and acquiring control parameter values of the interactive object corresponding to the second image frame.

According to the control parameter value, the feature information corresponding to the first image frame is marked to obtain a feature information sample.

In some embodiments, the feature information includes phoneme feature codes, and the control parameters include facial muscle control coefficients. According to the above method for obtaining feature information samples, the obtained facial muscle control coefficients are used to mark the feature codes of the phonemes corresponding to the first image frame, and then the feature information samples corresponding to the phoneme feature codes are obtained.

In some embodiments, the feature information includes feature codes of phonemes, and the control parameters include at least one local control vector of the interactive object. According to the above method for obtaining feature information samples, the feature code of the phoneme corresponding to the first image frame is marked by using the obtained at least one local control vector, and then the feature information sample corresponding to the feature code of the phoneme is obtained. .

In some embodiments, the feature information includes acoustic features of speech frames, and the control parameter includes at least one local control vector of the interactive object. According to the above method for obtaining feature information samples, using the obtained at least one local control vector to mark the acoustic features of the speech frame corresponding to the first image frame, the features corresponding to the acoustic features of the speech frame are obtained Information sample.

Those skilled in the art should understand that the feature information samples are not limited to the above, and corresponding feature information samples can be obtained corresponding to various features of various types of speech data units.

After obtaining the feature information samples, the initial recurrent neural network is trained according to the feature information samples, and the recurrent neural network is obtained by training after the change of the network loss satisfies the convergence condition, wherein the network loss Including the difference between the control parameter value predicted by the recurrent neural network and the marked control parameter value.

In the embodiment of the present disclosure, by splitting a video segment of a character into a plurality of corresponding first image frames and a plurality of voice frames, by converting the first image frame containing the real person into the interactive object containing the first image frame The second image frame is used to obtain the control parameter value corresponding to the feature information of at least one speech frame, so that the correspondence between the feature information and the control parameter value is better, so as to obtain high-quality feature information samples, so that the posture of the interactive object is closer to Corresponds to the real pose of the character.

FIG. 6 is a schematic structural diagram of an apparatus for driving an interactive object according to at least one embodiment of the present disclosure. As shown in FIG. 6 , the apparatus may include: a first acquiring unit 601 for acquiring driving data of the interactive object, and determining the driving mode of the driving data; the second obtaining unit 602 is configured to, in response to the driving mode, obtain the control parameter value of the interactive object according to the driving data; the driving unit 603 is configured to obtain the control parameter value according to the control parameter The value controls the pose of the interactive object.

In some embodiments, the apparatus further includes an output unit for controlling the display device to output voice and/or display text according to the driving data.

In some embodiments, when determining the driving mode corresponding to the driving data, the first obtaining unit is specifically configured to: according to the type of the driving data, obtain a voice data sequence corresponding to the driving data, the voice data The data sequence includes a plurality of voice data units; if it is detected that the voice data unit includes target data, the driving mode of the driving data is determined to be the first driving mode, and the target data and the preset control parameter value of the interactive object are determined. Correspondingly; in response to the driving mode, acquiring the control parameter value of the interactive object according to the driving data includes: in response to the first driving mode, converting the preset control parameter value corresponding to the target data As the control parameter value of the interactive object.

In some embodiments, the target data includes a keyword or a keyword, and the keyword or the keyword corresponds to a preset control parameter value of a setting action of an interactive object; or, the target data includes a syllable, The syllable corresponds to the preset control parameter value of the interactive object for setting the mouth shape action.

In some embodiments, when identifying the driving mode of the driving data, the first obtaining unit is specifically configured to: according to the type of the driving data, obtain a voice data sequence corresponding to the driving data, the voice data The sequence includes a plurality of voice data units; if it is not detected that the voice data unit includes target data, the driving mode of the driving data is determined to be the second driving mode, and the target data and the preset control parameter value of the interactive object are determined. Correspondingly; in response to the driving mode, acquiring the control parameter value of the interactive object according to the driving data includes: in response to the second driving mode, acquiring the value of at least one voice data unit in the voice data sequence Feature information; obtain the control parameter value of the interactive object corresponding to the feature information.

In some embodiments, the speech data sequence includes a phoneme sequence, and when the feature information of at least one speech data unit in the speech data sequence is obtained, the second obtaining unit is specifically configured to: perform an operation on the phoneme sequence. The feature coding is used to obtain a first coding sequence corresponding to the phoneme sequence; the feature coding corresponding to at least one phoneme is obtained according to the first coding sequence; the feature information of the at least one phoneme is obtained according to the feature coding.

In some embodiments, the voice data sequence includes a voice frame sequence, and when acquiring feature information of at least one voice data unit in the voice data sequence, the second acquiring unit is specifically configured to: acquire the voice frame The first acoustic feature sequence corresponding to the sequence, the first acoustic feature sequence includes an acoustic feature vector corresponding to each voice frame in the voice frame sequence; according to the first acoustic feature sequence, obtain at least one voice frame corresponding to The acoustic feature vector of ; obtain feature information corresponding to the at least one speech frame according to the acoustic feature vector.

In some embodiments, the control parameters of the interactive object include facial posture parameters, the facial posture parameters include facial muscle control coefficients, and the facial muscle control coefficients are used to control the motion state of at least one facial muscle; When the data obtains the control parameter value of the interactive object, the second obtaining unit is specifically used for: obtaining the facial muscle control coefficient of the interactive object according to the driving data; the driving unit is specifically used for: according to the obtained The facial muscle control coefficient drives the interactive object to make facial movements that match the driving data; the device further includes a limb driving unit for acquiring the driving data of the body posture associated with the facial posture parameters; The driving data of the body posture associated with the facial posture parameter value drives the interactive object to make body movements.

In some embodiments, the control parameter of the interactive object includes a control vector of at least one local area of the interactive object; when acquiring the control parameter value of the interactive object according to the driving data, the second acquiring unit It is specifically used for: acquiring the control vector of at least one local area of the interactive object according to the driving data; the driving unit is specifically used for: controlling the interactive object according to the acquired control vector of the at least one local area facial and/or body movements.

According to an aspect of the present disclosure, an electronic device is provided, the device includes a memory and a processor, the memory is used to store computer instructions that can be executed on the processor, and the processor is used to execute the computer instructions. At the same time, the driving method of the interactive object described in any one of the implementation manners provided by the present disclosure is implemented.

According to an aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the method for driving an interactive object according to any one of the implementation manners provided in the present disclosure.

At least one embodiment of the present specification further provides an electronic device, as shown in FIG. 7 , the device includes a memory and a processor, where the memory is used to store computer instructions that can be executed on the processor, and the processor is used to execute all computer instructions. The method for driving an interactive object described in any embodiment of the present disclosure is implemented when the computer instruction is used.

At least one embodiment of the present specification further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the method for driving an interactive object described in any embodiment of the present disclosure.

As will be appreciated by one skilled in the art, one or more embodiments of this specification may be provided as a method, system or computer program product. Accordingly, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may be implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) having computer-usable program code embodied therein in the form of a computer program product.

Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

The foregoing describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in an order different from that of the embodiments and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangible embodiment of computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or A combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, ie, one or more of computer program instructions encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, a data processing apparatus. Multiple mods. Alternatively or additionally, program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical or electromagnetic signals, which are generated to encode and transmit information to suitable receiver devices for data retrieval. The processing device executes. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or sequential access memory device, or a combination of one or more of these.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, eg, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

Computers suitable for the execution of a computer program include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from read-only memory and/or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operably coupled to, such mass storage devices to receive data therefrom, one or more mass storage devices, such as magnetic disks, optical disks, magneto-optical disks, or optical disks, etc., for storing data Or send data to it, or both. However, the computer does not have to have such a device. Additionally, a computer may be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver, or a universal serial bus (USB ) flash memory drives for portable storage devices, to name a few.

Computer-readable media suitable for storage of computer program instructions and data include all forms of non-volatile memory, media, and memory devices including, for example, semiconductor memory devices (eg, EPROM, EEPROM, and flash memory devices), magnetic disks (such as internal hard disks or removable disks), CD-ROMs, and CD-ROMs and DVD-ROMs. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.

While this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or what may be claimed, but rather are used primarily to describe features of specific embodiments of particular inventions. Certain features that are described in this specification in multiple embodiments can also be implemented in combination in a single embodiment. On the other hand, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may function as described above in certain combinations and even be originally claimed as such, one or more features from a claimed combination may in some cases be removed from the combination and the claimed A protected combination may point to a subcombination or a variation of a subcombination.

Similarly, although operations in the figures are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or sequentially, or that all illustrated operations be performed, to achieve the desired results . In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software package products, or packaged into multiple software products.

Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claim may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The above descriptions are only preferred embodiments of one or more embodiments of this specification, and are not intended to limit one or more embodiments of this specification. All within the spirit and principles of one or more embodiments of this specification, Any modifications, equivalent replacements, improvements, etc. made should be included within the protection scope of one or more embodiments of this specification.

201: the step of acquiring the driving data of the interactive object and determining the driving mode of the driving data 202: in response to the driving mode, the step of acquiring the control parameter value of the interactive object according to the driving data 203: the step of controlling the posture of the interactive object according to the control parameter value 601: The first acquisition unit 602: Second acquisition unit 603: Drive unit

FIG. 1 is a schematic diagram of a display device in a method for driving an interactive object provided by at least one embodiment of the present disclosure. FIG. 2 is a flowchart of a method for driving an interactive object provided by at least one embodiment of the present disclosure. FIG. 3 is a schematic diagram of a process of feature encoding for a phoneme sequence proposed by at least one embodiment of the present disclosure. FIG. 4 is a schematic diagram of a process of obtaining a control parameter value according to a phoneme sequence proposed by at least one embodiment of the present disclosure. FIG. 5 is a schematic diagram of a process of obtaining a control parameter value according to a sequence of speech frames according to at least one embodiment of the present disclosure. FIG. 6 is a schematic structural diagram of a driving device for an interactive object according to at least one embodiment of the present disclosure. FIG. 7 is a schematic structural diagram of an electronic device provided by at least one embodiment of the present disclosure.

201: the step of acquiring the driving data of the interactive object and determining the driving mode of the driving data

202: in response to the driving mode, the step of acquiring the control parameter value of the interactive object according to the driving data

203: the step of controlling the posture of the interactive object according to the control parameter value

Claims (13)

A method for driving an interactive object, wherein the interactive object is displayed in a display device, the driving method includes: acquiring driving data of the interactive object, and determining a driving mode of the driving data; in response to the driving mode, according to Obtaining the control parameter value of the interactive object from the driving data; controlling the posture of the interactive object according to the control parameter value, wherein determining the driving mode of the driving data includes: obtaining the driving data according to the type of the driving data a voice data sequence corresponding to the driving data, the voice data sequence including a plurality of voice data units; in response to not detecting that the voice data unit includes target data, determine that the driving mode of the driving data is the second driving mode ; wherein, in response to the driving mode, acquiring the control parameter value of the interactive object according to the driving data includes: in response to the second driving mode, acquiring the data of at least one voice data unit in the voice data sequence Feature information; obtain the control parameter value of the interactive object corresponding to the feature information. The driving method according to claim 1, further comprising: controlling the display device to output voice and/or display text according to the driving data. The driving method according to claim 1 or 2, wherein determining the driving mode of the driving data comprises: in response to detecting that the target data is included in the voice data unit, determining that the driving mode of the driving data is: In the first driving mode, the target data corresponds to the preset control parameter value of the interactive object; in response to the driving mode, acquiring the control parameter value of the interactive object according to the driving data includes: responding to the In the first driving mode, the preset control parameter value corresponding to the target data is used as the control parameter value of the interactive object. The driving method according to claim 3, wherein the target data includes a keyword or a keyword, and the keyword or the keyword corresponds to a preset control parameter value of the setting action of the interactive object; or , the target data includes syllables, and the syllables correspond to the preset control parameter values of the interactive object for setting the mouth shape action. The driving method according to claim 1, wherein the voice data sequence includes a phoneme sequence, and acquiring feature information of at least one voice data unit in the voice data sequence includes: performing feature encoding on the phoneme sequence to obtain The first coding sequence corresponding to the phoneme sequence; according to the first coding sequence, obtain the feature code corresponding to at least one phoneme; according to the feature code, obtain the feature information of the at least one phoneme. The driving method according to claim 1, wherein the voice data sequence includes a voice frame sequence, and acquiring feature information of at least one voice data unit in the voice data sequence includes: acquiring the first voice data unit corresponding to the voice frame sequence an acoustic feature sequence, the first acoustic feature sequence includes an acoustic feature vector corresponding to each speech frame in the speech frame sequence; according to the first acoustic feature sequence, acquire an acoustic feature vector corresponding to at least one speech frame ; According to the acoustic feature vector, obtain feature information corresponding to the at least one speech frame. The driving method according to claim 1 or 2, wherein the control parameters of the interactive object include facial posture parameters, the facial posture parameters include facial muscle control coefficients, and the facial muscle control coefficients are used to control at least one facial muscle obtaining the control parameter value of the interactive object according to the driving data, including: obtaining the facial muscle control coefficient of the interactive object according to the driving data; controlling the posture of the interactive object according to the control parameter value , which includes: driving the interactive object to perform facial actions matching the driving data according to the acquired facial muscle control coefficients. The driving method according to claim 7, further comprising: acquiring the driving data of the body posture associated with the facial posture parameter; driving the interactive object to do body movements. The driving method according to claim 1 or 2, wherein the control parameter of the interactive object includes a control vector of at least one local area of the interactive object; the control parameter value of the interactive object is obtained according to the driving data, The method includes: obtaining a control vector of at least one local area of the interactive object according to the driving data; controlling the posture of the interactive object according to the control parameter value, including: obtaining the control vector of the at least one local area according to the obtained control vector , to control the facial movements and/or body movements of the interactive object. The driving method according to claim 1, wherein acquiring the control parameter value of the interactive object corresponding to the feature information comprises: inputting the feature information into a pre-trained recurrent neural network, and obtaining the value of the control parameter of the interactive object corresponding to the feature information. The control parameter value of the interactive object corresponding to the feature information. A driving device for an interactive object, the interactive object is displayed on a display device, the device comprises: a first acquiring unit, configured to acquire driving data of the interactive object, and determine a driving mode of the driving data; a second acquiring unit an acquiring unit, configured to acquire a control parameter value of the interactive object according to the driving data in response to the driving mode; a driving unit, configured to control the posture of the interactive object according to the control parameter value, wherein the determined The driving mode of the driving data includes: obtaining the language corresponding to the driving data according to the type of the driving data a voice data sequence, the voice data sequence includes a plurality of voice data units; in response to not detecting that the voice data unit includes target data, determining that the driving mode of the driving data is the second driving mode; wherein, in response to the In the driving mode, acquiring the control parameter value of the interactive object according to the driving data includes: in response to the second driving mode, acquiring feature information of at least one voice data unit in the voice data sequence; The control parameter value of the interactive object corresponding to the feature information. An electronic device includes a memory and a processor, wherein the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the driving method described in claim 1 when the computer instructions are executed. A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the driving method described in claim 1.

Images