CN116778900A - Speech synthesis method, electronic device and storage medium - Google Patents

Abstract

The application provides a voice synthesis method, electronic equipment and a storage medium, and relates to the technical field of voice, wherein the method comprises the following steps: acquiring a first text, wherein the first text comprises N dialogue texts, each dialogue text corresponds to dialogue content of a role, and N is an integer greater than 1; determining azimuth information of each dialogue text according to target information of each dialogue text in the first text, wherein the target information comprises position information of a role corresponding to each dialogue text in an associated movie fragment of the dialogue text or position keywords associated with each dialogue text in the first text; and determining the sound source position of the audio to be synthesized of each dialogue text according to the azimuth information of each dialogue text, and generating the synthesized voice of each dialogue text according to the sound source position of the audio to be synthesized of each dialogue text. The embodiment of the application can convert the dialogue text into the dialogue voice with different sound source positions, thereby effectively improving the voice playing effect of the dialogue text.

Description

Speech synthesis method, electronic device and storage medium

Technical field

The present application relates to the field of speech technology, and in particular, to a speech synthesis method, electronic device and storage medium.

Background technique

"Listening to e-books" is a widely used technology. At present, text to speech (Text to Speech, TTS) technology can usually be used to synthesize text into a speech file in real time and play it back, allowing users to "hear" the content of e-books. Existing e-book readings usually use the sound library of a single speaker, and only convert the e-book text into a mono audio file for playback.

Many e-books are usually filled with a lot of character dialogue. The existing e-book reading only converts the e-book text into a mono audio file for playback, which results in a monotonous audio file and poor audio playback effect.

Contents of the invention

Embodiments of the present application provide a speech synthesis method, electronic equipment, and storage media to solve the problem that existing e-book reading only converts e-book text into monophonic audio files for playback, resulting in relatively monotonous audio files and poor audio playback effects. Poor question.

In order to solve the above technical problems, this application is implemented as follows;

In a first aspect, embodiments of the present application provide a speech synthesis method, which method includes:

Obtain the first text, wherein the first text includes N dialogue texts, each dialogue text corresponds to the dialogue content of a character, and N is an integer greater than 1;

Determine the orientation information of each dialogue text according to the target information of each dialogue text in the first text, wherein the target information includes the position information of the character corresponding to each dialogue text in the associated film and television clip of the dialogue text, or the location information of each dialogue text. Describe the positional keywords associated with each dialogue text in the first text;

According to the orientation information of each dialogue text, the sound source position of the audio to be synthesized of each dialogue text is determined, and based on the sound source position of the audio to be synthesized of each dialogue text, a synthesized speech of each dialogue text is generated.

Optionally, the target information includes position information of the character corresponding to each dialogue text in the film and television clip associated with the dialogue text;

Determining the orientation information of each dialogue text based on the target information of each dialogue text in the first text includes:

In the case where the e-book containing the first text has associated film and television resources, it is determined based on the content of each dialogue text and the film and television content of each video frame in the associated film and television resources, that each of the associated film and television resources is determined to be associated with each other. Related film and television clips matched with dialogue text;

Determine the target film and television character in the film and television clip associated with each dialogue text that matches the role of the corresponding dialogue text;

The orientation information of each dialogue text is determined based on the position information of the target film and television character in the associated film and television segment of each dialogue text in the film and television frame of the associated film and television segment.

Optionally, the determining the target film and television character in the associated film and television clip of each dialogue text that matches the role of the corresponding dialogue text includes:

Perform target detection on the first associated film and television segment, and determine the target image in the first associated film and television segment that emits a dialogue voice that matches the first dialogue text, wherein the film and television character indicated by the target image is the same as the first dialogue text. The role of the dialogue text matches the target film and television character, the first associated film and television segment is an associated film and television segment of the first dialogue text, and the first dialogue text is any dialogue text in the first text.

Optionally, determining the associated film and television segments in the associated film and television resources that respectively match each dialogue text based on the content of each dialogue text and the film and television content of each video frame in the associated film and television resources includes:

According to the content of the first dialogue text and the subtitle content or audio content of each video frame in the associated film and television resources, the first associated film and television segment matching the first dialogue text in the associated film and television resources is determined, and the first The dialogue text is any dialogue text in the first text;

Perform scene recognition on the first associated film and television clip to determine the target scene element in the first associated film and television clip;

Determine scene keywords in the context text of the first dialogue text in the first text;

When the target scene element matches the scene keyword, target detection is performed on the first associated film and television clip.

Optionally, the step of performing scene recognition on the first associated film and television clip and determining the target scene element in the first associated film and television clip includes:

Perform scene recognition on the first associated film and television clip, and determine each scene element and the credibility of each scene element in the first associated film and television clip;

Scene elements whose credibility exceeds a preset threshold in the first associated video clip are determined as the target scene elements.

Optionally, the target information includes position keywords associated with each dialogue text in the first text;

Determining the orientation information of each dialogue text based on the target information of each dialogue text in the first text includes:

Traverse the character names in the first text;

Determine the position keywords in the contextual text of the position where each character name appears in the first text;

According to the dialogue text corresponding to each character name in the first text, the position keyword associated with each dialogue text is determined.

Optionally, determining the location keywords in the context text where each character name appears in the first text includes:

When the qualified description of the first character name in the first text satisfies the preset discrimination mode, the first local word corresponding to the first character name is determined according to the qualified description, wherein the qualified description At least including a local word, the first character name is any character name in the first text;

Determine the first location corresponding to the first character name according to the first location word corresponding to the first character name;

The first character name, the first location word and the first location are correspondingly recorded in a location table, where, when the second location word corresponding to the second character name is the same as the first location word , the second orientation corresponding to the second character name recorded in the orientation table is the same as the first orientation, and the second character name is any character in the first text except the first character name. a role name;

Determining the orientation information of each dialogue text based on the target information of each dialogue text in the first text includes:

According to the location words and directions corresponding to each character name recorded in the location table, the location of the dialogue text corresponding to each character name is determined.

Optionally, before generating the synthesized speech of each dialogue text according to the sound source position of the audio to be synthesized of each dialogue text, the method further includes:

Determine the emotional attributes of each dialogue text in the first text;

Determine the audio parameters of the audio to be synthesized for each dialogue text according to the emotional attributes of each dialogue text;

Generating the synthesized speech of each dialogue text according to the sound source position of the audio to be synthesized of each dialogue text includes:

According to the sound source position and audio parameters of the audio to be synthesized of each dialogue text, the synthesized speech of each dialogue text is generated.

Optionally, determining the emotional attributes of each dialogue text in the first text includes:

Perform word segmentation processing on the first dialogue text and the context of the first dialogue text to obtain multiple emotional phrases, wherein the first dialogue text is any dialogue text in the first text;

Query the emotional attributes corresponding to each emotional phrase in the multiple phrases through the pre-constructed emotional dictionary;

According to the emotional attribute corresponding to each emotional phrase, the emotional attribute of the first dialogue text is determined.

Optionally, determining the emotional attribute of the first dialogue text based on the emotional attribute corresponding to each emotional phrase includes:

When the emotional attributes corresponding to at least two emotional phrases in the first dialogue text and the context of the first dialogue text are different, determine the weight of the emotional attribute corresponding to each emotional phrase;

The emotional attribute with the highest weight is determined as the emotional attribute of the first dialogue text.

Optionally, generating the synthesized speech of each dialogue text according to the sound source position of the audio to be synthesized of each dialogue text includes:

According to the sound source position of the audio to be synthesized in each dialogue text, the time delay or volume difference between the first channel and the second channel of the audio to be synthesized in each dialogue text is determined, and a sound source of each dialogue text is generated. The time delay or the two-channel synthesized speech with the volume difference.

Optionally, before generating the synthesized speech of each dialogue text according to the sound source position of the audio to be synthesized of each dialogue text, the method further includes:

Determine the character attributes of each dialogue text according to the name information of each character in the first text marked in the preset character dictionary;

Generating the synthesized speech of each dialogue text according to the sound source position of the audio to be synthesized of each dialogue text includes:

Determine the speaker corresponding to each dialogue text according to the role attribute of each dialogue text;

According to the sound source position of the audio to be synthesized in each dialogue text, the synthesized speech of each dialogue text is generated with the voice color of the speaker corresponding to each dialogue text.

In a second aspect, embodiments of the present application also provide a speech synthesis device, including:

The first acquisition module is used to acquire the first text, wherein the first text includes N dialogue texts, each dialogue text corresponds to the dialogue content of a character, and N is an integer greater than 1;

The first determination module is used to determine the orientation information of each dialogue text according to the target information of each dialogue text in the first text, wherein the target information includes the associated film and television clips of the characters corresponding to each dialogue text in the dialogue text. The location information in the first text, or the location keywords associated with each dialogue text in the first text;

The second determination module is configured to determine the sound source position of the audio to be synthesized for each dialogue text based on the orientation information of each dialogue text, and generate each sound source position of the audio to be synthesized for each dialogue text based on the sound source location of the audio to be synthesized. Synthetic speech for conversational text.

Optionally, the target information includes position information of the character corresponding to each dialogue text in the film and television clip associated with the dialogue text;

The first determination module includes:

A first determining unit configured to determine, when the e-book containing the first text has associated film and television resources, based on the content of each dialogue text and the film and television content of each video frame in the associated film and television resources, Related film and television clips in related film and television resources that match each dialogue text respectively;

a second determination unit configured to determine the target film and television character in the film and television clip associated with each dialogue text that matches the role of the corresponding dialogue text;

The third determination unit is configured to determine the orientation information of each dialogue text based on the position information of the target film and television character in the associated film and television segment of each dialogue text in the film and television frame of the associated film and television segment.

Optionally, the second determining unit is specifically used to:

Perform target detection on the first associated film and television segment, and determine the target image in the first associated film and television segment that emits a dialogue voice that matches the first dialogue text, wherein the film and television character indicated by the target image is the same as the first dialogue text. The role of the dialogue text matches the target film and television character, the first associated film and television segment is an associated film and television segment of the first dialogue text, and the first dialogue text is any dialogue text in the first text.

Optionally, the first determining unit includes:

The first determining subunit is configured to determine the first dialogue text in the associated film and television resource that matches the first dialogue text based on the content of the first dialogue text and the subtitle content or audio content of each video frame in the associated film and television resource. Associated with film and television clips, the first dialogue text is any dialogue text in the first text;

The second determination subunit is used to perform scene recognition on the first associated film and television clip and determine the target scene element in the first associated film and television clip;

A third determination subunit, configured to determine scene keywords in the context text of the first dialogue text in the first text;

The first detection subunit is configured to perform target detection on the first associated film and television clip when the target scene element matches the scene keyword.

Optionally, the second determination subunit is specifically used for:

Perform scene recognition on the first associated film and television clip, and determine each scene element and the credibility of each scene element in the first associated film and television clip;

Scene elements whose credibility exceeds a preset threshold in the first associated video clip are determined as the target scene elements.

Optionally, the target information includes position keywords associated with each dialogue text in the first text;

The first determination module also includes:

The first traversal unit is used to traverse the character names in the first text;

The fourth determination unit is used to determine the position keywords in the contextual text of the position where each character name appears in the first text;

The fifth determination unit is used to determine the position keyword associated with each dialogue text according to the dialogue text corresponding to each character name in the first text.

Optionally, the fourth determining unit is specifically used to:

When the qualified description of the first character name in the first text satisfies the preset discrimination mode, the first local word corresponding to the first character name is determined according to the qualified description, wherein the qualified description At least including a local word, the first character name is any character name in the first text;

Determine the first location corresponding to the first character name according to the first location word corresponding to the first character name;

The first character name, the first location word and the first location are correspondingly recorded in a location table, where, when the second location word corresponding to the second character name is the same as the first location word , the second orientation corresponding to the second character name recorded in the orientation table is the same as the first orientation, and the second character name is any character in the first text except the first character name. a role name;

The first determination module also includes:

The sixth determination unit is used to determine the location of the dialogue text corresponding to each character name based on the location word and location corresponding to each character name recorded in the location table.

Optionally, the speech synthesis device further includes:

The third determination module is used to determine the emotional attributes of each dialogue text in the first text;

The fourth determination module is used to determine the audio parameters of the audio to be synthesized of each dialogue text according to the emotional attributes of each dialogue text;

The second determination module includes:

The first generation unit is configured to generate the synthesized speech of each dialogue text according to the sound source position and audio parameters of the audio to be synthesized of each dialogue text.

Optionally, the third determination module includes:

The first processing module is used to perform word segmentation processing on the first dialogue text and the context of the first dialogue text to obtain multiple emotional phrases, wherein the first dialogue text is any dialogue in the first text. text;

The first query unit queries the emotional attributes corresponding to each of the plurality of emotional phrases through a pre-constructed emotion dictionary;

A seventh determination unit is configured to determine the emotional attribute of the first dialogue text according to the emotional attribute corresponding to each emotional phrase.

Optionally, the seventh determining unit is specifically used to:

When the emotional attributes corresponding to at least two emotional phrases in the first dialogue text are different, determine the weight of the emotional attribute corresponding to each emotional phrase;

The emotional attribute with the highest weight is determined as the emotional attribute of the first dialogue text.

Optionally, the first generation unit is specifically used for:

According to the sound source position of the audio to be synthesized in each dialogue text, the time delay or volume difference between the first channel and the second channel of the audio to be synthesized in each dialogue text is determined, and a sound source of each dialogue text is generated. The time delay or the two-channel synthesized speech with the volume difference.

Optionally, the speech synthesis device further includes:

The fifth determination module is used to determine the character attributes of each dialogue text according to the name information of each character in the first text marked in the preset character dictionary;

The first generation unit is specifically also used for:

Determine the speaker corresponding to each dialogue text according to the role attribute of each dialogue text;

According to the sound source position of the audio to be synthesized in each dialogue text, the synthesized speech of each dialogue text is generated with the voice color of the speaker corresponding to each dialogue text.

In a third aspect, embodiments of the present application also provide an electronic device, including: a transceiver, a memory, a processor, and a computer program stored on the memory and executable on the processor;

The processor is configured to read the program in the memory to implement the steps of the speech synthesis method described in the first aspect.

In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium for storing a computer program that implements the steps of the speech synthesis method as described in the first aspect when executed by a processor.

In the embodiment of the present application, by determining the orientation information of each character dialogue text in the text, and determining the sound source position of the audio to be synthesized for each dialogue text based on the orientation information, and then based on the sound source location of the audio to be synthesized for each dialogue text, Generating the synthesized speech of each dialogue text can make the synthesized speech have a stereo playback effect. It can be seen that the synthesized speech file according to the embodiment of the present application is rich in content and can effectively improve the speech playback effect of the dialogue text.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting any creative effort.

Figure 1 is one of the flow charts of a speech synthesis method provided by an embodiment of the present application;

Figure 2 is a schematic diagram of a sound source position provided by an embodiment of the present application;

Figure 3 is a second schematic diagram of the position of a sound source provided by an embodiment of the present application;

Figure 4 is a flow chart for determining the location information of dialogue text provided by an embodiment of the present application;

Figure 5 is a flow chart for determining the weight of emotional attributes provided by an embodiment of the present application;

Figure 6 is a second flow chart of a speech synthesis method provided by an embodiment of the present application;

Figure 7 is a structural diagram of a speech synthesis device provided by an embodiment of the present application;

Figure 8 is a structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

Referring to Figure 1, Figure 1 is a flow chart of a speech synthesis method provided by an embodiment of the present application. As shown in Figure 1, the method includes the following steps:

Step 101: Obtain the first text, where the first text includes N dialogue texts, each dialogue text corresponds to the dialogue content of a character, and N is an integer greater than 1;

Step 102: Determine the orientation information of each dialogue text according to the target information of each dialogue text in the first text, where the target information includes the position information of the character corresponding to each dialogue text in the associated film and television clip of the dialogue text, or the first Positional keywords in the text associated with each dialogue text;

Step 103: Determine the sound source location of the audio to be synthesized for each dialogue text based on the orientation information of each dialogue text, and generate a synthesized speech for each dialogue text based on the sound source location of the audio to be synthesized for each dialogue text.

In step 101, the communication device obtains a first text. The first text is used to synthesize speech. The first text is a text in an e-book. The type of e-book includes novels, etc. After obtaining the first text, the first text is identified to identify N dialogue texts included in the first text, where N is an integer greater than 1, and each of the N dialogue texts corresponds to a character. Conversation content. It should be noted that among the N dialogue texts, each dialogue text may correspond to a different role, or there may be two or more dialogue texts corresponding to the same role.

In step 102, the orientation information is such as "left, right and center". The orientation information of each dialogue text can reflect the orientation of the character corresponding to each dialogue text. In the first text, when the orientation of each character is different, the orientation information of the dialogue text corresponding to each character is also different.

In the embodiment of the present application, the orientation information of the dialogue text can be determined through two solutions. The two solutions are: determining the position information of each dialogue text in the first text based on the position information of the character corresponding to each dialogue text in the film and television clip associated with the dialogue text; or, determining the position information of each dialogue text in the first text based on The associated position keywords are used to determine the location information of each dialogue text in the first text.

Among them, the associated film and television clips of the dialogue text refer to the existence of subtitles or audio that match the dialogue text in the associated film and television clips. When there are no associated film and television clips for each dialogue text in the first text, the location information of each dialogue text can be determined directly based on the location keywords associated with each dialogue text in the first text. When each dialogue text has associated film and television clips, you can choose a method to determine the location information of each dialogue text according to your needs. For example, the above two schemes for determining the orientation information of the dialogue text can be tested in advance to determine the accuracy of the above two schemes for determining the orientation information, and then one of the corresponding schemes can be selected according to the accuracy priority rule.

In step 103, the sound source position of the audio to be synthesized in each dialogue text is associated with the orientation information of each dialogue text. Specifically, if the orientation information of the two dialogue texts is different, the sound source positions of the two dialogue texts are also different. According to the sound source position of the audio to be synthesized in each dialogue text, the synthesized voice of each dialogue text is generated, so that the synthesized voice of each dialogue text can feed back the sound source location to achieve a stereoscopic playback effect.

It should be noted that in this application, the number of vocal channels for playing synthetic speech can be preset, and all possible sound source positions can be predetermined based on the number of sound source positions that can be fed back by the number of vocal channels for playing synthetic speech. For example, if the synthesized speech is a binaural speech, the synthesized speech can feedback three sound source positions, and the binaural includes the left channel and the right channel. The three sound source positions that binaural speech can include are "left, right and center". As shown in Figure 2, if the sound source is in the middle, it means that the left and right channels in the two-channel audio are synchronized and there is no volume difference, so the listener will think that the sound source is directly in front; as shown in Figure 3, If the sound source position is on the left, it means that the audio in the right channel is delayed compared to the left channel, or the volume of the audio in the right channel is lower, so the position of the sound source perceived by the listener will be to the left. Based on the above, if the synthesized speech is a binaural speech, the three sound source positions that the synthesized speech can include are "left, right and middle". Therefore, when determining the orientation information of each dialogue text in the text, each The orientation information of the dialogue text is "left", "right" or "middle". Further, the corresponding sound source position is determined based on the orientation information being "left", "right" or "middle".

Among them, setting the synthesized voice to be a two-channel voice is only an exemplary description, and the synthesized voice can also be set to be a three-channel voice or a four-channel voice, etc.

In the embodiment of the present application, the position information of the narration in the text other than each dialogue text can also be determined. The position information of the narration can all be the same position information. For example, the position information of the narration can be "all of the above three kinds of position information". "Middle", according to the orientation information of the narration, the sound source position of the narration's audio to be synthesized can be determined, and the synthesized speech of the narration can be generated based on the sound source position of the narration's audio to be synthesized.

In the embodiment of the present application, by determining the orientation information of each character dialogue text in the text, and determining the sound source position of the audio to be synthesized for each dialogue text based on the orientation information, and then based on the sound source location of the audio to be synthesized for each dialogue text, Generating the synthesized speech of each dialogue text can make the synthesized speech have a stereo playback effect. It can be seen that the synthesized speech file according to the embodiment of the present application is rich in content and can effectively improve the speech playback effect of the dialogue text.

Optionally, the target information includes position information of the character corresponding to each dialogue text in the film and television clip associated with the dialogue text;

According to the target information of each dialogue text in the first text, the orientation information of each dialogue text is determined, including:

When the e-book containing the first text has associated film and television resources, determine the associated film and television clips in the associated film and television resources that match each dialogue text respectively based on the content of each dialogue text and the film and television content of each video frame in the associated film and television resources;

Determine the target film and television character in the film and television clip associated with each dialogue text that matches the role of the corresponding dialogue text;

The orientation information of each dialogue text is determined based on the position information of the target film and television character in the associated film and television segment of each dialogue text in the film and television frame of the associated film and television segment.

In this embodiment, the e-book containing the first text has associated film and television resources. The associated film and television resources are the film and television works of the e-book. The associated film and television resources include TV drama resources, animation resources, and movie resources associated with the e-book.

When the e-book containing the first text has associated film and television resources, the associated film and television segments in the associated film and television resources that match each dialogue text are determined based on the content of each dialogue text and the film and television content of each video frame in the associated film and television resources. The film and television content of each video frame in the associated film and television resources includes but is not limited to the voice information, background information, character information, subtitle information, etc. of each video frame. When the video content of multiple adjacent video frames matches any dialogue text in the first text, for example, when the subtitle information in multiple adjacent video frames matches any dialogue text, it is determined that the multiple adjacent video frames match any dialogue text. The video clip formed by adjacent video frames is the associated video clip matching the dialogue text.

After determining the film and television clips associated with each dialogue text, determine the target film and television character in the film and television clips associated with each dialogue text that matches the role of the corresponding dialogue text. There is at least one film and television character in the associated film and television segment of each dialogue text. The embodiment of the present application determines the target film and television character in the at least one film and television character, and the target film and television character in the associated film and television segment matches the role of the dialogue text corresponding to the associated film and television segment. , for example, the dialogue text corresponding to the associated film and television segment A is dialogue text A, and the character of dialogue text A is B, then the determined target film and television character in the associated film and television segment A is also B.

After determining the target film and television character in each associated film and television segment, the position information of the target film and television character in the film and television frame corresponding to the associated video segment is determined. The target film and television character in each associated film and television clip matches the role of the corresponding dialogue text, so that the position information of the target film and television character in the film and television screen of the associated film and television clip can reflect the character of the dialogue text corresponding to the associated film and television clip. Position information, so the orientation information of each dialogue text can be determined based on the position information of the target film and television character in the associated film and television clip of each dialogue text in the film and television frame of the associated film and television clip.

As an example, the position information of the target film and television character in the film and television screen can be the coordinate information of the target film and television character in the film and television screen. The orientation information of each dialogue text is determined based on the coordinate information of the target film and television character in the film and television screen. For example, multiple coordinate ranges can be predetermined, and each coordinate range is associated with a piece of orientation information. If the synthesized voice is set to a two-channel voice, the orientation information For example, "left, right, and center." When the coordinates of the target film and television character in the film and television screen are within the coordinate range associated with the left orientation information, the orientation information of the target film and television character is determined to be the left.

In this embodiment, the position information of the target film and television character in the film and television segment associated with each dialogue text in the film and television screen of the associated film and television segment can reflect the location information of the character in each dialogue text. By determining the associated film and television character of each dialogue text The position information of the target film and television character in the clip in the film and television frame of the associated film and television clip is used to determine the position information of each dialogue text, which can improve the accuracy of the determined position information of each dialogue text.

Optionally, determine the target film and television character in the film and television clip associated with each dialogue text that matches the role of the corresponding dialogue text, including:

Perform target detection on the first associated film and television clip, and determine the target image in the first associated film and television clip that emits a dialogue voice that matches the first dialogue text, wherein the film and television character indicated by the target image is a character that matches the first dialogue text. For the target film and television character, the first related film and television clip is a related film and television clip of the first dialogue text, and the first dialogue text is any dialogue text in the first text.

In this embodiment, target detection is performed on the first associated film and television segment, and the target image in the first associated film and television segment that emits a dialogue voice that matches the first dialogue text is determined, so that the film and television character indicated by the target image matches the first dialogue text. Role matching means that the film and television character indicated by the target image is a target film and television character that matches the role of the first dialogue text, where the target image is, for example, a human face image or an animation character image.

To determine the target image that emits a dialogue voice that matches the first dialogue text, target detection needs to be performed on the first associated film and television clip. If multiple character images are detected, the image information in the multiple character images indicates that the mouth shape is in a dialogue state. And when the mouth shape is in a conversational state, the audio is also determined to be the above-mentioned conversational voice as the target image.

It should be noted that in the embodiment of the present application, the matching of the first dialogue text and the dialogue voice does not require that the first dialogue text and the dialogue voice completely match. It can be set that the similarity value between the first dialogue text and the dialogue voice exceeds a certain threshold and is determined. The first dialogue text matches the dialogue speech.

In the embodiment of the present application, the target image in the first associated film and television clip that emits the dialogue voice that matches the first dialogue text is determined to determine the target film and television character that matches the role of the first dialogue text, so that the determined target film and television character can be compared Accurate, thereby making the position information of the dialogue text determined based on the position information of the target film and television character more accurate.

Optionally, based on the content of each dialogue text and the film and television content of each video frame in the associated film and television resources, determine the associated film and television segments in the associated film and television resources that match each dialogue text, including:

According to the content of the first dialogue text and the subtitle content or audio content of each video frame in the associated film and television resources, the first associated film and television segment in the associated film and television resources that matches the first dialogue text is determined, and the first dialogue text is the Any dialogue text;

Perform scene recognition on the first associated film and television clip to determine the target scene element in the first associated film and television clip;

Determining scene keywords in the context text of the first dialogue text in the first text;

When the target scene element matches the scene keyword, target detection is performed on the first associated film and television clip.

In this implementation, if subtitle content exists in each video frame in the associated film and television resource, the first associated film and television segment in the associated film and television resource that matches the first dialogue text can be determined directly based on the subtitle content; if each video frame in the associated film and television resource If only audio content exists, the audio content can be converted into text content, and then the first associated film and television clip in the associated film and television resource that matches the first dialogue text can be determined based on the text content.

Film and television resources associated with e-books are usually subject to certain adaptations. For example, the first associated film and television clip in the associated film and television resource that matches the first dialogue text is determined based on the subtitle content. The subtitle content of multiple adjacent video frames is consistent with the first dialogue text. If the text similarity of the text exceeds a certain threshold (for example, 70%), it can be determined that the video segment formed by the plurality of adjacent video frames is the first associated film and television segment that matches the first dialogue text.

Among them, the above text similarity can be determined based on Jaccard similarity. Specifically, first segment the subtitles of multiple video frames and the first dialogue text. It is assumed that the first dialogue text is segmented to form a word set A, and the subtitles of multiple video frames are segmented to form a word set B:

Word set A={please/ask/the bathroom/is/where/? }

Word set B={please/ask/the toilet/is/where/? }

Jaccard similarity = the number of the same word segments between word set A and word set B ÷ the number of all word segments between word set A and word set B, so the Jaccard similarity between word set A and word set B is:

Length ({please/ask/at/where/?})÷length ({please/ask/toilet/at/where/?/toilet}=5/7=71.43%

It can be seen that the text similarity between the above word set A and word set B is 71.43%.

After determining the associated film and television clips of the first dialogue text, further, perform scene recognition on the first associated film and television clips, and determine the scene elements in the associated film and television clips. The scene elements include character elements, object elements, scenery elements, etc. As examples, scene elements include trees, streets, and houses.

Among them, character elements may include specific role elements. A character library can be constructed based on the associated film and television resources of the e-book. The character library includes the main characters and the character images corresponding to the main characters. If the related film and television resources are TV drama resources or movie resources, the character library can include the main characters and the corresponding images of the main characters. Face images. After the character library is constructed, scene recognition is performed on the associated film and television clips. If there is a character image in the character library in the film and television screen of the related film and television clip, the character element corresponding to the character image can be determined based on the character library.

After determining the target scene element of the first associated film and television clip, the scene keywords in the context text of the first dialogue text are determined, and only when the target scene element matches the scene keyword, the first associated film and television clip is processed Target Detection. Among them, the matching between the target scene element and the scene keyword can be understood as the hit rate of the target scene element, that is, whether the scene keyword contains the target scene element.

In this embodiment of the present application, after determining the first associated film and television segment of the first dialogue text, the target scene element of the first associated film and television segment is matched with the contextual scene keyword of the first dialogue text. If the target scene element matches the scene keyword matching, it can be further determined that the content of the first associated film and television segment contains content that matches the first dialogue text, that is, it can be further determined that the first associated film and television segment contains a target that emits dialogue voice that matches the first dialogue text. image. It can be seen that this application only performs target detection on the first associated film and television clip when the target scene element matches the scene keyword, which is beneficial to improving the reliability of target detection.

Optionally, perform scene recognition on the first associated film and television clip to determine the target scene element in the first associated film and television clip, including:

Perform scene recognition on the first associated film and television clip, and determine each scene element and the credibility of each scene element in the first associated film and television clip;

Scene elements whose credibility exceeds a preset threshold in the first associated video clip are determined as target scene elements.

The credibility of each scene element is used to reflect the credibility of the existence of the scene element in the first associated film and television clip. Determine the scene element whose credibility exceeds the preset threshold in the first associated film and television clip as the target scene element. For example, if in the first associated film and television clip, the credibility of the scene element tree is 80%, and the credibility of the scene element sky is 80%. If the degree is 60%, the credibility of the scene element moon is 30%, and the preset threshold is 50%, then the scene element trees and scene element sky can be determined as the target scene elements.

Embodiments of the present application determine the scene elements whose credibility exceeds a preset threshold in the first associated video clip as target scene elements, thereby improving the reliability of the target scene elements.

Optionally, the target information includes position keywords associated with each dialogue text in the first text;

According to the target information of each dialogue text in the first text, the orientation information of each dialogue text is determined, including:

Traverse the character names in the first text;

Determining positional keywords in the contextual text where each character name appears in the first text;

According to the dialogue text corresponding to each character name in the first text, the position keyword associated with each dialogue text is determined.

The same character in the e-book containing the first text may include a character name and a character nickname. Based on this, a character dictionary can be constructed in advance. The character name of a character in the character dictionary is associated with the character's nickname. The character dictionary can be as shown in the following table :

Table 1 Character Dictionary

Role Name nickname Ai Ling Zhang Ailing Lan Cheng Hu Lancheng Lu Xun Zhou Shuren, Mr. Zhou, Yucai, me Yongji Guan Yongji Ruisheng Hu Lancheng Xiao Zhou Zhou Xunde Aizhen She Aizhen beautiful Fan Xiumei Huiwen Quan Huiwen Qiwu Shen Qiwu

When the character nickname corresponding to each character name in the first text is detected, the character nickname is marked as the corresponding character name. It should be noted that when the e-book is written from the first-person perspective, the character nickname is still including me". For example, as shown in Table 1, in the e-book with Lu Xun as the first person, Lu Xun’s corresponding aliases include Zhou Shuren, Mr. Zhou, Yucai and me. When it is recognized that the text contains Zhou Shuren, Mr. Zhou, Yucai and me , labeling Zhou Shuren, Mr. Zhou, Yucai and me as Lu Xun.

Before traversing the character names of the first text, the first text can be segmented and the part-of-speech of each phrase can be marked, so that relevant information such as each character name, character alias, and position keywords in the first text can be determined based on the part-of-speech of each phrase. , the parts of speech of each phrase include but are not limited to the following table:

Table 2 Parts of speech table

n common noun m Quantifier f location noun q quantifier s location noun r pronoun nz Other proper names p preposition nw Title c conjunction v common verb u particle vd verb adverb xc Other function words vn noun verb w Punctuation a adjective PER name ad adverb LOC Place name an noun ORG Organization name d adverb TIME time

When it is recognized that the first text includes a character alias, the character alias is marked with a character name corresponding to the character alias based on the character dictionary. After marking the role alias as the role name corresponding to the role alias, traverse the role names of the first text, and determine the position keywords in the context text where each role name appears in the first text, and determine the location keywords in the context text where each role name appears according to the location where each role name appears. The location keyword in the contextual text of the location determines whether the character corresponding to each character name is close to a certain character, an object, or a certain place, thereby determining the location information associated with each character name. After determining the location keywords associated with each role name, determine the location information associated with each dialogue text based on the location keywords associated with each role name and the dialogue text corresponding to each role name, such as the location key associated with the role name Zhang San. If the word contains "on the sofa", then the position keyword associated with the dialogue text corresponding to the character name Zhang San also contains "on the sofa".

Among them, each dialogue text in the first text corresponds to the dialogue content of a character, and the character name corresponding to each dialogue text can be determined according to the context of each dialogue text. For example: the first text contains: Lixiang stands quietly next to the sofa, After a long silence, he said word by word: "Let's go, now." It can be seen that according to the part of speech of each phrase in the context of the dialogue text "Let's go, now.", its corresponding character name can be identified For Lixiang.

In the embodiment of the present application, the position keywords associated with each character name are determined by determining the position keywords in the context of the position where each character name appears in the first text, so that according to the position keywords associated with each character name and each character name The corresponding dialogue text can determine the position keywords associated with each dialogue text. It can be seen that the location keywords associated with each dialogue text determined in the embodiment of the present application can more accurately reflect the location of the character corresponding to each dialogue text. Based on this, each dialogue text determined by the location keywords associated with each dialogue text can be The sound source position of the audio to be synthesized is more accurate, which is beneficial to improving the playback effect of the synthesized language.

Optionally, determine the location keywords in the contextual text where each character name appears in the first text, including:

When the qualified description of the first character name in the first text satisfies the preset discrimination mode, determine the first local word corresponding to the first character name according to the limited description, wherein the qualified description at least includes the local word, and the first role The name is the name of any character in the first text;

Determine the first location corresponding to the first character name according to the first location word corresponding to the first character name;

The first character name, the first location word and the first location are correspondingly recorded in the location table, where, when the second location word corresponding to the second character name is the same as the first location word, the second location word recorded in the location table The second position corresponding to the character name is the same as the first position, and the second character name is any character name in the first text except the first character name;

According to the target information of each dialogue text in the first text, the orientation information of each dialogue text is determined, including:

According to the location words and directions corresponding to each character name recorded in the location table, the location of the dialogue text corresponding to each character name is determined.

In this embodiment, place words are used to reflect spatial positions. Place words are such as back rooms, doorways, roofs, etc. A place word list can be constructed in advance. In addition to place words, the limited expression can also include location words. The location words represent relative positions in space. Positions, such as: on the sofa, under a big tree, up, down, left, right, next to, front, back, side, etc., a list of location words can be built in advance.

As shown in Figure 4, when the qualified description of the first character name in the first text satisfies the preset discrimination mode, the first local word corresponding to the first character name is determined according to the local word in the qualified description, where, The above limited description includes at least the word of location.

For ease of understanding, the preset discrimination mode is described below with specific examples.

The discrimination mode can be as follows (optional content in square brackets):

Character name + [preposition] + place word + [location word]

If the first text is: "Chu Ming collapsed on the sofa, expressionless, staring out the window blankly. Lixiang stood quietly next to the sofa, silent for a long time, and said word by word: 'Let's go. , right now.' Changwen leaned at the door and urged impatiently: 'Make a decision quickly, don't keep dilly-dallying!'" In this paragraph of text, "Chu Ming", "Li Xiang" and "Chang Wen" are the character names, and "sofa" and "doorway" are the place names, which meet the above default judgment mode: Chu Ming + on + sofa +; Li Xiang +Beside the +sofa, therefore, both people are associated with "sofa". Changwen + is at the door, which conforms to the judgment model. Changwen is related to "door".

After determining the first location word corresponding to the first character name, the first location corresponding to the first character name is determined. Wherein, when the second location word corresponding to the second character name is the same as the first location word, the second location corresponding to the second role name recorded in the location table is the same as the first location, and the second role name is the first location word. For any character name in the text except the first character name, it can be seen that in the position table, the position corresponding to the character name associated with the same local word is the same. An orientation table is provided below as an example:

Table 3 Orientation table

Role Name word of place position Chu Ming sofa Left Lixiang sofa Left Changwen doorway right

As shown in Table 3, the location words related to Chu Ming and Li Xiang are both couches, so the directions of Chu Ming and Li Xiang are the same. If the synthesized speech is binaural speech, the directions can include "left, right and middle". Chu Ming The orientation of Chuming and Lixiang is the same, so the orientation of Chuming and Lixiang can be one of the orientations of "left, right and middle". Changwen has different place words related to Chuming and Lixiang, so the orientation of Changwen is the same as that of Chuming and Lixiang. The orientation of Lixiang is different. If the orientations of Chuming and Lixiang are "left", then the orientation of Changwen can be determined to be different from "left". For example, the orientation of Changwen can be "right". Based on this, it can be determined that the orientation information of the dialogue texts corresponding to Chu Ming and Li Xiang is the same. Therefore, the sound source positions of the audio to be synthesized for the dialogue texts corresponding to Chu Ming and Li Xiang are also the same. The sound source positions can both be left. source, the sound source position of the audio to be synthesized corresponding to the dialogue text of Changwen can be the right sound source.

In the embodiment of this application, a position table is set up. In the position table, the character names associated with the same place word have the same position, and the character names associated with different place words have different positions, so that the dialogue text corresponding to the character name with the same place word is associated. The sound source positions of the audio to be synthesized are the same, and the sound source positions of the dialogue texts corresponding to the character names associated with different local words are different. Based on this, when the local words associated with the character names are different, according to the sound source position The determined synthesized voice can feedback that the character is in different directions, thereby improving the playback effect of the synthesized voice.

Optionally, before generating the synthesized speech of each dialogue text according to the sound source position of the audio to be synthesized of each dialogue text, the method further includes:

Determine the emotional attributes of each dialogue text in the first text;

According to the emotional attributes of each dialogue text, determine the audio parameters of the audio to be synthesized for each dialogue text;

According to the sound source position of the audio to be synthesized in each dialogue text, the synthesized speech of each dialogue text is generated, including:

According to the sound source position and audio parameters of the audio to be synthesized for each dialogue text, the synthesized speech of each dialogue text is generated.

In this implementation, the emotion attributes are preset emotion categories. As an example, emotional attributes can be divided into five attributes, namely excitement, calmness, agitation, sadness, and tension. Each emotional attribute is associated with a corresponding audio parameter, so by determining the emotional attribute of each dialogue text, the audio parameters of the audio to be synthesized for each dialogue text can be determined. The audio parameters include audio loudness, audio speaking speed, and audio pitch.

Each emotional attribute is associated with a corresponding audio parameter. Specifically, the audio parameters associated with each emotional attribute are as shown in the following table:

Table 4 Audio parameter table

Among them, in the first text, the emotional attribute of the narration text except each dialogue text can all be "calm". According to the sound source position of the narration audio to be synthesized and the audio parameters corresponding to "calm", a synthesis of the narration can be generated. voice.

Embodiments of the present application generate synthetic speech for each dialogue text based on the sound source position and audio parameters of the audio to be synthesized for each dialogue text, so that the synthesized speech can feed back not only the emotion of the dialogue text, but also the orientation of the dialogue text. , it can be seen that the embodiments of the present application can make the content of the synthesized speech richer and the playback effect of the synthesized speech better.

Optionally, determining the emotional attributes of each dialogue text in the first text includes:

Perform word segmentation processing on the first dialogue text and the context of the first dialogue text to obtain multiple emotional phrases, wherein the first dialogue text is any dialogue text in the first text;

Query the emotional attributes corresponding to each of multiple emotional phrases through the pre-built emotional dictionary;

According to the emotional attribute corresponding to each emotional phrase, the emotional attribute of the first dialogue text is determined.

In this implementation, an emotion dictionary can be constructed in advance. The emotion dictionary contains multiple emotion attributes, and there are multiple phrases under each emotion attribute. The emotion dictionary is not an emotion dictionary specifically built for the phrases of an e-book. The emotion dictionary can be an emotion dictionary common to all e-books. As an example, the emotion dictionary can contain five emotion attributes. The five emotion attributes are respectively Excited, calm, excited, sad and nervous, part of the emotional dictionary can be shown in the following table:

Table 4 Emotion Dictionary

emotional phrases part of speech emotional attributes strength at a loss adjective nervous 7 incoherent adjective nervous 5 cry bitterly verb sorrow 6 laughter noun excited 3

By performing word segmentation processing on the first dialogue text and the context of the first dialogue text, multiple phrases can be obtained. The multiple phrases include the emotion phrases recorded in the emotion dictionary and other phrases. Based on the emotion dictionary, the words in the multiple phrases are determined. Multiple emotional phrases, and the emotional attributes corresponding to each emotional phrase are determined, so that the emotional attributes of the first dialogue text can be determined based on the emotional attributes corresponding to each emotional phrase.

By constructing an emotion dictionary, the embodiments of the present application can query the dialogue text and the emotion phrases in the context of the dialogue text based on the emotion dictionary, and by determining the emotion attributes corresponding to the emotion phrases, the emotion attributes of the dialogue text can be quickly determined.

Optionally, determine the emotional attributes of the first dialogue text based on the emotional attributes corresponding to each phrase, including:

When the emotional attributes corresponding to at least two emotional phrases in the first dialogue text and the context of the first dialogue text are different, determine the weight of the emotional attribute corresponding to each emotional phrase;

The emotional attribute with the highest weight is determined as the emotional attribute of the first dialogue text.

In one implementation, as shown in Figure 5, the weight of the emotional attribute corresponding to each emotional phrase can be based on the intensity value corresponding to each emotional phrase. As shown in Table 4, the emotion phrases in the emotion dictionary also include intensity values. It should be noted that each emotional phrase belonging to the same emotional attribute can be set to have different intensity values. For example, in Table 4, the intensity values of "overwhelmed" and "incoherent" belonging to the same emotional attribute are different. Compared with "incoherent", it has a higher degree of tension, so its intensity value is greater.

In the embodiment of the present application, when the emotional attributes corresponding to at least two emotional phrases in the first dialogue text and the context of the first dialogue text are different, by determining the emotional attribute with the highest weight as the emotional attribute of the first dialogue text, it is possible to improve Reliability of emotional properties of first conversational texts.

In another implementation, when the emotional attributes corresponding to at least two emotional phrases in the first dialogue text and the context of the first dialogue text are different, the emotional attributes corresponding to each emotional phrase can be determined, and according to each emotion The emotional attribute corresponding to the phrase determines the multiple emotional attributes of the first dialogue text, adjusts the audio parameters of the audio to be synthesized corresponding to the text in the first dialogue text according to the multiple emotional attributes of the first dialogue text, and generates a plurality of emotions for the first dialogue text The synthetic voice can enrich the emotion of the synthetic voice of the first dialogue text and effectively improve the playback effect of the synthetic voice of the first dialogue text.

Optionally, generate synthetic speech for each dialogue text based on the sound source position of the audio to be synthesized for each dialogue text, including:

According to the sound source position of the audio to be synthesized for each dialogue text, the time delay or volume difference between the first channel and the second channel of the audio to be synthesized for each dialogue text is determined, and a time delay or volume difference for each dialogue text is generated. Binaural synthesized speech.

In this embodiment, the final synthesized speech is set to be a binaural speech, and the sound source positions of the audio to be synthesized in each dialogue text may include "left, right and middle".

The embodiment of the present application generates binaural synthetic speech with time delay for each dialogue text based on the Haas effect. The Haas effect is: the same sound is emitted from two different sound sources. If the time difference between the two sound waves reaching the listener is between 5 and 35 milliseconds, the human ear cannot distinguish the location of the two sounds. The sound that people hear first will be considered as the direction from which all sounds come. Using the Haas effect, the left and right channels of sound are played asynchronously, and a certain delay is added artificially: M milliseconds, usually M is between 5 and 35.

In the embodiment of the present application, the first channel and the second channel of the audio to be synthesized correspond to the left channel and the right channel respectively. When there is no volume difference between the first channel and the second channel, if the sound source position is "right", the first channel (left channel) is delayed by M milliseconds compared to the second channel (right channel); if If the source direction is "left", the second channel is delayed by M milliseconds from the first channel; if the sound source direction is "middle", the left and right channels are synchronized without delay.

There is no time delay between the first channel and the second channel. Based on the volume difference between the first channel and the second channel of the audio to be synthesized for each dialogue text, a two-channel synthesis with a volume difference for each dialogue text is generated. voice. When there is a volume difference between the first channel and the second channel, the position of the sound source perceived by the human ear is biased towards the side with higher volume. There is no time delay between the first channel and the second channel. If the sound source position is "right", the volume of the second channel (right channel) is higher than the first channel (left channel); if If the source direction is "left", the volume of the first channel is higher than that of the second channel; if the sound source direction is "middle", there is no volume difference between the left and right channels.

In the implementation of this application, binaural synthetic speech with time delay or volume difference is generated for each dialogue text, so that when the user listens to the synthesized speech with headphones or a two-channel stereo speaker, the user can feel the position of the sound source in the synthesized speech, and, Existing headphones and other listening devices usually play back in two channels. Synthesizing conversational text into two-channel synthetic speech can effectively improve the probability that users can feel the location of the sound source in the synthesized speech based on the listening device, thereby improving the speech playback effect.

Optionally, before generating the synthesized speech of each dialogue text according to the sound source position of the audio to be synthesized of each dialogue text, the method further includes:

Determine the character attributes of each dialogue text according to the name information of each character in the first text marked in the preset character dictionary;

According to the sound source position of the audio to be synthesized in each dialogue text, the synthesized speech of each dialogue text is generated, including:

According to the role attributes of each dialogue text, determine the speaker corresponding to each dialogue text;

According to the sound source position of the audio to be synthesized in each dialogue text, the synthesized speech of each dialogue text is generated with the voice of the speaker corresponding to each dialogue text.

In this embodiment, the character attributes of each dialogue text are determined based on the name information of each character in the first text marked in the preset character dictionary. The character attributes of each dialogue text include the gender and age of the character. According to the role attributes of each dialogue text, the speaker corresponding to each dialogue text can be determined from the speaker's voice library. In the speaker's voice library, each speaker is associated with the corresponding timbre characteristics, such as young men, young women, old people, children, etc. If the role attribute of the dialogue text matches the timbre characteristics of the speaker, the speaker is determined to be the speaker corresponding to the dialogue text.

Embodiments of the present application generate synthetic speech for each dialogue text based on the voice of the speaker corresponding to each dialogue text, thereby further enriching the content of the synthetic speech and further improving the playback effect of the synthetic speech.

In order to have a clearer understanding of the technical solutions of the embodiments of the present application, the process of generating synthetic speech for each dialogue text will be described below with reference to FIG. 6 .

Before generating the synthesized speech of each dialogue text, the orientation information, emotional attributes and speaker of each dialogue text are determined. As an example, taking a text in "Romeo and Juliet", the orientation information, emotional attributes and Pronounced by:

Romeo: [medium, voiceover, calm]

Believe me, love, you are the same in my eyes; sorrow drains our blood. Goodbye! Goodbye! [Left, magnetic male voice, sad]

Juliet: [center, voiceover, calm]

Destiny, destiny! Everyone says you are capricious; if you are really capricious, how do you treat a loyal person? May you not change your frivolous nature, for then perhaps you will send him back sooner. [Right, gentle female voice, excited]

Lady Capulet: [medium, voiceover, calm]

Hello, daughter! Are you up? [In the middle, intellectual Xiaoyan, nervous].

Among them, in the embodiment of the present application, in addition to determining the information of each dialogue text, the information of the narration other than the external text can also be determined, so that the listener can distinguish the dialogue voice and the narration voice. In the above [in, narration voice, Calm] is the attribute of the narration other than the external text.

After determining the orientation information, emotional attributes and speaker of each text, a synthetic language is generated. As shown in Figure 6: the audio file A to be synthesized is generated using the voice of the speaker of the dialogue text through speech synthesis (Text to Speech, TTS) technology, and the audio file A to be synthesized is converted into a two-channel audio file A1. Determine the sound source position of the two-channel audio file A1 based on the orientation information of the dialogue text, generate the audio file A2 based on the sound source location of the two-channel audio file A1, and update the audio file based on the emotional attributes of the dialogue text and the audio parameters corresponding to the emotional attributes. The pitch, loudness and speaking speed of A2 are processed to generate the final synthesized speech audio file A3.

Referring to Figure 7, Figure 7 is a structural diagram of a speech synthesis device provided by an embodiment of the present application. As shown in Figure 7, a speech synthesis device 200 includes:

The first acquisition module 201 is used to acquire the first text, where the first text includes N dialogue texts, each dialogue text corresponds to the dialogue content of a character, and N is an integer greater than 1;

The first determination module 202 is used to determine the orientation information of each dialogue text according to the target information of each dialogue text in the first text, where the target information includes the position of the character corresponding to each dialogue text in the associated film and television clip of the dialogue text. Information, or positional keywords in the first text associated with each dialogue text;

The second determination module 203 is configured to determine the sound source location of the audio to be synthesized for each dialogue text based on the orientation information of each dialogue text, and generate a synthesized speech for each dialogue text based on the sound source location of the audio to be synthesized for each dialogue text. .

Optionally, the target information includes position information of the character corresponding to each dialogue text in the film and television clip associated with the dialogue text;

The first determination module 202 includes:

The first determination unit is configured to determine, when the e-book containing the first text has associated film and television resources, based on the content of each dialogue text and the film and television content of each video frame in the associated film and television resources, that the associated film and television resources are associated with each dialogue. Text matching related film and television clips;

The second determination unit is used to determine the target film and television character in the film and television clip associated with each dialogue text that matches the role of the corresponding dialogue text;

The third determination unit is used to determine the orientation information of each dialogue text based on the position information of the target film and television character in the associated film and television clip of each dialogue text in the film and television frame of the associated film and television clip.

Optionally, the second determination unit is specifically used for:

Perform target detection on the first associated film and television clip, and determine the target image in the first associated film and television clip that emits a dialogue voice that matches the first dialogue text, wherein the film and television character indicated by the target image is a character that matches the first dialogue text. For the target film and television character, the first related film and television clip is a related film and television clip of the first dialogue text, and the first dialogue text is any dialogue text in the first text.

Optionally, the first determining unit includes:

The first determination subunit is used to determine the first associated film and television segment in the associated film and television resource that matches the first dialogue text based on the content of the first dialogue text and the subtitle content or audio content of each video frame in the associated film and television resource, the first The dialogue text is any dialogue text in the first text;

The second determination subunit is used to perform scene recognition on the first associated film and television clip and determine the target scene element in the first associated film and television clip;

The third determination subunit is used to determine the scene keywords in the context text of the first dialogue text in the first text;

The first detection subunit is used to perform target detection on the first associated film and television clip when the target scene element matches the scene keyword.

Optionally, the second determination subunit is specifically used for:

Perform scene recognition on the first associated film and television clip, and determine each scene element and the credibility of each scene element in the first associated film and television clip;

Scene elements whose credibility exceeds a preset threshold in the first associated video clip are determined as target scene elements.

Optionally, the target information includes position keywords associated with each dialogue text in the first text;

The first determination module 202 also includes:

The first traversal unit is used to traverse the character names in the first text;

The fourth determination unit is used to determine the position keywords in the contextual text of the position where each character name appears in the first text;

The fifth determination unit is used to determine the position keywords associated with each dialogue text based on the dialogue text corresponding to each character name in the first text.

Optionally, the fourth determination unit is specifically used for:

When the qualified description of the first character name in the first text satisfies the preset discrimination mode, determine the first local word corresponding to the first character name according to the limited description, wherein the qualified description at least includes the local word, and the first role The name is the name of any character in the first text;

Determine the first location corresponding to the first character name according to the first location word corresponding to the first character name;

The first character name, the first location word and the first location are correspondingly recorded in the location table, where, when the second location word corresponding to the second character name is the same as the first location word, the second location word recorded in the location table The second position corresponding to the character name is the same as the first position, and the second character name is any character name in the first text except the first character name;

The first determination module 202 also includes:

The sixth determination unit is used to determine the location of the dialogue text corresponding to each character name based on the local word and location corresponding to each character name recorded in the location table.

Optionally, the speech synthesis device 200 also includes:

The third determination module is used to determine the emotional attributes of each dialogue text in the first text;

The fourth determination module is used to determine the audio parameters of the audio to be synthesized for each dialogue text based on the emotional attributes of each dialogue text;

The second determination module 203 includes:

The first generation unit is used to generate the synthesized speech of each dialogue text according to the sound source position and audio parameters of the audio to be synthesized of each dialogue text.

Optionally, the third determination module includes:

The first processing module is used to perform word segmentation processing on the first dialogue text and the context of the first dialogue text to obtain multiple emotional phrases, wherein the first dialogue text is any dialogue text in the first text;

The first query unit queries the emotional attributes corresponding to each of the multiple emotional phrases through a pre-built emotional dictionary;

The seventh determination unit is used to determine the emotional attributes of the first dialogue text based on the emotional attributes corresponding to each emotional phrase.

Optionally, the seventh determination unit is specifically used for:

When the emotional attributes corresponding to at least two emotional phrases in the first dialogue text are different, determine the weight of the emotional attribute corresponding to each emotional phrase;

The emotional attribute with the highest weight is determined as the emotional attribute of the first dialogue text.

Optionally, the first generation unit is specifically used for:

According to the sound source position of the audio to be synthesized for each dialogue text, the time delay or volume difference between the first channel and the second channel of the audio to be synthesized for each dialogue text is determined, and a time delay or volume difference for each dialogue text is generated. Binaural synthesized speech.

Optionally, the speech synthesis device also includes:

The fifth determination module is used to determine the character attributes of each dialogue text based on the name information of each character in the first text marked in the preset character dictionary;

The first generation unit is also specifically used for:

According to the role attributes of each dialogue text, determine the speaker corresponding to each dialogue text;

According to the sound source position of the audio to be synthesized in each dialogue text, the synthesized speech of each dialogue text is generated with the voice of the speaker corresponding to each dialogue text.

It should be noted that the speech synthesis device provided by the embodiments of the present application can execute the above method embodiments, and its implementation principles and technical effects are similar, and will not be described again in this embodiment.

An embodiment of the present application also provides an electronic device. Since the problem-solving principle of the electronic device is similar to the speech synthesis method in the embodiment of the present application, the implementation of the electronic device can be referred to the implementation of the method, and repeated details will not be repeated. As shown in Figure 8, the electronic device according to the embodiment of the present application includes: a processor 300, configured to read the program in the memory 320 and perform the following processes:

The first text is obtained through the transceiver 310, where the first text includes N dialogue texts, each dialogue text corresponds to the dialogue content of a character, and N is an integer greater than 1;

Determine the orientation information of each dialogue text according to the target information of each dialogue text in the first text, where the target information includes the position information of the character corresponding to each dialogue text in the film and television clip associated with the dialogue text, or the location information in the first text corresponding to the dialogue text. Positional keywords associated with each dialogue text;

Based on the orientation information of each dialogue text, the sound source location of the audio to be synthesized for each dialogue text is determined, and based on the sound source location of the audio to be synthesized for each dialogue text, a synthesized speech for each dialogue text is generated.

Transceiver 310 for receiving and transmitting data under the control of processor 300.

Optionally, the target information includes the position information of the character corresponding to each dialogue text in the associated film and television clips of the dialogue text. The processor 300 is also configured to read the program in the memory 320 and perform the following steps:

When the e-book containing the first text has associated film and television resources, determine the associated film and television clips in the associated film and television resources that match each dialogue text respectively based on the content of each dialogue text and the film and television content of each video frame in the associated film and television resources;

Determine the target film and television character in the film and television clip associated with each dialogue text that matches the role of the corresponding dialogue text;

The orientation information of each dialogue text is determined based on the position information of the target film and television character in the associated film and television segment of each dialogue text in the film and television frame of the associated film and television segment.

Optionally, the processor 300 is also used to read the program in the memory 320 and perform the following steps:

Perform target detection on the first associated film and television clip, and determine the target image in the first associated film and television clip that emits a dialogue voice that matches the first dialogue text, wherein the film and television character indicated by the target image is a character that matches the first dialogue text. For the target film and television character, the first related film and television clip is a related film and television clip of the first dialogue text, and the first dialogue text is any dialogue text in the first text.

Optionally, the processor 300 is also used to read the program in the memory 320 and perform the following steps:

According to the content of the first dialogue text and the subtitle content or audio content of each video frame in the associated film and television resources, the first associated film and television segment in the associated film and television resources that matches the first dialogue text is determined, and the first dialogue text is the Any dialogue text;

Perform scene recognition on the first associated film and television clip to determine the target scene element in the first associated film and television clip;

Determining scene keywords in the context text of the first dialogue text in the first text;

When the target scene element matches the scene keyword, target detection is performed on the first associated film and television clip.

Optionally, the processor 300 is also used to read the program in the memory 320 and perform the following steps:

Perform scene recognition on the first associated film and television clip, and determine each scene element and the credibility of each scene element in the first associated film and television clip;

Scene elements whose credibility exceeds a preset threshold in the first associated video clip are determined as target scene elements.

Optionally, the target information includes position keywords associated with each dialogue text in the first text;

The processor 300 is also used to read the program in the memory 320 and perform the following steps:

Traverse the character names in the first text;

Determining positional keywords in the contextual text where each character name appears in the first text;

According to the dialogue text corresponding to each character name in the first text, the position keyword associated with each dialogue text is determined.

Optionally, the processor 300 is also used to read the program in the memory 320 and perform the following steps:

When the qualified description of the first character name in the first text satisfies the preset discrimination mode, determine the first local word corresponding to the first character name according to the limited description, wherein the qualified description at least includes the local word, and the first role The name is the name of any character in the first text;

Determine the first location corresponding to the first character name according to the first location word corresponding to the first character name;

The first character name, the first location word and the first location are correspondingly recorded in the location table, where, when the second location word corresponding to the second character name is the same as the first location word, the second location word recorded in the location table The second position corresponding to the character name is the same as the first position, and the second character name is any character name in the first text except the first character name;

According to the location words and directions corresponding to each character name recorded in the location table, the location of the dialogue text corresponding to each character name is determined.

Optionally, the processor 300 is also used to read the program in the memory 320 and perform the following steps:

Determine the emotional attributes of each dialogue text in the first text;

According to the emotional attributes of each dialogue text, determine the audio parameters of the audio to be synthesized for each dialogue text;

According to the sound source position of the audio to be synthesized in each dialogue text, the synthesized speech of each dialogue text is generated, including:

According to the sound source position and audio parameters of the audio to be synthesized for each dialogue text, the synthesized speech of each dialogue text is generated.

Optionally, the processor 300 is also used to read the program in the memory 320 and perform the following steps:

Perform word segmentation processing on the first dialogue text and the context of the first dialogue text to obtain multiple emotional phrases, wherein the first dialogue text is any dialogue text in the first text;

Query the emotional attributes corresponding to each emotional phrase in multiple phrases through the pre-built emotion dictionary;

According to the emotional attribute corresponding to each emotional phrase, the emotional attribute of the first dialogue text is determined.

Optionally, the processor 300 is also used to read the program in the memory 320 and perform the following steps:

When the emotional attributes corresponding to at least two emotional phrases in the first dialogue text and the context of the first dialogue text are different, determine the weight of the emotional attribute corresponding to each emotional phrase;

The emotional attribute with the highest weight is determined as the emotional attribute of the first dialogue text.

Optionally, the processor 300 is also used to read the program in the memory 320 and perform the following steps:

According to the sound source position of the audio to be synthesized for each dialogue text, the time delay or volume difference between the first channel and the second channel of the audio to be synthesized for each dialogue text is determined, and a time delay or volume difference for each dialogue text is generated. Binaural synthesized speech.

Optionally, the processor 300 is also used to read the program in the memory 320 and perform the following steps:

Determine the character attributes of each dialogue text according to the name information of each character in the first text marked in the preset character dictionary;

According to the role attributes of each dialogue text, determine the speaker corresponding to each dialogue text;

According to the sound source position of the audio to be synthesized in each dialogue text, the synthesized speech of each dialogue text is generated with the voice of the speaker corresponding to each dialogue text.

In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, each unit may be physically included separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute some steps of the transceiver method in various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code. .

The above are the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, several improvements and modifications can be made without departing from the principles of the present application, and these improvements and modifications should also be regarded as the present invention. The scope of protection applied for.

Claims (10)

1. A speech synthesis method, characterized in that the method includes: Obtain the first text, wherein the first text includes N dialogue texts, each dialogue text corresponding to the dialogue content of a character; Determine the orientation information of each dialogue text according to the target information of each dialogue text in the first text, wherein the target information includes the position information of the character corresponding to each dialogue text in the associated film and television clip of the dialogue text, or the location information of each dialogue text. Describe the positional keywords associated with each dialogue text in the first text; According to the orientation information of each dialogue text, the sound source position of the audio to be synthesized of each dialogue text is determined, and based on the sound source position of the audio to be synthesized of each dialogue text, a synthesized speech of each dialogue text is generated. 2. The method according to claim 1, wherein the target information includes position information of the character corresponding to each dialogue text in the film and television clip associated with the dialogue text; Determining the orientation information of each dialogue text based on the target information of each dialogue text in the first text includes: In the case where the e-book containing the first text has associated film and television resources, based on the content of each dialogue text and the film and television content of each video frame in the associated film and television resources, it is determined that the associated film and television resources are associated with each dialogue text. Matching related film and television clips; Determine the target film and television character in the film and television clip associated with each dialogue text that matches the role of the corresponding dialogue text; The orientation information of each dialogue text is determined based on the position information of the target film and television character in the associated film and television segment of each dialogue text in the film and television frame of the associated film and television segment. 3. The method according to claim 2, wherein determining the target film and television character matching the character of the corresponding dialogue text in the associated film and television segments of each dialogue text includes: Perform target detection on the first associated film and television segment, and determine the target image in the first associated film and television segment that emits a dialogue voice that matches the first dialogue text, wherein the film and television character indicated by the target image is the same as the first dialogue text. The role of the dialogue text matches the target film and television character, the first associated film and television segment is an associated film and television segment of the first dialogue text, and the first dialogue text is any dialogue text in the first text. 4. The method according to claim 2 or 3, characterized in that, according to the content of each dialogue text and the film and television content of each video frame in the associated film and television resources, it is determined that each dialogue in the associated film and television resources is Related film and television clips for text matching, including: According to the content of the first dialogue text and the subtitle content or audio content of each video frame in the associated film and television resources, the first associated film and television segment matching the first dialogue text in the associated film and television resources is determined, and the first The dialogue text is any dialogue text in the first text; Perform scene recognition on the first associated film and television clip to determine the target scene element in the first associated film and television clip; Determine scene keywords in the context text of the first dialogue text in the first text; When the target scene element matches the scene keyword, target detection is performed on the first associated film and television clip. 5. The method according to claim 4, wherein the step of performing scene recognition on the first associated film and television clip and determining the target scene element in the first associated film and television clip includes: Perform scene recognition on the first associated film and television clip, and determine each scene element and the credibility of each scene element in the first associated film and television clip; Scene elements whose credibility exceeds a preset threshold in the first associated video clip are determined as the target scene elements. 6. The method according to claim 1, characterized in that the target information includes position keywords associated with each dialogue text in the first text; Determining the orientation information of each dialogue text based on the target information of each dialogue text in the first text includes: Traverse the character names in the first text; Determine the position keywords in the contextual text of the position where each character name appears in the first text; According to the dialogue text corresponding to each character name in the first text, the position keyword associated with each dialogue text is determined. 7. The method according to claim 6, characterized in that determining the position keywords in the contextual text of the position where each character name appears in the first text includes: When the qualified description of the first character name in the first text satisfies the preset discrimination mode, the first local word corresponding to the first character name is determined according to the qualified description, wherein the qualified description At least including a local word, the first character name is any character name in the first text; Determine the first location corresponding to the first character name according to the first location word corresponding to the first character name; The first character name, the first location word and the first location are correspondingly recorded in a location table, where, when the second location word corresponding to the second character name is the same as the first location word , the second orientation corresponding to the second character name recorded in the orientation table is the same as the first orientation, and the second character name is any character in the first text except the first character name. a role name; Determining the orientation information of each dialogue text based on the target information of each dialogue text in the first text includes: According to the location words and directions corresponding to each character name recorded in the location table, the location of the dialogue text corresponding to each character name is determined. 8. The method according to claim 1, characterized in that, before generating the synthesized speech of each dialogue text according to the sound source position of the audio to be synthesized of each dialogue text, the method further includes: Determine the emotional attributes of each dialogue text in the first text; Determine the audio parameters of the audio to be synthesized for each dialogue text according to the emotional attributes of each dialogue text; Generating the synthesized speech of each dialogue text according to the sound source position of the audio to be synthesized of each dialogue text includes: According to the sound source position and audio parameters of the audio to be synthesized of each dialogue text, the synthesized speech of each dialogue text is generated. 9. An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor; characterized in that the processor is used to read the program in the memory The steps of implementing the speech synthesis method according to any one of claims 1 to 8. 10. A computer-readable storage medium for storing a computer program, characterized in that, when the computer program is executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 8 are implemented.