TWI840949B - Multi-speaker and multi-emotion speech synthesis system, method and computer readable medium - Google Patents

Abstract

The invention discloses a multi-speaker and multi-emotion speech synthesis system, method and computer readable medium. A speaker coding module generates an embedded speaker vector according to a multi-speaker speech reference signal, an emotion coding module generates an embedded emotion vector according to a multi-emotion speech reference signal, and a coding unit of a speech synthesis module generates an embedded linguistic vector according to a grapheme or phoneme sequence. Next, an integration unit of the speech synthesis module integrates the embedded speaker vector, the embedded emotion vector and the embedded linguistic vector into an embedded control vector, and then the integration unit controls a decoding unit of the speech synthesis module to synthesize speech spectrum by using the embedded control vector that integrates the embedded speaker vector, the embedded emotion vector and the embedded linguistic vector.

Description

Multilingual and multi-emotion speech synthesis system, method and computer-readable medium

The present invention relates to a speech synthesis technology, and more particularly to a multilingual and multi-emotion speech synthesis system, method and computer-readable medium.

Speech synthesis technology has been widely used in various electronic devices and human-computer interfaces, such as Apple's iPhone smartphone and voice assistant Siri, Amazon's Echo smart speaker and voice assistant Alexa, Google's smart speaker Google Home and voice assistant Google Assistant, etc.

Although speech synthesis technology can use human-computer interfaces to respond to or prompt relevant information to users of electronic devices, general speech synthesis technology can only synthesize neutral voices, but rarely considers synthesizing emotional voices, resulting in the inability to use emotional voices to conduct efficient interactive dialogues with users.

Furthermore, existing speech synthesis technology cannot generate embedded speaker vectors, embedded emotion vectors, and embedded context vectors based on speech reference signals of multiple speakers, speech reference signals of multiple emotions, and text or phonetic symbol sequences, nor can it generate personalized and emotional speech spectra and/or synthesized speech based on personalized needs, nor can it generate personalized and emotional speech spectra and/or synthesized speech based on transformation functions (such as vector concatenation transformation functions or learnable nonlinear neural networks). The embedded speaker vector, embedded emotion vector, and embedded context vector are integrated into an embedded control vector to control the speech synthesis module to synthesize a personalized and emotional speech spectrum. It is not possible to use a neural network to improve the quality of synthesized speech, and it is not possible for the vocoding module to use the speech spectrum (such as the Mel spectrum or the logarithmic Mel spectrum) synthesized by the speech synthesis module to generate personalized and emotional synthesized speech.

Therefore, how to provide an innovative speech synthesis technology to solve any of the above problems or provide related functions/services has become a major research topic for technical personnel in this field.

The present invention provides an innovative multilingual and multi-emotion speech synthesis system, method and computer-readable medium, which can generate an embedded speaker vector, an embedded emotion vector and an embedded context vector according to a multilingual speech reference signal, a multi-emotion speech reference signal and a text or phonetic symbol sequence, and can also generate a personalized and emotional speech spectrum and/or synthesized speech according to personalized needs, or according to a transformation function (such as a vector concatenated transformation function or a learnable The embedded speaker vector, the embedded emotion vector, and the embedded context vector are integrated into an embedded control vector to control the speech synthesis module to synthesize a personalized and emotional speech spectrum, or the quality of the synthesized speech is improved by using the neural network, or the speech spectrum (such as the Mel spectrum or the logarithmic Mel spectrum) synthesized by the speech synthesis module is used by the vocoding module to generate personalized and emotional synthesized speech.

The multilingual and multi-emotion speech synthesis system of the present invention comprises: a speaker encoding module, which generates a corresponding embedded speaker vector based on at least one of the speech reference signals of the multilingual speakers; an emotion encoding module, which generates a corresponding embedded emotion vector based on at least one of the multi-emotion speech reference signals; and a speech synthesis module, which has a coding unit, an integration unit and a decoding unit, wherein the coding unit of the speech synthesis module generates a corresponding embedded context vector based on a text or phonetic symbol sequence, and the speech synthesis module generates a corresponding embedded context vector based on a text or phonetic symbol sequence. The integration unit integrates the embedded speaker vector generated by the speaker encoding module based on the speech reference signals of multiple speakers, the embedded emotion vector generated by the emotion encoding module based on the speech reference signals of multiple emotions, and the embedded context vector generated by the encoding unit based on the text or phonetic symbol sequence into an embedded control vector. The integration unit of the speech synthesis module then uses the embedded speaker vector, the embedded emotion vector, and the embedded context vector to integrate the embedded control vector to control the decoding unit of the speech synthesis module to synthesize the speech spectrum.

The multilingual and multi-emotion speech synthesis method of the present invention comprises: a speaker encoding module generates a corresponding embedded speaker vector according to at least one of the multilingual speech reference signals, and an emotion encoding module generates a corresponding embedded emotion vector according to at least one of the multi-emotion speech reference signals, and a coding unit of the speech synthesis module generates a corresponding embedded context vector according to a text or phonetic symbol sequence; and an integration unit of the speech synthesis module integrates the speaker encoding module according to the multilingual speech reference signals. The embedded speaker vector generated by the speech reference signal, the embedded emotion vector generated by the emotion coding module based on the multi-emotion speech reference signal, and the embedded context vector generated by the coding unit based on the text or phonetic symbol sequence are integrated into an embedded control vector. The integration unit of the speech synthesis module then uses the embedded speaker vector, the embedded emotion vector, and the embedded context vector to integrate the embedded control vector to control the decoding unit of the speech synthesis module to synthesize the speech spectrum.

The computer-readable medium of the present invention is applied to a computing device or a computer, and stores instructions for executing the above-mentioned multilingual and multi-emotion speech synthesis method.

In order to make the above features and advantages of the present invention more clearly understandable, the following examples are given and detailed descriptions are provided in conjunction with the attached drawings. The following description will partially explain the additional features and advantages of the present invention, and these features and advantages will be partially known from the description or can be learned through the practice of the present invention. It should be understood that both the general description above and the detailed description below are exemplary and explanatory, and are not intended to limit the scope of the present invention.

1: Multilingual and multi-emotion speech synthesis system

10: Speaker encoding module

11: Neural network based on embedded speaker vector extraction

12: Sound frame level

13: Input features

14: First statistical pooling layer

15: Sentence level

16: Speaker's Mark

17:softmax classifier

18: Loss function

19: First database

20: Emotional coding module

21: Neural networks such as embedded emotion vector extraction

22: Sound frame level

23: Input features

24: Second statistical pooling layer

25: Sentence level

26: Emotional markers

27:softmax classifier

28: Loss function

29: Second database

30: Speech synthesis module

31: Coding unit

32: Integration unit

33: Attention unit

34: Decoding unit

40: Voice coding module

A1: Voice reference signal of multilingual speakers

A2: Embedded speaker vector

B1: Multi-emotional voice reference signals

B2: Embedded emotion vector

C1: text or phonetic symbol sequence

C2: Embedded context vector

C3: Embedded Control Vector

C4: Voice spectrum

C5: Synthetic Speech

S1 to S2: Steps

Figure 1 is a schematic diagram of the architecture of the multilingual and multi-emotion speech synthesis system of the present invention.

Figure 2 is a schematic diagram of the process of the multilingual and multi-emotion speech synthesis method of the present invention.

FIG3 is a schematic diagram of an implementation example of the embedded speaker vector analysis method in the multilingual and multi-emotion speech synthesis system and method of the present invention.

FIG4 is a schematic diagram of an implementation example of the embedded emotion vector analysis method in the multilingual and multi-emotion speech synthesis system and method of the present invention.

The following describes the implementation of the present invention through a specific concrete implementation form. People familiar with this technology can understand other advantages and effects of the present invention from the content disclosed in this manual, and can also implement or use it through other different specific equivalent implementation forms.

FIG1 is a schematic diagram of the structure of the multilingual and multi-emotion speech synthesis system 1 of the present invention, FIG2 is a schematic diagram of the process of the multilingual and multi-emotion speech synthesis method of the present invention, FIG3 is a schematic diagram of an embodiment of the multilingual and multi-emotion speech synthesis system 1 of the present invention and the method thereof, and FIG4 is a schematic diagram of an embodiment of the multilingual and multi-emotion speech synthesis system 1 of the present invention and the method thereof.

As shown in FIG1 , a multilingual and multi-emotion speech synthesis system 1 may include at least one speaker encoding module 10, at least one emotion encoding module 20, at least one speech synthesis module 30 and at least one voice encoding module 40 that are interconnected or communicated. In one embodiment, the speech synthesis module 30 may be respectively connected or communicated with the speaker encoding module 10, the emotion encoding module 20 and the voice encoding module 40. The speech synthesis module 30 may have at least one encoding unit 31, at least one integration unit 32, at least one attention unit 33 and at least one decoding unit 34. In one embodiment, the encoding unit 31 may be sequentially connected or communicated with the integration unit 32, the attention unit 33 and the decoding unit 34.

In one embodiment, the speaker encoding module 10 may be a speaker encoder, a speaker encoding chip, a speaker encoding circuit, a speaker encoding software, a speaker encoding program, etc., the emotion encoding module 20 may be an emotion encoder, an emotion encoding chip, an emotion encoding circuit, an emotion encoding software, an emotion encoding program, etc., the speech synthesis module 30 may be a speech synthesizer, a speech synthesis chip, a speech synthesis circuit, a speech synthesis software, a speech synthesis program, etc., and the vocoder module 40 may be a vocoder, a vocoder chip, a vocoder circuit, a vocoder software, a vocoder program, etc. The encoding unit 31 may be an encoder, an encoding chip, an encoding circuit, an encoding software, an encoding program, etc., the integration unit 32 may be an integration software, an integration program, etc., the attention unit 33 may be an attention software, an attention program, etc., and the decoding unit 34 may be a decoder, a decoding chip, a decoding circuit, a decoding software, a decoding program, etc.

In one embodiment, the "connection or communication" mentioned in the present invention may represent interconnection or communication in a wired manner (such as a wired network) or a wireless manner (such as a wireless network), "at least one" represents more than one (such as one, two or more than three), and "plurality" represents more than two (such as two, three, four, five or more than ten). However, the present invention is not limited to those mentioned in the above embodiments.

[張三]{生氣}我不這麼認為！

[李四]{厭惡}這個東西能用嗎？進一步地，多語者與多情緒語音合成系統1及其方法亦可在同一句話中變化情緒，例如：

[王五]{中性}只看外表，{傷心}會準嗎？

[趙六]{中性}每年到了這個時候^.....{快樂}聖誕樹就出來了。前述[張三]、[李四]、[王五]、[趙六]等為語者標記16，{生氣}、{厭惡}、{中性}、{傷心}、{快樂}等為情緒標記26。 The multilingual and multi-emotion speech synthesis system 1 and the method thereof generate personalized and emotional speech spectrum C4 and/or synthesized speech C5 (see FIG. 1 ) according to the text input by the user and the specified speaker tag 16 (see FIG. 3 ) and emotion tag 26 (see FIG. 4 ), for example:

[Zhang San]{Angry}I don’t think so!

[Li Si] Can this thing {disgust} be used? Furthermore, the multilingual person and multi-emotion speech synthesis system 1 and the method thereof can also change emotions in the same sentence, for example:

[Wang Wu] {Neutral} Only looking at the appearance, {Sad} will it be accurate?

[Zhao Liu] {neutral} Every year at this time ^... {happy} the Christmas tree comes out. The aforementioned [Zhang San], [Li Si], [Wang Wu], [Zhao Liu], etc. are speaker markers 16, and {angry}, {disgusted}, {neutral}, {sad}, {happy}, etc. are emotion markers 26.

The multilingual and multi-emotion speech synthesis system 1 and its method are characterized by constructing a speaker encoding module 10 to generate/obtain multiple (different) embedded speaker vectors A2 to represent the voice characteristics of multiple (different) speakers, and constructing an emotion encoding module 20 to generate/obtain multiple (different) embedded emotion vectors B2 to represent multiple (different) emotion types.

For example, the technical features of the present invention may include constructing a distribution model of an embedded speaker vector A2 by a speaker coding module 10 to represent the voice characteristics of the speaker, and constructing a distribution model of an embedded emotion vector B2 by an emotion coding module 20 to represent multiple (different) emotion types, so as to construct a multilingual and multi-emotion speech synthesis system 1 guided by both the embedded speaker vector A2 and the embedded emotion vector B2. That is, the multilingual and multi-emotion speech synthesis system 1 can construct a speaker coding module 10 that can generate/obtain multiple (different) embedded speaker vectors A2, and construct an emotion coding module 20 that can generate/obtain multiple (different) embedded emotion vectors B2. Therefore, the multilingual and multi-emotion speech synthesis system 1 constructed by the present invention can complete the speech synthesis processing according to the text input by the user and after specifying the relevant conditions of the speaker and emotion.

As shown in FIG1 , the speech synthesis module 30 may use end-to-end speech synthesis (such as Tacotron2) technology, a coding-decoding architecture (such as a coding unit 31 combined with a decoding unit 34) and/or an attention mechanism of an attention unit 33, so that the decoding unit of the speech synthesis module 30 can automatically synthesize a personalized and emotional speech spectrum C4.

The operation principle of the speech synthesis module 30 can be briefly described as follows: the encoding unit 31 of the speech synthesis module 30 first performs text analysis on the input grapheme or phoneme sequence C1 to automatically generate an embedded linguistic vector C2 corresponding to the grapheme or phoneme sequence C1, and then the decoding unit 34 of the speech synthesis module 30 automatically aligns the input grapheme or phoneme sequence C1 with the speech spectrum C4 output by the decoding unit 34 according to the embedded linguistic vector C2 (such as the context feature parameter) and through the weight mechanism of the attention unit 33, and the speech spectrum C4 can be, for example, a Mel spectrogram or a log-Mel spectrogram.

As shown in Figures 1 and 2, the multilingual and multi-emotion speech synthesis methods mainly include:

In step S1, the speaker encoding module 10 generates a corresponding embedded speaker vector A2 according to at least one of the multi-speaker speech reference signals A1, and the emotion encoding module 20 generates a corresponding embedded emotion vector B2 according to at least one of the multi-emotion speech reference signals B1, and the encoding unit 31 of the speech synthesis module 30 generates a corresponding embedded context vector C2 according to the text or phonetic symbol sequence C1.

Then, in step S2, the integration unit 32 of the speech synthesis module 30 integrates the embedded speaker vector A2 generated by the speaker encoding module 10 according to the multi-speaker speech reference signal A1, the embedded emotion vector B2 generated by the emotion encoding module 20 according to the multi-emotion speech reference signal B1, and the embedded context vector C2 generated by the encoding unit 31 according to the text or phonetic symbol sequence C1 into an embedded control vector C3, and then the integration unit 32 of the speech synthesis module 30 uses the embedded control vector C3 integrated with the embedded speaker vector A2, the embedded emotion vector B2, and the embedded context vector C3 to control the decoding unit 34 of the speech synthesis module 30 to synthesize the speech spectrum C4.

The multilingual and multi-emotion speech synthesis system 1 and the method thereof can utilize a neural network to improve the quality of synthesized speech C5, and can also continuously develop, pursue or demonstrate personalized and emotional (diversified) speech synthesis functions. To achieve personalized and emotional speech synthesis functions, the multilingual and multi-emotion speech synthesis system 1 and the method thereof can utilize multilingual and multi-emotion speech synthesis training corpus to respectively construct a speaker encoding module 10 and an emotion encoding module 20, so that the subsequent speech synthesis module 30 and the voice coding module 40 can respectively automatically synthesize personalized and emotional speech spectrum C4 and synthesized speech C5.

In detail, the technical contents of the multilingual and multi-emotion speech synthesis system 1 and its method can be divided into the following three parts for explanation. [1] The first part may include a processing procedure for controlling the multilingual and multi-emotion speech synthesis system 1 with an embedded speaker vector A2 and an embedded emotion vector B2 as shown in FIG1; [2] The second part may include: [2-1] an embedded speaker vector analysis method as shown in FIG3, and [2-2] an embedded emotion vector analysis method as shown in FIG4; [3] The third part may include an integration method of the speech synthesis module 30 32 as shown in FIG1 for integrating the embedded speaker vector A2, the embedded emotion vector B2, and the embedded context vector C2, and a training method of a related neural network (model).

[1] The processing procedure (processing mechanism) of the multilingual and multi-emotion speech synthesis system 1 is controlled by the embedded speaker vector A2 and the embedded emotion vector B2. As shown in FIG1 , the speaker encoding module 10, the emotion encoding module 20, the speech synthesis module 30 and the voice encoding module 40, etc., must be constructed by using the multilingual and multi-emotion speech synthesis training corpus with text, speaker tags 16 and emotion tags 26 for training, and the processing procedure of the multilingual and multi-emotion speech synthesis system 1 is controlled by the embedded speaker vector A2 and the embedded emotion vector B2, which may include the following procedures P11 to P16.

Procedure P11: The speaker encoding module 10 generates/obtains the corresponding embedded speaker vector A2 based on at least one of the input multi-speaker speech reference signals A1 (such as multi-speaker speech reference waveforms).

Procedure P12: The emotion coding module 20 generates/obtains the corresponding embedded emotion vector B2 based on at least one of the input multi-emotion speech reference signals B1 (such as multi-emotion speech reference waveforms).

Procedure P13: The encoding unit 31 of the speech synthesis module 30 generates/obtains the corresponding embedded context vector C2 according to the input text or phonetic symbol sequence C1.

Procedure P14: The integration unit 32 of the speech synthesis module 30 integrates the embedded speaker vector A2 generated by the speaker coding module 10, the embedded emotion vector B2 generated by the emotion coding module 20, and the embedded context vector C2 generated by the speech synthesis module 30 to obtain the embedded control vector C3.

Program P15: The integration unit 32 of the speech synthesis module 30 uses the embedded control vector C3 formed by integrating the embedded speaker vector A2, the embedded emotion vector B2, and the embedded context vector C2 to automatically control the attention unit 33 and the decoding unit 34 of the speech synthesis module 30 to synthesize a personalized and emotional speech spectrum C4 (such as a Mel spectrum or a logarithmic Mel spectrum).

Procedure P16: The voice coding module 40 uses the attention mechanism provided by the attention unit 33 of the speech synthesis module 30 and the speech spectrum C4 (such as the Mel spectrum or the logarithmic Mel spectrum) synthesized by the decoding unit 34 to automatically generate personalized and emotional synthesized speech C5 (such as the synthesized speech signal).

[2-1] Embedded speaker vector analysis method (processing procedure): As shown in Figures 1 and 3, the relevant processing procedures of the embedded speaker vector analysis method may include the following procedures P21 to P24.

Procedure P21: The speaker encoding module 10 uses the training corpus with speaker labels 16 to train a speaker identification neural network with a first statistics pooling layer 14, and the speaker identification neural network can be divided into two parts, namely, a frame level 12 and an utterance level 15. The first statistics pooling layer 14 of the speaker identification neural network converts the information of the frame level 12 of the speaker identification neural network into the information of the utterance level 15. The speaker encoding module 10 can choose to use Mel-Frequency Cepstral Coefficients (MFCC) as the input feature 13 of the speaker identification neural network, and the speaker encoding module 10 can also define the loss function 18 of the speaker identification neural network based on the softmax classifier 17.

Procedure P22: After the output layer of the trained speaker identification neural network is removed, the speaker encoding module 10 can use the output of the first statistical pooling layer 14 of the speaker identification neural network as the embedded speaker vector A2 of each input sentence, so that the trained speaker identification neural network becomes the embedded speaker vector extraction neural network 11.

Procedure P23: The speaker encoding module 10 uses the embedded speaker vector extraction neural network 11 to extract the embedded speaker vector A2 of each input sentence.

Procedure P24: The speaker encoding module 10 constructs a first database 19 according to the speaker tag 16 of each input sentence and the corresponding embedded speaker vector A2 provided by the embedded speaker vector extraction neural network 11. For example, the first database 19 may store a plurality of (different) speaker tags 16 and corresponding plurality of (different) embedded speaker vectors A2, the speaker tag 16 may include the first speaker tag, the second speaker tag to the Mth speaker tag, etc., the embedded speaker vector A2 may include the first embedded speaker vector, the second embedded speaker vector to the Mth embedded speaker vector, etc., and M represents a positive integer greater than 2 (such as 3, 4, 5 or more).

[2-2] Embedded emotion vector analysis method (processing procedure): As shown in Figures 1 and 4, the relevant processing procedures of the embedded emotion vector analysis method may include the following procedures P31 to P34.

Procedure P31: The emotion encoding module 20 uses the training corpus with emotion labels 26 to train a sentence emotion recognition neural network with a second statistical pooling layer 24, and the sentence emotion recognition neural network can be divided into two parts, namely, a sound frame level 22 and a sentence level 25. The second statistical pooling layer 24 of the sentence emotion recognition neural network converts the sound frame level information of the sentence emotion recognition neural network into sentence level information. The emotion coding module 20 can choose to use the Mel frequency cepstral coefficient (MFCC) and features such as pitch and energy that can reflect emotional changes as input features 23 of the sentence emotion recognition neural network, and the emotion coding module 20 can also define the loss function 28 of the sentence emotion recognition neural network based on the softmax classifier 27.

Procedure P32: After the output layer of the trained sentence emotion recognition neural network is removed, the emotion encoding module 20 can use the output of the second statistical pooling layer 24 of the sentence emotion recognition neural network as the embedded emotion vector B2 of each input sentence, so that the trained sentence emotion recognition neural network becomes the embedded emotion vector extraction neural network 21.

Procedure P33: The emotion encoding module 20 uses the embedded emotion vector extraction neural network 21 to extract the embedded emotion vector B2 of each input sentence.

Procedure P34: The emotion coding module 20 builds a second database 29 according to the emotion label 26 of each input sentence and the corresponding embedded emotion vector B2 provided by the embedded emotion vector extraction neural network 21. For example, the second database 29 may store a plurality of (different) emotion labels 26 and corresponding plurality of (different) embedded emotion vectors B2, the emotion label 26 may include the first emotion label, the second emotion label to the Nth emotion label, etc., the embedded emotion vector B2 may include the first embedded emotion vector, the second embedded emotion vector to the Nth embedded emotion vector, etc., and N represents a positive integer greater than 2 (such as 3, 4, 5 or more).

[3] The integration unit 32 of the speech synthesis module 30 integrates the embedded speaker vector A2, the embedded emotion vector B2, and the embedded context vector C2.

As shown in Figure 1 and the following formulas (1) to (2), F(.) represents the integration unit 32 of the speech synthesis module 30, and the integration unit 32 (such as F(.)) can define the corresponding transformation function (such as a vector concatenation transformation function or a learnable nonlinear neural network-like transformation function) according to different selection methods, so that the integration unit 32 (such as F(.)) automatically integrates the embedded speaker vector A2, the embedded emotion vector B2, and the embedded context vector C2 into an embedded control vector C3 (such as a fixed-length embedded control vector C3) according to the selected transformation function (such as a vector concatenation transformation function or a learnable nonlinear neural network-like transformation function). That is, the conversion function of the integration unit 32 (such as F(.)) can be selected as a vector concatenation conversion function as shown in formula (1), or a learnable nonlinear neural network conversion function as shown in formula (2).

、

、

分別代表嵌入式語者向量A2、嵌入式情緒向量B2、與嵌入式文脈向量C2， 而ω_s、ω_e、ω_l分別代表嵌入式語者向量A2之可調整加權值、嵌入式情緒向量B2之可調整加權值、與嵌入式文脈向量C2之可調整加權值。 In the above formulas (1) to (2), F(.) represents the integrated unit 32,

,

,

They represent the embedded speaker vector A2, the embedded emotion vector B2, and the embedded context vector C2 respectively, and ω _s , ω _e , ω _l represent the adjustable weighted value of the embedded speaker vector A2, the adjustable weighted value of the embedded emotion vector B2, and the adjustable weighted value of the embedded context vector C2 respectively.

When the integration unit 32 selects a learnable nonlinear neural network, the learnable nonlinear neural network can further integrate the attention unit 33 and the decoding unit 34 (such as a decoding network) of the speech synthesis module 30 to obtain an integrated neural network, and the integrated neural network can use the embedded control vector C3 formed by integrating the embedded speaker vector A2, the embedded emotion vector B2, and the embedded context vector C2 to complete the training construction.

Then, the decoding unit 34 of the speech synthesis module 30 may input the generated speech spectrum C4 (such as a Mel spectrum or a logarithmic Mel spectrum) to the vocoding module 40, so that the vocoding module 40 may synthesize the personalized (such as speaker characteristics) and emotional (such as speaker emotions) synthesized speech C5 using the speech spectrum C4. That is, the vocoding module 40 may select one of a plurality of neural networks such as WaveNet, WaveGlow, WaveGAN and HiFi-GAN to synthesize the speech spectrum C4 into the personalized (such as speaker characteristics) and emotional (such as speaker emotions) synthesized speech C5 according to different considerations such as the sound quality acceptability of the synthesized speech C5 and/or the system execution speed. It should be noted that the aforementioned WaveNet, WaveGlow, WaveGAN and HiFi-GAN are usually expressed in English and rarely in Chinese.

The established multilingual and multi-emotion speech synthesis system 1 and method thereof can generate personalized and emotional synthesized speech C5 according to the text input by the user and the specified speaker tag 16 and emotion tag 26, and can also find the embedded speaker vector A2 corresponding to the speaker tag 16 from the first database 19 shown in FIG. 3, and find the embedded emotion vector B2 corresponding to the emotion tag 26 from the second database 29 shown in FIG. 4, and then input the embedded speaker vector A2 and the embedded emotion vector B2 to the speech synthesis module 30 respectively for speech synthesis processing.

For example, the related processing procedures of the multilingual and multi-emotion speech synthesis system 1 and its method may include the contents described below.

[1] The user first collects multilingual and multi-emotion speech synthesis training corpus with text, speaker tags 16 and emotion tags 26 through the multilingual and multi-emotion speech synthesis system 1, and the method includes (1) recruiting speakers and designing recording texts and recording methods, or (2) using various public corpus. In terms of (1) recruiting speakers and designing recording texts and recording methods, the recording speakers can record corpus with multiple (different) emotions according to different needs. In terms of (2) using various public corpus, you can choose to use corpus such as TV series or podcasts.

Taking English as an example, Multimodal Emotion Lines Dataset (MELD; URL: https://affective-meld.github.io/) or Multimodal Signal Processing-Podcast corpus (MSP-Podcast; URL: https://ecs.utdallas.edu/research/researchlabs/msp-lab/MSP-Podcast.html) are both annotated multilingual and multi-emotion speech synthesis training corpora.

[2] The speaker labels 16 in the multilingual and multi-emotion speech synthesis training corpus are used to train the speaker encoding module 10 shown in FIG. 1 or the embedded speaker vector extraction neural network shown in FIG. 3, so that the speaker encoding module 10 uses the embedded speaker vector extraction neural network 11 to extract the embedded speaker vector A2 of each input sentence, and the embedded speaker vector extraction neural network 11 is trained using the x-vector training method, and the speaker encoding module 10 can choose to use the Mel frequency cepstral coefficient (MFCC) as the input feature 13 of the embedded speaker vector extraction neural network 11.

[3] The emotion markers 26 in the multilingual and multi-emotion speech synthesis training corpus are used to train the emotion coding module 20 shown in FIG. 1 or the embedded emotion vector extraction neural network 21 shown in FIG. 4, so that the emotion coding module 20 uses the embedded emotion vector extraction neural network 21 to extract the embedded emotion vector B2 of each input sentence, and the embedded emotion vector extraction neural network 21 is trained using an x-vector training method, and the emotion coding module 20 can choose to use Mel frequency inverse spectrum coefficients (MFCC) and features such as pitch and energy that can reflect emotional changes as input features 23 of the embedded emotion vector extraction neural network 21.

[4] Using all the text, speaker tags 16 and emotion tags 26 of the above-mentioned multilingual and multi-emotion speech synthesis training corpus, as well as the trained speaker encoding module 10 (such as the embedded speaker vector extraction neural network 11) and the emotion encoding module 20 (such as the embedded emotion vector extraction neural network 21) to train the speech synthesis module 30, including training the front-end encoding unit 31 and integration unit 32 (such as F(.)), the middle-end attention unit 33, and the back-end decoding unit 34 of the speech synthesis module 30, so that the speech synthesis module 30 generates the corresponding speech spectrum C4 (such as Mel spectrum or logarithmic Mel spectrum) information.

The architecture of the encoding unit 31, the attention unit 33 and the decoding unit 34 of the speech synthesis module 30 can utilize end-to-end speech synthesis (such as Tacotron2) technology, and the integration unit 32 (such as F(.)) can define the corresponding transformation function (such as a vector concatenation transformation function or a learnable nonlinear neural network-like transformation function) according to different selected methods, so that the integration unit 32 (such as F(.)) can automatically integrate the embedded speaker vector A2, the embedded emotion vector B2, and the embedded context vector C2 into an embedded control vector C3 (such as a fixed-length embedded control vector C3) according to the selected transformation function (such as a vector concatenation transformation function or a learnable nonlinear neural network-like transformation function). The conversion function of the integration unit 32 (such as F(.)) can be selected as a vector concatenation conversion function as shown in the above formula (1), or a learnable nonlinear neural network conversion function as shown in the above formula (2).

When the integration unit 32 selects a learnable nonlinear neural network, the learnable nonlinear neural network can further integrate the attention unit 33 and the decoding unit 34 (such as a decoding network) of the speech synthesis module 30 to obtain an integrated neural network, and the integrated neural network can complete the training construction using the embedded control vector C3 formed by integrating the embedded speaker vector A2, the embedded emotion vector B2, and the embedded context vector C2.

Then, the decoding unit 34 of the speech synthesis module 30 can input the generated speech spectrum C4 (such as the Mel spectrum or the logarithmic Mel spectrum) to the vocoding module 40, so that the vocoding module 40 can use the speech spectrum C4 to synthesize the personalized (such as the speaker's characteristics) and emotional (such as the speaker's emotions) synthesized speech C5. That is, the vocoding module 40 can select one of the multiple types of neural networks such as WaveNet, WaveGlow, WaveGAN and HiFi-GAN to synthesize the speech spectrum C4 into the personalized (such as the speaker's characteristics) and emotional (such as the speaker's emotions) synthesized speech C5 according to different considerations such as the sound quality acceptability of the synthesized speech C5 and/or the system execution speed.

[5] The established multilingual and multi-emotion speech synthesis system 1 and method thereof can generate personalized and emotional synthesized speech C5 according to the text input by the user and the specified speaker tag 16 and emotion tag 26, and can also find the embedded speaker vector A2 corresponding to the speaker tag 16 from the first database 19 shown in FIG. 3, and find the embedded emotion vector B2 corresponding to the emotion tag 26 from the second database 29 shown in FIG. 4, and then input the embedded speaker vector A2 and the embedded emotion vector B2 into the speech synthesis module 30 respectively for speech synthesis processing.

In addition, the present invention also provides a computer-readable medium for multilinguals and multi-emotion speech synthesis methods, which is applied to a computing device or computer having a processor and/or a memory, and the computer-readable medium stores instructions, and the computing device or computer can execute the computer-readable medium through the processor and/or the memory to execute the above content when executing the computer-readable medium. For example, the processor can be a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc., and the memory can be a random access memory (RAM), a memory card, a hard disk (such as a cloud/network hard disk), a database, etc., but is not limited to this.

In summary, the multilingual and multi-emotion speech synthesis system, method and computer-readable medium of the present invention have at least the following features, advantages or technical effects.

1. The speaker encoding module of the present invention can automatically generate/obtain an embedded speaker vector based on the speech reference signals of multiple speakers to represent the voice characteristics of the speaker, and the emotion encoding module can automatically generate/obtain an embedded emotion vector based on the speech reference signals of multiple emotions to represent the type of emotion, and the encoding unit of the speech synthesis module can automatically generate/obtain an embedded context vector based on a text or phonetic symbol sequence.

2. The speech synthesis module or integration unit of the present invention can integrate the embedded speaker vector, the embedded emotion vector, and the embedded context vector into an embedded control vector, so as to use the embedded control vector to automatically control the speech synthesis module (such as the attention unit and the decoding unit) to synthesize a personalized and emotional speech spectrum.

3. The present invention can generate personalized and emotional speech spectrum and/or synthesized speech according to personalized needs, and can also make the synthesized speech show the voice characteristics of multiple (different) speakers according to different needs.

4. The present invention can build a speaker coding module, an emotion coding module, a speech synthesis module and a voice coding module, and can also use a neural network to improve the quality of synthesized speech, and can also continue to develop, pursue or demonstrate personalized and emotional (diversified) speech synthesis functions.

5. The present invention can use multilingual and multi-emotion speech synthesis training corpus to respectively construct a speaker encoding module and an emotion encoding module, so that the subsequent speech synthesis module and voice coding module can automatically synthesize personalized and emotional speech spectra and/or synthesized speech.

6. The speech synthesis module of the present invention can use end-to-end speech synthesis (such as Tacotron2) technology, encoding-decoding architecture (such as the architecture of encoding unit combined with decoding unit) and/or attention mechanism of attention unit, so that the speech synthesis module can automatically synthesize personalized and emotional speech spectrum.

7. The speech synthesis module of the present invention can perform text analysis on text or phonetic symbol sequences to automatically generate/obtain embedded context vectors, and then use the embedded context vectors and the weight mechanism of the attention unit to automatically align the input text or phonetic symbol sequence with the output speech spectrum.

8. The speech synthesis module or integration unit of the present invention can define corresponding transformation functions (such as vector concatenation transformation functions or learnable nonlinear neural network transformation functions) according to different selected methods, so as to automatically integrate the embedded speaker vector, embedded emotion vector, and embedded context vector into an embedded control vector according to the selected transformation function (such as vector concatenation transformation function or learnable nonlinear neural network transformation function).

9. The speaker encoding module of the present invention can use speaker-labeled training corpus to train a speaker identification neural network (embedded speaker vector extraction neural network) with a statistical pooling layer, so as to automatically convert the frame-level information of the speaker identification neural network into sentence-level information using the statistical pooling layer.

10. The emotion encoding module of the present invention can use the training corpus with emotion labels to train a sentence emotion recognition neural network (embedded emotion vector extraction neural network) with a statistical pooling layer, so as to automatically convert the frame-level information of the sentence emotion recognition neural network into sentence-level information using the statistical pooling layer.

11. The voice coding module of the present invention can utilize the speech spectrum (such as Mel spectrum or logarithmic Mel spectrum) synthesized by the attention unit and decoding unit of the speech synthesis module to automatically generate personalized and emotional synthesized speech.

The above implementation forms are only illustrative of the principles, features and effects of the present invention, and are not intended to limit the scope of implementation of the present invention. Anyone familiar with this technology can modify and change the above implementation forms without violating the spirit and scope of the present invention. Any equivalent changes and modifications completed using the content disclosed by the present invention should still be covered by the scope of the patent application. Therefore, the scope of protection of the present invention should be as listed in the scope of the patent application.

1: Multilingual and multi-emotion speech synthesis system

10: Speaker encoding module

20: Emotional coding module

30: Speech synthesis module

31: Coding unit

32: Integration unit

33: Attention unit

34: Decoding unit

40: Voice coding module

A1: Voice reference signal of multilingual speakers

A2: Embedded speaker vector

B1: Multi-emotional voice reference signals

B2: Embedded emotion vector

C1: text or phonetic symbol sequence

C2: Embedded context vector

C3: Embedded Control Vector

C4: Voice spectrum

C5: Synthetic Speech

Claims (19)

A multilingual and multi-emotion speech synthesis system includes: a speaker encoding module, which uses the Mel frequency inverse spectrum coefficient as the input feature of the neural network of embedded speaker vector extraction, so that the speaker encoding module generates a corresponding embedded speaker vector according to at least one of the input speech reference signals of the multilingual speakers; an emotion encoding module, which uses the Mel frequency inverse spectrum coefficient and can reflect the emotion The characteristics of the pitch and volume of the voice are used as input characteristics of a neural network for extracting an embedded emotion vector, so that the emotion coding module generates a corresponding embedded emotion vector according to at least one of the input multi-emotional speech reference signals; and a speech synthesis module having a coding unit, an integration unit and a decoding unit, wherein the coding unit of the speech synthesis module converts the input text or phonetic symbol into a corresponding embedded emotion vector. The text analysis is performed on the sequence to generate an embedded context vector corresponding to the text or phonetic symbol sequence, and the integration unit of the speech synthesis module integrates the embedded speaker vector generated by the speaker encoding module according to the speech reference signal of the multi-speaker, the embedded emotion vector generated by the emotion encoding module according to the multi-emotion speech reference signal, and the embedded context vector generated by the encoding unit according to the text or phonetic symbol sequence into an embedded control vector through a conversion function, and then the integration unit of the speech synthesis module uses the embedded speaker vector, the embedded emotion vector, and the embedded control vector integrated with the embedded context vector to control the decoding unit of the speech synthesis module to synthesize the speech spectrum using end-to-end speech synthesis technology, encoding-decoding architecture or attention mechanism. The speech synthesis system as described in claim 1, wherein the speaker encoding module further uses speaker-labeled training corpus to train a speaker identification neural network with a statistical pooling layer, and the speaker identification neural network is divided into a sound frame level and a sentence level, so that the statistical pooling layer of the speaker identification neural network converts the information of the sound frame level of the speaker identification neural network into the information of the sentence level. The speech synthesis system as described in claim 1, wherein the speaker encoding module further uses the Mel frequency cepstrum coefficient as the input feature of the speaker identification neural network, and defines the loss function of the speaker identification neural network based on a classifier, so that after the output layer of the speaker identification neural network is removed, the speaker encoding module uses the output of the statistical pooling layer of the speaker identification neural network as the embedded speaker vector of each input sentence, so that the speaker identification neural network becomes a neural network extracted by the embedded speaker vector. The speech synthesis system as described in claim 1, wherein the speaker encoding module further utilizes the embedded speaker vector extraction neural network to extract the embedded speaker vector of each input sentence, so that the speaker encoding module builds a database based on the speaker tag of each input sentence and the corresponding embedded speaker vector provided by the embedded speaker vector extraction neural network. The speech synthesis system as described in claim 1, wherein the emotion encoding module further uses the training corpus with emotion labels to train a sentence emotion recognition neural network with a statistical pooling layer, and the sentence emotion recognition neural network is divided into a sound frame level and a sentence level, so that the statistical pooling layer of the sentence emotion recognition neural network converts the information of the sound frame level of the sentence emotion recognition neural network into the information of the sentence level. The speech synthesis system as described in claim 1, wherein the emotion coding module further uses the Mel frequency cepstrum coefficient and the characteristics of pitch and volume that can reflect emotion changes as input features of the sentence emotion recognition neural network, and uses a classifier as the basis to define the loss function of the sentence emotion recognition neural network, so that after the output layer of the sentence emotion recognition neural network is removed, the emotion coding module uses the output of the statistical pooling layer of the sentence emotion recognition neural network as the embedded emotion vector of each input sentence, so that the sentence emotion recognition neural network becomes the embedded emotion vector extraction neural network. The speech synthesis system as described in claim 1, wherein the emotion coding module further utilizes the embedded emotion vector extraction neural network to extract the embedded emotion vector of each input sentence, so that the emotion coding module builds a database based on the emotion tag of each input sentence and the corresponding embedded emotion vector provided by the embedded emotion vector extraction neural network. The speech synthesis system as described in claim 1, wherein the speech synthesis module further has an attention unit, and the decoding unit of the speech synthesis module aligns the input text or phonetic symbol sequence with the speech spectrum output by the decoding unit according to the embedded context vector and the weight mechanism of the attention unit. The speech synthesis system as described in claim 1 further includes a vocoding module, and the speech synthesis module further includes an attention unit, wherein the vocoding module uses the attention mechanism provided by the attention unit of the speech synthesis module and the speech spectrum synthesized by the decoding unit to generate personalized and emotional synthesized speech, wherein the speech spectrum synthesized by the decoding unit is a Mel spectrum or a logarithmic Mel spectrum, and the vocoding module selects a neural network such as WaveNet, WaveGlow, WaveGAN or HiFi-GAN to synthesize the speech spectrum into the personalized and emotional synthesized speech according to different considerations of the sound quality acceptability of the synthesized speech or the system execution speed. A method for synthesizing multilingual and multi-emotion speech, comprising: a speaker coding module uses a Mel frequency inverse spectrum coefficient as an input feature of a neural network for extracting an embedded speaker vector, so that the speaker coding module generates a corresponding embedded speaker vector according to at least one of the input multilingual speech reference signals; an emotion coding module uses the Mel frequency inverse spectrum coefficient and the pitch and volume features that can reflect the change of emotions as input features of a neural network for extracting an embedded emotion vector, so that the emotion coding module generates a corresponding embedded emotion vector according to at least one of the input multi-emotion speech reference signals; a coding unit of the speech synthesis module performs text analysis on the input text or phonetic symbol sequence to generate the text or phonetic symbol sequence The corresponding embedded context vector; and the integration unit of the speech synthesis module integrates the embedded speaker vector generated by the speaker encoding module according to the speech reference signal of the multi-language speaker, the embedded emotion vector generated by the emotion encoding module according to the multi-emotion speech reference signal, and the embedded context vector generated by the encoding unit according to the text or phonetic symbol sequence into an embedded control vector through a conversion function, and then the integration unit of the speech synthesis module uses the embedded speaker vector, the embedded emotion vector, and the embedded control vector integrated with the embedded context vector to control the decoding unit of the speech synthesis module to synthesize the speech spectrum using end-to-end speech synthesis technology, encoding-decoding architecture or attention mechanism. The speech synthesis method as described in claim 10 further includes the speaker encoding module using speaker-labeled training corpus to train a speaker identification neural network with a statistical pooling layer, and the speaker identification neural network is divided into a sound frame level and a sentence level, so that the statistical pooling layer of the speaker identification neural network converts the information of the sound frame level of the speaker identification neural network into the information of the sentence level. The speech synthesis method as described in claim 10 further includes the speaker encoding module using the Mel frequency cepstrum coefficient as an input feature of the speaker identification neural network, and using a classifier as a basis to define the loss function of the speaker identification neural network, so that after the output layer of the speaker identification neural network is removed, the speaker encoding module uses the output of the statistical pooling layer of the speaker identification neural network as the embedded speaker vector of each input sentence, so that the speaker identification neural network becomes a neural network extracted by the embedded speaker vector. The speech synthesis method as described in claim 10 further includes the speaker encoding module extracting the embedded speaker vector of each input sentence using the embedded speaker vector extraction neural network, so that the speaker encoding module builds a database based on the speaker tag of each input sentence and the corresponding embedded speaker vector provided by the embedded speaker vector extraction neural network. The speech synthesis method as described in claim 10 further includes the emotion coding module using the training corpus with emotion labels to train a sentence emotion recognition neural network with a statistical pooling layer, and the sentence emotion recognition neural network is divided into a sound frame level and a sentence level, so that the statistical pooling layer of the sentence emotion recognition neural network converts the information of the sound frame level of the sentence emotion recognition neural network into the information of the sentence level. The speech synthesis method as described in claim 10 further includes the emotion coding module using the Mel frequency cepstrum coefficient and the characteristics of pitch and volume that can reflect emotion changes as input features of the sentence emotion recognition neural network, and using a classifier as the basis to define the loss function of the sentence emotion recognition neural network, so that after the output layer of the sentence emotion recognition neural network is removed, the emotion coding module uses the output of the statistical pooling layer of the sentence emotion recognition neural network as the embedded emotion vector of each input sentence, so that the sentence emotion recognition neural network becomes the embedded emotion vector extraction neural network. The speech synthesis method as described in claim 10 further includes the emotion coding module extracting the embedded emotion vector of each input sentence using the embedded emotion vector extraction neural network, so that the emotion coding module builds a database based on the emotion label of each input sentence and the corresponding embedded emotion vector provided by the embedded emotion vector extraction neural network. The speech synthesis method as described in claim 10 further includes the decoding unit of the speech synthesis module aligning the input text or phonetic symbol sequence with the speech spectrum output by the decoding unit according to the embedded context vector and the weight mechanism of the attention unit of the speech synthesis module. The speech synthesis method as described in claim 10 further includes the vocoding module using the attention mechanism provided by the attention unit of the speech synthesis module and the speech spectrum synthesized by the decoding unit to generate personalized and emotional synthesized speech, wherein the speech spectrum synthesized by the decoding unit is a Mel spectrum or a logarithmic Mel spectrum, and the vocoding module selects a neural network such as WaveNet, WaveGlow, WaveGAN or HiFi-GAN to synthesize the speech spectrum into the personalized and emotional synthesized speech based on different considerations of the sound quality acceptability of the synthesized speech or the system execution speed. A computer-readable medium, used in a computing device or a computer, stores instructions for executing a multilingual and multi-emotion speech synthesis method as described in any one of claims 10 to 18.

Images