CN102005205B - Emotional speech synthesizing method and device - Google Patents

Abstract

The invention provides an emotional speech synthesis method and device. According to one aspect of the present invention, a method for emotional speech synthesis is provided, comprising the steps of: inputting a text sentence; utilizing the neutral feature model obtained by training the neutral speech bank of the first speaker, predicting that the above text sentence will be used in the above-mentioned first speech The neutral feature vector in the first feature space of the person; using the speaker regularization model obtained from the training of the above-mentioned neutral speech database and the parallel speech database of the second speaker, the above-mentioned neutral feature vector is transformed into the above-mentioned second speaker’s second Regular neutral feature vectors in the feature space; using the emotional conversion model obtained from the above-mentioned parallel speech library training, the above-mentioned regular neutral feature vectors are converted into regular emotional feature vectors in the second feature space; using the above-mentioned speaker regularization model, the Inversely transforming the regularized emotional feature vectors into emotional feature vectors in the first feature space; and synthesizing the emotional voice of the first speaker by using the emotional feature vectors in the first feature space.

Description

Emotional speech synthesis method and device

technical field

The present invention relates to information processing technology, in particular to speech synthesis technology, more specifically to emotion speech synthesis technology independent of speaker.

Background technique

At present, most speech synthesis systems based on large speech databases are based on speech in a neutral reading mode. For the synthesis of emotional speech, the common method is to convert the neutral speech into the prosody and spectrum conversion method of the target emotional speech, such as the method based on GMM (Gaussian mixture model, Gaussian mixture model) recorded in non-patent literature 1 and 2 and The method based on CART (Classification And Regression Tree, classification and regression tree) recorded in non-patent literature 2. These prosodic and spectral transformation methods only require building an additional small parallel corpus, which saves a lot of development time and expense compared to re-recording a large corpus of the target emotional speech. At the same time, these prosodic and spectral transformation methods can establish the connection between neutral speech features and target emotional speech features, such as GMM-based methods. Optionally, it is also possible to establish the link between linguistic information and the difference between neutral speech features and target emotional speech features, such as CART-based methods. The GMM-based method has better performance than the CART-based method. In addition, as described in Non-Patent Document 2, the CART method and the GMM method can also be combined, that is, first use the CART method to perform a preliminary classification based on linguistic information, and then use the GMM method to establish prosody for each category and spectral transformation models.

However, the above GMM-based prosodic and spectral transformation models are heavily speaker-dependent. That is, if the aforementioned large neutral corpus and the aforementioned small parallel corpus were not from the same speakers, the performance of the conversion would be severely degraded. Therefore, in the above-mentioned prosody and spectrum conversion method, in order to obtain a high-quality conversion effect, it is hoped that the above-mentioned large-scale neutral speech library and the above-mentioned small parallel speech library are from the same speaker. However, this is difficult to achieve in actual product support, because a customer's needs may arise at any time, for example, after several years of recording a neutral voice library, even if the speakers of the year can still be found, his/her The sound may also have changed considerably over time.

Non-Patent Document 1: L. Mesbahi, V.Barreaud and O.Boeffard, "Comparing GMM-based speech transformation systems", Proc.of INTERSPEECH 2007, Antwerp, Belgium, Aug.27-31, 2007, pp.1989-1992, The entire contents thereof are hereby incorporated by reference.

Non-Patent Document 2: J.Tao, Y.Kang and A.Li, "Prosodyconversion from neutral speech to emotional speech", IEEE Trans.OnAudio, Speech and Language Processing, Vol.14, No.4, 2006, pp.1145 -1154, the entire contents of which are hereby incorporated by reference.

Contents of the invention

The present invention is proposed in view of the above-mentioned problems in the prior art, and its purpose is to provide a speaker-independent emotional speech synthesis method and device, so as to effectively improve the performance of prosody and spectrum conversion.

According to one aspect of the present invention, a method for emotional speech synthesis is provided, comprising the steps of: inputting a text sentence; utilizing the neutral feature model obtained by training the neutral speech bank of the first speaker, predicting that the above text sentence will be used in the above-mentioned first speech The neutral feature vector in the first feature space of the person; using the speaker regularization model obtained from the training of the above-mentioned neutral speech database and the parallel speech database of the second speaker, the above-mentioned neutral feature vector is transformed into the above-mentioned second speaker’s second Regular neutral feature vectors in the feature space; using the emotional conversion model obtained from the above-mentioned parallel speech library training, the above-mentioned regular neutral feature vectors are converted into regular emotional feature vectors in the second feature space; using the above-mentioned speaker regularization model, the Inversely transforming the regularized emotional feature vectors into emotional feature vectors in the first feature space; and synthesizing the emotional voice of the first speaker by using the emotional feature vectors in the first feature space.

According to another aspect of the present invention, an emotional speech synthesis device is provided, including: an input unit, which inputs a text sentence; a prediction unit, which utilizes a neutral feature model obtained by training the neutral speech database of the first speaker to predict the above-mentioned The neutral feature vector of the text sentence in the first feature space of the above-mentioned first speaker; the transformation unit, which utilizes the speaker regularization model obtained by training the above-mentioned neutral speech database and the parallel speech database of the second speaker, converts the above-mentioned neutral features The vector is transformed into a regular neutral feature vector in the second feature space of the above-mentioned second speaker; a conversion unit, which uses the emotion conversion model obtained from the above-mentioned parallel speech database training, converts the above-mentioned regular neutral feature vector into the above-mentioned second feature space The regular emotional feature vector in the above; the inverse transformation unit, which uses the above-mentioned speaker regularization model, inversely transforms the above-mentioned regular emotional feature vector into the emotional feature vector in the above-mentioned first feature space; and a synthesis unit, which uses the above-mentioned first feature space The emotional feature vector in synthesizes the emotional speech of the first speaker.

Description of drawings

It is believed that people can better understand the above-mentioned characteristics, advantages and objectives of the present invention through the following description of specific embodiments of the present invention in conjunction with the accompanying drawings.

Fig. 1 is a flowchart of an emotional speech synthesis method according to an embodiment of the present invention.

Fig. 2 is an example of a speaker regularization model according to an embodiment of the present invention.

Fig. 3 is another example of a speaker normalization model according to an embodiment of the present invention.

Fig. 4 is a block diagram of an emotional speech synthesis device according to another embodiment of the present invention.

Detailed ways

Various preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Emotional Speech Synthesis Method

Fig. 1 is a flowchart of an emotional speech synthesis method according to an embodiment of the present invention. The present embodiment will be described below with reference to this figure.

As shown in FIG. 1 , first, at step 101, a text sentence is input. In this embodiment, the input text sentence may be any text sentence known to those skilled in the art, and may also be a text sentence in various languages, such as Chinese, English, Japanese, etc., and the present invention has no limitation on this.

Next, at step 105, linguistic information 60 is extracted from the input text sentence using text analysis. In the present embodiment, the linguistic information 60 includes the sentence length of the above-mentioned text sentence, the shape, pinyin, phoneme type, tone, part of speech, position in the sentence, and the distance between the characters (words) in the sentence. Boundary types and distances from front and back pauses, etc. In addition, in this embodiment, the text analysis method used to extract the linguistic information 60 from the input text sentence may be any method known to those skilled in the art, and the present invention has no limitation on this.

It should be noted that step 105 here is just an optional step, and it is also possible to directly proceed to step 110 after the text sentence is input in step 101 .

In step 110, use the neutral feature model 30 obtained from the training of the neutral speech library 10 of the first speaker to predict the neutral feature vector of the text sentence input in step 101 in the first feature space of the first speaker.

In this embodiment, the neutral speech database 10 includes the neutral speech of the first speaker, that is, the speech of neutral reading aloud. The neutral speech database 10 may be any speech database known to those skilled in the art, such as the neutral speech databases described in the above-mentioned non-patent documents 1 and 2. In addition, the method of training the neutral feature model 30 from the neutral speech library 10 may also be any method known to those skilled in the art, such as the training methods described in the above-mentioned non-patent documents 1 and 2. In addition, the neutral feature model 30 obtained through training may also be any model known to those skilled in the art, such as the neutral feature models described in the above-mentioned Non-Patent Documents 1 and 2. The feature vector in the neutral feature model 30 may include one or more of prosodic features (such as duration, fundamental frequency trajectory, pause, energy, etc.) and spectral features. The present invention only utilizes the neutral feature model 30 in step 110 , and has no limitations on the neutral speech library 10 , the training method of the neutral feature model 30 and the neutral feature model 30 .

In step 110, if the linguistic information 60 is not extracted in step 105, then use the neutral feature model 30 to predict the neutral feature vector of the text sentence input in step 101 in the first feature space of the first speaker. If the linguistic information 60 is extracted in step 105 , then the above-mentioned neutral feature vector is predicted by using the neutral feature model 30 according to the extracted linguistic information 60 . In this embodiment, the method for predicting the neutral eigenvector may be any method known to those skilled in the art, such as the prediction methods described in the above-mentioned Non-Patent Documents 1 and 2. In addition, the predicted neutral feature vector may contain one or more of prosodic features (such as duration, fundamental frequency trajectory, pause, energy, etc.) and spectral features.

Next, in step 115, the neutral feature vector predicted in step 110 is transformed into the second speaker's second Regular neutral eigenvectors in feature space. Here, the transformed regular neutral feature vector may also include one or more of prosodic features (such as duration, fundamental frequency trajectory, pause, energy, etc.) and spectral features.

In this embodiment, the second speaker's parallel speech bank 20 contains the second speaker's neutral speech and target emotional speech, which are paired, that is, the same text sentence is read aloud in both neutral and target emotional ways .

An example of speaker normalization model 50 and the transformation performed in step 115 is described in detail below with reference to FIG. 2 .

FIG. 2 is an example of a speaker normalization model 50 according to an embodiment of the present invention. As shown in FIG. 2 , in the process of training the speaker regularization model 50 , the neutral speech library 10 is first divided into m classes 1-1, 1-2, . . . , 1-m according to the classification rule 70 . The classification rules 70 can be different from feature to feature based on experience, for example, one or more of phoneme types for duration and spectrum features, tone types for fundamental frequency traces, and position in a sentence for energy. Next, according to the same classification rule 70, the parallel speech library 20 is also divided into corresponding m classes 2-1, 2-2, . . . , 2-m. Next, for each class 1-i and 2-i, calculate statistics 71-i and 72-i, wherein the above statistics can be the mean value of the feature vectors extracted from each class 1-i and 2-i μ and the covariance matrix Σ, etc. In this case, speaker normalization model 50 includes classification rules 70 and statistics 71-i and 72-i.

Returning to Fig. 1, in the case that the speaker regularization model 50 includes classification rules 70 and statistics 71-i and 72-i, in step 115, at first, the linguistic information 60 extracted in step 105 is used to find The class x corresponding to the obtained neutral feature vector, and then transform the neutral feature vector into a regular neutral feature vector in the second feature space of the second speaker according to the following formula (1), ${v_{\mathrm{no}}}^{\′}=(v_{\mathrm{no}}-{\mathrm{\μ}}_{1x}){\mathrm{\Σ}}_{1x}^{-1/2}{\mathrm{\Σ}}_{2x}^{1/2}+{\mathrm{\μ}}_{2x}---\left(1\right)$ Wherein, v' _n represents the above-mentioned regular neutral feature vector, v _n represents the above-mentioned neutral feature vector, μ _1x represents the mean value extracted from the xth class corresponding to the above-mentioned neutral feature vector of the above-mentioned neutral voice bank 10, and ∑ _1x represents the mean value extracted from the above-mentioned neutral feature vector The covariance matrix extracted in the xth class corresponding to the above-mentioned neutral eigenvector of the neutral speech bank 10, μ _2x represents the mean value extracted from the xth class corresponding to the above-mentioned neutral eigenvector of the above-mentioned parallel speech bank 20, and ∑ _2x represents the covariance matrix extracted from the xth class corresponding to the above-mentioned neutral feature vector of the above-mentioned parallel speech library 20 .

Another example of the speaker normalization model 50 and the transformation performed in step 115 is described in detail below with reference to FIG. 3 .

Fig. 3 is another example of a speaker normalization model according to an embodiment of the present invention. As shown in FIG. 3 , in the process of training the speaker regularization model 50 , the first feature space model (λ ₁ , μ ₁ , Σ ₁ ) 80 of the first speaker based on GMM is trained using the neutral speech library 10 . In the first feature space model 80, _λ1 represents the set of weights of each component in the first feature space model 80, _μ1 represents the set of the mean value of each component, and _Σ1 represents the covariance matrix of each component collection. Then, the parallel speech library 20 is used to adapt the first feature space model 80 of the first speaker to the second feature space model (λ ₂ , μ ₂ , Σ ₂ ) 90 of the second speaker. In the second feature space model 90, λ ₂ represents the set of weights occupied by each component in the second feature space model 90, μ ₂ represents the set of the mean value of each component, and ∑ ₂ represents the covariance matrix of each component collection. The number of components in the above feature space model should be large enough to enable the model to accurately describe the above feature space. In addition, it can be considered that the components corresponding to the two feature space models 80 and 90 before and after adaptation are coupled. In this embodiment, the adaptive method of the feature space model can be MAP (Maximum a Posteriori, maximum posterior probability), MCE (Minimum Classification Error, minimum classification error), MMI (Maximum Mutual Information, maximum mutual correlation information) or other The available algorithms are not limited by the present invention. It should be noted that because the second speaker's parallel speech library 20 has limited data, usually only the mean value μ of the model is adapted, so it is assumed that λ ₁ =λ ₂ =λ, ∑ ₁ =∑ ₂ =∑. In this embodiment, the method for training the GMM-based feature space model may be any method known to those skilled in the art, such as the training methods described in the above-mentioned non-patent documents 1 and 2. In this case, the speaker normalization model 50 includes a first feature space model 80 and a second feature space model 90 .

Returning to Fig. 1, in the case that the speaker regularization model 50 includes the first feature space model 80 and the second feature space model 90, in step 115, firstly, the neutral feature vector predicted in step 110 is calculated for the first feature space model ( λ ₁ , μ ₁ , Σ ₁ ) the probability P _i of each component i of 80. Optionally, the above probability P _i can be calculated according to the following formula (2): $p_i={\mathrm{\λ}}_i\&Center\; Dot;\frac{1}{{\left(2\mathrm{\π}\right)}^{\mathrm{no}/2}{\left|{\mathrm{\Σ}}_i\right|}^{1/2}}\exp (-\frac{1}{2}{(v_{\mathrm{no}}-{\mathrm{\μ}}_{1i})}^T{\mathrm{\Σ}}_i^{-1}(v_{\mathrm{no}}-{\mathrm{\μ}}_{1i}))---\left(2\right)$ Wherein, _λi represents the weight of each component i in the first feature space model 80, μ _1i represents the mean value of each component i in the first feature space model 80, and _Σi represents the weight of each component i in the first feature space model 80. The covariance matrix of each component i of , v _n represents the neutral eigenvector.

Next, the weight w _i of the above-mentioned probability P _i for each component i of the first feature space model (λ ₁ , μ ₁ , Σ ₁ ) 80 of the above-mentioned neutral feature vector is calculated. Optionally, the above weight w _i can be calculated according to the following formula (3): $w_i=\frac{p_i}{\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\Σ}}}{p}_i}---\left(3\right)$

Next, according to the above weight w _i and the second feature space model (λ ₂ , μ ₂ , Σ ₂ ) 90 , calculate the regularized neutral feature vector v′ _n transformed in step 115 . Optionally, the above regular neutral eigenvector v' _n can be calculated according to the following formula (4), ${v_{\mathrm{no}}}^{\′}=\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\Σ}}}(w_i\&Center\; Dot;{\mathrm{\μ}}_{2i})---\left(4\right)$ Wherein, μ _2i represents the average value of each component i in the second feature space model 90.

In step 115, the speaker normalization model 50 is used to transform the neutral feature vector v _n predicted in step 110 into a regular neutral feature vector v′ _n in the second feature space of the second speaker. The emotion conversion model 40 obtained through the training of the parallel speech library 20 converts the regularized neutral feature vector v' _n into the regularized emotional feature vector v' _e in the above-mentioned second feature space. Here, the converted normalized emotional feature vector v' _e may contain one or more of prosodic features (such as duration, fundamental frequency trajectory, pause, energy, etc.) and spectral features.

In this embodiment, the emotion conversion model 40 can be trained by using a GMM-based method, a CART-based method or other methods and various possible combinations thereof. Similar to the features contained in the above feature vectors, the emotion conversion model 40 may also include one or more of a duration conversion model, a fundamental frequency trajectory conversion model, a pause conversion model, an energy conversion model, and a spectrum conversion model.

In addition, in this embodiment, if the emotion conversion model 40 is trained using a CART-based method, it is necessary to extract the linguistic information 60 from the input text sentence in the above step 105, so that in step 120, according to the linguistic Information 60, using the emotion conversion model 40 to transform the regularized neutral feature vector v′ _n into a regularized emotional feature vector v′ _e .

Next, in step 125, the regularized emotional feature vector v' _e is inversely transformed into the emotional feature vector v _e in the above-mentioned first feature space by using the speaker normalized model 50 . Here, the inversely transformed emotional feature vector _ve may contain one or more of prosodic features (such as duration, pitch track, pause, energy, etc.) and spectral features.

Optionally, in the case that the speaker regularization model 50 includes classification rules 70 and statistics 71-i and ₇₂ -i as shown in FIG. is the emotional feature vector v _e , $v_e=({v_e}^{\′}-{\mathrm{\μ}}_{2x}){\mathrm{\Σ}}_{2x}^{-1/2}{\mathrm{\Σ}}_{1x}^{1/2}+{\mathrm{\μ}}_{1x}---\left(5\right)$ Among them, μ _2x represents the mean value extracted from the xth class corresponding to the neutral feature vector v _n of the parallel speech library 20, and ∑ _2x represents the extraction from the xth class corresponding to the neutral feature vector v _n of the parallel speech library 20 The covariance matrix of .

In addition, optionally, in the case that the speaker regularization model 50 includes the first feature space model 80 and the second feature space model ₉₀ as shown in FIG. The probability P′ _i of each component of the model (λ ₂ , μ ₂ , Σ ₂ ) 90 . Optionally, the above probability P′ _i can be calculated according to the following formula (6): ${p_i}^{\′}={\mathrm{\λ}}_i\&Center\; Dot;\frac{1}{{\left(2\mathrm{\π}\right)}^{\mathrm{no}/2}{\left|{\mathrm{\Σ}}_i\right|}^{1/2}}\exp (-\frac{1}{2}{({v_e}^{\′}-{\mathrm{\μ}}_{2i})}^T{\mathrm{\Σ}}_i^{-1}({v_e}^{\′}-{\mathrm{\μ}}_{2i}))---\left(6\right)$ Wherein, _λi represents the weight of each component i in the second feature space model 90, μ _2i represents the mean value of each component i in the second feature space model 90, and _Σi represents the weight of each component i in the second feature space model 90. The covariance matrix of each component i of , v′ _e represents the regular emotional feature vector.

Next, the _weight w′ i of the probability P′ _i of the regularized emotional feature vector v′ _e for each component i of the second feature space model 90 is calculated. Optionally, the above weight w′ _i can be calculated according to the following formula (7): ${w_i}^{\′}=\frac{{p_i}^{\′}}{\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\Σ}}}{p_i}^{\′}}---\left(7\right)$

Next, according to the weight w′ _i and the first feature space model (λ ₁ , μ ₁ , Σ ₁ ) 80 , calculate the emotion feature vector _ve after inverse transformation in step 125 . Optionally, the above emotional feature vector _ve can be calculated according to the following formula (8), $v_e=\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\Σ}}}({w_i}^{\′}\&Center\; Dot;{\mathrm{\μ}}_{1i})---\left(8\right)$ Wherein, μ _1i represents the mean value of each component i in the first feature space model 80.

Finally, in step 130, the emotional speech of the first speaker is synthesized by using the emotional feature vector _ve in the first feature space.

In this embodiment, the method of synthesizing the emotional feature vector into target emotional speech can be any method known to those skilled in the art, such as the synthesis method described in the above-mentioned non-patent literature 1 and 2, and the present invention has no limitation to this .

Through the emotional speech synthesis method of this embodiment, the emotional speech of the first speaker can be synthesized by using the parallel speech library of the second speaker different from the first speaker, so that the large-scale neutral speech library and the small parallel speech library are not from In the case of the same speaker, it can effectively improve the performance of prosody and spectral conversion.

Emotional Speech Synthesis Device

Under the same inventive conception, FIG. 4 is a block diagram of an emotional speech synthesis device according to another embodiment of the present invention. The present embodiment will be described below with reference to this figure. For those parts that are the same as those in the previous embodiments, descriptions thereof are appropriately omitted.

As shown in Figure 4, the emotional speech synthesis device 400 of the present embodiment includes: an input unit 401, which inputs a text sentence; a prediction unit 410, which utilizes the neutral feature model 30 obtained by training the neutral voice library 10 of the first speaker, Predict the neutral feature vector of the above-mentioned text sentence in the first feature space of the above-mentioned first speaker; the transformation unit 415, which utilizes the speaker regularization model obtained by training the above-mentioned neutral speech library 10 and the second speaker's parallel speech library 20 50. Transform the above-mentioned neutral feature vector into a regular neutral feature vector in the second feature space of the second speaker; a conversion unit 420, which utilizes the emotion conversion model 40 obtained by training the above-mentioned parallel speech library 20 to convert the above-mentioned regular neutral feature vector to The eigenvectors are converted into regular emotional feature vectors in the above-mentioned second feature space; the inverse transformation unit 425 uses the above-mentioned speaker regularization model 50 to inverse transform the above-mentioned regular emotional feature vectors into emotional feature vectors in the above-mentioned first feature space; And a synthesis unit 430, which uses the emotional feature vector in the first feature space to synthesize the emotional voice of the first speaker

In this embodiment, the text sentences input by the input unit 401 can be any text sentences known to those skilled in the art, and can also be text sentences in various languages, such as Chinese, English, Japanese, etc., and the present invention does not have this any restrictions.

The emotional speech synthesis device 400 of this embodiment optionally has an extraction unit 405 that uses text analysis to extract linguistic information 60 from the text sentence input by the input unit 401 . In the present embodiment, the linguistic information 60 includes the sentence length of the above-mentioned text sentence, the shape, pinyin, phoneme type, tone, part of speech, position in the sentence, and the distance between the characters (words) in the sentence. Boundary types and distances from front and back pauses, etc. In addition, in this embodiment, the text analysis method used by the extraction unit 405 to extract the linguistic information 60 from the input text sentence may be any method known to those skilled in the art, and the present invention has no limitation on this.

In this embodiment, the neutral speech database 10 includes the neutral speech of the first speaker, that is, the speech of neutral reading aloud. The neutral speech database 10 may be any speech database known to those skilled in the art, such as the neutral speech databases described in the above-mentioned non-patent documents 1 and 2. In addition, the method of training the neutral feature model 30 from the neutral speech library 10 may also be any method known to those skilled in the art, such as the training methods described in the above-mentioned non-patent documents 1 and 2. In addition, the neutral feature model 30 obtained through training may also be any model known to those skilled in the art, such as the neutral feature models described in the above-mentioned Non-Patent Documents 1 and 2. The feature vector in the neutral feature model 30 may include one or more of prosodic features (such as duration, fundamental frequency trajectory, pause, energy, etc.) and spectral features. The present invention only utilizes the neutral feature model 30 in the prediction unit 410 , and has no limitation on the neutral speech library 10 , the training method of the neutral feature model 30 and the neutral feature model 30 .

The prediction unit 410 uses the neutral feature model 30 to predict the neutral feature vector of the text sentence input by the input unit 401 in the first feature space of the first speaker without using the extraction unit 405 to extract the linguistic information 60 . When the linguistic information 60 is extracted by the extraction unit 405 , the above-mentioned neutral feature vector is predicted by using the neutral feature model 30 according to the extracted linguistic information 60 . In this embodiment, the method for predicting the neutral feature vector by the prediction unit 410 may be any method known to those skilled in the art, such as the prediction methods described in the above-mentioned Non-Patent Documents 1 and 2. In addition, the predicted neutral feature vector may contain one or more of prosodic features (such as duration, fundamental frequency trajectory, pause, energy, etc.) and spectral features.

The transformation unit 415 uses the speaker regularization model 50 trained by the neutral speech library 10 and the parallel speech library 20 of the second speaker to transform the neutral feature vector predicted by the prediction unit 410 into the second feature space of the second speaker Regular neutral eigenvectors in . Here, the transformed regular neutral feature vector may also include one or more of prosodic features (such as duration, fundamental frequency trajectory, pause, energy, etc.) and spectral features.

In this embodiment, the second speaker's parallel speech bank 20 contains the second speaker's neutral speech and target emotional speech, which are paired, that is, the same text sentence is read aloud in both neutral and target emotional ways .

In this embodiment, the speaker regularization model 50 may be the above-mentioned speaker regularization model 50 including classification rules 70 and statistics 71-i and 72-i as shown in FIG. The above-mentioned speaker normalization model 50 including the first feature space model 80 and the second feature space model 90 .

In the case that the speaker normalization model 50 includes classification rules 70 and statistics 71-i and 72-i, the transformation unit 415 includes a search unit and a calculation unit. The search unit uses the linguistic information 60 extracted by the extraction unit 405 to search for the class x corresponding to the neutral feature vector predicted by the prediction unit 410 . The calculation unit calculates the regular neutral eigenvector according to the above formula (1).

In the case that the speaker normalization model 50 includes the first feature space model 80 and the second feature space model 90 , the transformation unit 415 includes a probability calculation unit, a weight calculation unit, and a feature vector calculation unit. The probability calculation unit calculates the probability P _i of the neutral feature vector predicted by the prediction unit 410 for each component i of the first feature space model (λ ₁ , μ ₁ , Σ ₁ ) 80 . Optionally, the above probability P _i may be calculated according to the above formula (2).

The weight calculation unit calculates the weight w _i of the probability P _i of each component i of the first feature space model (λ ₁ , μ ₁ , Σ ₁ ) 80 of the neutral feature vector. Optionally, the above weight w _i may be calculated according to the above formula (3).

The feature vector calculation unit calculates the transformed regular neutral feature vector v′ _n according to the weight w _i calculated by the above weight calculation unit and the second feature space model (λ ₂ , μ ₂ , Σ ₂ ) 90 . Optionally, the regularized neutral eigenvector v' _n may be calculated according to the above formula (4).

After the transformation unit 415 uses the speaker normalization model 50 to transform the neutral feature vector _vn predicted by the prediction unit 410 into a regularized neutral feature vector v′ _n in the second feature space of the second speaker, the transformation unit 420 uses the The emotion conversion model 40 obtained through the training of the parallel speech library 20 converts the regularized neutral feature vector v' _n into the regularized emotional feature vector v' _e in the above-mentioned second feature space. Here, the converted normalized emotional feature vector v' _e may contain one or more of prosodic features (such as duration, fundamental frequency trajectory, pause, energy, etc.) and spectral features.

In this embodiment, the emotion conversion model 40 can be trained by using a GMM-based method, a CART-based method or other methods and various possible combinations thereof. Similar to the features contained in the above feature vectors, the emotion conversion model 40 may also include one or more of a duration conversion model, a fundamental frequency trajectory conversion model, a pause conversion model, an energy conversion model, and a spectrum conversion model.

In addition, in this embodiment, if the emotion conversion model 40 is trained using a CART-based method, the extraction unit 405 needs to be used to extract the linguistic information 60 from the input text sentence, so that the conversion unit 420 can use the linguistic information 60 , using the emotion conversion model 40 to transform the regularized neutral feature vector v′ _n into the regularized emotional feature vector v′ _e .

The inverse transformation unit 425 utilizes the speaker regularization model 50 to inverse transform the regularized emotional feature vector v' _e into the emotional feature vector v _e in the above-mentioned first feature space. Here, the inversely transformed emotional feature vector _ve may contain one or more of prosodic features (such as duration, pitch track, pause, energy, etc.) and spectral features.

Optionally, in the case that the speaker regularization model 50 includes classification rules 70 and statistics 71-i and ₇₂ -i as shown in FIG. is the emotional feature vector v _e .

In addition, optionally, when the speaker regularization model 50 includes the first feature space model 80 and the second feature space model 90 as shown in FIG. 3 , the inverse transformation unit 425 includes a probability calculation unit, a weight calculation unit and a feature Vector computing unit. The probability calculation unit calculates the probability P' _i of the regularized emotional feature vector v' _e for each component of the second feature space model (λ ₂ , μ ₂ , Σ ₂ ) 90 . Optionally, the probability calculation unit may calculate the above probability P' _i according to the above formula (6).

The weight calculation unit calculates the weight w' _i of the above-mentioned probability P' _i of each component i of the second feature space model 90 of the regularized emotional feature vector v' _e . Optionally, the weight calculation unit may calculate the above weight w' _i according to the above formula (7).

The feature vector calculation unit calculates the emotional feature vector v _e after inverse transformation according to the weight w′ _i calculated by the above weight calculation unit and the first feature space model (λ ₁ , μ ₁ , Σ ₁ ) 80 . Optionally, the feature vector calculation unit may calculate the aforementioned emotional feature vector _ve according to the aforementioned formula (8).

Finally, the synthesizing unit 430 synthesizes the emotional voice of the first speaker by using the emotional feature vector _ve in the first feature space.

In this embodiment, the method of synthesizing the emotional feature vector into target emotional speech can be any method known to those skilled in the art, such as the synthesis method described in the above-mentioned non-patent literature 1 and 2, and the present invention has no limitation to this .

Through the emotional speech synthesis device 400 of this embodiment, the emotional speech of the first speaker can be synthesized by using the parallel speech library of the second speaker different from the first speaker, so that the large-scale neutral speech library and the small parallel speech library are not From the same speaker case, it can effectively improve the performance of prosody and spectral conversion.

Although the emotional speech synthesis method and the emotional speech synthesis device of the present invention have been described in detail through some exemplary embodiments above, these above embodiments are not exhaustive, and those skilled in the art can understand the spirit and scope of the present invention Variations and modifications are implemented within. Therefore, the present invention is not limited to these embodiments, and the scope of the present invention is determined only by the appended claims.

That is to say, the idea of the present invention is to use the speaker regularization model to perform prosodic and spectral conversion independently of the speaker in the case of only an additional parallel speech library of another speaker, thereby effectively improving the performance of the conversion . The speaker regularization model described above is easy to combine with various existing prosody and spectrum conversion methods and is not limited to the methods described in the above embodiments. The application purpose of the present invention can also be not limited to emotional expression, but can be used for enriching the various expression types in TTS (Text-to-Speech, text-to-speech conversion, or be called speech synthesis) more widely, such as friendly speaking manner , semantic focus in dialogue, etc. The present invention is applicable to both the TTS system of unit splicing and the TTS system of parameter synthesis.

Claims (9)

1. emotional speech synthetic method may further comprise the steps:

The input text sentence;

Utilization is predicted the neutral proper vector of above-mentioned text sentence in above-mentioned first speaker's first feature space by the neutral characteristic model of first speaker's neutral sound bank training acquisition;

Utilization is by the regular model of speaker that the above-mentioned neutral sound bank and second speaker's parallel sound bank training obtains, and above-mentioned neutral proper vector is transformed to the regular neutral proper vector in above-mentioned second speaker's second feature space;

The emotion transformation model that utilization is obtained by above-mentioned parallel sound bank training converts above-mentioned regular neutral proper vector in above-mentioned second feature space regular affective characteristics vector;

Utilize the regular model of above-mentioned speaker, above-mentioned regular affective characteristics vector is inversely transformed into the affective characteristics vector in above-mentioned first feature space; And

Utilize affective characteristics vector in above-mentioned first feature space to synthesize first speaker's emotional speech.

2. emotional speech synthetic method according to claim 1, further comprising the steps of:

At the neutral characteristic model of above-mentioned utilization, predict and from above-mentioned text sentence, extract linguistic information before the step of the neutral proper vector of above-mentioned text sentence in above-mentioned first speaker's first feature space by first speaker's neutral sound bank training acquisition.

3. emotional speech synthetic method according to claim 2; Wherein, Above-mentioned utilization predicts that by the neutral characteristic model of first speaker's neutral sound bank training acquisition the step of the neutral proper vector of above-mentioned text sentence in above-mentioned first speaker's first feature space may further comprise the steps:

Based on above-mentioned linguistic information, utilize above-mentioned neutral characteristic model, predict above-mentioned neutral characteristic vector.

4. emotional speech synthetic method according to claim 2; Wherein, The emotion transformation model that above-mentioned utilization is obtained by above-mentioned parallel sound bank training, the step that above-mentioned regular neutral proper vector is converted into the regular affective characteristics vector in above-mentioned second feature space may further comprise the steps:

According to above-mentioned linguistic information, utilize above-mentioned emotion transformation model, convert above-mentioned regular neutral proper vector into above-mentioned regular affective characteristics vector.

5. emotional speech synthetic method according to claim 1; Wherein, the regular model of above-mentioned speaker comprise classifying rules, the average and the covariance matrix of the average of the proper vector from each type of dividing according to above-mentioned classifying rules of above-mentioned neutral sound bank, extracted and covariance matrix and the proper vector from each type of dividing according to above-mentioned classifying rules of above-mentioned parallel sound bank, extracted.

6. emotional speech synthetic method according to claim 5; Wherein, above-mentioned classifying rules comprise phoneme classification of type rule to duration and spectrum signature, to the tone classification of type rule of fundamental frequency track and to energy at least a in the classifying rules of position.

7. emotional speech synthetic method according to claim 5; Wherein, Above-mentioned utilization is by the regular model of speaker that the above-mentioned neutral sound bank and second speaker's parallel sound bank training obtains, and the step that above-mentioned neutral proper vector is transformed to the regular neutral proper vector in above-mentioned second speaker's second feature space may further comprise the steps:

According to following formula above-mentioned neutral proper vector is transformed to above-mentioned regular neutral proper vector,

Wherein, V _n ^/Represent above-mentioned regular neutral proper vector, v _nRepresent above-mentioned neutral proper vector, μ _1xThe average that representative is extracted from the x class corresponding with above-mentioned neutral proper vector of above-mentioned neutral sound bank, ∑ _1xThe covariance matrix that representative is extracted from the x class corresponding with above-mentioned neutral proper vector of above-mentioned neutral sound bank, μ _2xThe average that representative is extracted from the x class corresponding with above-mentioned neutral proper vector of above-mentioned parallel sound bank, and ∑ _2xThe covariance matrix that representative is extracted from the x class corresponding with above-mentioned neutral proper vector of above-mentioned parallel sound bank.

8. emotional speech synthetic method according to claim 5, wherein, the above-mentioned regular model of above-mentioned speaker that utilizes, the step that above-mentioned regular affective characteristics vector is inversely transformed into the affective characteristics vector in above-mentioned first feature space may further comprise the steps:

According to following formula above-mentioned regular affective characteristics vector is inversely transformed into above-mentioned affective characteristics vector,

Wherein, v _eRepresent above-mentioned affective characteristics vector, V _e ^/Represent above-mentioned regular affective characteristics vector, μ _1xThe average that representative is extracted from the x class corresponding with above-mentioned neutral proper vector of above-mentioned neutral sound bank, ∑ _1xThe covariance matrix that representative is extracted from the x class corresponding with above-mentioned neutral proper vector of above-mentioned neutral sound bank, μ _2xThe average that representative is extracted from the x class corresponding with above-mentioned neutral proper vector of above-mentioned parallel sound bank, and ∑ _2xThe covariance matrix that representative is extracted from the x class corresponding with above-mentioned neutral proper vector of above-mentioned parallel sound bank.

9. emotional speech synthesizer comprises:

Input block, its input text sentence;

Predicting unit, it utilizes the neutral characteristic model by first speaker's neutral sound bank training acquisition, predicts the neutral proper vector of above-mentioned text sentence in above-mentioned first speaker's first feature space;

Converter unit, the regular model of speaker that it utilizes the parallel sound bank training by the above-mentioned neutral sound bank and second speaker to obtain is transformed to above-mentioned neutral proper vector the regular neutral proper vector in above-mentioned second speaker's second feature space;

Converting unit, it utilizes the emotion transformation model that is obtained by above-mentioned parallel sound bank training, converts above-mentioned regular neutral proper vector in above-mentioned second feature space regular affective characteristics vector;

Inverse transformation block, it utilizes the regular model of above-mentioned speaker, and above-mentioned regular affective characteristics vector is inversely transformed into the affective characteristics vector in above-mentioned first feature space; And

Synthesis unit, it utilizes affective characteristics vector in above-mentioned first feature space to synthesize first speaker's emotional speech.

Images