JP2018084604A - Cross lingual voice synthesis model learning device, cross lingual voice synthesis device, cross lingual voice synthesis model learning method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a cross lingual voice synthesis technology which can synthesize voice in a language to be synthesized by a target speaker even when there is only one person's voice data in the language to be synthesized that is not voice data of the target speaker.SOLUTION: There are provided: a time information adjustment part 101 which generates after time information adjustment target speaker voice dataand after time information adjustment learning target language voice datafrom target speaker voice data and learning target language voice data; a vocal quality converter learning part 103 which learns a non-specific speaker vocal quality converter from a set of the after time information adjustment target speaker voice dataand the after time information adjustment learning target language voice data; a vocal quality conversion part 111 which uses the non-specific speaker vocal quality converter to generate after vocal quality conversion language to be synthesized voice data having a target speaker's vocal quality from the language to be synthesized voice data; and a synthesis model learning part 113 which learns a cross lingual voice synthesis model from the after vocal quality conversion language to be synthesized voice data and a set of the language to be synthesized speech information.SELECTED DRAWING: Figure 2

Description

  The present invention relates to speech synthesis technology, and more particularly to cross-lingual speech synthesis technology.

  In recent years, statistical parametric speech synthesis is a speech synthesis method that has become mainstream. For example, there are HMM (Hidden Markov Model) speech synthesis (Non-Patent Document 1) and DNN (Deep Neural Networks) speech synthesis (Non-Patent Document 2). These methods statistically model speech parameters such as spectral parameters (cepstrum, mel cepstrum, etc.) and pitch parameters (F0), which are feature amounts of speech. As a result, it is possible to generate a synthesized speech of a stable quality of an arbitrary speaker from a relatively small amount of learning speech data.

  When using this statistical parametric speech synthesis to achieve speech synthesis in any language, such as Japanese or English, with the voice quality of any speaker, generally the speech data spoken by the speaker in that language is required. It becomes. However, such voice data is not always available because the speaker cannot speak the language. In order to solve such a problem, a cross-lingual speech synthesis method has been proposed (Non-patent Documents 3 and 4).

  In this cross-lingual speech synthesis method, synthesized speech in the target language that has the voice quality of the target speaker is obtained by using speech data of the target language that is different from the target language of the target speaker to be synthesized. Can be generated.

Masashi Takashi, Tokuda Keiichi, Kobayashi Takao, Imai Kiyoshi, "HMM-based speech synthesis using dynamic features", IEICE Transactions D-II, vol.J79-D-II, No.12, pp. 2184-2190, 1996. Heiga Zen, Andrew Senior, Mike Schuster, “Statistical parametric speech synthesis using deep neural networks”, IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2013, pp.7962-7966, 2013. Masanobu Abe, Kiyohiro Shikano, Hisao Kuwabara, “Cross-language voice conversion”, IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP-90, pp.345-348, 1990. Yi-Jian Wu, Yoshihiko Nankaku, Keiichi Tokuda, “State mapping based method for cross-lingual speaker adaptation in HMM-based speech synthesis”, INTERSPEECH 2009, 10th Annual Conference of the International Speech Communication Association, pp.528-531, 2009 .

  However, the method of Non-Patent Document 3 requires speech data in both languages spoken by a bilingual speaker of a synthesis target language and a learning target language. In the method of Non-Patent Document 4, voice data from many speakers is required in each of the synthesis target language and the learning target language.

  Therefore, if it is difficult to collect speech data from bilingual speakers and collect speech data from many speakers in each language, neither method can be applied. In particular, such a problem becomes more prominent when the number of synthesis target languages increases.

  Therefore, the present invention can synthesize speech in the synthesis target language by the target speaker even when there is only one speech data in the synthesis target language that is not the target speaker's speech data. An object of the present invention is to provide a cross-lingual speech synthesis technology.

In one embodiment of the present invention, N is an integer greater than or equal to 1, n is an integer satisfying 1 ≦ n ≦ N, and target speaker voice data that is speech data in a target language by a target speaker and a target language input speaker by _n is audio data of the learning target language learning target language voice data _n (1 ≦ _n ≦ n) is a speech data uttered the same sentence, the said target speaker speech data learned language audio data A cross-lingual speech synthesis model learning device for learning a cross-lingual speech synthesis model for synthesizing speech in a synthesis target language by the target speaker from _n (1 ≦ n ≦ N), the target speaker speech adjust the time information data and the learned language audio data _n, and time information adjusted target speaker speech data _n and time information adjusted learned language time to generate audio data _n information adjustment unit, the time information After target speaker voice data _n after adjustment Learning an unspecified speaker voice quality converter that converts arbitrary voice data into voice data having the voice quality of the target speaker from a set (1 ≦ n ≦ N) of target language voice data _n after adjusting the recording time information Using the voice quality converter learning unit and the unspecified speaker voice quality converter, the voice quality of the target speaker is obtained from synthesis target language voice data which is voice data in a synthesis target language by a synthesis target language input speaker. From a voice quality conversion unit that generates post-voice quality conversion synthesis target language voice data, and a synthesis target language utterance information set that is a set of utterance information included in the synthesis target language voice data and the synthesis target language voice data after the voice quality conversion And a synthesis model learning unit for learning the cross-lingual speech synthesis model.

In one embodiment of the present invention, N is an integer greater than or equal to 1, n is an integer satisfying 1 ≦ n ≦ N, and target speaker voice data that is speech data in a target language by a target speaker and a target language input speaker by _n is audio data of the learning target language learning target language voice data _n (1 ≦ _n ≦ n) is a speech data uttered the same sentence, the said target speaker speech data learned language audio data a cross-lingual speech synthesis model learning device for learning a cross-lingual speech synthesis model for synthesizing speech in a synthesis target language by the target speaker from _n (1 ≦ n ≦ N), And the time information of the target speaker voice data and the learning target language voice data _n are adjusted, and the target information after adjusting the time information is adjusted. 's voice data _n and time information tone And the rear learning target language speech data _n the time information adjustment unit that generates, using the pronunciation vector construction rules, said a set of speech information speech included in the learning target language speech data _n learned the target language utterance information set _n A first pronunciation vector generation unit that generates a learning target language pronunciation vector set _n , which is a set of pronunciation vectors calculated from the utterance information, the time information adjusted target speaker voice data _n, and the time information adjusted Unspecified speaker voice quality conversion for converting arbitrary voice data into voice data having the voice quality of the target speaker from a set of learning target language voice data _n and the learning target language pronunciation vector set _n (1 ≦ n ≦ N) Included in the synthesis target language speech data, which is speech data in the synthesis target language by the synthesis target language input speaker, using the voice quality converter learning unit that learns the language and the pronunciation vector creation rules A second pronunciation vector generation unit that generates a synthesis target language pronunciation vector set that is a set of pronunciation vectors calculated from the speech information from a synthesis target language utterance information set that is a set of utterance information of the story; A voice quality conversion unit that generates post-speech synthesis target language voice data having voice quality of the target speaker from the synthesis target language voice data and the synthesis target language pronunciation vector set using a voice quality converter; and the voice quality A synthesis model learning unit that learns the cross-lingual speech synthesis model from the post-conversion synthesis target language speech data and the synthesis target language utterance information set.

  According to the present invention, by learning a voice quality converter that converts arbitrary voice data into voice data having the voice quality of the target speaker, one voice data in the synthesis target language that is not the voice data of the target speaker is obtained. Even if there is only the case, it is possible to generate a model for synthesizing speech in the synthesis target language by the target speaker.

The figure which shows an example of phoneme segmentation information. The figure which shows an example of a structure of the model learning apparatus 100 for cross-lingual speech synthesis. The figure which shows an example of operation | movement of the model learning apparatus 100 for cross-lingual speech synthesis. The figure which shows an example of a structure of the unspecified speaker voice quality converter learning device. The figure which shows an example of operation | movement of the unspecified speaker voice quality converter learning device 110. The figure which shows an example of a structure of the model learning apparatus 120 for speech synthesis. The figure which shows an example of operation | movement of the model learning apparatus 120 for speech synthesis. 1 is a diagram illustrating an example of a configuration of a cross-lingual speech synthesizer 200. FIG. The figure which shows an example of operation | movement of the cross-lingual speech synthesizer. The figure which shows an example of a pronunciation vector creation rule. The figure which shows an example of a structure of the model learning apparatus 300 for cross-lingual speech synthesis. The figure which shows an example of a structure of the unspecified speaker voice quality converter learning device 310. The figure which shows an example of operation | movement of the unspecified speaker voice quality converter learning device 310. The figure which shows an example of a structure of the model learning apparatus 320 for speech synthesis. The figure which shows an example of operation | movement of the model learning apparatus 320 for speech synthesis. The figure which shows an example of a structure of the pronunciation vector creation rule production | generation apparatus 400. FIG. The figure which shows an example of operation | movement of the pronunciation vector creation rule production | generation apparatus 400. FIG. The figure which shows an example of operation | movement of the 1st pronunciation statistics information generation part 410. FIG. The figure which shows an example of operation | movement of the pronunciation similarity determination part 420. The figure which shows an example of a structure of the model learning apparatus 520 for speech synthesis. The figure which shows an example of operation | movement of the model learning apparatus 520 for speech synthesis. The figure which shows an example of a structure of the model learning apparatus 620 for speech synthesis. The figure which shows an example of operation | movement of the model learning apparatus 620 for speech synthesis. The figure which shows an example of operation | movement of the speaker similarity determination part 612. The figure which shows an example of a structure of the model learning apparatus 720 for speech synthesis. The figure which shows an example of operation | movement of the model learning apparatus 720 for speech synthesis. The figure which shows an example of a structure of the model learning apparatus 820 for speech synthesis. The figure which shows an example of operation | movement of the model learning apparatus 820 for speech synthesis.

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the structure part which has the same function, and duplication description is abbreviate | omitted.

<Notation>
_ (Underscore) represents a subscript. For example, ^xy_z represents that _yz is a superscript to x, and _{xy_z} represents that _yz is a subscript to x.

<Definition>
Hereinafter, terms used in each embodiment will be described.

[Model for speech synthesis]
The speech synthesis model is a statistical model of speech feature quantity / acoustic feature quantity (hereinafter referred to as speech parameters) required for speech synthesis. Examples of audio parameters include spectral parameters such as cepstrum and mel cepstrum, and pitch parameters such as fundamental frequency (F0). The speech synthesis model learns using speech data and utterance information described later. Several methods for learning a model for speech synthesis have been proposed. For example, there are HMM speech synthesis (Non-Patent Document 1) and DNN speech synthesis (Non-Patent Document 2).

[Audio data]
The voice data is voice data recorded in advance for use in each learning (specifically, learning of an unspecified speaker voice quality converter and learning of a model for cross-lingual voice synthesis). The voice data is a voice of a sentence uttered by a speaker, and may be recorded as a voice parameter (spectrum parameter, pitch parameter) obtained as a result of performing signal processing on the voice data.

For non-specific speaker voice quality converter learning, voice data in the target language of the target speaker (hereinafter referred to as target speaker voice data), voice data in the target language of learning by N speakers (hereinafter referred to as the target speaker voice data) Learning language speech data _n (referred to as 1 ≦ n ≦ N) is required. Here, the target speaker is a subject that is a target for generating synthesized speech in a synthesis target language. The target speaker voice data and the learning target language voice data _n (1 ≦ n ≦ N) need to be voice data obtained by uttering the same sentence.

In addition, learning of a model for cross-lingual speech synthesis requires speech data in a synthesis target language (hereinafter referred to as synthesis target language speech data _m (1 ≦ m ≦ M)) by M speakers. However, M = 1 may be sufficient. That is, it suffices if there is at least one person to be synthesized speech data. At this time, instead of the synthesis target language speech data ₁ , it is simply referred to as synthesis target language speech data.

Note that a speaker who utters the learning target language voice data _n is referred to as a learning target language input speaker _n . A speaker who utters the synthesis target language speech data _m is referred to as a synthesis target language input speaker _m . In general, the learning target language input speaker _n (1 ≦ n ≦ N) and the synthesis target language input speaker _m (1 ≦ m ≦ M) are different from the target speaker.

[Speech information]
The utterance information is information such as pronunciation information given to each utterance (speech of a sentence uttered by a speaker) in voice data. One utterance information is assigned to each utterance in the voice data.

  The utterance information includes at least pronunciation information (reading and phonemes) corresponding to each utterance. In addition, phoneme segmentation information that is information on the start time and end time of each phoneme may be included. The start time and end time here are the elapsed time when the start point of each utterance is set to 0 [seconds]. An example of phoneme segmentation information is shown in FIG.

  Further, the speech information may include information other than phonemes and phoneme segmentation information, for example, accent information (accent type, accent phrase length), and part of speech information.

<First embodiment>
The cross-lingual speech synthesis model learning device 100 and the cross-lingual speech synthesis device 200 will be described below with reference to FIGS.

[Cross-Lingual Speech Synthesis Model Learning Device 100]
As shown in FIG. 2, the cross-lingual speech synthesis model learning device 100 includes an unspecified speaker voice quality converter learning device 110, a speech synthesis model learning device 120, and a recording unit 190. The recording unit 190 is a component that appropriately records information necessary for processing by the model learning device 100 for cross-lingual speech synthesis. The cross-lingual speech synthesis model learning apparatus 100 includes target speaker speech data, learning target language speech data _n (1 ≦ n ≦ N), synthesis target language speech data, and speech information included in the synthesis target language speech data. Is used as an input to learn and output a cross-lingual speech synthesis model for synthesizing speech in the synthesis target language by the target speaker.

The operation of the cross-lingual speech synthesis model learning apparatus 100 will be described with reference to FIG. The unspecified speaker voice quality converter learning device 110 receives the target speaker voice data and the learning target language voice data _n (1 ≦ n ≦ N) as input, and converts any voice data into voice data having the target speaker voice quality. An unspecified speaker voice quality converter to be converted is learned and output (S110). The speech synthesis model learning device 120 receives the synthesis target language speech data and the synthesis target language utterance information set as input, learns and outputs a cross-lingual speech synthesis model (S120). At that time, the unspecified speaker voice quality converter is used to convert the synthesis target language voice data into voice data having the voice quality of the target speaker (hereinafter referred to as synthesized voice language data after voice quality conversion).

  The configurations and operations of the unspecified speaker voice quality converter learning device 110 and the speech synthesis model learning device 120 will be described in detail below.

  First, the unspecified speaker voice quality converter learning device 110 will be described with reference to FIGS. As shown in FIG. 4, the unspecified speaker voice quality converter learning device 110 includes a time information adjustment unit 101 and a voice quality converter learning unit 103. The operation of the unspecified speaker voice quality converter learning device 110 will be described with reference to FIG.

The time information adjustment unit 101 receives the target speaker voice data and the learning target language voice data _n (1 ≦ n ≦ N) as inputs, and the voices of the sentences included in the target speaker voice data and the learning target language voice data _n ( In other words, voice data (hereinafter referred to as target speaker voice data _n after time information adjustment, and target language voice data _n after time information adjustment) is generated and output (S101). For the target speaker voice data, the time that is the voice data after time adjustment for each of the N learning target language input speakers in order to adjust the time information with each of the learning target language voice data _n (1 ≦ n ≦ N) Information-adjusted target speaker voice data _n (1 ≦ n ≦ N) is generated. For adjusting the time information, a method of associating the frames of the target speaker voice data and the learning target language voice data _n may be used. For example, as with many voice quality conversion methods, dynamic time warping (DTW) and DP (Dynamic Programming) matching can be used.

The voice quality converter learning unit 103 receives a set (1 ≦ n ≦ N) of the target speaker voice data _n after time information adjustment and the target language voice data _n after time information adjustment (1 ≦ n ≦ N) as an unspecified speaker voice quality converter. Learning and outputting (S103). For example, when the target speaker voice data _n after time information adjustment and the learning target language voice data _n after time information adjustment are expressed as spectrum parameters, the unspecified speaker voice quality converter is learned as follows.

Spectral parameters of target speaker voice data _n after time information adjustment are C _{n, target} (t), and spectral parameters of target language voice data _n after time information adjustment are C _{n, train} (t) (t is the frame number) Represents). The set of spectral parameters C _{n, train} (t) of the target language speech data _n after time information adjustment and the spectral parameters C _{n, target} (t) of the target speaker speech data _n after time information adjustment is used as the learning data. An unspecified speaker voice quality converter f _{train → target} is converted from the speaker spectrum parameter c _train (t) to the target speaker spectrum parameter c ^ _target (t) corresponding to the voice quality of the target speaker.

Here, the unspecified speaker voice quality converter f _{train → target} is a voice quality converter that converts an arbitrary input speaker spectrum parameter of a certain frame t into a target speaker spectrum parameter.

In a general voice quality converter learning method, when the voice data of speaker A is used to learn a voice quality converter that converts voice data having the voice quality of speaker B, the speech used for the voice quality converter learning is used. Unless voice data of the speaker A is input, the voice data having the voice quality of the speaker B cannot be converted. Therefore, here, a method based on the neural network of (Reference Non-Patent Document 1) is used as a learning method of the voice quality converter.
(Non-patent document 1) Takashi Nose, Takahiro Shinozaki, Yojiro Ito, Akinori Ito, “Examination of many-to-one voice quality conversion that does not require user speech based on neural network”, Transactions of the Acoustical Society of Japan, March 2015, 3 -2-1, pp.271-274, 2015
(Reference Non-Patent Document 1), when learning a voice quality converter, voice data of a large number of target language input speakers that are input to the voice quality converter and one target speaker that is the output of the voice quality converter Are used as learning data. As a result, it is possible to convert any voice data of the speaker into voice data having the voice quality of the target speaker.

  Next, the speech synthesis model learning device 120 will be described with reference to FIGS. As shown in FIG. 6, the speech synthesis model learning device 120 includes a voice quality conversion unit 111 and a synthesis model learning unit 113. The operation of the speech synthesis model learning device 120 will be described with reference to FIG.

The voice quality conversion unit 111 receives the synthesis target language voice data as input and generates and outputs voice quality converted synthesis target language voice data using an unspecified speaker voice quality converter (S111). For example, when the unspecified speaker voice quality converter is expressed by Equation (1), the target speaker of the synthesis target language having the voice quality of the target speaker from the spectrum parameter C _{synth, org} (t) of the synthesis target language speech data. Convert to spectrum parameter C ^ _{synth, target} (t) (t is frame number).

When (Reference Non-Patent Document 1) is used for learning of the voice quality converter, the voice quality conversion may be performed according to (Reference Non-Patent Document 1). The target speaker spectrum parameter C ^ _{synth, target} (t) calculated according to the equation (1) is the speech data to be synthesized after the voice quality conversion.

  The synthesis model learning unit 113 learns and outputs a cross-lingual speech synthesis model, using the post-voice quality conversion synthesis target language speech data generated in S111 and the synthesis target language utterance information set as inputs (S113). For learning of a model for cross-lingual speech synthesis, (Non-patent Document 1), (Non-Patent Document 2), etc. can be used.

  Hereinafter, the cross-lingual speech synthesis apparatus 200 will be described. It is assumed that the cross-lingual speech synthesis model is an HMM speech synthesis model as described in (Non-Patent Document 1).

[Cross-Lingual Speech Synthesizer 200]
As shown in FIG. 8, the cross-lingual speech synthesizer 200 includes a text analysis unit 210, a speech parameter generation unit 220, a speech waveform generation unit 230, and a recording unit 290. The recording unit 290 is a component that appropriately records information necessary for processing of the cross-lingual speech synthesizer 200. The cross-lingual speech synthesizer 200 receives a synthesized text that is a text in a synthesis target language that is a target of speech synthesis, generates a synthesized speech that reads out the synthesized text using a cross-lingual speech synthesis model, and outputs the synthesized speech. To do.

The operation of the cross-lingual speech synthesizer 200 will be described with reference to FIG. The text analysis unit 210 receives the synthesized text as input and generates and outputs context information such as reading of the synthesized text and accent by text analysis (S210). The voice parameter generation unit 220 receives the context information generated in S210, generates a voice parameter (spectrum parameter or pitch parameter) using the cross-lingual voice synthesis model, and outputs it (S220). The speech waveform generation unit 230 receives the speech parameter generated in S220, generates a synthesized speech using a speech synthesis filter, and outputs the synthesized speech (S230). As the speech synthesis filter, for example, a filter described in (Reference Non-Patent Document 2) may be used.
(Reference Non-Patent Document 2) Sei Imai, Kazuo Sumita, Chieko Furuichi, “Mel Log Spectrum Approximation (MLSA) Filter for Speech Synthesis”, IEICE Transactions A, Vol. J66-A, No. 2, pp.122-129, 1983.

Instead of target speaker voice data and learning target language voice data _n (1 ≤ n ≤ N), time information adjusted target speaker voice data _n (1 ≤ n ≤ N), time The information-adjusted learning target language speech data _n (1 ≦ n ≦ N) may be input to the model learning device for cross-lingual speech synthesis.

  According to the invention of the present embodiment, an unspecified speaker voice quality converter that is trained by using target speaker voice data prepared in advance and N target language voice data for learning as input is different from the target speaker. The synthesis target language speech data of the name is converted into the synthesis target language speech data after voice quality conversion having the voice quality of the target speaker. Next, a speech model in cross-lingual speech synthesis that is statistical parametric speech synthesis, that is, speech in the target language by the target speaker, is obtained from a set of speech data after speech quality conversion and speech information of the target language speech data. Learn a model to synthesize.

  This makes it possible to generate a model for cross-lingual speech synthesis from N language learning language speech data and one synthesis target language speech data. By using this model, cross-lingual speech synthesis is possible. Is possible. For this reason, it is not necessary to prepare speech data by a bilingual speaker in the learning target language and the synthesis target language, or speech data by a large number of speakers for each of the learning target language and the synthesis target language. Therefore, cross-lingual speech synthesis can be realized at a lower cost than in the past.

<Second embodiment>
In the first embodiment, only voice data (spectrum parameters) is used for learning of the unspecified speaker voice quality converter and voice quality conversion by the unspecified speaker voice quality converter. That is, voice quality conversion is performed using an unspecified speaker voice quality converter that has been learned without using the utterance information accompanying the voice data. For this reason, the accuracy of voice quality conversion may deteriorate, and the quality of the final synthesized speech may deteriorate.

  Therefore, in the present embodiment, it is considered that the speech information is also used for the learning of the unspecified speaker voice quality converter and the voice quality conversion by the unspecified speaker voice quality converter. However, generally speaking, if the language is different, pronunciation information such as phoneme system will be different.For this reason, the utterance information will be used for learning of the unspecified speaker voice quality converter and voice quality conversion by the unspecified speaker voice quality converter. Cannot be used. In order to use the utterance information, a correspondence relationship between similar pronunciation information (phonemes) in both languages is created in order to absorb the difference in pronunciation information between the learning target language and the synthesis target language. Thereby, learning of the unspecified speaker voice quality converter considering the difference in pronunciation information between the two languages, and voice quality conversion by the unspecified speaker voice quality converter are realized.

  Hereinafter, the correspondence between similar pronunciation information (phonemes) between the learning target language and the synthesis target language is referred to as a pronunciation vector creation rule. An example is shown in FIG. Each row in the table of FIG. 10 is obtained by indexing a phoneme of a similar learning target language and a phoneme of a synthesis target language, and the number of rows is the number L of phonemes of the learning target language. In this example, the language to be learned is Japanese and the language to be synthesized is English. For example, looking at the third line (the line where the index is 3), the Japanese phoneme "s" and the English phoneme "s" , “Th” is in a similar relationship and can be seen to correspond.

  The expression of the pronunciation vector creation rule is limited to the table shown in FIG. 10 that shows the phonemes of the synthesis target language similar to the phonemes of the learning target language, where the number of phonemes L of the learning target language is the number of rows. However, in the following description, it is assumed that the pronunciation vector creation rules are represented by a table with the number of rows L as shown in FIG.

  The cross-lingual speech synthesis model learning apparatus 300 will be described below with reference to FIGS. Note that the cross-lingual speech synthesizer 200 can be used to generate synthesized speech.

[Cross-Lingual Speech Synthesis Model Learning Device 300]
As shown in FIG. 11, the cross-lingual speech synthesis model learning device 300 includes an unspecified speaker voice quality converter learning device 310, a speech synthesis model learning device 320, and a recording unit 190. It is assumed that a pronunciation vector creation rule is recorded in the recording unit 190 in advance. The model learning device 300 for cross-lingual speech synthesis includes a target speaker speech data, learning target language speech data _n (1 ≦ n ≦ N), and a set of utterance information of speech included in the learning target language speech data _n (hereinafter referred to as learning). The target language utterance information set _n (referred to as 1 ≦ n ≦ N), the synthesis target language speech data, and the synthesis target language utterance information set are input and the cross-lingual speech synthesis model is learned and output.

  Hereinafter, the configuration and operation of the unspecified speaker voice quality converter learning device 310 and the speech synthesis model learning device 320 will be described in detail.

  First, the unspecified speaker voice quality converter learning device 310 will be described with reference to FIGS. As shown in FIG. 12, the unspecified speaker voice quality converter learning device 310 includes a time information adjustment unit 101, a first pronunciation vector generation unit 302, and a voice quality converter learning unit 303. The operation of the unspecified speaker voice quality converter learning device 310 will be described with reference to FIG.

The time information adjustment unit 101 receives the target speaker voice data and the learning target language voice data _n (1 ≦ n ≦ N) as inputs, and the voices of the sentences included in the target speaker voice data and the learning target language voice data _n ( The target speaker voice data _n after adjusting time information and the learning target language voice data _n after adjusting time information are generated and output (S101).

The first pronunciation vector generation unit 302 receives the learning target language utterance information set _n as an input, generates and outputs a learning target language pronunciation vector set _n using the pronunciation vector creation rules read from the recording unit 190 (S302). . Here, the pronunciation vector is a vector calculated for each frame of speech in the speech data, and is a vector indicating what kind of pronunciation information (phoneme) the frame is. When the pronunciation vector creation rule is expressed by a table with the number of rows L as shown in FIG. 10, the dimension of the pronunciation vector is L. A pronunciation vector (hereinafter referred to as a learning target language pronunciation vector) is calculated from the utterance information of the learning target language utterance information set _n , and the set is generated as a learning target language pronunciation vector set _n .

Hereinafter, a method for calculating a pronunciation vector from pronunciation information (phonemes) included in certain utterance information will be specifically described. First, for the pronunciation information (phoneme), a corresponding index idx is obtained using a pronunciation vector creation rule (where 1 ≦ idx ≦ L). In the example of FIG. 10, a row having a learning target language phoneme (here, Japanese phoneme) is found from the middle column, and an index idx is obtained. Next, the obtained index idx is converted into a numerical vector to calculate a pronunciation vector. As a numerical vectorization method, for example, the following 1-of-K representation of intention information is used, and the pronunciation vector V = (v ₁ , v ₂ ,…, v _L ) (L is the learning target) from the index idx Language phoneme number).

However, j = 1,..., L.

  The 1-of-K expression is used for the pronunciation vector in general when 1-dimensional numerical values are used instead of 1-dimensional numerical values when L-valued information (here, index idx) is used as the input of the neural network. This is because a higher-performance learning result (that is, an unspecified speaker voice quality converter) can be obtained by treating as an of-K expression.

The voice quality converter learning unit 303 receives a set (1 ≦ n ≦ N) of the target speaker voice data _n after time information adjustment, the learning target language voice data _n after time information adjustment, and the learning target language pronunciation vector set _n as inputs. An unspecified speaker voice quality converter is learned and output (S303). For example, when the target speaker voice data _n after time information adjustment and the learning target language voice data _n after time information adjustment are expressed as spectrum parameters, the unspecified speaker voice quality converter is learned as follows.

Spectral parameters of target speaker speech data _n after time information adjustment is C _{n, target} (t), spectral parameters of learning target language speech data _n after time information adjustment is C _{n, train} (t), and learning target language pronunciation vector set _Let V _{n, train} (t) be the learning target language pronunciation vector that is an element of _n (t represents the frame number). Spectral parameters C _{n, train} (t) of the target language speech data _n after time information adjustment and the corresponding language target pronunciation vector V _{n, train} (t) (C _{n, train} (t), V _{n, train} (t)) and spectral parameter C _{n, target} (t) of target speaker speech data _n after time information adjustment are used as training data, and extended input speaker spectral parameters (c _train (t), v _train (t) ) To an unspecified speaker voice quality converter f _{train → target} for conversion to a target speaker spectrum parameter c ^ _target (t) corresponding to the voice quality of the target speaker.

Here, the unspecified speaker voice quality converter f _{train → target} is a voice quality converter for converting any extended input speaker spectrum parameter of a certain frame t into a target speaker spectrum parameter.

The learning algorithm of the unspecified speaker voice quality converter f _{train → target} may be the same method as described in the first embodiment (reference non-patent document 1). However, as learning data, is that it uses a set of spectral parameters of the spectral parameters and the set time information adjusted target speaker speech data _n corresponding learning target language pronunciation vector after time information adjustment learned language audio data _n different .

  Next, the speech synthesis model learning device 320 will be described with reference to FIGS. As shown in FIG. 14, the speech synthesis model learning device 320 includes a voice quality conversion unit 311, a second pronunciation vector generation unit 312, and a synthesis model learning unit 313. The operation of the speech synthesis model learning device 320 will be described with reference to FIG.

The second pronunciation vector generation unit 312 receives the synthesis target language utterance information set as an input, and generates and outputs a synthesis target language pronunciation vector set using the pronunciation vector creation rules read from the recording unit 190 (S312). The generation method of the synthesis target language pronunciation vector set is the same as the generation method of the learning target language pronunciation vector set _n in the first pronunciation vector generation unit 302, and the pronunciation vector calculated from the utterance information of the synthesis target language utterance information set ( Hereinafter, a set of synthesis target language pronunciation vectors) is a synthesis target language pronunciation vector set.

The voice quality conversion unit 311 receives the synthesis target language voice data and the synthesis target language pronunciation vector set generated in S312 as input, generates voice quality converted synthesis target language voice data using an unspecified speaker voice quality converter, and outputs it. (S311). For example, when the unspecified speaker voice quality converter is expressed by Equation (2), the synthesis target language pronunciation vector V _{synth, org} (t) corresponding to the spectrum parameter C _{synth, org} (t) of the synthesis target language speech data. (C _{synth, org} (t), V _{synth, org} (t)) is converted into a target speaker spectrum parameter C ^ _{synth, target} (t) of the synthesis target language having the target speaker's voice quality.

When (Reference Non-Patent Document 1) is used for learning of the voice quality converter, the voice quality conversion may be performed according to (Reference Non-Patent Document 1). The target speaker spectrum parameter C ^ _{synth, target} (t) calculated according to equation (2) is the speech data to be synthesized after voice quality conversion.

  The synthesis model learning unit 313 learns and outputs a cross-lingual speech synthesis model using the post-voice quality conversion synthesis target language speech data generated in S311 and the synthesis target language utterance information set as inputs (S313).

  Hereinafter, the pronunciation vector creation rule generation device 400 will be described with reference to FIGS. 16 to 17. Here, in order that the pronunciation vector creation rule can be generated from the speech data of the learning target language and the speech data of the synthesis target language, it is necessary that the speech information of each speech data includes phoneme segmentation information.

When an international phonetic alphabet (IPA) can be used, a pronunciation vector creation rule can be easily generated. Here, the international phonetic symbol is a phonetic symbol determined by the International Phonetic Society, and is a symbol that can express pronunciation information (phonemes) in any language (reference URL).
(Reference URL: https://www.internationalphoneticassociation.org/content/full-ipa-chart)

  If there are rules for converting speech information in the target language to IPA and rules for converting speech information in the target language into IPA, correspondences are established via the conversion rules from each language to IPA. That is, a pronunciation vector creation rule can be created.

[Pronunciation vector creation rule generator 400]
As shown in FIG. 16, the pronunciation vector creation rule generation device 400 includes a first pronunciation statistical information generation unit 410, a second pronunciation statistical information generation unit 415, a pronunciation similarity determination unit 420, and a recording unit 490. The recording unit 490 is a component that appropriately records information necessary for the processing of the pronunciation vector creation rule generation device 400. The pronunciation vector creation rule generation device 400 includes a target speaker voice data, a set of speech information included in the target speaker voice data (hereinafter referred to as a target speaker speech information set), a synthesis target language voice data, and a synthesis target language. Using the utterance information set as an input, a pronunciation vector creation rule is generated and output.

  Hereinafter, the operation of the pronunciation vector creation rule generation device 400 will be described with reference to FIG.

The first pronunciation statistic information generation unit 410 receives the target speaker voice data and the target speaker utterance information set as input, and generates and outputs learning target language pronunciation statistic information (S410). The operation of the first pronunciation statistic information generation unit 410 is specifically as follows (see FIG. 18). The speech segment of the target language phoneme _j (j = 1, ..., L, where L is the number of phonemes in the target language) is identified from the target speaker voice data, and the target language phoneme _j speech data (eg, spectrum) Parameter) is extracted (S410-1). Using the voice data of the extracted learned target language phoneme _j, calculates pronunciation statistics learning target language phoneme _j (S410-2). Learning target language pronunciation statistical information is generated as a collection of pronunciation statistical information of the learning target language phoneme ₁ ,..., Pronunciation statistical information of the learning target language phoneme _L (S410-3). Here, as the statistical information, it is possible to use an average value, variance, quartile, or a statistical model such as a normal mixture distribution (Gaussian Mixture Model, GMM) of the entire speech data of the language phoneme _{j to be} learned. it can. Note that the learning target language speech data _n and the learning target language utterance information set _n may be used instead of the target speaker voice data and the target speaker utterance information set (where n is an integer of 1 to N).

The second pronunciation statistic information generation unit 415 receives the synthesis target language speech data and the synthesis target language utterance information set as input, and generates and outputs synthesis target language pronunciation statistical information (S415). Operation of the second sound information statistics acquisition unit 415 is similar to the operation of the first sound statistical information generating unit 410, a set of sound statistics synthesized target language phoneme _1, ..., pronunciation statistics synthesized target language phoneme _K As a synthesis target language pronunciation statistical information (where K is the number of phonemes in the synthesis target language).

The pronunciation similarity determination unit 420 receives the learning target language pronunciation statistical information generated in S410 and the synthesis target language pronunciation statistical information generated in S415, and generates and outputs a pronunciation vector creation rule (S420). The operation of the pronunciation similarity determination unit 420 is specifically as follows (see FIG. 19). First, the distance d _ij between the pronunciation statistical information (i = 1,..., K) of the synthesis target language phoneme _i and the pronunciation statistical information (j = 1,..., L) of the learning target language phoneme _j is calculated (S420-1). ). The calculation method of the distance between pronunciation statistical information changes with pronunciation statistical information to be used. For example, when GMM is used as pronunciation statistical information, KL divergence, which is a measure for measuring a difference between probability distributions, can be used. Further, when the average value and the variance are used, the Mahalanobis distance and when the quartile is used, the distance _dij may be calculated using the Euclidean distance of the quartile. Next, for the synthesis target language phoneme _i , the learning target language phoneme _{j_min} (1 ≦ j _min ≦) is the smallest among the distances d _i1 to the learning target language phoneme ₁ and the distances d _{iL to} the learning target language phoneme _L. L) is specified (S420-2). _Assuming that the target language phoneme _i and the target language phoneme _{j_min} correspond to each other, the index, the target language phoneme _{j_min} , and the target language phoneme _i are added to the table, and finally the pronunciation vector creation rule is generated (S420-3).

  According to the invention of this embodiment, a cross-lingual speech synthesis model that improves the quality of synthesized speech can be generated by generating an unspecified speaker voice quality converter using speech information of a language to be learned. . In addition, the quality of synthesized speech can be improved by using the cross-lingual speech synthesis model.

<Third embodiment>
In the first embodiment, voice data of one speaker is used as the synthesis target language voice data.

  In the present embodiment, voice data of a plurality of speakers is used as the synthesis target language voice data. As a result, the quality of the synthesized speech can be improved.

  Hereinafter, the speech synthesis model learning device 520 will be described with reference to FIGS. The unspecified speaker voice quality converter learning device 110 can be used to generate the unspecified speaker voice quality converter. That is, a non-specific speaker voice quality converter learning device 110, a speech synthesis model learning device 520, and a cross-lingual speech synthesis model learning device including the recording unit 190 can be configured. In addition, the cross-lingual speech synthesizer 200 can be used to generate synthesized speech.

[Speech Synthesis Model Learning Device 520]
As shown in FIG. 20, the speech synthesis model learning device 520 includes a voice quality conversion unit 511 and a synthesis model learning unit 513. The operation of the speech synthesis model learning device 520 will be described with reference to FIG. Here, synthesis target language speech data _m (1 ≦ m ≦ M) is required.

The voice quality conversion unit 511 receives the synthesis target language voice data _m as input, generates post-voice quality conversion synthesis target language voice data _m using an unspecified speaker voice quality converter, and outputs it (S511). For example, when the unspecified speaker voice quality converter is expressed by the equation (1), the synthesis target language having the voice quality of the target speaker is obtained from the spectrum parameter C _{synth, org, m} (t) of the synthesis target language voice data _m . Convert to target speaker spectrum parameter _m C ^ _{synth, target, m} (t).

When (Reference Non-Patent Document 1) is used for learning of the voice quality converter, the voice quality conversion may be performed according to (Reference Non-Patent Document 1). The target speaker spectrum parameter _m C ^ _{synth, target, m} (t) calculated according to the equation (1) is the speech data _{m to be} synthesized after voice conversion.

The synthesis model learning unit 513 learns and outputs a cross-lingual speech synthesis model using the post-voice quality conversion synthesis target language speech data _m generated in S511 and the synthesis target language utterance information set _m as inputs (S513). (Non-Patent Document 1) and (Non-Patent Document 2) can be used for learning a model for cross-lingual speech synthesis. Further, as in (Reference Non-Patent Document 3), an average voice model learned using voice data of a plurality of speakers can be used as a voice synthesis model.
(Non-patent document 3) Masanori Tamura, Takashi Masuko, Keiichi Tokuda, Takao Kobayashi, “Speaker adaptation of pitch spectrum in speech synthesis based on HMM”, IEICE Transactions D, Vol.J85-D2, No .4, pp.545-553, 2002.

  In general, when learning a speech synthesis model, it is possible to generate higher-quality synthesized speech as the amount of speech data and speech information that can be used for learning increases. In the present embodiment, M name synthesis target language speech data is used. In the case of using the M synthesis target language speech data, the M name speech quality-converted synthesis target language speech data having the voice quality of the target speaker can be used as learning data for the speech synthesis model. This increases the amount of speech data that can be used for speech synthesis model learning compared to the case of using one target language speech data for synthesis, so cross-lingual speech synthesis improves the quality of synthesized speech. Model can be generated. In addition, the quality of synthesized speech can be improved by using the cross-lingual speech synthesis model.

<Fourth embodiment>
In the third embodiment, all post-voice-quality synthesis target language speech data generated from synthesis target language speech data by M speakers is used for speech synthesis model learning.

However, when the speech characteristics of the synthesis target language input speaker _m and the target speaker are greatly different, the conversion accuracy of the post-voice quality conversion target language speech data _m generated by the voice quality conversion unit 511 may deteriorate. Accordingly, it is not always appropriate to use the target language speech data for synthesis after voice quality conversion for all M names for speech synthesis model learning.

Therefore, in the present embodiment, in order to avoid this problem, only speech data with a high conversion accuracy among the M language voice synthesis target language speech data for speech synthesis conversion is used for learning the speech synthesis model.
As a result, the quality of the synthesized speech can be improved.

  Hereinafter, the speech synthesis model learning device 620 will be described with reference to FIGS. 22 to 23. The unspecified speaker voice quality converter learning device 110 can be used to generate the unspecified speaker voice quality converter. That is, a non-specific speaker voice quality converter learning device 110, a speech synthesis model learning device 620, and a cross-lingual speech synthesis model learning device including the recording unit 190 can be configured. In addition, the cross-lingual speech synthesizer 200 can be used to generate synthesized speech.

[Speech Synthesis Model Learning Device 620]
As illustrated in FIG. 22, the speech synthesis model learning device 620 includes a voice quality conversion unit 511, a speaker similarity determination unit 612, and a synthesis model learning unit 513. The operation of the speech synthesis model learning device 620 will be described with reference to FIG.

The voice quality conversion unit 511 receives the synthesis target language voice data _m as input, generates post-voice quality conversion synthesis target language voice data _m using an unspecified speaker voice quality converter, and outputs it (S511).

The speaker similarity determination unit 612 receives the voice quality-converted synthesis target language voice data _m (1 ≦ m ≦ M) and the target speaker voice data generated in S511 and inputs the voice quality-converted target language voice data _m and the target voice data _m. calculating a similarity _m indicating the degree of similarity of the speaker's speech data, if after voice conversion synthesized target language voice data _m, based on the similarity _m is determined to be similar to the target-speaker speech data, voice By adding the post-conversion synthesis target language speech data _m to the target speaker-similar synthesis target language speech data set, which is a set of post-conversion speech synthesis target language speech data similar to the target speaker speech data, A speaker-like synthesis target language speech data set is generated (S612). For the calculation of the similarity, the difference between the speech quality conversion target speech data _m after the voice quality conversion and the target speaker voice data, for example, the difference in the spectral parameters is used. As such a method, there is a method of using a normal mixture distribution (GMM) (reference non-patent document 4).
(Reference Non-Patent Document 4) DA Reynolds, “Speaker identification and verification using Gaussian mixture speaker models”, Speech Communication, vol.17, Issues 1-2, pp.91-108, 1995.

The operation of the speaker similarity determination unit 612 is specifically as follows (see FIG. 24). First, the target speaker's voice data is used to learn the target speaker's GMM (S612-1). Then, the target speaker GMM learned, enter the post-voice conversion synthesized target language voice data _m, and calculates the likelihood of the synthesis target language input speaker _m as the similarity _m (S612-2). Finally, when the similarity _m is equal to or higher than a predetermined threshold (or larger than the predetermined threshold), it is determined that the synthesis target language voice data _m after voice quality conversion and the target speaker voice data are similar (that is, synthesis) The target language input speaker _m is determined to be a speaker having high similarity to the target speaker), and the speech-to-synthesized synthesis target language speech data _m is determined as an element of the target speaker similar synthesis target language speech data set (S612-). 3).

The synthesis model learning unit 513 learns a cross-lingual speech synthesis model using the target speaker similar synthesis target language speech data set generated in S612 and the synthesis target language utterance information set _m (1 ≦ m ≦ M) as inputs. Are output (S513). Here, only the synthesis target language utterance information set _m corresponding to the synthesis target language speech data _m after voice quality conversion, which is an element of the target speaker similar synthesis target language speech data set, is used.

Note that, instead of generating the target speaker similar synthesis target language speech data set in S612, the synthesized speech language data after voice quality conversion having the highest similarity may be generated as the target speaker similar synthesis target language speech data. In this case, the synthesis model learning unit 513 receives the target speaker similar synthesis target language speech data generated in S612 and the synthesis target language utterance information set _m (1 ≦ m ≦ M) as input, and generates a cross-lingual speech synthesis model. Learn and output.

  In the present embodiment, only the speech data after speech quality conversion with high similarity is used as learning data for the speech synthesis model. This makes it possible to generate a cross-lingual speech synthesis model that improves the quality of the synthesized speech. In addition, the quality of synthesized speech can be improved by using the cross-lingual speech synthesis model.

  In particular, when only speech data after speech quality conversion with the highest degree of similarity is used as learning data, it is based on a speech synthesis model that is generated when speech data in the synthesis target language by the target speaker is available. A model for cross-lingual speech synthesis close to the quality of synthesized speech can be generated. In addition, by using the cross-lingual speech synthesis model, a synthesized speech that is close to the quality of the synthesized speech by the speech synthesis model that is generated when speech data in the target language by the target speaker is available is generated. be able to.

<Fifth embodiment>
In the second embodiment, voice data of one speaker is used as the synthesis target language voice data.

  In the present embodiment, voice data of a plurality of speakers is used as the synthesis target language voice data. As a result, the quality of the synthesized speech can be improved.

  Hereinafter, the speech synthesis model learning apparatus 720 will be described with reference to FIGS. Note that the unspecified speaker voice quality converter learning device 310 can be used to generate the unspecified speaker voice quality converter. That is, a cross-lingual speech synthesis model learning device including the unspecified speaker voice quality converter learning device 310, the speech synthesis model learning device 720, and the recording unit 190 can be configured. Here, it is assumed that a pronunciation vector creation rule is recorded in the recording unit 190 in advance. In addition, the cross-lingual speech synthesizer 200 can be used to generate synthesized speech.

[Speech Synthesis Model Learning Device 720]
As shown in FIG. 25, the speech synthesis model learning device 720 includes a voice quality conversion unit 711, a second pronunciation vector generation unit 712, and a synthesis model learning unit 713. The operation of the speech synthesis model learning device 720 will be described with reference to FIG. Here, synthesis target language speech data _m (1 ≦ m ≦ M) is required.

The second pronunciation vector generation unit 712 receives the synthesis target language utterance information set _m as an input, generates a synthesis target language pronunciation vector set _m using the pronunciation vector creation rules read from the recording unit 190, and outputs it (S712). . The generation method of the synthesis target language pronunciation vector set _{m is the same as} the generation method of the synthesis target language pronunciation vector set in the second pronunciation vector generation unit 312, and the pronunciation vector calculated from the utterance information of the synthesis target language utterance information set _m A set of synthesis target language pronunciation vectors (hereinafter referred to as a synthesis target language pronunciation vector) is a synthesis target language pronunciation vector set _m .

The voice quality conversion unit 711 receives the synthesis target language speech data _m and the synthesis target language pronunciation vector set _m generated in S712 as input, and generates the post-voice quality converted synthesis target language speech data _m using an unspecified speaker voice quality converter. And output (S711). For example, when the unspecified speaker voice quality converter is expressed by Equation (2), the synthesis target language pronunciation vector V _{synth, org} corresponding to the spectrum parameter C _{synth, org, m} (t) of the synthesis target speech data _{m , m} (t) (C _{synth, org, m} (t), V _{synth, org, m} (t)) from the target speaker spectral parameter _m C ^ _synth, Convert to _{target, m} (t).

When (Reference Non-Patent Document 1) is used for learning of the voice quality converter, the voice quality conversion may be performed according to (Reference Non-Patent Document 1). The target speaker spectrum parameter _m C ^ _{synth, target, m} (t) calculated according to the equation (2) is the speech data _{m to be} synthesized after voice conversion.

The synthesis model learning unit 713 learns and outputs a cross-lingual speech synthesis model using the post-voice-quality-converted synthesis target language speech data _m generated in S711 and the synthesis target language utterance information set _m as inputs (S713). (Non-patent document 1), (Non-patent document 2), and (Reference non-patent document 3) can be used for learning of a model for cross-lingual speech synthesis.

  In the present embodiment, M name synthesis target language speech data is used. In the case of using the M synthesis target language speech data, the M name speech quality-converted synthesis target language speech data having the voice quality of the target speaker can be used as learning data for the speech synthesis model. This increases the amount of speech data that can be used for speech synthesis model learning compared to the case of using one target language speech data for synthesis, so cross-lingual speech synthesis improves the quality of synthesized speech. Model can be generated. In addition, as in the second embodiment, since the unspecified speaker voice quality converter learned using the speech information of the language to be learned is used, a cross-lingual speech synthesis model that further improves the quality of the synthesized speech is generated. be able to. In addition, the quality of synthesized speech can be improved by using the cross-lingual speech synthesis model.

<Sixth embodiment>
In the fifth embodiment, all post-voice quality conversion target language speech data generated from synthesis target language speech data by M speakers is used for speech synthesis model learning.

However, if the speech characteristics of the synthesis target language input speaker _m and the target speaker are greatly different, the conversion accuracy of the post-voice quality conversion synthesis target language speech data _m generated by the voice quality conversion unit 711 may deteriorate. Accordingly, it is not always appropriate to use the target language speech data for synthesis after voice quality conversion for all M names for speech synthesis model learning.

Therefore, in the present embodiment, in order to avoid this problem, only speech data with a high conversion accuracy among the M language voice synthesis target language speech data for speech synthesis conversion is used for learning the speech synthesis model.
As a result, the quality of the synthesized speech can be improved.

  Hereinafter, the speech synthesis model learning device 820 will be described with reference to FIGS. 27 to 28. Note that the unspecified speaker voice quality converter learning device 310 can be used to generate the unspecified speaker voice quality converter. That is, a cross-lingual speech synthesis model learning device including the unspecified speaker voice quality converter learning device 310, the speech synthesis model learning device 820, and the recording unit 190 can be configured. Here, it is assumed that a pronunciation vector creation rule is recorded in the recording unit 190 in advance. In addition, the cross-lingual speech synthesizer 200 can be used to generate synthesized speech.

[Speech Synthesis Model Learning Device 820]
As shown in FIG. 27, the speech synthesis model learning device 820 includes a voice quality conversion unit 711, a second pronunciation vector generation unit 712, a speaker similarity determination unit 812, and a synthesis model learning unit 713. The operation of the speech synthesis model learning device 820 will be described with reference to FIG.

The second pronunciation vector generation unit 712 receives the synthesis target language utterance information set _m as an input, generates a synthesis target language pronunciation vector set _m using the pronunciation vector creation rules read from the recording unit 190, and outputs it (S712). .

The voice quality conversion unit 711 receives the synthesis target language speech data _m and the synthesis target language pronunciation vector set _m generated in S712 as input, and generates the post-voice quality converted synthesis target language speech data _m using an unspecified speaker voice quality converter. And output (S711).

The speaker similarity determination unit 812 receives the post-voice quality conversion synthesis target language voice data _m (1 ≦ m ≦ M) and the target speaker voice data generated in S711, and inputs the post-voice quality conversion target language voice data _m and the target. calculating a similarity _m indicating the degree of similarity of the speaker's speech data, if after voice conversion synthesized target language voice data _m, based on the similarity _m is determined to be similar to the target-speaker speech data, voice By adding the post-conversion synthesis target language speech data _m to the target speaker-similar synthesis target language speech data set, which is a set of post-conversion speech synthesis target language speech data similar to the target speaker speech data, A speaker-like synthesis target language speech data set is generated (S812). The operation of the speaker similarity determination unit 812 may be the same as the operation of the speaker similarity determination unit 612.

The synthesis model learning unit 713 learns a cross-lingual speech synthesis model using the target speaker similar synthesis target language speech data set generated in S812 and the synthesis target language utterance information set _m (1 ≦ m ≦ M) as inputs. Are output (S713). Here, only the synthesis target language utterance information set _m corresponding to the synthesis target language speech data _m after voice quality conversion, which is an element of the target speaker similar synthesis target language speech data set, is used.

Note that, instead of generating the target speaker similar synthesis target language speech data set in S812, the speech quality converted synthesis target language speech data having the highest degree of similarity may be generated as the target speaker similar synthesis target language speech data. In this case, the synthesis model learning unit 713 receives the target speaker similar synthesis target language speech data generated in S812 and the synthesis target language utterance information set _m (1 ≦ m ≦ M) as input, and generates a cross-lingual speech synthesis model. Learn and output.

  In the present embodiment, only the speech data after speech quality conversion with high similarity is used as learning data for the speech synthesis model. This makes it possible to generate a cross-lingual speech synthesis model that improves the quality of the synthesized speech. In addition, as in the second embodiment, since the unspecified speaker voice quality converter learned using the speech information of the language to be learned is used, a cross-lingual speech synthesis model that further improves the quality of the synthesized speech is generated. be able to. In addition, the quality of synthesized speech can be improved by using the cross-lingual speech synthesis model.

  In particular, when only speech data after speech quality conversion with the highest degree of similarity is used as learning data, it is based on a speech synthesis model that is generated when speech data in the synthesis target language by the target speaker is available. A model for cross-lingual speech synthesis close to the quality of synthesized speech can be generated. In addition, by using the cross-lingual speech synthesis model, a synthesized speech that is close to the quality of the synthesized speech by the speech synthesis model that is generated when speech data in the target language by the target speaker is available is generated. be able to.

<Modification>
The present invention is not limited to the above-described embodiment, and it goes without saying that modifications can be made as appropriate without departing from the spirit of the present invention. The various processes described in the above embodiment may be executed not only in time series according to the order of description, but also in parallel or individually as required by the processing capability of the apparatus that executes the processes or as necessary.

<Supplementary note>
The apparatus of the present invention includes, for example, a single hardware entity as an input unit to which a keyboard or the like can be connected, an output unit to which a liquid crystal display or the like can be connected, and a communication device (for example, a communication cable) capable of communicating outside the hardware entity. Can be connected to a communication unit, a CPU (Central Processing Unit, may include a cache memory or a register), a RAM or ROM that is a memory, an external storage device that is a hard disk, and an input unit, an output unit, or a communication unit thereof , A CPU, a RAM, a ROM, and a bus connected so that data can be exchanged between the external storage devices. If necessary, the hardware entity may be provided with a device (drive) that can read and write a recording medium such as a CD-ROM. A physical entity having such hardware resources includes a general-purpose computer.

  The external storage device of the hardware entity stores a program necessary for realizing the above functions and data necessary for processing the program (not limited to the external storage device, for example, reading a program) It may be stored in a ROM that is a dedicated storage device). Data obtained by the processing of these programs is appropriately stored in a RAM or an external storage device.

  In the hardware entity, each program stored in an external storage device (or ROM or the like) and data necessary for processing each program are read into a memory as necessary, and are interpreted and executed by a CPU as appropriate. . As a result, the CPU realizes a predetermined function (respective component requirements expressed as the above-described unit, unit, etc.).

  The present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the spirit of the present invention. In addition, the processing described in the above embodiment may be executed not only in time series according to the order of description but also in parallel or individually as required by the processing capability of the apparatus that executes the processing. .

  As described above, when the processing functions in the hardware entity (the apparatus of the present invention) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entity should have are described by a program. Then, by executing this program on a computer, the processing functions in the hardware entity are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used. Specifically, for example, as a magnetic recording device, a hard disk device, a flexible disk, a magnetic tape or the like, and as an optical disk, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only). Memory), CD-R (Recordable) / RW (ReWritable), etc., magneto-optical recording medium, MO (Magneto-Optical disc), etc., semiconductor memory, EEP-ROM (Electronically Erasable and Programmable-Read Only Memory), etc. Can be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, a hardware entity is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

Claims (8)

N is an integer of 1 or more, n is an integer 1 ≦ n ≦ N,
Target language speech data _n (1 ≦ n ≦ N), which is the target speaker speech data that is speech data in the target language by the target speaker, and speech data in the target language by the target language input speaker _n , Voice data uttering the same sentence,
Cross-lingual speech synthesis for learning a cross-lingual speech synthesis model for synthesizing speech in the synthesis target language by the target speaker from the target speaker speech data and the learning target language speech data _n (1 ≦ n ≦ N) Model learning device for
The target speaker speech data and adjusts the time information of the learned language audio data _n, the time information adjusted target speaker speech data _n and time information adjusted learned language time to generate audio data _n information adjustment unit When,
From the set (1 ≦ n ≦ N) of the target speaker voice data _n after the time information adjustment and the learning target language voice data _n after the time information adjustment, arbitrary voice data is converted into voice data having the voice quality of the target speaker. A voice quality converter learning unit for learning an unspecified speaker voice quality converter to be converted;
Using the unspecified speaker voice quality converter, the voice to be synthesized after synthesis of speech having the voice quality of the target speaker from the synthesis target language voice data which is voice data in the synthesis target language by the synthesis target language input speaker A voice quality conversion unit for generating data;
A synthesis model learning unit that learns the cross-lingual speech synthesis model from the synthesis target language speech information set that is a set of speech information included in the synthesis target language speech data after the voice quality conversion and the synthesis target language speech data A model learning device for cross-lingual speech synthesis including and. N is an integer of 1 or more, n is an integer 1 ≦ n ≦ N,
Target language speech data _n (1 ≦ n ≦ N), which is the target speaker speech data that is speech data in the target language by the target speaker, and speech data in the target language by the target language input speaker _n , Voice data uttering the same sentence,
Cross-lingual speech synthesis for learning a cross-lingual speech synthesis model for synthesizing speech in the synthesis target language by the target speaker from the target speaker speech data and the learning target language speech data _n (1 ≦ n ≦ N) Model learning device for
A recording unit that records a pronunciation vector creation rule indicating a correspondence relationship between a phoneme of a language to be learned and a phoneme of a language to be synthesized;
The target speaker speech data and adjusts the time information of the learned language audio data _n, the time information adjusted target speaker speech data _n and time information adjusted learned language time to generate audio data _n information adjustment unit When,
Learning, which is a set of pronunciation vectors calculated from the utterance information, from the learning target language utterance information set _n , which is a set of utterance information of utterances included in the learning target language speech data _n , using the pronunciation vector creation rule A first pronunciation vector generation unit for generating a target language pronunciation vector set _n ;
From the set (1 ≦ n ≦ N) of the target speaker speech data _n after the time information adjustment, the learning target language speech data _n after the time information adjustment and the learning target language pronunciation vector set _n (1 ≦ n ≦ N), the target speech data A voice quality converter learning unit for learning an unspecified speaker voice quality converter for converting into voice data having the voice quality of the speaker;
Using the pronunciation vector creation rule, from the synthesis target language utterance information set, which is a set of utterance information of the utterance included in the synthesis target language speech data that is speech data in the synthesis target language by the synthesis target language input speaker, A second pronunciation vector generation unit that generates a synthesis target language pronunciation vector set that is a set of pronunciation vectors calculated from the utterance information;
Using the unspecified speaker voice quality converter, a voice quality conversion unit that generates voice-converted synthesis target language voice data having the voice quality of the target speaker from the synthesis target language voice data and the synthesis target language pronunciation vector set When,
A cross-lingual speech synthesis model learning device comprising: a synthesis model learning unit that learns the cross-lingual speech synthesis model from the synthesis target language speech data after voice quality conversion and the synthesis target language utterance information set. A model learning device for cross-lingual speech synthesis according to claim 1,
M is an integer of 1 or more, m is an integer satisfying 1 ≦ m ≦ M,
The voice quality conversion unit generates post-voice quality synthesis target language voice data _m having the voice quality of the target speaker from synthesis target language voice data _m which is voice data in a synthesis target language by a synthesis target language input speaker _m. And
The synthesis model learning unit includes a synthesis target language utterance information set _m (1 ≦ m ≦ M), which is a set of utterance information included in the synthesis target language speech data _m and the synthesis target language speech data _m. ) To learn the cross-lingual speech synthesis model. A cross-lingual speech synthesis model learning device. A model learning device for cross-lingual speech synthesis according to claim 3,
further,
A similarity _m indicating a degree of similarity between the voice quality converted synthesis target language voice data _m and the target speaker voice data is calculated, and the voice quality converted synthesis target language voice data _m is calculated based on the similarity _m. When it is determined that the speech data is similar to the speaker voice data, the target speech data _m is a set of post-voice quality synthesis target language speech data similar to the target speaker voice data. A speaker similarity determination unit that generates the target speaker similar synthesis target language speech data set by adding to the speaker similar synthesis target language speech data set;
The synthesis model learning unit learns the cross-lingual speech synthesis model from the target speaker similar synthesis target language speech data set and the synthesis target language utterance information _m (1 ≦ m ≦ M). Model learning device for cross-lingual speech synthesis. Synthetic text that is text in a synthesis target language that is a target of speech synthesis using the cross-lingual speech synthesis model learned by the cross-lingual speech synthesis model learning device according to any one of claims 1 to 4. A cross-lingual speech synthesizer for generating synthesized speech of the synthesized text from
A text analysis unit that generates context information of the synthesized text from the synthesized text;
Using the cross-lingual speech synthesis model, a speech parameter generation unit that generates speech parameters from the context information;
A cross-lingual speech synthesizer comprising: a speech waveform generator that generates the synthesized speech from the speech parameters. N is an integer of 1 or more, n is an integer 1 ≦ n ≦ N,
Target language speech data _n (1 ≦ n ≦ N), which is the target speaker speech data that is speech data in the target language by the target speaker, and speech data in the target language by the target language input speaker _n , Voice data uttering the same sentence,
The cross-lingual speech synthesis model learning device synthesizes speech in the synthesis target language by the target speaker from the target speaker speech data and the learning target language speech data _n (1 ≦ n ≦ N). A cross-lingual speech synthesis model learning method for learning a synthesis model,
The cross-lingual speech synthesis model learning device adjusts time information between the target speaker speech data and the learning target language speech data _n, and adjusts the time information adjusted target speaker speech data _n and the time information adjusted learning target. A time information adjustment step for generating language audio data _n ;
The cross-lingual speech synthesis model learning device is configured to convert arbitrary speech data from the set (1 ≦ n ≦ N) of the target speaker speech data _n after the time information adjustment and the target language speech data _n after the time information adjustment. A voice quality converter learning step for learning an unspecified speaker voice quality converter for converting into voice data having the voice quality of the target speaker;
The cross-lingual speech synthesis model learning device uses the unspecified speaker voice quality converter from the synthesis target language speech data which is speech data in a synthesis target language by a synthesis target language input speaker, and the target speaker A voice quality conversion step of generating speech data to be synthesized after voice quality conversion having the voice quality of:
The cross-lingual speech synthesis model learning device uses the cross-lingual speech from a synthesis target language utterance information set, which is a set of utterance information included in the synthesis target language speech data and the synthesis target language speech data after the voice quality conversion. A model learning method for cross-lingual speech synthesis, comprising: a model learning step for synthesis for learning a model for synthesis. N is an integer of 1 or more, n is an integer 1 ≦ n ≦ N,
Target language speech data _n (1 ≦ n ≦ N), which is the target speaker speech data that is speech data in the target language by the target speaker, and speech data in the target language by the target language input speaker _n , Voice data uttering the same sentence,
A model learning device for cross-lingual speech synthesis having a recording unit that records a pronunciation vector creation rule indicating a correspondence relationship between a phoneme of a language to be learned and a phoneme of a language to be synthesized, the target speaker speech data and the learning target language speech data A cross-lingual speech synthesis model learning method for learning a cross-lingual speech synthesis model for synthesizing speech in a synthesis target language by the target speaker from _n (1 ≦ n ≦ N),
The cross-lingual speech synthesis model learning device adjusts time information between the target speaker speech data and the learning target language speech data _n, and adjusts the time information adjusted target speaker speech data _n and the time information adjusted learning target. A time information adjustment step for generating language audio data _n ;
The cross-lingual speech synthesis model learning device uses the pronunciation vector creation rule to generate the utterance information from the learning target language utterance information set _n , which is a set of utterance information of utterances included in the learning target language speech data _n. A first pronunciation vector generation step of generating a learning target language pronunciation vector set _n , which is a set of pronunciation vectors calculated from
The model learning device for cross-lingual speech synthesis includes a set of the target speaker speech data _n after time information adjustment, the learning target language speech data _n after time information adjustment and the learning target language pronunciation vector set _n (1 ≦ n ≦ N), a voice quality converter learning step for learning an unspecified speaker voice quality converter for converting arbitrary voice data into voice data having the voice quality of the target speaker;
The cross-lingual speech synthesis model learning device uses the pronunciation vector creation rules to collect speech utterance information included in synthesis target language speech data that is speech data in a synthesis target language by a synthesis target language input speaker. A second pronunciation vector generation step of generating a synthesis target language pronunciation vector set, which is a set of pronunciation vectors calculated from the utterance information, from the synthesis target language utterance information set of
The model learning device for cross-lingual speech synthesis uses the unspecified speaker voice quality converter, and after voice quality conversion having the voice quality of the target speaker from the synthesis target language voice data and the synthesis target language pronunciation vector set A voice conversion step for generating speech data to be synthesized;
The cross-lingual speech synthesis model learning device includes a synthesis model learning step of learning the cross-lingual speech synthesis model from the synthesis target language speech data after the voice quality conversion and the synthesis target language utterance information set. Model learning method for speech synthesis.   A program for causing a computer to function as the cross-lingual speech synthesis model learning device according to any one of claims 1 to 4 or the cross-lingual speech synthesis device according to claim 5.

Images