JP2018005048A - Voice quality conversion system - Google Patents

Abstract

A voice quality conversion system capable of converting the voice quality of an actor into the voice quality of a target is provided. A voice quality learning device (10) extracts a first feature quantity from a first speech signal of a target (2). The voice quality learning apparatus 10 transforms the fundamental frequency contained in the second audio signal of the actor 1 by a predetermined magnification, and extracts the second feature quantity from the second audio signal whose fundamental frequency has been transformed. Voice quality learning apparatus 10 stores model data obtained by modeling the correspondence relationship between the extracted first feature amount and second feature amount in database DB. The voice conversion device 20 converts the fundamental frequency included in the third audio signal of the actor 1 by a predetermined magnification, and extracts the third feature amount from the third audio signal whose fundamental frequency has been converted. The voice quality conversion device 20 generates a fourth voice signal in which the voice quality of the actor 1 is converted into the voice quality of the target 2 based on the model data stored in the database DB and the extracted third feature amount. [Selection drawing] Fig. 1

Description

  The present invention relates to a voice quality conversion system.

  In recent years, attracting customers has been performed by characters appearing in theme parks or event venues.

  This character includes various characters appearing in, for example, movies, anime, comics, games, and the like. Further, in the music field, a character may act as an artist.

  For example, the character may appear as a costume in the real space or may appear as a video. Such a visual element of the character can sufficiently entertain the audience by improving the accuracy of the costume and improving the technology of computer graphics.

Kazuhiro Kobayashi, Tomoki Toda, Graham Neubig, Sakriani Sakti, Satoshi Nakamura (Nara Institute of Science and Information), “Statistical Voice / Voice Conversion Based on Difference Spectrum Correction”, [online], March 2014, Proc. [Search on June 27, 2016], Internet <URL: http: www.phontron.com/paper/kobayashi14asj.pdf>

  By the way, it is common practice to make the character appear to be speaking by playing the voice recording the voice of the character at the same time or by being played by an actor (voice actor) in the field.

  However, there are cases where the playback timing and recorded content of the recorded voice do not match the actual site, or the voice quality of the actor is different from the voice quality of the character. In this case, the audience may feel uncomfortable.

  For this reason, the voice of a specific character can be seen as the actual voice actor himself acting as an actor, or only by a specific actor who can make a voice similar to the character, but at the theme park or event venue In the case of a character that appears for a long time, a situation may occur in which an actor hurts his throat. In such a situation, since the actor overuses the throat, even if the actor is the same, the voice quality may change between morning and evening, for example.

  Characters appearing in theme parks or event venues are also required to have real-time conversations with the audience on the stage or on the screen, for example. It is difficult.

  Therefore, there is a demand for a technique that allows different actors to speak freely with the voice of the same character (target).

  Therefore, an object of the present invention is to provide a voice quality conversion system capable of converting an actor's voice quality into a target voice quality.

  According to one aspect of the present invention, a voice quality conversion system that includes a voice quality learning device and a voice quality conversion device and converts an actor's voice quality to a target voice quality is provided. The voice quality learning device includes: a first input unit that inputs a first audio signal of the target; a first extraction unit that extracts a first feature amount from the input first audio signal; A second input means for inputting a second audio signal of the actor corresponding to the first audio signal; and a first input for converting a fundamental frequency contained in the input second audio signal at a predetermined magnification. Correspondence between the converting means, the second extracting means for extracting the second feature value from the second audio signal whose fundamental frequency has been converted, and the extracted first feature value and second feature value And a database that stores model data obtained by modeling the relationship. The voice quality conversion device includes: a third input unit that inputs a third audio signal of the actor; and a second conversion that converts a fundamental frequency included in the input third audio signal at the predetermined magnification. Means, third extraction means for extracting a third feature amount from the third audio signal whose fundamental frequency has been converted, model data stored in the database, and the extracted third feature amount. And generating means for generating a fourth voice signal in which the voice quality of the actor is converted to the voice quality of the target, and output means for outputting the generated fourth voice signal.

  The present invention makes it possible to convert an actor's voice quality to a target voice quality.

The figure which shows schematically an example of a structure of the voice quality conversion system which concerns on embodiment of this invention. The block diagram which shows an example of a function structure of a voice quality learning apparatus. The block diagram which shows an example of a function structure of a voice quality conversion apparatus. The flowchart which shows an example of the process sequence of a voice quality learning apparatus. The flowchart which shows an example of the process sequence of a voice quality conversion apparatus.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram schematically showing a configuration of a voice quality conversion system according to the present embodiment. The voice quality conversion system according to the present embodiment includes, for example, a character (hereinafter simply referred to as a target) 2 called a voice quality of a person (hereinafter simply referred to as an actor) 1 called an actor such as a voice actor. Used to convert to voice quality.

  Specifically, the voice quality conversion system converts the voice quality of the actor 1 into the voice quality of the target 2 and outputs the voice when the actor 1 utters, for example, in a theme park or an event venue. Even if the actor 1 has a different voice quality, the voice quality of the target 2 can be used.

  In the description of the present embodiment, the actor 1 is described as being a person. However, the actor 1 only needs to emit sound, for example, it may emit mechanically generated sound. Good. Further, the target 2 (character) may be a person who plays the voice of the character, or may be a device that mechanically emits the voice of the character. The target 2 may be a voice of a person such as a celebrity, an actor, or a singer instead of a character.

  As shown in FIG. 1, the voice quality conversion system includes a voice quality learning device 10 and a voice quality conversion device 20.

  The voice quality learning apparatus 10 includes a personal computer including a processor (computer) such as a CPU that can execute various programs (software). The voice quality learning apparatus 10 includes an analysis engine 10a and a database (DB) 10b.

  The analysis engine 10a executes the learning process using the voice of the target 2 (voice generated by the target 2) and the voice of the actor 1 which is pronounced by imitating the intonation, the sound generation timing, and the pitch of the target 2 voice. (Learn voice conversion model). The analysis engine 10a performs analysis based on the learning process result (learning result) to create the database 10b.

  Although not shown in FIG. 1, the voice quality learning device 10 includes a microphone and the like for inputting the voices of the actor 1 and the target 2 described above. Note that the voice quality learning device 10 may be configured to input a voice file in which voices of the actor 1 and the target 2 are recorded in advance, for example.

  Similar to the voice quality learning device 10, the voice quality conversion device 20 includes various programs (such as a personal computer including a processor (computer) such as a CPU capable of executing software).

  The voice quality conversion device 20 converts the voice quality of the actor 1 into the voice quality of the target 2 using the database 10b created in the voice quality learning device 10 described above. The voice whose voice quality is converted in the voice quality conversion apparatus 20 in this way is output from, for example, a speaker 20 a provided in the voice quality conversion apparatus 20. Note that the voice whose voice quality is converted in the voice quality conversion apparatus 20 may be output as, for example, a voice file and managed in the voice quality conversion apparatus 20, or may be transmitted to an external server apparatus or the like.

  Although not shown in FIG. 1, the voice quality conversion device 20 includes a microphone or the like for inputting the voice of the actor 1. Similarly to the voice quality learning apparatus 10 described above, the voice quality conversion apparatus 20 may be configured to input a voice file in which the voice of the actor 1 is recorded in advance.

  In the voice quality conversion system according to the present embodiment, the voice quality learning device 10 and the voice quality conversion device 20 will be described as separate devices, but the voice quality learning device 10 and the voice quality conversion device 20 are realized as one device. It does not matter.

  Furthermore, the voice quality learning device 10 and the voice quality conversion device 20 may be realized as an electronic device other than a personal computer, such as a smartphone or a tablet terminal. Further, the voice quality learning device 10 and the voice quality conversion device 20 may have a form such as a microphone in which each function described in the present embodiment is stored in a chip or the like, or other forms. It may be realized as a dedicated device.

  The outline of the voice quality conversion method in the voice quality conversion system according to this embodiment will be described below.

  In the voice quality conversion system according to the present embodiment, a technique for converting voice quality based on a mixed normal distribution model (GMM: Gaussian Mixture Model) (hereinafter referred to as voice conversion based on GMM) is adopted. In the voice quality conversion based on this GMM, the learning process is executed by the voice quality learning apparatus 10 described above, and the conversion process is executed by the voice quality conversion apparatus 20.

  First, the learning process will be briefly described. In the learning process, for example, voice signals (voice data) of the actor 1 and the target 2 when the same speech (sentence, etc.) is uttered with the same intonation and pitch are prepared.

  The voice quality learning device 10 inputs the voice signals of the actor 1 and the target 2 (that is, voice signals of the same content uttered by the actor 1 and the target 2).

  The voice quality learning device 10 divides both audio signals into frames and executes short-time analysis processing. Normally, both speech signals are divided into short-time speech waveforms by shifting the analysis interval by a fixed length (for example, 5 ms).

  The voice quality learning device 10 performs analysis (spectrum analysis) of the feature amount representing the feature of the voice for each divided frame, expands and contracts the local time frame sequence, and synchronizes the time frames, thereby obtaining both voice frames. Match between. Data obtained by combining the spectra for the corresponding frames is calculated sequentially, and the combined probability density function is modeled by GMM.

  In the present embodiment, model data (voice quality conversion model data) obtained by such learning processing is accumulated in the database 10b.

  That is, in the learning process described above, a conversion rule for converting the voice quality of the actor 1 to the voice quality of the target 2 from the pair of voice (waveform) feature quantities of the actor 1 and the target 2 associated with time. Are modeled statistically.

  Next, the conversion process will be briefly described. In the above-described conversion processing of voice quality conversion based on GMM, a basic speech is generally generated using a speech synthesis technique. However, in the conversion processing according to the present embodiment, relatively natural speech is output instead of mechanical speech. Therefore, the voice (waveform) of the actor 1 is used as it is.

  That is, in this conversion process, the feature amount of the voice is not converted, but the difference in the feature amount between the voice of the actor 1 and the voice of the target 2 to be output is estimated based on the GMM, and the voice of the actor 1 ( The voice quality of the actor 1 is converted to the voice quality of the target 2 by convolving the waveform with a synthesis filter (that is, applying a difference spectrum correction).

  As described above, in voice quality conversion based on the GMM, the voice quality of the actor 1 is converted into the voice quality of the target 2 by the conversion processing using the database 10b (model data accumulated therein) created by the learning processing. Can do.

  For voice quality conversion based on GMM adopted in this embodiment, for example, “Kazuhiro Kobayashi, Tomoki Toda, Graham Neubig, Sakriani Sakti, Satoshi Nakamura (NAIST, Information),“ Statistical singing voice quality based on differential spectrum correction ” "Conversion", [online], March 2014, Proceedings of the Acoustical Society of Japan, [Search June 27, 2016], Internet <URL: http: www.phontron.com/paper/kobayashi14asj.pdf> "Tomochi Toda," Speech and acoustic signal processing-Voice conversion using statistical methods ", [online], January 20, 2014, [Search June 27, 2016], Internet <http: //hil.tu -tokyo.ac.jp/~kameoka/SAP/SAP13_11.pdf> ”, etc., and detailed description thereof is omitted here.

  By the way, for example, when the genders of the actor 1 and the target 2 are different, the sound ranges that can be uttered by the actor 1 and the target 2 are different.

  In this way, when the sound range is different between the actor 1 and the target 2, even if the voice quality of the actor 1 is converted to the voice quality of the target 2 by voice quality conversion based on the GMM, the output voice cannot be recognized as the voice of the target 2 It will be about.

  For this reason, in the conversion process described above, it is necessary to execute a process for converting the fundamental frequency (f0) of the voice of the actor 1 in accordance with the range of the target 2 (speech) (hereinafter referred to as fundamental frequency conversion). The fundamental frequency is one of the feature quantities of voice that expresses the pitch (pitch) of the voice. In addition, since the voice of the actor 1 is used as it is in the conversion process, in the present embodiment, basic frequency conversion is required in which the voice (waveform) of the actor 1 is processed by signal processing.

  Here, as a fundamental frequency conversion technique for processing the above-described speech waveform by signal processing, for example, there is a relatively simple technique using time stretching and resampling such as W-SOLA. Basic frequency conversion by a relatively simple method of processing such a speech waveform on the time axis does not require, for example, speech fundamental frequency estimation processing, does not require speech analysis / synthesis processing by a vocoder, and has a CPU load. There is an advantage of less. Note that other methods may be used as the fundamental frequency conversion.

  However, when such fundamental frequency conversion is executed, the voice spectrum of the actor 1 changes because the voice spectrum (formant) of the actor 1 expands and contracts. When such a fundamental frequency conversion is executed in the preceding stage of the conversion process in the voice quality conversion apparatus 20, the formant of the voice of the actor 1 given during the learning process in the voice quality learning apparatus 10 and the conversion process Therefore, it is difficult to appropriately convert the voice quality of the actor 1 to the voice quality of the target 2 in the conversion process.

  On the other hand, when the fundamental frequency conversion is performed after the conversion process in the voice quality conversion device 20 (that is, the basic frequency of the voice is converted after the voice quality is converted by the conversion process), it is already obtained by the conversion process. The voice quality of the target 2 is changed to a different one by the expansion / contraction of the formant caused by the fundamental frequency conversion. In this case, the voice of the voice quality of the target 2 cannot be output.

  Here, as described above, the fundamental frequency conversion causes the expansion and contraction of the formant (spectrum) of the speech. However, if the fundamental frequency is always converted at a constant magnification, the voice quality will change, but as with the formant. It stabilizes by expanding and contracting at a certain magnification (that is, it is possible to obtain a formant voice quality with a stable personality).

  In this embodiment, paying attention to such characteristics, the fundamental frequency conversion is performed on the voice of the actor 1 before the learning process and the conversion process in the voice quality conversion system. In this case, the fundamental frequency conversion is executed at a predetermined magnification (pitch conversion magnification).

  Hereinafter, functional configurations of the voice quality learning device 10 and the voice quality conversion device 20 included in the voice quality conversion system according to the present embodiment will be described.

  FIG. 2 is a block diagram illustrating a functional configuration of the voice quality learning device 10. The voice quality learning device 10 learns the voice (voice quality) of the target 2 and the voice (voice quality) of the actor 1 in order to convert the voice quality of the actor 1 into the voice quality of the target 2 in the voice quality conversion device 20 as described above. It has a function.

  As shown in FIG. 2, the voice quality learning device 10 includes a first voice input unit 11, a first analysis processing unit 12, a magnification determination unit 13, a second voice input unit 14, a fundamental frequency conversion unit 15, and a second analysis processing unit. 16 and a model learning unit 17.

  In the present embodiment, the first speech input unit 11, the first analysis processing unit 12, the magnification determination unit 13, the second speech input unit 14, the fundamental frequency conversion unit 15, the second analysis processing unit 16, and the model learning unit 17 are: It is a functional unit constituting the analysis engine 10a shown in FIG. 1, and is realized by causing a computer such as a CPU provided in the voice quality learning apparatus 10 to execute a program, that is, by software. Part or all of these units 11 to 17 may be realized by hardware such as an IC (Integrated Circuit), or may be realized as a combined configuration of software and hardware. The program to be executed by the computer may be stored and distributed in a computer-readable storage medium, or may be downloaded to the voice quality learning device 10 through a network.

  Here, the voice of the target 2 uttered to the voice quality learning device 10 for the above-described learning process is converted into an analog electric signal via a microphone, for example. The sound converted into the analog electric signal through the microphone is further converted into a digital electric signal through the A / D converter, and input by the first sound input unit 11. Hereinafter, the voice (signal) input by the first voice input unit 11 is referred to as a learning voice signal of the target 2 for convenience. The first voice input unit 11 may input the above-described voice file as a learning voice signal for the target 2.

  In addition, the learning speech signal of the target 2 includes, as parameters (features) representing the features of the speech of the target 2, for example, a spectral feature amount (spectrum envelope) representing phonological characteristics and voice quality, and a voice pitch ( Pitch) and a basic frequency and a non-periodic component expressing voice blur.

  The first analysis processing unit 12 analyzes the learning speech signal of the target 2 (the spectral feature amount, the fundamental frequency, and the non-periodic component included therein), and the spectral feature amount (the first feature amount from the learning speech signal of the target 2). (Feature amount) is extracted.

  The magnification determination unit 13 determines a constant magnification (f0rate) when executing the above-described fundamental frequency conversion. Specifically, the magnification determination unit 13 determines the magnification based on, for example, the average value of the frequency band (ie, sound range) of the voice of the actor 1 and the average value of the frequency band (ie, sound range) of the sound of the target 2. . Hereinafter, the magnification determined by the magnification determination unit 13 is referred to as a fixed magnification for convenience. This fixed magnification is held in the voice quality learning device 10. The fixed magnification is also held in the voice quality conversion apparatus 20 as described later by being transmitted to the voice quality conversion apparatus 20, for example.

  Here, the voice of the actor 1 uttered to the voice quality learning device 10 for the learning process is converted into an electric signal (voice signal) via a microphone, for example. The second audio input unit 14 inputs an audio signal (second audio signal) converted through a microphone. Hereinafter, the audio signal input by the second audio input unit 14 is referred to as a learning audio signal of the actor 1 for convenience.

  Note that the learning speech signal of the actor 1 includes, as parameters (features) representing the characteristics of the speech of the actor 1, for example, a spectral feature amount (spectrum envelope) representing phonological characteristics and voice quality, and a voice pitch ( Pitch) and a basic frequency and a non-periodic component expressing voice blur.

  The fundamental frequency conversion unit 15 converts the fundamental frequency included in the learning speech signal of the actor 1 at a fixed magnification. In other words, in the present embodiment, the fundamental frequency conversion unit 15 performs fundamental frequency conversion on the learning speech signal of the actor 1 in the previous stage of the learning process.

  The second analysis processing unit 16 analyzes the learning speech signal of the actor 1 (the spectrum feature amount, the fundamental frequency, and the non-periodic component included) after the fundamental frequency is converted at a fixed magnification, and learns the actor 1 A spectral feature amount (second feature amount) is extracted from the audio signal.

  The model learning unit 17 is a functional unit that executes the learning process described above. The model learning unit 17 extracts the spectrum feature amount extracted by the first analysis processing unit 12 (that is, the feature amount of the target 2 speech) and the spectrum feature amount extracted by the second analysis processing unit 16 (that is, the actor 1 Conversion rules for pairs of speech features are statistically modeled. The model learning unit 17 stores (accumulates) the model data obtained by such learning processing in the database 10b.

  FIG. 3 is a block diagram showing a functional configuration of the voice quality conversion device 20. The voice quality conversion device 20 has a function of converting the voice quality of the actor 1 into the voice quality of the target 2.

  As shown in FIG. 3, the voice quality conversion device 20 includes a conversion table 21, a voice input unit 22, a fundamental frequency conversion unit 23, an analysis processing unit 24, a difference estimation unit 25, a voice quality conversion unit 26, and a voice output unit 27.

  In the present embodiment, the conversion table 21 holds the model data generated by installing the database 10b of the voice quality learning device 10 described above and accumulated in the database 10b. The conversion table 21 is stored in, for example, a storage device provided in the voice quality conversion device 20.

  In the present embodiment, the voice input unit 22, the fundamental frequency conversion unit 23, the analysis processing unit 24, the difference estimation unit 25, the voice quality conversion unit 26, and the voice output unit 27 are, for example, a CPU or the like provided in the voice quality conversion device 10. It is assumed that the program is executed by a computer, that is, realized by software. Note that some or all of these units 22 to 27 may be realized by hardware such as an IC (Integrated Circuit) or may be realized as a combined configuration of software and hardware. The program to be executed by the computer may be stored and distributed in a computer-readable storage medium, or may be downloaded to the voice quality conversion device 20 through a network.

  The voice of the actor 1 uttered to the voice quality conversion device 20 for the conversion process is converted into an analog electric signal via a microphone, for example. The sound converted into the analog electric signal through the microphone is further converted into a digital electric signal through the A / D converter and input by the sound input unit 22. Hereinafter, the voice (signal) input by the voice input unit 22 will be referred to as a conversion voice signal of the actor 1 for convenience. Note that the voice input unit 22 may input the above-described voice file as the conversion voice signal of the actor 1.

  As described above, the conversion voice signal of the actor 1 includes a spectral feature quantity, a fundamental frequency, an aperiodic component, and the like as parameters (feature quantities) representing the voice characteristics of the actor 1.

  Here, the fixed magnification (that is, the magnification determined by the magnification determination unit 13) held in the voice quality learning apparatus 10 is also held in the voice quality conversion apparatus 20 as described above.

  The fundamental frequency conversion unit 23 converts the fundamental frequency included in the voice signal for conversion of the actor 1 at a fixed magnification held in the voice quality conversion device 20. That is, in the present embodiment, the fundamental frequency conversion is performed on the conversion audio signal of the actor 1 by the basic frequency conversion unit 23 in the previous stage of the conversion process.

  The analysis processing unit 24 analyzes the conversion voice signal of the actor 1 (the spectrum feature amount, the basic frequency, and the non-periodic component included therein) after the fundamental frequency is converted at a fixed magnification, and the conversion voice of the actor 1 is analyzed. A spectral feature amount (third feature amount) is extracted from the signal.

  The difference estimation unit 25 and the voice quality conversion unit 26 are functional units that execute the conversion processing described above.

  Here, the difference estimation unit 25 and the voice quality conversion unit 26 convert the voice quality of the actor 1 to the target 2 by the conversion process based on the model data held in the conversion table 21 and the spectrum feature amount extracted by the analysis processing unit 24. A voice signal (fourth voice signal) converted into voice quality is generated. The audio signal generated in this way corresponds to the audio signal of the target 2 corresponding to the conversion audio signal of the actor 1.

  Specifically, the difference estimation unit 25 refers to the conversion table 21 (that is, model data), and the spectrum feature amount extracted by the analysis processing unit 24 (that is, the spectrum included in the conversion voice signal of the actor 1). The difference between the feature amount) and the spectral feature amount of the audio signal of the target 2 corresponding to the conversion audio signal of the actor 1 (hereinafter referred to as difference feature amount) is estimated.

  The voice quality conversion unit 26 executes a process (filtering process) for applying the difference feature amount estimated by the difference estimation unit 25 to the conversion voice signal (voice waveform) of the actor 1. Thereby, the voice quality of the actor 1 can be converted into the voice quality of the target 2 in the voice signal for conversion of the actor 1 input by the voice input unit 22.

  The audio output unit 27 outputs the audio signal whose voice quality has been converted by the voice quality conversion unit 26 via, for example, the speaker 20a. Note that the audio signal whose properties are converted by the voice quality conversion unit 26 may be output as an audio file as described above.

  Hereinafter, the operation of the voice quality conversion system (voice quality learning apparatus 10 and voice quality conversion apparatus 20) according to the present embodiment will be described.

  First, the processing procedure of the voice quality learning device 10 will be described with reference to the flowchart of FIG.

  When the process shown in FIG. 4 is executed, for example, a text including a phoneme (phrase) often spoken by the target 2 (specific character) is prepared.

  The target 2 (the person who plays the voice) creates an image of the voice of the target 2 and utters based on the prepared text in consideration of intonation, change in pitch, and the like based on the image. The text prepared here includes, for example, about 50 to 100 sentences (such as words).

  Thereby, the 1st audio | voice input part 11 inputs the audio | voice signal for learning of the said target 2 (namely, specific character) according to the utterance of the target 2 (step S1).

  The first analysis processing unit 12 extracts a spectrum feature amount from the learning speech signal of the target 2 input in step S1 (step S2).

  Next, the actor 1 utters based on the text by imitating the same intonation and pitch change as the utterance by the target 2 based on the text.

  As a result, the second voice input unit 14 performs the learning voice signal of the actor 1 corresponding to the voice of the actor 1 (that is, the learning voice of the actor 1 corresponding to the learning voice signal of the target 2 input in step S1). Audio signal) is input (step S3).

  Here, as described above, the voice quality learning device 10 holds the magnification (fixed magnification) for the fundamental frequency conversion determined by the magnification determination unit 13 as a pre-process. The fixed magnification is determined based on the average value of the frequency band of the voice of the actor 1 and the average value of the frequency band of the voice of the target 2 as described above. Specifically, for example, when the average value of the frequency band of the voice of the actor 1 is 100 Hz and the average value of the frequency band of the voice of the target 2 is 130 Hz, the fixed magnification is 1.3 (130/100). It is. Here, the fixed magnification is described as “the average value of the frequency band of the voice of the target 2 / the average value of the frequency band of the voice of the actor 1”, but the fixed magnification is determined by another method. There may be. In addition, the average value of the frequency band of the voice of the actor 1 and the average value of the frequency band of the voice of the target 2 may be measured in advance.

  Based on the fixed magnification described above, the fundamental frequency conversion unit 15 performs fundamental frequency conversion on the learning speech signal of the actor 1 input in step S3 (step S4). As a result, the fundamental frequency included in the learning audio signal of the actor 1 is converted at a fixed magnification.

  The second analysis processing unit 16 extracts a spectrum feature amount from the learning speech signal of the actor 1 after the fundamental frequency is converted in step S4 (step S5).

  The model learning unit 17 executes the above-described learning process, and the spectral feature amount extracted in step S2 (the spectral feature amount of the target 2 speech) and the spectral feature amount extracted in step S5 (the speech spectrum of the actor 1). A correspondence relationship between the feature quantities) is modeled (step S6). Specifically, as described above, the model learning unit 17 statistically models the conversion rule based on the spectral feature amount associated with each time (frame) (modeling with GMM) to obtain the model data. Can be obtained.

  The model data obtained by the model learning unit 17 is accumulated in the database 10b (step S7).

  According to the process shown in FIG. 4 described above, the learning process is performed after the fundamental frequency conversion based on the fixed magnification is performed on the learning speech signal of the actor 1 as described above, and the model obtained in the learning process is obtained. Data is accumulated in the database 10b.

  Next, the processing procedure of the voice quality conversion apparatus 20 will be described with reference to the flowchart of FIG.

  In the present embodiment, the voice quality conversion device 20 causes the actor 1 wearing a costume to play a voice of a specific character (target 2) and talk (interact) with the audience in real time, for example, at a theme park or event venue. Used when Note that the voice quality conversion device 20 may be used when the actor 1 plays the voice of a specific character projected as an image.

  Note that the model data obtained by modeling the correspondence between the spectral feature quantity of the voice of the actor 1 and the spectral feature quantity of the voice of the target 2 (specific character) is obtained by the process shown in FIG. It is assumed that the voice quality learning device 10 (database 10b) accumulates the data by executing the command. It is assumed that this model data is installed in the voice quality conversion device 20 and held in the conversion table 21.

  When the actor 1 uses the voice quality conversion device 20, the actor 1 utters with the intonation of the target 2 and the change in the pitch, etc., which are imitated when the process shown in FIG. 4 is executed (for example, the audience) Talk to).

  In this case, the voice input unit 22 inputs the conversion voice signal of the actor 1 according to the utterance of the actor 1 (step S11).

  Here, as described above, in the voice quality conversion apparatus 10, the same fixed magnification as the fixed magnification (magnification determined by the magnification determination unit 13) held in the voice quality learning apparatus 10 is held.

  The fundamental frequency conversion unit 15 performs fundamental frequency conversion on the conversion audio signal of the actor 1 input in step S11 based on the fixed magnification held in the voice quality conversion apparatus 10 (step S12). As a result, the fundamental frequency included in the conversion audio signal of the actor 1 is converted at a fixed magnification.

  The analysis processing unit 24 extracts a spectrum feature amount from the conversion voice signal of the actor 1 after the fundamental frequency is converted in step S12 (step S13).

  Hereinafter, the difference estimation unit 25 and the voice quality conversion unit 26 execute the conversion process described above. Specifically, the difference estimation unit 25 sets the spectral feature value (the spectral feature value of the voice of the actor 1) extracted in step S13 and the target 2 corresponding to the voice signal for conversion of the actor 1 input in step S11. A difference feature amount from the audio signal is estimated based on the model data (GMM) held in the conversion table 21 (step S14). In the estimation process in step S14, for example, variable conversion is performed on the GMM to derive a GMM that models the joint probability density of the spectrum feature quantity (vector) and the difference feature quantity (vector) of the voice of the actor 1 The difference feature amount is estimated based on the GMM thus derived.

  Next, the voice quality conversion unit 26 convolves (synthesizes) the differential feature quantity estimated in step S14 with the synthesis filter for the spectral feature quantity extracted in step S13, so that the voice quality of the actor 1 is the target 2 The voice signal converted into the voice quality is generated (step S15). As the synthesis filter, for example, an MLSA (Mel-Log Spectrum Approximation) filter used for speech synthesis can be used.

  The voice signal whose voice quality has been converted by the voice quality conversion unit 26 is output by the voice output unit 27 (step S16).

  According to the process shown in FIG. 5 described above, the conversion process is executed after the fundamental frequency conversion based on the fixed magnification is performed on the conversion audio signal of the actor 1, and the voice quality of the actor 1 is set to the target 2 by the conversion process. It is possible to output a voice signal converted in real time to the voice quality of the voice. Note that the process shown in FIG. 5 is executed each time an audio signal of the actor 1 is input. Specifically, the voice quality conversion in real time can be realized by processing the voice signal of the actor 1 that is continuously input for each fixed length of, for example, about 5 ms.

  As described above, in the present embodiment, the voice quality learning device 10 includes the learning audio signal (first audio signal) of the target 2 and the learning audio signal (first audio signal of the actor 1 corresponding to the learning audio signal of the target 2. 2 audio signals). The voice quality learning device 10 converts the fundamental frequency included in the learning speech signal of the actor 1 at a predetermined magnification (fixed magnification) before the learning process. Further, the voice quality learning device 10 is extracted from the learning speech signal of the actor 1 obtained by converting the spectral feature amount (first feature amount) and the fundamental frequency extracted from the learning speech signal of the target 2 as the learning process. The model data (voice quality conversion model data) obtained by modeling the correspondence relationship between the spectral feature amounts (second feature amounts) is stored (accumulated) in the database 10b.

  On the other hand, the voice quality conversion device 20 receives the conversion audio signal (third audio signal) of the actor 1 and sets the basic frequency included in the conversion audio signal of the actor 1 to the predetermined frequency described above before the conversion process. Convert with magnification (fixed magnification). The voice quality conversion device 20 performs conversion processing based on the model data stored in the database 10b and the spectral feature value (third feature value) extracted from the conversion voice signal of the actor 1 whose fundamental frequency is converted. A voice signal (fourth voice signal) in which the voice quality of the actor 1 is converted into the target voice quality is generated.

  In the conversion process by the voice quality conversion device 20, the spectral feature value of the target 2 based on the model feature data stored in the database 10b and the spectral feature value extracted from the conversion speech signal of the actor 1 whose fundamental frequency has been converted. And the difference feature quantity is applied to the spectrum feature quantity as a filter, so that the voice quality of the actor 1 is converted to the voice quality of the target 2.

  Here, in the present embodiment, in order to reduce the influence due to the difference in pitch between the actor 1 and the target 2, the fundamental frequency conversion is executed before both the learning process and the conversion process. That is, in this embodiment, since the learning process is executed with the voice (signal) of the actor 1 after the fundamental frequency conversion, the fundamental frequency conversion is executed with respect to the voice signal for conversion of the actor 1 before the conversion process. Even in this case, it is possible to appropriately convert the voice quality based on the model data obtained by the learning process.

  In this embodiment, the voice quality of the actor 1 is converted into the voice quality of the target 2 according to the voice signal input based on the voice of the actor 1 at a theme park or an event venue, for example. Since an audio signal can be output in real time, for example, the actor 1 can easily utter (that is, utter) a voice of a specific character (target 2).

  In addition, since the basic frequency conversion that is relatively simple can be used in the present embodiment, a voice quality conversion system can be realized even with low-performance electronic devices (voice quality learning device 10 and voice quality conversion device 20). it can.

  In the present embodiment, since the magnification in the fundamental frequency conversion executed in the preceding stage of both the learning process and the conversion process only needs to be fixed, the magnification may be changed as appropriate. Also, for example, during the learning process, the basic frequency of the audio signal of the target 2 and the actor 1 is always measured to determine the dynamic magnification. In the conversion process, the basic frequency of the input audio signal of the actor 1 is determined. Alternatively, the fundamental frequency may be converted at a different magnification.

  In the present embodiment, for convenience of explanation, the actor 1 and the target 2 have been described as having a one-to-one relationship. However, the voice (signal) of each of the plurality of actors 1 and the voice of the target 2 are described. By accumulating model data (that is, model data for each actor 1) that models the correspondence between feature quantities with (signal), each of the plurality of actors 1 utters with the same character voice quality. It becomes possible to do. According to this, it is possible to reduce the physical burden on each actor 1 by facilitating the replacement of the actor 1 who plays the voice of a specific character, and to improve the similarity of voice quality among a plurality of actors 1 Can be realized. When model data for each actor 1 is accumulated, the fixed magnification described above is determined for each actor 1.

  In addition, model data (that is, model data for each target 2) that models the correspondence relationship between the voice (signal) of the actor 1 and the voice (signal) of each of the plurality of targets 2 is stored. Thus, the actor 1 can select a desired target 2 and output a voice signal converted into the voice quality of the selected target 2.

  Hereinafter, examples of usage modes of the voice quality conversion system according to the present embodiment will be described. In this embodiment, according to the utterance of the actor 1, the voice signal of the voice quality of the target 2 having a different sound range from the actor 1 can be output. For this reason, for example, a female actor 1 can have a conversation with the voice quality of a male target 2. Further, in the present embodiment, since the difference in the pitch of the individual voice can be compensated, the actor 1 can output a voice in a range that cannot normally be uttered. You can sing with the voice quality of the singer and solve the problem of the range.

  Further, by storing the voice of a specific character (target 2) in advance, for example, after the person (voice actor) who played the voice of the specific character has died, other persons (actor 1) It is also possible to obtain a configuration in which model data obtained by modeling the correspondence between feature amounts between the voice and the stored voice of the specific character is obtained. According to such a configuration, even after the person who was playing the voice of a specific character died, an animated movie in which the character appears in response to the voice of another person (Actor 1) is produced. It becomes possible. That is, the voice quality conversion system according to the present embodiment can be applied to a field different from conventional speech synthesis, such as eliminating the time limit of production of an animated movie.

  In addition, for example, a voice actor or the like (target 2) may use the voice quality changing system according to the present embodiment as insurance in preparation for a future voice quality change. In other words, when the voice of a person such as a voice actor is saved in advance and the voice quality actually changes due to illness, injury or aging, etc., the feature amount of the current voice and the accumulated past voice It is also possible to obtain a configuration in which model data obtained by modeling the correspondence between the two is obtained. According to such a configuration, even after the voice quality has changed, it is possible to say a speech or have a conversation with a past voice (voice quality) such as when young. In this case, for example, voice can be stored as insurance for free or at a low price, and a service that pays a fee when actually using the voice quality change system can be provided. In recent years, devices have been developed that are capable of producing artificial voice even after they cannot speak. However, it is possible to have conversations with past voices. In addition, there may be a problem that the active voice actor will lose the opportunity to appear in this system, but rather the voice actor's voice will be used at every place in the world at the same time. It is possible to provide the service without the need and rather the opportunity to use it. Therefore, it is possible to provide a business model in which the usage fee is returned to the voice actor in the form of a royalty or the like according to the use.

  In addition, when the voice quality conversion system is used, if the actor 1 is replaced by a third party who has not registered the model data, a warning or the like is issued in advance, or the actor 1 cannot be used. The voice print registration is performed, and when voice is input to the voice quality conversion device 20, voice print authentication is performed. If the voice print does not match the voice print of the actor 1, an error is not displayed or conversion is not executed. I can do it.

  Similarly, by storing model data that models the correspondence between feature amounts of current voices and past voices of characters (eg, actors) of movies or dramas, for example, the movie or drama In a reminiscence scene such as a drama, the voice quality changing system may be used for such purposes as speaking the speech with the voice quality (that is, past voice) of the person at the time drawn in the recollection scene.

  Furthermore, for example, when dubbing in Japanese is performed in an overseas movie or drama, it is possible to speak Japanese words with the voice quality of an actor who actually appears in the movie or drama.

  As described above, since the voice quality conversion system according to the present embodiment is less influenced by language, it is possible to convert voice quality even if it is a special language or the like generated by a character that has no linguistic meaning, for example. . In addition, the voice quality conversion system according to the present embodiment can be used for various purposes by taking advantage of real-time characteristics. For example, the voice quality conversion system can also be used for the above-mentioned karaoke, and converts the voice quality of voice synthesis guidance by a device. It is also possible to use it for the purpose of outputting such voice.

  In the voice quality conversion system according to the present embodiment, the voice quality conversion based on GMM has been mainly described. However, as described above, the difference in the feature amount between the voice of the actor 1 and the voice of the target 2 is calculated. For example, voice quality conversion based on deep learning in which modeling is performed using a neural network instead of GMM is a method for converting the voice quality of actor 1 into the voice quality of target 2 by synthesizing the voice of actor 1 (applying differential spectrum correction). It can also be applied to. For this reason, in the voice quality conversion system according to the present embodiment, such voice quality conversion based on deep learning may be employed, or learning may be performed by inputting the voice of the actor 1 and the voice of the target 2. For example, other voice quality conversion (method) may be adopted.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

  DESCRIPTION OF SYMBOLS 10 ... Voice quality learning apparatus, 10a ... Analysis engine, 10b ... Database, 11 ... 1st audio | voice input part (1st input means), 12 ... 1st analysis process part (1st extraction means), 13 ... Magnification determination part , 14 ... second voice input unit (second input unit), 15 ... fundamental frequency conversion unit (first conversion unit), 16 ... second analysis processing unit (second extraction unit), 17 ... model learning unit 20 ... voice quality conversion device, 20a ... speaker, 21 ... conversion table, 22 ... speech input unit (third input unit), 23 ... basic frequency conversion unit (second conversion unit), 24 ... analysis processing unit (first) 3 extraction means), 25 ... difference estimation unit, 26 ... voice quality conversion unit, 27 ... voice output unit.

Claims (1)

In a voice quality conversion system that includes a voice quality learning device and a voice quality conversion device, and converts an actor's voice quality to a target voice quality,
The voice quality learning device includes:
First input means for inputting a first audio signal of the target;
First extraction means for extracting a first feature quantity from the input first audio signal;
Second input means for inputting a second audio signal of the actor corresponding to the first audio signal;
First conversion means for converting a fundamental frequency included in the input second audio signal at a predetermined magnification;
Second extraction means for extracting a second feature quantity from the second audio signal converted from the fundamental frequency;
A database that stores model data obtained by modeling the correspondence between the extracted first feature quantity and second feature quantity, and
The voice quality conversion device includes:
Third input means for inputting a third audio signal of the actor;
Second conversion means for converting a fundamental frequency included in the input third audio signal at the predetermined magnification;
Third extraction means for extracting a third feature amount from the third audio signal converted from the fundamental frequency;
Generating means for generating a fourth audio signal in which the voice quality of the actor is converted to the voice quality of the target based on the model data stored in the database and the extracted third feature quantity;
An output means for outputting the generated fourth voice signal. A voice quality conversion system comprising:

Images