JP2009162879A - Speech support method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a format close to a speaker's original voice in an environment where it is difficult to communicate with clear voice, for example, utterance support for an esophageal utterance method of a person with speech dysfunction or communication assistance in a high noise environment It is an object to provide a method that reproduces the voice of the voice and facilitates communication.
A transfer function is generated for each subword based on an input signal such as a body conduction sound of a speaker and voice data as a model of output speech, and the input signal is continuously subworded when the speaker utters. Each sound quality is converted with a transfer function, the subword after sound quality conversion is convolved with a digital filter, and continuous output speech is reproduced, thereby realizing speech support with speech quality close to the output speech. .
[Selection] Figure 2

Description

  The present invention relates to an utterance support method in a situation where communication through normal utterance is difficult.

  In recent years, there has been an increase in the number of patients who have had their entire pharynx removed due to diseases such as pharyngeal cancer. Total pharyngectomy is one of the common treatments for pharyngeal cancer, but vocal cords are also lost by total pharyngectomy, making normal speech impossible.

  Many patients who have removed vocal cords use the esophageal voicing method as an alternative voicing method, but the sound produced by the esophageal voicing method is not always clear. There is a problem with communication in high noise environments.

  The problem of voice communication in a high noise environment is not limited to the vocal cord extractor. The voice recognition rate is significantly reduced.

  In the past, the inventor of the present application has conducted research on a speech recognition apparatus using bone conduction, that is, body conduction sound, using a bone conduction system in Non-Patent Document 1. In addition, in the non-patent document 2, the inventor is conducting research on a technique for converting a vocal cord vibration signal into a voice signal.

In addition, as an example of an utterance support method for a total pharyngectomy person using a body conduction sound, devices and methods as disclosed in Non-Patent Document 3, Patent Document 1, and Patent Document 2, for example, are disclosed. Non-Patent Document 3 is a paper on a method for supporting voice communication by extracting a non-audible murmur (NAM) and converting it into a whisper. Japanese Patent Application Laid-Open No. 2004-228561 is a voice acting device characterized by synthesizing a voice by detecting the movement of a muscle around the mouth using a myoelectric potential. Patent Document 2 registers the target utterance model in text, matches the content of the speaker's utterance with the registered target utterance model, outputs if there is a match, and utterance characteristics if there is no matching target utterance model This is a conversation support apparatus characterized in that it generates and outputs from a plurality of target utterance models that coincide with each other, and at the same time registers as a new target utterance model.
JP-A-7-181888 JP 2004-287209 A "Study of bone conduction recognition system for marine engine operation support" Journal of Japan Marine Engineering Society Vol.39 No.4, P35-40 “Speech generation from vocal cord vibration signals using vocal tract filter characteristics” Proceedings of the Annual Meeting of the Japan Society of Mechanical Engineers, 2003, P121-122 "Voice communication support system for total pharyngectomy based on the conversion of meat conduction artificial speech" IEICE Transactions D Vol. J90-D No. 3, P780-787

  However, none of the inventions described in the above-mentioned literatures have been able to generate clear speech while maintaining the voice quality of the speaker, and are not practical.

  In Non-Patent Document 1, which is a prior study of the inventor of the present application, extraction of bone conduction (in-body conduction) has been successful, but it has not yet reached the stage of conversion into clear speech. In Non-Patent Document 2, it is necessary to simultaneously input a vocal cord signal and a voice signal, which is invalid in a situation where a normal voice signal cannot be used. Further, in Non-Patent Document 2, signal conversion is confirmed by a word registered in advance (in this document, “huchu”), and is not taken into account in unregistered words and conversations.

  Non-Patent Document 3 is an invention that performs conversion from “non-audible murmur (NAM)” to whispering voice. However, whispering voice is different from actual speech sound and does not consider the fundamental frequency, so it is male voice. There is no distinction between female voices and it cannot be said that the voice quality of the speaker is left. Further, the present invention only converts the input signal by NAM as it is, and since no examination has been made regarding a method for recognizing and analyzing the input signal, it is vulnerable to ambiguity.

  Patent Document 1 is an invention that synthesizes and outputs speech based on the myoelectric potential around the mouth, but the voice quality is not determined only by the movement of the muscle, and cannot be said to reproduce the voice quality.

  Further, Patent Document 2 is a device that holds a target utterance model as a database in advance, collates the content of a speaker's utterance with the target utterance model on the database, and converts and outputs it. The unit. Even if new content that is not registered as the target utterance model is uttered, it is possible to respond by automatic synthesis from the registered utterance model, and a database corresponding to the new utterance is automatically constructed to supplement the data However, if there is no utterance model similar to a new utterance in the database, automatic synthesis cannot be performed. In addition, given the infinite number of utterances and words, it must be said that the feasibility is poor.

  The present invention uses a speaker's body conduction sound or voice as input signal information, and creates a transfer function that associates the input signal information with the output voice signal model in advance using a cross spectrum method for each subword shorter than a word. And storing and holding, a step of inputting a conduction sound or voice of the speaker as an input signal, a step of identifying the input signal for each subword, and the transfer function corresponding to each subword The method includes a step of converting the sound quality of the subword, a step of synthesizing the subword subjected to the sound quality conversion as a series of output sound signals, and a step of outputting the series of output sound signals. Therefore, according to the present invention, by generating in advance a transfer function that associates a speaker's internal conduction sound or speech with the speech model to be output, it is close to the speech model to be output prepared in advance when the speaker actually speaks. It can be output as audio.

  In the present invention, the subword is a phoneme, semi-syllable or syllable. Therefore, in the present invention, pre-registration of words or the like is not necessary, and it is possible to deal with free utterances.

  In the present invention, the transfer function is a function for converting the input signal into sound quality to be output in units of subwords, and is stored and held in association with each subword. Therefore, the transfer function held in the present invention need only be the number of subwords, and the amount of data can be suppressed.

  In the present invention, the transfer function is called continuously for each subword when the input signal is input. Therefore, in the present invention, since continuous conversion is performed for continuous utterances, smooth and real-time output is possible.

  In the present invention, the step of synthesizing the series of output sounds is a convolution operation using a digital filter. Therefore, in the present invention, a more natural output sound can be obtained by connecting data subjected to sound quality conversion for each subword into a series of smooth formats.

  According to the present invention, by performing sound quality conversion for each subword by a transfer function in parallel with the speaker's utterance, it is possible to output a speech close to the model of the output speech signal used when generating the transfer function. Therefore, if the speaker's own voice is used as a model of the output voice signal, it is possible to reproduce a voice close to the speaker's own voice.

  Furthermore, according to the present invention, by setting the unit of sound quality conversion as a subword, it is possible to perform precise conversion compared to the case of batch conversion of input signals, and there is an advantage that clear sound can be output.

  Furthermore, according to the present invention, the unit of sound quality conversion is a subword, so that the number of transfer functions to be held is small compared to the conversion for each word or each sentence, and the amount of data can be suppressed. For example, when subwords are syllables, it is only necessary to prepare transfer functions corresponding to pronunciation, such as so-called fifty sounds, muddy sounds, semi-muddy sounds, stuttering sounds, and prompt sounds.

  Furthermore, according to the present invention, by setting the unit of sound quality conversion as a subword, it is possible to flexibly reproduce the output speech without being restricted by a word or a sentence model.

  Further, according to the present invention, when the input signal is ambiguous and the subword identification is wrong, a transfer function that is different from the intention of the speaker is called (for example, the sound function is converted by the transfer function of “Shi” which is intended to speak “Hi”). The original speech is ambiguous, and the transfer function that is called is similar, so the output speech is also similar, thus making it possible to perform transformations that are robust against ambiguity. .

  Furthermore, according to the present invention, after performing sound quality conversion for each subword, it is possible to smooth the output sound by performing convolution calculation with a digital filter and compensating for the connected portion of the converted sound quality signal for each subword. To do.

  Furthermore, according to the present invention, there is no restriction on the model of the input signal and output audio signal of the transfer function. Therefore, the input signal is the body conduction sound of the esophageal vocalization method, the body conduction sound at the time of vocal cord vocalization, the normal voice of the normal person, the voice of the esophageal vocalization method, the inaudible murmur, and the model of the output voice signal is the speaker At the time of generating a transfer function, such as the current voice of the speaker, the voice of a past speaker recorded on a recording medium such as a cassette tape, the voice of a relative, the voice of a celebrity, the ideal voice of the speaker, etc. It is possible to set freely according to this.

  As described above, in the present invention, both the input signal and the output voice are not limited. However, as one preferred embodiment, a voice support method for a voice function disabled person by total pharyngectomy is illustrated. An example of an apparatus configuration for carrying out the invention will be described with reference to FIG.

  First, as a preliminary process, a transfer function is generated. In the embodiment, for the purpose of assisting the speech of a person with speech disabilities, the input signal is a body conduction sound when the speaker performs the esophageal speech method. As a model of the output audio signal, the voice of the speaker himself before losing the vocal cords recorded on a recording medium such as a cassette tape is prepared. Subwords are in syllable units.

  The voice recorded in the recording medium is reproduced with the apparatus for extracting the body conduction sound as the signal input unit 11 attached to the speaker's body, and the reproduced voice is transferred to the transfer function generation unit 22 by the voice model input unit 21. Simultaneously with the input, the speaker utters the same content simultaneously with the reproduced speech by the esophageal utterance method, and inputs it to the transfer function generator 22. In other words, if the voice recorded in the medium speaks “Asahi” (/ a / sa / hi /), the speaker speaks “Asahi” simultaneously with the playback sound. As a result, the recorded voice and the body conduction sound of the speaker's esophageal utterance method are associated in the transfer function generator 22.

  Since the utterance content of the playback voice can be specified in advance, the playback voice is subworded according to the utterance content of the playback voice specified in advance by superimposing the playback voice and the body conduction sound based on the time axis and vibration characteristics, etc. In addition, the corresponding body conduction sound can be extracted for each subword.

A transfer function is generated in the transfer function generator 22 based on the reproduced voice and the body conduction sound extracted for each subword. When the autospectrum of the body conduction sound is Hxx and the cross spectrum of the reproduced sound and the body conduction sound is Hxxd, the expression for generating the transfer function H _(f) is H _(f) = Hxd / Hxx.

  As shown in FIG. 3, information is collected multiple times for the same subword, and “asa (/ sa /)” of “Asahi” is different from “sa (/ sa /)” of “Sapporo”. By collecting information of the same subword by word, the transfer function for the fluctuation of the input signal can be made flexible, and the accuracy is improved. In FIG. 3, the upper waveform in the “voice” column and the “body conduction sound” column is a voice waveform, and the shade of the lower color is a spectrogram.

  By performing the above process for all subwords, transfer functions corresponding to all subwords are generated and stored in the transfer function storage unit 13 to complete the preliminary process.

  Note that the speech model input unit 21 and the transfer function generation unit 22 in FIG. 1 are used only in the process of generating the transfer function, and therefore, there is no problem even if separated from other device configuration units.

  Hereinafter, the utterance support method for actually uttering in the present invention will be described. FIG. 2 illustrates an image of the following utterance support method.

  As a first step, a speaker speaks, and the body conduction sound is input at the signal input unit 11 in FIG. Since the input signal varies depending on where the body conduction sound is extracted, and the measurement characteristics of the device may be considered, it is desirable to attach the same device used to generate the transfer function at the same position.

  As a second step, the voice recognition unit 12 in FIG. 1 identifies the input signal for each subword. Identification is performed by generally known speech recognition.

  As a third step, the above-described transfer function corresponding to each subword is called from the transfer function storage unit 13 in FIG. 1, and the sound quality conversion unit 14 in FIG. 1 corrects the frequency characteristics for each subword. Convert sound quality.

  As a fourth step, the subwords subjected to sound quality conversion by the digital filter unit 15 in FIG. 1 are synthesized as a series of output sound signals.

  As a method of synthesizing a sound quality conversion result cut for each subword as a series of continuous output audio signals, for example, a convolution operation is performed by a digital filter based on a FIR filter (Finite Impulse Response Filter). As illustrated in the lower part of FIG. 2, this is arbitrary n pieces of past data that are continuous with the latest data (in this application, the subword last converted by the sound quality conversion unit 14 in FIG. 1). Is obtained, a convolution operation is performed to compensate for each data, and this is repeated for each subword, thereby compensating for the difference in the joint portion of each broken subword and smoothing the output audio signal. The digital filter is not limited to the FIR filter. Any type of digital filter can be used as long as it is a digital filter that complements each sub-word by a convolution operation, such as an IIR filter (Infinite Impulse Response Filter).

  As a fifth step, a series of output audio signals are output from the output unit 16 in FIG. The output unit 16 can be freely selected according to the purpose and application, such as a speaker or a communication device such as a mobile phone for output as voice, or an e-mail or word processor for output as character data via voice recognition software or the like. Can be set.

  One of the features of the present invention is that it has strong ambiguity as described above, and an example will be described below.

  If each subword is identified incorrectly, that is, if the place to divide is incorrect, for example, “Asahi” (/ a / sa / ra / hi / i / ), Of course, the transfer function corresponding to “A”, “SA”, “RA”, “HI” and “I” will be called. However, in this case, conversely, since the original input speech is analyzed in more detail, the output speech itself is close to the utterance of the speaker.

  In addition, if the utterance is not recognized in place of the separation place, for example, as described above, “Asahi” (/ a / sa / hi /) is mistakenly recognized as “Asashi” (/ a / sa / shi /), naturally “ The transfer function corresponding to “a” “sa” “shi” will be called. In this case, “hi” (/ hi /) in the input speech is misidentified as “shi” (/ shi /), but in other words, the speaker's utterance is ambiguous. It can be said that the voice that is mistaken for the original voice is similar, and in this case, the transfer function is also similar. Therefore, the output speech is not much different from what the speaker intended, and does not interfere with the speaker's communication.

  In the above-described embodiment, since the description is based on the assumption of the speech support method for the speech function disabled person, the input signal is the body conduction sound when the speaker's esophagus is generated, and the output speech is the speaker's own past. Although described as a voice, it is not limited to this. The input signal can be appropriately changed, such as a body conduction sound when a normal person's vocal cord is uttered, an actual utterance sound, a non-audible murmur sound, and the like. In addition, when there is no sample of the voice of the speaker itself, the output voice functions in the same manner even if another person's voice is used. Furthermore, as described above, the output method itself can be freely set such as voice through a speaker, communication equipment, and character output.

  In other words, if the user and purpose of use are clear at the stage of creating the transfer function, select a suitable input / output method and create a transfer function tailored to the purpose. Can do.

  For example, when the voice of a speaker capable of vocal voicing is used as the input signal and the internal conduction sound when the vocal cord is uttered, and the output method is a communication device such as a mobile phone, the influence of external noise is suppressed under high noise. It is possible to provide a method capable of communicating clearly and clearly.

  In addition, if a transfer function is created by using a normal voice microphone for the input signal and the output voice as the voice of another person such as a family member or celebrity, the voice of the speaker is converted into the voice of another person, and the speaker's free speech is converted into the voice of another person. It is also possible to provide a voice transmission method as a voice changer that reproduces the above.

  Regarding the conversion of the word “Asahi”, we verified the effectiveness of sound quality conversion based on the body conduction sound when the vocal cords were uttered by the healthy person and the voice of the healthy person himself. The body conduction sound was extracted from the upper left of the upper lip with an acceleration pickup. The sampling frequency is 16 kHz. The extraction position of the body conduction sound is a position that has been confirmed to be effective in Non-Patent Document 1 described above.

  In addition, as a method of identifying a break for each subword, commercially available free software “Julius 3.4.2” was used. Subwords are syllable units. The acoustic model and the language model necessary for operating the software were an unspecified speaker model attached to “Dictionation Kit Ver.3.0” and a web-based 60,000 word bigram provided with the software. . The voice identification software is not particularly limited, and the operation can be confirmed with other software.

  The voice of the speaker himself as a model of the output voice signal was collected at a position 30 cm from the speaker using a microphone.

  The measurement results are shown in FIGS. The upper waveform in each figure is a speech waveform, and the shade of the lower color is a spectrogram. FIG. 4 shows the speech waveform and spectrogram of the “Asahi” utterance collected from the microphone, and FIG. 5 shows the speech waveform and spectrogram of the body conduction sound when the “Asahi” utterance. By comparison, it can be seen that the frequency characteristics of the broad component are mainly lost in the body conduction sound.

  A transfer function is generated from the voice shown in FIG. 4 and the body conduction sound shown in FIG. 5, and the output result obtained by converting the sound of the body conduction sound of FIG. FIG. From FIG. 6, it can be seen that the frequency characteristic of the high frequency component lost in the body conduction sound is restored. Also, comparing FIG. 4 and FIG. 6, it can be seen that the speech waveform and spectrogram are similar, and therefore the output speech has a voice quality similar to the speech model and the utterance content is also similar. Note that “/ SIL /” in FIG. 6 indicates a silent state.

  For reference, FIG. 7 shows the result of sound quality conversion performed on the utterance “a, hi, hi” using the transfer function “hi, hi, a” with different vowels and consonants. It can be seen that the high frequency characteristics can be recovered from the spectrogram, although the speech waveforms are very different.

  In environments where it is difficult to communicate with clear voice, such as voice support for the esophageal vocalization method for people with speech dysfunction, support for communication using communication devices such as mobile phones in high noise environments, etc. Clear speech can be reproduced. It can also be used as a voice changer, such as a doll that speaks in the mother's voice.

It is a block diagram of the apparatus which implement | achieves the speech support method in this invention. It is a schematic diagram of the speech support method with respect to a person with a speech function disorder in this invention. It is a transfer function generation | occurrence | production image figure in this invention. It is the waveform and spectrogram which show the voice of the speaker in this invention. It is the waveform and spectrogram which show the inside-body sound of a speaker in this invention. It is the waveform and spectrogram which were decompress | restored from the body conduction sound using the correct transfer function in this invention. It is the waveform and spectrogram which were decompress | restored from the body conduction sound using the incorrect transfer function in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Signal input part 12 Voice identification part 13 Transfer function memory | storage part 14 Sound quality conversion part 15 Digital filter part 16 Output part 21 Voice model input part 22 Transfer function generation part

Claims (5)

A transfer function that associates the body's internal conduction sound or voice with the input signal information and associates the input signal information with the model of the output voice signal is created and stored in advance using the cross spectrum method for each subword shorter than the word. Process,
Inputting the conducted sound or voice of the speaker as an input signal;
Identifying the input signal for each subword;
Converting the sound quality of the subword by the transfer function corresponding to each subword;
Synthesizing the sound quality converted subword as a series of output audio signals;
And a step of outputting the series of output audio signals. The utterance support method according to claim 1, wherein the subword is a phoneme, a semi-syllable, or a syllable. The utterance support method according to claim 1, wherein the transfer function is a function for converting the input signal into a sound quality to be output in units of subwords, and is stored and held in association with each subword. The utterance support method according to claim 1, wherein the transfer function is continuously called for each subword when the input signal is input. The speech support method according to claim 1, wherein the step of synthesizing as a series of output speech is a convolution operation using a digital filter.