JPH11119791A - System and method for voice feeling recognition - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To recognize the emotion level of a speaker in a speech recognition system. Kind Code: A1 A dictionary unit that collects words to be subjected to speech recognition, a speech analysis unit that performs speech analysis processing, an acoustic model unit that has a speech pattern in units of phonemes, and an utterance representing deformation of a phoneme spectrum due to emotion. A modified emotion model unit, and a speech recognition unit that performs a speech recognition process by connecting the acoustic model unit, the uttered modified emotion model unit, and the dictionary unit to the speech analysis result. The target word is output as a speech recognition result, and an emotion level indicating the degree of the emotion of the speaker having the speech is output. The other voice analysis unit outputs the emotion level from the characteristics of the power of the voice.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a speech recognition system and method, and relates to a small information device represented by a car navigation system, an in-vehicle PC, a PDA (personal digital assistant), and a handheld PC, and a portable speech translator. Machines, games, and voice recognition systems used in home appliances, especially words and sentences that express emotions, as well as words and sentences to be subjected to voice recognition,
The present invention relates to a voice emotion recognition system and method for recognizing a degree of emotion.

[0002]

2. Description of the Related Art In recent years, small information systems using voice recognition technology have been around for a long time. These include car navigation systems, small information devices such as PDAs, and portable translators.

[0003] As an example of such a speech recognition system,
Japanese Patent Application No. 5-35776 discloses a "translation device with an automatic language selection function". The translation device recognizes the voice of an operator input from a microphone, translates the voice, and outputs the voice of the translated language. A technology relating to a translation device is disclosed.

[0004] An outline of such a conventional speech translation apparatus will be described below with reference to FIG.

FIG. 7 is a block diagram showing the configuration of a speech translation apparatus according to the prior art. The control unit 701 includes a microprocessor or the like, and controls each unit of the device. The voice section cutout unit 702 converts the voice input from the microphone 709 into a digital signal and cuts out the digital signal, and outputs the digital signal.
Send to The voice recognition unit 703 receives an operation signal 711 from a keyboard, a switch, or the like, and receives a command from the control unit 701, passes through the microphone 709 and the voice section cutout unit 702,
Analyze the extracted audio. Then, by comparing the result with a standard speech pattern stored in the speech recognition dictionary unit 707, speech recognition is performed. Voice synthesis unit 70
Reference numeral 5 reads a translated word corresponding to the voice recognized by the voice recognition unit 703 from the translated word data memory card 706, converts the translated word into a voice signal, and outputs the voice signal via the speaker amplifier 710 and the speaker 708.

[0006] The display unit 704 provides instructions to the user of the translation apparatus, displays translated characters, and the like. The translation data memory card 706 is composed of a ROM card or the like,
When a translated word is synthesized and output, audio data is stored. Further, a character code corresponding to the translated word is read from the translated word data memory card 706 and displayed on the display unit 704. By exchanging the translated word data memory card 706 with one for another language, translation into a plurality of languages becomes possible. The voice recognition dictionary unit 707 includes a RAM or the like, and stores a standard voice pattern according to the occurrence of the operator. This standard voice pattern is stored in advance by the operator.

[0007]

SUMMARY OF THE INVENTION Such speech recognition,
In the field of speech synthesis technology, its development is expected from the demand that the system should provide a more human-like user interface with the improvement of semiconductor technology. In the above-mentioned small information systems using the conventional speech recognition technology, as car navigation systems, portable information devices represented by PDAs, portable translators, and information home appliances with voice interfaces, the future will continue. It is expected to become more and more popular.

However, in speech recognition, the amount of information to be processed is enormous. Therefore, in the prior art, it is necessary to limit the number of words to be recognized in order to prevent the performance of the recognition rate and the recognition response time from lowering. There is. To do this, for a word or sentence registered in advance, the statistical characteristics of the speaker's voice in the character string and the characteristics of the voice actually spoken by the speaker are compared, and the probability is calculated. The value closest to is used as the recognition result.

[0009] In the future, it is conceivable that the performance of the recognition rate and the recognition response time will be improved due to technical innovation in speech recognition and improvement of the software and hardware for realizing it. Therefore, in order to provide a more human-like user interface, conventional speech recognition technology should not only recognize words and sentence strings registered in advance, but also recognize emotions and intentions of speakers. Even if the number of recognized words is limited, improvement in usability can be expected. However, the conventional speech recognition system is a speech recognition system in which only a character string of a word or sentence registered in advance is collated with a speech and a character string closest to the inputted speech is output as a speech recognition result. I cannot recognize the emotions and intentions of the speaker who did it.

The present invention has been made to solve the above problems so that the system can have a human interface at all.

An object of the present invention is to provide a speech recognition system for use in a small information system, which recognizes a character string of a word or a sentence registered in a dictionary with respect to an inputted speech and has the inputted speech. It is an object of the present invention to provide a voice emotion recognition system and method capable of recognizing a speaker's emotion and intention.

Another object of the present invention is to provide a speech recognition system used in a small-sized information system, in which a speaker's feeling or intention of an input voice is converted into a number or a modifier expressing the degree of the feeling. Conversion to achieve a good voice interface between humans and the system.

[0013]

In order to achieve the above object, in the speech emotion recognition system and method of the present invention, words and sentences to be subjected to speech recognition are collected and defined as a dictionary, and the speech recognition result is obtained. In a speech recognition system that picks up those words and sentences from the dictionary unit and outputs them using character string display and speech synthesis, a speech analysis unit that performs speech analysis processing on the captured speech, and a speech pattern Acoustic model unit with a phoneme unit, a vocal deformation emotion model unit that represents the deformation of the phoneme spectrum due to emotion, and a voice analysis unit that connects the acoustic model unit, the utterance deformation emotion model unit, and the dictionary unit for speech analysis. A speech recognition unit that performs processing, outputs words and sentences to be subjected to speech recognition based on the features of the speech as speech recognition results, and the speaker's emotions of the speech. Is obtained so as to output a degree.

In one more detailed embodiment, the level indicating the degree of the emotion of the speaker possessed by the voice is a number 0 to N.
(N is an integer).

Further, in the speech emotion recognition system and method of the present invention, the degree of the speaker's emotion possessed by the speech is determined by a dictionary which collects words and sentences to be subjected to speech recognition, and a dictionary of those words and sentences. Equipped with a dictionary that has a dictionary that collects modifiers that express the level of emotion for the sentence, and picks up those words and sentences as speech recognition results, and also picks up modifiers that express the level of emotion. In addition, a modifier is added to a word or a sentence, and the word or the sentence is output using characters or speech synthesis.

In a further detailed embodiment, words and sentences to be subjected to speech recognition are collected and defined as a dictionary.
In a voice recognition system that picks up these words and sentences as voice recognition results and outputs them using character string display and voice synthesis, a voice analysis unit that performs voice analysis processing on the captured voice, and a voice pattern And a voice recognition unit that performs voice recognition processing by connecting the voice model unit and the dictionary unit to voice analysis results, and performs voice analysis processing on the captured voice. The voice analysis unit can output the degree of emotion from the power characteristic indicating the strength of the sound in which the degree of emotion appears.

[0017]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments according to the present invention will be described below with reference to FIGS.

FIG. 1 is a block diagram showing the functions of the voice and emotion recognition system according to the present invention and the flow of processing thereof.

In order to perform voice and emotion recognition, voice is taken in from the microphone 101 shown in FIG. An analog signal that is a captured voice is converted by an A / D converter 102 that converts the analog signal into a digital signal.
Analog data is converted into digital data at an arbitrarily determined sampling period. The digital data of the converted voice is subjected to preprocessing such as noise processing, voice analysis, and speaker adaptation by the voice analysis unit 103, and voice and emotion recognition is performed by the voice emotion recognition unit 104. Here, speech and emotion recognition execute two processes.

The first process is to analyze a speech signal, analyze it as phonemes for each short time, analyze its pattern, and select a corresponding word or sentence from a dictionary.

The second process is to analyze a voice signal, analyze it as a phoneme for each short time (5 to 20 ms), analyze its pattern, and measure the degree of emotion of the voice uttered by the speaker. The level to be shown is to be selected for each word or sentence.

From the above two processes, a speech recognition result and a speech emotion level 109 are generated as outputs of the speech emotion recognition system.

The voice emotion recognition unit 104 includes the voice analysis unit 10
The voice analysis result of the input voice analyzed in step 3 is compared with the acoustic model unit 105, the uttered deformation emotion model unit 106, and the word dictionary unit 107 for each phoneme connected by the model connection unit 108, and the word dictionary unit The word closest to the words registered in 107 is picked up. Further, a level indicating the degree of the emotion having the input voice of the picked-up word is selected. Note that, in the embodiment shown in FIG. 1, when the power is turned on, the level indicating the degree of the words and emotions of the phoneme units connected by the model connection unit 108 is stored in the voice emotion recognition unit 104, and the voice analysis is performed. Part 103
It can be collated immediately with the voice analysis result from.

The acoustic model unit 105 is a model used for speech recognition. Specifically, the acoustic model unit 105 is a correspondence between a character and a phoneme used in the word dictionary unit 107, and a distribution of a probability that a feature of the phoneme appears. , Which stores the distribution of probabilities that the characteristics of the phoneme that has appeared transition to a state in which the next characteristic appears. The distribution of the probability that a feature of a phoneme appears will be described. For example, the voice spectrum of "A" is different for each person, so the horizontal axis represents the voice spectrum of the phoneme "A" and the phoneme appears on the vertical axis. This means that the probability of recognizing "a" in the speech spectrum changes when the probability of taking the speech spectrum is changed. Next, a description will be given of the distribution of the probability that the feature of the phoneme that has appeared changes to a state in which the next feature appears. For example,
The phoneme "A" may be followed by "Tsu" like "Atsu", "Sai" followed by "S", or "Ama" It may transition to "ma". The probability that "A" will transition to the next phoneme changes with each phoneme. In other words, the probability that a feature of one phoneme changes to the feature of the next phoneme changes,
This distribution of probability is called.

The so-called "unspecified speaker correspondence" in which the speaker can recognize the voice without registering the voice in advance is becoming common.
As such an acoustic model, for example, a Hidden Markov Model (HMM) can be used.

The utterance transformation emotion model unit 106 focuses on the transformation element of the phoneme spectrum due to the change in emotion, and is the correspondence between the characters and phonemes used in the word dictionary unit 107 when the emotion changes. That is, the distribution of the probability of phonemes changes when emotions are included, but the distribution of the probability that the characteristics of the phoneme appear at that time, and the distribution of the probability that the characteristics of the phoneme that has appeared transitions to the state where any of the following characteristics appear: Is stored. The distribution of the probability that the feature of the appearing phoneme transitions to a state in which the following feature appears is, for example, “hot”
It does not change even if you say "hot" with emotion, but it changes when it changes to "achi" or "chii". As such an utterance deformation emotion model unit 106, for example, a hidden Markov model (HMM: Hidden Mar
kov Model) can be used.

The word dictionary unit 107 stores words, words (nouns,
Verbs, etc.) and sentences. For example, in a car navigation system, it is a collection of words necessary for a minimum necessary conversation, such as street names, place names, building names, town names, street addresses, intersection names, private houses (personal names), telephone numbers, and the like. . However, in the speech recognition emotion system, in particular, among words, it is a word that expresses an emotion, or a collection of words formed of words in which an emotion appears. More specifically, the speaker speaks “hot”, “cold”, “hot”, “cold”, “fast”, “slow”, “large”, “small”, “red”, “white”, “high”, “low”, “run” These words are "advance,""return,""turn," and "fly." Also, words that are not emotional expressions such as nouns are included. This word dictionary 107
Consists of words of, for example, 10 to 5000 words per one dictionary according to the capability of the system.

From the above, speech emotion recognition means analyzing speech, analyzing it as phonemes at short time intervals, analyzing its pattern, selecting a corresponding word or sentence from a dictionary, and selecting a speaker. Is to select, for each word or sentence, a level indicating the degree of emotion of the voice uttered.

Each processing block shown in FIG. 1 may be a system constituted by a plurality of LSIs or memories, or one or a plurality of system-on-chips constituted on semiconductor elements. Further, each process may be hardware processed by a dedicated LSI or a dedicated IC, or middleware realized by software such as a DSP or a RISC microcomputer.

FIG. 2 shows a hidden Markov model (HMM: Hidden
This is an example of modeling a Japanese phoneme by Markov Model).

Reference numerals 201, 202, and 203 represent the states of the phoneme distribution. The voice uttered by the speaker is "hot" and is "atsui" represented by an example of phonetic symbols.
For the sake of simplicity, "a" in FIG.
Corresponds to the state of 201, “tsu” corresponds to the state of 202, and “i” corresponds to the state of 203. In actual speech recognition, the state is further subdivided and represented.
Speech is a non-stationary signal, and conveys linguistic information by changing the nature of the spectrum every moment, such as the spectrum of “a” at one time and the spectrum of “tsu” at some times. This non-stationary signal can be regarded as a continuation of phonemic segments of a stationary signal having different properties. Each of the phonemic segments of the stationary signal having a different property is denoted by 201 to 2
This corresponds to the state of the HMM state transition network shown in FIG. In this state, that is, the spectrum of the voice is observed as the output from the non-stationary signal source. The observation value may be an LPC analysis result of the audio signal for each short-time frame or a vector-quantized code. Therefore, the HMM is the state transition probability 207 between the state 201 and the state 202 and the probability 201 that the spectrum of the sound output from the state 201 is output. The probability 201 indicates which probability value of the phoneme distribution of “a” is output. That is, reference numerals 201 to 203 denote phoneme distributions, each of which indicates a probability that a spectrum of a sound output from each state is output. Reference numerals 204 to 209 denote the probabilities to which state each state transitions next. Of these, 204 to 206 indicate that, for example, when a certain sound is generated for a long time, when the sound is analyzed, the sound returns to the same sound within a certain time interval.

Next, an example of an HMM used for speech recognition will be described.

A curve 211 shown in FIG.
The probability that the spectrum of the voice output from No. 1 is output is represented by a continuous distribution. Here, the spectrum of the voice is n when the feature parameter of the voice is i-dimensional.
The feature parameter. In other words, there are many types of expression methods representing the features of the voice, but if there are i expression methods, it means the feature parameter of the n-th expression method. The horizontal axis is the spectrum of the sound output from the state 201, and the vertical axis is the probability value. This distribution is a continuous distribution having mean μ_a and variance σ_a. Similarly, a curve 212 shown in FIG. 2B represents the probability that the spectrum of the sound output from the state 202 is output by a continuous distribution. The horizontal axis is state 20
2 is the spectrum of the voice output from the audio signal 2, and the vertical axis is the probability value. This distribution has a mean μ_tsu, variance σ
_Tsu with a continuous distribution.

The curve 213 shown in FIG.
3 represents the probability that the spectrum of the voice output from No. 3 is output by a continuous distribution. The horizontal axis is the spectrum of the sound output from the state 203, and the vertical axis is the probability value. This distribution is a continuous distribution having mean μ_i and variance σ_i.

Here, the speaker "Atsui" is entered in the word dictionary 107 of "atsui" registered as a recognition target word.
And voice input. A voice analysis is performed on the voice of “a”, and the characteristics of the voice are output. For example, assuming that the n-th feature parameter when the feature parameter of the voice is i-dimensional is used, the feature f_n1 of “a” is output. At this time, in the word dictionary “a”, the feature f
The probability of occurrence of _n1 is calculated from the continuous distribution curve 211, and a probability value p_n1 is output. Similarly, "one"
Voice analysis is performed on the voice of “i”, and the characteristics of the voice are output. In the word dictionary “tsu”, the probability that the feature f_n2 appears is represented by the continuous distribution curve 212
And a probability value p_n2 is output. Further, in the word dictionary “i”, the probability of occurrence of the feature f_n3 is calculated from the continuous distribution curve 213, and the probability value p_n3 is output. Further, the same processing is performed on the state transition probability from the phoneme distribution state 201 to the phoneme distribution state 202, and the state transition destination is determined to be the state 207. Finally, with respect to the registered word dictionary “atsui”, the probability value of the appearance of “hot” input by voice is P_ats
ui = p_n1 + p_n2 + p_n3. This series of processing is calculated for all the registered word dictionaries, and the one with the highest probability value is the recognition result. The above is a series of processing of speech recognition.

Further, referring to FIG. 2 (e) to FIG. 2 (g), an utterance deformation emotion model in speech emotion recognition is used.
An example of the HMM will be described.

The utterance transformation emotion model unit 106 focuses on the transformation element of the phoneme spectrum due to the change of the emotion, and is the correspondence between the character of the word stored in the word dictionary unit 107 and the phoneme when the emotion changes. , The distribution of the probability that the feature of the phoneme appears, and the distribution of the probability that the feature of the phoneme that has appeared transitions to a state in which the following feature appears.

The curve 211 corresponds to the state 2 as described above.
This is a representation of the probability that the spectrum of the voice output from No. 01 is output as a continuous distribution. Here, the speech spectrum is the n-th feature parameter when the feature parameter of the speech is i-dimensional. The horizontal axis is the spectrum of the sound output from the state 201, and the vertical axis is the probability value. This distribution is a continuous distribution having mean μ_a and variance σ_a. At this time, in the n-th feature parameter when the feature parameter of the voice is i-dimensional,
Suppose that the deformation of the phonological spectrum due to the change of the emotion has appeared remarkably. Therefore, the continuous distribution curve of the speech spectrum when the speaker utters with normal emotion is set to 211, and the continuous distribution curve of the speech spectrum when the speaker utters with emotion, that is, when the speaker is deformed by a change in emotion. Is set to 214. Therefore, in addition to the HMM of the acoustic model used for the conventional speech recognition, a probability distribution composed only of the feature parameters in which the deformation of the phoneme spectrum appears due to the change of the emotion is prepared as the HMM of the speech emotion recognition model. Curve 2
Reference numeral 14 denotes a continuous distribution of the probability that a spectrum of a voice due to a change in the emotion output from the state 201 is output in the voice emotion recognition model. The horizontal axis is the spectrum of the sound output from the state 201, and the vertical axis is the probability value. This distribution has a mean μ_a_s,
It is a continuous distribution with variance σ_a_s. Here, in the word dictionary of “atsui” registered as a word to be recognized, the speaker actually inputs a voice with a feeling of “hot”. Voice analysis is performed on the voice of "A",
The features of the audio are output. For example, if the feature parameter of the speech is i-dimensional and the feature parameter of the n-th speech is adopted, the feature “a” of f_n1_e is output. At this time, in the word dictionary “a”, the feature f
The probability of occurrence of _n1_e is calculated from the continuous distribution curve 214, and a probability value p_n1_e is output. Here, regarding the continuous distribution curve 214, the probability value p_n1_e
Takes a value higher than the probability value p_n1 at the feature f_n1 when the speaker speaks normally.

A curve 222 represents the probability that the spectrum of the sound output from the state 202 is output by a continuous distribution. The horizontal axis is the spectrum of the sound output from the state 202, and the vertical axis is the probability value. This distribution has a mean μ_tsu_s, a variance σ_tsu
_S. Curve 223 represents state 203
Is a continuous distribution of the probability that the spectrum of the voice output from is output. The horizontal axis is the spectrum of the sound output from the state 203, and the vertical axis is the probability value. This distribution has a mean μ_i_s, a variance σ_i_
It is a continuous distribution with s.

As in the case of the curve 214, "T" and "I"
The voice analysis is performed on the voice and the characteristics of the voice are output. In each of the word dictionaries “tsu”, the probability of occurrence of the feature f_n2_e is calculated from the continuous distribution curve 222, and the probability value p_n2_e is output. Further, in the word dictionary “i”, the probability of occurrence of the feature f_n3_e is calculated from the continuous distribution curve 223, and the probability value p_n3
_E is output. Further, similar processing is performed for the state transition probability between states, and the state transition destination is determined. Finally, the registered word dictionary “ats
With respect to “ui”, the probability value of the appearance of “Autum” that is voice-inputted with emotion is P_attui = p_n1_e
+ P_n2_e + p_n3_e. This series of processing is calculated in all the registered word dictionaries, and the emotion level is output according to the calculated range of the probability value. The above is a series of processing for recognizing the emotion level of the voice.

FIG. 3 is a block diagram showing functions of another voice and emotion recognition system according to the present invention and a flow of processing thereof.

In FIG. 3, a voice is taken in from a microphone 301 in order to perform voice and emotion recognition.
An analog signal, which is a captured voice, is converted from analog data to digital data by an A / D converter 302 that converts the analog signal into a digital signal at an arbitrarily determined sampling period. The digital data of the converted voice is output by the voice analysis unit 303.
Preprocessing such as noise processing, speech analysis, and speaker adaptation is performed, and speech power is analyzed by a speech power analysis unit 303a included in the speech analysis unit 303, and the emotion level is output. The output of the voice analysis unit 303 is the voice emotion recognition unit 3
At 04, speech and emotion recognition is performed. Here, the voice emotion recognition performed by the voice emotion recognition unit 304 executes two processes.

The first process is to analyze a voice signal, analyze it as a phoneme for each short time, analyze its pattern, and select a corresponding word or sentence from a dictionary.

The second process is to analyze a voice signal, analyze it as a phoneme for each short time (5 to 20 ms), analyze its pattern, and measure the degree of emotion of the voice uttered by the speaker. The level to be shown is to be selected for each word or sentence.

From the above two processes, a speech recognition result and a speech emotion level 309 are generated as outputs of the speech emotion recognition system.

The voice emotion recognition unit 304 includes the voice analysis unit 30
The acoustic model 305 and the word dictionary 307 are collated by the phoneme unit connected by the model connecting unit 308 with the voice analysis result of the input voice analyzed in
7, the closest word is picked up from the word dictionary 307 registered in. Further, a level indicating the degree of emotion of the input voice of the picked-up word is selected.

The acoustic model unit 305 is a model used for speech recognition. Specifically, the acoustic model unit 305 is a correspondence between a character stored in the word dictionary unit 307 and a phoneme, and a distribution of the probability that a feature of the phoneme appears. , Which stores the distribution of probabilities that the characteristics of the phoneme that has appeared transition to a state in which the next characteristic appears. The so-called “unspecified speaker correspondence” in which the speaker can recognize the voice of the acoustic model unit 305 without registering the voice in advance is becoming common. As such an acoustic model, for example, a Hidden Markov Model (HMM) can be used.

The word dictionary unit 307 stores words, words (nouns,
Verbs, etc.) and sentences. For example, in a car navigation system, it is a collection of words necessary for a minimum necessary conversation, such as street names, place names, building names, town names, street addresses, intersection names, private houses (personal names), telephone numbers, and the like. . However, in the speech recognition emotion system, in particular, among words, it is a word that expresses an emotion, or a collection of words formed of words in which an emotion appears. More specifically, the speaker speaks “hot”, “cold”, “hot”, “cold”, “fast”, “slow”, “large”, “small”, “red”, “white”, “high”, “low”, “run” These words are "advance,""return,""turn," and "fly." Also, words that are not emotional expressions such as nouns are included. This word dictionary unit 307
The number of words stored in each dictionary is determined according to the capability of the system, but is, for example, 10 to 5000 words per dictionary.

As described above, the voice emotion recognition system or the voice emotion recognition method is to analyze a voice signal, analyze it as a phoneme for each short time, analyze its pattern,
In addition to selecting a corresponding word or sentence from the dictionary, a level indicating the degree of emotion of the voice uttered by the speaker is selected for each word or sentence.

Each processing block shown in FIG. 3 may be a system constituted by a plurality of LSIs and memories, or one or a plurality of system-on-chips constituted on semiconductor elements. Further, each process may be hardware processed by a dedicated LSI or a dedicated IC, or middleware realized by software such as a DSP or a RISC microcomputer.

FIG. 4 (a) shows a voice input waveform of a voice "hot" uttered by the speaker in the voice emotion recognition system described in FIG. 3, in which the horizontal axis represents time and the vertical axis represents voice level. Is shown. FIG. 4B shows the power of the "hot" sound, in which the horizontal axis represents time and the vertical axis represents the sound power.

The voice input waveform 401 is a voice waveform when the speaker utters “Aut” with normal voice. The audio signal is an unsteady signal that changes every moment. If this audio signal is cut out and viewed in a short time of 20 ms, the same spectral audio analysis as that of a stationary signal can be performed. When the autocorrelation function is calculated from the sampled values of the extracted audio signal in, for example, LPC analysis widely used in audio analysis, the power of the audio is obtained as one of the characteristic parameters of the audio.

A curve 402 showing the audio power is the power calculated from the audio signal of the audio waveform 401. Time t
Represents the change in power with respect to. Here, a threshold value is arbitrarily set for this power information, and it is observed whether or not the threshold value is exceeded for each input voice.
This observation is performed by the voice analysis unit 303. Further, a plurality of thresholds are provided, and it is observed whether or not each of the inputted voices exceeds the threshold. For example, in the case of the sound of the sound power curve 402, the threshold value TH1 is exceeded, but the threshold value TH2 is not exceeded. That is, when the power is continuously between the threshold values TH1 and TH2, the emotion level is regarded as 1, and the voice analysis unit 303 outputs the emotion level 1.

Next, FIGS. 4 (c) and 4 (d) show the speech waveform and the speech power, respectively, when the speaker utters "hot" with a strong tone with emotion. FIG.
FIG. 4C shows time on the horizontal axis and audio level on the vertical axis.
In (d), the horizontal axis indicates time, and the vertical axis indicates audio power. FIG.
In (c), reference numeral 403 denotes an audio waveform. , FIG.
In (d), reference numeral 404 denotes the power of the sound calculated from the sound signal of the sound waveform 403, which represents a change in the power with respect to time t. For example, in the case of the audio power 404, the threshold value TH1 is exceeded and the threshold value TH is further increased.
Over two. That is, when the power continuously exceeds the threshold value TH2, the emotion level is regarded as 2, and the voice analysis unit 303 outputs the emotion level 2. In this example, the emotion level is set to two levels, but by increasing the threshold value, the emotion level is set to N (N is an integer).
Can be set in stages.

Further, from the speech analysis unit 303, the feature parameters of the speech for speech recognition together with the emotion level are input to the speech emotion recognition unit 304 from moment to moment, and finally, from the speech emotion recognition unit 304. The speech recognition result "atu
data 309 indicating “si” (= hot) and emotion level N
Is output.

Next, the hardware configuration of the speech recognition system according to the present invention will be described with reference to FIG.

The microphone 501 for taking in voice is a car navigation system, a portable information terminal, a PDA,
Handheld PCs, games, portable translators, home appliances such as air conditioners, and the like are directional microphones having directivity so as not to capture ambient noise. 50
Reference numeral 4 denotes an A / D converter that converts analog audio data captured by the microphone 501 into digital audio data.

The voice input button 502 is a button for designating a section in which voice is being input. While the button is being pressed, or from the time the button is pressed, the system is notified that a voice has been input. 505 is
This is an interface for connecting the voice input button 502 and the system.

The key input device 509 is, for example, a pen input digitizer in the case of a portable information terminal, and a keyboard in the case of a handheld PC. In the case of a game console such as a NES, a keypad for operating a character or the like or a joystick is used. 5
Reference numeral 10 denotes an interface for connecting the key input device 509 to the system.

The CPU 503 controls a main system such as a car navigation system, a portable information terminal, a PDA, a handheld PC, a game, a portable translator, and a home appliance, and performs voice recognition and emotion recognition in a voice emotion recognition system. Perform processing. The voice analysis unit 303, the voice emotion recognition unit 304, and the model connection unit 308 of the voice emotion recognition system of the present invention shown in FIG. A recent trend is to use a RISC microcomputer or DSP for the CPU 503.

The ROM 506 is a storage device for storing a word dictionary for speech recognition, an acoustic model, an utterance deformation emotion model, and a program. Further, a memory card may be used to store a plurality of dictionaries, acoustic models, and utterance deformation emotion models.

The RAM 507 stores a part of the dictionary, acoustic model, and program transferred from the ROM 506, and is a minimum necessary work memory required for the voice emotion recognition processing. A semiconductor element having a short time is used. Also, here is C
Data 309 indicating a speech recognition result and an emotion level is input from PU 503.

The bus 508 is used as a data bus, an address bus, and a control signal bus in the system.

A display 512 for outputting and displaying the voice emotion recognition result is an LC such as a TFT liquid crystal display.
D to display the speech recognition result and the emotion level of the speech. An interface 511 connects the display 512 to the system.

A speaker 514 for outputting the voice emotion recognition result as sound synthesizes and outputs the voice recognition result and the emotion level of the voice. A D / A converter 513 converts the speech recognition result and the emotion level of the speech from text to speech synthesis data, and then converts the digital speech synthesis data to an analog speech signal.

Hereinafter, an example of an embodiment according to the present invention will be described with reference to FIGS.

In the present embodiment, a case where the voice emotion recognition system of the present invention is applied to a car electronics product will be described.

FIG. 6A is a block diagram when the voice emotion recognition system according to the present invention is used for operating an air conditioner of a car electronics product. FIG. 6B shows an example of voice input in the voice emotion recognition system. It is a schematic diagram showing the recognition result.

In FIG. 6A, reference numeral 601 denotes a voice input microphone, 602 denotes a voice emotion recognition system, and 603 denotes a voice emotion recognition result and a conversational format for bidirectional communication with a speaker. A display 604 for outputting character information is a speaker for synthesizing and outputting conversational character information in order to perform a voice emotion recognition result and bidirectional communication with a speaker.

Next, with reference to FIG. 6B, an example of a voice input uttered by a speaker and an example of a recognition result output by a voice emotion recognition system will be described.

Reference numeral 605 denotes “hot = emotional level 3” which is a recognition result of the voice emotion system 602 when the speaker utters “hot” with a normal voice to the voice emotion system 602. Next, reference numeral 606 denotes a recognition result of the voice emotion system 602 when the speaker utters "hot" in a strong tone with respect to the voice emotion system, and indicates "hot = emotion level 5".

Reference numeral 607 indicates “is it a little hot” as a recognition result of the voice emotion system 602 when the speaker utters “hot” with a normal voice to the voice emotion system. Next, 608 is a recognition result of the voice emotion system 602 when the speaker utters “hot” in a strong tone with respect to the voice emotion system, “Is it quite hot?”
Is shown.

Further, "Yes" is given to the recognition result 608.
Then, the voice emotion system 602 outputs "set the interior of the vehicle to 25 ° C." as a recognition result. Actually, the inside of the vehicle is set to 25 ° C.

In another embodiment, a case will be described in which the voice emotion recognition system of the present invention is applied to a game product such as Family Computer (registered trademark).

FIGS. 8 (a) to 8 (d) show the operation of a character of a game machine such as a family computer.
It is an example using a voice emotion recognition system, and is a schematic diagram showing a voice input example of an interface by voice and an operation example based on a recognition result.

In FIG. 8, reference numerals 801, 802, 809, and 810 denote screens of a game machine main body display and a TV or the like to which the game machine is connected.

In FIG. 8A, the character 805 appearing in the game is, for example, moving from left to right toward the screen 801. This operation is performed using voice emotion recognition. Therefore, the operator (speaker) utters “forward” as a normal voice as shown in utterance example 803. The voice emotion recognition system recognizes the progress and further recognizes the emotion level. For example, in the voice emotion recognition system in this game machine, if the emotion level is set to five levels, the emotion level is recognized as three. Therefore, in the system of the game machine main body, the character 80
5 is moved to the position of the character 806.

In FIG. 8B, the character 807 appearing in the game moves from left to right toward the screen 802, for example. This operation is performed using voice emotion recognition. Then, the operator (speaker) utters “advance” in a strong tone as shown in a voice example 804. The voice emotion recognition system recognizes the progress and further recognizes the emotion level. For example, in the voice emotion recognition system of this game machine, if the emotion level is set to five levels, the emotion level is recognized as 5. Therefore, in the system of the game machine main body, the character 80
7 largely moves to the position of the character 808. Here, the movement amount of the character is the recognized voice
Is proportional to the emotion level.

In FIG. 8C, the character 813 appearing in the game advances from left to right toward the screen 809, for example. At this time, the obstacle 81
Suppose 6 appears. Therefore, it is necessary to jump over the obstacle 816. This operation is performed using voice emotion recognition. Then, the operator (speaker) utters “jump” with a normal voice as shown in voice example 811. The voice emotion recognition system recognizes a jump (jyanpu) and further recognizes an emotion level. For example, in the voice emotion recognition system in this game machine, if the emotion level is set to five levels, the emotion level is recognized as three. Therefore, in the system on the game machine main body side, the character 813 is moved to the position of the character 814, and further moved to the position of the character 815.

In FIG. 8D, a character 817 appearing in the game is moving from left to right toward the screen 810, for example. At this time, obstacle 8
Suppose 20 appears. The obstacle 820 is displayed on the screen 80
9 is larger than the obstacle 816. Therefore, it is necessary to jump over the obstacle 820 high. This operation is performed using voice emotion recognition. Therefore, the operator (speaker)
Utters “jump” in a strong tone as shown in a voice example 812. The voice emotion recognition system uses the jump (jy
anpu), and also the emotion level.
For example, in the voice emotion recognition system of this game machine, if the emotion level is set to five levels, the emotion level is recognized as 5. Therefore, in the system of the game machine main body side, the character 817 is largely moved to the position of the character 818 and further moved to the position of the character 819. The movement amount of the character is proportional to the emotion level of the recognized voice “jump”.

[0081]

According to the present invention, in a car navigation system, a small information system, and a voice recognition system used in a game, a character string of a word in a registered dictionary is recognized by voice,
It is possible to provide a voice emotion recognition system capable of recognizing a level of emotion of a speaker's voice.

According to the present invention, the emotion level of voice can be recognized in a car navigation system, a small information system, and a game using voice recognition.
Even with a limited number of words, it is possible to increase the variations of the interface by voice recognition, and to realize a good voice recognition interface.

[Brief description of the drawings]

FIG. 1 is a block diagram showing one embodiment of a voice emotion recognition system according to the present invention.

FIG. 2 is a schematic diagram for explaining an acoustic model and an utterance deformation emotion model of the speech emotion recognition system shown in FIG. 1;

FIG. 3 is a block diagram showing another embodiment of the voice emotion recognition system according to the present invention.

FIG. 4 is a schematic diagram for explaining a relationship between a voice waveform and voice power and a feeling level of the voice emotion recognition system shown in FIG. 3;

FIG. 5 is a block diagram showing a hardware configuration of the present invention.

FIG. 6 is a schematic diagram showing a voice input example of a voice interface and a recognition result example in a car navigation system to which the voice recognition emotion system of the present invention is applied.

FIG. 7 is a block diagram of a conventional portable translator.

FIG. 8 is a schematic diagram showing a voice input example of a voice interface and an operation example based on a recognition result in a game machine to which the voice recognition emotion system of the present invention is applied.

[Explanation of symbols]

101, 301, 501, 601 ... microphone, 102,
302, 504: A / D converter, 103, 303: Voice analysis unit, 104, 304: Voice emotion recognition unit, 10
8, 308: Model connection unit, 105, 305: Acoustic model unit, 106: Speech deformation emotion model unit, 107, 3
07: Word dictionary unit, 201: “A” state in HMM acoustic model connection, 202: “T” state in HMM acoustic model connection, 203: “I” state in HMM acoustic model connection, 204: State Probability of transition from "A" to state "A", 205 ... Probability of transition from state "T" to state "T", 206 ... Probability of transition from state "I" to state "I", 207 ... State "A" From the state "i" to the state "i", 208 ... the probability of a transition from the state "i" to the state "i", 209 ... the probability of a transition from the state "i" to another state, 211 ... the output of the state "a" Continuous distribution of probabilities, 212 ... Continuous distribution of output probabilities of state "T", 2
13 ... Continuous distribution of output probability of state "i", 214 ... HM
Continuous distribution of output probabilities of state "A" in the M utterance deformation emotion model, 221 ... Continuous distribution of output probabilities of state "A" in the HMM utterance deformation emotion model, 222 ... Output of state "T" in the HMM utterance deformation emotion model Continuous distribution of probabilities, 223 ... continuous distribution of output probabilities of state "i" in the HMM utterance deformation emotion model, 502 ... button, 50
3 CPU, 506 ROM, 507 RAM, 5
09 ... key, 602 ... voice emotion recognition system.

Continuing on the front page (72) Inventor Kazuo Kondo 5-2-1, Josuihonmachi, Kodaira-shi, Tokyo Inside the Semiconductor Division, Hitachi, Ltd. Hitachi Central Research Laboratory (72) Inventor Tetsushi Toshita 5-2-1, Josuihoncho, Kodaira-shi, Tokyo Inside the Semiconductor Division, Hitachi, Ltd. (72) Yasushi Ishikawa 5--22, Josuihoncho, Kodaira-shi, Tokyo No. 1 in Hitachi microcomputer system

Claims (8)

[Claims] 1. A method for collecting words and sentences to be subjected to speech recognition, defining the dictionary as a dictionary, picking up the words and sentences from a dictionary unit as a speech recognition result, and outputting the words and sentences using character string display and speech synthesis. A speech recognition unit that performs a speech analysis process on the captured speech, an acoustic model unit that has a speech pattern in phoneme units,
An utterance deformation emotion model unit that represents a deformation of a phoneme spectrum due to emotion, and a speech recognition unit that performs speech recognition processing by connecting the acoustic model unit, the utterance deformation emotion model unit, and the dictionary unit to the speech analysis result. A voice emotion recognition system comprising: outputting words and sentences to be subjected to voice recognition as voice recognition results based on voice characteristics; and outputting a level indicating a degree of emotion of a speaker having voice. . 2. A speech emotion recognition system according to claim 1, wherein the level indicating the degree of the emotion of the speaker possessed by the speech is a numeral 0 to N (N is an integer). Recognition system. 3. The voice emotion recognition system according to claim 1, wherein the dictionary unit has a dictionary in which words and sentences to be subjected to speech recognition are collected, and a voice is provided for those words and sentences. A dictionary that collects modifiers that express the level of emotions, and picks up those words and sentences as speech recognition results, and also picks up modifiers that express the level of emotions, and modifies words and sentences into words. A voice emotion recognition system characterized by adding characters and outputting using characters or voice synthesis. 4. A speech recognition system which collects words and sentences to be subjected to speech recognition, defines them as a dictionary, picks up the words and sentences as a speech recognition result, and outputs them using character string display or speech synthesis. In the system, a voice analysis unit that performs a voice analysis process on the captured voice, an acoustic model unit that has a voice pattern in phoneme units,
A voice recognition unit that performs a voice recognition process by connecting the acoustic model unit and the dictionary unit to the voice analysis result, and performs a voice analysis process on the captured voice; A voice emotion recognition system that outputs a level indicating a degree of emotion from a characteristic of power indicating the strength of a sound that appears. 5. A dictionary that collects words and sentences to be subjected to speech recognition, a speech analysis unit that performs speech analysis processing on the acquired speech, an acoustic model unit that has speech patterns in phoneme units, A speech transformation emotion model unit representing a transformation of a phoneme spectrum by a, and a speech recognition unit that performs speech recognition processing by connecting the acoustic model unit, the speech transformation emotion model unit, and the dictionary unit to the speech analysis result, It outputs words and sentences to be subjected to speech recognition as speech recognition results based on the features of the speech, and outputs the degree of the emotion of the speaker having the speech using data from the utterance deformation emotion model unit. Voice emotion recognition method. 6. Speech analysis capable of collecting words and sentences to be subjected to speech recognition and performing a speech analysis process on the fetched speech and analyzing the power of the sound and outputting an emotion level. Unit, a sound model unit having a voice pattern in phoneme units, and a voice recognition unit for performing voice recognition processing by connecting the voice model unit and the dictionary to the voice analysis result. A voice emotion recognition method comprising: recognizing a degree of emotion from a characteristic of power indicating a strength of a sound in which the degree of emotion appears, and outputting the output emotion level. 7. The voice emotion recognition method according to claim 5, wherein the level indicating the degree of the emotion of the speaker possessed by the voice is a numeral 0 to N (N is an integer). Voice emotion recognition method. 8. The voice emotion recognition method according to claim 5, wherein the dictionary unit includes a dictionary in which words and sentences to be subjected to voice recognition are collected, and a speech and / or a voice for each of the words and sentences. And a dictionary that collects modifiers that express the level of emotion that the user has, and as a result of speech recognition, picks up those words and sentences, and also picks up modifiers that express the level of emotion, A speech emotion recognition system characterized by adding a modifier to a sentence and outputting it using characters or speech synthesis.