JPH06162168A - Composite image display system - Google Patents

Abstract

(57) [Summary] (Correction) [Purpose] In a synthetic image display system applicable to AV, etc., which can convey a message by a synthetic moving image and synthetic voice of a sender by simply sending text data, especially, a synthetic voice. And the movement of the mouth of the composite image can be accurately synchronized. [Structure] Speech synthesis means 31 for synthesizing and outputting synthesized speech
And a conversion means 32 for converting the sentence information into a series of mouth-shaped codes representing a series of mouth-shaped movements when uttered, and a pronunciation time of each syllable output from the speech synthesizing means to calculate the timing of each syllable break. Pronunciation time calculation means 33 for estimating
And a display control means 34 for controlling the display image to be switched to an image corresponding to the mouth-shaped code from the converting means at the timing of the estimated break of each syllable, and the sounding time calculating means calculates the sounding time of the synthetic voice. In the calculation, the relationship between the display mode such as the display size of the synthetic face moving image of the person on the screen and the time required for the display is obtained in advance, and the delay time corresponding to each voice is corrected based on the above relationship. Composed.

Description

Detailed Description of the Invention

[0001]

INDUSTRIAL APPLICABILITY According to the present invention, a message can be transmitted to the other party only by sending text data, using a synthetic moving image of a face spoken by the sender and a synthesized voice as if the sender were a TV phone. The present invention relates to a synthetic image display system applicable to AV (audio / video) e-mails, etc., and particularly to a synthetic image display system capable of accurately synchronizing a synthetic voice and a mouth movement of the synthetic image.

[0002]

2. Description of the Related Art A technique for freely generating and pronouncing synthetic speech corresponding to arbitrary text information is called "regular speech synthesis", and a regular speech synthesizing device for realizing this is already made. There is. This rule speech synthesis technology is applied in various fields to improve the interface between human and machine. Also, in recent years, similar to voice synthesis, a technique has been developed that generates a moving image of a person including the movement of the mouth when speaking it from arbitrary text information by analyzing the text information. A more natural interface can be realized by combining the above with the above speech synthesis technology.

For example, when such a technique of synthesizing a voice and a facial moving image is applied to an electronic mail, a data file such as a face image of the mail sender is prepared in advance on the receiving side, so that the screen of the receiving side is conventionally used. It is possible to convey to the recipient an expressive message such as a moving image of the face of the sender of the mail appearing and reading it aloud with a synthetic voice in response to the e-mail whose text was only displayed in.

FIG. 3 shows an example of the configuration of a voice / moving image output apparatus for synthesizing and outputting a voice and a face moving image based on such text information. In FIG. 3, 1 is a sentence (text)
This is a sentence decomposition unit where information is input. This sentence decomposition unit 1
Analyzes the input sentence information, generates pronunciation control data for voice output, and generates the regular voice synthesis unit 2 and the voice / mouth conversion unit 3.
Output to. For example, when a sentence of "I'm home" is input as the sentence information, this is changed to "T, A, D, A, I,
It is decomposed into vowel and consonant phoneme data of "M, A" and output.

The regular voice synthesizing unit 2 is a device for generating and outputting a synthetic voice for reading out a sentence based on pronunciation control data for the arbitrary sentence.

The voice / mouth conversion unit 3 is a device for converting phoneme data of an arbitrary sentence into a series of mouth codes for representing a series of mouth movements when the sentence is pronounced. As the mouthpiece code, for example, A (vowel a), I
(Vowel a), U (vowel u), E (vowel d), O
There are seven types of (vowel vowel), S (consonant), and C (closed mouth), and mouth-shaped images for uttering them are prepared in advance corresponding to each mouth-shaped code. For example, when the above-mentioned sentence "Itaima" is input as the sentence information, based on this phoneme data "TADAIMA", "T" → mouth code S, "A" → mouth code A, "D" → mouth code S,
“I” → mouth shape code I, “M” → mouth shape code C, and “A” → mouth shape code A are respectively assigned and output to the image display control unit 5 as a series of mouth shape codes.

Composite image data is filed in the image memory 6. As the composite image data, one file is obtained by collecting the one-frame shoulder image of the speaker and the data of the seven types of mouth region images corresponding to the above seven types of mouth shape codes synthesized based on the image. I am trying.

The pronunciation time calculation unit 4 calculates the time until each syllable is pronounced when synthesizing a voice based on the phoneme data from the sentence decomposition unit 1 using the same algorithm as the regular voice synthesis unit 2. To do. That is, when the input sentence is synthesized by the regular speech synthesizer 2 and is output as a pronunciation, the timing of the break of each syllable constituting the sentence is estimated starting from the beginning of the sentence. And outputs the result to the image display controller 5.

The image display control unit 5 selects, from the image memory 6, a mouthpiece image corresponding to the mouthpiece code of the corresponding syllable when the sounding timing of each syllable arrives, based on the timing signal from the sounding time calculation unit 4. The image display is controlled so that the image is output after being output. That is, the synchronization control is performed so that the movement of the speaker's mouth displayed on the screen matches the voice produced by the regular voice synthesizing unit 2, that is, the synthesized voice and the face moving image can be synchronized. Is.

The parameter input unit 7 has a voice quality of the voice synthesized by the regular voice synthesis unit 2, a display location of the face moving image on the screen,
This is a part for inputting various parameters such as display magnification using a keyboard or the like. Parameters regarding synthetic voice are passed to the regular voice synthesizing unit 2, and parameters regarding face moving images are passed to the image display control unit 5 and the image memory 6. Be done.

The operation of the apparatus thus configured will be described. When the sentence information is input, the sentence information is analyzed by the sentence disassembling unit 1 and the phoneme data is collectively passed to the regular voice synthesizing unit 2 and output as a synthetic voice. In parallel with this sounding operation, the phoneme data is converted into a series of mouth code by the voice / mouth conversion unit 3. In addition, the pronunciation time calculation unit 4 estimates the time of each syllable break from the phoneme data, and passes this time data to the image display control unit 5. The image display control unit 5 matches the timing of the mouthpiece code with the sounding timing of each syllable, and corresponds to the mouthpiece code obtained by the voice / mouthpiece conversion unit 3 from the images of each mouthpiece code developed on the image memory 6. The face moving image data is transferred to the VRAM, and the face moving image of the speaker is displayed on the screen of the display device via the VRAM. As a result, the text information is transmitted to the receiver as a message by the synthetic voice that actually pronounces it and the face moving image of the speaker whose timing of mouth movement is in the synthetic voice.

The apparatus shown in FIG. 3 uses a conventional small-sized speech synthesizing unit for the regular speech synthesizing unit 2 and uses a personal computer or the like for the other portions to realize a compact and economical system. can do.

In such a voice / face moving image output apparatus, parameters such as voice quality, image display location on the screen, and display magnification are previously set as initial values to the display system when generating the synthesized voice and the face moving image. The set one (the one previously input by the parameter input unit 7) is used. In this way, when the voice quality of the synthesized voice and the generation mode of the face moving image are preset on the receiving side, the person and voice quality of the face image registered in advance are not balanced with respect to the message, for example. In such a case, it will give an unnatural feeling to the viewer.

Also, as typified by the case where this device is used for AV e-mail, etc., the person who created the text information is different from the person who actually displays the text information by voice and moving images. In this case, the voice quality and image size desired by the person who created the text information are not always displayed and displayed as an image on the receiving side, and as a result, an impression completely different from the intention of the person on the sending side is obtained. It may be given to the recipient.

Therefore, when a synthetic image of a face is created on the side of creating text information, the voice quality of the synthetic voice on the display side, the display magnification when displaying the synthetic moving image, the display position, and other various parameters are synthesized. To be passed to the display side so that the display side can synthesize the face and the moving image according to the display magnification and display position of the synthesized image intended by the creator and the voice quality of the synthesized voice. Is proposed.

[0016]

In the system described above,
In order to match the voice with the movement of the mouth of the face on the display side, the image display control unit controls to match the movement of the mouth of the face image with the pronunciation timing of each syllable of the voice generated by the regular voice synthesis unit. It is carried out.

By the way, when there is no means for measuring the actual time of the timing at which the face image corresponding to each syllable should be displayed in the environment where the system is operated (personal computer in this example) like the above-mentioned system, A counter or the like is realized on software, and by using the counter or the like, the process of switching the mouth-shaped image is delayed by a necessary time interval until the sounding timing of each syllable, and the mouth-shaped movement is matched with the synthesized voice.

That is, as shown in FIG. 4, for example, the time for uttering a syllable (a) or (ta) is uniquely determined by the rule voice synthesizing unit 2, but each sounding time T _TOTAL is examined. Looking at the pronunciation time T _TOTAL
Is the sum of the data processing time T _P , the transfer time T _{T of the} composite image data to the VRAM, and the delay time T _D different for each syllable. Of these, the data processing time T _P is always a constant value, and the transfer time T _T to the VRAM is a constant value for any syllable if the display magnification is the same. Therefore, the difference in sounding time for each syllable is adjusted by the delay time T _D preset for each syllable. That is, the delay time T _{D of the} syllable is predetermined for each syllable,
The sounding time calculation unit 4 adds the delay time T _{D of the} syllable to the data processing time T _P and the transfer time T _T for each syllable to calculate the sounding time T _{TOTAL of the} syllable.

However, the transfer time T _T of the composite image data to the VRAM changes depending on the display magnification because the transfer data amount changes when the display magnification (display size) of the display image on the screen changes. T _T will also change. As a result, when the system is configured such that the display magnification and the like can be arbitrarily changed according to the intention of the sender, such as the above-mentioned AV electronic mail system, the display magnification of the delay time T _D corresponding to each syllable is increased. However, if the values are the same, the pronunciation time T _TOTAL of each syllable calculated by the pronunciation time calculation unit 4 changes depending on the display magnification and the like, and thus perfect synchronization between the synthetic voice and the synthetic image cannot be obtained. I will end up. In order to prevent this, the delay time corresponding to each syllable may be predetermined and held for each display magnification.
In that case, there is a problem that the amount of held data becomes large.

The present invention has been made in view of the above problems, and enables the pronunciation time of each syllable to be correctly calculated even when the display size and the display magnification of the screen are changed, so that the natural time synchronized with the voice is obtained. It is intended to be able to generate a face moving image of mouth movement.

[0021]

FIG. 1 is a diagram illustrating the principle of the present invention. In the present invention, as one form,
A voice synthesizing means 31 for synthesizing and outputting a synthetic voice based on the sentence information, a converting means 32 for converting the sentence information into a series of mouth-shaped codes representing a series of mouth movements when the sentence information is uttered, and the sentence information. On the basis of the above, the pronunciation time calculation means 3 for estimating the pronunciation time of each syllable of the synthetic speech output from the speech synthesis means 31 and estimating the timing of the break of each syllable.
3 and a display control means 34 for performing control to switch the display image to the mouth shape image corresponding to the mouth shape code from the conversion means 32 at the timing of the break of each syllable estimated by the pronunciation time calculation means 33. In calculating the pronunciation time of each syllable of the synthetic voice, the pronunciation time calculation means 33 preliminarily relates the relationship between the display mode such as the display size of the synthetic face moving image of the person on the screen and the time required for the display processing. A synthetic image display system is provided, which is configured to correct the delay time corresponding to each voice for calculating the pronunciation time of each syllable based on the above relationship.

According to another aspect of the present invention, in a synthetic image display system for generating a synthetic voice corresponding to arbitrary sentence information and a synthetic facial moving image of a person whose mouth moves in accordance with the synthetic voice, In calculating the pronunciation time of each syllable of the synthesized voice in order to switch the mouth-shaped image of the synthesized face moving image at the timing of the break of each syllable, the display mode such as the display size of the synthetic face moving image of the person on the screen and A composition characterized in that the delay time corresponding to the type of each syllable for calculating the pronunciation time of each syllable is corrected in advance based on the above relation, An image display system is provided.

The correction of the delay time described above is performed such that the delay time is the first delay time that is uniquely determined depending on the image display mode.
It can be performed by dividing into a second delay time that is uniquely determined depending on the type of syllable, a first delay time is obtained according to the image display mode, and a second delay time corresponding to each syllable is added to it. it can.

[0024]

The delay time used when calculating the pronunciation time of each syllable is the first delay time that is uniquely determined depending on the mode such as the image display magnification and the second delay time that is uniquely determined depending on the type of syllable. And the delay time. As a result, when the image display magnification is determined, the first delay time is determined accordingly, and the second delay time corresponding to each syllable is added thereto, whereby the delay time of each syllable is determined. If the sounding time of each syllable is obtained using this delay time, this sounding time becomes a unique value for each syllable and does not depend on the image display magnification or the like. Therefore, the voice and the moving image can be synchronized without depending on the size of the image or the like.

[0025]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 2 is a diagram for explaining the calculation process of the sound generation time in the composite image display system as one embodiment of the present invention. The hardware configuration of the voice / facial moving image output device as this composite image display system is almost the same as that shown in FIG. 3, but the difference is that the pronunciation time calculation unit 4 calculates the pronunciation time as shown in FIG. It depends on the method. Less than,
This calculation method will be described.

As shown in the figure, as a phoneme (a),
It is assumed that there are (a), (c), and (d). These _{tone generation} times T _TOTAL are different from each other and are adjusted by the delay time T _D. That is, the pronunciation time T of each phoneme
_TOTAL consists sum of processing time T _P and VRAM transfer time T _T and the delay time T _D, the sounding time T _TOTAL is a different value for each phoneme by varying the delay time T _D for each phoneme.

Of the above times, the processing time T _P is a constant value that does not depend on the image display magnification or the like. Transfer time T _T to VRAM is a value whose value varies depending on the display magnification of the image, for example, as shown in FIG. 2, the transfer time when a display magnification a is at T _{T (a)} If so, the transfer time when the display magnification is b is T _{T (b)} .

Now, assuming the maximum transfer time T _{T (max)} when the display magnification is maximum, a constant time T _C (time between A and B in FIG. 2 ₎ which is a little larger than this value is assumed. Set. Then, the delay time T _D is divided into two using this constant time T _C. That is, the delay time T _D is divided into the delay time T _D1 included in the constant time T _C and the delay time T _D2 deviating from the delay time T _D1 . In this case, T _D = T _D1 + T _D2 .

In this way, when the delay time T _D is divided into T _D2 and T _D2 , the delay time T _D1 depends on the display magnification (display size) of the image on the screen regardless of the type of phoneme. It's time to change. That is, the constant time T _C is VR
Since it is the sum of the AM transfer time T _T and the delay time T _D1 , T
_D1 = _{T C} -T T next, these transfer time T _T is because a value which changes the display magnification without depending on the type of phonemes. On the other hand, the delay time T _D2 is a time when its value changes depending on the type of phoneme, and does not change depending on the display magnification.

[0030] Thus setting the delay time as described above, the display magnification of the facial image picture on the screen is determined, it determines the transfer time T _T to VRAM in accordance with the display magnification, delayed by the transfer time T _T The time T _D1 is uniquely determined. For example, if the transfer time T _T the corresponding display magnification a _(a), the delay time depending on display magnification T _{D1 (a) is, T D1 (a) = T} C -T T (a) , and the if the transfer time is a T _{T (b)} the corresponding display magnification b, the delay time depending on display magnification T _{D1 (b) is, T D1 (b) = T} C -
It becomes T _{T (b)} .

Therefore, by adding the delay time T _D2 corresponding to each phoneme to this delay time T _D1 , the delay time T _D can be determined for each phoneme. In the case of this method, the data to be retained is the delay time T corresponding to each phoneme.
_{Since only D2} and the transfer time T _T to the VRAM corresponding to each display magnification are required, the amount of held data can be reduced.

Various modifications are possible in carrying out the present invention. For example, in the above embodiment, the mouth shape is 7
Although the case of using images of different types has been described, the present invention is not limited to this, and the types of images of this mouth shape may be further increased in order to synthesize a more natural movement of the mouth. Further, in the above-described embodiment, the movement of the mouth area is taken up as the moving portion of the face image to be synthesized on the receiving side, but the present invention is not limited to this. By also changing the above, a more expressive AV message can be sent to the receiving side.

Further, although the present invention is applied to the voice / face moving image output system in the above-mentioned embodiments, the present invention is not limited to this, and it can be applied to AV electronic mails, or voiced voices can be used to produce voiced voices in real time. If the phoneme can be recognized, it can be applied to a service such as a pseudo-videophone in which the facial expression of the talker's face can be displayed on the receiver side as a moving image only by making a normal call.

[0034]

As described above, according to the present invention, even if the display mode such as the screen magnification changes in the synthetic image display system, it is not affected by the change and the natural speech synchronized with the synthetic voice is obtained. A moving face moving image can be combined and displayed.

[Brief description of drawings]

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 is a diagram illustrating a calculation process of a sounding time of a voice / moving image output device in a composite image display system as an example of the present invention.

FIG. 3 is a diagram showing a conventional audio / moving image output apparatus.

FIG. 4 is a diagram illustrating the concept of sound generation time in a conventional device.

[Explanation of symbols]

 1 Text Decomposing Unit 2 Ruled Speech Synthesizing Unit 3 Voice / Variety Converting Unit 4 Sounding Time Calculation Unit 5 Image Display Control Unit 6 Image Memory 7 Parameter Input Unit

Claims (3)

[Claims] 1. A voice synthesizing means (31) for synthesizing and outputting a synthetic voice based on sentence information, and converting the sentence information into a series of mouth-shaped codes representing a series of mouth-shaped movements when the sentence information is uttered. Converting means (3
2), and a pronunciation time calculation means (33) for estimating the timing of each syllable of the synthetic speech output from the speech synthesis means on the basis of the sentence information to estimate the break timing of each syllable; Display control means (34) for controlling switching of the display image to the mouth shape image corresponding to the mouth shape code from the converting means at the timing of the break of each syllable estimated by the time calculating means.
In the synthetic image display system including the above, the pronunciation time calculation means calculates the pronunciation time of each syllable of the synthesized voice, and the display mode of the synthetic face moving image of the person on the screen and the time required for the display process. Is obtained in advance, and the delay time corresponding to each voice for calculating the pronunciation time of each syllable is corrected based on the above relation. 2. A synthetic image display system for generating a synthetic voice corresponding to arbitrary text information and a synthetic facial moving image of a person whose mouth moves in accordance with the synthetic voice, in a timing of a break of each syllable of the synthetic voice. In calculating the pronunciation time of each syllable of the synthesized voice in order to switch the mouth-shaped image of the synthesized face moving image, the relationship between the display mode of the person's synthesized facial moving image on the screen and the time required for the display processing is described. A synthetic image display system, characterized in that the delay time corresponding to the type of each syllable for calculating the pronunciation time of each syllable is corrected based on the above relationship. 3. The correction of the delay time is performed such that the delay time is divided into a first delay time that is uniquely determined depending on an image display mode and a second delay time that is uniquely determined depending on a syllable type. 3. The composite image according to claim 1 or 2, wherein the first delay time is calculated according to the image display mode, and the second delay time corresponding to each syllable is added to the first delay time. Display system.