DE60131521T2 - Method and device for controlling the operation of a device or a system, and system having such a device and computer program for carrying out the method - Google Patents

Description

The The invention relates to the field of emotion synthesis, in which a Emotion is simulated, for example in a speech signal, and In particular, it has the provision of a new degree of freedom in controlling the possibilities to the goal offered by emotion synthesis systems and algorithms become.

in the Case of a voice data to be forwarded or transmitted Emotion can this understandable words or incomprehensible expressions or sounds, like a murmur or animal-like sounds.

A Such emotion synthesis finds applications in animation of communication objects, such as robotic animals, among others humanlike Beings, interactive machines, training training devices, Systems for reading texts, creating soundtracks for movies, in animations, etc.

1 illustrates the basic concept of a classical voiced emotion synthesis system 2 based on an emotion simulation algorithm.

The system receives at an input 4 Speech data Vin, which is typically neutral, and it generates at an output 6 Voice data Vout showing an emotion-emphasized form of the input-side voice data Vin. The speech data is typically in the form of a stream of data elements each corresponding to a sound element such as a phoneme or a syllable. A data item generally specifies one or more values concerning the pitch and / or intensity and / or duration of the corresponding clipping element. The speech-emotion synthesis works by performing algorithm steps that modify at least one of these values in a predetermined manner to produce the required emotion.

The emotion simulation algorithm is determined by a series of input parameters P1, P2, P3, ..., PN, referred to as emotion adjustment parameters, and those at an appropriate input 8th of the system 2 be supplied. These parameters are usually numerical values and possibly indicators for parameterization of the emotion simulation algorithm, and they are generally determined empirically.

each Emotional E to be displayed has its specific set of emotion adjustment parameters on. In the example, the values are the emotion adjustment parameters P1, P2, P3, ..., PN given with C1, C2, C3, ..., CN for rest, with A1, A2, A3, ..., AN for annoyance or trouble, with H1, H2, H3, ..., HN for Luck, with S1, S2, S3, ..., SN for Sadness.

It exist as well Emotional simulation algorithms that are entirely generative insofar as they do not implement an input stream of speech data, but generate the emotion-stressed voice data Vout internally. These Systems also use series of parameters P1, P2, P3, ..., PN analogous to the system described above to determine the type of the To determine emotion.

As whatever the emotion simulation algorithm system looks like Meanwhile these parametrizations effectively synthesize the corresponding emotions can, a need to do so, in addition to be able, a size or a measure of one to be able to assign synthesized emotion E. That's the way it is, for example advantageous to be able to a certain emotion E a range of a quantity or a measure to generate the emotion to be represented in the speech data Vout, for example, from easy to intense.

One possibility would be to generate empirically determined additional sets of parameters for a particular emotion, each parameter corresponding to one degree of the emotion to be represented. However, such a solution suffers from significant disadvantages:
the development of the additional rates would be extremely labor-intensive,
their storage in an application would occupy a portion of the memory, which could be detrimental to a memory-constrained device, such as a small robot animal,
the management and processing of the additional sets consumes significant processing power,
and from the standpoint of performance it would not be possible
Consider variations, the gentle changes in the strength or level of emotion cause.

One Example of a Speech Synthesizer with Emotion Modeling is in the publication: Sato et al., "Emotion Modeling in Speech Production Using Emotion Space ", IEEE Int'1 Workshop on Robot and Human Communication, 1996, pp. 472-477. The described therein Emotions-speech synthesizer creates an emotion content by having the neutral language in the level of their time, pitch and intensity is played.

The invention proposes, according to a first object, a method for controlling the operation of an apparatus for synthesizing an emotion which is transmitted into a sound, the apparatus having at least one input for a parameter whose value is used to determine a type of emotion to be relayed set. The method comprises the steps:
Programming the input signal (s) with a parameterization to generate a particular type of emotion and
Lending a variability in a measure of the particular type of emotion to be passed on.

This Method is characterized in that the variability in terms the size or of measure of a certain type of emotion is obtained by that within a certain control range at least one parameter (Pi), which is used to determine the type of emotion, a deflection or Deviation from its initial default value.

Preferably is at least one variable parameter corresponding to a local one Model over the control area made variable, the model is based on a measure or a quantity refers to the emotion control variable variable variable. As a result, the measure or the quantity the emotion control variable used to variable a value of the variable parameter set up or set.

The local model can be based on the assumption that - while different Sets of one or more parameter values produce identifiable emotions - a selected set of the parameter / s Parameter values sufficient for determining a particular type of emotion is stable to local excursions or deviations from the parameter value / To allow parameter values without an uncontrolled change to evoke the corresponding emotion in nature. As it seems the change occurs in the measure or in quantity the emotion. The particular control area will then be within of the range of local deflections or deviations.

The Model is advantageously a local linear model for the control area and for a certain or predetermined type of emotion, the variable Parameter is formed so that it is linear over the control range through the size of the emotion control variable changes linearly.

VPi

den Wert des fraglichen variablen Parameters,

A und B

Werte, die durch den Steuerungsbereich zugelassen sind, und

δ

die Größe der Emotions-Steuerungsvariablen.

In a preferred embodiment, the magnitude of the emotion control variable (δ) modifies the variable parameter according to a relationship given by the formula: VPi = A + δB; mean here

VPi

the value of the variable parameter in question,

A and B

Values permitted by the control area, and

δ

the size of the emotion control variable.

Preferably A is a value within the control area; this is the Size or the measure of Emotions control variables in an interval variable, which contains the value zero.

Of the Value A can be essentially the mean of the control range and the size of the emotion control variables may be variable in an interval whose center value is zero.

The Size or the measure of Emotions control variables are preferably in one interval from 1 to +1 variable.

In the preferred embodiment, the value B is determined by:
B = (Eimax - A) or by
B = (Eimin + A).

Here, Eimax means the value of the input parameter for generating the maximum size or the maximum measure of the emotion type to be transmitted in the control area, and
Eimin means the value of the parameter for generating the minimum size of the emotion type to be transmitted in the control area.

Of the Value A may be equal to the default parameter originally used for Determination of a to be transferred Emotion type has been specified.

Of the Value Eimax or Eimin can be determined by deflecting the default parameter value, originally specified been to transfer one Determine emotion type and by determining a maximum displacement in an increase or Acceptance direction leading to a desired limit for the size or the measure of Emotion leads, which or which to transmit through the control area is to be determined experimentally.

The Invention allows it, the same size or same measure of a Emotions control variables too to collectively use a variety of variable parameters of the Emotions-synthesizer to set up.

According to one second aspect, the invention relates to a device for Controlling the operation of a system for synthesizing an emotion, which translate into a sound with the system having at least one input for a parameter, its value is used to determine an emotion type to be transmitted becomes. This device is characterized in that it has a Means for programming the input signal (s) Input signals with a parameterization for generating a determine Emotion type and change facilities to transfer a variability in a size or in a measure of to be transferred Contains emotion type, and it is characterized in that the changing means are capable are within a certain control range at least one Parameter used to define an emotion type undergo a deflection from its initial default value.

The optional features of the above related to the first aspect (the method) invention are with the necessary changes in the second aspect (device) according to applicable and should be short half not be repeated.

According to one third aspect, the invention relates to the use of the above Device for setting an emotion size in a device for synthesizing one in a sound transmitted Emotion.

According to one fourth aspect, the invention relates to a system which an emotion synthesizer, the at least one entrance for has the inclusion of at least one parameter whose value for Determination of a to be transferred Emotionstyps is used, and a device according to the third aspect includes, the work-wise so connected is that it gives a variable to the at least one input, thereby a variability in terms of a size of a to be transferred Emotion type to transfer.

According to one fifth Aspect, the invention relates to a computer program, which by a computer executable Provides commands. This program then initiates when it is in a data processor is loaded, the data processor, accordingly to work the above procedure. The computer program can be in one record carrier of any suitable form be.

The The invention and its advantages will become apparent with reference to the accompanying drawings from the following description of the preferred embodiments closer which are given purely as non-limiting examples.

In show the drawings

1 Already described, a classic emotion simulation algorithm of the type that implements neutral speech data,

2 10 is a block diagram of a magnitude of an emotion alteration system according to a preferred embodiment of the invention;

3 10 is a block diagram of an example of an operator-based emotion generation system that determines the size of the emotion change system according to FIG 2 realized,

4 a schematic representation of pitch operators, by the system according to 3 be used,

5 a schematic representation of intensity operators, optionally in the system according to 3 can be used

6 a schematic representation of persistent operators by the system according to 3 be used, and

7A and 7B a flow chart of an emotion generation process, which is based on syllable data by the system according to 3 is carried out, where 7B a continuation of 7A represents.

2 illustrates the functional units and the operation of a size of a emotion change system 10 according to a preferred embodiment of the invention, wherein the operation is in conjunction with a speech-based emotion simulation algorithm system 12 he follows. In the example, the latter system is of the generative type, that is, it has its own means for generating speech data transmitting a particular emotion E. Of course, embodiment 10 may also work with any other type of emotion simulation algorithm system, such as that described with reference to FIG 1 has been described, in which a stream of neutral speech data is supplied to an input. These two types of emotion simulation algorithm systems as well as other types of systems with which the embodiment can operate are known in the art. More information regarding speech-based emotion simulation algorithms and systems can be found inter alia in: Cahn, J. (1990) "The Generation of Affected in Synthesised Speech", Journal of the 1/0 Voice American Society, 8: 1-19; Iriondo I., et al. (2000) "Validation of an Acoustical Modeling of Emotional Expression in Spanish using speech synthesis techniques", Proceedings of ISCA workshop on speech and emotion; Edington MD (1997) "Investigating the limitations of concatenative speech synthesis", Proceedings of EuroSpeech '97, Rhodes, Greece; Iida A., et al (2000) "A speech synthesis system with emotion for assisting communication", ISCA workshop on speech and emotion.

In addition, emotion synthesis methods and apparatus are described in the applicant's copending European patent applications, the priority of which is claimed in the present application: European Published Applications EP-A-1256931 , filed on 11.05.2001 and EP-A-1256932 , filed on 13.07.2001.

The emotion simulation algorithm system 12 uses a number N of emotion adjustment parameters P1, P2, P3, ..., PN (indicated generally by P) to generate a particular emotion E, as described above with reference to FIG 1 has been explained. The number N of these parameters can vary considerably from one algorithm to another, typically from 1 to 16, or considerably more. These parameters P are empirically determined numerical values or indicators that are utilized in computation or decision steps of the algorithm. You can enter the emotion simulation algorithm system 12 either loaded by a dedicated interface or by a parameter loader routine. In the example, the insertion of the parameter P is symbolically represented by lines in the system 12 occur, with a suitable interface or a suitable charging unit are integrated to allow the introduction of these parameters from the outside.

The emotion simulation algorithm system 12 may thus cause different types of emotions E, such as rest, anger, happiness, sadness, etc., through a suitable series of N values for the respective parameters P1, P2, P3, ..., PN. In the case considered, the system is 12 programmed first for the following parameterization: P1 = E1, P2 = E2, P3 = E3, ..., PN = EN to generate a certain emotion E. The values E1-EN are already determined in order to achieve the emotion E.

The emotion change system 10 works with respect to the size of the emotion so that a change is imposed on these values E1-EN according to a linear model. In other words, it is assumed that a linear - or progressive - change of E1-EN is a progres sive change in the response of the emotion simulation algorithm system 12 causes. As remarkably discovered by the applicant, the response in question will cause a change in the magnitude, ie the intensity of the emotion E, at least with respect to a certain range of variation of the values E1-EN.

To evoke the above variations in the values E1-EN, first, a range of possible change is determined for each of these values. For a given parameter Pi (i is any integer between 1 and N inclusive) becomes a study of the emotion simulation algorithm system 12 during which a parameter Pi is subjected to a deflection from its initial default value Ei to a value Eimax determined to correspond to a maximum intensity of the emotion E. This value Eimax is determined experimentally. It will generally correspond to a value above which the parameter either no longer contributes to a significant increase in the intensity of the emotion E (ie a saturation occurs), or below which the emotion type E is modified or disfigured. It should be noted that the value Eimax may be either greater or less than the standard value Ei: depending on the parameter Pi, the increase in the intensity of the emotion may result from an increase or decrease in the standard value Ei.

The Determination of the maximum intensity value Eimax for the parameter Pi can either be made by all the other parameters at the initial default value be held or that some or all of the other parameters according to a knowledge of the interaction of the different Parameter P1-PN be varied.

The The above procedure follows a local model of controllable behavior around the standard parameter values Pi, which assumed to be sufficiently stable are local deviations from their initially chosen value to enable and a controlled change to achieve within the emotion to which they belong. The particular control area is then within the range the local deflections or deviations.

After this initial setup phase, a set of maximum intensity parameter values E1max, E2max, E3max, ..., ENmax are obtained, each corresponding to the maximum intensity of the emotion E determined by the respective parameter P1, P2, P3, ..., PN is caused. These maximum intensity parameter values are stored in a memory unit 14 stored in association with the corresponding default initial parameter value Ei. Thus, the storage unit arranges 14 with respect to a parameter Pi two values to: Ei and Eimax. In a typical application, the above procedure is performed for each type of emotion E to be processed by the emotion simulation algorithm unit 12 to be generated and with respect to which a size or a measure of the relevant emotion must be controlled. Each emotion E has its corresponding associated set of values Ei and Eimax stored in the memory unit 14 are stored.

The in the storage unit 14 stored values are passed through a parameter generator unit 16 exploited for a variable parameter whose function consists of the parameters P1-PN of the emotion simulation algorithm system 12 to replace VP1-VPN with appropriate variable parameters.

The parameter generator unit 16 for the variable parameter generates each variable parameter VPi based on a common control variable and the associated values Ei and Eimax according to the following formula: VPi = Ei + δ (Eimax - Ei) (1).

It can be found that this equation is a linear model with a standard form y = mx + c, where y is given by VPi, where m is given by (Eimax - Ei), where x is given with δ and where c is given with egg.

The way through the parameter generator unit or parameter generation unit 16 variable parameter values generated for variable parameters VP1-VPN are at the respective outputs 17-1 to 17-N delivered with the respective parameter inputs 13-1 to 13-N of the emotion simulation algorithm system 12 are connected. Of course, the schematic representation of these connections from the parameter generator unit 16 for variable parameters to the emotion simulation algorithm system 12 may be embodied in any suitable form: parallel or serial data bus, wireless connection, etc. using any suitable data transmission protocol. Loading the variable para meter VP can through a routine in the plane of the emotion simulation algorithm system 12 to be controlled.

The control variable δ is in the range of -1 to +1 inclusive. Their value is determined by an emotion size selector 18 which may be a user-accessible interface or electronic control unit which operates in accordance with a program which determines the magnitude of the emotion to be generated, such as a function of an external command indicating the size or dimension in question Indicates measure, or automatically depending on the environment, the history, the context, etc. of the operation, such as a robot animal or the like.

In the figure, the range of change of δ is a scale 20 represented, along which a pointer 22 can be shifted to set the required value of δ in the interval [-1, 1]. In a case where the level of emotion is controllable by a user, the scale can 20 and the pointer 22 may be embodied by a graphics interface to be displayed as cursors on a monitor display screen of a computer or form part of a robot animal. The pointer 22 can then through a keyboard, buttons, a mouse or the like. be displaceable.

The Scale can also determined by a potentiometer or similar variable component be.

The Values of δ can be in Basically continuously or incrementally incrementally over the Range [-1, +1] be.

The one by the pointer 20 certain value of δ is determined by an emotion selection unit 18 generated and sent to an entrance 22 the parameter generation unit 16 for variable parameters capable of picking up the control variable to be inserted into the above formula (1).

The application of a scale standardized in the interval [-1, +1] is advantageous in that it simplifies the management of the values generated by the parameter generation unit 16 be used for variable parameters. More specifically, this allows the values of the storage unit 14 to use directly as contained in the formula (1) without the requirement of introducing a scaling factor. However, other intervals may be considered for the range of δ, including ranges that are asymmetric with respect to the position δ = 0 (for which formula (1) returns the default parameter setting VPi = Ei). The implementation of formula (1) makes it possible to calculate the range of variable parameter VPi values from the minimum emotion intensity value Eimin = 2Ei-Eimax (in the case of δ = -1) to Eimax (in the case of δ = +1 ). This numerical value for Eimin has been found to be within the expected range of emotion that can be controlled by such a solution based on a linear model. In other words, it has been found that the value of Eimin thus obtained is indeed the acceptable lowest level of emotion to be expressed at a standard parameter setting Ei (corresponding to δ = 0) effectively given to the term to be a substantially middle-range measure of emotion adjustment. However, it may be provided to select any midrange value Emr which is not necessarily equal to Ei. Formula (1) would then be given more generally by VPi = Emr + δ (Eimax-Emr).

The embodiment is noteworthy as the same variable δ for changing each of the N variable parameter values VPi for the emotion simulation algorithm system 12 serves while the respective ranges of values for the parameters P1-PN are covered.

It It should be noted that the amendment law accordingly of the formula (1) is capable of managing both parameters whose Increase value is to an increased degree To evoke emotion, as well as to manage parameters whose value to downsize to evoke an increased level of emotion. In the latter case, the value Eimax in question becomes smaller be as an egg. The parenthetical term of formula (1) then becomes be negative, with a size that increases when that is through the variable δ chosen emotion measure in the range between 0 and +1 increases. For an increasing negative magnitude δ becomes the Term δ (Eimax - Ei) positive and carries to enlarge VPi and thus to reduce the emotion at.

Moreover, for all the values of δ, the variable parameters VP will each have the same relative position in their respective area, which will be determined by the emotion measure selector 14 here well-balanced change and is homogeneous across the variable parameters.

Of course, the embodiment allows many variations including: forming the number of parameters P as variable parameters VP. It may be considered that not all N parameters P are controlled, but that only one subset of one or more parameters is controlled by the parameter generation unit 16 for variable parameters is accessed; the remaining parameters remain at their default value;
the choice of formula (1), both in terms of their form and in terms of values. The choice of constant Ei and Eimax in the formula (1) is advantageous in that egg is already known from the outset and as Eimax is simply the experimentally determined value, which greatly simplifies the realization. Other computational operations using these values or other values may, however, be considered. For example, formula (1) may be adjusted to accommodate an Eimin value that is independently determined and that is not subordinate to the value of Eimax. In this case, formula (1) can be reworded as: VPi = EI + δ (Eimin + Ei) (1 ')

Of the Value of Eimin can be for each parameter must be determined experimentally to be in an analogue Way to be made variable as described above is: Eimin is referred to as the lowest value useful emotional dimension, below of which either no practically usable reduction in emotional intensity or a Disfigurement of the emotion type present. The memory will be the Save values Eimin instead of Eimax.

In addition, the mid-range value may be a value different from the default value Ei;
the choice of control value δ and its interval, as discussed above. In addition, other more complex variants may be considered that use more than one controllable variable;
the choice of the emotion simulation algorithm as explained above. In fact, it should be understood that the teachings of the invention are fairly universal in terms of emotion simulation algorithms. These teachings may also be considered as appropriate for other simulation systems,
for example, to create variability for parameters that determine facial expressions to express speech, emotions, etc.

The The above teachings are applicable to all emotions E, which are simulated by emotion simulation algorithms: rest, Luck or Happiness, anger, sadness, timidity, Etc.

Two examples will now be given to illustrate how an emotion simulation algorithm system measures a magnitude of the emotion change system 10 can use, as with reference to 2 is described.

example 1: A robot animal capable of modulating sounds to Expressing that are generated by a speech synthesizer, which contains a number of input parameters, one through the language be expressed Set emotion state.

The example is based on the content of the earlier published application EP-A-1256931 by the applicant, filed on 11 May 2001, entitled 'Method and apparatus for voice synthesis and robot apparatus', claiming priority.

Of the Emotions synthesis algorithm is based on the idea that one Emotion can be expressed in a feature space that consists of a Excitation component and a valence component. So be for example, trouble, Sadness, happiness and cosiness in particular regions within the excitation valence trait space shown.

The algorithm refers to tables representing a set of parameters P which include at least the duration (DUR), the pitch (PITCH) and the sound (VOLUME) of a phoneme defined in advance for each basic emotion. These parameters are numeric values or states (such as "rise" or "fall"). These state parameters can be maintained per default, and they can not be adjusted by the level of the emotion change system 10 to be controlled.

Table I below shows an example of the parameters and their attached values for the emotion "cheerfulness". The mentioned parameters are valid for incomprehensible words of one or several syllables or Phonemes that are specified in terms of pitch characteristics, duration, contour, volume, etc. in recognized units. These characteristics are expressed in a formatted data structure recognized by the algorithm. Table I: Parameter settings for the emotion "happiness" characteristics parameter numeric value or state emphasizes last word right middle pitch 400 Hz pitch change 100 Hz maximum pitch 600 Hz medium duration 170 ms Time change 50 ms Probability of 0.3 (30%) Emphasis on a word Standard contour sloping Contour of the last word sloping volume 2 (specific units)

different Emotions become their own parameter values or states for the same Have characteristics.

The Robot animal in which this algorithm is involved is designed to make it from one set of parameter values to one other set of parameter values switches to the decision, the To represent emotion.

In this case, the parameters of the characteristics in the table I having numerical values are no longer fixed to a particular emotion, but they become variable parameters VP by using the measure of the emotion changing system 10 ,

For example, in the case of the mean pitch characteristic for the "cheerfulness" emotion, the default parameter value of 400 Hz becomes the value Ei in equation (1) for that parameter. A step is performed for determining i) the direction (increase / decrease) in which this value can be modified to produce a more intense representation of happiness. Then, a step ii) is performed to determine how far in the direction in question that parameter can be changed to usefully increase that intensity. This limit is Eimax in equation (1). In this way, all the necessary information for generating the variability scale for the variable parameter VPi of the characteristic in question is obtained. The same procedure is used for all other characteristics with which it is decided to set the parameter by the size of the emotion change system 10 to make a variable parameter VP.

example 2: A system capable of arriving at an emotion content Speech data understandable accordingly words or incomprehensible sounds to add in a neutral tone, so the added Emotion can be felt when the thus processed voice data be played or played back.

The example is based on the content of the earlier published application EP-A-1256932 by the applicant, filed on 13 July 2001, "Method and apparatus for synthesizing an emotion conveyed to a sound", from which the priority is claimed.

The system includes an emotion simulation algorithm system which, as in the case of 1 has an input for recording sound data and an output for outputting landforms in the same format, but with modified data values corresponding to the emotion to be expressed. The system may thus be effectively arranged along a chain between a sound data source and a sound data player, such as an interpolator plus a synthesizer, in a completely transparent manner.

The modification of the data values is performed by operators which are to be modified Influence values. Typically, the sound data is in the form of successive data elements, each of which corresponds to a sound element, such as a syllable or a phoneme, to be reproduced by a synthesizer. For example, a data item specifies the duration of the beat element and a pitch value or multiple pitch values that are present over that duration. The data element may also identify the syllable to be played, and an indication may be associated with whether the data element can be emphasized or not. For example, a data element for the syllable "be" may have the following data structure: "be": 100, P1, P2; P3, P4, P5 "The first number 100 expresses the duration in milliseconds The following five values (symbolized by P1-P5) indicate the pitch value (F0) at five corresponding and successive intervals during the duration in question.

different Types or types of possible Operators of the system call different modifications regarding the Data elements to which they are applied.

3 illustrates in a block diagram in functional terms, such as the emotion simulation algorithm system with the above emotion synthesizer 26 is integrated to produce variable emotion-weighted speech data in intensity.

The emotion simulation algorithm system 12 works by selectively applying the operators O to the syllable data resulting from a mood data file 28 are read out. Depending on their type, these operators can modify either the pitch data (pitch operator) or the syllable duration (duration operator) data. These modifications take place before an interpolator 30 instead of, for example, a voice data decoder 32 so that the interpolation with respect to the operator-modified values is carried out. As explained below, the modification is made to selectively transform a neutral form of speech selectively into a language that expresses a selected emotion (sadness, calm, happiness, anger) to a selected extent.

The basic operator forms are in an operator set library 34 stored in the on it selectively by an operator set configuration unit 36 can be accessed. The latter unit serves to create and parameterize the operators in accordance with current requirements. For this purpose, an operator parameterization unit 38 which determines the parameterization of the operators, in accordance with: i) the emotion to be expressed in language (rest, sadness, happiness, anger, etc.), ii) the extent - or intensity - of the emotion to be exercised, and iii ) the context of the syllable, as explained below. For the realization of the embodiment according to 2 are in the operations parameterization unit 38 the parameter generation unit 16 for variable parameters and the memory 14 of the emotion measure change system 10 included.

The emotion and the degree of emotion become the operator parameterization unit 38 through an emotion selection interface 40 supplied by command representing input signals accessible to a user U. For implementation of the embodiment, the emotion measurement device is included in this user interface 18 (please refer 2 ), the pointer 22 is a user-displaceable physical or electronic device. Among those of the interface unit 40 Accordingly, given instructions will be the variable δ. The emotion selection interface 40 may be in the form of a computer interface with menus and icons on a display screen that allow the user U to display all the necessary emotion characteristics as well as other operating parameters.

In the example, the context of the syllable that is operator-sensitive is given by: i) the position of the syllable in an expression or phrase when some operator sets are applied to only the first and last syllables of the phrase or sentence ii) whether the syllable refers to understandable phrases or unintelligible sounds (marbles, etc.), and iii) whether or not a given syllable is allowed to be emphasized or not, as in the mood data file 28 is specified.

For this purpose, an identification unit 42 for first and last syllables and an emphasis detection unit 44 for authorized syllables, with both units accessing the mood data file unit 28 own and the operator parameterization unit 38 inform about the appropriate context-sensitive parameters.

As explained in detail below, there are operator sets that are especially applicable to syllables that are to be emphasized ("conspicuous" syllables). These operators are not systematically applied to all conspicuous syllables, but only to those that are randomly selected from candidate syllables selection are selected. The candidate syllables depend on the mood data. If the latter contain indications of which syllables are allowed to be emphasized, then the candidate syllables are merely taken out of those conspicuous syllables.

This becomes common be the case for understandable Texts in which the emphasis on a few syllables is forbidden a natural one Ensure sound reproduction. If the mood library does not contain such ads or instructions, then put all Syllables candidates for random selection. This is usually incomprehensible sounds be the case.

The random selection is made by a random sampling unit 46 with controllable probability, with the unit in question operatively between the accent unit 44 for authorized syllables and the operator parameterization unit 38 connected. The random sampling unit 46 has a controllable degree of probability of choosing a syllable from the candidates. More specifically, in the event that N is the probability that a candidate is selected, with N controllably ranging from 0 to 1, then for P candidate syllables N.P syllables on average should be selected to be exposed to a specific set of operators which belongs to a random stress. The distribution of the randomly selected candidates is substantially uniform over the sequence of syllables.

The appropriately configured operator sets from the operator set configuration unit 26 become a syllable data modifier unit 48 in which they are applied to the syllable data. For this purpose, the syllable data modifier unit receives 48 the syllable data directly from the mood data file 28 , The thus received syllable data is passed through the unit 48 modified as a function of the operator set, in particular with regard to the pitch and duration data. The resulting modified syllable data (new syllable data) is then received by the syllable data modifier unit 48 to the decoder 32 with the same structure as in the mood data file. In this way, the decoder can accurately process the new syllable data as if it originated directly from the vocalization data file. From there, the new syllable data are interpolated (interpolator unit 30 ) and processed by an audio frequency sound processor, an audio amplifier and speakers. However, the sound produced by the speaker no longer corresponds to a neutral tone, but rather corresponds to the sound with a simulation of an emotion determined by the user U.

All the above functional units are under the overall control of an operation sequencer 50 which directs the entire execution of the emotion generation procedure according to a prescribed set of rules.

4 graphically illustrates the effect of the pitch operator set OP on a pitch curve of a synthesized clipping element originally specified by its sound data. For each operator, the figure shows a pitch curve (fundamental frequency f as a function of time t) - in left and right columns, respectively - before the effect of the pitch operator and after the action of a pitch operator. In the example, the input pitch curves are identical for all operators and they appear relatively flat.

There are four operators in the illustrated sentence, the following ones (from top to bottom in the figure):
an "ascending waveform" pitch operator OPrs which imposes a waveform on each input-side pitch curve, which increases in time, ie, the operator concerned causes the original pitch contour to increase in frequency over time;
a pitch operator OPfs having a "falling waveform" which imposes a waveform on each input pitch curve, which decays in time, ie, the operator concerned causes the original pitch contour to decrease in frequency over time;
a pitch operator OPsu with an "upward shift" which imposes a uniform upward shift in the fundamental frequency on each input pitch curve, the shift being equal for all points in time so that the pitch contour is simply shifted up on the fundamental frequency axis becomes; and
a "downshift" pitch operator OPsd which imposes a uniform downward shift in the fundamental frequency on each input pitch curve, the shift being the same for all points in time so that the pitch contour is simply shifted down on the fundamental frequency axis is.

In the embodiment, the operators OPrs for the rising waveform and OPfs for the falling curve the following characteristic: The pitch in the middle point at the time (1/2 t1 for a pitch duration of t1) remains essentially unchanged after the operator or after its action. In other words, the operators operate to panning the input pitch curve by the pitch value at the central point in time to impose the required curve or slope. That is, in the case of the rising-waveform operator OPrs, the pitch values before the central timing are actually lowered and, in the case of a falling-waveform operator OPfs, the pitch values before the central timing are actually raised, as illustrated by the figure.

Optionally, intensity operators labeled OI may also be provided. The effects of these operators are in 5 illustrates the appearance of 4 is directly analog. These operators are also four in number, and are identical to those of the pitch operators OP, except that they act on the intensity I curve over time t. Accordingly, these operators should not be considered separately in detail for the sake of simplicity.

The pitch and intensity operators can each be parameterized as follows:
For the rising and falling operators (OPrs, OPfs, OIrs, OIfs): The gradient of the slope to be imposed on the input contour. The slope may be expressed in terms of normalized slope values. For example, 0 means that no slope is imposed; the operator has no effect on the input signal in this case (such an operator is called a neutralized or neutral operator). At the other extreme, a maximum value causes the input curve to have an infinite gradient, ie, to rise or fall almost vertically. Between these extremes, any parameter value may be assigned to the operator in question to impose the required slope of the input contour.

For the shift operators (OPsu, OPsd, OIsu, OIsd): The amount of upward or downward shift that the input contour with respect to the absolute fundamental frequency (for the pitch) or of the intensity value. The corresponding parameters can thus in unit increments or decrements along the pitch or intensity axis expressed become.

6 graphically illustrates the effect of a continuous (or time) operator OD on a syllable. The illustration illustrates in left and right columns the duration of the syllable (in a horizontal line expressing an initial time length t1) of the input syllable before the action of a persistent operator or after the action of a persistent operator.

The duration operator can be:
a stretch operator that causes the duration of the syllable to increase. The magnification is expressed in terms of a parameter D, which is called a positive D parameter. For example, the parameter D may simply be a number of milliseconds of duration to add to the initial input duration value, if the latter is also expressed in milliseconds, so that the effect of the operator is obtained simply by adding the value D to the Duration specification t1 is added for the syllable in question. As a result, the processing of the data by the interpolator 30 and subsequent units cause the period of time over which the syllable is expressed to be stretched;
a shrinking operator that causes the duration of the syllable to be reduced. The reduction is expressed in terms of the same parameter D, which in this case is a negative parameter. For example, the parameter D may simply be a number of milliseconds of duration to be subtracted from the initial input duration value, if the latter is also expressed in milliseconds, so that the operator's effect is achieved simply by setting the value D is subtracted from the duration specification for the syllable in question. As a result, the processing of the data by the interpolator causes 30 and by the following units, that the period of time over which the syllable is expressed, shrunk (shortened).

Of the Operator can as well be a neutralized or neutral operator by simply the value 0 for the parameter D inserted becomes.

It It should be noted that the duration operator as an operator of two various types of stretching and contraction However, it is clear that the only difference is in the Sign plus or minus, which is set before the parameter D. is. Thus, one and the same operator mechanism can have both operator functions (stretching and contracting), if positive and can process negative numbers.

Of the Range of possible Values for D and its possible Incremental values in the range may be according to requirements chosen become.

in the Following is the parameterization of each of the operators OP, OI and OD expressed by a variable value, that by the last Letters of the respective operator plus the one specific to the respective operator Annex is marked, d. H. Prs = the value of the positive slope for the Pitch operator OPrs with rising curve; Pfs = value of the negative slope for the pitch operator OPs for the sloping curve; Psu = value of the size of the upward shift for the pitch operator OPsu with an upward shift; Psd = value of the pitch operator OPsd with a downward shift; Irs = value of the positive slope for the intensity operator OIrs with rising curve; Ifs = value of the negative slope for the Intensity operator OIfs with sloping curve; Isu = value of the size of the upward shift for the Intensity operator OIsu with an upward shift; Isd = value of the intensity operator OIsd with a downward shift; Dd = value of the temporal increment for the duration-stretching operator ODd; Dc = Value of the temporal decrement (contraction) for the duration shrink operator ODc.

The embodiment also uses a separate operator that has the probability N for the random sampling unit 46 sets. This value is selected from a range of 0 (no possibility of selection) to 1 (certainty of selection). The value N is used to control the density of the accented syllables in the output signal as appropriate for the reproduction of the emotional quality.

In the example, each selection or a selection of the above values which parameterize the operators OP, OI, OD, and N, for variable parameters by the parameter generation unit 16 made variable in conjunction with the store 14 and the emotion measure selector 18 works like this with reference to 2 has been described. Thus, a particular variable parameter VPi may correspond to one of the following parameter values defined above: Prs, Pfs, Psu, Psd, Irs, Ifs, Isu, Isd, Dd, Dc. The number and selection of these values to be made variable is through the user interface 40 selectable.

7A and 7B FIG. 14 is a flow chart illustrating the process for selectively forming and applying the above operators to syllabic data based on with reference to FIG 3 illustrated system illustrates. 7B is a continuation of 7A ,

The process begins with an initialization phase P1, which is the loading of the input-side syllable data from the vocalization data file 28 (Step S2).

Then, the emotion to be transmitted into the sentence or passage from which the loaded syllable data is a part is made using the interface unit 40 loaded (step S4). The emotions can be rest, sadness, happiness, anger, etc. The interface also inputs the measure (degree) of the emotion to be provided, for example, by adding a weighting value (step S6). This weighting value can be expressed as a displacement of the variable parameter value or variable parameter values VPi from the standard value Pi (= Ei), which is defined by the variable δ, as described with reference to FIG 2 has been described.

The system then enters a universal operator phase P2 in which a universal operator set OS (U) is systematically applied to all syllables. The universal operator set OS (U) contains all operators of 4 and 6 , ie OPrs, OPfs, OPsu, OPsd, which form the four pitch operators, plus ODd and ODc, which form the two duration operators. Each of these operators of the operator set OS (U) is parameterized by a corresponding value corresponding to Prs (U), Pfs (U), Psu (U), Psd (U), Dd (U) and Dc (U), respectively has been explained above (step S8). This step includes the addition of numeric values to these parameters, and is done by the operator set configuration unit 26 executed. The choice of the parameter values for the universal operator set OS (U) is made by the operator parameterization unit 8th determined as a function of the programmed emotion and the measure of the emotion plus any other factors. In the example, suppose that each of these parameters is made variable by the variable δ, whereby the respective parameters are represented as VPrs (U), VPfs (U), VPsu (U), VPsd (U), VDd (U) and VDc (U) should be designated. (Generally, in the following each parameter value or operator / operator set made variable by the variable δ is identified as such by the letter "V", which is used as the initial letter of its name.) The universal operator set VOS (U) then becomes systematic applied to all the syllables of a sentence or phrase or group of sentences (step S10). The effect includes a modifi the numerical values t1, P1-P5 of the syllable data. For the pitch operators, the pitch parameter VPrs or VPfs is translated into a set of five different values to be arithmetically applied to the values P1-P5. These difference values are chosen such that each of the values P1-P5 is shifted according to the parameterized slope, the mean value P3 remaining largely unchanged, as explained earlier. For example, the first two values of the rising slope parameters will be negative to cause the first half of the pitch to be lowered and the last two values will be positive to cause the last half of the pitch to be raised. Thus, the rising curve or the rise curve is articulated, which is articulated around the time center, as in 6 is illustrated. The degree of slope that constitutes the variable parameterization is expressed in terms of these difference values. A corresponding solution in the reverse direction is used for the parameter with falling inclination or falling curve.

The shift up or downshift operators can before or after the slope operators. They add or just subtract one and the same value by the Parameterization is determined, to or from the five pitch values P1-P5. The Operators form mutually exclusive pairs, i. H. an operator for the rising inclination or the rising curve will not applied if an operator for to apply a sloping curve or a sloping slope and similarly applies to the upshift and downshift and persistent operators.

The application of the operators (ie, the calculation for modifying the data parameters t1, P1-P5) is performed by the syllable data modifying unit 48 performed.

After this the syllables are processed by the universal operator set VOS (U) they become provisional for one further processing, if necessary, buffered.

The system then enters a probable emphasis phase P2 for which another operator stress parameter set VOS (PA) is created. This operator set has the same operators as the universal operator set, but has different variable values for parameterization. Using the convention applied to the universal operator set, the operator set VOS (PA) is replaced by corresponding values: VPrs (PA), VPfs (PA), VPsu (PA), VPsd (PA), VDd (PA), and VDc (PA) parameterized. These parameter values are correspondingly determined by the operator parameterization unit 38 calculated as a function of the emotion, the emotion level of the emotion, and other factors passing through the interface unit 40 to be provided. The choice of parameters is generally made by adding a degree of intonation (teaching) to the language according to the considered emotion. An additional parameter of the probable emphasis operator set VOS (PA) is the value of the probability N, as defined above, which is also made variable by the variable δ (VN). This value depends on the emotion and emotion measure as well as other factors, for example the nature of the syllable file.

After the parameters have been obtained, they are put into the operator set configuration unit 26 to form the complete probabilistic stress set parameter set VOS (PA) (step S12).

It is then determined which of the syllables are to be subjected to this set of operators VOS (PA), as by the random unit 46 is determined (step S14). The latter unit supplies the list of random syllables for emphasis by this set of operators. As discussed above, the candidate syllables are all syllables if they are dealing with unpronounceable sounds, or if there are no forbidden accents on syllables, or only the allowed syllables, if specified in the file. This will usually be the case for meaningful words.

The syllables randomly selected from the candidates are then processed by the syllable data modifier unit for processing by means of the probabilistic stressor operator set VOS (PA) 48 (Step S16). The actual processing that is performed is the same as explained above with respect to the universal operator set, with the same technical considerations. The only difference lies in the parameter values involved.

It should be noted that the processing is performed by the probabilistic emphasis operator set VOS (PA) on syllable data already represented by the universal Operational set VOS (U) have been processed. Mathematically, this circumstance may be represented for a syllable data item Si of the file which has been processed after being used in step S14, as follows: VOS (PA) .VOS (U) .Si → Sipacc, where Sipacc the resulting data for the stressed processed syllable represent.

For all syllables except for the syllables of the first and last words of a sentence, the one in the vocalization data file unit 28 is contained, supplies the syllable data modifying unit 48 the following modified forms of the syllable data (indicated generally by S), originally in the file 28 :
VOS (U) .S → Spna for the syllable data not extracted at step S14, where Spna denotes a processed unstressed syllable, and
VOS (PA) .VOS (U) .S → Spacc for the syllable data taken at step S14, where Spacc designates the processed stressed syllable.

Finally occurs the process enters a phase P4 to process an emphasis, the for the first and last syllables of a sentence or phrase specific is. When a sentence or phrase is made identifiable or recognizable words this phase P4 acts, all syllables of the first and last words to emphasize the sentence or phrase. The term sentence or phrase can be understood in the ordinary grammatical sense Text be understood to be spoken, for example regarding breaks during the lecture. In case of incomprehensible Sound, as with mumbles or animal imitations, becomes a sentence or clause understood with respect to a beginning and end of the utterance, and marked by a break. Typically, a such sentence or phrase of about one to three or four seconds to last; to endure, to continue. For incomprehensible sounds becomes the phase P4 to emphasize the last syllable at least the first and last syllables and preferably at the first m and the last n syllables where m or n is more typical Are equal to about 2 or 3 and be the same or different can.

As with the previous phases, a specific parameterization of the same primitives VOPrs, VOPfs, VOPsu, VOPsd, VODd, VODc is performed, yielding a first and last syllabic operator set VOS (FL) parameterized by a corresponding value, namely VPrs ( FL), VPfs (FL), VPsu (FL), VPsd (FL), VDd (FL) and VDc (FL) (step S18). These parameter values are determined by the operator parameterization unit 28 similarly calculated as a function of emotion, degree of emotion and other factors generated by the interface unit 30 to be provided.

The resulting operator set VOS (FL) is then applied to the first and last syllables of each sentence (step S20), these syllables being represented by the syllable detector unit 34 be identified for the first / last syllables.

As explained above become the syllable data to which the operator set VOS (FL) applies is, previously through the universal operator set VOS (U) at the step S10 has been processed. additionally It may happen that a first or last syllable or first or last syllables at the random selection step S14 also taken and thus also by means of the probabilistic theory Emphasis operator set VOS (PA).

Thus, there are two ways of processing for a first or last syllable, expressed below, using the convention set forth above:
Possibility 1: Processing by the operator set VOS (U) and then by the operator set VOS (FL), resulting in VOS (FL) .VOS (U) .S → Spfl (1), and
Possibility 2: successive processing by the operator sets VOS (U), VOS (PA) and VOS (FL), resulting in VOS (FL) .VOS (PA) .VOS (U) .S → Spfl (2).

These simple operator-based solution has come up as a solution proven that leads to results, which are at least comparable to those that are essential more complicated systems are achieved, both in terms of meaningful statements as well as in the pronunciation of a recognizable language.

The choice of parametrizations to express a particular emotion is extremely subjective and varies considerably depending on the form of the utterance, the language, etc. However, because of the availability of simple, well-defined parameters that do not require much real-time processing, it is easy Scanning through many possible combinations of parametrizations to make the most satisfactory lendsten operator sets.

For every parameterization, that belongs to a certain emotion may be a range of variability in the parameter values according to the invention which control of the amount of the respective generated generated Emotion allows.

Applicant has found, for purposes of exemplification only, that good results can be achieved with the following parameterizations:
Sadness: the pitch for the universal operator set = sloping curve with low slope,
Duration operator = stretch,
Probability of taking N for an emphasis: low;
Rest: no operator set is applied or just a slightly parameterized universal operator;
Happiness: Pitch for universal operator set = rising curve, moderately high gradient,
Duration for universal operator set = shrinkage,
Duration for stressed operator set = stretch;
Annoyance: pitch for all operator sets = declining curve, moderately high slope,
Duration for all operator sets = shrinkage.

For one Operator set not specified in the above example is, the parameterization is of the same general type for all Operator sets. Generally speaking, the type or type of changes (increasing curve, Shrinkage, etc.) the same for all operator sets; only the actual Values are different. Here are the values usually chosen so that the least amount of change is caused by the universal operator set and that the largest amount of change is caused by the first and last syllabic emphasis, where the probabilistic emphasis operator set one mean amount of change causes.

The System can also be designed so that there are intensity operators OI in his Sentence used, depending on from the used parameterization.

The interface unit 40 can be integrated into a computer interface to provide various controls. Among these controls may be a direct selection of parameters of the above-mentioned various operator sets to allow user U to fine-tune the system. The interface may be made user-friendly by providing visual scales that graphically illustrate, for example, the slope values, the displacement values, the shrinkage / strain values for the various parameters.

The Invention can be many more types of emotion synthesis systems include. While She is particularly suited to fair synthesis systems that have an emotion express in speech or sound, the invention can also for other types of emotion synthesis systems to be considered in which the emotion is in other forms is displayed: facial or body expressions, visual effects, etc., the movement of animated objects involving those involved Parameters reflect a type or type of emotion to be represented.

Claims (20)

Method for controlling the operation of a device ( 2 ; 12 ) for synthesizing an emotion transmitted to a sound, the apparatus having at least one input for a parameter (Pi) whose value (Ei) is used to set an emotion type to be transmitted, comprising the steps of: - programming the input signal / the input signals with a parameterization to generate a certain type of emotion (E), and - imparting variability in an amount to the respective type of emotion to be transmitted; characterized in that the variability is obtained to the degree of said emotion type by, within a predetermined control range, determining at least one parameter (Pi) used to set an emotion type, a deviation from its initial default value (Ei ). The method of claim 1, wherein the at least one variable parameter (VPi) is made variable according to a local model over the control region, wherein the model is a measure of an emotion control variable (VP). 6 ) refers to the variable parameter (VPi) and wherein the measure or the Size of the emotion control variable is used to variably set a value of the variable parameter. The method of claim 2, wherein the local model is a local linear model for the control domain and for a given emotion type, the variable parameter (VPi) being determined to be linear over the control domain by means of the measure of the emotion control variable (VPi). 6 ) changes. Method according to one of claims 1 to 3, wherein the measure of the emotion is determined by a control variable (δ) which modifies the variable parameter (VPi) according to a relationship given by the formula: VPi = A + δB, in VPi is the value of the variable parameter in question, A and B are values allowed for the control region, and δ is the measure of the emotion control variable. The method of claim 4, wherein A is a value within of the control region, where is the measure of the emotion control variables (δ) in one Interval is variable, which contains the value zero. The method of claim 5, wherein A is substantially the mean value (Emr) of the control region is and where is the measure of the emotion control variable (δ) in one Interval is variable whose mean is zero. The method of claim 6, wherein the measure of the emotion control variables (δ) in one Interval of -1 until +1 is variable. A method according to any one of claims 4 to 7, wherein B is determined is going through B = (Eimax - A) or by B = (Eimin + A), where Eimax is the value of the input parameter is to the maximum degree of to be transferred To create emotion type in the control area and where Eimin the value of the parameter is to calculate the minimum dimension of the Create emotion type in the control area. Method according to one of claims 4 to 8, wherein A is the same is the default parameter value (Ei), which is originally specified became one to transfer Define or set emotion type. Method according to claim 8 or 9, wherein the value Eimax or Eimin by deviating from the default parameter value (Ei), which originally was specified to be transmitted Determine or set emotion type, and by determining a Maximum deviation in increase or decrease direction, which is a desired limit for the Measure of through the control area Emotion results, is determined experimentally. Method according to one of claims 1 to 10, wherein the same measure of the emotion control variable (δ) is used to generate a plurality of variable parameters (VP1-VPN) of the emotion synthesizer ( 2 ; 12 ) together. Contraption ( 10 ) for controlling the operation of a system ( 2 ; 12 ) for synthesizing an emotion transmitted in a sound, the system having at least one input for a parameter (Pi) whose value (Ei) is used to set an emotion type to be transmitted, characterized in that the device comprises: - means for programming the input signal (s) with a parameterization to generate a particular type of emotion (E), and - variation means ( 14 . 16 . 18 ) for imparting variability to the extent of the type of emotion to be transmitted and characterized in that the variation means ( 14 . 16 . 18 ) are capable, within a predetermined control range, of at least one parameter (Pi) used to determine an emotion type submit to a deviation from its initial default value (Ei). Device according to claim 12, wherein the variation means ( 14 . 16 . 20 ) is operable to cause at least one variable parameter (VPi) to change to a measure of the emotion control variable (δ) that is accessible to variably set a value of the variable parameter. Apparatus according to claim 13, wherein the variation means ( 14 . 16 . 18 ) cause the variable parameter (VPi) to correspond to a local linear model with a variation in the extent of the emotion control variable (VPi) 6 ) changed linearly. Apparatus according to claim 13 or 14, wherein the Measure of Emotion control variables (δ) is variable in an interval containing the value zero. The apparatus of claim 15, wherein the measure of emotion control variables (δ) in one Interval of -1 until +1 is variable. Device according to one of claims 12 to 16, wherein the variation means ( 14 . 16 . 20 ) cause at least one variable parameter (VPi) to relate to a measure of the emotion control variables ( 5 ) according to one of the following formulas: VPi = Emr + δ (Eimax - Emr) or VPi = Emr + δ (Eimin + Emr) where δ is the value of the measure of the emotion control variable, Emr is essentially the average of the control region and preferably equal to the default parameter value (Ei) originally specified to specify an emotion type to be transmitted, Eimax the value of the parameter for generating the maximum measure of Eimin is the value of the parameter for generating the minimum amount of emotion type to be transmitted in the control area. Apparatus according to any one of claims 12 to 17, operable to apply, with the same degree of emotion control variable (δ), a plurality of variable parameters (VP1-VPN) of the emotion-synthesizing system (16). 2 ; 12 ) to variably set a value of the variable parameter. System comprising an emotion synthesizer ( 2 ; 12 ) having at least one input for receiving at least one parameter (Pi) whose value (Ei) is used to determine an emotion type to be transmitted, and a device ( 10 ) according to any one of claims 13 to 19, which is operatively connected to provide a variable (VPi) to the at least one input thereby to impart variability in the degree of emotion type to be transmitted. Computer program which executes computer-executable instructions whose execution in a data processor causes the data processor all Steps of the method according to one of claims 1 to 11 executes.