TWI867419B - Method and system of generating audio - Google Patents

Abstract

A method includes: extracting character information; matching the character information and preset voice parameters; when the character information does not match the preset voice parameters, selecting at least one original voice parameter corresponding to the character information from the preset voice parameters, and generating at least one new voice parameter according to the at least one original voice parameter which corresponds to the character information; and generating corresponding audio according to the at least one new voice parameter.

Description

Method and system for generating speech

The present disclosure relates to a technology for generating speech, and more particularly to a method for generating speech and a system for generating speech.

When making an audiobook, it is necessary to dub the characters in the audiobook with appropriate voiceprints to improve the quality of the audiobook. However, the voiceprints in the default database may not be sufficient to correspond to all the characters in the audiobook. Therefore, how to design to solve the above problem is an important topic in this field.

The embodiment of the present invention includes a method for generating speech. The method includes: extracting character information; matching the character information with a plurality of preset sound parameters; when the character information does not match the preset sound parameters, selecting at least one original sound parameter corresponding to the character information from the preset sound parameters, and generating at least one additional sound parameter based on the at least one original sound parameter, wherein the at least one additional sound parameter corresponds to the character information; and generating corresponding speech based on the at least one additional sound parameter.

The embodiment of the present invention includes a system for generating speech. The system includes a storage unit and a processor. The storage unit stores a plurality of preset sound parameters. The processor is coupled to the storage unit, accesses the preset sound parameters through the storage unit, and generates at least one newly added sound parameter corresponding to the character information according to at least one original sound parameter in the preset sound parameters when the character information does not conform to the preset sound parameters, and generates speech according to the at least one newly added sound parameter.

In this article, when an element is referred to as "connected" or "coupled", it may refer to "electrically connected" or "electrically coupled". "Connected" or "coupled" may also be used to indicate that two or more elements cooperate with each other or interact with each other. In addition, although the terms "first", "second", etc. are used in this article to describe different elements, the terms are only used to distinguish between elements or operations described with the same technical terms. Unless the context clearly indicates otherwise, the terms do not specifically refer to or imply an order or sequence, nor are they used to limit the present case.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by ordinary technicians in the field to which this case belongs. It will be further understood that those terms as defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and this case, and will not be interpreted as an idealized or overly formal meaning unless expressly defined as such in this document.

FIG. 1 is a schematic diagram of a system 100 according to some embodiments of the present invention. In some embodiments, the system 100 generates speech according to preset sound parameters stored in a storage element. As shown in FIG. 1 , the system 100 includes a storage unit 110, a processor 120, an input interface 130, and a speaker 150. The processor 120 is coupled to the storage unit 110, the input interface 130, and the speaker 150.

In operation, the processor 120 receives the text 140 preset in the system 100, or receives the text 140 from the outside or the character information CR1 input through the input interface 130, and receives the default sound parameters DS1-DS4 from the storage unit 110. When the character information CR1 does not match any of the default sound parameters DS1-DS4, the processor 120 generates a new sound parameter (such as the new sound parameter VD1 described below) that matches the character information CR1 through the default sound parameters DS1-DS4, and generates voice AB2 according to the new sound parameter, and the speaker 150 plays the voice AB2.

In some embodiments, the text 140 may be information representing a story or a script, and the present invention is not intended to be limiting. In some embodiments, the text 140 may also be information consisting of only one or a few words. In various embodiments, the text 140 may be pre-stored in various storage media, such as the text 140 may be pre-stored in the storage unit 110, such as a flash drive, a hard drive, a flash drive, a cloud, or any other storage media, and the present invention is not intended to be limiting.

In some practices, when dubbing a text, it is necessary to hire voice actors who have the same number of characters as the characters in the text and have corresponding characteristics, which makes the cost high.

Compared with the above method, in the embodiment of the present invention, the processor 120 can generate new sound parameters that match the character information CR1 based on the preset sound parameters DS1-DS4 according to the character information CR1. In this way, the text 140 can be dubbed by the preset sound parameters DS1-DS4, which makes the cost lower.

As shown in FIG. 1 , in some embodiments, the storage unit 110 has a database 111. The database 111 stores preset sound parameters DS1-DS4. In various embodiments, the database 111 may include various numbers of preset sound parameters, and FIG. 1 is only an example.

As shown in FIG. 1 , in some embodiments, the processor 120 includes a text extraction module 121, a sound parameter expansion module 122, and a text-to-speech conversion module 123. In various embodiments, the processor 120 may include other modules different from the text extraction module 121, the sound parameter expansion module 122, and the text-to-speech conversion module 123, and the operations of the text extraction module 121, the sound parameter expansion module 122, and the text-to-speech conversion module 123 described below are performed by other modules, so the embodiment shown in FIG. 1 is only an example and is not intended to limit the present invention. The above modules may also be implemented by other systems, hardware, software, or a combination thereof.

In some embodiments, the text extraction module 121 extracts the character information CR1 and the text content TX1 from the text 140, and provides the character information CR1 and the text content TX1 to the sound parameter expansion module 122 and the text-to-speech conversion module 123, respectively.

In other embodiments, the input interface 130 receives the input data FT1, extracts the character information CR1 from the input data FT1, and provides the character information CR1 to the sound parameter expansion module 122. In some embodiments, the input data FT1 may include user-defined feature parameters, such as feature parameters corresponding to various age features and/or gender features.

In some embodiments, the voice parameter expansion module 122 accesses the preset voice parameters DS1-DS4 from the storage unit 110, and matches the character information CR1 with the preset voice parameters DS1-DS4. For example, the voice parameter expansion module 122 compares the age characteristics and/or gender characteristics of the character information CR1 and the preset voice parameters DS1-DS4, and assigns one of the preset voice parameters DS1-DS4 that matches the character information CR1 to the character information CR1 according to the comparison result. In some embodiments, the voice parameter expansion module 122 matches the character information CR1 by using the speaker embedding technology.

When the character information CR1 matches one of the preset sound parameters DS1-DS4, the sound parameter expansion module 122 selects the corresponding preset sound parameter as the original sound parameter DS and transmits the original sound parameter DS to the text-to-speech conversion module 123, so that the text-to-speech conversion module 123 generates the corresponding voice AB1 according to the original sound parameter DS and the text content TX1.

When the character information CR1 does not match any of the preset sound parameters DS1-DS4, the sound parameter expansion module 122 selects the sound parameter corresponding to the character information CR1 from the preset sound parameters DS1-DS4 as the original sound parameter DX, and generates the newly added sound parameter VD1 corresponding to the character information CR1 according to the original sound parameter DX. Then, the text-to-speech conversion module 123 generates the corresponding newly added voice AB2 according to the newly added sound parameter VD1 and the text content TX1. In some embodiments, the voice AB1 and/or AB2 correspond to the audio book information represented by the text 140, and the speaker 150 plays the voice AB1 and/or AB2.

In some embodiments, the character information CR1 may include various characteristic parameters, which may represent, for example, age characteristics and/or gender characteristics. When the characteristic parameters of the character information CR1 do not conform to any of the preset voice parameters DS1-DS4, the voice parameter expansion module 122 generates corresponding newly added voice parameters VD1 according to the original voice parameters DX and the characteristic parameters of the character information CR1, so that the corresponding newly added voice AB2 can be further generated based on it, and the newly added voice AB2 can express the character information CR1 that does not conform to the preset situation, as described below.

FIG. 2 is a schematic diagram of the default sound parameters DS1-DS4 in the vector space 200 according to some embodiments of the present invention. In some embodiments, different positions in the vector space 200 correspond to different sound parameters, such as different sound frequencies. As shown in FIG. 2, the default sound parameters DS1-DS4 and the newly added sound parameters NS1, NS2, NZ1, NN1, NN2 are represented in the vector space 200. Specific embodiments will be described below.

In some embodiments, when the number of characters corresponding to the character information CR1 is greater than the number of parameters of the original sound parameters DX, the sound parameter expansion module 122 generates additional sound parameters such that the sum of the number of additional sound parameters and the number of parameters of the original sound parameters DX is the same as the number of characters.

For example, the default voice parameters DS1-DS4 respectively have the age characteristics of 20, 40, 60, and 80 years old, and the character information CR1 corresponds to three 20-year-old characters. At this time, the voice parameter expansion module 122 selects the default voice parameter DS1 corresponding to 20 years old as the original voice parameter DX according to the character information CR1, and generates two newly added voice parameters NS1 and NS2 corresponding to 20 years old according to the default voice parameter DS1, wherein the newly added voice parameters NS1 and NS2 are different from the default voice parameter DS1. Then, the voice parameter expansion module 122 assigns the newly added voice parameters NS1, NS2 and the default voice parameter DS1 to the three twenty-year-old characters respectively, so that the text-to-speech conversion module 123 generates voice AB2 according to the newly added voice parameters NS1, NS2, the default voice parameter DS1 and the text content TX1, thereby enabling the voice AB2 to emit the voices of the three twenty-year-old characters.

In some embodiments, the sound parameter expansion module 122 adjusts the default sound parameter DS1 to generate the newly added sound parameters NS1 and NS2. In the vector space 200, the distance between each of the newly added sound parameters NS1 and NS2 and the default sound parameter DS1 is closer than the distance between each of the newly added sound parameters NS1 and NS2 and other default sound parameters (e.g., default sound parameters DS2-DS4). Therefore, the characteristics of each of the newly added sound parameters NS1 and NS2 are closer to the characteristics of the default sound parameter DS1. For example, each of the default sound parameter DS1, the newly added sound parameters NS1, and NS2 has the same age characteristics. In some embodiments, the aforementioned distance may be the distance between any two feature points (sound parameters) of the newly added sound parameters NS1, NS2 and the default sound parameters DS1-DS4 in the feature space, that is, the distance between the feature vectors of any two sound parameters.

In some embodiments, when the first age characteristic of the character information CR1 does not match any of the preset voice parameters DS1-DS4, the voice parameter expansion module 122 selects the first original voice parameter and the second original voice parameter from the preset voice parameters DS1-DS4 as the original voice parameter DX according to the first age characteristic, and generates a newly added voice parameter that matches the first age characteristic according to the first original voice parameter and the second original voice parameter. The first original voice parameter has the second age characteristic, the second original voice parameter has the third age characteristic, and the first age characteristic is between the second age characteristic and the third age characteristic.

For example, the default sound parameters DS1-DS4 have the age characteristics of 20, 40, 60, and 80 years old, respectively, and the character information CR1 corresponds to a 30-year-old character. Compared with 60 and 80 years old, 20 and 40 years old are closer to 30 years old. Correspondingly, the sound parameter expansion module 122 selects the default sound parameters DS1 and DS2 as the first original sound parameter and the second original sound parameter, and generates the newly added sound parameter NZ1 according to the default sound parameters DS1 and DS2. For example, the newly added sound parameter NZ1 has the age characteristic of 30 years old.

As shown in FIG. 2 , the newly added sound parameter NZ1 is located between the default sound parameters DS1 and DS2 in the vector space 200. In some embodiments, the sound parameter expansion module 122 generates the newly added sound parameter NZ1 by performing an interpolation operation on the default sound parameters DS1 and DS2.

In some embodiments, when the character information CR1 corresponds to multiple characters with the first age characteristic, the sound parameter expansion module 122 generates more new sound parameters based on the new sound parameters, so that the number of parameters of the new sound parameters is the same as the number of characters corresponding to the character information CR1.

For example, the default voice parameters DS1-DS4 have age characteristics of 20, 40, 60, and 80 years old, respectively, and the character information CR1 corresponds to three 30-year-old characters. After the voice parameter expansion module 122 generates the newly added voice parameter NZ1 according to the default voice parameters DS1 and DS2, it further generates the newly added voice parameters NN1 and NN2 according to the newly added voice parameter NZ1, wherein the newly added voice parameters NZ1, NN1, and NN2 are different from each other, and each of the newly added voice parameters NN1 and NN2 has an age characteristic of 30 years old. Then, the voice parameter expansion module 122 assigns the newly added voice parameters NZ1, NN1, and NN2 to the three 30-year-old characters, respectively.

In some embodiments, the sound parameter expansion module 122 adjusts the newly added sound parameter NZ1 to generate the newly added sound parameters NN1 and NN2. In the vector space 200, the distance between each of the newly added sound parameters NN1 and NN2 and the newly added sound parameter NZ1 is closer than the distance between each of the newly added sound parameters NN1 and NN2 and the default sound parameters (e.g., the default sound parameters DS1-DS4). Therefore, the characteristics of each of the newly added sound parameters NN1 and NN2 are closer to the characteristics of the newly added sound parameter NZ1. For example, each of the newly added sound parameters NZ1, NN1, and NN2 has the same age characteristics.

In some embodiments, the processor 120 stores the newly added sound parameters NZ1, NN1, and NN2 in the database 111 in the storage unit 110.

In some embodiments, the above-mentioned algorithm for generating new sound parameters NS1 and NS2 based on the default sound data DS1, and the algorithm for generating new sound parameters NN1 and NN2 based on the default sound data NZ1, can be implemented by different oversampling techniques, for example: synthetic minority oversampling technique (SMOTE), borderline synthetic minority oversampling technique (borderline-SMOTE) and/or oversampling technology based on Generative Adversarial Network (GAN).

FIG. 3 is a flow chart of a method 300 of the system 100 according to some embodiments of the present invention. The operation of the method 300 is described below with reference to the components of the system 100, but the present invention is not limited thereto. In various embodiments, the method 300 may be performed by other systems different from the system 100. As shown in FIG. 3, the method 300 includes operations S31-S36, wherein the order of the operations S31-S36 in the method 300 of the present invention is not limited thereto.

In operation S31, the processor 120 extracts the character information CR1 from the text 140. For example, in the embodiment of FIG. 1, the text extraction module 121 in the processor 120 extracts the character information CR1 from the text 140 for subsequent generation of the sound parameters corresponding to the character information CR1. In a different example, the processor 120 may also receive the character information CR1 through the input interface 130.

In operation S32, the processor 120 obtains the default sound parameters DS1-DS4. For example, in the embodiment of FIG. 1, the processor 120 reads the default sound parameters DS1-DS4 from the storage unit 110 in the temporary memory for subsequent operation.

In operation S33, the processor 120 determines whether the character information CR1 matches the preset sound parameters DS1-DS4. For example, in the embodiment of FIG. 1, the sound parameter expansion module 122 in the processor 120 determines whether the character information CR1 matches the preset sound parameters DS1-DS4 obtained in operation S32.

In some embodiments, when the character information CR1 meets the preset sound parameters DS1-DS4, the processor 120 performs operation S34 after operation S33. When the character information CR1 does not meet the preset sound parameters DS1-DS4, the processor 120 performs operation S35 after operation S33.

In operation S34, the processor 120 sets the corresponding one of the preset sound parameters DS1-DS4 matching the character information CR1 in operation S33 as the original sound parameter DS, and then generates the corresponding voice AB1 according to the original sound parameter DS. For example, referring to FIG. 1, the sound parameter expansion module 122 sets one of the preset sound parameters DS1-DS4 matching the character information CR1 as the original sound parameter DS, and then transmits the original sound parameter DS to the text-to-speech conversion module 123 to generate the corresponding voice AB1.

In operation S35, the processor 120 selects the original sound parameter DX corresponding to the character information CR1 from the preset sound parameters DS1-DS4 to generate the newly added sound parameter VD1. For example, referring to FIG. 1, the sound parameter expansion module 122 generates the newly added sound parameter VD1 according to the character information CR1 and the original sound parameter DX, so that the parameter quantity and characteristics of the newly added sound parameter VD1 conform to the character information CR1, and further transmits the newly added sound parameter VD1 to the text-to-speech conversion module 123.

In operation S35, the processor 120 generates the corresponding voice AB2 according to the newly added voice parameter VD1. For example, in the embodiment of FIG. 1, the text-to-speech conversion module 123 in the processor 120 generates the corresponding voice AB2 according to the newly added voice parameter VD1 and the text content TX1, and the speaker 150 plays it.

In this way, when the character information cannot obtain the corresponding default sound parameters from the storage unit, the sound parameters that match the character information can be generated, and different voices can be emitted even when the character information is the same, giving the listener a better experience.

Although the contents of this disclosure have been disclosed as above by way of embodiments, they are not intended to limit the contents of this disclosure. Any person with ordinary knowledge in the relevant technical field may make some changes and modifications without departing from the spirit and scope of the contents of this disclosure. Therefore, the protection scope of the contents of this disclosure shall be subject to the scope defined by the attached patent application.

100: System 110: Storage unit 120: Processor 130: Input interface 140: Text 150: Speaker 111: Database DS1~DS4: Default sound parameters DS, DX: Original sound parameters 121: Text extraction module 122: Sound parameter expansion module 123: Text-to-speech conversion module CR1: Character information TX1: Text content FT1: Input data AB1, AB2: Voice 200: Vector space VD1, NS1, NS2, NZ1, NN1, NN2: Add new sound parameters 300: Method S31~S36: Operation

FIG. 1 is a schematic diagram of a system according to some embodiments of the present invention. FIG. 2 is a schematic diagram of a vector space corresponding to the preset sound parameters shown in FIG. 1 according to some embodiments of the present invention. FIG. 3 is a flow chart of a method corresponding to the system shown in FIG. 1 according to some embodiments of the present invention.

Domestic storage information (please note in the order of storage institution, date, and number) None Foreign storage information (please note in the order of storage country, institution, date, and number) None

100: System

110: Storage unit

120: Processor

130: Input interface

140: Text

150: Speaker

111: Database

DS1~DS4: Default sound parameters

DS, DX: original sound parameters

121: Text extraction module

122: Sound parameter expansion module

123: Text-to-speech module

CR1: Character information

TX1: text content

FT1: Input data

AB1, AB2: Voice

VD1: Added new sound parameters

Claims (10)

A method for generating speech, comprising: extracting character information from a text; comparing the character information with a plurality of preset sound parameters to match the character information with the preset sound parameters; judging whether the character information matches or does not match the preset sound parameters based on the comparison result of the character information with the plurality of preset sound parameters; when the character information does not match the preset sound parameters, selecting a corresponding character information from the preset sound parameters; At least one original sound parameter of the character information is calculated by an algorithm to generate at least one newly added sound parameter, and the at least one newly added sound parameter corresponds to the character information; and the at least one newly added sound parameter is transmitted to a text-to-speech conversion module, and the text-to-speech conversion module generates a corresponding speech according to the at least one newly added sound parameter and a text content of the text. The method as described in claim 1 further comprises: when at least one characteristic parameter of the character information does not match any of the preset sound parameters, generating at least one additional sound parameter that matches the at least one characteristic parameter based on the at least one original sound parameter. The method as described in claim 1 further includes: when the number of characters corresponding to the character information is greater than the number of parameters of the at least one original sound parameter, the at least one newly added sound parameter is generated, and the sum of the number of the at least one newly added sound parameter and the number of the parameter is the same as the number of characters. The method as described in claim 1, wherein selecting the at least one original voice parameter comprises: when a first age characteristic of the character information does not match each of the preset voice parameters, selecting a first original voice parameter and a second original voice parameter from the preset voice parameters according to the first age characteristic, wherein the first original voice parameter has a second age characteristic, and the second original voice parameter has a third age characteristic, wherein the first age characteristic is between the second age characteristic and the third age characteristic. The method as described in claim 1, wherein extracting the role information comprises: extracting the role information from the pre-stored text; or extracting the role information from data received from an input interface. A system for generating speech includes: a storage unit storing a plurality of preset sound parameters; and a processor coupled to the storage unit, accessing the preset sound parameters through the storage unit, judging whether the character information meets or does not meet the preset sound parameters according to the comparison result between the character information and the preset sound parameters, and when the character information does not meet the preset sound parameters, calculating at least one original sound parameter among the preset sound parameters by an algorithm to generate at least one newly added sound parameter corresponding to the character information, and generating a speech according to the at least one newly added sound parameter. A system as described in claim 6, wherein when the character information does not conform to the preset sound parameters, the at least one original sound parameter includes at least a first original sound parameter and a second original sound parameter that are different from each other, and the processor generates the at least one newly added sound parameter based on the first original sound parameter and the second original sound parameter. A system as described in claim 7, wherein the processor generates a parameter quantity of the at least one newly added sound parameter that is the same as a character quantity corresponding to the character information. The system as described in claim 6, wherein the processor compares at least one characteristic parameter of the character information with the preset sound parameters, and when the at least one characteristic parameter does not match any of the preset sound parameters, the processor generates the at least one new sound parameter that matches the at least one characteristic parameter based on the preset sound parameters. A system as described in claim 6, wherein the processor receives the role information through an input interface, or extracts the role information from a text stored in the storage unit.

Images