CN114420084A - Voice generation method, device and storage medium - Google Patents

Abstract

The application relates to a voice generation method, a voice generation device and a storage medium. The method comprises the following steps: performing text analysis processing on a text to be processed to obtain text information of each text unit in the text to be processed; determining target prosody information of each text unit according to the text information and symbol information of punctuations contained in the text to be processed; and generating target voice according to the text information and the target prosody information. According to the method and the device, the symbol information and the text information of the punctuation marks are combined, so that the punctuation marks in the text to be processed can act on the generated target voice, and the generated target voice can better accord with the emotional scene and the like when the characters are written; in the second aspect, by introducing the symbol information of punctuation, the target voice can better meet the requirement of the context where the text is located, and the target voice is more natural, real and close to the natural language expression mode.

Description

Voice generation method, device and storage medium

technical field

The present application relates to the field of computer technology, and in particular, to a voice generation method, device, and storage medium.

Background technique

Voice interaction is the most direct, natural and effective way people use to communicate information. With the rapid development of electronic devices such as mobile phones in recent years, new voice interaction methods have also become a hot spot in scientific research such as computer, linguistics, and communication. Speech interaction is a human-machine dialogue technology based on speech recognition, natural language understanding and speech synthesis. Speech synthesis is one of the cores of speech interaction. Its purpose is to enable electronic devices to emit natural sounds like humans. Therefore, how to make mobile phones, computers and other electronic devices can "speak" exactly like people, broadcast a natural and smooth voice with an emotional tone, so that the broadcast voice is more in line with the corresponding tone scene of the text content input by the user, etc., is problems to be solved at present.

SUMMARY OF THE INVENTION

In order to overcome the problems existing in the related art, the present application provides a voice generation method, device and storage medium.

According to a first aspect of the embodiments of the present application, a method for generating speech is provided, including:

Performing text analysis processing on the text to be processed to obtain text information of each text unit in the text to be processed;

Determine the target prosody information of each of the text units according to the text information and the symbol information of the punctuation marks contained in the text to be processed;

A target speech is generated according to the text information and the target prosody information.

In some embodiments, the text information includes: text phonemes and text tones; the target prosody information of each of the text units is determined according to the text information and the symbol information of punctuation marks included in the text to be processed ,include:

According to the text phoneme and the text pitch of each of the text units, determine the text prosody information of each of the text units;

According to the symbol information, determine the symbol prosody information of each of the text units;

The target prosody information is determined according to the text prosody information and the symbol prosody information.

In some embodiments, the symbol information includes: symbol position; and the determining, according to the symbol information, symbol prosody information of each of the text units includes:

According to the symbol positions of the punctuation marks, determine the text units that each punctuation mark acts on; wherein, one punctuation mark acts on at least one text unit;

Symbolic prosody information of the text unit to which the punctuation marks act is determined.

In some embodiments, the symbol information includes: symbol type and number of symbols; and determining the symbol prosody information of each of the text units according to the symbol information, including:

Determine the tone type according to the symbol type of the punctuation mark;

Determine the tone degree according to the number of the punctuation marks;

The symbol prosody information is determined according to the tone type and/or the tone degree.

In some embodiments, generating the target speech according to the text information and the target prosody information includes:

Matching the text information and the target prosody information with preset text information and preset prosody information in a preset voice library; wherein the preset voice library is used to store audio clips, and the preset text information and the mapping relationship between the preset prosody information and the audio segment;

Using the speech generation strategy corresponding to the matching result, the target speech is generated according to the text information and the target prosody information; wherein, different matching results correspond to different speech generation strategies.

In some embodiments, generating the target speech according to the text information and the target prosody information by using the speech generation strategy corresponding to the matching result includes:

In the case that the matching result indicates that the text information and the target prosody information exist in the preset voice library, determine the audio corresponding to the text information and the target prosody information from the preset voice library fragment;

Perform splicing processing on each of the audio segments to generate the target speech.

In some embodiments, generating the target speech according to the text information and the target prosody information by using the speech generation strategy corresponding to the matching result includes:

In the case that the matching result indicates that the text information and the target prosody information do not exist in the preset speech library, feature conversion processing is performed on the text information and the target prosody information of each of the text units , get the acoustic features;

Decoding each of the acoustic features is performed to generate the target speech.

According to a second aspect of the embodiments of the present application, there is provided a voice generation apparatus, including:

an analysis module, configured to perform text analysis processing on the text to be processed, and obtain text information of each text unit in the text to be processed;

a determining module, configured to determine the target prosody information of each of the text units according to the text information and the symbol information of the punctuation marks contained in the text to be processed;

A generating module is configured to generate a target speech according to the text information and the target prosody information.

In some embodiments, the text information includes: text phonemes and text tones; the determining module is configured to:

According to the text phoneme and the text pitch of each of the text units, determine the text prosody information of each of the text units;

According to the symbol information, determine the symbol prosody information of each of the text units;

The target prosody information is determined according to the text prosody information and the symbol prosody information.

In some embodiments, the symbol information includes: symbol position; and the determining module is configured to:

According to the symbol positions of the punctuation marks, determine the text units that each punctuation mark acts on; wherein, one punctuation mark acts on at least one text unit;

Symbolic prosody information of the text unit to which the punctuation marks act is determined.

In some embodiments, the symbol information includes: symbol type and symbol quantity; and the determining module is configured to:

Determine the tone type according to the symbol type of the punctuation mark;

Determine the tone degree according to the number of the punctuation marks;

The symbol prosody information is determined according to the tone type and/or the tone degree.

In some embodiments, the generating module is configured to:

Matching the text information and the target prosody information with preset text information and preset prosody information in a preset voice library; wherein the preset voice library is used to store audio clips, and the preset text information and the mapping relationship between the preset prosody information and the audio segment;

Using the speech generation strategy corresponding to the matching result, the target speech is generated according to the text information and the target prosody information; wherein, different matching results correspond to different speech generation strategies.

In some embodiments, the generating module is configured to:

In the case that the matching result indicates that the text information and the target prosody information exist in the preset voice library, determine the audio corresponding to the text information and the target prosody information from the preset voice library fragment;

Perform splicing processing on each of the audio segments to generate the target speech.

In some embodiments, the generating module is configured to:

In the case that the matching result indicates that the text information and the target prosody information do not exist in the preset speech library, feature conversion processing is performed on the text information and the target prosody information of each of the text units , get the acoustic features;

Decoding each of the acoustic features is performed to generate the target speech.

According to a third aspect of the embodiments of the present application, there is provided a voice generation apparatus, including:

processor;

a memory configured to store processor executable instructions;

Wherein, the processor is configured to implement the steps in any one of the voice generation methods in the first aspect when executed.

According to a fourth aspect of the embodiments of the present application, a non-transitory computer-readable storage medium is provided, when an instruction in the storage medium is executed by a processor of a speech generating apparatus, the apparatus can execute the above-mentioned first aspect Steps in any speech generation method.

The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

The present application determines the text information of each text unit in the text to be processed and the symbol information of the punctuation marks contained in the text to be processed, and jointly determines the target prosody information of each text unit, and then generates the text information and target prosody information through the obtained text information and target prosody information. target voice. In this way, by combining the symbol information of the punctuation marks with the text information, in the first aspect, the punctuation marks in the text to be processed can be made to have an effect on the generated target voice, thereby making the generated target voice more in line with the emotional scene when writing the text etc.; secondly, by introducing the symbolic information of punctuation marks, the target speech can be made more in line with the needs of the context in which the text is located, more natural and realistic and close to natural language expressions.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application.

FIG. 1 is a flowchart of a method for generating speech according to an exemplary embodiment of the present application.

FIG. 2 is a schematic diagram of a speech synthesis system according to an exemplary embodiment of the present application.

Fig. 3 is a block diagram of a voice generation apparatus according to an exemplary embodiment of the present application.

FIG. 4 is a block diagram of a hardware structure of a speech generating apparatus according to an exemplary embodiment of the present application.

FIG. 5 is a block diagram of the hardware structure of a voice generation apparatus according to an exemplary embodiment of the present application.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as recited in the appended claims.

Fig. 1 is a flow chart of a method for generating speech according to an exemplary embodiment. As shown in Fig. 1 , the method mainly includes the following steps:

In step 101, text analysis processing is performed on the text to be processed to obtain text information of each text unit in the text to be processed;

In step 102, according to the text information and the symbol information of the punctuation marks contained in the text to be processed, determine the target prosody information of each of the text units;

In step 103, a target speech is generated according to the text information and the target prosody information.

In some embodiments, the voice generation method of the present application may be applied to an electronic device, where the electronic device may include a terminal device, such as a mobile terminal, a fixed terminal, or a vehicle-mounted terminal. The mobile terminal may include devices such as mobile phones, tablet computers, notebook computers, or wearable devices, and may also include smart home devices, such as smart speakers. Fixed terminals may include: desktop computers or smart TVs. The vehicle-mounted terminal may include a front-end device of a vehicle monitoring and management system, and may also be called a vehicle dispatch monitoring (Telematics Control Unit, TCU) terminal, such as a vehicle-machine terminal and the like. The vehicle terminal can integrate global positioning system (Global Positioning System, GPS) technology, mileage positioning technology and car black box technology, and can be used for modern management of vehicles, including: driving safety monitoring management, operation management, service quality management, intelligent Centralized scheduling management, electronic stop sign control management, etc.

In this embodiment of the present application, the text to be processed may refer to the text that needs to be processed currently, and the text may include text content and punctuation marks. A text can be a sentence or a combination of sentences with complete, systematic meaning with punctuation marks, a text can be a sentence, a paragraph or a chapter, such as short texts such as short sentences, proverbs, aphorisms, titles, etc., or articles , document text, etc. The text to be processed may be represented by a vector sequence or the like (for example, but not limited to [I do not like the story of the movie, very much!]), which is not specifically limited in this embodiment of the present application. The text unit may refer to a unit with its own attributes contained in the text to be processed. For example, the text unit may be a single word or word in the text to be processed. For example, the text to be processed is "Zhang San takes the high-speed train to a certain city.", Then the text units included in the text to be processed are "Zhang San", "Chang", "High-speed rail", "Go to" and "A certain city" in sequence.

Text information may refer to information used to represent attributes of text units, such as information that can identify attributes such as semantics and pronunciation of text units, and text information may include at least one of the following: phoneme information; pitch information; location information; grammar information ; pronunciation duration; frequency information, etc. For example: if the text to be processed is Chinese text (for example, there is a reason...), the electronic device can perform phoneme conversion processing on the Chinese text to obtain a phoneme sequence (that is, a continuous pinyin string) (for example, [youyi ge dao li ...]), and then perform pitch tagging processing on the phoneme sequence to obtain a phoneme sequence with pitch tags (for example, [you3 yi1 ge4 dao4 li3...]), and the electronic device can convert the phoneme sequence with pitch tags etc. as text information.

In some embodiments, the electronic device may obtain text information of each text unit in the text to be processed by performing text analysis processing on the text to be processed. Text analysis (also known as linguistic analysis, which mainly simulates the process of human understanding of natural language) can refer to the feature extraction or feature conversion of the text to be processed in order to obtain text information. Text analysis can mainly include: text regularization Processing, word segmentation, part-of-speech tagging, polyphonic word disambiguation, etc. Electronic devices can take sentences as processing units, and perform lexical, grammatical and semantic analysis of the text to be processed sentence by sentence, so as to determine the underlying structure of the sentence (such as the structure of syllables or the structure of words). structure) and the composition of each character phoneme, etc., text analysis may also include: text segmentation, word segmentation, processing of numbers, characters, processing of abbreviations, etc., to obtain text information that can be understood by electronic devices.

Punctuation marks can refer to symbols used in writing to indicate the reading and tone of sentences. Punctuation marks are symbols that assist written language to record language and are an integral part of written language. They can be used to indicate pauses, tone, and the nature and function of words. Punctuation marks can be Including labels and dots, etc. In this embodiment of the present application, the text to be processed may include text content and punctuation marks, and the symbol information of the punctuation marks may act on the text content. Symbol information may refer to information used to characterize the attributes of punctuation marks. For example, symbol information may include information on attributes such as type and location of punctuation marks. Symbol information of punctuation marks may include information such as symbol name, symbol position, and number of symbols. . Taking the text to be processed as "in class?" as an example, it can be determined that the symbol name of the punctuation mark is a question mark, and the symbol position is after the text unit "class", then the electronic device can determine that the text unit "class" in the text to be processed should be It is broadcast in an interrogative tone.

The rhythm of speech is acoustically characterized by pauses, pitch, length, intensity, etc., also known as supersegmental features. When electronic equipment broadcasts speech, it broadcasts different text units "priority, cadence" and other different Tone characteristics, to reflect the rhythm of speech. Prosody information (also called prosodic parameters or prosodic sequence, etc.) may refer to attribute information used to characterize prosody characteristics. The target prosody information is used to represent the prosody information corresponding to each text unit in the text to be processed, and is distinguished from the prosody information corresponding to the entire text to be processed.

In some embodiments, the electronic device may determine the prosody information by performing prosody prediction processing on the text to be processed, and the prosody prediction processing may be understood as performing intonation, accent, duration, and pause processing on the text to be processed, so as to determine the specific value of each text unit. Pronunciation, pause position and pronunciation, such as cadence characteristics and other information. When the electronic device performs prosody prediction, it can process the text to be processed according to different prosody levels, and obtain prosody information corresponding to different prosody levels, which can include: prosody word (Prosody Word, PW) level, prosody phrase (Prosody Phrase, PP) level and Intonation Phrase (IP) level, etc., which are not specifically limited in this application.

In this embodiment of the present application, the electronic device may determine the target prosody information of each text unit according to the text information and the symbol information of the punctuation marks included in the text to be processed. The electronic device may jointly determine the prosodic information based on the text information corresponding to the text unit in the text to be processed and the symbol information corresponding to the punctuation marks. In a possible embodiment, the electronic device may first generate the first prosody information based on the text unit, then generate the second prosody information based on the symbol information, and then perform comprehensive processing on the first prosody information and the second prosody information (such as , adding the first prosody information and the second prosody information, etc.) to obtain the target prosody information. The electronic device may also directly generate target prosody information and the like according to text information and symbol information, which is not specifically limited in this application.

In a possible embodiment, the electronic device may input the text unit into a prosody prediction model to obtain the first prosody information, and the prosody prediction model may refer to a trained neural network for generating the first prosody information according to the text unit Model. Taking the prosody prediction model as an example based on a bidirectional long short-term memory neural network model, the electronic device can first determine each text unit of the text to be processed, then convert the text unit into a word embedding vector, and input each word embedding vector into the neural network model. For the predicted labels of each text unit, the first prosodic information is determined according to the predicted punctuation.

In some embodiments, the electronic device may pre-define the correspondence between the device symbol information and the second prosody information, and after the electronic device determines the symbol information, the second prosody information may be determined according to the symbol information and the corresponding relationship. For example, the electronic device can preset the question mark corresponding to the interrogative tone, the exclamation mark corresponding to the emotional tone, etc., the second prosodic information corresponding to the interrogative tone is a, the second prosody information corresponding to the emotional tone is b, and so on. If the electronic device recognizes that there is a question mark in the text to be processed, it can determine that the second prosody information is a, wherein the second prosody information a can be applied to the text unit before or after the question mark alone, or can be applied to the sentences before and after the question mark. Moderate, the scope of the role of the second prosody information is not specifically limited, and the settings can be customized according to actual usage requirements.

After the electronic device determines the target prosody information of each text unit, it can generate the target speech according to the text information and the target prosody information. The target voice may refer to audio data broadcast based on the text to be processed, and the audio data may include data such as sound signals or voice waveform data. According to the main idea of design, the method of generating target speech by electronic equipment can be divided into two categories: rule-driven method and data-driven method. The main idea of rule-driven is to formulate rules according to the physical process of human pronunciation to simulate and reproduce the pronunciation process, while data-driven is to use the data in the speech database to complete speech synthesis through statistical modeling, and to improve the quality of the speech corpus and the smallest unit. And with more reliance on scale, the synthesized speech is more natural and smooth. For example, speech generation methods such as formant synthesis, waveform splicing, harmonic plus noise model, neural network and end-to-end deep neural network model, etc., the specific method for generating target speech is not specifically limited in this application.

The present application determines the text information of each text unit in the text to be processed and the symbol information of the punctuation marks contained in the text to be processed, and jointly determines the target prosody information of each text unit, and then generates the text information and target prosody information through the obtained text information and target prosody information. target voice. In this way, by combining the symbol information of the punctuation marks with the text information, in the first aspect, the punctuation marks in the text to be processed can be made to have an effect on the generated target voice, thereby making the generated target voice more in line with the emotional scene when writing the text etc.; secondly, by introducing the symbolic information of punctuation marks, the target speech can be made more in line with the needs of the context in which the text is located, more natural and realistic and close to natural language expressions.

In some embodiments, the text information includes: text phonemes and text tones; the target prosody information of each of the text units is determined according to the text information and the symbol information of punctuation marks included in the text to be processed ,include:

According to the text phoneme and the text pitch of each of the text units, determine the text prosody information of each of the text units;

According to the symbol information, determine the symbol prosody information of each of the text units;

The target prosody information is determined according to the text prosody information and the symbol prosody information.

In this embodiment of the present application, the text information may at least include text phonemes and text tones. Phoneme is the smallest phonetic unit divided according to the natural properties of speech. It is analyzed according to the pronunciation action in the syllable, and an action constitutes a phoneme. For example, the Chinese syllable ah (ā) has only one phoneme, love (ài) has two phonemes, and dai (dài) has three phonemes. The level of sound frequency is called pitch (Pitch), which is one of the three main subjective attributes of sound, namely volume (loudness), pitch, and timbre (also known as fret). The pitch can indicate the degree to which the user's hearing can distinguish the pitch of a sound. The pitch is mainly determined by the frequency of the sound, and is also related to the intensity of the sound. The text phoneme may refer to the text content based on the text to be processed, the determined phoneme, and the text pitch may refer to the text content based on the text to be processed, the determined pitch, in order to distinguish it from the phoneme and pitch determined based on other content such as punctuation marks.

In a possible embodiment, the electronic device may convert the text to be processed into text phonemes through phoneme conversion processing. For example, an electronic device can use the trained phoneme conversion model to convert the to-be-processed text of "going to school tomorrow" into text phonemes in the form of "[ming tian qu xue xiao]". Then the electronic device can perform pitch labeling processing on the text phonemes. For example, the electronic device can use the syllable as the processing unit to query the pitch corresponding to each syllable in the preset dictionary to obtain the text pitch [2,1,4,2,4 ], etc., the electronic device can use a sequence of text phonemes and text tones to comprehensively represent text information, for example, [ming2 tian1 qu4 xue2 xiao4] and so on.

After the electronic device determines the text phoneme and the text pitch, it can perform prosody prediction processing according to the text phoneme and the text pitch to obtain text prosody information of each text unit. For example, the electronic device can use machine learning methods to predict the text phoneme and text pitch to be predicted according to the pre-trained prosody prediction model, so as to obtain the pause prediction result corresponding to the text unit (for example, the pause position, including the upper pause and the short pause) The text prosody information such as the pause type of the pause and the probability value relative to the pause type, etc.).

After the electronic device determines the symbol information, it can determine the symbol prosody information of each text unit according to the symbol information. The electronic device may preset the correspondence between the symbol information and the symbol prosody information, and then determine the symbol prosody information according to the specific symbol information and the corresponding relationship. For example, the electronic device presets that the first symbol information corresponds to the first symbol prosody information, the second symbol information corresponds to the second symbol prosody information, etc., and then the electronic device can store the corresponding relationship in the memory of the electronic device, after determining the first symbol information Afterwards, the first symbol prosody information and the like can be obtained by querying the memory. In another possible embodiment, the electronic device may further obtain the symbol prosody information according to the symbol information through the symbol prosody prediction model. The symbolic prosody prediction model may refer to a trained neural network model, which inputs symbolic information and outputs corresponding symbolic prosody information, etc. For example, an electronic device can obtain specific symbolic prosody information by inputting the symbolic information into the trained symbolic prosody prediction model Wait. During the training process, the electronic device can input the marked symbolic information and symbolic prosody information into the initialized symbolic prosody prediction model to train the configuration parameters, and obtain a trained symbolic prosody prediction model.

After the electronic device determines the text prosody information and the symbol prosody information, it can determine the target prosody information according to the text prosody information and the symbol prosody information. In an optional embodiment, the electronic device may add and fuse text prosody information and symbol prosody information to obtain target prosody information. For example, the electronic device determines that the text prosody information is in the form of a [0.3, 0.2, 0.7] sequence, where the numerical values in the sequence can represent the attribute values of different attributes of prosody (for example, pitch is 0.3, length is 0.2, and intensity is 0.7, etc. ). The electronic device may determine that the symbol prosody information is [0.6, 0.5, 0.1], etc., then the electronic device may determine that the target prosody information is [0.9, 0.7, 0.8], etc.

In a possible embodiment, the electronic device may also perform weighted average processing on the text prosody information and the symbol prosody information. For example, the electronic device first determines that the weight of the text prosody information is 0.4, the weight of the symbol prosody information is 0.6, etc., and then The weighted mean of text prosody information and symbol prosody information is calculated according to specific weights. The weights between the text prosody information and the symbol prosody information corresponding to different text units may be the same or different. The electronic device can determine the weight according to the part of speech corresponding to the text unit. For example, the text unit is the subject, and the weights corresponding to the text prosody information and the symbol prosody information can be 0.3 and 0.7 respectively; the text unit is the predicate, the text prosody information and the symbol prosody information. The corresponding weights can be 0.4 and 0.6, etc., respectively.

In the embodiment of the present application, the text prosody information of each text unit is determined according to the text phonemes and text tones of each text unit, the symbol prosody information of each text unit is determined according to the symbol information, and the symbol prosody information of each text unit is determined according to the text prosody information and the symbol prosody information. Target prosody information. Compared with the related art, only the prosody information is determined based on the text information of the text to be processed, and other information of the text to be processed is not considered to determine the prosody information. information and symbolic prosody information, so as to determine more accurate and rich target prosody information.

In some embodiments, the symbol information includes: symbol position; and the determining, according to the symbol information, symbol prosody information of each of the text units includes:

According to the symbol positions of the punctuation marks, determine the text units that each punctuation mark acts on; wherein, one punctuation mark acts on at least one text unit;

Symbolic prosody information of the text unit to which the punctuation marks act is determined.

In this embodiment of the present application, the symbol information of the punctuation mark may include at least the position of the symbol. The electronic device can determine the symbol position of each punctuation mark by performing character string recognition processing on the text to be processed. For example, the electronic device determines that the first punctuation mark is a comma, and the symbol position of the first punctuation mark is 10. The second punctuation mark is a period, and the symbol position of the second punctuation mark is 23 and so on. Since the electronic device does not broadcast the punctuation directly when broadcasting the voice, it is necessary to determine the text unit that each punctuation mark acts on. After the electronic device determines the symbol position of the punctuation mark, it can determine the text unit that each punctuation mark acts on. Among them, a punctuation mark acts on at least one text unit.

In some embodiments, the electronic device may preset the scope of action, and the electronic device may use a text unit before the punctuation mark as the text unit to be used, or a text unit after the punctuation mark as the text unit to be used. One or more text units between punctuation marks can be used as the text unit to be used, and so on. For example, for the to-be-processed text "in class?", the electronic device determines that the text unit used by the punctuation mark "?" can be "in class". For the to-be-processed text "?? Is Zhang San telling the truth?", the electronic device determines that the text unit used by the punctuation mark "??" can be "Zhang San" or the like. For the to-be-processed text "Li Si yesterday! He skipped class to watch a movie", the electronic device determines that the text units used by the punctuation mark "!" can be "yesterday" and "skipped class".

After the electronic device determines the text unit acted by each punctuation mark, it can separately determine the symbol prosody information of the text unit acted by the punctuation mark, for example: the unsigned prosody of the first, second and third text units in the text to be processed information, the symbolic prosody information of the fourth text unit can be [0.2, 0.5, 0.7], etc. When an electronic device broadcasts the text of a sentence-level unit composed of multiple text units, it can use the symbol prosody information to broadcast the corresponding tone, or it can broadcast only some text units in the text to be processed, using the symbol prosody information to broadcast the corresponding tone. Wait. In a possible embodiment, the electronic device may further jointly determine the text units to which the punctuation marks act according to the combination of different types of punctuation marks. For example, the electronic device can determine that the text to be processed is "Today, the teacher read "Nostalgia!", then the electronic device can determine the text unit used by the exclamation corresponding to the exclamation mark "!", which can be the text content "Nostalgia" in the title number. Wait. Electronic devices can also use quotation marks (such as, "How are you?", etc.) or consecutive, identical multiple punctuation marks (such as, ? Not going to work tomorrow?. etc.) and other forms to determine the role of punctuation marks text unit.

In the embodiment of the present application, by determining the text units acted by each punctuation mark according to the symbol positions of the punctuation marks, wherein one punctuation mark acts on at least one text unit, and determining the symbol prosody information of the text unit acted by the punctuation mark, you can Accurately determining the symbol prosody information corresponding to the text to be processed can effectively reduce the workload compared to determining the symbol prosody information corresponding to each text unit, and make the broadcast tone more natural and smooth in the process of broadcasting the text to be processed.

In some embodiments, the symbol information includes: symbol type and number of symbols; and determining the symbol prosody information of each of the text units according to the symbol information, including:

Determine the tone type according to the symbol type of the punctuation mark;

Determine the tone degree according to the number of the punctuation marks;

The symbol prosody information is determined according to the tone type and/or the tone degree.

In this embodiment of the present application, the symbol information at least includes: symbol type and symbol quantity. Symbol types can include: pause, read-free, or tone. For example, the pause class may include: commas, commas, periods, semicolons, hyphens and other punctuation marks; the read-free class may include: book title, brackets and other punctuation marks; the tone class may include question marks, exclamation marks and other punctuation marks; Classes and read-free classes can also be referred to as voiceless types. The tone types may at least include: declarative, imperative, exclamatory, interrogative and other types of tone, and the electronic device may pre-set the correspondence between different punctuation marks and different tone types. For example, the period corresponds to the declarative tone, the exclamation mark corresponds to the exclamation tone, the question mark corresponds to the interrogative tone, etc. For punctuation marks without obvious mood characteristics, the electronic device can set the punctuation marks as the default mood type, for example: the electronic device can set the comma, semicolon and other punctuation marks as the default mood type, etc.

The electronic device can determine the tone level according to the number of punctuation marks. For example, a question mark "?" can indicate a common question mark, two or more consecutive question marks "??" can indicate a strong question mark, etc.; an exclamation mark " !" can express a common exclamation level, and two or more consecutive exclamation marks "!!" can express a strong exclamation level, etc. In a possible embodiment, different types of punctuation marks can also be combined, such as "?!" or "??!!", etc., which can be used to express strong doubts and surprises, so before the question mark Text units can be marked with a questionable surprise level or a strong questionable surprise level (if there are two or more), etc.

After the electronic device determines the symbol type and the number of symbols, it can determine the symbol prosody information according to the tone type and/or tone degree. For example, if the electronic device determines that the text to be processed has 5 text units, and there is a question mark after the last text unit, the electronic device may determine that the symbol prosody information corresponding to the fifth text unit may be an ordinary interrogative tone, etc.

In the embodiment of the present application, by determining the mood type according to the symbol type of punctuation marks, determining the mood degree according to the number of punctuation marks, and determining the symbol prosody information according to the mood type and/or mood degree, the symbol prosody can be determined simply and effectively information, improve the operating efficiency of electronic equipment, etc.

In some embodiments, generating the target speech according to the text information and the target prosody information includes:

Matching the text information and the target prosody information with preset text information and preset prosody information in a preset voice library; wherein the preset voice library is used to store audio clips, and the preset text information and the mapping relationship between the preset prosody information and the audio segment;

Using the speech generation strategy corresponding to the matching result, the target speech is generated according to the text information and the target prosody information; wherein, different matching results correspond to different speech generation strategies.

In this embodiment of the present application, the preset speech library may be used to store audio clips, and the mapping relationship between preset text information and preset prosody information and audio clips. For example, the first preset text information and the first preset prosody information correspond to the first audio segment, the second preset text information and the second preset prosody information correspond to the second audio segment, and so on. The audio segment may refer to audio data corresponding to a short text (eg, text units such as words or words), and the electronic device may combine multiple audio segments to form a target speech corresponding to the complete text to be processed.

There are both textual information and target prosody information

The electronic device may match the text information and the target prosody information with the preset text information and the preset prosody information in the preset speech library. The matching result can at least include: matching success and matching failure. If both the preset text information corresponding to the text information and the preset prosody information corresponding to the target prosody information exist in the preset voice library, the electronic device may determine that the matching is successful. If the preset text information corresponding to the text information and the preset prosody information corresponding to the target prosody information do not exist in the preset voice library at the same time (for example, the preset voice library only contains preset text information corresponding to the text information, or the preset voice There is only preset prosody information corresponding to the target prosody information in the library), or there is neither preset text information corresponding to the text information nor preset prosody information corresponding to the target prosody information in the preset voice library, then the electronic device can determine Match failed. For example: the electronic device determines that the text information matches the first preset text information successfully, and the target prosody information matches the first preset prosody information successfully, then the electronic device can determine that the matching result is successful; if the electronic device determines that the text information matches the first preset prosody information successfully 2. The preset text information is successfully matched, and the target prosody information does not match all the preset prosody information, then the electronic device may determine that the matching result is a matching failure, etc.

After the electronic device determines the matching result, it can generate the target speech according to the text information and the target prosody information by using the speech generation strategy corresponding to the matching result, wherein different matching results correspond to different speech generation strategies. For example, successful matching may correspond to the first voice generation strategy, and failure of matching may correspond to the second voice generation strategy. The speech generation strategy may refer to the way or rules for the electronic device to generate the target speech, such as: formant synthesis strategy, pronunciation process synthesis strategy, waveform splicing strategy, neural network model strategy, etc. For example, if the matching result is successful, the electronic device can use the first voice generation strategy to generate the target voice; if the matching result is a match failure, the electronic device can use the second voice generation strategy to generate the target voice, etc.

In the embodiment of the present application, by matching the text information and the target prosody information with the preset text information and the preset prosody information in the preset voice database, using the voice generation strategy corresponding to the matching result, according to the text information and the target prosody information , to generate the target speech, a variety of different strategies can be used to ensure the complete target speech accurately and effectively.

In some embodiments, generating the target speech according to the text information and the target prosody information by using the speech generation strategy corresponding to the matching result includes:

In the case that the matching result indicates that the text information and the target prosody information exist in the preset voice library, determine the audio corresponding to the text information and the target prosody information from the preset voice library fragment;

Perform splicing processing on each of the audio segments to generate the target speech.

In this embodiment of the present application, when the matching result indicates that text information and target prosody information exist in the preset voice library, that is, when the matching result is successful, the electronic device can determine the text information and target prosody information from the preset voice library. The audio segment corresponding to the target prosody information. For example, the electronic device determines that the text information and target prosody information of the first text unit correspond to the first audio segment, and the text information and target prosody information of the second text unit correspond to the second audio segment, and so on. The electronic device can then perform splicing processing on each audio segment to generate the target speech.

The electronic device can determine the splicing sequence according to the position information between each text unit. For example, the electronic device determines that the first text information is the first text unit in the text to be processed, and the second text information is the first text unit in the text to be processed. two text units, then the electronic device can determine that the first audio segment is connected to the second audio segment, and the first audio segment is in front. When splicing each audio segment, the electronic device can also perform scaling, filtering, enhancement, etc. on each audio segment, so as to ensure the natural and smoothness of the generated target speech.

In the embodiment of the present application, when the matching result indicates that there are text information and target prosody information in the preset voice library, audio fragments corresponding to the text information and target prosody information are determined from the preset voice library, and each audio fragment is analyzed. The splicing process generates the target voice, and the audio clip can be simply and accurately determined directly from the preset voice library, which has a relatively stable synthesis effect and improves the operation efficiency of electronic equipment.

In some embodiments, generating the target speech according to the text information and the target prosody information by using the speech generation strategy corresponding to the matching result includes:

In the case that the matching result indicates that the text information and the target prosody information do not exist in the preset speech library, feature conversion processing is performed on the text information and the target prosody information of each of the text units , get the acoustic features;

Decoding each of the acoustic features is performed to generate the target speech.

In the embodiment of the present application, when the matching result indicates that there is no text information and target prosody information in the preset voice library, that is, when the matching result is that the matching fails, the electronic device can compare the text information and target of each text unit. The prosodic information is subjected to feature conversion processing to obtain acoustic features. Acoustic features (also referred to as acoustic parameters or speech parameters, etc.) can be physical quantities that represent the acoustic characteristics of speech, and are also a general term for the acoustic performance of various elements of the sound. For example, it represents the energy concentration area, formant frequency, formant intensity and bandwidth of the timbre, as well as the duration, fundamental frequency, and average voice power, which represent the prosody characteristics of speech. For example, electronic devices can convert text information and target prosody information into acoustic features through a trained acoustic model.

After the electronic device obtains the acoustic features, it can decode each acoustic feature to generate the target speech. For example, the electronic device can combine the acoustic features corresponding to each text unit into an acoustic feature sequence or set according to the position between the text units. etc., and input it into the trained vocoder to obtain the target speech (also referred to as the target waveform corresponding to the text to be processed, etc.). Vocoder refers to a neural network model used to convert the previously obtained high-level features (ie, acoustic features) into sound waveforms.

In a possible embodiment, the electronic device can also generate the target speech by using strategies such as end-to-end, and the end-to-end synthesis strategy can directly input the text to be processed or the phonetic characters of the text to be processed, and directly output the audio waveform (also the target voice). The electronic device can utilize an end-to-end model (eg, TACOTRON model) to implement an end-to-end synthesis strategy, and the electronic device can input text information and target prosody information into the trained end-to-end model and input target speech. In the training process of the end-to-end model, it needs to be trained according to data such as labeled text information, prosody information, and speech. In the embodiment of the present application, the electronic device uses the end-to-end strategy to generate the target speech, which can reduce the requirements for linguistic knowledge, and at the same time, it can be copied in different languages conveniently and quickly, and the synthesis of dozens or even more languages can be realized in batches. strategy, and the end-to-end strategy has strong and rich pronunciation style and prosodic expressiveness.

In a possible embodiment, the electronic device may first generate the first voice according to the text prosody information, and then adjust the first voice according to the symbol prosody information to generate the target voice. The electronic device presets an adjustment method corresponding to the symbol prosody information. For example, if the text to be processed contains an exclamation mark, the electronic device can indicate that the amplitude of the first voice is reduced by 10%, etc., so as to obtain the target voice. In the above manner, the electronic device can directly use the existing speech synthesis strategy to generate the first speech through the text prosody information, and only need to add a small number of adjustment processing modules to the process logic in the prior art to generate the target speech, reducing the Developer workload, etc.

In a possible embodiment, the speech generation method in the present application may be applied in a speech synthesis system, as shown in FIG. 2 , which may represent a schematic diagram of a speech synthesis system. As shown in FIG. 2, the speech synthesis system may include: a front-end processing module 201 and a back-end processing module, wherein the back-end processing module may include: a first back-end module and/or a second back-end module, the first back-end module An acoustic model 202 and a vocoder 203 may be included, and the second back-end module may include a preset speech library 204 and the like. During the processing, the electronic device may determine the text to be processed, where the text to be processed includes text content and punctuation marks. Electronic devices can be processed by crawlers to obtain pending texts in the form of articles on the Internet, and can also receive pending texts uploaded by users through components such as mouse or keyboard. Then the electronic device can input the text to be processed into the front-end processing module 201 for processing to obtain target prosody information.

Here, the speech synthesis system can be mainly divided into two modules: a front-end processing module 201 and a back-end processing module. The front-end processing module 201 can mainly perform the processing of links (sub-modules) such as text regularization, word segmentation, part-of-speech tagging, polyphonic word disambiguation, and prosody prediction, wherein in the prosody prediction link, a punctuation detection sub-module is added in this embodiment. , the user determines the symbol prosody information and the like according to the symbol information of the punctuation symbols in the text to be processed.

The front-end processing module 201 can parse the text to be processed input into the speech synthesis system. In the parsing process, the front-end processing will first normalize the text, and mark some special characters (for example, the interpretation of Arabic numerals can be numerals or phone numbers) according to certain rules. During this process, word segmentation processing is completed. Word segmentation is to cut the text to be processed according to the rules and phrases of the natural language, and the word segmentation will also be based on some special characters and punctuation information that were processed during the regularization of the text to be processed. After the word segmentation, enter the part-of-speech tagging link. The role of the part-of-speech tagging is to tag each word in the word segmentation result with a correct part-of-speech type (noun, verb, adjective or other parts of speech).

Then carry out the polyphonic word disambiguation link. This link will take the word segmentation as the processing unit (that is, the text unit). For each word segmentation, it can be compared with the vocabulary in the pronunciation library according to the longest matching principle to determine the phoneme and tone of each word segmentation, etc. information. After the matching is completed, the electronic device can also record the phoneme level (previous phoneme, next phoneme), syllable level (the syllable number of the word), word level (part of speech/position in the sentence), etc. corresponding to each participle. information, which can be used for subsequent temperament prediction. If the electronic device cannot find a matching result at the "word" level in the pronunciation library, the electronic device can use the "word" level that is smaller than the "word" level as a processing unit to perform matching. In this case, there may be polyphonic words. For the problem, you can simply select the pronunciation with the highest probability in the pronunciation database to get the text information. Then the electronic device can perform prosody prediction. In the process of prosody prediction, in addition to the prosody prediction of each word segment to obtain the text prosody information, the punctuation marks of each sentence can be processed in units of "sentence" to obtain the symbol prosody information. . For example, when a punctuation mark is encountered in the parsing process, the front-end processing module 201 can identify and record the type and range of the punctuation mark. The electronic device can then determine the target prosody information according to the text prosody information and the symbol prosody information.

In a possible embodiment, taking Chinese as an example, the speech synthesis system supports all the punctuation symbols specified in the national standard. For example, the punctuation marks for pauses can include, commas (","): commas are generally used for pauses in sentences. Generally, there is no special tone and the default tone level is used. Therefore, the processing logic is relatively simple. It is enough to record a "comma pause time" in the attribute of a participle. The comma (“,”): similar to the comma, it is generally used to pause between words and words in a sentence. Generally, there is no special tone and the default tone level is used, so the processing logic is relatively simple, and only the first 1 of the sentence is required. It is enough to record a "pause time of comma" in the attribute of each word segment, where the pause time of comma can be slightly less than the pause time of comma. Semicolon (";"): Semicolon is similar to comma, and is generally used for pauses between sentences. Generally, there is no special tone and the default tone level is used, so the processing logic is relatively simple, and only the attributes of the first participle at the sentence break are required. It is enough to record a "semicolon pause time", in which the general semicolon pause time can be slightly longer than the comma pause time. Period (“.”): The period is similar to the comma. It is generally used to pause at the end of the sentence. Generally, there is no special tone and the default tone level is used. Therefore, the processing logic is relatively simple, and it only needs to be recorded in the attribute of the first participle of the sentence. One "period pause time" is sufficient, among which, the pause time of the period is generally relatively large, which can be longer than the pause time of other punctuation marks. The hyphen (“-”): The hyphen is similar to the comma, and is generally used to connect two words. Generally, there is no special tone and the default tone level is used, so the processing logic is relatively simple, and only the first participle is required at the sentence break. It is enough to record a "connection number pause time" in the properties of , in which the general connection number pause time can be consistent with the pause time of the pause number. Ellipsis ("..."): The ellipsis is generally used to omit the scene. Generally, there is no special tone and the default tone level is used. It is only necessary to record an "ellipsis pause time" in the attribute of the first participle of the sentence. Among them, Generally, the ellipsis pause time can be slightly longer than the comma pause time. Dash (“—”): Dash is generally used for explanations or topic change scenarios. Generally, there is no special tone and the default tone level is used. It is only necessary to record a “dash pause time” in the attribute of the first participle of the sentence. . There is a special case of the dash, which is sometimes used to indicate a prolonged sound. This situation is currently not supported because the text content needs to be analyzed and processed. Generally, the dash pause time can be slightly longer than the comma pause time.

Read-free punctuation can include parentheses ("()"): The text in the parentheses is ignored and not processed. Book title number (""", "<>"): The content in the book title number is also not subject to special tone treatment, the default tone level is used, and there is no pause time.

The punctuation marks of the mood class can include, question mark ("?"): question marks are used in interrogative sentences, so the participle before the question mark will mark the "general interrogative mood level". If there are 2 or more question marks ("???"), the participle before the question mark will be marked as "strong question mark". Exclamation mark ("!"): The exclamation mark is used in exclamatory sentences, so the participle before the exclamation mark marks the "normal exclamation level". If there are 2 or more question marks ("!!!"), the participle before the question mark will be marked with "strong exclamation level". Question mark and exclamation mark combination ("?!" or "???!!!"): This combination is a special kind of usage, generally used for strong doubts and surprises, so the participle before the question mark will mark doubtful surprises Tone level or strong interrogative surprise tone level (if there are 2 or more).

The tone degree in the symbolic prosodic information can be divided into different levels, such as: no tone, ordinary interrogative tone, strong interrogative tone, ordinary exclamatory tone, strong exclamatory tone, etc. (more and more delicate tone types can be expanded). By detecting the last punctuation mark of the text to be processed, it is determined what kind of tone is used to broadcast the text to be processed. The tone will act on the last text unit after the word segmentation. For example, the text to be processed is: In class? The interrogative mood can be used, and the result after participle is: in /class/? , the word class will be read in a questioning tone; while in class! This text can be read out in an exclamatory tone, or in an exclamatory tone in class.

After the electronic device obtains the target prosody information after processing by the front-end processing module 201, the target prosody information can be input into the back-end processing module for back-end processing, wherein. The role of the back-end processing module is to generate target speech according to the target prosody information. The back-end processing module may include: a first back-end module and/or a second back-end module, and the first back-end module may include an acoustic model 202 and a vocoder 203. The second back-end module may include a preset voice library 204 and the like. The back-end processing can include speech synthesis based on statistical parameter modeling (or called parametric synthesis or parametric method, etc.), and speech synthesis based on unit selection and waveform splicing (or called splicing synthesis or splicing method, etc.). For the parametric method, the electronic device may first generate acoustic features through the acoustic model 202 . If the electronic device adopts the parametric method to generate the target voice, the electronic device can process the acoustic features through the vocoder 203 to obtain the target voice. If the electronic device uses the splicing method to generate the target speech, the electronic device can obtain the audio segments corresponding to each text unit from the preset speech library 204, and then perform splicing processing on the audio segments to generate the target speech.

The present application determines the text information of each text unit in the text to be processed and the symbol information of the punctuation marks contained in the text to be processed, and jointly determines the target prosody information of each text unit, and then generates the text information and target prosody information through the obtained text information and target prosody information. target voice. In this way, by combining the symbol information of the punctuation marks with the text information, in the first aspect, the punctuation marks in the text to be processed can be made to have an effect on the generated target voice, thereby making the generated target voice more in line with the emotional scene when writing the text etc.; secondly, by introducing the symbolic information of punctuation marks, the target speech can be made more in line with the needs of the context in which the text is located, more natural and realistic and close to natural language expressions.

Fig. 3 is a block diagram of a voice generating apparatus according to an exemplary embodiment. As shown in FIG. 3, the voice generation device 300 mainly includes:

The analysis module 301 is configured to perform text analysis processing on the text to be processed, and obtain text information of each text unit in the text to be processed;

The determining module 302 is configured to determine the target prosody information of each of the text units according to the text information and the symbol information of the punctuation marks contained in the text to be processed;

The generating module 303 is configured to generate a target speech according to the text information and the target prosody information.

In some embodiments, the text information includes: text phonemes and text tones; the determining module 302 is configured to:

According to the text phoneme and the text pitch of each of the text units, determine the text prosody information of each of the text units;

According to the symbol information, determine the symbol prosody information of each of the text units;

The target prosody information is determined according to the text prosody information and the symbol prosody information.

In some embodiments, the symbol information includes: symbol position; the determining module 302 is configured to:

According to the symbol positions of the punctuation marks, determine the text units that each punctuation mark acts on; wherein, one punctuation mark acts on at least one text unit;

Symbolic prosody information of the text unit to which the punctuation marks act is determined.

In some embodiments, the symbol information includes: symbol type and symbol quantity; the determining module 302 is configured to:

Determine the tone type according to the symbol type of the punctuation mark;

Determine the tone degree according to the number of the punctuation marks;

The symbol prosody information is determined according to the tone type and/or the tone degree.

In some embodiments, the generating module 303 is configured to:

Matching the text information and the target prosody information with preset text information and preset prosody information in a preset voice library; wherein the preset voice library is used to store audio clips, and the preset text information and the mapping relationship between the preset prosody information and the audio segment;

Using the speech generation strategy corresponding to the matching result, the target speech is generated according to the text information and the target prosody information; wherein, different matching results correspond to different speech generation strategies.

In some embodiments, the generating module 303 is configured to:

In the case that the matching result indicates that the text information and the target prosody information exist in the preset voice library, determine the audio corresponding to the text information and the target prosody information from the preset voice library fragment;

Perform splicing processing on each of the audio segments to generate the target speech.

In some embodiments, the generating module 303 is configured to:

In the case that the matching result indicates that the text information and the target prosody information do not exist in the preset speech library, feature conversion processing is performed on the text information and the target prosody information of each of the text units , get the acoustic features;

Decoding each of the acoustic features is performed to generate the target speech.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

Fig. 4 is a block diagram of a hardware structure of a voice generating apparatus according to an exemplary embodiment. For example, apparatus 400 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, and the like.

4, the apparatus 400 may include one or more of the following components: a processing component 402, a memory 404, a power supply component 406, a multimedia component 408, an audio component 410, an input/output (I/O) interface 412, a sensor component 414, and communication component 416 .

The processing component 402 generally controls the overall operation of the device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 402 may include one or more modules that facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

Memory 404 is configured to store various types of data to support operations at device 400 . Examples of such data include instructions for any application or method operating on device 400, contact data, phonebook data, messages, pictures, videos, and the like. Memory 404 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

Power supply assembly 406 provides power to various components of device 400 . Power supply components 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 400 .

Multimedia component 408 includes screens that provide an output interface between the device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, multimedia component 408 includes a front-facing camera and/or a rear-facing camera. When the apparatus 400 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 410 is configured to output and/or input audio signals. For example, audio component 410 includes a microphone (MIC) that is configured to receive external audio signals when device 400 is in operating modes, such as call mode, recording mode, and voice recognition mode. The received audio signal may be further stored in memory 404 or transmitted via communication component 416 . In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

The I/O interface 412 provides an interface between the processing component 402 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 414 includes one or more sensors for providing status assessment of various aspects of device 400 . For example, the sensor assembly 414 can detect the open/closed state of the device 400, the relative positioning of components, such as the display and keypad of the device 400, and the sensor assembly 414 can also detect a change in the position of the device 400 or a component of the device 400 , the presence or absence of user contact with the device 400 , the orientation or acceleration/deceleration of the device 400 and the temperature change of the device 400 . Sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 416 is configured to facilitate wired or wireless communication between apparatus 400 and other devices. The device 400 may access a wireless network based on a communication standard, such as WI-FI, 4G or 5G, or a combination thereof. In one exemplary embodiment, the communication component 416 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, apparatus 400 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation is used to perform the above method.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory 404 including instructions, executable by the processor 420 of the apparatus 400 to perform the method described above. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a speech generation device, enabling the speech generation device to execute a speech generation method, comprising:

Performing text analysis processing on the text to be processed to obtain text information of each text unit in the text to be processed;

Determine the target prosody information of each of the text units according to the text information and the symbol information of the punctuation marks contained in the text to be processed;

A target speech is generated according to the text information and the target prosody information.

FIG. 5 is a block diagram of the hardware structure of an apparatus 500 for speech generation according to an exemplary embodiment. For example, the apparatus 500 may be provided as a server. 5, apparatus 500 includes processing component 522, which further includes one or more processors, and a memory resource, represented by memory 532, for storing instructions executable by processing component 522, such as an application program. An application program stored in memory 532 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 522 is configured to execute instructions to perform a method of speech generation, including:

Performing text analysis processing on the text to be processed to obtain text information of each text unit in the text to be processed;

Determine the target prosody information of each of the text units according to the text information and the symbol information of the punctuation marks contained in the text to be processed;

A target speech is generated according to the text information and the target prosody information.

Device 500 may also include a power supply assembly 526 configured to perform power management of device 500 , a wired or wireless network interface 550 configured to connect device 500 to a network, and an input/output (I/O) interface 558 . Device 500 may operate based on an operating system stored in memory 532, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

Other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of this application that follow the general principles of this application and include common knowledge or conventional techniques in the technical field not disclosed in this application . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

It is to be understood that the present application is not limited to the precise structures described above and illustrated in the accompanying drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (10)

1. a speech generation method, it is characterised in that the method comprises: Performing text analysis processing on the text to be processed to obtain text information of each text unit in the text to be processed; Determine the target prosody information of each of the text units according to the text information and the symbol information of the punctuation marks contained in the text to be processed; A target speech is generated according to the text information and the target prosody information. 2 . The method according to claim 1 , wherein the text information comprises: text phonemes and text tones; the determining according to the text information and the symbol information of punctuation marks contained in the text to be processed The target prosody information of each of the text units, including: According to the text phoneme and the text pitch of each of the text units, determine the text prosody information of each of the text units; According to the symbol information, determine the symbol prosody information of each of the text units; The target prosody information is determined according to the text prosody information and the symbol prosody information. 3 . The method according to claim 2 , wherein the symbol information comprises: symbol position; the determining, according to the symbol information, the symbol prosody information of each of the text units, comprising: 3 . According to the symbol positions of the punctuation marks, determine the text units that each punctuation mark acts on; wherein, one punctuation mark acts on at least one text unit; Symbolic prosody information of the text unit to which the punctuation marks act is determined. 4. The method according to claim 2, wherein the symbol information comprises: symbol type and number of symbols; and determining the symbol prosody information of each of the text units according to the symbol information, comprising: Determine the tone type according to the symbol type of the punctuation mark; Determine the tone degree according to the number of the punctuation marks; The symbol prosody information is determined according to the tone type and/or the tone degree. 5. The method according to claim 1, wherein the generating a target speech according to the text information and the target prosody information comprises: Matching the text information and the target prosody information with preset text information and preset prosody information in a preset voice library; wherein the preset voice library is used to store audio clips, and the preset text information and the mapping relationship between the preset prosody information and the audio segment; Using the speech generation strategy corresponding to the matching result, the target speech is generated according to the text information and the target prosody information; wherein, different matching results correspond to different speech generation strategies. 6. The method according to claim 5, wherein, generating the target voice according to the text information and the target prosody information by using a voice generation strategy corresponding to the matching result, comprising: In the case that the matching result indicates that the text information and the target prosody information exist in the preset voice library, determine the audio corresponding to the text information and the target prosody information from the preset voice library fragment; Perform splicing processing on each of the audio segments to generate the target speech. 7. The method according to claim 5, wherein, generating the target voice according to the text information and the target prosody information by using a voice generation strategy corresponding to the matching result, comprising: In the case that the matching result indicates that the text information and the target prosody information do not exist in the preset speech library, feature conversion processing is performed on the text information and the target prosody information of each of the text units , get the acoustic features; Decoding each of the acoustic features is performed to generate the target speech. 8. A voice generation device, comprising: an analysis module, configured to perform text analysis processing on the text to be processed, and obtain text information of each text unit in the text to be processed; a determining module, configured to determine the target prosody information of each of the text units according to the text information and the symbol information of the punctuation marks contained in the text to be processed; The generating module is configured to generate the target speech according to the text information and the target prosody information. 9. A voice generation device, comprising: processor; memory configured to store processor executable instructions; Wherein, the processor is configured to: implement the steps in any one of the above-mentioned speech generation methods in claims 1 to 7 when executed. 10. A non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a processor of a speech generating apparatus, enables the apparatus to perform any one of the above-mentioned speech generating methods of claims 1 to 7 steps in .

Images