CN111816157B - Music score intelligent video-singing method and system based on voice synthesis - Google Patents

Abstract

The invention provides a music score intelligent video-singing method and a music score intelligent video-singing system based on voice synthesis, wherein the method comprises the following steps: step one, data preparation, namely inputting and analyzing an abc music score to obtain pitch and duration information of each note in the specific abc music score; step two, training parameters, namely generating notes with the length within 5 when training data are made, namely dividing all notes into a group of 5 notes when a complete abc music score is processed; step three, synthesizing audio splicing, specifically comprising three substeps of music score segmentation identification, segment splicing, waveform alignment and blank segment filling; and step four, performing visual display on the synthesized audio. The invention solves the technical problems of large calculation amount in the training process, obvious splicing trace in direct splicing, splicing noise and the like, and the difference is difficult to distinguish by comparing the effects of the generated audio and the original data.

Description

A method and system for intelligent solfeggio based on speech synthesis

technical field

The invention belongs to the field of computers, and in particular relates to an intelligent solfeggio method and system based on speech synthesis.

Background technique

In recent years, with the continuous development of the Internet, the continuous update and iteration of mobile phone applications and computer applications, online education is also developing continuously, while art subjects have certain particularities, which emphasize communication and mutual feedback between teachers and students , students need to receive feedback in time to correctly recognize their own mistakes when learning this type of course. For example, in music learning, each student will have different problems simply by reading and singing scores. Teachers cannot take care of every student at the same time. When our students are learning music, it is best to combine seeing with eyes, listening with ears, and singing with the mouth, so as to cooperate with multiple senses, so that we can better learn subjects that stimulate the senses such as music.

What the present invention researches is the sight-singing teaching section of the music online teaching platform. It is hoped that the sight-singing of music scores can be changed from the traditional way of asking a professional teacher to sing each score well into a computer to simulate the singing method of a real person to sing the score, so that not only students can Learning various music scores flexibly, and it also has a great effect on the teacher or the platform to expand the database, so the problem studied by the present invention is to complete the task of sight-singing music scores by computer.

Music score sight-singing is a problem similar to speech synthesis, and the current speech synthesis technology has already reached the level of being put into the market. Music score can be regarded as a special language. What the computer needs to do is to learn the pronunciation of each note. The music score will give information such as tempo in the information at the beginning, so this information needs to be brought into all the notes in the string when processing. The string contains information about the duration and pitch of each note, but the specific application Sometimes, when the training data is too large, the training cannot be performed, the splicing traces will be more obvious when splicing directly, and noise will be generated.

Contents of the invention

The present invention provides an intelligent solfeggio method and system based on speech synthesis, which can translate the music score into a new language so that it fits the corresponding mapping relationship of <text, pronunciation> in a certain language, and solves the problem of training The calculation process is heavy, the splicing traces will be more obvious when splicing directly, and there are technical problems such as splicing noise. It is difficult to distinguish the difference between the generated audio and the original data.

In order to solve the above problems, the present invention provides a method for intelligent sight-singing of music scores based on speech synthesis, said method comprising:

Step 1, data preparation, input and analyze the abc score, and obtain the pitch and duration information of each note in a specific abc score;

Step 2, training parameters, when making training data, notes with a length of less than 5 are generated, that is, when processing a complete abc score, all notes are divided into groups of 5 notes;

Step 3. Synthetic audio splicing, specifically including three sub-steps of music score segmentation recognition, fragment splicing, waveform alignment and blank segment filling;

Step 4: Visually display the synthesized audio.

In the second aspect, the embodiment of the present application provides a music score intelligent sight-singing system based on speech synthesis, the system comprising:

The data preparation module is used to input and analyze the abc score, and obtain the pitch and duration information of each note in a specific abc score;

The training parameter module generates notes with a length of less than 5 when making training data, that is, when processing a complete abc score, divide all its notes into groups of 5 notes;

Synthetic audio splicing module, including score segmentation recognition sub-module, fragment splicing sub-module, waveform alignment and blank segment filling sub-module;

The visualization module is used to visualize the synthesized audio.

In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor executes the The method described in the embodiment of the present application is implemented when the computer program is described.

In a fourth aspect, the embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program is used for:

When the computer program is executed by the processor, the method described in the embodiment of the present application is realized.

Description of drawings

Fig. 1 is the intelligent solfeggio flow chart of musical notation based on speech synthesis of the present invention;

Fig. 2 is an effect diagram of the result of audio synthesis based on music score in the present invention.

Detailed ways

In order to clarify the invention purpose, technical process and technical innovation points of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

To achieve the above object, the present invention provides a method for intelligent sight-singing of music score based on speech synthesis. The main process is shown in Figure 1, and the method includes:

Step 1, data preparation, input and analyze the abc score, and obtain the pitch and duration information of each note in a specific abc score;

The data to be processed in this application is the abc file. The abc file contains two parts of information. The first five lines are the rhythm and key information of the score, and the following content is the note. The difference from the language is that the score is specified at the beginning of the score The singing method of the whole score, and the language is that each word has independent pronunciation rules, so when processing the abc score, it is necessary to consider a similar processing method to bring the score information of the first part into each note.

Step 2, training parameters, when making training data, notes with a length of less than 5 are generated, that is, when processing a complete abc score, all notes are divided into groups of 5 notes;

Step 3. Synthetic audio splicing, specifically including three sub-steps of music score segmentation recognition, fragment splicing, waveform alignment and blank segment filling;

Step 4: Visually display the synthesized audio.

Preferably, the data preparation step also includes: each note setting is composed of three parts: duration, note, and pitch, the shortest unit of duration is 1/8 beat, and the duration of the note is 1/8, 1/4, 3/8 , 1/2, 3/4, 1 beat, the vocal range is f3～f#5, the data is difficult to collect vocal data, use vocaloid software to synthesize audio to replace the human voice, and call the raised half tone with a lowered tone to represent , in order to label accurately, all the notes represented by the "black keys" are uniformly represented by sharp half-tones. Through this combination, the first information of the score can be brought into each note, which solves the translation problem of abc score.

Preferably, the training process also includes: the application uses a script to generate a total of 11,000 data, the data set is divided into 10,000 data as training data, and the remaining 1,000 data are used for testing data to test the effect of the model, and the testing method is artificial listening. Identify the duration and pronunciation of sounds. Every 1000 steps of training will generate a checkpoint and the current alignment of the encoder and decoder. If you disconnect the training midway, you can resume training from the checkpoint; replace # and Arabic numerals with English letters, and the length of the section is still fixed. Short, using data of variable length, the "r" sound is added directly to the synthesized audio during subsequent synthesis.

A mel-scale spectrogram with a bandwidth of 80 is used as the target output of the decoder, the sample rate is 20000hz, the frame length is 50ms, and the frame offset is 12.5ms. The optimizer uses conventional adam, the batch size is 32, and the learning rate is delayed It can effectively reduce the situation of model overfitting.

Preferably, the synthetic audio splicing specifically includes:

Music score segmentation recognition, for a section of abc file, split it into multiple measures with 5 notes as a group, and divide it when encountering the "r" symbol, and finally the remaining notes together as a measure, and then for each The section uses the trained model to process the synthesized audio and generate the wav file corresponding to each section;

Fragment splicing, splicing the audio synthesized by segments;

Waveform alignment and blank segment filling. For continuous singing scores, rough splicing will have obvious splicing traces. First, use software to perform simple splicing observations and find that two basic problems need to be dealt with. There will be a short blank time at the beginning and end of the audio, so the blank space needs to be trimmed. Since the blank time at the beginning and end of the synthesized audio is almost the same, it can be processed uniformly. The second problem is the rough splicing of the "pop" sound. , the main reason is that if there is a voiced sound after a certain sound, that is, the four sounds of "Duo", "唻", "Mi" and "La", this will happen. After zooming in on the waveform diagram, it is found that the noise is Because of the sudden change of the waveform here. In order to solve these problems, this application adopts the following scheme: 'r' in the abc score represents an empty segment without pronunciation, when synthesizing the audio, directly add the audio with a blank duration as a buffer to the part that needs to be added, and use sox to perform the subsection Synthesis, the r sound is composed of two parts, one is the pronunciation length, and the other is r. According to the requirements of abc score, the corresponding "r" audio is selected for processing. A very short splicing part can eliminate noise without affecting The r sound experienced by the duration is processed and output. This approach has had unexpected results in murmur cancellation. After testing, as shown in Figure 2, after zooming in on the music score synthesized audio results, you can see the blank r segment added at the splicing, and there is no obvious feeling in the sense of hearing. The actual effect is no different from direct artificially synthesized audio.

As another aspect, the present application also provides a music score intelligent solfeggio system based on speech synthesis, the system comprising:

The data preparation module is used to input and analyze the abc score, and obtain the pitch and duration information of each note in a specific abc score;

The training parameter module generates notes with a length of less than 5 when making training data, that is, when processing a complete abc score, divide all its notes into groups of 5 notes;

Synthetic audio splicing module, including score segmentation recognition sub-module, fragment splicing sub-module, waveform alignment and blank segment filling sub-module;

The visualization module is used to visualize the synthesized audio.

Preferably, the data preparation module also includes: each note setting is composed of three parts: duration, note, and pitch, the shortest unit of duration is 1/8 beat, and the duration of the note is 1/8, 1/4, 3/8 , 1/2, 3/4, 1 beat, the vocal range is f3～f#5, the data is difficult to collect vocal data, use vocaloid software to synthesize audio to replace the human voice, and call the raised half tone with a lowered tone to represent , in order to label accurately, all the notes represented by the "black keys" are uniformly represented by sharp half-tones.

Preferably, the training parameter module is also used for:

Every 1000 steps of training will generate a checkpoint and the current alignment status of the encoder and decoder, and the training can be resumed from the checkpoint if the training is disconnected halfway;

Both # and Arabic numerals are replaced with English letters, and the length of the bar is still fixed. If the pronounced word is very short, use data with an unfixed length, and the "r" sound will be directly added to the synthesized audio during subsequent synthesis.

Preferably, the synthetic audio splicing module specifically includes:

Music score segmentation recognition sub-module, used to split a piece of abc file, split into multiple measures with 5 notes as a group, and also divide when encountering the "r" symbol, and finally the rest of the notes together as a measure, Then use the trained model to process and synthesize audio for each subsection, and generate a wav file corresponding to each subsection;

Fragment splicing sub-module, used for splicing audio synthesized by segments;

The waveform alignment and blank section filling sub-module is used for the 'r' in the abc score to indicate the blank section that is not pronounced. When synthesizing the audio, directly add the audio of the blank duration as a buffer to the part that needs to be added, and use sox to measure the section Synthesis, the r sound is composed of two parts, one is the pronunciation length, and the other is r. According to the requirements of abc score, the corresponding "r" audio is selected for processing. A very short splicing part can eliminate noise without affecting The r sound experienced by the duration is processed and output.

As another aspect, the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein the processor executes the The computer program implements the method as described in the embodiment of the present application.

As another aspect, the present application also provides a computer-readable storage medium, which may be the computer-readable storage medium contained in the aforementioned devices in the above-mentioned embodiments; computer-readable storage media stored in the device. The computer-readable storage medium stores one or more programs, and the aforementioned programs are used by one or more processors to execute the methods described in the embodiments of this application.

It should be understood that each part of the present application may be realized by hardware, software, firmware or a combination thereof. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one or a combination of the following techniques known in the art: a discrete Logic circuits, ASICs with appropriate combinational logic gate circuits, Programmable Gate Array (hereinafter referred to as: PGA), Field Programmable Gate Array (Field Programmable Gate Array; hereinafter referred to as: FPGA), etc.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. During execution, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present application, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (8)

1. A music score intelligent video-singing method based on voice synthesis comprises the following steps:

step one, data preparation, namely inputting and analyzing an abc music score to obtain pitch and duration information of each note in the specific abc music score;

step two, training parameters, namely generating notes with the length within 5 when training data are made, namely dividing all notes of a complete abc music score into a group of 5 notes when the complete abc music score is processed;

step three, synthesizing audio splicing, specifically comprising three substeps of music score segmentation identification, segment splicing, waveform alignment and blank segment filling;

fourthly, visually displaying the synthesized audio;

the synthesized audio concatenation specifically includes:

segmenting and identifying a music score, namely segmenting an abc file, segmenting the abc file into a plurality of measures by taking 5 notes as a group, segmenting the abc file when meeting an r symbol, taking the rest notes as a measure, processing each measure by using a trained model to synthesize audio, and generating a wav file corresponding to each measure;

segment splicing, namely splicing the audio synthesized by the segments;

the waveform is aligned and the blank section is filled, r' in the abc music score represents an unvoiced blank sound section, when the audio is synthesized, the audio with blank duration is directly added to the part needing to be added to serve as a buffer zone, sox is used for synthesizing the measure, r sound is composed of two parts, namely pronunciation length and r, the corresponding r audio is selected according to the requirements of the abc music score for processing, and each section of spliced part is processed and output by using a very short r sound which can eliminate the noise and does not influence the duration feeling.

2. The method of claim 1, wherein the data preparation step further comprises: each note setting comprises three parts of duration, note and pitch, wherein 1/8 beat is selected in the shortest unit of duration, the duration of the note is 1/8, 1/4, 3/8, 1/2, 3/4 and 1 beat, the range of the note is f 3-f #5, because the human voice data is difficult to collect, audio is synthesized by using vocaloloid software to replace the human voice, the falling half tone is expressed by the rising half tone of the falling half tone, and in order to mark accurately, the tones expressed by all black keys are uniformly expressed by the rising half tone.

3. The method of claim 1, wherein the training process further comprises:

the alignment condition of the check point and the current encoder and decoder can be generated every 1000 training steps, the training is interrupted in the midway, and the training is recovered from the check point again;

the number # and Arabic numerals are both replaced by English letters, the length of the bar is still fixed, if the pronunciation word is short, the data with unfixed length is used, and the 'r' sound is directly added into the synthesized audio in the subsequent synthesis.

4. A music score intelligent video-song system based on speech synthesis, the system comprising:

the data preparation module is used for inputting and analyzing the abc music score to obtain the pitch and duration information of each note in the specific abc music score;

the training parameter module generates notes with the length within 5 when training data are produced, namely, all notes of a complete abc music score are divided into a group of 5 notes when the complete abc music score is processed;

the synthesis audio splicing module comprises a music score segmentation identification sub-module, a segment splicing sub-module and a waveform alignment and blank segment filling sub-module;

the visualization module is used for visually displaying the synthesized audio;

the synthesized audio splicing module specifically comprises:

the music score segmentation recognition sub-module is used for splitting an abc file, splitting the abc file into a group of 5 notes and a plurality of measures, splitting the abc file when the r symbol is met, using the rest notes as a measure, processing each measure by using a trained model to synthesize audio, and generating a wav file corresponding to each measure;

the segment splicing submodule is used for splicing the audio synthesized by segments;

the waveform alignment and blank section filling submodule is used for 'r' in an abc music score to represent an unvoiced blank section, when the audio is synthesized, the audio with blank duration is directly added to a part needing to be added to serve as a buffer zone, sox is used for synthesizing a measure, r sound is composed of two parts, namely pronunciation length and r, the corresponding 'r' audio is selected according to the requirements of the abc music score to be processed, and each spliced part is processed and output by using a very short r sound which can eliminate the noise and does not influence the duration feeling.

5. The system of claim 4, wherein the data preparation module further comprises: each note setting comprises three parts of duration, note and pitch, wherein 1/8 beat is selected in the shortest unit of duration, the duration of the note is 1/8, 1/4, 3/8, 1/2, 3/4 and 1 beat, the range of the note is f 3-f #5, because the human voice data is difficult to collect, audio is synthesized by using vocaloloid software to replace the human voice, the falling half tone is expressed by the rising half tone of the falling half tone, and in order to mark accurately, all the tones expressed by the black keys are uniformly expressed by the rising half tone.

6. The system of claim 4, wherein the training parameter module is further configured to:

the alignment condition of the check point and the current encoder and decoder can be generated every 1000 training steps, the training is interrupted in the midway, and the training is recovered from the check point again;

the number # and Arabic numerals are replaced by English letters, the lengths of the bars are still fixed, if the pronunciation words are short, data with unfixed lengths are used, and the 'r' sound is directly added into the synthesized audio in the subsequent synthesis.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 3 when executing the computer program.

8. A computer-readable storage medium having stored thereon a computer program for: the computer program, when executed by a processor, implements the method of any one of claims 1-3.

Images