CN111816148A - A method and system for virtual vocal sight-singing based on generative adversarial network - Google Patents

Abstract

The invention provides a method and system for virtual vocal solfeggio based on a generative confrontation network. The method includes: step 1, inputting abc notation files and vocal notation audio produced by Vocaloid; step 2, converting abc The file is converted into a text file in a custom format, and the custom text file and vocal audio are used as the input of the Tacotron‑2 neural network model; Step 3. In the Tacotron‑2 neural network, the characters in the input text file pass through the 512-dimensional The character embedded in Character Embedding represents; in step 4, the synthesis of the virtual vocal waveform file is completed; in step 5, a complete piece of virtual vocal sight-singing music is obtained. The invention uses virtual human voice to sing musical scores, and the output voice rhythm is smooth and natural, so that the listener will feel natural when listening to the information, and will not feel mechanical and jerky in the voice output of the device.

Description

A method and system for virtual vocal sight-singing based on generative adversarial network

technical field

The invention belongs to the field of computers, and in particular, relates to a virtual vocal sight-singing method and system based on a generative confrontation network.

Background technique

When learning music, reading scores and singing is the basis of learning music, which is called "sight-singing". Solfeggio is particularly important, but the problem is that extracurricular activities such as solfeggio require a lot of practice to achieve the learning effect. Obviously, for the vast majority of ordinary families, teachers cannot accompany students to complete exercises around the clock. Because learning and practice are inseparably linked, and practice is crucial for singing and listening, it can be seen that there are some defects in the traditional listening training courses, and some improvements need to be made. "Real person singing notation is realistic and easy to learn." The virtual vocal solfeggio music notation is equivalent to the real person singing notation, which makes up for the shortcomings of musical instruments such as piano and electronic organ that are different from human voices, and greatly improves the speed of reading music and the ability to sing music. The current music-related application software has developed rapidly, adding the timbre of various musical instruments, and it can also be played automatically according to the score, but no human voice is added to the timbre.

The traditional method of simulating vocal singing adopts some or all methods such as splicing. For example, each roll call is simply spliced according to the time length and rhythm. Although this method is simple to operate, the result is blunt, which is quite different from real singing, and the effect is not good. Ideal. How to sing a score with a real human voice is a problem that needs to be solved.

SUMMARY OF THE INVENTION

The invention provides a virtual vocal sight-singing method and system based on a generative confrontation network. The virtual vocal can be used to sing musical scores, and the output voice rhythm is smooth and natural, so that the listener will feel natural when listening to the information, but not The voice output of the device feels mechanical and jerky.

In order to solve the above-mentioned problems, the present invention provides a virtual vocal sight-singing method based on a generative adversarial network, the method comprising:

Step 1. Input the abc notation file and the vocal score audio produced by Vocaloid, and the vocal score audio corresponds to the abc file;

Step 2: Convert the abc file into a text file in a custom format, and use the custom text file and vocal audio as the input of the Tacotron-2 neural network model;

Step 3. In the Tacotron-2 neural network, the characters in the input text file are represented by the 512-dimensional character embedding Character Embedding, and then through 3 convolutional layers, the output of the convolutional layer is passed to a bidirectional LSTM layer, At the same time, the use of Location Sensitive Attention makes the model always move forward in the process of input, and the model generated by the Tacotron-2 neural network, that is, the Mel spectrum, will be used as the input of the MelGAN model;

Step 4. Use the model trained by the Tacotron-2 neural network and the original vocal audio file as the input of the MelGAN generative adversarial neural network model. Through the generator and the discriminator, the feature map and the synthesized vocal will eventually be obtained. spectrum audio file, and the synthesis of the virtual vocal waveform file is completed;

Step 5: Glue and splicing the corresponding audio clips according to the scene, and finally a complete piece of virtual vocal sight-singing music will be obtained.

In a second aspect, an embodiment of the present application provides a virtual vocal sight-singing system based on a generative adversarial network, the system comprising:

The input module is used to input the abc notation file and the vocal score audio produced by Vocaloid, and the vocal score audio corresponds to the abc file;

The conversion module is used to convert the abc file into a text file in a custom format, and use the custom text file and vocal audio as the input of the Tacotron-2 neural network model;

Processing module, used in the Tacotron-2 neural network, the characters in the input text file are represented by 512-dimensional character embedding Character Embedding, and then through 3 convolutional layers, the output of the convolutional layer is passed to a bidirectional LSTM layer At the same time, the use of Location Sensitive Attention makes the model always move forward during the input process. The model generated by the Tacotron-2 neural network, that is, the Mel spectrum, will be used as the input of the MelGAN model, and the Tacotron-2 neural network will be trained. The good model and the original vocal audio file are used as the input of the MelGAN generative adversarial neural network model. Through the generator and the discriminator, the feature map and the synthesized vocal score audio file will finally be obtained, and the virtual vocal waveform is completed. synthesis of documents;

The splicing module is used for gluing and splicing the corresponding audio clips according to the scene, and finally a complete piece of virtual vocal solfeggio music will be obtained.

In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes all When the computer program is described, the methods described in the embodiments of the present application are implemented.

In a fourth aspect, the embodiments of the present application provide a computer-readable storage medium on which a computer program is stored, the computer program being configured to: implement the method described in the embodiments of the present application when the computer program is executed by a processor .

Description of drawings

Fig. 1 is the virtual vocal solfeggio flow chart of the present invention based on generative confrontation network;

Fig. 2 is the audio frequency waveform diagram of real person singing notation of the present invention;

FIG. 3 is a frame diagram of the virtual vocal solfeggio based on the generative adversarial network of the present invention.

Detailed ways

In order to make the invention purpose, technical process and technical innovation points of the present invention more clearly described, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

In order to achieve the above purpose, the present invention provides a virtual vocal sight-singing method based on a generative confrontation network. The main process is shown in Figure 3, and the method includes:

Step 1. Input the abc notation file and the vocal score audio produced by Vocaloid, and the vocal score audio corresponds to the abc file;

Step 2: Convert the abc file into a text file in a custom format, and use the custom text file and vocal audio as the input of the Tacotron-2 neural network model;

Step 3. In the Tacotron-2 neural network, the characters in the input text file are represented by the 512-dimensional character embedding Character Embedding, and then through 3 convolutional layers, the output of the convolutional layer is passed to a bidirectional LSTM layer, At the same time, the use of Location Sensitive Attention makes the model always move forward in the process of input, and the model generated by the Tacotron-2 neural network, that is, the Mel spectrum, will be used as the input of the MelGAN model;

Step 4. Use the model trained by the Tacotron-2 neural network and the original vocal audio file as the input of the MelGAN generative adversarial neural network model. Through the generator and the discriminator, the feature map and the synthesized vocal will eventually be obtained. spectrum audio file, and the synthesis of the virtual vocal waveform file is completed;

Step 5: Glue and splicing the corresponding audio clips according to the scene, and finally a complete piece of virtual vocal sight-singing music will be obtained.

Preferably, step 2 specifically includes: bringing the musical score information into each note, so that the musical score can be read out by the neural network like text, and using three key points: pitch, duration and pronunciation to express the syllables in the musical score; Convert abc notation to formal notation in another language with custom rules, and save it in a txt file, that is, score voice, the generated file is a score parsing file, and the first item of the score parsing file is the note and Tone information; 'b' means lower halftone; '#' means rising halftone; 'r' means empty tone, replace numbers and special symbols such as "#" with pure English, specifically, it will indicate the original pitch Symbols 3, 4, and 5 are replaced by n, o, and p, respectively, and 1/8 beat, 1/4 beat, 3/8 beat, 1/2 beat, 3/4 beat, and 1 beat are represented by q, 1 beat, and 1 beat respectively. r, s, t, u, v are replaced, and the c, c#, d, d#, e, f, f#, g, g#, a, a#, b of the notes are replaced by a, b, c, d, e, f, g, h, i, j, k, l instead.

Preferably, step 3 also includes:

Use position-sensitive attention to make the model always move forward during the input process, pass the prediction results through a pre-network Pre-Net containing 2 fully connected layers, and then, the output of the pre-network and the attention context vector Attention Context The Vector will be passed to 2 unidirectional LSTM layers. The output of the LSTM layer and the attention context vector will be linearly projected to generate the Mel spectrum. Finally, the predicted feature results will be passed to a post-convolutional layer with 5 layers. into the network to improve overall reconstruction.

Preferably, it also includes improving the adversarial network, including:

Placing an inductive bias in the generator, i.e. there is a long-range correlation between audio time steps, a residual block with dilation is added after each upsampling layer, so that the temporally distant output activations of each subsequent layer have significant Overlapping inputs, the receptive field of a stack of dilated convolutional layers grows exponentially with the number of layers, increasing the induced receptive field at each output time step, and greater overlap in the induced receptive field at distant time steps, leading to better long-range correlation;

Use the kernel size as a multiple of the stride, ensuring that the dilation grows with the kernel size, so that the receptive field of the stack is a fully balanced and symmetric tree, with the kernel size as the branching factor;

Use weight normalization in all layers of generator and discriminator;

Adopt a multi-scale architecture with 3 discriminators D1, D2, D3, the 3 discriminators have the same network structure and operate at different audio scales, D1 operates at the scale of the original audio, D2, D3 respectively at the downscale 2 times and 4 times the original audio, downsampling is performed using strided average pooling with a kernel size of 4; window-based discriminators are chosen because they have been shown to capture fundamental high-frequency structures, requiring less parameter, runs faster, and can be applied to variable-length audio sequences.

The generator is trained using the feature matching target.

As another aspect, the present application also provides a virtual vocal sight-singing system based on a generative adversarial network. As shown in FIG. 1, the system includes:

The input module is used to input the abc notation file and the vocal score audio produced by Vocaloid, and the vocal score audio corresponds to the abc file;

The conversion module is used to convert the abc file into a text file in a custom format, and use the custom text file and vocal audio as the input of the Tacotron-2 neural network model;

Processing module, used in the Tacotron-2 neural network, the characters in the input text file are represented by 512-dimensional character embedding Character Embedding, and then through 3 convolutional layers, the output of the convolutional layer is passed to a bidirectional LSTM layer At the same time, the use of Location Sensitive Attention makes the model always move forward during the input process. The model generated by the Tacotron-2 neural network, that is, the Mel spectrum, will be used as the input of the MelGAN model, and the Tacotron-2 neural network will be trained. The good model and the original vocal audio file are used as the input of the MelGAN generative adversarial neural network model. Through the generator and the discriminator, the feature map and the synthesized vocal score audio file will finally be obtained, and the virtual vocal waveform is completed. synthesis of documents;

The splicing module is used for gluing and splicing the corresponding audio clips according to the scene, and finally a complete piece of virtual vocal solfeggio music will be obtained.

Preferably, the conversion module is further used to: bring the score information into each note, so that the score can be read out by the neural network like text, and use three key points: pitch, duration and pronunciation to express the syllables in the score ;

Convert abc notation to formal notation in another language with custom rules, and save it in a txt file, that is, score voice, the generated file is a score parsing file, and the first item of the score parsing file is the note and Tone information; 'b' means lower halftone; '#' means rising halftone; 'r' means empty tone, replace numbers and special symbols such as "#" with pure English, specifically, it will indicate the original pitch Symbols 3, 4, and 5 are replaced by n, o, and p, respectively, and 1/8 beat, 1/4 beat, 3/8 beat, 1/2 beat, 3/4 beat, and 1 beat are represented by q, 1 beat, and 1 beat respectively. r, s, t, u, v are replaced, and the c, c#, d, d#, e, f, f#, g, g#, a, a#, b of the notes are replaced by a, b, c, d, e, f, g, h, i, j, k, l instead.

Preferably, the processing module is also used for:

Use position-sensitive attention to make the model always move forward during the input process, pass the prediction results through a pre-network Pre-Net containing 2 fully connected layers, and then, the output of the pre-network and the attention context vector Attention Context The Vector will be passed to 2 unidirectional LSTM layers. The output of the LSTM layer and the attention context vector will be linearly projected to generate the Mel spectrum. Finally, the predicted feature results will be passed to a post-convolutional layer with 5 layers. into the network to improve overall reconstruction.

Preferably, the processing module is also used for:

Placing an inductive bias in the generator, i.e. there is a long-range correlation between audio time steps, a residual block with dilation is added after each upsampling layer, so that the temporally distant output activations of each subsequent layer have significant Overlapping inputs, the receptive field of a stack of dilated convolutional layers grows exponentially with the number of layers, increasing the induced receptive field at each output time step, and greater overlap in the induced receptive field at distant time steps, leading to better long-range correlation;

Use the kernel size as a multiple of the stride, ensuring that the dilation grows with the kernel size, so that the receptive field of the stack is a fully balanced and symmetric tree, with the kernel size as the branching factor;

Use weight normalization in all layers of generator and discriminator;

Adopt a multi-scale architecture with 3 discriminators D1, D2, D3, the 3 discriminators have the same network structure and operate at different audio scales, D1 operates at the scale of the original audio, D2, D3 respectively at the downscale 2 Runs at times and 4 times the original audio, downsampling is performed using a strided average pool of kernel size 4;

The generator is trained using the feature matching target.

As another aspect, the present application also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the When the computer program is described, the method as described in the embodiments of the present application is implemented.

As another aspect, the present application also provides a computer-readable storage medium, and the computer-readable storage medium may be the computer-readable storage medium included in the aforementioned apparatus in the foregoing embodiment; computer-readable storage medium in the device. The computer-readable storage medium stores one or more programs, and the aforementioned programs are used by one or more processors to execute the methods described in the embodiments of the present application.

It should be understood that various parts of this application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one of the following techniques known in the art, or a combination thereof: discrete with logic gates for implementing logic functions on data signals Logic circuits, ASICs with suitable combinational logic gate circuits, Programmable Gate Arrays (hereinafter referred to as PGA), Field Programmable Gate Arrays (hereinafter referred to as FPGAs), etc.

Those skilled in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program can be stored in a computer-readable storage medium. When executed, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations to the present application. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (10)

1. A virtual human voice-over-video-singing method based on generation of an antagonistic network, the method comprising:

inputting an abc notation file and a vocal music score audio frequency manufactured by Vocaloloid, wherein the vocal music score audio frequency corresponds to the abc file;

step two, converting the abc file into a text file with a custom format, and taking the custom text file and the voice audio as the input of a Tacotron-2 neural network model;

step three, in a Tacotron-2 neural network, characters in an input text file are represented by Embedding 512-dimensional characters into Character Embedding, then the output of the convolutional layers is transmitted to a bidirectional LSTM layer through 3 convolutional layers, meanwhile, a position Sensitive Attention Sensitive Attention is used to enable a model to move forwards all the time in the input process, and a Mel frequency spectrum, which is a model generated by the Tacotron-2 neural network, is used as the input of a MelGAN model;

step four, taking the model trained by the Tacotron-2 neural network and the original human voice audio file as the input of the MelGAN generation countermeasure neural network model, and finally obtaining a Feature Map and a synthesized human voice record audio file through a generator and a discriminator to complete the synthesis of the virtual human voice waveform file;

and step five, splicing the corresponding audio clips according to the scene to finally obtain a complete section of virtual human audio-video-singing music.

2. The method according to claim 1, wherein step two specifically comprises: score information is brought to each note so that the score can be read out as text by a neural network, using three key points: pitch, duration and pronunciation to express syllables in the score;

converting abc notation into another language formalized notation by using a self-defined rule, and storing the notation in a txt file, namely, a music score voice, wherein the generated file is a music score analysis file, and the first item of the music score analysis file is musical notes and tone information; 'b' represents a falling half tone; '#' indicates half-tone; 'r' represents null sound, the number and special symbol such as "#" are replaced by pure English, concretely, original symbols 3, 4 and 5 representing pitch are respectively replaced by n, o and p, 1/8 beats, 1/4 beats, 3/8 beats, 1/2 beats, 3/4 beats and 1 beat representing duration are respectively replaced by q, r, s, t, u and v, c #, d #, e, f #, g #, a #, b representing note are respectively replaced by a, b, c, d, e, f, g, h, i, j, k and l.

3. The method of claim 1, wherein step three further comprises: the method comprises the steps of enabling a model to move forwards all the time in the input process by using position sensitive Attention, enabling a prediction result to pass through a preposed network Pre-Net comprising 2 fully connected layers, then transmitting the output of the preposed network and an Attention Context Vector to 2 one-way LSTM layers, generating a Mel frequency spectrum after the output of the LSTM layers and the Attention Context Vector are subjected to linear projection, and finally transmitting the predicted characteristic result to a postposition network comprising 5 convolutional layers to improve total reconstruction.

4. The method of claim 1, further comprising retrofitting the antagonistic network, in particular comprising:

placing an induced bias in the generator, i.e., there is a long range correlation between audio time steps, adding a residual block with dilation after each upsampled layer, so that the time-out activation of each subsequent layer has significant overlapping inputs, the receive field of a stack of dilated convolutional layers grows exponentially with the number of layers, the induced receive field of each output time step can be increased after the generator is incorporated, achieving greater overlap in the induced receive field of the long range time steps, resulting in better long range correlation;

using kernel size as a multiple of the span to ensure that inflation grows as kernel size increases, so that the acceptance field of the stack is a fully balanced and symmetric tree, with kernel size as a branching factor;

using weight normalization in all layers of the generator and discriminator;

adopting a multi-scale architecture with 3 discriminators D1, D2 and D3, wherein the 3 discriminators have the same network structure and operate on different audio scales, the D1 operates on the scale of original audio, the D2 and the D3 operate on the original audio which is reduced by 2 times and 4 times respectively, and downsampling is performed by using a stride average pool with the kernel size of 4;

the generator is trained using feature matching targets.

5. A virtual human voice-over-video-singing system based on a generation confrontation network, the system comprising:

the input module is used for inputting an abc notation file and vocal music score audio made by Vocaloloid, wherein the vocal music score audio corresponds to the abc file;

the conversion module is used for converting the abc file into a text file in a custom format, and taking the custom text file and the voice audio as the input of a Tacotron-2 neural network model;

the processing module is used for Embedding characters in an input text file into Character Embed representation through 512-dimensional characters in a Tacotron-2 neural network, then transmitting the output of the convolutional layers to a bidirectional LSTM layer through 3 convolutional layers, meanwhile, enabling the model to move forwards all the time in the input process by using position Sensitive Attention positioning, enabling a model generated by the Tacotron-2 neural network, namely a Mel frequency spectrum, to be used as the input of a MelGAN model, enabling a model trained by the Tacotron-2 neural network and an original human voice audio file to be used as the input of a MelGAN to generate an anti-neural network model, and finally obtaining a Feature Map and a synthesized human voice spectrum audio file through a generator and a discriminator to complete the synthesis of a virtual human voice shape file; and the splicing module is used for splicing the corresponding audio clips together in a bonding way according to the scene, and finally obtaining a complete section of virtual human audio-video-singing music.

6. The system of claim 5, wherein the conversion module is further configured to: score information is brought to each note so that the score can be read out as text by a neural network, using three key points: pitch, duration and pronunciation to express syllables in the score;

converting abc notation into another language formalized notation by using a self-defined rule, and storing the notation in a txt file, namely, a music score voice, wherein the generated file is a music score analysis file, and the first item of the music score analysis file is musical notes and tone information; 'b' represents a falling half tone; '#' indicates half-tone; 'r' represents null sound, the number and special symbol such as "#" are replaced by pure English, concretely, original symbols 3, 4 and 5 representing pitch are respectively replaced by n, o and p, 1/8 beats, 1/4 beats, 3/8 beats, 1/2 beats, 3/4 beats and 1 beat representing duration are respectively replaced by q, r, s, t, u and v, c #, d #, e, f #, g #, a #, b representing note are respectively replaced by a, b, c, d, e, f, g, h, i, j, k and l.

7. The system of claim 5, wherein the processing module is further configured to:

the method comprises the steps of enabling a model to move forwards all the time in the input process by using position sensitive Attention, enabling a prediction result to pass through a preposed network Pre-Net comprising 2 fully connected layers, then transmitting the output of the preposed network and an Attention Context Vector to 2 one-way LSTM layers, generating a Mel frequency spectrum after the output of the LSTM layers and the Attention Context Vector are subjected to linear projection, and finally transmitting the predicted characteristic result to a postposition network comprising 5 convolutional layers to improve total reconstruction.

8. The system of claim 5, wherein the processing module is further configured to:

placing an induced bias in the generator, i.e., there is a long range correlation between audio time steps, adding a residual block with dilation after each upsampled layer, so that the time-out activation of each subsequent layer has significant overlapping inputs, the receive field of a stack of dilated convolutional layers grows exponentially with the number of layers, the induced receive field of each output time step can be increased after the generator is incorporated, achieving greater overlap in the induced receive field of the long range time steps, resulting in better long range correlation;

using kernel size as a multiple of the span to ensure that inflation grows as kernel size increases, so that the acceptance field of the stack is a fully balanced and symmetric tree, with kernel size as a branching factor;

using weight normalization in all layers of the generator and discriminator;

adopting a multi-scale architecture with 3 discriminators D1, D2 and D3, wherein the 3 discriminators have the same network structure and operate on different audio scales, the D1 operates on the scale of original audio, the D2 and the D3 operate on the original audio which is reduced by 2 times and 4 times respectively, and downsampling is performed by using a stride average pool with the kernel size of 4;

the generator is trained using feature matching targets.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 4 when executing the computer program.

10. A computer-readable storage medium having stored thereon a computer program for: the computer program, when executed by a processor, implements the method of any one of claims 1-4.

Images