CN114203154A - Training method and device of voice style migration model and voice style migration method and device - Google Patents

Abstract

The disclosure provides a method and a device for training a voice style migration model and a voice style migration model, and relates to the technical field of artificial intelligence, in particular to the field of voice synthesis. The method comprises the following steps: respectively inputting the voice spectrum characteristics of the sample audio into a content coding network and a speaker coding network of the voice style migration model to obtain a content characteristic vector and a speaker characteristic vector of the sample audio, and extracting the fundamental tone frequency of the sample audio; determining mutual information of the content characteristic vector, the speaker characteristic vector and the fundamental tone frequency; inputting the content feature vector, the speaker feature vector and the fundamental tone frequency into a voice spectrum decoding network of the voice style migration model to obtain predicted voice spectrum features; and updating the parameters of the voice style migration model according to the predicted voice spectrum characteristics and the mutual information. The method optimizes the effect of voice style migration.

Description

Training of voice style transfer model, voice style transfer method and device

technical field

The present disclosure relates to speech synthesis technology in the technical field of artificial intelligence, and in particular, to a method and device for training a speech style transfer model, and for speech style transfer.

Background technique

Voice style transfer is often used in voice-changing systems, voice chats, online games and other scenarios, outputting one voice style voice in other voice styles to hide user identity or increase entertainment.

Voice style transfer is often achieved by using a voice style transfer model. The voice style transfer model is used to extract the content features of the source audio and the speaker features of the target audio. The output content is the content of the source audio but the voice style is that of the target audio speaker. The spectral features of the style are finally converted into audio output using a vocoder.

When training the speech style transfer model, the content features and speaker features of the sample audio are extracted, and the predicted spectral features of the sample audio are output, and the difference between the predicted spectral features and the real spectral features of the sample audio is used. to update. Due to the dependency between the content features of the sample audio and the speaker features, the speech style transfer effect in the application process is poor.

SUMMARY OF THE INVENTION

The present disclosure provides a voice style transfer model training, voice style transfer method and device with optimized voice style transfer effect.

According to one aspect of the present disclosure, there is provided a

Training methods for speech style transfer models, including:

Inputting the spectral features of the sample audio into the content coding network and the speaker coding network of the speech style transfer model respectively, obtaining the content feature vector and the speaker feature vector of the sample audio, and extracting the pitch frequency of the sample audio;

determining the mutual information of the content feature vector, the speaker feature vector and the pitch frequency;

Inputting the content feature vector, the speaker feature vector and the pitch frequency into the sound spectrum decoding network of the voice style transfer model to obtain predicted sound spectrum features;

According to the predicted spectral features and the mutual information, parameters of the speech style transfer model are updated.

According to another aspect of the present disclosure, there is provided a voice style transfer method, comprising:

obtaining the first sound spectrum feature of the source audio and the second sound spectrum feature of the target audio;

Inputting the first sound spectrum feature into the content coding network of the speech style transfer model, obtaining the content feature vector of the source audio, and extracting the pitch frequency of the source audio;

Inputting the second sound spectrum feature into the speaker coding network of the speech style transfer model to obtain the speaker feature vector of the target audio;

The content feature vector of the source audio, the speaker feature vector of the target audio and the fundamental frequency of the source audio are input into the sound spectrum decoding network of the voice style transfer model to obtain the to-be-output sound spectrum feature, and according to the Describe the features of the sound spectrum to be output, and generate the audio to be output;

Wherein, the speech style transfer model is obtained by training according to the aforementioned method.

According to another aspect of the present disclosure, a training device for a speech style transfer model is provided, including:

The feature extraction module is used to input the spectral features of the sample audio into the content coding network and the speaker coding network of the speech style transfer model respectively, obtain the content feature vector and the speaker feature vector of the sample audio, and extract the sample audio the pitch frequency;

a mutual information calculation module, used for determining the mutual information of the content feature vector, the speaker feature vector and the pitch frequency;

A sound spectrum decoding module for inputting the content feature vector, the speaker feature vector and the pitch frequency into the sound spectrum decoding network of the voice style transfer model to obtain predicted sound spectrum features;

An update module, configured to update the parameters of the speech style transfer model according to the predicted sound spectrum feature and the mutual information.

According to another aspect of the present disclosure, a voice style transfer apparatus is provided, comprising:

an acquisition module for acquiring the first sound spectrum feature of the source audio and the second sound spectrum feature of the target audio;

A content coding module, for inputting the first sound spectrum feature into the content coding network of the speech style transfer model, obtaining the content feature vector of the source audio, and extracting the pitch frequency of the source audio;

a speaker encoding module, for inputting the second sound spectrum feature into the speaker encoding network of the speech style transfer model, to obtain the speaker feature vector of the target audio;

The sound spectrum decoding module is used to input the content feature vector of the source audio, the speaker feature vector of the target audio and the fundamental frequency of the source audio into the sound spectrum decoding network of the voice style transfer model, and obtain the output to be output sound spectrum features, and generate to-be-output audio according to the to-be-output sound spectrum features;

Wherein, the speech style transfer model is obtained by training according to the aforementioned method.

According to yet another aspect of the present disclosure, an electronic device is provided, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of the first aspect above.

According to yet another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of the first aspect above.

According to yet another aspect of the present disclosure, there is provided a computer program product, the program product comprising: a computer program, the computer program being stored in a readable storage medium, from which at least one processor of an electronic device can read The storage medium reads the computer program, and the at least one processor executes the computer program to cause the electronic device to perform the method of the first aspect.

According to the technical solutions of the present disclosure, the effect of voice style transfer is optimized.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present disclosure. in:

1 is a schematic flowchart of a training method for a speech style transfer model provided according to an embodiment of the present disclosure;

2 is a schematic diagram of a speech style transfer model provided according to an embodiment of the present disclosure;

3 is a schematic diagram of a content encoding network provided according to an embodiment of the present disclosure;

4 is a schematic flowchart of a voice style transfer method provided according to an embodiment of the present disclosure;

5 is a schematic structural diagram of a training apparatus for a speech style transfer model provided according to an embodiment of the present disclosure;

6 is a schematic structural diagram of a voice style transfer apparatus provided according to an embodiment of the present disclosure;

7 is a schematic block diagram of an electronic device used to implement the method of an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

The speech style transfer model usually includes at least a content encoding network, a speaker encoding network, and a sound spectrum decoding network. Among them, the content coding network is used to extract the content features of the source audio, the speaker coding network is used to extract the speaker features of the target audio, and the sound spectrum decoding network is used to output the audio according to the content features of the source audio and the speaker features of the target audio. Spectral features, and finally, convert the spectral features into audio, which is audio with the content of the source audio but the voice style of the speaker of the target audio.

When training the voice style transfer model, for the sample audio, the content features of the sample audio are extracted by the content coding network, and at the same time, the speaker features of the sample audio are extracted by the speaker coding network. The content feature and the speaker feature of the sample audio, output the predicted spectral feature of the sample audio, the content of the audio corresponding to the predicted spectral feature is the content of the sample audio and the voice style is the style of the speaker of the sample audio, using the predicted sound spectrum The difference between the features and the real spectral features of the sample audio updates the model parameters. In this way, after the model training is completed, when using the voice style transfer model for voice style transfer, it is only necessary to change the input of the speaker coding network to the target audio, so that the speaker coding network outputs the speaker characteristics of the target audio, and then the voice can be realized. style transfer.

However, when training the speech style transfer model, the content features and speaker features extracted from the sample audio may not be decoupled, that is, there may be a dependency between the content features and the speaker features, which may lead to the training process. In the finished application, after changing the input of the speaker encoding network to the target audio, the transfer of speech style is poor.

To this end, it is proposed in the embodiments of the present disclosure that in the speech style transfer model, in addition to using the content coding module to extract content features and the speaker coding module to extract speaker features, the pitch frequency of the audio is also extracted, and the features of these three aspects are used. To determine the predicted spectral features, when training the model, in addition to updating the model parameters according to the predicted spectral features, it also uses the mutual information (Mutual Information, MI) between the content features, speaker features and pitch frequency, two features Mutual information between two features can be seen as the amount of information contained in one feature about another random feature, or the uncertainty reduced by one feature due to the known other feature, which is used to measure the difference between two features. reciprocity. In the training process, by reducing the mutual information, the content features, speaker features and pitch frequencies are decoupled as much as possible, that is, the dependencies between them are reduced, so that the information contained in their respective features does not overlap, so as to ensure that in the follow-up In the process of application, the input of the speaker coding network is changed to the target audio, and after obtaining the speaker characteristics of the target audio, the spectral characteristics of the voice style of the target audio can be obtained by using the model, which is not affected or interfered by other characteristics. Voice style transfer is better.

The present disclosure provides a voice style transfer model training, voice style transfer method and device, which are applied to the field of speech synthesis in the field of artificial intelligence technology, and can be specifically applied to scenarios such as voice changing systems, voice chat, and online games to optimize voice style. effect of migration.

Hereinafter, the training method of the speech style transfer model and the speech style transfer method provided by the present disclosure will be described in detail through specific embodiments. It can be understood that the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

FIG. 1 is a schematic flowchart of a training method of a speech style transfer model according to an embodiment of the present disclosure. The execution body of the method is a training device of the speech style transfer model, and the device can be implemented by means of software and/or hardware. As shown in Figure 1, the method includes:

S101. Input the spectral features of the sample audio into the content coding network and the speaker coding network of the speech style transfer model respectively, obtain the content feature vector and the speaker feature vector of the sample audio, and extract the pitch frequency of the sample audio.

For example, the sample audio may use open source data sets Aishell, LJSpeech, VCTK, etc., and the spectral features of the sample audio may be MFCC, PLP, or Fbank features, etc., which are not limited in this embodiment of the present application. Input the spectral features of the sample audio into the content coding network to obtain the content feature vector of the sample audio; input the spectral features of the sample audio into the speaker coding network to obtain the speaker feature vector of the sample audio.

The speaker coding network models the characteristics of the speaker (speaker identity) and extracts the characteristics of the speaker. In addition, for phonemes that are not related to the characteristics of the speaker, such as stress (stress), pause (pause), etc., Characterized by the pitch frequency, the pitch frequency of the sample audio can be obtained by means of digital signal processing.

S102. Determine the mutual information of the content feature vector, the speaker feature vector and the pitch frequency.

In order to realize the decoupling between the three features of the content feature vector, the speaker feature vector and the pitch frequency through model training, and reduce the dependency between them, these three features are introduced in the model training process in the embodiment of the present disclosure. Mutual information between features, it can be understood that the mutual information between the three features here can refer to their respective mutual information. The lower the mutual information between the two features, the more The lower the correlation between them, the less overlapping the information they each contain.

S103. Input the content feature vector, the speaker feature vector and the pitch frequency into the sound spectrum decoding network of the speech style transfer model to obtain the predicted sound spectrum feature.

The input of the sound spectrum decoding network is the content feature vector, the speaker feature vector and the pitch frequency, and the output is the predicted sound spectrum feature. Since the content feature vector, the speaker feature vector and the pitch frequency are all features of the sample audio, therefore, in this step The sound spectrum decoding network actually reconstructs the sound spectrum features of the sample audio according to the above three features, and obtains the predicted sound spectrum features.

S104. Update parameters of the speech style transfer model according to the predicted sound spectrum feature and mutual information.

The aforementioned steps have already explained that the predicted spectral features are to reconstruct the spectral features of the sample audio, so the parameters of the speech style transfer model can be adjusted according to the predicted spectral features and the spectral features of the sample audio. At the same time, in order to ensure the content features The decoupling between the three features of vector, speaker feature vector and pitch frequency will also reduce mutual information as the goal of training optimization. According to the predicted spectral features and mutual information, the parameters of the speech style transfer model are updated to ensure prediction. Based on the accuracy of the spectral features, the content feature vector, speaker feature vector and pitch frequency are decoupled.

In the method of the embodiment of the present disclosure, the content feature vector, the speaker feature vector and the pitch frequency of the sample audio are extracted, and the features of these three aspects are used to determine the predicted spectral features. The parameters are updated, and the mutual information between the content feature vector of the sample audio, the speaker feature vector and the pitch frequency is also added. The reduction of mutual information is also used as the optimization goal of model training. In the training process, by reducing the mutual information, the content features The vector, the speaker feature vector and the pitch frequency are decoupled, so as to ensure that the input of the speaker encoding network is changed to the target audio in the subsequent application process, so that the speaker encoding network outputs the speaker feature vector of the target audio, that is, The sound spectrum decoding network can accurately output the sound spectrum features of the voice style of the target audio without being affected or interfered by other features, so that the voice style transfer effect is better.

On the basis of the above-mentioned embodiment, the implementation manner of each step will be further described.

Optionally, in step S101, the spectral features of the sample audio are input into the content coding network of the speech style transfer model to obtain the content feature vector of the sample audio, including:

The sound spectrum feature is input into the first feature extraction network in the content coding network to obtain a first feature vector, and the frame number of the first feature vector is lower than the frame number of the sound spectrum feature; the first feature vector is input into the content coding network. In the quantizer, the second eigenvector with the highest similarity with the first eigenvector is determined; the second eigenvector is updated by using the comparative prediction coding method to obtain the content eigenvector.

Optionally, in step S104, updating the parameters of the speech style transfer model according to the predicted sound spectrum feature and mutual information, including: determining the first loss of the content coding network according to the content feature vector; determining the sound spectrum according to the predicted sound spectrum feature. The second loss of the decoding network; according to the first loss, the second loss and the mutual information, the parameters of the speech style transfer model are updated.

Optionally, the first sub-loss is determined according to the second feature vector, and the second sub-loss is determined according to the content feature vector; the sum of the first sub-loss and the second sub-loss is determined as the first loss.

The following description will be made with reference to FIGS. 2 and 3 .

The content coding network consists of {h-net+quantizer+g-net}, where h-net is the first feature extraction network, optional, h-net consists of {Conv+{LayerNorm+Linear+ReLU}*4+Linear } composition; quantizer (quantizer) is a codebook (codebook), composed of multiple learnable vectors, for example, composed of 512 64-dimensional learnable vectors; g-net is composed of unidirectional RNN.

The input of the content coding network is the spectral feature of the sample audio. For example, the dimension of the spectral feature can be 20, the duration of each frame is 25ms, and the frame shift is 10ms. The content coding network can be calculated with a certain amount of context, such as context 2 frames each, the output of the content coding network is the content feature vector extracted from the sample audio.

这样将第一特征向量Z压缩到更低维度紧凑的空间，去除了不重要的细节，保留了基础的语言学信息。通过最小化矢量量化损失，来完成对第一特征向量Z的压缩映射。根据第二特征向量

确定第一子损失，即矢量量化损失L_VQ如下：Among them, h-net is to map the spectral feature X (T frame) of the input sample audio to the first feature vector Z (for example, T/2 frame), and by compressing the spectral feature X of the sample audio, from the sample audio Extract high-level features from the spectral features of , so that the obtained first feature vector Z has less redundant information. After that, the vector quantization (Vector Quantization, VQ) method of the sampling dictionary learning algorithm (DictionaryLearning) is used to find the most similar vector to each column of the first feature vector Z from the codebook to form the second feature vector.

In this way, the first feature vector Z is compressed into a lower-dimensional compact space, unimportant details are removed, and basic linguistic information is preserved. The compression mapping to the first feature vector Z is done by minimizing the vector quantization loss. According to the second eigenvector

Determine the first sub-loss, the vector quantization loss L _VQ as follows:

Among them, K is the number of sample audios, and sg represents stop-gradient.

g-net用于对第二特征向量

进行更新得到更新后的

更新后的

即为内容编码网络输出的内容特征向量。其中，g-net利用参数矩阵W乘以第二特征向量

以对第二特征向量

进行更新。在训练过程中，为了使第二特征向量

能够学习到更多的局部结构信息，同时丢弃掉底层的低级信息和噪声，本公开实施例中引入了无监督的对比预测编码(Contrastive Predictive Coding，CPC)方法，利用未来或上下文的信息，来提取紧凑潜在的特征表示，将自回归建模和噪声对比估计与预测编码相结合，以一种无监督的方式学习抽象表示，通过最小化信息噪声对比估计(Information Noise-Contrastive Estimation，lnfoNCE)损失L_CPC对VQ(作为编码器)和g-net(作为自回归模型)进行联合训练，也就是，根据内容特征向量确定第二子损失，即信息噪声对比损失L_CPC如下：The input of g-net is the second feature vector

g-net is used for the second eigenvector

update get updated

after the update

It is the content feature vector output by the content coding network. Among them, g-net uses the parameter matrix W to multiply the second eigenvector

to the second eigenvector

to update. During training, in order to make the second feature vector

It can learn more local structural information, while discarding the low-level information and noise of the bottom layer. Extract compact latent feature representations, combine autoregressive modeling and noise-contrastive estimation with predictive coding, and learn abstract representations in an unsupervised manner by minimizing Information Noise-Contrastive Estimation (lnfoNCE) loss The L _CPC jointly trains VQ (as an encoder) and g-net (as an autoregressive model), that is, the second sub-loss is determined according to the content feature vector, that is, the information noise contrast loss L _CPC is as follows:

是一个正样本，是从负样本集合Ω_k，t，m中向前移动m步得到的，示例的集合样本数取10；r_k，t是g-net中用于预测t时刻之后的第二特征向量

的中间值。Among them, T′=T/2-M, W _m (m=1, 2, ..., M); K is the number of sample audios; W is the parameter matrix that needs to be trained in g-net, and M is a constant , the M of the example is 6;

is a positive sample, which is obtained by moving forward m steps from the negative sample set Ω _{k, t, m} , and the number of samples in the sample set is 10; r _{k, t} is used in g-net to predict the th Two eigenvectors

the median value of .

In this way, the above-mentioned sum of the first sub-loss and the second sub-loss is determined as the first loss of the content coding network, that is, the total loss of each part in the content coding network. The calculation can ensure the accuracy of the information contained in the content feature vector output by the content coding network obtained by training.

Optionally, the speaker coding network consists of {ConvBank*8+Conv1d*12+average-pooling+Linear*4}, the input of the speaker coding network is the spectral feature X of the sample audio, and the ConvBank layer is used to expand the experience Wild, capture long-term information, and finally use the pooling layer to aggregate the information features, and finally output the speaker feature vector s.

The speaker coding network mainly models the characteristics of the speaker. The speaking characteristics of the speaker are composed of conscious and unconscious activities, and are generally highly variable. From the perspective of sound spectrum, it includes three aspects. On the one hand, it is inherently stable. Coarse-grained characteristics, this part is determined by the physiological structure of the speaker. Different speakers have different subjective auditory perceptions, which are manifested in the low-frequency region of the sound spectrum, such as the average pitch, which reflects the channel impulse. The spectral envelope of the excitation response, the relative amplitude and position of the formant, etc., on the other hand, the unstable short-term acoustic characteristics, the speed of the pronunciation, the sharp and rapid situation, the frequency, the loudness (intensity), the refined structure of the spectrum, etc., these can reflect the changes in the speaker's psychology or spirit, and different emotions and intentions can be expressed by changing these during conversation. The above two features are extracted by the speaker coding network. The multi-layer convolutional network in the speaker coding network extracts features at different levels, and then integrates them, thereby ensuring the accuracy of the speaker features.

The last aspect is that stress, pause, etc. have nothing to do with the speaker itself, which is characterized by the fundamental frequency p, which can be determined by discrete Fourier transform (DFT), harmonic summation and other numbers. The method of signal processing is determined.

After the content feature vector, speaker feature vector and pitch frequency of the sample audio are determined, the mutual information between them can be determined. In the embodiment of the present disclosure, the upper limit algorithm of the logarithm ratio is used to determine the upper limit of mutual information between the content feature vector, the speaker feature vector, and the pitch frequency; The sum of the upper limit of mutual information between , is determined as the mutual information of the content feature vector, the speaker feature vector and the pitch frequency.

As described in the foregoing embodiments, if there are no explicit constraints on the above-mentioned three features, namely the content feature vector, the speaker feature vector and the pitch frequency, the three features are likely to depend on each other. For example, the content feature contains some speakers. The speaker features contain part of the pitch frequency, etc., which leads to an unsatisfactory effect of speech style transfer. Therefore, in the embodiments of the present disclosure, from the perspective of information theory, mutual information is used to constrain the dependencies between the above features. In order to ensure that the mutual information between them is reduced, the comparison logarithm is introduced in the embodiments of the present disclosure. The algorithm of Contrastive Log-ratio Upper Bound (CLUB) is used to estimate the upper limit of the mutual information between them, and the upper limit is taken as the mutual information between them, and the reduction of mutual information is used as the optimization goal during the training process. , so as to ensure that the above three features extracted by the model obtained after training have the lowest correlation.

In the related art, the way to calculate the mutual information I(u, v) between the two features u and v is to calculate the relative entropy (Kullback-Leibler Divergence) between the joint distribution and the marginal distribution of the two, namely:

I(u,v)= _DKL (P(u,v);P(u)P(v))

After the introduction of the CLUB algorithm, the upper limit of the mutual information obtained is as follows:

表示给定v后对u的接近真实分布的估计，

可以通过神经网络θ_u，v去计算得到。可选的，通过变量估计神经网络{Linear+ReLU+Linear}估计出

的均值，通过{Linear+ReLU+Linear+Tanh}估计出

的对数方差，然后在计算出对数似然(log-likehood)，通过最大化如下损失更新神经网络θ_u，v：in,

represents an estimate of the near-true distribution of u given v,

It can be calculated by the neural network θ _{u, v} . Optionally, estimated by the variable estimation neural network {Linear+ReLU+Linear}

The mean of , estimated by {Linear+ReLU+Linear+Tanh}

Then, after calculating the log-likelihood, update the neural network θ _{u, v} by maximizing the loss as follows:

Substitute the content feature vector, speaker feature vector and pitch frequency respectively, and the mutual information between these three features is obtained as follows:

故内容特征向量、说话人特征向量和基音频率的整体的互信息为L_MI：in

Therefore, the overall mutual information of the content feature vector, the speaker feature vector and the pitch frequency is L _MI :

Optionally, the sound spectrum decoding network is composed of {LSTM+Conv1d+LSTM*2+Linear}. The input of the sound spectrum decoding network is the concatenation of the content feature vector of the sample audio, the speaker feature vector and the pitch frequency, and the output is To predict the spectral features, in order to make the predicted spectral features approach the spectral features of the real sample audio, the error loss is calculated by the mean square error (MSE), that is, the second loss, to update the spectral decoding network, the error loss _LRECs are as follows:

When splicing the content feature vector, the speaker feature vector and the pitch frequency, the content feature vector may be up-sampled, that is, extended to T frames, for example, linear interpolation may be used for extension. In addition, the speaker feature vector is copied and repeated to T frames.

Through the above process, the first loss, the second loss and the mutual information are obtained, and the sum of the first loss, the second loss and the mutual information is determined as the total loss; according to the total loss, the parameters of the speech style transfer model are updated.

That is, the total loss _LVC of the speech style transfer model is:

L _VC =L _VQ +L _CPC +L _REC +λ _MI L _MI

Among them, λ _MI is a constant weight (generally 0.1 or 0.01, etc.), which is used to adjust the weight of L _MI in the total loss.

Through the method of the embodiment of the present disclosure, in the process of training the speech style transfer model, the vector quantization keeps important linguistic information in the content feature vector, removes unnecessary redundant information, and uses mutual information as a loss pair. The model is updated, and the content feature vector, the speaker feature vector and the pitch frequency are decoupled through unsupervised training, so that in the application process after the training is completed, the target audio is input into the speaker coding network, which can be used. The model outputs the phonetic spectral features of the speech style of the target audio.

FIG. 4 is a schematic flowchart of a voice style transfer method provided according to an embodiment of the present disclosure. The execution body of the method is a voice style transfer device, and the device can be implemented by means of software and/or hardware. As shown in Figure 4, the method includes:

S401. Acquire a first sound spectrum feature of the source audio and a second sound spectrum feature of the target audio.

S402. Input the first sound spectrum feature into the content coding network of the speech style transfer model, obtain the content feature vector of the source audio, and extract the pitch frequency of the source audio.

S403. Input the second sound spectrum feature into the speaker coding network of the speech style transfer model to obtain the speaker feature vector of the target audio.

S404. Input the content feature vector of the source audio, the speaker feature vector of the target audio, and the fundamental frequency of the source audio into the sound spectrum decoding network of the speech style transfer model to obtain the sound spectrum features to be output, and generate a sound spectrum according to the sound spectrum features to be output. Audio to be output.

Wherein, the speech style transfer model is obtained by training according to the method of the foregoing embodiment.

In the embodiment of the present disclosure, the functions of each part in the speech style transfer model are similar to those in the foregoing embodiments, and will not be repeated here. In the application process, the content encoding network extracts the content feature vector of the source audio, and the speaker encoding network extracts the target The speaker feature vector of the audio is further obtained by the sound spectrum decoding network according to the content feature vector of the source audio, the speaker feature vector of the target audio and the fundamental frequency of the source audio, and finally the output sound spectrum is obtained through the output sound spectrum. audio. The content of the output audio is the content of the source audio, and the voice style is the voice style of the target audio. By training the speech style transfer model by the method in the foregoing embodiment, in the application process, there is no dependency and interference between features, and the speech style transfer effect is better.

FIG. 5 is a schematic structural diagram of an apparatus for training a speech style transfer model according to an embodiment of the present disclosure. As shown in FIG. 5, the training device 500 of the speech style transfer model includes:

The feature extraction module 501 is used to input the spectral features of the sample audio into the content coding network and the speaker coding network of the speech style transfer model respectively, obtain the content feature vector and the speaker feature vector of the sample audio, and extract the fundamental frequency of the sample audio. ;

Mutual information calculation module 502, for determining the mutual information of the content feature vector, the speaker feature vector and the pitch frequency;

The sound spectrum decoding module 503 is used for inputting the content feature vector, the speaker feature vector and the pitch frequency into the sound spectrum decoding network of the speech style transfer model to obtain the predicted sound spectrum feature;

The updating module 504 is configured to update the parameters of the speech style transfer model according to the predicted spectral features and mutual information.

In one embodiment, the update module 504 includes:

a first determining unit, configured to determine the first loss of the content coding network according to the content feature vector;

a second determining unit, configured to determine the second loss of the audio spectrum decoding network according to the predicted audio spectrum feature;

The updating unit is used for updating the parameters of the speech style transfer model according to the first loss, the second loss and the mutual information.

In one embodiment, the update unit includes:

a first determination subunit, configured to determine the sum of the first loss, the second loss and the mutual information as the total loss;

The update subunit is used to update the parameters of the speech style transfer model according to the total loss.

In one embodiment, the feature extraction module 501 includes:

The first extraction unit is used for inputting the sound spectrum feature into the first feature extraction network in the content coding network to obtain a first feature vector, and the frame number of the first feature vector is less than the frame number of the sound spectrum feature;

The second extraction unit is used to input the first feature vector into the quantizer in the content coding network, to determine the second feature vector with the highest similarity with the first feature vector;

The third extraction unit is used for updating the second feature vector by adopting the contrastive predictive coding method to obtain the content feature vector.

In one embodiment, the first determining unit includes:

a second determination subunit, configured to determine the first subloss according to the second feature vector, and determine the second subloss according to the content feature vector;

The third determination sub-unit is used for determining the sum of the first sub-loss and the second sub-loss as the first loss.

In one embodiment, the mutual information calculation module 502 includes:

The third determination unit is used to determine the upper limit of mutual information between the content feature vector, the speaker feature vector and the pitch frequency by comparing the logarithm ratio upper limit algorithm;

The fourth determining unit is configured to determine the sum of the upper limit of mutual information between the content feature vector, the speaker feature vector and the pitch frequency as the mutual information of the content feature vector, the speaker feature vector and the pitch frequency.

The apparatus of the embodiment of the present disclosure can be used to execute the training method of the speech style transfer model in the above method embodiment, and the implementation principle and technical effect thereof are similar, and are not repeated here.

FIG. 6 is a schematic structural diagram of a voice style transfer apparatus provided according to an embodiment of the present disclosure. As shown in FIG. 6, the voice style transfer apparatus 600 includes:

An acquisition module 601 is used to acquire the first sound spectrum feature of the source audio and the second sound spectrum feature of the target audio;

The content coding module 602 is used to input the first sound spectrum feature into the content coding network of the speech style transfer model, obtain the content feature vector of the source audio, and extract the pitch frequency of the source audio;

The speaker encoding module 603 is used to input the second sound spectrum feature into the speaker encoding network of the speech style transfer model to obtain the speaker feature vector of the target audio;

The sound spectrum decoding module 604 is used for inputting the content feature vector of the source audio, the speaker feature vector of the target audio and the fundamental frequency of the source audio into the sound spectrum decoding network of the speech style transfer model, to obtain the sound spectrum features to be output, and according to the sound spectrum decoding network to be output. Output spectral features to generate audio to be output;

Wherein, the speech style transfer model is obtained by training according to the method of the foregoing embodiment.

The apparatus in the embodiment of the present disclosure can be used to execute the speech style transfer method in the foregoing method embodiment, and the implementation principle and technical effect thereof are similar, and are not repeated here.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device and a non-transitory computer-readable storage medium storing computer instructions.

According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, the program product includes: a computer program, the computer program is stored in a readable storage medium, and at least one processor of the electronic device can be read from the readable storage medium A computer program, where at least one processor executes the computer program to cause the electronic device to execute the solution provided by any of the foregoing embodiments.

7 is a schematic block diagram of an electronic device used to implement the method of an embodiment of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 7 , the electronic device 700 includes a computing unit 701 that can be programmed according to a computer program stored in a read only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage unit 708 . Various appropriate actions and processes are performed. In the RAM 703, various programs and data necessary for the operation of the device 700 can also be stored. The computing unit 701 , the ROM 702 , and the RAM 703 are connected to each other through a bus 704 . An input/output (I/O) interface 705 is also connected to bus 704 .

Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc. ; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Computing unit 701 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 701 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as a training method of a voice style transfer model or a voice style transfer method. For example, in some embodiments, a method of training a speech style transfer model or a method of speech style transfer may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 700 via ROM 702 and/or communication unit 709 . When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the training method of the speech style transfer model or the speech style transfer method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform a training method of a speech style transfer model or a speech style transfer method by any other suitable means (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or a cloud host. It is a host product in the cloud computing service system to solve the problem of traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). , there are the defects of difficult management and weak business expansion. The server can also be a server of a distributed system, or a server combined with a blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present application can be executed in parallel, sequentially or in different orders, and as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, no limitation is imposed herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included within the protection scope of the present disclosure.

Claims (17)

1. A training method of a speech style migration model comprises the following steps:

respectively inputting the voice spectrum characteristics of the sample audio into a content coding network and a speaker coding network of a voice style migration model to obtain a content characteristic vector and a speaker characteristic vector of the sample audio, and extracting the fundamental tone frequency of the sample audio;

determining mutual information of the content feature vector, the speaker feature vector and the fundamental tone frequency;

inputting the content feature vector, the speaker feature vector and the fundamental tone frequency into a sound spectrum decoding network of the voice style migration model to obtain predicted sound spectrum features;

and updating the parameters of the voice style migration model according to the predicted voice spectrum characteristics and the mutual information.

2. The method of claim 1, wherein said updating parameters of said speech style migration model based on said predicted sonographic features and said mutual information comprises:

determining a first loss of the content encoding network based on the content feature vector;

determining a second loss of the sound spectrum decoding network according to the predicted sound spectrum characteristics;

and updating parameters of the voice style migration model according to the first loss, the second loss and the mutual information.

3. The method of claim 2, wherein said updating parameters of said speech style migration model based on said first loss, said second loss, and said mutual information comprises:

determining the sum of the first loss, the second loss and the mutual information as a total loss;

and updating the parameters of the voice style migration model according to the total loss.

4. The method according to claim 2 or 3, wherein inputting the sound spectrum feature of the sample audio into the content coding network of the speech style migration model to obtain the content feature vector of the sample audio comprises:

inputting the sound spectrum characteristics into a first characteristic extraction network in the content coding network to obtain a first characteristic vector, wherein the frame number of the first characteristic vector is less than that of the sound spectrum characteristics;

inputting the first feature vector into a quantizer in the content coding network to determine a second feature vector having the highest similarity with the first feature vector;

and updating the second feature vector by adopting a contrast prediction coding method to obtain the content feature vector.

5. The method of claim 4, wherein said determining a first loss of the content encoding network from the content feature vector comprises:

determining a first sub-loss according to the second feature vector, and determining a second sub-loss according to the content feature vector;

determining a sum of the first sub-loss and the second sub-loss as the first loss.

6. The method of any of claims 1-5, wherein the determining mutual information of the content feature vector, the speaker feature vector, and the pitch frequency comprises:

determining the mutual information upper limit between the content characteristic vector, the speaker characteristic vector and the fundamental tone frequency by comparing a comparison ratio upper limit algorithm;

and determining the sum of the content characteristic vector, the speaker characteristic vector and the mutual information upper limit between every two pitch frequencies as the mutual information of the content characteristic vector, the speaker characteristic vector and the pitch frequencies.

7. A method of voice style migration, comprising:

acquiring a first sound spectrum characteristic of a source audio and a second sound spectrum characteristic of a target audio;

inputting the first voice spectrum characteristic into a content coding network of a voice style migration model to obtain a content characteristic vector of the source audio, and extracting a fundamental tone frequency of the source audio;

inputting the second acoustic spectrum characteristic into a speaker coding network of a speech style migration model to obtain a speaker characteristic vector of the target audio;

inputting the content characteristic vector of the source audio, the speaker characteristic vector of the target audio and the fundamental tone frequency of the source audio into a sound spectrum decoding network of the voice style migration model to obtain a sound spectrum characteristic to be output, and generating the audio to be output according to the sound spectrum characteristic to be output;

wherein the speech style migration model is trained according to the method of any one of claims 1-6.

8. A training apparatus for a speech style migration model, comprising:

the characteristic extraction module is used for respectively inputting the sound spectrum characteristics of the sample audio into a content coding network and a speaker coding network of the voice style migration model to obtain a content characteristic vector and a speaker characteristic vector of the sample audio and extracting the fundamental tone frequency of the sample audio;

a mutual information calculation module for determining mutual information of the content feature vector, the speaker feature vector and the fundamental tone frequency;

a voice spectrum decoding module, configured to input the content feature vector, the speaker feature vector, and the pitch frequency into a voice spectrum decoding network of the speech style migration model to obtain a predicted voice spectrum feature;

and the updating module is used for updating the parameters of the voice style migration model according to the predicted sound spectrum characteristics and the mutual information.

9. The apparatus of claim 8, wherein the update module comprises:

a first determining unit, configured to determine a first loss of the content encoding network according to the content feature vector;

a second determining unit, configured to determine a second loss of the sound spectrum decoding network according to the predicted sound spectrum feature;

and the updating unit is used for updating the parameters of the voice style migration model according to the first loss, the second loss and the mutual information.

10. The apparatus of claim 9, wherein the updating unit comprises:

a first determining subunit, configured to determine a sum of the first loss, the second loss, and the mutual information as a total loss;

and the updating subunit is used for updating the parameters of the voice style migration model according to the total loss.

11. The apparatus of claim 9 or 10, wherein the feature extraction module comprises:

a first extraction unit, configured to input the audio spectrum feature into a first feature extraction network in the content coding network to obtain a first feature vector, where a frame number of the first feature vector is smaller than a frame number of the audio spectrum feature;

a second extraction unit, configured to input the first feature vector into a quantizer in the content coding network to determine a second feature vector having a highest similarity with the first feature vector;

and the third extraction unit is used for updating the second feature vector by adopting a contrast predictive coding method to obtain the content feature vector.

12. The apparatus of claim 11, wherein the first determining unit comprises:

a second determining subunit, configured to determine a first sub-loss according to the second feature vector, and determine a second sub-loss according to the content feature vector;

a third determining subunit, configured to determine a sum of the first sub-loss and the second sub-loss as the first loss.

13. The apparatus according to any one of claims 8-12, wherein the mutual information calculation module comprises:

a third determining unit, configured to determine, by comparing a comparison ratio upper limit algorithm, a mutual information upper limit between each two of the content feature vector, the speaker feature vector, and the pitch frequency;

a fourth determining unit, configured to determine a sum of mutual information upper limits between each two of the content feature vector, the speaker feature vector, and the pitch frequency as mutual information of the content feature vector, the speaker feature vector, and the pitch frequency.

14. A speech style migration apparatus comprising:

the acquisition module is used for acquiring a first sound spectrum characteristic of source audio and a second sound spectrum characteristic of target audio;

a content coding module, configured to input the first spectrum feature into a content coding network of a speech style migration model, obtain a content feature vector of the source audio, and extract a pitch frequency of the source audio;

the speaker coding module is used for inputting the second acoustic spectrum characteristic into a speaker coding network of a speech style migration model to obtain a speaker characteristic vector of the target audio;

a sound spectrum decoding module, configured to input the content feature vector of the source audio, the speaker feature vector of the target audio, and the pitch frequency of the source audio into a sound spectrum decoding network of the speech style migration model, to obtain a sound spectrum feature to be output, and generate a sound spectrum to be output according to the sound spectrum feature to be output;

wherein the speech style migration model is trained according to the method of any one of claims 1-6.

15. An electronic device, comprising:

at least one processor; and a memory communicatively coupled to the at least one processor;

wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-7.

Images