CN109215680B - A speech restoration method based on convolutional neural network - Google Patents

Abstract

The invention relates to the technical field of voice processing, and in particular to a voice restoration method based on a convolutional neural network, comprising: step S1, collecting electronic camouflage voice; Converting the camouflaged voice into a standard voice sequence with preset dimensions; Step S3, using a restoration model to restore the standard voice sequence to the original voice sequence; wherein, in Step S2, the preprocessing process includes formant breakage cleaning, formant merge optimization And formant sequence adjustment; the convolutional neural network is used for electronic camouflage voice restoration, which has a high enough restoration effect to meet the high-demand electronic camouflage voice restoration scene.

Description

A speech restoration method based on convolutional neural network

technical field

The invention relates to the technical field of speech processing, in particular to a speech restoration method based on a convolutional neural network.

Background technique

With the continuous development of information technology, the speaker recognition technology has made great progress, and the analysis and research of the speaker's personality characteristics of speech has received extensive attention. However, the appearance of camouflaged speech makes the research work of speaker recognition face unprecedented challenges. In a broad sense, disguised speech means that any change, distortion or deviation from normal speech, regardless of the cause, can be called disguised speech. In a narrow sense, camouflage refers to deliberate camouflage, that is, the intentional distortion of normal speech for the purpose of concealing one's identity.

In criminal cases, in order to cover up their real identities, criminals often disguise their pronunciation by using electronic camouflage voices to avoid being attacked. Electronic camouflaged speech refers to the distorted speech generated by using electronic equipment or speech processing software to process the speaker's original speech through voice-changing camouflage. Electronic camouflage speech has a high degree of camouflage for the speaker's identity. It changes the acoustic personality characteristics of the speaker's original speech. Not only the speaker cannot be identified from the human ear, but also it is difficult to confirm through the detection of electro-acoustic instruments. Voice is an important biological feature for personal identification. The emergence of electronic camouflage voice will make voice identification even more difficult. Therefore, the characteristics of various electronic camouflage voices are deeply studied, the characteristic parameters with the most obvious changes are extracted, the changing laws of electronic camouflage voices are summarized, and the restoration method for electronic camouflage voices is designed, so that it can restore the changing electronic camouflage voices. It is of great significance for the identification of the speaker and the exertion of its evidential effect.

Electronic camouflage voice restoration refers to the process of weakening or eliminating the camouflage characteristics of electronic camouflage voice through a certain algorithm or model, and generating a restored voice that is closer to the original voice than the electronic camouflage voice. Since electronic camouflage speech is generally based on a certain algorithm to realize the change of its own acoustic characteristics, there is a certain change rule in the process of converting the original speech into electronic camouflage speech. And the voice has the characteristics of short-term stability, so by statistical comparison of the voiceprint deviation characteristics between the original voice and the electronic camouflage voice, it provides a basis for the restoration of the electronic camouflage voice.

However, the restoration effect of the existing electronic camouflage voice is not ideal, and cannot meet the scenes with high requirements for restoration effect.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention proposes a voice restoration method based on a convolutional neural network, wherein a preprocessing model and a restoration model are preset;

The reduction model includes a convolutional neural network and an initial reduction factor, and the reduction model trains the initial reduction factor through the convolutional neural network to generate a reduction for controlling the reduction process of the reduction model. factor;

The convolutional neural network includes a connected expanded causal convolution layer and a sub-control layer, and the sub-control layer is used to convert the initial reduction factor into a sequence of preset dimensions;

The voice restoration method also includes:

Step S1, collecting the electronic camouflage voice, and simultaneously extracting acoustic parameters in the electronic camouflage voice;

Step S2, using the preprocessing model to preprocess the acoustic parameters to convert the acoustic parameters into a standard sequence with the preset dimension;

Step S3, using the restoration model to restore the electronic camouflage voice to a restored voice sequence, and the restoration model completes the restoration of the standard voice sequence according to the standard sequence;

Wherein, in the step S2, the preprocessing process includes formant breakage cleaning, formant merging optimization, and formant sequence adjustment.

In the above-mentioned speech restoration method, wherein, a gate activation unit is used in the expanded causal convolution layer to perform a nonlinear transformation process on the input from the sub-control layer.

The above-mentioned voice restoration method, wherein, the gate activation unit adopts the following function to perform the nonlinear conversion process:

为输入的电子伪装语音序列，*表示卷积运算，σ()表示 Sigmoid函数，W_f,k表示学习型的卷积滤波器指数，

表示滤波卷积滤波器，

表示门控卷积滤波器，

表示门激活函数，h表示所述还原因子，k表示层数，f表示滤波系数，g表示门控系数。In the formula,

is the input electronic camouflage speech sequence, * represents the convolution operation, σ() represents the Sigmoid function, W _f,k represents the learned convolution filter index,

represents the filtered convolution filter,

represents a gated convolution filter,

represents the gate activation function, h represents the reduction factor, k represents the number of layers, f represents the filter coefficient, and g represents the gating coefficient.

The above-mentioned speech restoration method, wherein the convolutional neural network further comprises a jump layer residual structure connecting the output of the causal convolutional layer;

The jump layer residual structure is used to identically map the input data before any pre-convolutional layer to the post-convolutional layer separated by a preset number of layers, and superimpose it on the residual calculated by the post-convolutional layer. output later.

The above-mentioned voice restoration method further includes: step S4, classifying and outputting the data of the restored voice sequence.

The above-mentioned voice restoration method, wherein, the step S4 specifically includes:

Step S41, using Softmax function to discretize and classify the restored data, so as to realize the normalization of the data sequence of the restored speech sequence;

Step S42, performing μ-law companding transformation on the restored speech sequence after the normalization of the data sequence, so as to reduce the amount of output operation.

The above-mentioned voice restoration method, wherein, the Softmax function is:

为输入的还原语音序列，N为向量维度。In the formula,

is the input restored speech sequence, and N is the vector dimension.

The above-mentioned voice restoration method, wherein, the μ-law companding conversion is completed by the following functions:

为输入的数据序列归一化后的所述还原语音序列。In the formula,

is the restored speech sequence after normalization of the input data sequence.

The above-mentioned voice restoration method, wherein, the formant damage cleaning comprises:

Step A1, extracts the formant data pair of the non-homogeneous vowels in the described electronic camouflage voice, and confirms the described formant data pair that has broken after the voiceprint comparison;

Step A2, according to the size of the number of non-zero formants, the data in the pair of formant data are distinguished and marked;

Step A3, performing cross-subtraction calculation on the corresponding formant center frequency parameter in the differentiated data to form an absolute value matrix about the difference;

Step A4, according to the absolute value matrix, take the formant data pair corresponding to the matrix with the minimum sum of the elements of different rows and different columns;

Step A5, performing the steps A1 to A4 on all formant data pairs of non-homogeneous finals to form a set of the formant data pairs after damage and cleaning.

The above-mentioned voice restoration method, wherein, the formant merging optimization comprises:

Step B1, extracts the formant data pair of the non-homogeneous final vowel in the described electronic camouflage voice according to a preset formant extraction rule;

Step B2, according to the size of the number of non-zero formants, the data in the pair of formant data is distinguished and marked;

Step B3, make a difference between the formant center frequency parameter in the data with the small number of non-zero formants and the adjacent formant center frequency parameter, and extract a group of the described formants with the smallest absolute value of the difference. The formant center frequency parameter, denoted as (f _v,1 ,f _v+1,1 ), and do the following transformation:

In step B4, the steps B1 to B3 are performed on all formant data pairs of non-homogeneous finals to form a set of the formant data pairs after combining and optimizing.

The above-mentioned voice restoration method, wherein, the formant sequence adjustment comprises:

Step C1, extracting the formant center frequency parameter of the non-homogeneous finals in the electronic camouflage voice is divided into different sub-sets A ₁ , A ₂ , A ₃ , A ₄ according to the numerical range, and judges any sub-set A _j ( The number of elements in j=1,2,3,4);

进行S₀中对应的字韵母共振峰

进行同化操作，以保持同一字韵母的共振峰数量和实际位置一致；When there is only 1 or 0 elements in any subset A _j (j=1,2,3,4), if A _j =A _j+1 ≠φ, then keep the j that minimizes |ji| The corresponding A _j , and A _j+1 is set as the empty set, and then the formants of the letters and finals of S _tm or S _pn or S _rr

Carry out the corresponding letter-final formant in S ₀

Perform assimilation operation to keep the same number of formants and actual positions of finals of the same letter;

的集合；此时位于同一中心频率分布重叠区域的共振峰不超过2个，即card(M_j,j+1)≤2，q＝1,2,3，记B₁＝A₁-M₁₂，B₂＝A₂-M₁₂-M₂₃， B₃＝A₃-M₂₃-M₃₄，B₄＝A₄-M₃₄；令

其中

记

为集合M_j-1,j中中心频率最大的共振峰，记

为集合M_j,j+1中的中心频率最小的共振峰，则对每一个j＝1,2,3,4进行如下计算：Otherwise, let M ₁₂ =A ₁ ∩A ₂ , M ₂₃ =A ₂ ∩A ₃ , M ₃₄ =A ₃ ∩A ₄ , where M _j,j+1 (j=1,2,3) represents the Resonant peaks appearing in both A _j and A _j+1 while distributing overlapping regions

At this time, there are no more than 2 resonance peaks located in the overlapping area of the same center frequency distribution, that is, card(M _j,j+1 )≤2, q=1,2,3, denoted B ₁ =A ₁ -M ₁₂ , B ₂ =A ₂ -M ₁₂ -M ₂₃ , B ₃ =A ₃ -M ₂₃ -M ₃₄ , B ₄ =A ₄ -M ₃₄ ; let

in

remember

is the formant with the largest center frequency in the set M _j-1,j , denoted

is the formant with the smallest center frequency in the set M _j,j+1 , then the following calculation is performed for each j=1,2,3,4:

放入对应的集合A_j当中，使得A_j(j＝1,2,3,4)中至多只有一条非零共振峰；In the formula, [x] represents the largest integer not greater than x, and x refers to the operation result of the operation expression in the operation symbol []; then, set the sets A ₁ , A ₂ , A ₃ , and A ₄ to be empty, and set the

Put it into the corresponding set A _j , so that there is at most one non-zero formant in A _j (j=1, 2, 3, 4);

Step C2, performing the step C1 on all formant data pairs of non-homogeneous finals to form a set of the formant data pairs after sequence adjustment.

Beneficial effects: The voice restoration method based on the convolutional neural network proposed by the present invention adopts the convolutional neural network to restore the electronic camouflage voice, which has a high enough restoration effect and can meet the high-demand electronic camouflage voice restoration scene.

Description of drawings

1 is a flowchart of steps of a method for restoring speech based on a convolutional neural network according to an embodiment of the present invention;

2 is a schematic diagram of the receptive field of the enlarged convolution as the enlargement coefficient increases according to an embodiment of the present invention;

3 is a schematic diagram of a training effect generated by combining enlarged convolution with causal convolution in an embodiment of the present invention;

4 is a schematic diagram of the basic structure of speech restoration based on a convolutional neural network according to an embodiment of the present invention;

FIG. 5 is a structural diagram of a jump layer connection and a residual block in a deep transfer optimization process according to an embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

In a preferred embodiment, as shown in FIG. 1 , a voice restoration method based on a convolutional neural network is proposed, wherein a preprocessing model and a restoration model are preset;

The restoration model includes a convolutional neural network and an initial restoration factor, and the restoration model trains the initial restoration factor through the convolutional neural network to generate a restoration factor for controlling the restoration process of the restoration model;

The convolutional neural network includes a connected expanded causal convolution layer and a sub-control layer, and the sub-control layer is used to convert the initial reduction factor into a sequence of preset dimensions;

The voice restoration method may also include:

Step S1, collecting the electronic camouflage voice, and simultaneously extracting the acoustic parameters in the electronic camouflage voice;

Step S2, using a preprocessing model to preprocess the acoustic parameters to convert the acoustic parameters into a standard sequence with preset dimensions;

Step S3, adopts the restoration model to restore the electronic camouflage voice to the restored voice sequence, and the restoration model completes the restoration of the standard voice sequence according to the standard sequence;

Wherein, in step S2, the preprocessing process includes formant breakage cleaning, formant merging optimization, and formant sequence adjustment.

In the above technical solution, the acoustic parameters may include the center frequency, bandwidth and sound intensity of the formant.

In a preferred embodiment, the gate activation unit is used in the augmented causal convolution layer to perform a nonlinear transformation process on the input from the sub-control layer.

In the above embodiment, preferably, the gate activation unit adopts the following function to perform the nonlinear conversion process:

为输入的电子伪装语音序列，*表示卷积运算，σ()表示 Sigmoid函数，W_f,k表示学习型的卷积滤波器指数，

表示滤波卷积滤波器，

表示门控卷积滤波器，

表示门激活函数，h表示还原因子，k表示层数，f表示滤波系数，g表示门控系数。In the formula,

is the input electronic camouflage speech sequence, * represents the convolution operation, σ() represents the Sigmoid function, W _f,k represents the learned convolution filter index,

represents the filtered convolution filter,

represents a gated convolution filter,

represents the gate activation function, h represents the reduction factor, k represents the number of layers, f represents the filter coefficient, and g represents the gating coefficient.

In a preferred embodiment, the convolutional neural network further includes a layered residual structure connecting the outputs of the causal convolutional layers;

The transition layer residual structure is used to map the input data before any pre-convolutional layer to the post-convolutional layer separated by a preset number of layers, and superimpose it on the residual calculated by the post-convolutional layer.

In the above technical solution, the jump-layer residual structure may include a jump-layer connection structure and a residual structure; the pre-convolutional layer and the post-convolutional layer form a connected causal convolutional layer.

In a preferred embodiment, the method further includes: step S4, classifying and outputting the data of the restored speech sequence.

In the above embodiment, preferably, step S4 specifically includes:

Step S41, using the Softmax function to discretize the restored data to achieve normalization of the data sequence of the restored speech sequence;

In step S42, μ-law companding transformation is performed on the restored speech sequence after the normalization of the data sequence, so as to reduce the amount of output operation.

In the above-mentioned embodiment, preferably, the Softmax function is:

为输入的还原语音序列，N为向量维度。In the formula,

is the input restored speech sequence, and N is the vector dimension.

In the above embodiment, preferably, the μ-law companding conversion is completed by the following function:

为输入的数据序列归一化后的还原语音序列。In the formula,

The restored speech sequence normalized for the input data sequence.

In a preferred embodiment, the formant break cleaning may include:

Step A1, extracts the formant data pair of the non-homogeneous final vowel in the electronic camouflage voice, and confirms the formant data pair that appears damaged after the voiceprint comparison;

Step A2, according to the size of the number of non-zero formants, the data in the formant data pair is discriminated and marked;

Step A3, performing cross-subtraction calculation on the corresponding formant center frequency parameter in the differentiated data to form an absolute value matrix about the difference;

Step A4, according to the absolute value matrix, get the formant data pair corresponding to the matrix with the minimum sum of the elements of different rows and different columns;

Step A5: Steps A1 to A4 are performed on all formant data pairs of non-homogeneous finals to form a set of formant data pairs after breakage and cleaning.

In a preferred embodiment, the formant merging optimization may include:

Step B1, extracts the formant data pair of the non-homogeneous final vowel in the electronic camouflage voice according to a preset formant extraction rule;

Step B2, according to the size of the number of non-zero formants, the data in the formant data pair is discriminated and marked;

Step B3, make a difference between the formant center frequency parameter in the data with a small number of non-zero formants and the adjacent formant center frequency parameter, and extract a group of formant center frequency parameters with the smallest absolute value of the difference, record is (f _v,1 ,f _v+1,1 ), and do the following transformation:

In step B4, steps B1-B3 are performed on all formant data pairs of non-homogeneous finals to form a set of combined and optimized formant data pairs.

In a preferred embodiment, the formant sequence adjustment may include:

Step C1, extracting the formant center frequency parameters of the non-homogeneous finals in the electronic camouflage speech are divided into different subsets A ₁ , A ₂ , A ₃ , A ₄ according to the numerical range, and determine any subset A _j (j= 1,2,3,4) the number of elements;

进行S₀中对应的字韵母共振峰

进行同化操作，以保持同一字韵母的共振峰数量和实际位置一致；When there is only 1 or 0 elements in any subset A _j (j=1,2,3,4), if A _j =A _j+1 ≠φ, then keep the j that minimizes |ji| The corresponding A _j , and A _j+1 is set as the empty set, and then the formants of the letters and finals of S _tm or S _pn or S _rr

Carry out the corresponding letter-final formant in S ₀

Perform assimilation operation to keep the same number of formants and actual positions of finals of the same letter;

的集合；此时位于同一中心频率分布重叠区域的共振峰不超过2个，即card(M_j,j+1)≤2，q＝1,2,3，记B₁＝A₁-M₁₂，B₂＝A₂-M₁₂-M₂₃， B₃＝A₃-M₂₃-M₃₄，B₄＝A₄-M₃₄；令

其中

记

为集合M_j-1,j中中心频率最大的共振峰，记

为集合M_j,j+1中的中心频率最小的共振峰，则对每一个j＝1,2,3,4进行如下计算：Otherwise, let M ₁₂ =A ₁ ∩A ₂ , M ₂₃ =A ₂ ∩A ₃ , M ₃₄ =A ₃ ∩A ₄ , where M _j,j+1 (j=1,2,3) represents the Resonant peaks appearing in both A _j and A _j+1 while distributing overlapping regions

At this time, there are no more than 2 resonance peaks located in the overlapping area of the same center frequency distribution, that is, card(M _j,j+1 )≤2, q=1,2,3, denoted B ₁ =A ₁ -M ₁₂ , B ₂ =A ₂ -M ₁₂ -M ₂₃ , B ₃ =A ₃ -M ₂₃ -M ₃₄ , B ₄ =A ₄ -M ₃₄ ; let

in

remember

is the formant with the largest center frequency in the set M _j-1,j , denoted

is the formant with the smallest center frequency in the set M _j,j+1 , then the following calculation is performed for each j=1,2,3,4:

放入对应的集合A_j当中，使得A_j(j＝1,2,3,4)中至多只有一条非零共振峰；In the formula, [x] represents the largest integer not greater than x, and x refers to the operation result of the operation expression in the operation symbol []; then, set the sets A ₁ , A ₂ , A ₃ , and A ₄ to be empty, and set the

Put it into the corresponding set A _j , so that there is at most one non-zero formant in A _j (j=1, 2, 3, 4);

Step C2: Step C1 is performed on all formant data pairs of non-homogeneous finals to form a set of formant data pairs after sequence adjustment.

Below is the more detailed embodiment content that the technical solution of the present invention is provided:

The formant parameters are assumed to be as follows:

The cleaning steps for formant breakage are as follows:

放入W_l；(1) Respectively extract W _Nh in each group of S _c , by observing the formant and voiceprint imaging of the corresponding broadband spectrogram, select the formant data pair of non-homogeneous finals according to the aforementioned formant breakage reasons

put in W _l ;

记其中非零共振峰

数量较少者为

较多者为

(2) For a certain non-homogeneous final formant data pair in W _l

note where the non-zero formant

The lesser is

more

中的每个f_v,1(即共振峰中心频率)分别与

的f_1,1，…，f_u,1相减，记其差的绝对值为d_u,v，即d_u,v＝|f_u,1-f_v,1|，并放入u×v的矩阵D；(3) will

Each f _v,1 (that is, the formant center frequency) in the

Subtract the f _1,1 ,... _, f _{u, 1 of the} matrix D of v;

当

取最小值时，数组

所对应的

即为受损共振峰清洗所得该字的数据集；(4) Set a set

when

When taking the minimum value, the array

corresponding to

is the data set of the word obtained by cleaning the damaged formant;

进行上述(2)- (4)步操作，最终获得清洗共振峰折损后的数据对

的集合W_l′。(5) to each non-homogeneous final _formant data pair in W1

Carry out above-mentioned (2)-(4) step operation, finally obtain the data pair after cleaning formant breakage

The set W _l '.

The formant merging optimization steps are as follows:

放入W_m；(1) Extract W _Nh in each group of S _c respectively. Since W _l ∩ W _m ≠φ, it is necessary to observe the formant and voiceprint imaging of the corresponding broadband spectrogram, and select the non-homogeneous final formant data pair according to the aforementioned reasons for the merge of formants.

put in W _m ;

记其中非零共振峰

数量较少者为

较多者为

(2) For a certain non-homogeneous final formant data pair in W _l

note where the non-zero formant

The lesser is

more

中每个f_v，1(即共振峰中心频率，v＝1,2，…且v＜4)与其相邻的共振峰中心频率f_v+1，1做差，取其差的绝对值最小的一组共振峰中心频率(f_v,1,f_v+1,1)，就其所对应的

与

做出如下变换：(3) will

Each f _{v, 1} (that is, the formant center frequency, v=1, 2, ... and v < 4) makes a difference with its adjacent formant center frequency f _v+1 , 1, and the absolute value of the difference is the smallest A set of formant center frequencies (f _v,1 ,f _v+1,1 ) of

and

Make the following transformations:

进行上述2、3步操作，最终获得共振峰合并优化后的数据对

的集合W′_m。(4) For each inhomogeneous final formant data pair in W _m

Carry out the above 2 and 3 steps, and finally obtain the data pair after formant merging and optimization

The set _W'm of .

The formant sequence adjustment steps are as follows:

(1) Set A ₁ , A ₂ , A ₃ , and A ₄ to be empty, and select a certain letter final with less than 4 formants in the original voice set S ₀ of a certain group S _c sequentially according to the preset formant center frequency The ranges are put into A ₁ , A ₂ , A ₃ , A ₄ respectively,

When any A _j (j=1,2,3,4) has only 1 or 0 elements, that is

card(A _j )≤1, j=1,2,3,4

的原始顺序)，并将另一集合置为空集。j的值即为

经序列调整后的位置排序，并由此跳至第(3)步操作。否则，当存在至少一A_j(j＝1,2,3,4)中有至少2个元素时，还需进行第(2)步操作；If A _j =A _j+1 ≠φ, keep the A _j corresponding to the j that minimizes |ji| (i is

in the original order) and set the other set to the empty set. The value of j is

The position after the sequence adjustment is sorted, and thus jump to step (3). Otherwise, when there are at least 2 elements in at least one A _j (j=1, 2, 3, 4), step (2) needs to be performed;

由于在使用司法鉴定智能语音工作站进行检测时，已设定了共振峰中心频率的最小频率间隔为80Hz，故而位于同一中心频率分布重叠区域的共振峰不会超过2个，即(2) Suppose M ₁₂ =A ₁ ∩A ₂ , M ₂₃ =A ₂ ∩A ₃ , M ₃₄ =A ₃ ∩A ₄ , then the set M _j,j+1 (j=1,2,3) represents the factor Resonant peaks in both A _j and A _j+1 in the overlapping region of the center frequency distribution

Since the minimum frequency interval of the formant center frequency has been set to 80Hz when using the forensic identification intelligent voice workstation for detection, there will be no more than two formants located in the overlapping area of the same center frequency distribution, that is,

card(M _j,j+1 )≤2, q=1,2,3

其中

又记

集合M_j-1,j中中心频率最大的共振峰，记

集合M_j,j+1中中心频率最小的共振峰(当M_j-1,j或M_j,j+1为空集φ时，令

或

为向量0)。则对每一个j＝1,2,3,4都可进行如下计算：Denote B ₁ =A ₁ -M ₁₂ , B ₂ =A ₂ -M ₁₂ -M ₂₃ , B ₃ =A ₃ -M ₂₃ -M ₃₄ , and B ₄ =A ₄ -M ₃₄ . make

in

remember again

The formant with the largest center frequency in the set M _j-1,j , denoted

The formant with the smallest center frequency in the set M _j,j+1 (when M _j-1,j or M _j,j+1 is an empty set φ, let

or

is the vector 0). Then for each j=1, 2, 3, 4, the following calculation can be performed:

放入对应的集合A_j当中，使得A_j(j＝1,2,3,4)中至多只有一条非零共振峰。j的值即为

经序列调整后的位置排序；Among them, [x] represents the largest integer not greater than x, and x refers to the operation result of the operation expression in the operation symbol []. Then, set the sets A ₁ , A ₂ , A ₃ , and A ₄ to be empty, and set the

Put them into the corresponding set A _j , so that there is at most one non-zero formant in A _j (j=1, 2, 3, 4). The value of j is

Position sorting after sequence adjustment;

进行S₀中对应的字韵母共振峰

进行同一操作以保持同一字韵母的共振峰数量和实际位置一致；(3) Correspondingly, for S _tm or S _pn or S _rr letter final formants

Carry out the corresponding letter-final formant in S ₀

Carry out the same operation to keep the same number of formants and actual positions of the finals of the same letter;

进行上述(1)-(3) 步操作，最终获得全部共振峰序列调整后的数据对

的集合 W′_h。(4) For each homogeneous final formant data pair in W _h

Perform the above (1)-(3) steps, and finally obtain the adjusted data pairs of all formant sequences

The set W′ _h of .

Through the voice parameter preprocessing method of original voice and electronic camouflage voice, the electronic camouflage voice under different camouflage degrees is converted into a standard format with a sampling frequency of 8000Hz, 16-bit, and monophonic, and imported into the forensic identification intelligent voice workstation. The audio of the multi-person dialogue is separated, and then the wideband spectrogram of each word in each audio is collected, and the long-term average LPC (linear predictive coding, LPC) analysis method is used to generate each word. The center frequency, bandwidth and sound intensity of the formant of the pronunciation part of the final vowel. Calculate the difference between the center parameters of each formant of the word corresponding to the original voice and the electronic camouflage voice of each degree of camouflage, and record it as the initial restoration feature h ₀ .

The reason why the present invention utilizes the multi-layer enlarged causal convolutional neural network for machine learning is as follows: (1) Convolution Neural Network (CNN for short) was first proposed by Hubel and Wiesel, and is a feedforward neural network. Usually, each neuron of CNN is composed of a feature extraction layer responsible for extracting local features of the previous neuron and a feature mapping layer composed of multiple feature mapping planes required in the neuron calculation process; (2) Let the convolution input be an m×n matrix A, and the convolution kernel (Kernel, also known as Filter) is a p×q matrix B:

The essence of convolution is Weighted Stacking, and the new matrix formed after its operation is the output (Feature Map) generated by the convolution, denoted as C. The value of any element in C can be calculated as follows:

(3) Among them, 1≤x≤m-p+1, 1≤y≤n-q+1. It can be seen that the output dimension O of the convolution has the following relationship with the input dimension I and the convolution kernel dimension K:

O=I-K+1

Generally, the data dimension of the output C of the convolution calculation is smaller than the data dimension of the input A. To keep the dimensions consistent, it can be achieved by padding;

(4) There are two convolution methods, causal convolution and enlarged convolution, in the expanded causal convolution (DC-CNN). Causal convolution is mostly used for data with a certain order, and has a good modeling effect for long serialized data processing. Dilated convolution is a sparse convolution kernel, which increases the range of the receptive field by ignoring part of the input data, that is, adding zeros to the original convolution kernel according to certain rules to generate an "expanded" convolution kernel. As shown in Figure 2, the gray rectangle represents the receptive field under this expansion factor, and the dots represent the actual convolution kernel. Obviously, as the expansion factor increases, the receptive field expands exponentially;

(5) If only causal convolution is simply used, a large number of neural network layers or a large convolution kernel are required to obtain better training results. However, too deep neural network and too large convolution will not only greatly reduce the operation efficiency, but also prone to the phenomenon that the model training is difficult to converge or degenerate. The combination of enlarged convolution and causal convolution can make the model expand the receptive field without increasing the number of neural network layers and the size of the convolution kernel, so that the model has excellent performance in processing serialized signal data. The training effect is shown in the figure 3 shown.

The function and principle of the control reduction sub-network designed by the present invention are as follows: In order to ensure that the reduction factor h and the input speech signal keep the same dimension, a reduction control sub-network is established in the model. The restoration control sub-network first reduces the dimension of the initial restoration feature h ₀ of the input model system through 1×1 convolution and fuses the features of different channels, then uses the tanh function to achieve nonlinear excitation, and finally upsamples the data through deconvolution. Consistent with the sampling rate of the input speech signal, the reduction factor h is obtained.

与转置门控卷积滤波器

处理，并与卷积后的电子伪装语音按如下方式整合：The function of the reduction factor h of the gate activation function in the present invention is: in the modeling of the speech signal, the effect of the nonlinear model is better than that of the general linear model. In order to make the DC-CNN electronic camouflage speech restoration model better To solve the problem of speech generation, Gated Activation Units (GAU for short) are used to enable the neural network to have hierarchical nonlinear mapping learning capabilities. Transpose filtering convolution filter on the reduction factor h generated via the reduction control sub-network

Gated convolution filter with transpose

processed and integrated with the convoluted electronically camouflaged speech as follows:

的多维联合变量分布可表示为：The basic structure of the present invention is shown in Figure 4, the speech signal x _t is predicted and restored by the input speech signal before time t, and x _t only depends on x ₁ , x ₂ ,..., x _t-1 and the restoration factor h . The reduction factor h is converted from the reduction initial feature h ₀ via the reduction control sub-network. A sequence of speech signals over a period of time

The multidimensional joint variable distribution of can be expressed as:

In order to realize the generation of the restored speech sequence according to the above conditional probability, in the neural network main body based on the DC-CNN electronic camouflage speech restoration model, a multi-layer enlarged causal convolution block stacking and gate activation function is used to model.

通过跃层连接恒等映射至输出，并叠加于卷积后的残差输出，经优化计算即可得到期望输出。The present invention uses a layered residual structure to promote model convergence and transfer the gradient to a deeper level, so as to alleviate the performance degradation caused by the deepening of the neural network. In a residual block structure, the input of this layer of neural network

The identity is mapped to the output through the jump layer connection, and superimposed on the residual output after the convolution, and the desired output can be obtained after optimization and calculation.

The output layer of the present invention adopts the Softmax function to discretize and classify the data after the synthetic operation, and the steps are as follows:

，则其函数表达式为：(1) The input of the Softmax function is an N-dimensional real vector, set as

, then its function expression is:

(2) In its essence, the Softmax function can map an N-dimensional arbitrary real number vector to an N-dimensional vector with the values of each element in (0, 1) to achieve vector normalization:

and a′ ₁ ,a′ ₂ ,…,a′ _N ∈(0,1)

中元素a′₁,a′₂,…,a′_N分别代表了原向量

中a₁,a₂,…,a_N的分类输出概率。故对于给定的N维

可知其每个分类的概率为：(3) Mapping generated

The elements a' ₁ , a' ₂ ,..., a' _N represent the original vector respectively

The classification output probabilities of a ₁ , a ₂ ,...,a _N in . So for a given N dimension

It can be known that the probability of each classification is:

(4) In order to reduce the computational complexity of the model system, the output data volume is reduced to 2 ⁸ through μ-law companding transformation, and the prediction efficiency of the model is improved.

To sum up, a voice restoration method based on a convolutional neural network proposed by the present invention includes: step S1, collecting electronic camouflage voice, and extracting acoustic parameters in the electronic camouflage voice; step S2, using a preprocessing model to analyze the acoustic The parameters are preprocessed to convert the acoustic parameters into a standard sequence with preset dimensions; Step S3, the restoration model is used to restore the electronic camouflage voice to a restored voice sequence, and the restoration model completes the restoration of the standard voice sequence according to the standard sequence; wherein , in step S2, the preprocessing process includes formant breakage cleaning, formant merging optimization and formant sequence adjustment; convolutional neural network is used for the first time to restore electronic camouflage voice, which has a high enough restoration effect and can meet high requirements The electronic camouflage voice restoration scene.

Typical examples of specific structures of specific embodiments are given through the description and drawings, and other transformations may be made based on the spirit of the present invention. Although the above-described invention provides existing preferred embodiments, these are not intended to be limiting.

Various changes and modifications will no doubt become apparent to those skilled in the art upon reading the above description. Therefore, the appended claims should be construed to cover all changes and modifications of the true intent and scope of this invention. Any and all equivalent ranges and content within the scope of the claims should still be considered to be within the intent and scope of the invention.

Claims (11)

1. a speech restoration method based on convolutional neural network, is characterized in that, comprises preset a preprocessing model and a restoration model; The reduction model includes a convolutional neural network and an initial reduction factor, and the reduction model trains the initial reduction factor through the convolutional neural network to generate a reduction for controlling the reduction process of the reduction model. factor; The convolutional neural network includes a connected expanded causal convolution layer and a sub-control layer, and the sub-control layer is used to convert the initial reduction factor into a sequence of preset dimensions; The voice restoration method also includes: Step S1, collecting an electronic camouflage voice, and simultaneously extracting acoustic parameters in the electronic camouflage voice; Step S2, using the preprocessing model to preprocess the acoustic parameters to convert the acoustic parameters into a standard sequence with the preset dimension; Step S3, using the restoration model to restore the electronic camouflage voice to a restored voice sequence, and the restoration model completes the restoration of the restored voice sequence according to the standard sequence; Wherein, in the step S2, the preprocessing process includes formant breakage cleaning, formant merging optimization, and formant sequence adjustment. 2 . The speech restoration method according to claim 1 , wherein a gate activation unit is used in the expanded causal convolution layer to perform a nonlinear transformation process on the input from the sub-control layer. 3 . 3. speech restoration method according to claim 2, is characterized in that, described gate activation unit adopts following function to carry out described nonlinear conversion process:

In the formula,

is the input electronic camouflage speech sequence, * represents the convolution operation, σ() represents the Sigmoid function, W _f,k represents the learned convolution filter index,

represents the filtered convolution filter,

represents a gated convolution filter,

represents the gate activation function, h represents the reduction factor, k represents the number of layers, f represents the filter coefficient, and g represents the gating coefficient. 4. The speech restoration method according to claim 1, wherein the convolutional neural network further comprises a jump layer residual structure connecting the output of the expanded causal convolution layer; The jump layer residual structure is used to identically map the input data before any pre-convolutional layer to the post-convolutional layer separated by a preset number of layers, and superimpose it on the residual calculated by the post-convolutional layer. output later. 5 . The voice restoration method according to claim 1 , further comprising: step S4 , classifying and outputting the data of the restored voice sequence. 6 . 6. The voice restoration method according to claim 5, wherein the step S4 specifically comprises: Step S41, using Softmax function to discretize and classify the restored data, so as to realize the normalization of the data sequence of the restored speech sequence; Step S42, performing μ-law companding transformation on the restored speech sequence after the normalization of the data sequence, so as to reduce the amount of output operation. 7. speech restoration method according to claim 6, is characterized in that, described Softmax function is:

In the formula,

is the input restored speech sequence, and N is the vector dimension. 8. speech restoration method according to claim 6 is characterized in that, described μ-law companding conversion adopts following function to complete:

and μ=255 In the formula,

is the restored speech sequence after normalization of the input data sequence. 9. voice restoration method according to claim 1, is characterized in that, described formant breakage cleaning comprises: Step A1, extracts the formant data pair of the non-homogeneous vowels in the described electronic camouflage voice, and confirms the described formant data pair that has broken after the voiceprint comparison; Step A2, according to the size of the number of non-zero formants, the data in the pair of formant data are distinguished and marked; Step A3, performing cross-subtraction calculation on the corresponding formant center frequency parameter in the differentiated data to form an absolute value matrix about the difference; Step A4, according to the absolute value matrix, take the formant data pair corresponding to the matrix with the minimum sum of the elements of different rows and different columns; In step A5, the steps A1 to A4 are performed on all formant data pairs of non-homogeneous finals to form a set of the formant data pairs after breakage and cleaning. 10. speech restoration method according to claim 1, is characterized in that, described formant merging optimization comprises: Step B1, extracts the formant data pair of the non-homogeneous final vowel in the described electronic camouflage voice according to a preset formant extraction rule; Step B2, according to the size of the number of non-zero formants, the data in the pair of formant data is distinguished and marked; Step B3, make a difference between the formant center frequency parameter in the data with the small number of non-zero formants and the adjacent formant center frequency parameter, and extract a group of the described formants with the smallest absolute value of the difference. The formant center frequency parameter, denoted as (f _v,1 ,f _v+1,1 ), and do the following transformation:

Step B4, performing the steps B1 to B3 on all formant data pairs of non-homogeneous finals to form a set of the formant data pairs after combining and optimizing. 11. The speech restoration method according to claim 1, wherein the formant sequence adjustment comprises: Step C1, extracting the formant center frequency parameter of the non-homogeneous finals in the electronic camouflage voice is divided into different sub-sets A ₁ , A ₂ , A ₃ , A ₄ according to the numerical range, and judges any sub-set A _j ( The number of elements in j=1,2,3,4); When there is only 1 or 0 elements in any subset A _j (j=1,2,3,4), if A _j =A _j+1 ≠φ, then keep the j that minimizes |ji| The corresponding A _j , and A _j+1 is set as the empty set, and then the formants of the letters and finals of S _tm or S _pn or S _rr

Carry out the corresponding letter-final formant in S ₀

Perform assimilation operation to keep the same number of formants and actual positions of finals of the same letter; Otherwise, let M ₁₂ =A ₁ ∩A ₂ , M ₂₃ =A ₂ ∩A ₃ , M ₃₄ =A ₃ ∩A ₄ , where M _j,j+1 (j=1,2,3) means that the frequency is at the center frequency Resonant peaks appearing in both A _j and A _j+1 while distributing overlapping regions

At this time, there are no more than 2 resonance peaks located in the overlapping area of the same center frequency distribution, that is, card(M _j,j+1 )≤2, q=1,2,3, denoted B ₁ =A ₁ -M ₁₂ , B ₂ =A ₂ -M ₁₂ -M ₂₃ , B ₃ =A ₃ -M ₂₃ -M ₃₄ , B ₄ =A ₄ -M ₃₄ ; let

in

remember

is the formant with the largest center frequency in the set M _j-1,j , denoted

is the formant with the smallest center frequency in the set M _j,j+1 , then the following calculation is performed for each j=1,2,3,4:

In the formula, [x] represents the largest integer not greater than x, and x refers to the operation result of the operation expression in the operation symbol []; then, set the sets A ₁ , A ₂ , A ₃ , and A ₄ to be empty, and set the

Put it into the corresponding set A _j , so that there is at most one non-zero formant in A _j (j=1, 2, 3, 4); Step C2, performing the step C1 on all formant data pairs of non-homogeneous finals to form a set of the formant data pairs after sequence adjustment.

Images