CN114220419A - A voice evaluation method, device, medium and equipment - Google Patents

Abstract

The present application discloses a speech evaluation method, the method includes: acquiring speech to be evaluated and reference text, extracting acoustic features from the speech, inputting the acoustic features into K sub-acoustic models of the acoustic model, and obtaining comprehensive state posterior probability; Synthesizing the state posterior probability and the segmentation network constructed by the reference text, the phonemes of the speech are aligned with the reference text, and the scoring feature is obtained based on the alignment result; the scoring feature is input into the scoring model to regress the score of the voice. Since the K sub-acoustic models are obtained by training their respective in-set and out-of-set data, each sub-acoustic model has different out-of-set data, so the acoustic model can distinguish the out-of-set data, thereby improving the scoring accuracy of abnormal speech, solving the problem of It solves the problem of unstable abnormal speech score, and has high reliability and usability.

Description

A voice evaluation method, device, medium and equipment

technical field

The present application relates to the technical field of speech processing, and in particular, to a speech evaluation method, apparatus, storage medium and device.

Background technique

With the deepening of education reform and the improvement of language ability requirements of enterprises and other institutions, more and more people are committed to improving their oral expression ability, and try to obtain corresponding certificates by participating in oral examinations. Among them, the reading questions are the most common type of questions in the oral exam, so the accuracy of the reading questions is very important.

In recent years, speech recognition technology has achieved remarkable results in many fields. Therefore, many researchers try to apply speech recognition to oral examinations to automatically evaluate the participants' speech, thereby reducing the workload of teachers.

At present, the scoring process of reading questions in the speaking test is mainly divided into the following stages: The first stage is based on the pre-trained acoustic model to do complex network segmentation and alignment of the reading text and the candidate's voice, and obtain the information of increased and missed reading, word and phoneme levels Information such as the time boundary and the state posterior probability of each frame; the second stage is based on the forced segmentation results, and the scoring features related to completeness, fluency and pronunciation accuracy are extracted; the third stage is based on normal speech (for example, according to text). Normal speech) and abnormal speech (such as speech not read according to text) are detected by using a binary classification model trained by scoring features; the fourth stage is to train the scoring feature regression model based on manual scores, and the machine score is regressed in the testing phase.

However, the training data of the acoustic model is usually transcribed from normal speech, and the acoustic model has not seen abnormal speech during the training process, so the classification effect of the acoustic model on abnormal speech is unknown. Therefore, the scoring features extracted based on the acoustic model are in There will be instability in the distinction, which will lead to uncertainty in the scoring results of reading questions. How to provide a voice evaluation method so as to obtain an accurate score has become a key concern of the industry.

SUMMARY OF THE INVENTION

The main purpose of the embodiments of the present application is to provide a voice evaluation method, system, computing device cluster, computer-readable storage medium, and computer program product, which can improve the scoring accuracy of abnormal voice and solve the problem of unstable abnormal voice scoring. , with high reliability and availability.

The embodiment of the present application provides a voice evaluation method, including:

Obtain the speech and reference text to be evaluated;

Acoustic features are extracted from the speech, and the acoustic features are input into K sub-acoustic models of the acoustic model to obtain comprehensive state posterior probability. The out-of-set data of each sub-acoustic model are different;

According to the comprehensive state posterior probability and the segmentation network constructed by the reference text, the phonemes of the speech are aligned with the reference text, and scoring features are obtained based on the alignment results;

The scoring features are input into a scoring model to regress the scores for the speech.

The embodiment of the present application also provides a voice evaluation system, including:

The acquisition module is used to acquire the speech to be evaluated and the reference text;

The acoustic processing module is used for extracting acoustic features from the speech, and inputting the acoustic features into K sub-acoustic models of the acoustic model to obtain comprehensive state posterior probability, and the K sub-acoustic models pass the respective data in the set and the sub-acoustic model. The out-of-set data is obtained by training, and the out-of-set data of each sub-acoustic model is different;

A post-processing module, configured to align the phonemes of the speech with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, and obtain scoring features based on the alignment results;

The evaluation module is used for inputting the scoring feature into a scoring model to regress the score of the speech.

An embodiment of the present application further provides a computing device cluster, where the computing device cluster includes at least one computing device, and the at least one computing device includes: a processor, a memory, and a system bus.

Wherein, the processor and the memory are connected through the system bus;

The memory is used to store one or more programs, the one or more programs comprising instructions that, when executed by the processor, cause the cluster of computing devices to perform any one of the above-described speech evaluation methods. Way.

Embodiments of the present application further provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computing device cluster, the computing device cluster is made to execute the above method. any implementation.

Embodiments of the present application further provide a computer program product, which, when running on a computing device cluster, enables the computing device cluster to execute any one of the above voice evaluation methods.

A voice evaluation method provided by an embodiment of the present application. Specifically, the speech evaluation system obtains the speech to be evaluated and the reference text (for example, reading text), extracts acoustic features from the speech, inputs the acoustic features into the K sub-acoustic models of the acoustic model, and obtains the comprehensive state posterior probability , according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, align the phonemes of the speech with the reference text, obtain scoring features based on the alignment results, and input the scoring features into the scoring model, to return the score for the speech.

This method draws on the idea of K-fold cross-validation to obtain the in-set data and out-set data of each of the K sub-acoustic models, and the out-of-set data of each sub-acoustic model is different. By training the sub-acoustic model with the respective in-set data and out-of-set data, the acoustic model can be differentiated from the out-of-set data, thereby improving the scoring accuracy of abnormal speech, solving the problem of unstable score of abnormal speech, and has high performance. reliability and availability.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are For some embodiments of the present application, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a system architecture diagram of a voice evaluation method provided by an embodiment of the present application;

2 is a flowchart of a voice evaluation method provided by an embodiment of the present application;

3 is a schematic flowchart of a training acoustic model provided by an embodiment of the present application;

4 is a schematic structural diagram of a voice evaluation system provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a computing device cluster according to an embodiment of the present application.

Detailed ways

The terms "first" and "second" in the embodiments of the present application are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature defined as "first", "second" may expressly or implicitly include one or more of that feature.

First, some technical terms involved in the embodiments of this application are introduced.

Speech recognition is also known as Automatic Speech Recognition (ASR), computer speech recognition or speech-to-text. Speech recognition is used to process speech (eg, human speech) into written text.

Speech recognition has achieved remarkable results in many fields. Therefore, many researchers try to apply speech recognition to oral examinations to automatically evaluate the participants' speech, thereby reducing the workload of teachers. Similarly, speech recognition can also be applied to oral language training to evaluate the speech during training, thereby assisting oral language learning.

Reading questions are included in the speaking test or training. A reading question is a question type that indicates reading aloud according to the text. At present, the scoring process of reading questions is mainly divided into the following stages: The first stage is based on the pre-trained acoustic model to do complex network segmentation and alignment of the reading text and the candidate's voice, and obtain the information of increased and missed reading, word and phoneme-level time boundaries, and The state posterior probability of each frame and other information; the second stage is based on the forced segmentation results, and the scoring features related to completeness, fluency and pronunciation accuracy are extracted; the third stage is based on normal speech (such as normal reading according to text) Speech) and abnormal speech (such as speech that is not read according to text) are detected by using a binary classification model trained by scoring features; the fourth stage is to train a regression model of scoring features based on manual scores, and the machine score is returned in the testing phase.

However, the training data of the acoustic model is usually transcribed from normal speech, and the acoustic model has not seen abnormal speech during the training process, so the classification effect of the acoustic model on abnormal speech is unknown. Therefore, the scoring features extracted based on the acoustic model are in There will be instability in the distinction, which will lead to uncertainty in the scoring results of reading questions.

In order to solve the above problem, an embodiment of the present application provides a voice evaluation method. The method may be performed by a speech evaluation system. In some embodiments, the speech evaluation system may be a software system, and a computing device or a cluster of computing devices executes the speech evaluation method by running the program code of the software system. In other embodiments, the speech evaluation system may also be a hardware system for speech evaluation. The embodiment of the present application uses a voice evaluation system as a software system for illustration.

Specifically, the speech evaluation system obtains the speech to be evaluated and the reference text (for example, reading text), extracts acoustic features from the speech, inputs the acoustic features into the K sub-acoustic models of the acoustic model, and obtains the comprehensive state posterior probability , according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, align the phonemes of the speech with the reference text, obtain scoring features based on the alignment results, and input the scoring features into the scoring model, to return the score for the speech.

This method draws on the idea of K-fold cross-validation, divides the state categories of the output layer of the acoustic model into K groups, and obtains the respective in-set and out-set data of the K sub-acoustic models based on the K-group state categories, and each sub-acoustic model The out-of-set data is different. Using the respective in-set data and out-of-set data to train the sub-acoustic model can make the acoustic model distinguishable from the out-of-set data, thereby improving the scoring accuracy of abnormal speech, solving the problem of unstable abnormal speech scoring, and having high reliability sex and availability.

In order to make the technical solutions of the present application clearer and easier to understand, the following describes the application scenarios of the present application with reference to the accompanying drawings.

Referring to the schematic diagram of the application scenario of the speech evaluation method shown in FIG. 1 , as shown in FIG. 1 , a communication connection is established between the server 10 and the terminal 20 . The communication connection may, for example, implement a wired network connection or a wireless wireless network connection.

A voice evaluation system (not shown in FIG. 1 ) is deployed in the server 10 to provide a voice evaluation service. The terminal 20 has a recording function for presenting the test paper of the oral test to the user, and collecting the user's answering voice for the test paper. Wherein, the oral test may be an English oral test or a Putonghua oral test, and the present embodiment uses the English oral test as an example for description. Specifically, the test paper includes reading questions, and the terminal 20 may present the reading text corresponding to the reading question to the user, and then collect the user's voice when reading aloud according to the reading text. The terminal 20 may use the collected voice as the voice to be evaluated, and the read-aloud text as the reference text, and send it to the server 10, and the server 10 will evaluate through the voice evaluation system.

Specifically, after acquiring the speech to be evaluated and the reference text, the speech evaluation system in the server 10 extracts the acoustic features from the speech, inputs the acoustic features into the K sub-acoustic models of the acoustic model, obtains the comprehensive state posterior probability, and then according to the The integrated state posterior probability and the segmentation network constructed by the reference text, align the phonemes of the voice with the reference text, obtain scoring features based on the alignment results, and then input the scoring features into a scoring model to regression out a score for the speech.

Further, the server 10 can also return the score of the voice to the terminal 20, so that the terminal 20 can present the score of the voice to the user.

It should be noted that FIG. 1 only illustrates that the voice evaluation system is deployed on the server. In other possible implementations of the embodiments of the present application, the voice evaluation system may also be deployed on a terminal, or a system formed by being deployed on multiple computing devices. A computing device cluster, which is not limited in this embodiment.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

Referring to the flowchart of the speech evaluation method shown in Figure 2, the method includes the following steps:

S202: The speech evaluation system acquires the speech to be evaluated and the reference text.

The voice to be evaluated refers to the voice of a user (such as a test taker of an oral language test, an oral language student) reading aloud according to the reference text. The read-aloud text is a preset text, for example, the text in the oral examination paper. The reference text may be English text or Chinese text, and correspondingly, the speech to be evaluated may be English speech or Mandarin speech, which is not limited in this embodiment.

S204: The speech evaluation system extracts acoustic features from the speech, and inputs the acoustic features into the K sub-acoustic models of the acoustic model to obtain comprehensive state posterior probability.

The acoustic features can be Linear Predictive Cepstral Coding (LPCC) features, Filter Bank (fBank) features, Mel-frequency CeptraIC Coefficients (MFCC) features or Perceptual Linear Prediction (Perceptual Linear Prediction) features. Any one or more of Predictive, PLP) features.

The speech evaluation system can extract the corresponding features from the speech through the acoustic feature extraction algorithm. The following is an example of extracting MFCC features. Specifically, the speech evaluation system can preprocess the speech, such as framing, pre-emphasis and windowing, so that multiple speech frames can be obtained, and then the speech evaluation system can convert the multiple speech frames from time domain to frequency domain. Domain, for example, a speech evaluation system may perform a fast Fourier transform (FFT) on multiple speech frames. The feature obtained through FFT is a complex matrix. The complex number matrix represents an energy spectrum. Since the phase spectrum in the energy spectrum contains very little information, the speech evaluation system can discard the phase spectrum and retain the amplitude spectrum by squaring or finding the absolute value of the complex numbers in the complex number matrix. The speech evaluation system can perform Mel filtering on the magnitude spectrum to convert linear frequencies into non-linearly distributed Mel frequencies. The features after Mel filtering are also called fBank features.

Since the human ear's perception of sound increases in pairs, the speech evaluation system can perform a logarithmic operation on the fBank feature to simulate the human ear's perception. Further, the speech evaluation system can also perform discrete cosine transform (Discrete Cosine Transform, DCT) after logarithmic operation to eliminate the correlation between different orders, map the signal into a low-dimensional space, and obtain MFCC features. .

The above-mentioned MFCC features are static, and the speech evaluation system can also perform differential operations on the static MFCC features to obtain dynamic MFCC features. The speech evaluation system can combine static MFCC features and dynamic MFCC features to improve the recognition performance.

The state category of the output layer of the acoustic model refers to the hidden state of tri-phone modeling based on Hidden Markov Model (HMM) in speech recognition. For example, a state class may include the hidden state "cup" modeled by the phonemes /k/, /uh/, /p/. When the user utters the speech "would u like a cup of tea", the state category of one speech frame of the speech may be the above-mentioned "cup".

The state categories of the output layer of the acoustic model can be divided into K groups. Among them, the state categories can be equally divided into K groups. When grouping, grouping can be done in a random manner or according to phoneme types. For example, in English, groups can be grouped according to vowels and consonants, and in Mandarin, groups can be grouped according to initials and finals.

The acoustic model has K sub-acoustic models. The K acoustic sub-models are trained by their respective in-set and out-of-set data. Wherein, the out-of-set data of each sub-acoustic model is the data whose state category in the dataset belongs to one of the K groups. The out-of-set data is different for each sub-acoustic model. The intra-set data of each sub-acoustic model is the data in the data set whose state category belongs to the remaining K-1 groups in the K groups. Among them, the label of the data in the set is the vector of the state category, such as a one-hot vector, and the data outside the set has no corresponding state category, and its label is set to the uniform distribution of the state category of the corresponding group.

Based on this, after the speech evaluation system inputs the acoustic features into the K sub-acoustic models of the acoustic model, for each state type, K-1 acoustic sub-models in the K acoustic sub-models can output the posterior probability of the state type . The speech evaluation system can obtain the comprehensive state posterior probability according to K-1 posterior probability. For example, the speech evaluation system can determine the average value of K-1 posterior probabilities as the comprehensive state posterior probability.

S206: The speech evaluation system aligns the phonemes of the speech with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, and obtains scoring features based on the alignment result.

The segmentation network is used for Force Alignment of speech and text. The segmentation network can segment the reference text, so as to obtain various possibilities of the user's reading path for the reference text, such as the normal reading path, the word or sentence addition, omission, stress or sentence readback path. Usually, the segmentation network can be obtained by constructing the reference text based on the set rules.

Forced segmentation and alignment refers to the use of the Viterbi algorithm to find the optimal reading path based on the state posterior probability output by the segmentation network and the acoustic model, so as to realize the speech and reference text according to the optimal reading path. Align.

Among them, the Viterbi algorithm is a dynamic programming algorithm. The algorithm is used to find a Viterbi path that is most likely to generate an observation event sequence (or called an observation event sequence), and the Viterbi path is characterized by a hidden state sequence (or called a hidden state sequence). Specific to the speech evaluation scenario, speech (such as phoneme sequence in speech) can be regarded as the sequence of observation events, and the sequence of characters in the reference text can be regarded as the sequence of hidden states. Therefore, the speech evaluation system can apply the Viterbi algorithm to the speech to find the most probable sequence of hidden states, so as to achieve the alignment of speech and reference text.

Specifically, the speech evaluation system can determine the target reading path through the Viterbi algorithm according to the comprehensive state posterior probability and the segmentation network constructed by the reference text. For example, the target reading path can be the above optimal reading path, and then the speech The evaluation system may align the phonemes of the speech with the reference text according to the target reading path, and obtain time boundary information of the phonemes.

When evaluating speech, it can be evaluated from different dimensions. Correspondingly, the scoring features may include one or more of the following: goodness of pronunciation (GOP), the proportion of words that are added or omitted, and the average phoneme pronunciation duration.

In this embodiment, the speech evaluation system can further calculate scoring features such as good pronunciation GOP, word increase and omission reading ratio, average phoneme pronunciation duration, etc. based on the time boundary information of phonemes, so as to measure the user's pronunciation accuracy, completeness and accuracy. Fluency.

GOP is calculated based on the time boundary information of the phoneme and the posterior probability of the comprehensive state, and is used to evaluate the correctness of the user's pronunciation. The GOP calculation formula is as follows:

Among them, p is a phoneme, o is an observation vector (also called observation vector), and each observation vector is generated by a state sequence with a corresponding probability density distribution. Q is the set of all phonemes, p(o|p) is the likelihood probability of the phoneme p, S _i is the ith possible state sequence, the state sequence is found by the Viterbi algorithm, s _{i, t} are the tri- phone status.

The speech evaluation system may further determine at least one of the added word and the missed word based on the time boundary information of the phonemes. For example, the speech evaluation system may determine the word read-increase ratio based on the number of added words and the total number of words, and determine the word miss-read ratio based on the number of missed words and the total number of words. In addition, the speech evaluation system can also determine the pronunciation duration of each phoneme by the user based on the time boundary information of the above factors, and based on the pronunciation duration, the average phoneme pronunciation duration can be determined.

In this embodiment, the comprehensive state posterior probability is distinguishable, and the scoring features such as GOP determined based on the comprehensive state posterior probability are also distinguishable, and the scoring feature is used for speech evaluation with high accuracy.

S208: The speech evaluation system inputs the scoring feature into a scoring model to regress the speech score.

A scoring model is a model that evaluates speech. The scoring model takes the scoring features of speech as input, for example, the GOP of speech, the ratio of word augmentation and omission reading, and the average phoneme pronunciation time as input, and the speech score as output. The scoring model is usually a regression model, such as any one of a linear regression model, a decision tree regression model, a Gaussian regression model, or a neural network regression model. Correspondingly, the scoring model can output the score of speech through regression, which is also called machine score.

The scoring model can be obtained by training the initialized regression model based on the scoring features extracted from the data set and the corresponding manual scores (specifically, manually labeled scores). The speech evaluation system can use the data set to pre-train the scoring model, and then use the scoring model to perform inference in the inference stage, so as to evaluate the speech.

Based on the above content description, an embodiment of the present application provides a voice evaluation method. This method draws on the idea of K-fold cross-validation, and divides the state categories of the output layer of the acoustic model into K groups. The K sub-acoustic models are obtained by training their respective in-set and out-of-set data. The out-of-set data of each sub-acoustic model is The state category in the dataset belongs to the data of one of the K groups, and the out-of-set data is different for each sub-acoustic model. The intra-set data of each sub-acoustic model is the data in the data set whose state category belongs to the remaining K-1 groups in the K groups. In this way, the acoustic model can distinguish the data outside the set, thereby improving the scoring accuracy of abnormal speech, solving the problem of unstable scoring of abnormal speech, and having high reliability and usability.

The speech evaluation method provided by the embodiment of the present application relies on an acoustic model, and the acoustic model can be obtained by pre-training. Next, the training process of the acoustic model is introduced.

Referring to the schematic flowchart of training an acoustic model shown in FIG. 3, the method includes:

S302: The speech evaluation system acquires a data set.

Each sample data in the dataset has K labels. Among them, K-1 labels are vectors of state categories corresponding to the sample data, such as one-hot vectors, and the remaining 1 label is a uniform distribution of state categories.

Specifically, the user can divide the state categories of the output layer of the acoustic model into K groups. Assuming that the number of state categories in the output layer is N, the user can equally divide the state categories into K groups, each group including N/K state categories. When users are grouped, they can use a random method or group them according to phonemes. Then, referring to the idea of K-fold cross-validation, one group of state categories is removed in turn, and the remaining K-1 groups of state categories are formed into a state category set, and finally K state category sets can be constructed, and each state category is included in K - 1 state category set.

Then, based on each state category set, the user can trigger the operation of generating labels for the sample data in the dataset. Taking one of the state category sets as an example, the sample data in the dataset can be divided into in-set data and out-of-set data. In-set data refers to sample data (such as speech frames) whose state category is in the state category set, and out-set data refers to sample data whose state category is not in the state category set. For the data in the set, the state category of the data in the set is determined, and the one-hot vector of the state category is used as the label. For the data outside the set, since the state category cannot be determined, the uniform distribution of the state categories in the current group can be used as the label.

It should be noted that the data set in this embodiment may include not only sample data generated by normal speech, but also sample data generated by abnormal speech. Abnormal speech includes speech that is not read according to the reference text and/or speech with abnormal sound quality. Taking the English speaking test scenario as an example, the speech that is not read out according to the reference text may include Mandarin, dialect, Chinese English, typing on a keyboard, blowing a microphone, singing or coughing, and so on.

S304: The speech evaluation system trains K acoustic sub-models in the acoustic model according to the data set.

Wherein, each sample data in the dataset has K labels, that is, the dataset includes K groups of labels. Based on this, the speech evaluation system can train an acoustic sub-model according to a set of labels of the dataset, and each set of labels is used to train a different acoustic sub-model.

In order to ensure that the final acoustic model is distinguishable between in-set data and out-of-set data, two training objectives (such as two loss functions) are used to update the parameters of the acoustic sub-model during the training process. The following is an example of the training of one of the acoustic sub-models.

在一些实施例中，也可以采用Focal Loss函数计算损失，该损失记作

如此，可以缓解模型训练过程中各类样本数量比例不均衡的情况，让模型更加关注难分类样本。其中，

和

计算公式如下所示：For the data in the set, the label is a one-hot vector, so the standard cross entropy (CrossEntropy) function can be used to calculate the loss, where the loss is also called the standard cross entropy loss, denoted as

In some embodiments, the loss can also be calculated using the Focal Loss function, and the loss is recorded as

In this way, the unbalanced proportion of various samples in the model training process can be alleviated, and the model can pay more attention to the difficult-to-classify samples. in,

and

The calculation formula is as follows:

是指x_i帧数据在所述类别标签为c的模型预测结果，α_c是类别c的权重，γ为衰减系数。Among them, X is the data in the set used to train the acoustic sub-model, Y is the label of the data in the set, f(x _i ) refers to the model prediction result of the _xi frame data,

Refers to the model prediction result of the x _i frame data in the category label c, α _c is the weight of category c, and γ is the attenuation coefficient.

也可以使用JS散度(Jensen-Shannon Divergence)计算损失

两者之间的差异在于JS散度解决了KL散度不对称的问题。其中，

和

计算公式如下所示：For out-of-set data, the label is a uniform distribution of state categories, and the loss can be calculated using KL divergence (Kullback-Leibler Divergence)

Loss can also be calculated using JS Divergence (Jensen-Shannon Divergence)

The difference between the two is that JS divergence solves the problem of KL divergence asymmetry. in,

and

The calculation formula is as follows:

为用于训练该声学子模型的集外数据，μ为集外数据的标签，f(x_i)是指x_i帧数据的模型预测结果。in,

is the out-of-set data used to train the acoustic sub-model, μ is the label of the out-of-set data, and f( _xi ) refers to the model prediction result of the x _i frame data.

According to the above losses, the loss of the acoustic sub-model can be determined as follows:

Among them, λ is a hyperparameter with a value in the [0,1] interval, which can usually be determined by fine-tuning based on the validation set in the dataset.

The method solves the problem of instability of abnormal response speech scores through the K-fold acoustic model training framework, and does not need to introduce a separate abnormality detection module. Specifically, first, according to the idea of K-fold cross-validation, the state categories of the output layer of the acoustic model are divided into K state category sets; then the sample data in the dataset is divided into in-set data and out-of-set data according to each state category set. Labels are represented by a one-hot vector and a uniform distribution of categories respectively; then K independent sub-acoustic models are trained respectively. When training each sub-acoustic model, the data in the set and the data outside the set use CE Loss and KL Loss. Objective function training; finally, the posterior probability of K-1 models is averaged when calculating the posterior probability of frame-level state, and further forced segmentation and extraction of scoring features are performed. In this case, by introducing the K-fold acoustic model training framework when training the acoustic model, the model can distinguish the data outside the set, thereby improving the scoring accuracy of the abnormal speech data outside the set.

Based on the voice evaluation method provided by the embodiment of the present application, the embodiment of the present application further provides a voice evaluation system. The speech evaluation system is used to execute the above-mentioned speech evaluation method. Next, the speech evaluation system of the embodiment of the present application will be introduced with reference to the accompanying drawings.

Referring to the schematic structural diagram of the speech evaluation system shown in FIG. 4 , as shown in FIG. 4 , the speech evaluation system 400 includes:

an acquisition module 402, configured to acquire the speech to be evaluated and the reference text;

The acoustic processing module 404 is configured to extract acoustic features from the speech, input the acoustic features into the K sub-acoustic models of the acoustic model, and obtain the comprehensive state posterior probability. The out-of-set data training is obtained, and the out-of-set data of each sub-acoustic model is different;

A post-processing module 406, configured to align the phonemes of the speech with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, and obtain scoring features based on the alignment result;

The evaluation module 408 is configured to input the scoring feature into a scoring model to regress the score of the speech.

In some possible implementations, the system 400 further includes:

The training module is used to divide the state categories of the output layer of the acoustic model into K groups; the data whose state category belongs to the first group to the data whose state category belongs to the Kth group is the out-of-set data, and the state category is the remaining data. The data of the K-1 group is the data in the set, the label of the data outside the set is the uniform distribution of the state category corresponding to the group, and the label of the data in the set is the vector corresponding to the state category; through the K acoustic sub-models The in-set and out-of-set data of each acoustic sub-model in train the acoustic sub-model.

In some possible implementations, the objective function used for training the acoustic sub-model is a multi-objective function, and the multi-objective function is obtained according to a first objective function and a second objective function, and the first objective function is obtained by the The objective function when the acoustic sub-model is trained by the data in the set, and the second objective function is the objective function when the acoustic sub-model is trained by the data outside the set.

In some possible implementations, the acoustic processing module 404 is specifically used for:

Inputting the acoustic features into the K sub-acoustic models of the acoustic model to obtain K-1 posterior probabilities for each state type;

The comprehensive state posterior probability is determined according to the K-1 posterior probabilities of each state type.

In some possible implementations, the post-processing module 406 is specifically used for:

According to the comprehensive state posterior probability and the segmentation network constructed by the reference text, determine the target reading path through the Viterbi algorithm;

The phonemes of the speech are aligned with the reference text according to the target reading path, and time boundary information of the phonemes is obtained.

In some possible implementations, the scoring feature includes at least one of good pronunciation, word augmentation and omission reading ratio, and average phoneme pronunciation duration;

The post-processing module 406 is specifically used for:

According to the time boundary information of the phonemes, the pronunciation good degree, the word increase and omission reading ratio and/or the average phoneme pronunciation duration are determined.

In some possible implementations, the data set also includes sample data generated by abnormal speech.

The speech evaluation system 400 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above-mentioned and other operations and/or functions of the various modules/units of the speech evaluation system 400 are for the purpose of realizing FIG. 2 and FIG. 3 , respectively. For the sake of brevity, the corresponding processes of the respective methods in the illustrated embodiment are not repeated here.

Further, an embodiment of the present application also provides a computing device cluster, where the computing device cluster includes at least one computing device, and any computing device in the at least one computing device may be a server from a cloud environment or an edge environment , or a terminal device. The computing device cluster or computing device is specifically used to implement the functions of the speech evaluation system 400 in the embodiment shown in FIG. 4 .

FIG. 5 provides a schematic structural diagram of a computing device cluster. As shown in FIG. 5 , the computing device cluster 50 includes multiple computing devices 500 . The processor 502 , the memory 504 and the communication interface 503 communicate through the bus 501 .

The bus 501 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 5, but it does not mean that there is only one bus or one type of bus.

The processor 502 may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc. any one or more of the devices.

The communication interface 503 is used for external communication. For example, the communication interface 503 is used to obtain the speech to be evaluated and the reference file, return the score of the speech, and so on.

Memory 504 may include volatile memory, such as random access memory (RAM). The memory 504 may also include non-volatile memory (non-volatile memory) such as read-only memory (ROM), flash memory, hard disk drive (HDD) or solid state drive (SSD) ).

Computer-readable instructions are stored in the memory 504, and the processor 502 executes the computer-readable instructions to cause the computing device 500 or the computing device cluster 50 to perform the aforementioned voice evaluation method (or implement the aforementioned functions of the voice evaluation system 400).

Specifically, in the case of implementing the embodiment of the system shown in FIG. 4 , and each module of the speech evaluation system 400 described in FIG. In the case where the functions are implemented by software, the software or program codes required to perform the functions of the modules in FIG. 4 may be stored in at least one memory 504 in the computing device cluster 50 . At least one processor 502 executes program code stored in memory 504 to cause computing device 500 or computing device cluster 50 to perform the aforementioned speech evaluation method.

It should be noted that when the computing device cluster includes multiple computing devices, each computing device can independently execute the voice evaluation method, that is, each computing device can provide a complete voice evaluation service. In other embodiments, a plurality of computing devices may cooperate to perform the speech evaluation method. For example, some computing devices may perform several steps of the speech evaluation method, and other computing devices may perform other steps of the speech evaluation method. Complete voice evaluation service.

Further, an embodiment of the present application further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed on the computing device cluster 50, the computing device cluster 50 is made Any implementation method of the above-mentioned speech evaluation method is performed.

Further, an embodiment of the present application also provides a computer program product, which, when running on the computing device cluster 50 , enables the computing device cluster 50 to execute any one of the above voice evaluation methods.

From the description of the above embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in storage media, such as ROM/RAM, magnetic disks , CD, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the various embodiments or parts of the embodiments of the present application. method.

It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments may be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

It should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply those entities or operations There is no such actual relationship or order between them. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, this application is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A speech evaluation method, comprising:

acquiring a voice to be evaluated and a reference text;

extracting acoustic features from the voice, inputting the acoustic features into K sub-acoustic models of an acoustic model to obtain a comprehensive state posterior probability, wherein the K sub-acoustic models are obtained by training respective in-set data and out-set data, and the out-set data of each sub-acoustic model is different;

aligning the phoneme of the voice with the reference text according to the comprehensive state posterior probability and a segmentation network constructed by the reference text, and obtaining a scoring characteristic based on an alignment result;

inputting the scoring characteristics into a scoring model to regress the score of the voice.

2. The method of claim 1, wherein the K sub-acoustic models are trained by:

dividing the state categories of the output layer of the acoustic model into K groups;

respectively taking data of which the state class belongs to a first group to data of which the state class belongs to a K-th group as off-set data, taking data of which the state class belongs to the remaining K-1 group as in-set data, wherein labels of the off-set data are uniformly distributed in the state classes of the corresponding groups, and labels of the in-set data are vectors of the corresponding state classes;

training the acoustic submodels through the in-set data and the out-set data of each acoustic submodel in the K acoustic submodels.

3. The method of claim 2, wherein an objective function used for training the acoustic submodel is a multi-objective function, the multi-objective function is obtained according to a first objective function and a second objective function, the first objective function is an objective function when the acoustic submodel is trained by the intra-set data, and the second objective function is an objective function when the acoustic submodel is trained by the out-set data.

4. The method according to any one of claims 1 to 3, wherein the inputting the acoustic features into K sub-acoustic models of an acoustic model to obtain a comprehensive state posterior probability comprises:

inputting the acoustic features into the K sub-acoustic models of the acoustic model to obtain K-1 posterior probabilities of each state type;

and determining the comprehensive state posterior probability according to the K-1 posterior probabilities of each state type.

5. The method according to any one of claims 1 to 3, wherein aligning the phonemes of the speech with the reference text according to the integrated state posterior probabilities and a segmentation network constructed by the reference text comprises:

determining a target reading path through a Viterbi algorithm according to the comprehensive state posterior probability and a segmentation network constructed through the reference text;

and aligning the phoneme of the voice with the reference text according to the target reading path to obtain the time boundary information of the phoneme.

6. The method of claim 5, wherein the scoring characteristics comprise at least one of pronunciation goodness, word miss-add ratio, average phoneme pronunciation duration;

the obtaining scoring characteristics based on the alignment result comprises:

and determining the pronunciation goodness, the word increase and omission ratio and/or the average phoneme pronunciation duration according to the time boundary information of the phonemes.

7. The method according to any one of claims 1 to 3, wherein the data set further comprises sample data generated by abnormal speech.

8. A speech assessment system, the system comprising:

the acquisition module is used for acquiring the voice to be evaluated and the reference text;

the acoustic processing module is used for extracting acoustic features from the voice, inputting the acoustic features into K sub-acoustic models of an acoustic model to obtain comprehensive state posterior probability, wherein the K sub-acoustic models are obtained through respective in-set data and out-set data training, and the out-set data of each sub-acoustic model is different;

the post-processing module is used for aligning the phonemes of the voice with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, and obtaining scoring characteristics based on an alignment result;

and the evaluation module is used for inputting the scoring characteristics into a scoring model so as to regress the score of the voice.

9. A cluster of computing devices, the cluster of computing devices comprising at least one computing device, the at least one computing device comprising: a processor, a memory, a system bus;

the processor and the memory are connected through the system bus;

the memory to store one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the cluster of computing devices to perform the method of any of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, which when executed on a cluster of computing devices, cause the cluster of computing devices to perform the method of any of claims 1-7.

Images