CN1763843A - Pronunciation quality evaluating method for language learning machine - Google Patents

Abstract

The invention discloses a pronunciation quality evaluation method of language study machine in the computer subsidiary language study and phonetic technique domain, which is characterized by the following: extracting exercise phonetic feature; exercising standard pronunciation model; forming standard pronunciation network; detecting phonetic end; extracting evaluation phonetic feature; searching optimum path; calculating the mark of pronunciation quality. The method displays objective and stable evaluation, which constitutes imbedded English study system and mutual human-machine education and oral English self-detection.

Description

Pronunciation quality assessment method for language learning machine

technical field

The invention belongs to the field of computer-aided language learning and speech technology, and in particular relates to a pronunciation quality evaluation method realized by adopting 16-bit and above digital signal processing chips.

Background technique

In recent years, embedded language learning products have developed rapidly at home and abroad. In the early days, it was mainly a repeater. It is an analog tape recorder with a device that can digitally store a short section of speech. At present, the mainstream language learning machines on the market are the second-generation products using digital signal processing chip (DSP, Digital Signal Processing) technology. The hardware system generally includes a microcontroller (Micro Control Unit, MCU), digital signal processing chip (DSP), multimedia digital signal codec (CODEC), ROM, SRAM, flash memory (Flash Memory), universal serial bus ( USB), keyboard and liquid crystal display (Liquid Crystal Display, LCD), etc.; among them, MCU is used as the main control chip to execute system control programs such as device drivers and program scheduling, and DSP executes application algorithm programs. The application program includes basic modules such as recording, playback, and speech rate adjustment, and some products also have an mp3 module. In terms of functions, it has repeat reading, follow-up reading, follow-up reading comparison, synchronous display of text, content retrieval query and adjustable speech speed playback, etc. Most of these learning machines can download and update learning materials through the Internet. The Haojixing English learning machine of Shenzhen Haojixing Company is a typical representative of the second-generation digital English learning machine.

The key to language learning, especially oral language learning, lies in the interactive learning process, that is, teachers make timely and targeted judgments and guidance during the learning process. However, in the traditional teacher-centered language learning, this task cannot be accomplished due to the lack of teachers. However, the existing language learning machines do not have the ability to evaluate the learner's pronunciation.

Contents of the invention

The purpose of the present invention is to overcome the weak point of prior art, propose a kind of pronunciation quality evaluation method that is used for language learning machine, can realize high-performance text and speaker-independent pronunciation quality evaluation on embedded language learning machine , which has the characteristics of moderate complexity, high evaluation accuracy and good robustness. In particular, the accuracy of the evaluation of people with a Chinese accent has reached or even exceeded the current international advanced level.

The method for evaluating the pronunciation quality of a language learning machine proposed by the present invention includes extraction of speech features for training, training of standard pronunciation models, generation of standard pronunciation networks, detection of speech endpoints, extraction of speech features for evaluation, and optimal path Search, and the calculation parts of pronunciation quality score; It is characterized in that, the implementation method of each part specifically comprises the following steps:

A. Speech feature extraction for training:

(1) pre-establish a training database that includes a large amount of reading speech;

(2) carry out pre-emphasis and frame-by-frame windowing processing to the digital voice in each voice file in said training database, obtain the frame-by-frame voice with quasi-stationarity;

(3) extracting speech feature to said sub-frame speech, this speech feature is cepstral coefficient;

B. Standard pronunciation model training

(1) Utilize the said phonetic feature training of step A to obtain the standard pronunciation model based on phonemes;

(2) Carry out the self-adaptation of the accent of the Chinese population to the said standard pronunciation model as the final standard pronunciation model, and optimize the evaluation performance of the model to the Chinese population;

C. Generation of standard pronunciation network

Segment the given text into words, search the pronunciation dictionary to get the phoneme annotation, and finally use the phoneme-based standard pronunciation model to obtain a linear standard pronunciation network with the state as the node;

D. Voice endpoint detection:

(1) The analog voice signal undergoes A/D conversion to obtain digital voice;

(2) carry out pre-emphasis and frame-by-frame windowing process to said digital voice, obtain the frame-by-frame voice with quasi-stationarity;

(3) calculating the time-domain logarithmic energy to said framed speech;

(4) adopt the method for moving-average filter (moving-average filter) to obtain the feature (hereinafter referred to as the end detection feature) that is used for endpoint detection by said time-domain logarithmic energy;

(5) adopt the method that upper limit and lower limit double threshold value and finite state machine combine, carry out endpoint detection to said end detection feature, obtain the start and end endpoint of speech;

E. Speech feature extraction for evaluation

The said framed speech of step D is extracted speech feature, and the process is identical with step (3) of step A.

F. Optimal path search:

(1) carry out forcible matching with the said speech feature of step E and the said standard pronunciation network of step C, obtain all possible path information in the network;

(2) Utilize said path information, trace out the optimal path from the terminal node allowed by the network;

G. Calculation of pronunciation quality score:

(1) Utilize said optimal path information in the step F to calculate the confidence score of every frame speech feature;

(2) Utilize the confidence score of each state on the optimal path information calculation path mentioned in the step F; The confidence score of all states on the optimal path is averaged to obtain the confidence score of the whole sentence;

(3) Use the mapping function to map the confidence score of the entire sentence to the subjective evaluation score interval to obtain the final pronunciation quality score.

The cepstrum coefficients in said step A may be Mel-Frequency Cepstrum Coefficients (MFCC, Mel-Frequency Cepstrum Coefficients), which utilizes the frequency resolution characteristics of the human ear.

The standard pronunciation model in the said step B (1) is a phoneme-based Hidden Markov Model (HMM, HiddenMarkov Model), and the specific training process of this model is: adopt all speech features to initialize a Gaussian model, utilize this model Copy all the phoneme models, and use the Baum-Welth method to train the model multiple times. Continuously increasing the number of Gaussian components of each phoneme model, re-training Baum-Welth.

The realization method of carrying out the self-adaptation of the accent of the Chinese population to the standard pronunciation model in said step B (2) is: the standard pronunciation model obtained is based on maximum likelihood linear regression (Maximum Likelihood LinearRegression, MLLR) and maximum a posteriori Accent adaptation of the probability (Maximum A Posteriori, MAP) method to obtain the final standard pronunciation model.

The standard pronunciation network in step C can be a linear network with a definite start node and end node, the current node is only related to its predecessor node, and the state of the HMM is used as the node without considering the grammar.

The optimal path search method in step F uses a frame-synchronous Viterbi (Viterbi) beam search method.

In order to be able to realize the pronunciation quality evaluation method of the present invention on the limited memory resource of embedded system, step D, E, F and G all carry out in segments in time with the preset fixed frame number step size, can like this Greatly reduce the requirements for system resources, so that the embedded learning system can handle relatively long speech;

The pronunciation quality evaluation method of the invention enables the language learning machine to have an interactive function. The embedded English learning system realized by this method has achieved good performance in practical application.

The present invention has following characteristics:

(1) The present invention has the characteristics of high evaluation accuracy, good robustness, and low system resource overhead;

(2) The phoneme-based standard pronunciation model is adopted, so that the embedded learning system can easily change the content of the courseware without retraining;

(3) Taking into account the influence of the native accent on English pronunciation, the phoneme model is adapted to the accent;

(4) Real-time endpoint detection is carried out by using technologies such as moving average filtering and finite state machine, which improves the accuracy and robustness of endpoint detection for English speech;

(5) It can be used in embedded language learning systems based on DSP, and has the outstanding characteristics of small size, light weight, low power consumption and low cost;

(6) The pronunciation quality evaluation method of the present invention, combined with rich courseware forms, can change the traditional learning machine working mode and classroom teaching mode.

Description of drawings

Fig. 1 is a schematic flow chart of the overall method of the embodiment of the present invention.

Fig. 2 is the standard pronunciation model training flowchart of the embodiment of the present invention; Fig. 2 (a) represents the whole process of standard pronunciation model training, Fig. 2 (b) represents the training process of a specific hidden Markov model.

Fig. 3 is a topological structure diagram of a standard pronunciation model of an embodiment of the present invention; Fig. 3(a) shows a pause model, and Fig. 3(b) shows a phoneme and silence model.

Fig. 4 is a flow chart of HMM accent adaptation according to an embodiment of the present invention.

Fig. 5 is the topological structure schematic diagram of the standard pronunciation network of the embodiment of the present invention; Fig. 5 (a) represents the linear network structure that whole sentence takes word as node, and Fig. 5 (b) represents the linear network structure that each word takes phoneme as node .

Fig. 6 is a schematic diagram of the generation process of the standard pronunciation network according to the embodiment of the present invention.

Fig. 7 is a detailed flow chart of the pronunciation quality evaluation method realized on the embedded platform according to the embodiment of the present invention.

Detailed ways

A kind of pronunciation quality evaluation method embodiment that is used for language learning machine that the present invention proposes is described in detail as follows in conjunction with each figure:

The overall flow of the embodiment of the inventive method is divided into as shown in Figure 1, is divided into: A, the speech feature extraction that is used for training; B, the training of standard pronunciation model; Complete); D, speech endpoint detection; E, speech feature extraction for evaluation; F, optimal path search; G, calculation and output of pronunciation quality scores (the above steps are completed using an embedded platform). Examples of each step are detailed below.

A. Speech feature extraction for training:

(1) Pre-establish a training database that includes a large amount of English reading voices (the content required to be included has a certain amount of coverage for each phoneme);

(2) carry out pre-emphasis processing to the digital speech in each speech file in said training database, pre-emphasis filter is taken as H (z)=1-0.9375z ^-1 ; The speech after pre-emphasis is divided Frame windowing (using Hamming window) processing, the frame length can be 32ms, the frame shift can be 16ms, and the framed speech with quasi-stationarity can be obtained;

(3) Extract the Mel-Frequency Cepstral Coefficient (MFCC) as the speech feature for the said framed speech; the short-time frequency domain feature of the speech can accurately describe the change of the speech, and the MFCC is calculated according to the frequency resolution characteristics of human hearing A eigenvector is obtained based on Fourier spectrum analysis. The calculation method of MFCC is as follows: first, perform Fast Fourier Transformation (FFT) on the framed speech to obtain the short-term spectrum of the signal, and secondly, according to the MEL frequency standard Divide the short-time spectrum into several band-pass groups equally, and the frequency response of the band-pass is triangular, calculate the signal energy of the corresponding filter group again, and finally calculate the corresponding cepstral coefficient through discrete cosine transform;

MFCC features mainly reflect the static features of speech, and the dynamic features of speech signals can be described by the first-order difference spectrum and second-order difference spectrum of static features. The entire speech feature is composed of MFCC parameters, MFCC first-order and second-order difference coefficients, normalized energy coefficients and their first-order and second-order difference coefficients. Each frame contains a total of 39 dimensional features;

B. Training of standard pronunciation model:

(1) Utilize the said speech feature of step A to train the process of the standard pronunciation model based on phonemes, as shown in Figure 2:

a. Establish a prototype of a multidimensional Gaussian distribution whose covariance matrix is a diagonal form of a single data stream according to the dimensionality of speech features, and use all the speech data to estimate the mean vector and covariance matrix of the Gaussian distribution.

b. Determine the pronunciation dictionary and phonetic symbol system, and complete the phoneme-level labeling of all voices. The phonetic symbol system in this embodiment includes 40 phonemes, 1 mute mark, and 1 pause mark.

c. In this embodiment, a phoneme-based Hidden Markov Model (HMM) is used as a standard pronunciation model, and HMM is a statistical speech recognition model widely used at present. The state transition model of HMM from left to right can well describe the pronunciation characteristics of speech. The phoneme and silence models used in the present invention are 3-state HMMs, and the pause model is a single-state spanable HMM, and its topological structure is shown in FIG. 3 . where q _i represents the state of the HMM. a _ij represents the jump probability of the HMM. b _j (O _t ) is the multi-stream mixed Gaussian density probability distribution function of the state output of the HMM model. As shown in formula (1).

${\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; the\; s</span>}}{\mathrm{<span\; class="notranslate">\; \&Pi;</span>}}}{<span\; class="notranslate">\; [</span>\underset{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; jsm</span>}}\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; st</span>}}<span\; class="notranslate">\; ;</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; jsm</span>}}<span\; class="notranslate">\; ;</span>{\mathrm{<span\; class="notranslate">\; \&phi;</span>}}_{\mathrm{<span\; class="notranslate">\; jsm</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ]</span>}^{{\mathrm{<span\; class="notranslate">\; \&gamma;</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Where S is the number of data streams, M _s is the number of mixed Gaussian density distributions in each data stream; N is a multidimensional Gaussian distribution, as shown in formula (2):

$\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; \&mu;</span>}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; \&phi;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\sqrt{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&phi;</span>}<span\; class="notranslate">\; |</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&mu;</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; \&phi;</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&mu;</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

The standard pronunciation model of the present embodiment comprises 40 phoneme HMM models and a mute HMM model, a pause HMM model; Said Gaussian distribution prototype is copied into each HMM model; Then utilize Baum-Welch algorithm to carry out multiple HMM models to each HMM model Second valuation, the number of valuations can be 5;

d. Gradually increase the number of Gaussian components in the HMM model, and perform Baum-Welch training on the obtained model again; the number of Gaussian components increases in order of 2, 4, 6, and 8; when the number of Gaussian components increases to 8, repeat the training 10 times , the training process ends.

(2) Carry out the self-adaptation of Chinese population accent to said standard pronunciation model, the embodiment of the present invention has adopted the accent self-adaptation method based on global MLLR and MAP serial, self-adaptation number of times is set to 4, and concrete process is as Fig. 4 shown.

a. MLLR is an adaptive algorithm based on model transformation. The basic assumption of this type of algorithm is that the transformation relationship between similar speech in the speaker-independent speech model space and the adapted human speech space is also similar, so the transformation relationship can be calculated by using the speech that has appeared in the training speech. The model of the non-appearing speech uses this transformation to realize the mapping from the speaker-independent model to the speech space of the adapted person, so as to realize the adaptation. The speech model space is divided into R classes according to a certain measure (such as Euclidean distance, likelihood, etc.), the transformation of each class is T _r (*), and the corresponding training speech set of each class is X _r , r=1, 2 ,..., R, the model parameter is λ _r , r=1, 2,..., R, then the adaptive training satisfies:

${\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2</span>}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

Adapted parameters ${\widehat{\mathrm{\&lambda;}}}_{r}r=\mathrm{1,2},...R$ satisfy

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1,2</span>}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; .</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Because this type of algorithm makes full use of the interrelationship between speech, multiple models share a transformation, and the parameters to be estimated are the coefficients of each transformation, and it is easier to accumulate data to estimate parameters, which can take effect in the case of less adaptive data, so it has the advantages of Faster adaptive speed. The embodiment of the present invention adopts unclassified global MLLR adaptation.

b. The basic principle of the MAP algorithm is to maximize the posterior probability, so it has theoretical optimality:

${\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

The mean vector estimation formula of the standard MAP algorithm is:

$\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&tau;</span>}}\stackrel{<span\; class="notranslate">\; \_</span>}{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; \&tau;</span>}}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&tau;</span>}}\mathrm{<span\; class="notranslate">\; \&mu;</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

where L _t is the probability of the observation vector at time t to the Gaussian mixture component, τ is the weight of the adaptive speech data based on prior knowledge, μ is the mean vector of the adaptive speech, and μ is the mean vector of the speaker-independent model. It can also be seen from this that when there are enough adaptive data, the mean value vector after adaptation will tend towards the speaker-dependent mean vector μ. The purpose of adopting MAP adaptation after MLLR adaptation in the embodiments of the present invention is to make full use of adaptive voice data and further provide the effect of accent adaptation.

Store the final standard pronunciation model into the external memory of the embedded system.

C. Generation of standard pronunciation network:

The standard pronunciation network of the embodiment of the present invention is as shown in Figure 5, wherein (a) is a linear network example with words as nodes, the starting node is the beginning "sil", and the ending node is the last "sil", (b) is a linear network with phonemes as nodes inside each word, and a network with states as nodes inside each phoneme as shown in Figure 3. The network generation process is shown in Figure 6: first, the original text is segmented into words as shown in Figure 5(a), and then the pronunciation dictionary is searched for each word to get the result as shown in Figure 5(b). Considering the multi-pronunciation situation of a word, in order to save storage space and improve search efficiency, this embodiment has carried out phoneme string comparison based on dynamic programming between multiple pronunciations of a word, and multiple phoneme sequences are fused into a phoneme is a network of nodes such that the same phoneme is shared among the various pronunciations. Finally, the phoneme HMM model is used to expand the network into a network with states as nodes. Each state node records the current node's state identifier, phoneme identifier, word identifier, number of preorder nodes, and preorder node identifier information. So far, the standard pronunciation network of this embodiment with the definite starting node P and terminating node T, the current node is only related to its predecessor node and the state of the HMM is obtained without considering the grammar.

The said standard pronunciation network is stored in the external memory of the embedded system.

D. Voice endpoint detection:

(1) The voice signal is first low-pass filtered, and then sampled and quantized by a 16-bit linear A/D to become a digital voice. The sampling frequency is 8kHz;

(2) carry out pre-emphasis and frame-by-frame windowing process to said digital voice, obtain the frame-by-frame voice with quasi-stationarity; method is identical with the (2) step of step A;

(3) Calculate the short-time logarithmic energy for the framed speech.

(4) Adopt the method for moving average filtering to obtain the end detection feature by said time-domain logarithmic energy: the end point detection is carried out in real time, and the real-time end point detection method needs to meet the following requirements: a, have consistency to different background noise levels output; b, can detect the start point and end point; c, short delay; d, limited response interval; e, maximize the signal-to-noise ratio at the end point; f, accurately locate the detected end point; g, Suppress detection errors to the greatest extent; comprehensively consider the objective function defined by the above requirements, which is very similar to the graphic boundary detection function (sliding average filter) usually used in image processing. The moving average filter is shown in formula (7), wherein g(·) is the time-domain logarithmic energy, t is the current frame number, and h(·) is the moving average filter, as shown in formula (8) , it can be seen that h(·) is an odd symmetric function, W can be 13, f(·) is shown in formula (9), its parameters can be: A=0.2208, s=0.5383, [K ₁ ...K ₆ ] =[1.583, 1.468, -0.078, -0.036, -0.872, -0.56].

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; W</span>}}{\overset{\mathrm{<span\; class="notranslate">\; W</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

$\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; W</span>}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 8</span>}<span\; class="notranslate">\; )</span>$

f(x)＝e ^Ax [K ₁ sin(Ax)+K ₂ cos(Ax)]+e ^-Ax [K ₃ sin(Ax)+K ₄ cos(Ax)]+K ₅ +K ₆ e ^sx ( 9)

(5) adopt the method that upper limit and lower limit double threshold value and finite state machine combine, carry out endpoint detection to said end detection feature, obtain the starting point and the end point of speech: said end detection feature F(t) is in speech Positive values at the beginning, negative values at the end, and close to zero in silent segments. According to the pre-set upper limit, lower limit threshold and the shortest duration of the voice, each frame of voice is controlled to jump between voice, mute and leave the voice state. The initial setting is the mute state, and when F(t) reaches the upper threshold, the starting endpoint of the output voice is entered into the voice state. In the speech state, when F(t) reaches the lower threshold, it enters the leaving speech state. When the time in the state of leaving the voice reaches a set threshold, the end point of the output voice is closed, the recording channel is closed, and the end point detection ends.

E. Speech feature extraction for evaluation:

The said framed speech of step D is extracted speech feature, and the process is identical with step (3) of step A.

F. Optimal path search:

(1) carry out forcibly matching the said speech feature of step E with the said standard pronunciation network of step C, obtain all possible path information in the network: the standard pronunciation network of the embodiment of the present invention is a linear network from left to back ( As shown in Figure 5), the optimal path can be obtained by using the frame-synchronized Viterbi beam search algorithm. Given the HMM model Φ and the observation vector sequence O={o ₁ ,...,o _T }, it is necessary to find the best state sequence S={s ¹ ,...s ^T } that produces this observation vector sequence, namely

$\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; the\; s</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; \&Phi;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \}</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; )</span>$

In the Viterbi algorithm, the optimal path likelihood at time t is defined as

V _i (t)=P(o ₁ ,..., o _t , s ¹ ,..., s ^t-1 , s ^t =i |Φ) (11)

In a linear network, the optimal path at any time is only related to the information of the current frame and the previous frame, which satisfies the principle of no aftereffect. Therefore, if the globally optimal path passes through node i at time t, then the part of the path between 0 and time t must be optimal among the paths with node i as the final node at time t. If we only want to obtain the optimal path, then at time t, only one path with node i as the final node needs to be reserved.

According to the above principles, the search algorithm in this embodiment is:

Definition: PreNode(i) is the preorder node set of node i. BestPre(t, i) is the optimal preorder node of node i at time t. L(t, i) is the likelihood score of the speech frame corresponding to node i at time t. L_Path(-1, i) and L_Path(0, i) are the likelihood scores of the optimal path with node i as the final node in the previous frame and the current frame, respectively.

Step 1: At time t=0

$\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; path</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; Entry</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&NotElement;</span>\mathrm{<span\; class="notranslate">\; Entry</span>}\end{array}\right.-<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 12</span>}<span\; class="notranslate">\; )</span>$

Where i∈Entry means that i is the starting node.

Step 2: At time t, for any node i, the likelihood score L(t, i) of the current frame has been obtained, then the optimal path score of the current frame is:

$\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; path</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; path</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; \&ForAll;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; PreNode</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 13</span>}<span\; class="notranslate">\; )</span>$

Record the optimal preorder node into BestPre(t, i), and exchange the data of L_Path(-1, i) and L_Path(0, i) to prepare for the calculation of the next frame.

Step 3: If t<T, go to step 2; otherwise, end.

(2) When the speech ends, the optimal state path of mandatory matching can be obtained by backtracking BestPre(t, i) from the terminating node allowed by the network;

G. Calculation of pronunciation quality score

(1) Utilize the optimal path information mentioned in step F to calculate the confidence score of each frame of speech features, as shown in formula (14):

${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; log</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; log</span>}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; o</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 14</span>}<span\; class="notranslate">\; )</span>$

(2) Use the optimal path information mentioned in step F to calculate the confidence score of each state on the path; average the confidence scores of all states on the optimal path to obtain the confidence score of the entire sentence, as shown in formula (15) , where N is the number of states contained in the optimal path.

$\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; js</span>}}{\overset{\mathrm{<span\; class="notranslate">\; je</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}{\mathrm{<span\; class="notranslate">\; je</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; js</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 15</span>}<span\; class="notranslate">\; )</span>$

(3) Use the mapping function to map the confidence score of the whole sentence to the subjective evaluation score interval: the value interval of the confidence score obtained by direct calculation is usually between (-∞, a], where a is a constant, which is consistent with the subjective evaluation score interval. The evaluation score interval is inconsistent, and the embodiment of the present invention uses a piecewise linear function to map it to the subjective score interval, as shown in formula (16), where a and b are determined by experiments, and a is an adjustment factor:

$\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; =</span>\left\{\begin{array}{ccc}\mathrm{<span\; class="notranslate">\; \&alpha;C</span>}& \mathrm{<span\; class="notranslate">\; if</span>}& \mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; b</span>}\\ \mathrm{<span\; class="notranslate">\; 100</span>}& \mathrm{<span\; class="notranslate">\; if</span>}& \mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; <figure-callout\; id="0"\; label="b"\; filenames="A20051011484800131.png"\; state="\{\{state\}\}">b</figure-callout></span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; if</span>}& \mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; a</span>}\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 16</span>}<span\; class="notranslate">\; )</span>$

The obtained S can also be further quantified into pronunciation quality grades of excellent, good, medium and poor.

Considering the limitation of memory resources, steps D, E, F and G of the embodiment of the present invention are all performed in time segments with a preset fixed frame number step, and the size of each segment can be 40 frames.

This embodiment develops an embedded English learning system based on pronunciation quality evaluation based on the above method. Learning content can be automatically updated at any time according to teaching requirements. The use of pronunciation quality evaluation technology can enable interactive learning between humans and computers, greatly reducing the workload of oral English teaching in the classroom, alleviating the problem of tight supply and demand of teachers, and realizing independent learning and automatic testing of spoken English. The invention can evaluate the English pronunciation quality of the Mandarin Chinese population. This method has a correlation of 0.74 with the subjective evaluation of the English pronunciation quality evaluation of the Chinese population when the grades are 4 (excellent, good, medium, and poor).

Claims (5)

1, a kind of pronunciation quality evaluating method that is used for language learner, comprise that the phonetic feature that is used to train extracts, the Received Pronunciation model training, the generation of Received Pronunciation network, sound end detects, the phonetic feature that is used to estimate extracts, optimum route search, and the calculating each several part of voice quality mark; It is characterized in that the implementation method of each several part specifically may further comprise the steps:

A, the phonetic feature that is used to train extract:

(1) foundation comprises the tranining database of reading aloud voice in a large number in advance;

(2) digital speech in each voice document in the said tranining database is carried out pre-emphasis and divides frame windowing process, the branch frame voice that obtain having accurate stationarity;

(3) said minute frame voice are extracted phonetic feature, this phonetic feature is a cepstrum coefficient;

B, Received Pronunciation model training

(1) utilize the said phonetic feature of steps A to train the Received Pronunciation model that obtains based on phoneme;

(2) self-adaptation that said Received Pronunciation model is carried out Chinese crowd accent is as final Received Pronunciation model, and Optimization Model is to Chinese crowd's assess performance;

The generation of C, Received Pronunciation network

Given text is carried out the segmentation of words, search Pronounceable dictionary and obtain the phoneme mark, utilizing said Received Pronunciation model based on phoneme to obtain with the state at last is the linear Received Pronunciation network of node;

D, sound end detect:

(1) analog voice signal obtains digital speech through the A/D conversion;

(2) said digital speech is carried out pre-emphasis and divides frame windowing process, the branch frame voice that obtain having accurate stationarity;

(3) said minute frame voice are calculated time domain logarithm energy;

(4) method of employing moving average filter is obtained being used for the end inspection feature of end-point detection by said time domain logarithm energy;

(5) method of employing upper and lower bound dual threshold and finite state machine combination is carried out end-point detection to said end inspection feature, obtains the starting and ending end points of voice;

E, the phonetic feature that is used to estimate extract

Said minute frame voice of step D are extracted phonetic feature, and process is identical with (3) step of steps A.

F, optimum route search:

(1) said phonetic feature of step e and the said Received Pronunciation network of step C are forced coupling, obtain all possible routing information in the network;

(2) utilize said routing information, the terminal node that allows from network is recalled and optimal path;

The calculating of G, voice quality mark:

(1) utilize said optimal path information in the step F to calculate the confidence score of every frame phonetic feature;

(2) utilize in the step F confidence score of each state on the said optimal path information calculating path; Confidence score to all states on the optimal path is averaged the confidence score that obtains whole sentence;

(3) utilize mapping function that said whole sentence confidence score is mapped to subjective assessment and divide number interval, obtain final voice quality mark.

2, the pronunciation quality evaluating method that is used for language learner as claimed in claim 1 is characterized in that, the cepstrum coefficient in the said steps A is the Mei Er frequency marking cepstrum coefficient that utilizes the frequency discrimination characteristic of people's ear.

3, the pronunciation quality evaluating method that is used for language learner as claimed in claim 1, it is characterized in that, Received Pronunciation model among the said step B (1) is the hidden Markov model based on phoneme, the concrete training process of this model is: adopt Gauss model of all phonetic feature initialization, utilize this model copy to go out all phoneme models, adopt the method for Baum-Welth that model is repeatedly trained; Constantly increase the quantity of the gauss component of each phoneme model, carry out the Baum-Welth training again.

4, the pronunciation quality evaluating method that is used for language learner as claimed in claim 1, it is characterized in that, the adaptive implementation method that the Received Pronunciation model is carried out Chinese crowd accent among the said step B (2) is: the Received Pronunciation model that obtains is carried out returning and the accent self-adaptation of maximum a posteriori probability method based on maximum likelihood is linear, obtain final Received Pronunciation model.

5, the pronunciation quality evaluating method that is used for language learner as claimed in claim 1, it is characterized in that, the Received Pronunciation network of said step C is one and has definite start node and terminal node that the state with HMM of not considering the syntax that present node is only relevant with its preorder node is the Linear Network of node.

Images