CN111916058A - Voice recognition method and system based on incremental word graph re-scoring - Google Patents

Abstract

The invention discloses a speech recognition method and system based on incremental word graph re-score, which acquires speech signals to be recognized and extracts acoustic features; calculates the likelihood probability corresponding to the acoustic features by a trained acoustic model; the decoding network, obtain the state-level word map from the decoding network and obtain the word-level word map by updating the word map deterministically; determine the state-level word map of the remaining decoding network, and compare the obtained word-level word map with the obtained word-level word map Merge to generate a decoded word graph once; use the re-scoring language model obtained by training the decoded word graph and the small corpus to obtain the target word graph through the finite state transcription machine merging algorithm; obtain the optimal cost path word graph of the target word graph, and then obtain the corresponding word sequence as the final recognition result. The invention reduces the calculation amount of the determination after the decoding of the common decoder is completed, and speeds up the decoding speed; the word error rate of speech recognition in a specific scene is reduced, and the accuracy is improved.

Description

A speech recognition method and system based on incremental word graph re-score

technical field

The invention belongs to the technical field of speech recognition, and in particular relates to a speech recognition method and system based on incremental word graph re-score.

Background technique

In recent years, with the rapid development of the artificial intelligence industry, speech recognition technology has received more and more attention from academia and industry. As a front-end technology in the field of voice interaction, voice recognition plays a vital role. It is widely used in many human-computer interaction systems, such as intelligent customer service systems, chat robots, personal intelligent assistants, and smart homes.

At present, the traditional speech recognition technology is mainly based on the HMM-DNN framework. The advantage of this modeling is that a speech recognition system with good accuracy can be obtained through relatively less data training. The decoder is an extremely important component in the speech recognition system. Its function is to process the input speech features in series with the acoustic model, pronunciation dictionary and language model to construct a decoding network to obtain a series of state sequences and their corresponding word graphs, and then select from them. The word sequence corresponding to the best state sequence outputs the final recognition result.

In the existing method, in order to ensure the recognition accuracy of one-pass decoding, it is necessary to use a larger beam search width, which will make the final word map too large and the recognition speed is still not fast enough. There is a method to replace one-step decoding with a larger beam search width. Although the decoding speed is about 2-3 times faster at lower WERs, there may be a large difference between the two-step decoding because the beam value is too small. use. The cost of using GPU parallel computing is relatively high, and the wide-scale use of this decoder in industrialized scenarios is still open to question.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a speech recognition method and system based on incremental word graph re-score for the deficiencies in the above-mentioned prior art. When processing the remaining audio, a new word-level word map is generated on the basis of the previous word map, which greatly improves the speed of word map generation and reduces the real-time stream decoding scene without affecting the accuracy of one-pass decoding speech recognition. The delay after the end of medium and long utterances facilitates the output of intermediate results, and then re-scores the decoding results through an advanced language model trained with small sample corpus to achieve self-learning in specific scenarios, thereby reducing word error rates and improving speech recognition. accuracy.

The present invention adopts the following technical solutions:

A speech recognition method based on incremental word graph re-score, comprising the following steps:

S1. Acquire the speech signal to be recognized and perform acoustic feature extraction through preprocessing;

S2. Calculate the likelihood probability corresponding to the acoustic feature by the trained acoustic model;

S3, the decoder constructs a corresponding decoding network through the trained decoding map and the acoustic information calculated in step S2, obtains the word map of the state level from the decoding network, and obtains the word map of the word level by updating the word map deterministically;

S4. After decoding, the state-level word graph of the remaining decoding network is determined, and combined with the obtained word-level word graph to generate a decoded word graph;

S5. The re-scoring language model obtained by decoding the word graph and the small corpus training is obtained through the finite state transcription machine merging algorithm to obtain the target word graph;

S6. Obtain the optimal cost path word graph of the target word graph, and then obtain the word sequence corresponding to the optimal state sequence of the word graph, and use it as the final recognition result.

Specifically, in step S1, preprocessing such as adding Gaussian white noise, pre-emphasis, and windowing is performed on the speech signal; fast Fourier transform is performed on the preprocessed speech signal to convert the time-domain signal into a frequency-domain signal and obtain Power spectrum; point product with a set of triangular filters to obtain the mel energy to obtain the acoustic features of the corresponding dimension.

Specifically, in step S2, the acoustic features calculated in step S1 are used as the input of the acoustic model, the voice features of multiple frames before and after the central frame are input into the acoustic model, and each pronunciation unit corresponding to the central voice frame is obtained after the neural network calculation The acoustic posterior probability of .

Specifically, in step S3, the decoder searches the decoding graph through the Viterbi dynamic programming algorithm, constructs a decoding network by combining the acoustic cost calculated in step S2 and the graph cost in the decoding graph, and constrains the grid size by setting a threshold pruning path; then The state-level word graph is obtained from the decoding network and a new word-level word graph is obtained by updating the word graph deterministically.

Further, the specific steps are as follows:

S301. Obtain the state-level word graph of F from the state sequence of the decoding network, including the state number and transition edge;

S302. Perform a deterministic operation on the first part of F. The last frame of the first part is a repeated deterministic state, and adding a termination state to the transition edge of the repeated deterministic state constitutes a finite state receiver A; a, merge jumps with the same input label from the original image, and gradually add them to the new image that is initially empty;

S303, processing the second part, the first frame of the second part is the last frame of the first part, that is, the repeated deterministic state; the last state of the repeated deterministic state is taken as the initial state to construct a finite state receiver B, B complex The processing result of the repeated determinized state by the finite state receiver A; the arc edge label of the repeated determinized state is found through the mapping table of the state and the arc edge label; the arc edge label is mapped to the first part of the repeated determinized state after the determinization. State number; the new state number and the repeated determination state are in one-to-one correspondence, and the state number and transition edge of the following frame are added in turn to obtain the second part of the finite state receiver B, and B is determined to obtain b;

S304. Combine a and b together to form a finite state receiver C. The state in C normally consists of the following two parts: all transition edges in a are not states of arc edge labels; all transition edges in b except the first one state; the arc edges in receiver C include all arc edges in b except the initial state arcs, and all arc edges in a that start and end in non-repeated deterministic states; if the initial state of a is not a repeated deterministic state , set as the initial state of the finite state receiver C, otherwise the initial state of b is used as the initial state of the receiver C; finally, the final result G is obtained by removing the empty label in C, that is, the incremental word graph generation is realized .

Further, the present invention is also characterized in that step S302 is specifically:

S3021. Create a new empty graph, add the initial state of the original graph and the corresponding initial weight to the new graph, create a new queue, and put the state into the queue;

S3022. Take a state p from the head of the queue, traverse all the input labels of the jumps derived from the state p, and for each input label x, add a new state and a corresponding jump to the new graph, and the input label of the new jump is x, the weight is the ⊕ operation of all jumps corresponding to x in the original image, combining several jumps in the original image into one jump;

S3023, adding the new state of step S3024 to the queue;

S3024, go back to step S3023 to continue processing the queue until the queue is empty, and the determined result is called a.

Specifically, in step S4, the incrementally generated word-level word map is obtained after decoding; then the last part of the state-level word map corresponding to the decoding network is determined; finally, the two parts of the word-level word map are merged to generate the target word map , complete the last part of incremental word graph generation.

Specifically, in step S5, the re-scoring language model obtained by decoding the word graph and the small corpus training is obtained through the finite state transcription machine merging algorithm to obtain the target word graph; the long-term memory neural network is trained by the importance sampling based on the Monte Carlo method. Language model; remember that the decoded word graph is T ₁ , the G.fst converted by the LSTM-RNN language model is T ₂ , and T ₁ and T ₂ generate the target word graph T based on the breadth-first search merging algorithm.

Specifically, in step S6, the optimal cost path word graph of the target word graph is obtained; the word-level word graph is converted into a state-level word graph to obtain the optimal state sequence; finally, the corresponding optimal state sequence is obtained by backtracking to find the optimal precursor node The excellent word sequence is used as the final recognition result.

Another technical solution of the present invention is a speech recognition system based on incremental word graph re-score, according to the method, comprising:

The signal acquisition and detection module is used to obtain and detect the voice signal to be recognized, and retain the effective voice signal;

The preprocessing module is used to preprocess the valid speech signal;

A feature extraction module, which performs feature extraction on the preprocessed speech signal to obtain an acoustic feature sequence;

The incremental decoding module uses the decoder to combine the decoding map and the acoustic model to decode the acoustic feature sequence to construct a decoding network, and incrementally generate a word-level word map;

The word map generation module is used to generate the word-level word map corresponding to the last part of the state sequence of the decoding network, and combine it with the word-level word map of the incremental decoding module to obtain a decoded word map;

The re-scoring module re-scores the decoded word map through the language model trained on the specific scene corpus to generate the target word map;

The recognition module obtains the final recognition result from the target word graph.

Compared with the prior art, the present invention at least has the following beneficial effects:

The present invention is a speech recognition method and system based on incremental word graph re-score. During the speech recognition process, the state-level word graph obtained from the decoding network needs to be determined to ensure that each state does not have the same input label. Therefore, the size of the word graph can be reduced, and because of the uniqueness of the input label, the speed of generating the optimal cost path word graph can be accelerated. However, ordinary decoders can only start determinization from the state sequence corresponding to the first frame of speech signal in the decoding network, so the determinization operation will bring a perceptible delay when the long utterance ends. It is completed during the decoding process, which greatly reduces the calculation amount of the deterministic part after decoding. The word-level word map can be obtained in the decoding process. On this basis, the intermediate recognition results are dynamically generated, and the high-computational language model is newly trained through a small corpus. The weighted sum calculation of the one-pass decoded word graph increases its weight through the merging algorithm, which improves the recognition accuracy in specific scenarios and realizes the self-adaptation of speech recognition in different fields.

Further, for the voice signal in step S1, a part of long silent frames is first filtered through voice activity detection (Voice Activity Detection, VAD), and the effective voice signal is retained to achieve the effect of noise reduction. Then, the effective speech blocks are preprocessed and features are extracted to obtain the acoustic features of the corresponding dimensions. The 200 sample points of a frame of 25ms with a sampling rate of 8k can be extracted as 40-dimensional Fbank features after Mel filter, which greatly reduces the noise of the speech signal. dimension.

Further, using the speech features calculated in step S1 as the input of the acoustic model, generally calculating the acoustic probability matrix of multiple frames at a time, splicing the multi-frame speech signal data blocks with its context information and inputting the acoustic model together, reducing the acoustic posterior probability. The number of matrix calculations to improve the decoding speed.

Further, the decoding network is constructed using the Viterbi dynamic programming algorithm based on beam word graph pruning. All paths that exceed the beam threshold will be pruned. Before obtaining the state-level word graph, state pruning will be performed again to simplify the decoding network to speed up the process. The speed of incremental word graph generation.

Further, the state-level word graph is determined, so that any input sequence on each state node only corresponds to a unique jump, which can greatly reduce the amount of calculation of matching sequences in the graph, and determine the redundancy of the word graph after the determination. The degree is much lower than before determinization.

Further, determining the word graph after decoding is the reason for the high delay of the traditional decoder. Here, the incremental word graph generation decoder only needs to determine the last small piece of undetermined state-level word graph, which is fast. Generate recognition results.

Further, re-score the one-pass decoded word graph generated in step S4, and train the language model through a small sample corpus of a specific scene, which can be a traditional N-gram language model, or a recurrent neural network language model based on long-term and short-term memory, Improve the accuracy of speech recognition in the training corpus scene and reduce the word error rate.

Further, after obtaining the best word sequence as the final recognition result, the final output is obtained through the punctuation service.

A speech recognition system based on incremental word graph re-score, pruning is performed again before the decoding network is determined, and the decoding network generation is smaller than the non-incremental, so the network generation speed is accelerated and the real-time rate is improved. After the incremental word graph is generated, the decoded word graph is the optimal word graph. When it is combined with the re-scoring language model, it will consume less time and reduce the delay after the end of the speech. By post-processing the decoded word graph in one pass, it is possible to fine-tune the speech recognition results in any scenario to improve its recognition accuracy, and to achieve higher accuracy and wider customization at lower costs.

To sum up, the present invention incrementally generates word-level word graphs in the decoding process to provide convenience for outputting intermediate results; reduces the amount of deterministic computation after the decoding of the ordinary decoder is completed, and speeds up the decoding speed; reduces the number of words recognized by speech in specific scenarios. Error rate improves accuracy.

The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments.

Description of drawings

Fig. 1 is the flow chart of a kind of speech recognition method provided by the invention;

Fig. 2 is the schematic diagram of the incremental word graph generation method provided by the present invention;

Fig. 3 is the incremental word graph provided by the present invention to determine the flow chart of data block size;

Fig. 4 is the incremental word graph repetition determination state processing flow chart provided by the present invention;

5 is a schematic flow chart of a specific module of the system of the present invention;

6 is a performance comparison diagram of the method of the present invention and a common decoder in terms of real-time rate;

7 is a performance comparison diagram of the method of the present invention and a common decoder in terms of delay;

FIG. 8 is a performance comparison diagram of the method of the present invention and an ordinary decoder in terms of word error rate in a specific scenario.

Detailed ways

First, the methods and terminology involved in the present invention will be described.

1) Finite State Receiver (FSA): Weighted Finite State Transcriber (WFST) consists of a set of states and directed jumps between states, where each jump stores three kinds of information, namely input labels, output labels and weight, recorded in the format of "input_label:output_label/weight", the decoding network mentioned in the present invention is a WFST. FSA can be seen as a simplification of FST, with only input labels for each jump.

2) State-level word graph: a directed acyclic graph with input labels, output labels and weight values on its transition edges. The input label is the alignment information, and the output label is the word result.

3) Word-level word map: also called compressed word map, which is obtained by determining the state-level word map. Unlike the state-level word map, its alignment information is changed from input label storage to weight storage.

4) Deterministic: The classical algorithm in the finite state transcription machine, whose role is to ensure that the same input label does not exist on all transition edges starting from any state in the transcription machine, and to ensure the uniqueness of the input label sequence.

5) Acoustic feature: the frequency domain information obtained by the preprocessing and Fourier transform of the speech signal is obtained by processing, and the acoustic feature used in the experiment of the present invention is the filter bank (Filter Bank, FBank) feature.

6) Acoustic model: The information related to pronunciation is modeled and obtained by iterative training based on acoustic features. The main function of the acoustic model is to obtain the matching degree between the input acoustic feature sequence and the pronunciation unit sequence, which is usually expressed by probability. The acoustic model based on Hidden Markov-Deep Neural Network (HMM-DNN) is used in the experiments of the present invention.

7) Language model: used to model the correlation between words in the language to be recognized. In the experiment of the present invention, a 3-gram language model based on probability statistics is used in one-pass decoding. The re-scoring language model uses the Long Short-Term Memory (LSTM) recurrent neural network language model that has a better modeling effect on continuous spaces. Through the language model, each word sequence W={w1, w2, The probability of occurrence of ..., wn}.

8) Viterbi algorithm: It is an algorithm based on the idea of dynamic programming to solve the optimal path in the decoding network. In the decoding process of speech recognition, the Viterbi algorithm is usually combined with certain threshold constraints to construct a corresponding decoding network.

Please refer to Fig. 1, a kind of speech recognition method based on incremental word graph re-score of the present invention, comprises the following steps:

S1. Acquire the speech signal to be recognized and perform acoustic feature extraction through preprocessing;

S101, preprocessing the speech signal, first adding a random Gaussian value to each frame sample point, then subtracting the average value of the frame sample point to remove the DC component, and then subtracting the value of each frame sample point The value of the previous frame is multiplied by 0.97 for pre-emphasis, and finally it is dot-multiplied with a window function whose size is the frame length to obtain a more stable speech signal.

S102, perform acoustic feature extraction, first obtain a filter bank object whose length is expanded to the nth power of 2, and then perform fast Fourier transform on the preprocessed speech signal to convert the time-domain signal into a frequency-domain signal to obtain a frequency spectrum Sample point, then multiply the real part of the spectrum sample point by the real part plus the imaginary part by the imaginary part to get its power spectrum, and finally take the sample point of half of the power spectrum plus one and roll it with the triangular filter bank distributed according to the Mel scale. Product to get FBank acoustic features.

S2. Calculate the likelihood probability corresponding to the acoustic feature by the trained acoustic model;

The acoustic features calculated in step S1 are used as the input of the acoustic model. In order to consider the acoustic context information of each frame feature, the multiple frames of speech features before and after the central frame are input into the acoustic model together. The features are input into the Time-Delay Neural Network (TDNN) together, and the first layer of the neural network is mapped to the acoustic features of 7 frames. The latter three hidden layers map the 7 frames of acoustic features into one frame through subsampling. Finally, Through the softmax normalization operation, the acoustic probability matrix of 5696 pronunciation results of each frame of speech signal after triphone clustering is obtained.

S3. The decoder constructs a corresponding decoding network through the trained decoding map and the acoustic information calculated in step S2, obtains the state-level word map from the decoding network, and determines it by updating the word map, and incrementally generates the word-level word map ;

The decoding is carried out frame by frame. The state that can be reached at time t starts from time t-1. The state generated in each frame is linked by transition edges. It is formed by the Viterbi algorithm based on dynamic programming and setting the tolerance estimation value to constrain the grid size. A directed acyclic graph containing all the recognition results is created, which is the decoding network.

The present invention proposes to divide the state-level word graph corresponding to the decoding network into a plurality of consecutive blocks for determination, and these consecutive blocks correspond to the sequential frame state sequence range in the word graph. Incremental word graph generation can be achieved by determining the state sequences of these sequential frames and connecting them together in a certain way. The method is to introduce special symbols, namely arc edge labels, at the decomposition points of consecutive blocks. The state sequence with arc edge labels is called repeated deterministic state. The specific steps of incremental word graph generation are:

Referring to Figure 2, the determinization of the decoding network is divided into many blocks, and two adjacent parts (F) are processed each time, and the frame state sequence where the two parts overlap is the repeated determinization state. Arc edge label unique identifier.

S301, decode to obtain the first part of the state, obtain the state number of each state and the transition edge to form the first part of the state-level word graph of F, that is, the first data block;

S302. Perform a deterministic operation on the first part of the state-level word graph of F, the last frame of which is a repeated determinized state, and a finite state receiver A is formed by adding a termination state to the transition edge of the repeated deterministic state. The main idea of determining A is to continuously merge jumps with the same input label from the original image, and gradually add them to the new image that is initially empty, specifically:

S3021. Create a new empty graph, add the initial state of the original graph and the corresponding initial weight to the new graph, create a new queue, and put these states into the queue.

S3022 , take out a state p from the head of the queue, and traverse all the jumped input labels drawn by the state p. For each input label x, a new state and corresponding jump are added to the new graph. The input label of the new jump is x, and the weight is the ⊕ operation of all jumps corresponding to x in the original graph. This step combines several jumps in the original image into one jump.

S3023. Add the new state of step S3024 to the queue.

S3024, go back to step S3023 to continue processing the queue until the queue is empty.

Call the finalized result a, the state number of the state sequence uniquely identified by the arc edge label may have changed.

S303, it is necessary to consider how to obtain the new data block of the second part of F. Please refer to Figure 3. In the experiment, the minimum block size threshold and the maximum delay threshold are limited by two parameters. The former defines the minimum number of frames in each block; the latter determines how many new frames need to be decoded. The new block can continue to be determined. When the number of newly decoded frames exceeds the maximum delay threshold, the frame with the least number of states between the minimum block frame threshold and the maximum delay threshold is selected, which is the last frame of the new block. From the last frame to the last frame in the determined word graph is a new data block, and its first frame is the last frame of the first part, that is, the repeated determination state. Taking the last state of the repeated deterministic state as the initial state to construct a finite state receiver B, B needs to reuse the processing result of the repeated deterministic state by the finite state receiver A;

Referring to Figure 4, the core of incremental word graph generation is to process the repeated determinization state so that new blocks can reuse the previous determinization results. First, find the arc edge label of the repeated determinized state through the mapping table between the state and the arc edge label; map the arc edge label to the state number of the repeated determinized state after the first part of the determinization; update the state node and state number mapping table to convert the new The state number corresponds to the repeated determination state one by one, and the state number and transition edge of the following frame are added in turn, and the second part of the finite state receiver B can be obtained. Perform steps S3021, S3022, S3023, and S3024 to determine B to obtain b ;

S304. Combine a and b together to form a finite state receiver C. The state in C normally consists of the following two parts: all transition edges in a are not states of arc edge labels; all transition edges in b except the first one state.

The arc edges in receiver C include all arc edges in b except for the initial state arcs, and all arc edges in a that start and end with non-repeated deterministic states; if the initial state of a is not a repeated deterministic state, then Let it be the initial state of the finite state receiver C, otherwise we use the initial state of b as the initial state of the receiver C;

Finally, the final result G is obtained by removing the empty labels in C, that is, the incremental word graph generation is realized.

S305: Repeat steps S303 and S304 with the G generated in step S304 as the first part until the state of the acoustic feature of the last frame is generated and state pruning is performed on the last frame.

S4, after the decoding in step S3 is completed, the state-level word graph of the last remaining decoding network is determined, and combined with the obtained word-level word graph to generate a decoded word graph;

S401, after the decoding is completed, an incrementally generated word-level word map is obtained;

S402, determining the last part of the state-level word graph corresponding to the decoding network;

S403: Combine the two parts of word-level word graphs to generate a target word graph, and complete the generation of the last part of the incremental word graph.

S5. The re-scoring language model obtained by decoding the word graph and the small corpus training is obtained through the finite state transcription machine merging algorithm to obtain the target word graph;

S501. Train a long-short-term memory neural network language model (LSTM-RNNLM) through importance sampling based on Monte Carlo method;

S502, record the decoded word graph as T ₁ , the G.fst converted by the LSTM-RNN language model is T ₂ , and the steps of generating the target word graph T based on the breadth-first search merging algorithm of T ₁ and T ₂ are as follows:

S5021, denote the sets _of T1 and T2 jumps as _E1 and _{E2 respectively} .

S5022, traverse all the jumps e ₁ of T ₁ and T ₂ belong to E ₁ and e ₂ belong to E ₂ , in the traversal process, if the output label o[e ₁ ] of a certain e ₁ and the input label i[e of e _{2 2} ] is the same, then take the source state pair (p[e ₁ ], p[e ₂ ]) and target state pair (n[e ₁ ], n[e ₂ ]) of e ₁ and e ₂ as T respectively. Two states, and add a jump from state (p[e ₁ ], p[e ₂ ]) to (n[e ₁ ], n[e ₂ ]) in T, and its input label is i[e ₁ ], the output label is o[e ₂ ], the weight is the weighted sum operation of the weight of e ₁ and e ₂ .

S5023. Repeat step S5022 until all e ₁ and e ₂ satisfying o[e ₁ ]=i[e ₂ ] are processed, and after setting the starting state and ending state and their weights, T ₁ and T are obtained The composite result of ₂ T.

S6. Obtain the optimal cost path word graph of the above target word graph, and then obtain the word sequence corresponding to the optimal state sequence of the word graph, and use it as the final recognition result.

S601. Determine the optimal precursor node of each state node through the idea of dynamic programming to obtain the optimal cost path word graph of the above target word graph;

S602, taking out the transition ID in the weight information in the word-level word map and converting the replacement input label into a state-level word map to obtain an optimal state sequence;

S603: Obtain word labels whose output labels are not 0 on the arc edge through the optimal state sequence, find the Chinese characters corresponding to these word labels according to the word symbol table, and output them in sequence to realize the process of recognizing speech from audio to Chinese characters.

Referring to Fig. 3, the present invention provides a speech recognition system based on incremental word graph re-score, including:

The signal acquisition and detection module is used to obtain and detect the speech signal to be recognized, and retain the valid speech signal;

The preprocessing module is used to preprocess the valid speech signal;

The preprocessing module includes: a first processing module, which performs topological sorting on the state of each frame of the target word graph to ensure its topological ordering. The second processing module, through the idea of dynamic programming, traverses the entire word graph to determine the optimal precursor node of each state node, and obtains the state ID of the optimal termination state; the third processing module, through the state ID, traces back to obtain the optimal precursor node. All the state IDs on the path, and the arc edges connecting these state IDs are obtained, which together constitute a compressed word graph containing the optimal state sequence.

A feature extraction module, which performs feature extraction on the preprocessed speech signal to obtain an acoustic feature sequence;

The incremental decoding module uses the decoder to combine the decoding map and the acoustic model to decode the acoustic features to construct a decoding network, and incrementally generate the word-level word map;

The incremental decoding module includes: the first determination module, which obtains the acoustic observation probability of each frame feature corresponding to all pronunciation units from the chain acoustic model based on the time-delay neural network; the second determination module is based on the 3-gram language model in the decoding diagram. The acoustic feature sequence determines the probability of the possible target word sequence, that is, the graph cost; the third determining module, the decoder combines the acoustic probability and the graph cost in the decoding graph to construct a decoding network according to the Viterbi dynamic programming algorithm; the fourth determining module, The state-level word graph is obtained according to the decoding network, and the word-level word graph is incrementally generated by deterministically.

The word map generation module is used to generate the word-level word map corresponding to the last part of the state sequence of the decoding network, and combine it with the word-level word map of the previous module to obtain the first-pass decoded word map;

The re-scoring module re-scores the decoded word graph through a smaller language model trained on a specific scene corpus to generate the target word graph;

The re-scoring module includes: a merging module, which multiplies the graph cost of the decoded word graph plus the re-scoring language model score by 0.2 to generate a new word graph according to the merging algorithm; a pruning module, which transfers the weights that are too large in the new word graph. The edge points to a specific state ID, then the transition edge that points to the specific state ID is deleted and the state ID whose in-arc is empty is also deleted.

The recognition module obtains the final recognition result from the target word graph.

The identification module includes: a processing module, which obtains a compressed word graph containing only the optimal cost state sequence through a dynamic programming algorithm for the target word graph; a conversion module, which converts the compressed word graph into an input label as a transition ID, and an output label as a word ID , a state-level word graph whose weight is the graph cost; the generation module sequentially obtains the corresponding word sequence of the word ID mapping on the arc edge of the state-level word graph, which is the final recognition result.

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. The components of the embodiments of the invention generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Thus, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

This embodiment compares the incremental word graph re-score speech recognition method and the mainstream decoding method under multiple real scene test sets. The data set used is real dialogue data collected in a telephone scene, and its content covers multiple industries. , the average duration of each dataset is about 2h. During the experiment, the parameter configuration of the incremental word graph re-score method is three more parameters than that of the mainstream decoder, namely the deterministic maximum delay threshold parameter and the deterministic minimum block size parameter, and the finite state converted by the incoming re-score language model. Transcription machine.

The indicators referenced by the present invention are mainly the real-time rate (Real-Time factor, RTF) of speech recognition, the word error rate (Word Error Rate, WER), and the delay caused by the last frame state pruning and word map determination after decoding. . The calculation formula of the real-time rate is Real_time_factor=total_time_taken_/total_audio_, that is, the total decoding duration of the audio is divided by the total duration of the audio. The purpose of the three indicators is mainly to verify the improvement of decoding speed, the increase of recognition accuracy and the reduction of delay brought by the incremental word graph re-score method in the present invention. As shown in Table 1 and Table 2, the decoding real-time rate and delay of the deterministic method (DCG) of the traditional decoder and the incremental word graph re-score method (ADCG) of the present invention are compared.

Table 1

Combining Table 1 and Figure 4, it can be seen that the performance of the incremental word graph re-score method in terms of decoding real-time rate (RTF) is far superior to the mainstream deterministic-based method, and the real-time rate has dropped by nearly 25%, which greatly improves the decoding speed.

Table 2

The delay parameter more accurately reflects the performance of incremental word graph generation based on the real-time rate, because it mainly considers the delay of re-score after decoding and also includes the time change consumed by re-score. Combining with Table 2 and Fig. 5, the incremental deterministic decoder 7 reduces the delay of the method by an average of 25% compared with the traditional decoder, which fully demonstrates the superiority of the method of the present invention.

Finally, combined with Table 3, Figure 6, Figure 7 and Figure 8, the word error rate is calculated according to the recognition results of the test set in a specific scenario. In these scenarios, the recognition accuracy is improved by nearly 3.14%, and the accuracy is slightly better than the speech recognition of big manufacturers in the industry.

table 3

To sum up, the present invention is a speech recognition method and system based on incremental word graph re-score. Through a large number of experiments, it has demonstrated better performance than traditional speech recognition methods, accelerated decoding speed, reduced delay and improved specificity. The recognition accuracy in the scene proves the superiority of incremental word graph re-score from experiments.

The above content is only to illustrate the technical idea of the present invention, and cannot limit the protection scope of the present invention. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention all fall within the scope of the claims of the present invention. within the scope of protection.

Claims (10)

1. a speech recognition method based on incremental word graph re-scoring, is characterized in that, comprises the following steps: S1. Acquire the speech signal to be recognized and perform acoustic feature extraction through preprocessing; S2. Calculate the likelihood probability corresponding to the acoustic feature by the trained acoustic model; S3, the decoder constructs a corresponding decoding network through the trained decoding map and the acoustic information calculated in step S2, obtains the word map of the state level from the decoding network, and obtains the word map of the word level by updating the word map deterministically; S4. After decoding, the state-level word graph of the remaining decoding network is determined, and combined with the obtained word-level word graph to generate a decoded word graph; S5. The re-scoring language model obtained by decoding the word graph and the small corpus training is obtained through the finite state transcription machine merging algorithm to obtain the target word graph; S6. Obtain the optimal cost path word graph of the target word graph, and then obtain the word sequence corresponding to the optimal state sequence of the word graph, and use it as the final recognition result. 2. the speech recognition method based on incremental word graph re-scoring according to claim 1, is characterized in that, in step S1, adds Gaussian white noise, pre-emphasis, the preprocessing work such as windowing is carried out to speech signal; The processed speech signal is subjected to fast Fourier transform to convert the time-domain signal into a frequency-domain signal and obtains the power spectrum; point product with a set of triangular filters to obtain the Mel energy to obtain the acoustic features of the corresponding dimension. 3. the speech recognition method based on incremental word graph re-scoring according to claim 1, is characterized in that, in step S2, use the acoustic feature that step S1 calculates to obtain as the input of acoustic model, the multiple frames before and after the center frame The speech features are input into the acoustic model together, and the acoustic posterior probability of each pronunciation unit corresponding to the central speech frame is obtained after the neural network calculation. 4. the speech recognition method based on incremental word graph re-scoring according to claim 1, is characterized in that, in step S3, decoder searches decoding graph by Viterbi dynamic programming algorithm, in conjunction with the acoustic cost that step S2 calculates and The decoding network is constructed from the graph cost in the decoding graph, and the grid size is constrained by setting a threshold pruning path; then the state-level word graph is obtained from the decoding network and a new word-level word graph is obtained by updating the word graph deterministically. 5. the speech recognition method based on incremental word graph re-scoring according to claim 4, is characterized in that, concrete steps are as follows: S301. Obtain the state-level word graph of F from the state sequence of the decoding network, including the state number and transition edge; S302. Perform a deterministic operation on the first part of F. The last frame of the first part is a repeated deterministic state, and adding a termination state to the transition edge of the repeated deterministic state constitutes a finite state receiver A; a, merge jumps with the same input label from the original image, and gradually add them to the new image that is initially empty; S303, processing the second part, the first frame of the second part is the last frame of the first part, that is, the repeated deterministic state; the last state of the repeated deterministic state is taken as the initial state to construct a finite state receiver B, B complex The processing result of the repeated determinized state by the finite state receiver A; the arc edge label of the repeated determinized state is found through the mapping table of the state and the arc edge label; the arc edge label is mapped to the first part of the repeated determinized state after the determinization. State number; the new state number and the repeated determination state are in one-to-one correspondence, and the state number and transition edge of the following frame are added in turn to obtain the second part of the finite state receiver B, and B is determined to obtain b; S304. Combine a and b together to form a finite state receiver C. The state in C normally consists of the following two parts: all transition edges in a are not states of arc edge labels; all transition edges in b except the first one state; the arc edges in receiver C include all arc edges in b except the initial state arcs, and all arc edges in a that start and end in non-repeated deterministic states; if the initial state of a is not a repeated deterministic state , set as the initial state of the finite state receiver C, otherwise the initial state of b is used as the initial state of the receiver C; finally, the final result G is obtained by removing the empty label in C, that is, the incremental word graph generation is realized . 6. the speech recognition method based on incremental word graph re-scoring according to claim 5, is characterized in that, step S302 is specifically: S3021. Create a new empty graph, add the initial state of the original graph and the corresponding initial weight to the new graph, create a new queue, and put the state into the queue; S3022. Take a state p from the head of the queue, traverse all the input labels of the jumps derived from the state p, and for each input label x, add a new state and a corresponding jump to the new graph, and the input label of the new jump is x, the weight is the ⊕ operation of all jumps corresponding to x in the original image, combining several jumps in the original image into one jump; S3023, adding the new state of step S3024 to the queue; S3024, go back to step S3023 to continue processing the queue until the queue is empty, and the determined result is called a. 7. the speech recognition method based on incremental word graph re-scoring according to claim 1, is characterized in that, in step S4, after decoding finishes, obtain the word-level word graph of incremental generation; Then to the last part corresponding to decoding network The state-level word map is determined; finally, the two parts of the word-level word map are merged to generate the target word map, and the last part of the incremental word map is generated. 8. the speech recognition method based on incremental word graph re-scoring according to claim 1, is characterized in that, in step S5, the re-scoring language model obtained by decoding word graph and small corpus training once is merged by finite state transcription machine The algorithm obtains the target word map; trains the long-term memory neural network language model through importance sampling based on Monte Carlo method; remembers the decoded word map as T ₁ , and the G.fst converted by the LSTM-RNN language model is T ₂ , T ₁ and T ₂ generate the target word graph T based on the breadth-first search merging algorithm. 9. the speech recognition method based on incremental word graph re-scoring according to claim 1, is characterized in that, in step S6, obtains the optimal cost path word graph of target word graph; The word level word graph is converted into state level The word graph obtains the optimal state sequence; finally, the corresponding optimal word sequence is obtained as the final recognition result by backtracking to find the optimal precursor node. 10. a speech recognition system based on incremental word graph re-scoring, is characterized in that, the method according to claim 1, comprises: The signal acquisition and detection module is used to obtain and detect the voice signal to be recognized, and retain the effective voice signal; The preprocessing module is used to preprocess the valid speech signal; A feature extraction module, which performs feature extraction on the preprocessed speech signal to obtain an acoustic feature sequence; The incremental decoding module uses the decoder to combine the decoding map and the acoustic model to decode the acoustic feature sequence to construct a decoding network, and incrementally generate a word-level word map; The word map generation module is used to generate the word-level word map corresponding to the last part of the state sequence of the decoding network, and combine it with the word-level word map of the incremental decoding module to obtain a decoded word map; The re-scoring module re-scores the decoded word map through the language model trained on the specific scene corpus to generate the target word map; The recognition module obtains the final recognition result from the target word graph.

Images