TWI659411B - Multi-language hybrid speech recognition method - Google Patents

Abstract

The invention discloses a multilingual mixed speech recognition method, which belongs to the technical field of speech recognition. The method includes: step S1, configuring a multilingual mixed dictionary including a plurality of different languages; step S2, according to the multilingual mixed dictionary and a plurality of different languages. Multilingual speech data training forms an acoustic recognition model; step S3, forming a language recognition model based on multilingual text corpora training including multiple different languages; step S4, forming a speech using a multilingual mixed dictionary, acoustic recognition model, and language recognition model Recognition system; subsequently, the speech recognition system is used to recognize the mixed speech and output the corresponding recognition result. The above technical solution has the beneficial effects that it can support the recognition of mixed speech in multiple languages, improve the accuracy and efficiency of recognition, and therefore improve the performance of the speech recognition system.

Description

Multilingual mixed speech recognition method

The invention relates to the technical field of speech recognition, in particular to a multilingual mixed speech recognition method.

In the expression of everyday speech, people often inadvertently use one language and another language or other languages. For example, some English words will directly use their original names in Chinese, such as "ipad", "iphone" , "USB" and other proper nouns, so it will cause a mixed phenomenon of Chinese and English, which will bring some difficulties and challenges to speech recognition.

The recognition principle of the early multilingual mixed speech recognition system was to establish separate speech recognition systems, then cut the mixed speech, and send the speech fragments of different languages to the corresponding speech recognition system for recognition, and finally each speech The recognition results of the segments are merged to form the recognition result of the mixed speech. This recognition method is difficult to guarantee the accuracy of segmenting the mixed speech according to the language, on the other hand, the context information of each segmented speech segment is too short, which affects the recognition accuracy.

In recent years, the practice of multilingual mixed speech recognition methods has begun to change. Specifically, it is to expand the dictionary of a single speech recognition system, that is, use the phoneme set of one language to piece together another language. For example, the "iphone" in English is in Chinese. The pronunciation in the dictionary will be pieced together as "love crazy". Although this recognition method can recognize individual vocabularies in different languages, on the one hand, it requires that the user's pronunciation is very strange (for example, "iphone" must be accurately pronounced as "love crazy"); The accuracy rate will drop significantly.

According to the above problems in the prior art, a technical solution of a multilingual mixed speech recognition method is now provided, which aims to support the recognition of multilingual mixed speech, improve the recognition accuracy and efficiency, and thus improve the performance of the speech recognition system.

The above technical solution specifically includes: a multilingual mixed speech recognition method, in which a speech recognition system for identifying multilingual mixed speech is first formed, and the method for forming a speech recognition system includes: Step S1, configuring a multi-language multi-language Language mixed dictionary; Step S2, forming an acoustic recognition model based on multilingual mixed dictionary and multilingual speech data including multiple different languages; Step S3, forming a language recognition model based on multilingual text corpus training including multiple different languages; In step S4, a multilingual mixed dictionary, an acoustic recognition model, and a language recognition model are used to form a speech recognition system. Subsequently, the speech recognition system is used to recognize the mixed speech and output the corresponding recognition result.

Preferably, in the multilingual mixed speech recognition method, in step S1, a three-phoneme modeling method is adopted, and a multilingual mixed dictionary is configured according to a single language dictionary corresponding to each different language.

Preferably, in the multilingual mixed speech recognition method, in step S1, a multilingual mixed dictionary is configured by using a three-phoneme modeling method; when the multilingual mixed dictionary is configured, each language included in the multilingual mixed dictionary is configured. A corresponding language mark is added before each phoneme of the phoneme to distinguish phonemes in different languages.

Preferably, the multilingual mixed speech recognition method, wherein step S2 specifically includes: step S21, forming an acoustic model according to multilingual speech data mixed with multiple different languages and multilingual mixed dictionary training; step S22, multilingual speech data Extract the speech features and use the acoustic model to perform frame alignment operations on the speech features to obtain the output label corresponding to each frame of the speech feature; Step S23: Use the speech feature as the input data of the acoustic recognition model and output the corresponding speech feature. The labels are used as output labels in the output layer of the acoustic recognition model to train to form an acoustic recognition model.

Preferably, in the multilingual mixed speech recognition method, the acoustic model is a hidden Markov-Gaussian mixed model.

Preferably, in the multilingual mixed speech recognition method, in step S23, after the acoustic recognition model is trained, the output layer of the acoustic recognition model is adjusted, which specifically includes: Step S231, calculating the priors of each language separately The probability and the prior probability of mute that is common to all kinds of languages; step S232, the posterior probability of each language is calculated and the posterior probability of mute is calculated; step S233, according to the priors of each language The probability and posterior probability, as well as the silent prior and posterior probability, adjust the output layer of the acoustic recognition model.

Preferably, in the multilingual mixed speech recognition method, in step S231, the prior probability of each language is calculated according to the following formula: ; among them, An output label used to represent the i-th state of the j-th language in the multilingual speech data; Used to represent the output labels in multilingual speech data as Probability of a priori Used to represent the output labels in multilingual speech data as total; An output label for representing the m-th state of silence in multilingual speech data; Used to represent the output labels in multilingual speech data as total; Used to represent the total number of states in the jth language in the multilingual speech data; The total number used to indicate the status of silence in multilingual speech data.

Preferably, in the multilingual mixed speech recognition method, in step S231, the prior probability of silence is calculated according to the following formula: ; among them, An output label for representing the m-th state of silence in multilingual speech data; Used to represent the output labels in multilingual speech data as Probability of a priori Used to represent the output labels in multilingual speech data as total; An output label used to represent the i-th state of the j-th language in the multilingual speech data; Used to represent the output labels in multilingual speech data as total; Used to represent the total number of states in the jth language in the multilingual speech data; It is used to represent the total number of muted states in the multilingual voice data; L is used to represent all languages in the multilingual voice data.

Preferably, in the multilingual mixed speech recognition method, in step S232, the posterior probability of each language is calculated according to the following formula: ; among them, An output label used to represent the i-th state of the j-th language in multilingual speech data; x is used to represent speech features; Used to represent the output labels in multilingual speech data as Post-test probability Input data for representing the i-th state of the j-th language in the multilingual speech data; Input data for representing the i-th state of silence; Used to represent the total number of states in the jth language in the multilingual speech data; It is used to indicate the total number of mute states in multilingual speech data; exp is used to indicate the exponential function calculation method.

Preferably, in the multilingual mixed speech recognition method, in step S232, the posterior probability of silence is calculated according to the following formula: ; among them, An output label used to represent the m-th state of silence in multilingual speech data; x is used to represent speech features; Used to represent the output labels in multilingual speech data as Post-test probability Input data for representing the i-th state of the j-th language in the multilingual speech data; Input data for representing the i-th state of silence; Used to represent the total number of states in the jth language in the multilingual speech data; It is used to indicate the total number of muted states in multilingual speech data; L is used to indicate all languages in multilingual speech data; exp is used to indicate the exponential function calculation method.

Preferably, in the multilingual mixed speech recognition method, in step S2, the acoustic recognition model is an acoustic model of a deep neural network.

Preferably, in the multilingual mixed speech recognition method, in step S3, n-Gram model training is used to form a language recognition model, or recursive neural network training is used to form a language recognition model.

Preferably, in the multilingual mixed speech recognition method, after the speech recognition system is formed, weight adjustments are first performed for different types of languages in the speech recognition system. The weight adjustment steps include: Step A1, determining each Step A2: Adjust the posterior probability of each language according to the posterior probability weight value to complete the weight adjustment.

Preferably, in the multilingual mixed speech recognition method, in step A2, weight adjustment is performed according to the following formula: ; among them, An output label used to represent the i-th state of the j-th language in multilingual speech data; x is used to represent speech features; Used to represent the output labels in multilingual speech data as Post-test probability A posteriori probability weight value used to represent the jth language in multilingual speech data; The output label used to represent the weighted multilingual speech data is Post-test probability.

The above technical solution has the beneficial effect of providing a multilingual mixed speech recognition method capable of supporting recognition of multilingual mixed speech, improving recognition accuracy and efficiency, and thereby improving performance of a speech recognition system.

In the following, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that, in the case of no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

The present invention is further described below with reference to the accompanying drawings and specific embodiments, but is not intended to limit the present invention.

Based on the above problems in the prior art, the present invention provides a multilingual mixed speech recognition method. The so-called mixed speech refers to a mixture of speech data of multiple different languages. For example, a user inputs a speech "I need a USB interface". The segment voice includes both Chinese voice and English proper noun "USB", then the segment voice is mixed voice. In other embodiments of the present invention, the foregoing mixed voice may also be a mixture of two or more voices, which is not limited herein.

In the above-mentioned multilingual mixed speech recognition method, it is first necessary to form a speech recognition system for identifying the mixed speech. The method for forming the speech recognition system is specifically shown in FIG. 1 and includes: Step S1, configuring a multilingual mixed dictionary including multiple different languages; Step S2, training according to the multilingual mixed dictionary and multilingual speech data including multiple different languages Forming an acoustic recognition model; Step S3, forming a language recognition model according to training of a multilingual text corpus including a plurality of different languages; Step S4, forming a speech recognition system using a multilingual mixed dictionary, an acoustic recognition model, and a language recognition model.

After the speech recognition system is formed, the speech recognition system can be used to recognize the mixed speech and output the corresponding recognition result.

Specifically, in this embodiment, the above-mentioned multilingual mixed dictionary is a mixed dictionary including a plurality of different languages, and the mixed dictionary is configured to a phoneme level. In a preferred embodiment of the present invention, a triphone model is used to configure the above-mentioned mixed dictionary, and a dictionary model that is more stable than word modeling can be obtained. In addition, since dictionaries in different languages may contain phonemes expressed by the same character, it is necessary to add a corresponding language mark before the phonemes of each language included in the multi-language mixed dictionary when configuring the mixed dictionary, in order to change a variety of different The phonemes of the language are distinguished.

For example, both the Chinese and English phoneme sets include phonemes such as "b" and "d". In order to distinguish them, a language tag is added in front of all the English phoneme sets (for example, “en” is added as a prefix) to distinguish the English phoneme set from the Chinese phoneme set, as shown in FIG. 2.

The above language mark can be empty. For example, if there are two languages in the mixed dictionary, only one language mark needs to be added to the two languages to distinguish the two languages. Similarly, if there are three languages in the mixed dictionary, only two languages need to be added with the language mark, that is, the three languages can be distinguished, and so on.

In the above mixed dictionary, it is also possible to add language tags only to phoneme sets of languages that may cause confusion. For example, a mixed dictionary includes Chinese, English, and other languages, and only the Chinese and English phoneme sets may have a problem of confusion. , So you only need to add a language tag in front of the English phoneme set.

In this embodiment, after forming a multilingual mixed dictionary, an acoustic recognition model is formed based on the mixed dictionary and multilingual speech data including multiple languages. Specifically, the multilingual voice data is prepared in advance and includes mixed voice data for training in multiple different languages, and the hybrid dictionary provides phonemes in different languages in the process of forming an acoustic recognition model. Therefore, in the process of training to form a multilingual acoustic recognition model, in order to obtain the triphoneme relationship of the mixed language phonemes, it is necessary to prepare the above-mentioned multilingual mixed multilingual speech data and perform the multilingual mixed dictionary formed according to the above.

In this embodiment, a multilingual text corpus is subsequently trained to form a language recognition model, and finally a multilingual mixed dictionary, an acoustic recognition model, and a language recognition model are included in a voice recognition system, and according to the voice The recognition system recognizes a mixed voice including multiple languages input by a user, and outputs a recognition result.

In this embodiment, after the above processing, the above-mentioned mixed speech recognition process is similar to the recognition process for monolingual speech in the prior art. The acoustic features in a piece of speech data are identified into corresponding phonemes or words through an acoustic recognition model. Sequence, and recognize the word sequence as a complete sentence through the language recognition model, thus completing the recognition process of mixed speech. The above identification process will not be repeated here.

In summary, in the technical solution of the present invention, a multilingual mixed dictionary including a plurality of languages is first formed according to a plurality of monolingual language dictionaries, and language marks of different languages are marked in the languages to distinguish them. Then, an acoustic recognition model is formed according to multilingual mixed speech data and a multilingual mixed dictionary training, and a language recognition model is formed according to multilingual mixed text corpus training. Then, a complete speech recognition system is formed according to the multilingual mixed dictionary, the acoustic recognition model and the language recognition model to recognize the multilingual mixed speech input by the user.

In a preferred embodiment of the present invention, as shown in FIG. 3, the above-mentioned step S2 specifically includes: step S21, forming an acoustic model according to multilingual voice data mixed with multiple different languages and multilingual mixed dictionary training; step S22, Multilingual speech data is used to extract speech features and use the acoustic model to perform frame alignment operations on the speech features to obtain the output label corresponding to each frame of speech features. Step S23, using the speech features as input data of the acoustic recognition model, and the speech The output labels corresponding to the features are used as output labels in the output layer of the acoustic recognition model to train and form an acoustic recognition model.

Specifically, in this embodiment, before training to form an acoustic recognition model, an acoustic model is first formed by training based on multilingual speech data mixed with multiple different languages. The acoustic model may be a Hidden Markov Model-Gaussian Mixture Model (HMM-GMM) model. Aiming at the problem of parameter re-evaluation in the three-phoneme modeling, the parameter sharing technique can be selected in the process of training to form an acoustic model, thereby reducing the parameter scale. The modeling technology of the acoustic model based on HMM-GMM is quite mature at present, and will not be repeated here.

In this embodiment, after the above-mentioned acoustic model is formed, it is necessary to use the acoustic model to perform frame alignment operation on the above-mentioned multilingual speech data, so that the speech features extracted from each frame of the multilingual speech data correspond to an output label. Specifically, after frame alignment, each frame of speech features corresponds to a GMM number. The output labels in the output layer of the acoustic recognition model are the labels corresponding to the speech features of each frame. Therefore, the number of output tags in the output layer of the acoustic recognition model is the number of GMM in the HMM-GMM model. One output node corresponds to one GMM.

In this embodiment, a voice feature is used as input data of the acoustic recognition model, and an output tag corresponding to the voice feature is used as an output tag in the output layer of the acoustic recognition model to train and form an acoustic recognition model.

Figure 4 shows the general structure of an acoustic recognition model according to an embodiment of the present invention. The acoustic recognition model is a deep neural network model built from a fully connected neural network structure. The neural network contains a total of 7 Fully connected neural network unit, each layer has 2048 nodes, and every two neural networks contain a sigmoid non-linear unit. The output layer is implemented with a softmax non-linear unit. S51 in FIG. 4 is used to represent the output layer of the acoustic recognition model, and L1, L2, and L3 respectively represent output labels on the output layer that are associated with different kinds of languages.

In a preferred embodiment of the present invention, after the acoustic recognition model is trained in the above step S23, operations such as adjusting and prioring the output layer of the acoustic recognition model for multiple languages need to be performed, as shown in FIG. 5, It includes: Step S231, calculating a priori probability of each language, and calculating a priori probability of mute common to all kinds of languages; Step S232, separately calculating a posterior probability of each language, and calculating a mute Post-test probability; Step S233: Adjust the output layer of the acoustic recognition model according to the pre-test probability and post-test probability of each language, and the pre-test probability and post-test probability of silence.

Specifically, in a preferred embodiment of the present invention, when an acoustic recognition model is used for speech recognition, for a given speech feature, the character string of the output result is generally determined by the following formula: (1) Among them, A string used to represent the output result, w is a possible string, x is the input speech feature, P (w) is used to represent the probability of the above-mentioned language recognition model, and P (x | w) is used to represent the above-mentioned acoustic recognition model Chance.

Then the above P (x | w) can be further expanded as: (2) Among them, x _{t is} used to represent the speech features input at time t, q _{t is} used to represent the state of the three phonemes bound at time t, π (q ₀ ) is used to represent the probability distribution of the initial state q ₀ , P (x _t | q _t ) is used to indicate the probability that the speech feature is x _t in the q _t state.

Then the above P (x _t | q _t ) can be further expanded as: (3) where P (x _t | q _t ) is the posterior probability of the output layer of the above-mentioned acoustic recognition model, P (q _t ) is the prior probability of the above-mentioned acoustic recognition model, and P (x _t ) represents x probability of _t . P (x _t ) is not related to the string sequence and can be ignored.

According to the above formula (3), it can be obtained that the string of the output result can be adjusted by calculating the prior probability and the posterior probability of the output layer of the acoustic recognition model.

In a preferred embodiment of the present invention, the prior probability P (q) of the neural network is usually calculated by the following formula: (4) Among them, Count (q ⁱ ) is used to represent the total number of tags q ⁱ in the multilingual speech data, and N is used to represent the total number of all output tags.

In a preferred embodiment of the present invention, since the amount of training voice data for different kinds of languages may be different, the above-mentioned prior probability cannot be calculated uniformly, and needs to be calculated separately according to different kinds of languages.

In a preferred embodiment of the present invention, in step S231, the prior probability of each language is first calculated and the prior probability of mute common to all kinds of languages is calculated.

First, the prior probability of each language is calculated according to the following formula: (5) Among them, An output label used to represent the i-th state of the j-th language in the multilingual speech data; Used to represent the output labels in multilingual speech data as Probability of a priori Used to represent the output labels in multilingual speech data as total; An output label for representing the m-th state of silence in multilingual speech data; Used to represent the output labels in multilingual speech data as total; Used to represent the total number of states in the jth language in the multilingual speech data; The total number used to indicate the status of silence in multilingual speech data.

Then, the prior probability of silence is calculated according to the following formula: ; (6) where Used to represent the output labels in multilingual speech data as A priori probability; L is used to represent all languages in multilingual speech data.

In the preferred embodiment of the present invention, after calculating the prior probability of each of the languages and the prior probability of silence, the posterior probability of the acoustic recognition model is continuously calculated. The posterior probability P (q _i | x) of the neural network output is usually calculated by the output layer. When the output layer is implemented as a softmax non-linear unit, the posterior probability is usually calculated according to the following formula: (7) Among them, y ^{i is} used to represent the input value in the i-th state, and N is the number of all states.

Similarly, in the acoustic recognition model, the uneven amount of training data of different kinds of languages will cause the distribution of the state value calculation results of different kinds of languages to be uneven, so the post-test probability still needs to be calculated separately for different kinds of languages.

Then, in a preferred embodiment of the present invention, in step S232, the posterior probability of each language is calculated according to the following formula: (8) where x is used to represent the speech features; Used to represent the output labels in multilingual speech data as Post-test probability Input data for representing the i-th state of the j-th language in the multilingual speech data; Input data used to represent the i-th state of silence; exp is used to indicate how the exponential function is calculated.

In a preferred embodiment of the present invention, in step S232, the posterior probability of silence is calculated according to the following formula: ; (9) where Used to represent the output labels in multilingual speech data as Post-test probability.

In the present invention, the a priori probability and the a priori probability in each language and mute state can be calculated by using the improved formulas (6) to (9), so that the acoustic recognition model can meet the output requirements of multilingual mixed modeling , Can more accurately describe each language and mute state. It should be noted that after the above formula is adjusted, the sum of the probabilities of the first and the second probabilities is no longer 1.

In a preferred embodiment of the present invention, in the above step S3, a language recognition model may be formed by training with an n-Gram model, or a language recognition model may be formed by training using a recurrent neural network. The above multilingual text corpus needs to include multilingual separate text corpora and multilingual mixed text data.

In a preferred embodiment of the present invention, after forming a speech recognition system, first perform weight adjustment on different kinds of languages in the speech recognition system. The steps for performing weight adjustment are shown in FIG. 6, including: Step A1, based on real speech data Determine the posterior probability weight value of each language separately; Step A2, adjust the posterior probability of each language according to the posterior probability weight value to complete the weight adjustment.

Specifically, in this embodiment, after the above-mentioned speech recognition system is formed, since the problem of imbalanced amount of training data may occur during training, a language with a larger amount of data will have a relatively large prior probability. The recognition probability is the posterior probability divided by the prior probability, so the actual recognition probability of a language with more training data is rather small, which will cause the recognition result of the recognition system to tend to recognize a certain language and fail to recognize another. A language that causes bias in recognition results.

In order to solve this problem, before the above speech recognition system is put into practical use, it is necessary to use real data as a development set to measure it to adjust the weight of each language. The above weight adjustment is usually applied to the posterior probability of the output of the acoustic recognition model, so its formula is as follows: ; (10) where An output label used to represent the i-th state of the j-th language in multilingual speech data; x is used to represent speech features; Used to represent the output labels in multilingual speech data as Post-test probability It is used to represent the posterior probability weight value of the j-th language in the multilingual speech data, and the posterior probability weight value is determined by actually measuring the acoustic recognition model by using the development set composed of the real data.

The output label used to represent the weighted multilingual speech data is Post-test probability.

After the weight adjustment is made, the speech recognition system can obtain a good recognition effect in different application scenarios.

In a preferred embodiment of the present invention, for a Chinese-English mixed speech recognition system, after the actual data is measured, the Chinese posterior probability weight value can be set to 1.0, and the English posterior probability weight value is set in English. Set it to 0.3 and set the mute posterior probability weight to 1.0.

In other embodiments of the present invention, the above-mentioned posterior probability weight value may be repeatedly adjusted by using development sets composed of different real data multiple times to finally determine the optimal value.

The above are only preferred embodiments of the present invention, and therefore do not limit the implementation and protection scope of the present invention. For those skilled in the art, they should be able to realize that equivalent substitutions and Obvious changes should be included in the protection scope of the present invention.

S1‧‧‧Step 1

S2‧‧‧Step 2

S3‧‧‧Step 3

S4‧‧‧Step 4

S21‧‧‧Step 21

S22‧‧‧Step 22

S23‧‧‧Step 23

Output layer of S51‧‧‧acoustic recognition model

L1‧‧‧ output label

L2‧‧‧ output label

L3‧‧‧ output label

S231‧‧‧Step 231

S232‧‧‧Step 232

S233‧‧‧Step 233

A1‧‧‧Step A1

A2‧‧‧Step A2

FIG. 1 is a schematic diagram of an overall process of forming a speech recognition system in a multilingual mixed speech recognition method in a preferred embodiment of the present invention; FIG. 2 is a schematic view of a multilingual mixed dictionary in a preferred embodiment of the present invention FIG. 3 is a schematic flowchart of training to form an acoustic recognition model based on FIG. 1 in a preferred embodiment of the present invention; FIG. 4 is a schematic structural diagram of an acoustic recognition model in a preferred embodiment of the present invention; 5 is a schematic flowchart of adjusting an output layer of an acoustic recognition model based on FIG. 2 in a preferred embodiment of the present invention; FIG. 6 is a voice recognition system in a preferred embodiment of the present invention Schematic diagram of the process of weight adjustment.

Claims (11)

A method for multilingual mixed speech recognition, first forming a speech recognition system for identifying multilingual mixed speech. The method for forming the speech recognition system includes: Step S1, configuring a multilingual mixed dictionary including multiple different languages; Step S2 Forming an acoustic recognition model based on the multilingual mixed dictionary and multilingual speech data including multiple different languages; the step S2 specifically includes: step S21, forming an acoustic model based on the multilingual speech data and the multilingual mixed dictionary training Step S22, extract the speech features from the multilingual speech data, and use the acoustic model to perform frame alignment operations on the speech features to obtain the output label corresponding to the speech feature for each frame; Step S23, use the speech feature as The input data of the acoustic recognition model and the output tags corresponding to the speech features are used as output tags in the output layer of the acoustic recognition model to train to form the acoustic recognition model. In step S23, the acoustic recognition model is performed. After training, the output layer of the acoustic recognition model The adjustment includes: step S231, calculating a priori probability of each language, and calculating the mute priori probability, common to all types of languages; step S232, calculating a posterior probability of each language, and Step S233: Adjust the output layer of the acoustic recognition model according to the prior and posterior probability of each language, and the prior and posterior probability of the mute. Step S3. A language recognition model is formed according to training of a multilingual text corpus including a plurality of different languages. In step S4, the multilingual mixed dictionary, the acoustic recognition model, and the language recognition model are used to form the speech recognition system. Subsequently, the speech recognition system is adopted The mixed voice is recognized and the corresponding recognition result is output. In step S231, the prior probability of each language is calculated according to the following formula: among them, The output label used to represent the i-th state of the j-th language in the multilingual speech material; P ( ) Is used to indicate that the output label in the multilingual voice data is Probability of priors; Count ( ) Is used to indicate that the output label in the multilingual voice data is total; The output label used to indicate the m-th state of the mute in the multilingual voice data; Count ( ) Is used to indicate that the output label in the multilingual voice data is M _{j is} used to indicate the total number of states in the j-th language of the multilingual voice material; M _{sil is} used to indicate the total number of the muted states in the multilingual voice material. The multilingual mixed speech recognition method as described in claim 1, wherein in step S1, a three-phoneme modeling method is adopted, and the multilingual mixed dictionary is configured according to a single language dictionary corresponding to each different language. The multilingual mixed speech recognition method according to item 1 of the claim, wherein in step S1, the multilingual mixed dictionary is configured in a three-phoneme modeling manner; when the multilingual mixed dictionary is configured, the multilingual A corresponding language mark is added before the phonemes of each language included in the mixed dictionary to distinguish the phonemes of multiple different languages. The multilingual mixed speech recognition method according to claim 1, wherein the acoustic model is a hidden Markov-Gaussian mixed model. The multilingual mixed speech recognition method according to item 1 of the claim, wherein, in step S231, the prior probability of the silence is calculated according to the following formula: among them, The output label used to indicate the m-th state of silence in the multilingual voice data; P ( ) Is used to indicate that the output label in the multilingual voice data is Probability of priors; Count ( ) Is used to indicate that the output label in the multilingual voice data is total; The output label used to represent the i-th state of the j-th language in the multilingual voice data; Count ( ) Is used to indicate that the output label in the multilingual voice data is M _{j is} used to indicate the total number of states in the j-th language in the multilingual voice material; M _{sil is} used to indicate the total number of muted states in the multilingual voice material; L is used to indicate the multi language Language All materials in the audiobook. The multilingual mixed speech recognition method according to item 1 of the claim, wherein in step S232, the posterior probability of each language is calculated according to the following formula: among them, The output label used to represent the i-th state of the j-th language in the multilingual speech material; x is used to represent the speech feature; P ( | X ) is used to indicate that the output label in the multilingual voice data is Y ⁱ _{j is} used to represent the input data of the i-th state of the j-th language in the multilingual voice data; y ⁱ _{sil is} used to represent the input data of the m-th state of the mute; M _{j is} used to indicate the total number of states in the j-th language in the multilingual voice material; M _{sil is} used to indicate the total number of the muted states in the multilingual voice material; exp is used to indicate the exponential function calculation method. The multilingual mixed speech recognition method according to item 1 of the claim, wherein in step S232, the mute posterior probability is calculated according to the following formula: among them, The output label used to represent the m-th state of silence in the multilingual voice material; x is used to represent the voice feature; P ( | X ) is used to indicate that the output label in the multilingual voice data is Y ⁱ _{j is} used to represent the input data of the i-th state of the j-th language in the multilingual voice data; y ⁱ _{sil is} used to represent the input data of the m-th state of the mute; M _{j is} used to indicate the total number of states in the j-th language of the multilingual voice material; M _{sil is} used to indicate the total number of the muted states in the multilingual voice material; L is used to indicate the All languages; exp is used to indicate how the exponential function is calculated. The multilingual mixed speech recognition method according to claim 1, wherein in step S2, the acoustic recognition model is an acoustic model of a deep neural network. The multi-language mixed speech recognition method according to claim 1, wherein in step S3, the language recognition model is formed by training using an n-Gram model, or the language recognition model is formed by training using a recurrent neural network. The multi-language mixed speech recognition method according to claim 1, wherein after the speech recognition system is formed, weight adjustments of different kinds of languages in the speech recognition system are first performed; the step of performing the weight adjustment includes: step A1 , Determine the posterior probability weight value of each language according to the real voice data; step A2, adjust the posterior probability of each language according to the posterior probability weight value to complete the weight adjustment. The multilingual mixed speech recognition method according to claim 10, wherein in step A2, the weight adjustment is performed according to the following formula: among them, The output label used to represent the i-th state of the j-th language in the multilingual speech material; x is used to represent the speech feature; P ( | X ) is used to indicate that the output label in the multilingual voice data is A posterior probability; a _{j is} used to represent the posterior probability weight value of the j-th language in the multilingual speech data; ( | X ) is used to indicate that the output label in the multilingual voice material after the weight adjustment is Post-test probability.