CN111369978A - A data processing method, device and device for data processing - Google Patents

Abstract

The embodiment of the invention provides a data processing method and device and a device for data processing. The method specifically comprises the following steps: determining the language type of a voice frame in the voice information according to the multi-language acoustic model; the multi-language acoustic model is obtained by training according to acoustic data of at least two language types; decoding the voice frame according to a decoding network corresponding to the language type of the voice frame to obtain a first decoding result of the voice frame; and determining the recognition result corresponding to the voice information according to the first decoding result. The embodiment of the invention can improve the accuracy of voice recognition.

Description

A data processing method, device and device for data processing

technical field

The present invention relates to the field of computer technology, and in particular, to a data processing method, device and device for data processing.

Background technique

Speech recognition technology, also known as ASR (Automatic Speech Recognition), whose goal is to convert lexical content in speech into computer-readable input, such as keystrokes, binary codes, or sequences of characters.

In daily language expression, there may be mixed expressions of multiple languages. Taking the mixed expression of Chinese and English as an example, the user can use English words and phrases interspersed in the process of expressing in Chinese. For example, "I bought the latest iPhone", "Let me have a song Yesterday once more".

However, the current speech recognition technology is relatively accurate for the speech recognition of a single language, but in the case that the speech contains multiple languages, the recognition accuracy is obviously reduced.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a data processing method, an apparatus, and a data processing apparatus, which can improve the accuracy of speech recognition when the speech contains multiple languages.

In order to solve the above problems, an embodiment of the present invention discloses a data processing method, the method includes:

Determine the language type of the speech frame in the speech information according to the multilingual acoustic model; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types;

According to the decoding network corresponding to the language type of the speech frame, the speech frame is decoded to obtain the first decoding result of the speech frame;

According to the first decoding result, the recognition result corresponding to the voice information is determined.

On the other hand, an embodiment of the present invention discloses a data processing apparatus, and the apparatus includes:

a type determination module, used for determining the language type of the speech frame in the speech information according to the multilingual acoustic model; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types;

a first decoding module, configured to decode the speech frame according to a decoding network corresponding to the language type of the speech frame to obtain a first decoding result of the speech frame;

A result determination module, configured to determine a recognition result corresponding to the voice information according to the first decoding result.

In another aspect, an embodiment of the present invention discloses an apparatus for data processing, comprising a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more programs Execution of the one or more programs by the above processor includes instructions for:

Determine the language type of the speech frame in the speech information according to the multilingual acoustic model; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types;

According to the decoding network corresponding to the language type of the speech frame, the speech frame is decoded to obtain the first decoding result of the speech frame;

According to the first decoding result, the recognition result corresponding to the voice information is determined.

In yet another aspect, an embodiment of the present invention discloses a machine-readable medium with instructions stored thereon that, when executed by one or more processors, cause an apparatus to perform one or more of the data processing methods described above.

The embodiments of the present invention include the following advantages:

In this embodiment of the present invention, a multilingual model can be obtained by training acoustic data of at least two language types. Through the multilingual acoustic model, the language types of the speech frames in the speech information can be determined. Therefore, the speech information includes multiple language types. In this case, the embodiment of the present invention can accurately distinguish the voice frames of different language types in the voice information, and can decode the voice frames according to the decoding network of the corresponding language type, so as to obtain the first decoding result of the voice frame. The result is obtained by decoding the decoding network corresponding to the speech type of the speech frame, which can ensure the accuracy of decoding, and thus can improve the accuracy of speech recognition.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the drawings that are used in the description of the embodiments of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention. , for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

Fig. 1 is the step flow chart of a kind of data processing method embodiment of the present invention;

2 is a structural block diagram of an embodiment of a data processing apparatus of the present invention;

FIG. 3 is a block diagram of an apparatus 800 for data processing of the present invention; and

FIG. 4 is a schematic structural diagram of a server in some embodiments of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Method embodiment

Referring to FIG. 1, a flow chart of steps of an embodiment of a data processing method of the present invention is shown, which may specifically include the following steps:

Step 101: Determine the language type of the speech frame in the speech information according to the multilingual acoustic model; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types;

Step 102, according to the decoding network corresponding to the language type of the speech frame, decode the speech frame to obtain the first decoding result of the speech frame;

Step 103: Determine a recognition result corresponding to the voice information according to the first decoding result.

The data processing method according to the embodiment of the present invention can be used in a scenario of recognizing voice information including at least two language types, and the data processing method can be applied to an electronic device, where the electronic device includes but is not limited to: a server, a smart phone, a Tablet PC, e-book reader, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 (Moving Picture Experts Group Audio Layer IV) player , laptops, car computers, desktop computers, set-top boxes, smart TVs, wearables, and more.

It can be understood that the manner of acquiring the voice information to be recognized is not limited in this embodiment of the present invention. For example, the electronic device may acquire the voice information to be recognized from a client or a network through a wired connection or a wireless connection, or , the voice information to be recognized can be obtained by real-time recording of the electronic device, or the voice information to be recognized can also be obtained according to the instant messaging message obtained in the instant messaging application.

In this embodiment of the present invention, the speech information to be recognized may be divided into multiple speech frames according to the preset window length and frame shift, wherein each speech frame may be a speech segment, and further Speech information is decoded frame by frame. If the voice information to be recognized is analog voice information (for example, a recording of a user's call), the analog voice information needs to be converted into digital voice information first, and then the voice information is segmented.

The window length can be used to represent the duration of each frame of speech segment, and the frame shift can be used to represent the time difference between adjacent frames. For example, when the window length is 25ms and the frame shift is 15ms, the first frame of voice segment is 0-25ms, the second frame of voice segment is 15-40ms, and so on, the segmentation of the voice information to be recognized can be realized. It can be understood that the specific window length and frame shift can be set by themselves according to actual requirements, which are not limited in this embodiment of the present invention.

Optionally, before segmenting the to-be-recognized speech information, the electronic device may further perform noise reduction processing on the to-be-recognized speech information, so as to improve the subsequent processing capability of the speech information.

In the embodiment of the present invention, speech information may be input into a pre-trained multilingual acoustic model, and a speech recognition result may be obtained based on the output of the multilingual acoustic model. The multilingual acoustic model may be a classification model incorporating various neural networks. The neural network includes but is not limited to at least one of the following or a combination, superposition and nesting of at least two: CNN (Convolutional Neural Network, convolutional neural network), LSTM (Long Short-TermMemory, long short-term memory) network, RNN (Simple Recurrent Neural Network, recurrent neural network), attention neural network, etc.

In order to improve the accuracy of recognizing speech information containing multiple language types, the embodiment of the present invention obtains a multi-language acoustic model by pre-training according to acoustic data of at least two language types. According to the multi-language acoustic model, it is possible to determine the The language type of the speech frame, so the speech frame can be decoded according to the decoding network corresponding to the language type to obtain the first decoding result corresponding to the speech frame, and then the speech frame can be determined according to the first decoding result. The identification result corresponding to the information.

It can be understood that the embodiments of the present invention do not limit the number of language types and language types included in the acoustic data for training the multilingual acoustic model. For ease of description, in the embodiments of the present invention, speech information including two language types, Chinese and English, is used as an example for description, that is, the multilingual acoustic model may be obtained by training according to the collected Chinese acoustic data and English acoustic data. . Of course, it is also possible to collect acoustic data of more than two language types, such as Chinese, English, Japanese, German and other language types, to train a multilingual acoustic model. For application scenarios of two or more language types, the implementation process is similar to that of the two language types, and it is sufficient to refer to each other.

The decoding network in this embodiment of the present invention may include decoding networks corresponding to at least two language types. For example, in the scenario of recognizing speech information mixed in Chinese and English, a Chinese decoding network and an English decoding network may be constructed respectively. Specifically, Chinese text corpus can be collected to train a Chinese language model, and a Chinese decoding network can be constructed based on knowledge sources such as a Chinese language model and a Chinese pronunciation dictionary; similarly, an English language model can be trained by collecting English text corpus, and the English language model and English pronunciation dictionary and other knowledge sources to build an English decoding network.

In the process of decoding the speech information frame by frame, if the language type of the speech frame is determined to be Chinese according to the multilingual acoustic model, the speech frame can be decoded according to the Chinese decoding network. If the language of the speech frame is determined according to the multilingual acoustic model If the type is English, the speech frame can be decoded according to the English decoding network.

In an application example of the present invention, it is assumed that the voice information to be recognized is "I like apple". Specifically, first, the language type of the first frame of speech in the speech information can be determined according to the multilingual acoustic model. Assuming that the language type of the first frame of speech is Chinese, the first frame of speech can be decoded according to the Chinese decoding network. frame decoding to obtain the first decoding result of the first frame of speech frame; then determine the language type of the second frame of speech frame according to the multilingual acoustic model, and input the second frame of speech frame into the decoding network corresponding to its language type Decoding is performed to obtain the first decoding result of the second frame of speech; by analogy, if the language type of the mth speech frame is determined to be English according to the multilingual acoustic model, then the mth frame can be decoded according to the English decoding network. The speech frame is decoded to obtain the first decoding result of the mth speech frame, until the decoding of the last speech frame is completed; finally, the recognition result of the speech information can be obtained according to the first decoding result of each speech frame, such as this The recognition result may include text information "I like apple" as follows.

It can be seen that in the embodiment of the present invention, the language type of the speech frame in the speech information can be determined by using the trained multilingual acoustic model, so that the speech frame can be decoded according to the decoding network of the corresponding language type, so as to obtain a more accurate recognition result.

In an optional embodiment of the present invention, determining the language type of the speech frame in the speech information according to the multilingual acoustic model may specifically include:

Step S11, according to the multilingual acoustic model, determine the posterior probability of the speech frame corresponding to each state; wherein, there is a corresponding relationship between the state and the language type;

Step S12, according to the posterior probability of the speech frame corresponding to each state and the language type corresponding to each state, determine the probability ratio of the posterior probability of the speech frame corresponding to the state of each language type;

Step S13: Determine the language type of the speech frame according to the probability ratio.

The multilingual acoustic model can convert the input features of the speech frame into the posterior probability of each state, and the state can specifically be an HMM (Hidden Markov Model, Hidden Markov Model) state. A state can correspond to a phoneme, multiple phonemes can correspond to a word, and multiple words can form a sentence.

For example, it is assumed that the multilingual acoustic model can output posterior probabilities corresponding to (M1+M2) states in the output layer, wherein the language type corresponding to the M1 states can be Chinese, and the language type corresponding to the M2 states can be English.

The voice frame is input into the multilingual acoustic model, and the posterior probability of the voice frame corresponding to each state can be output. According to the posterior probability of the voice frame corresponding to each state and the language type corresponding to each state, the voice frame can be determined. The posterior probability of is corresponding to the probability ratio of the state of each language type, such as the probability ratio of the corresponding Chinese state and the English state in the posterior probability of the speech frame, and then the language type corresponding to the speech frame can be determined according to the probability ratio.

For example, the probability value obtained by adding the posterior probabilities of M1 Chinese states is p1, and the probability value obtained by adding the posterior probabilities of M2 English states is p2, and p1+p2=1. If p1 is greater than p2, it means that the posterior probability of the speech frame corresponds to a higher probability of the Chinese state, and the language type of the speech frame can be determined to be Chinese; otherwise, the language type of the speech frame can be determined to be English.

However, in the speech information mixed with Chinese and English, the posterior probability of English is usually small and rarely exceeds 0.5. Therefore, in order to reduce misjudgment, the embodiment of the present invention can set a preset threshold, by comparing the probability ratio with all The preset threshold is compared to determine the language type of the speech frame.

Taking the mixture of Chinese and English as an example, it is assumed that the probability ratio of the posterior probability of the speech frame corresponding to the English state and the Chinese state is p2/p1. If p2/p1 exceeds a preset threshold (such as 0.25), the language type of the speech frame can be determined. In the same way, the probability ratio of the posterior probability of the speech frame corresponding to the Chinese state and the English state is p1/p2. If p1/p2 exceeds 4, it can be determined that the language type of the speech frame is Chinese. The preset threshold may be adjusted according to experiments, and it can be understood that a specific value of the preset threshold is not limited in this embodiment of the present invention.

Of course, since p1+p2=1, p2/p1>0.25 is equivalent to p2>0.2, so the judgment can also be made simply by the value of p1 or p2.

In a specific application, if the user frequently switches the language type, or the voice information is short, judging the language type of the voice frame according to a single frame of voice frame may lead to a judgment error.

In order to improve the accuracy of determining the language type of the speech frame, in an optional embodiment of the present invention, the probability ratio of the state of each language type corresponding to the posterior probability of the consecutive speech frames within the preset window length of the speech frame may be to determine the language type of the speech frame.

It can be understood that the embodiment of the present invention does not limit the specific value of the preset window length. For example, the preset window length can be set to include a time length of 10 consecutive speech frames. Specifically, 10 consecutive speech frames including the speech frame can be obtained, and the posterior probability of each of the 10 speech frames corresponding to the probability ratio p2/p1 of the English state and the Chinese state is calculated respectively, and then the 10 p2/p1 sums are averaged, and if the average value exceeds a preset threshold of 0.25, it can be determined that the language type of the speech frame is English, so as to avoid the probability of misjudgment through single-frame judgment, and thus can improve the determination of The accuracy of the language type of the speech frame.

In an optional embodiment of the present invention, before determining the language type of the speech frame in the speech information according to the multilingual acoustic model, the method may further include:

Step S21, determining the target language type from the at least two language types;

Step S22, decoding each speech frame in the speech information according to the decoding network corresponding to the target language type, to obtain the second decoding result of each speech frame;

After determining the language type of the speech frame in the speech information according to the multilingual acoustic model, the method may further include:

From the voice frame of the voice information, determine the target voice frame, and determine the second decoding result of the target voice frame; wherein, the language type of the target voice frame is a non-target language type;

The decoding network corresponding to the language type of the speech frame to decode the speech frame to obtain the first decoding result of the speech frame may specifically include: a decoding network corresponding to the language type of the target speech frame. a decoding network, which decodes the target speech frame to obtain a first decoding result of the target speech frame;

The determining the recognition result corresponding to the speech information according to the first decoding result may specifically include: replacing the second decoding result of the target speech frame with the first decoding result of the language type corresponding to the target speech frame , and use the replaced second decoding result as the recognition result corresponding to the voice information.

In specific applications, users usually use a mixture of two types of languages to express, and most of the sentences use one of the language types, and only a small part of the sentences will be interspersed with the other language type. In addition, when the speech information is short, for example, the speech information contains only one English word, since a word has no context information during decoding, the decoding result may be inaccurate.

Therefore, in this embodiment of the present invention, a target language type may be determined from the at least two language types, and the target language type may be the main language used in mixed language expression. For example, it may be determined that the target language type is Chinese. In the process of decoding the voice information, each frame of the voice frame in the voice information is decoded according to the Chinese decoding network to obtain the second decoding result (such as R1) corresponding to each frame of voice frame, and the R1 is Chinese decoding result. Since the second decoding result is obtained by decoding a complete piece of speech information, during the decoding process of each speech frame, its corresponding context information can be referred to, so the accuracy of the second decoding result can be improved.

After the decoding network corresponding to the target language type completes the decoding of all the speech frames in the speech information, the target speech frame can be determined from the speech frames of the speech information; wherein, the language type of the target speech frame is a non-target language type. For example, for speech information mixed with Chinese and English, if it is determined that the target language type is Chinese, then English is the non-target language type, that is, it can be determined from the speech information that the speech frame whose language type is English is the target speech frame, and the The target speech frame corresponds to the first decoding result of English (eg R2), and the R2 is obtained by decoding the target speech frame according to the English decoding network, that is, the R2 is the English decoding result. Finally, replace the corresponding R1 with R2, and the recognition result corresponding to the speech information can be obtained.

In an application example of the present invention, it is assumed that the speech information to be recognized is "I like apple" and the target language type is Chinese. Specifically, firstly, input the voice information into the multilingual acoustic model to obtain the state posterior probability sequence corresponding to each frame of voice frame, and decode the posterior probability sequence of the Chinese state of each frame of voice frame according to the Chinese decoding network, In order to obtain the second decoding result of each frame of speech frame, it is assumed that the second decoding result of the speech information is "I like love broken"; then, according to the posterior probability of each speech frame corresponding to each state, and the corresponding Language type, determine the language type of each speech frame, and determine the speech frame whose language type is English as the target speech frame; and then decode the target speech frame according to the English decoding network, to obtain the first English corresponding to the target speech frame. The decoding result is assumed to be "apple"; finally, replace the "aipo" corresponding to "apple" in the second decoding result "I like lovepo" with "apple", and the second decoding result after replacement can be obtained as Text like: "I like apple".

It should be noted that, in this embodiment of the present invention, for a speech frame whose language type is the target language type, the first decoding result and the second decoding result are the same. For example, in the above example, the speech frame corresponding to "I like", If the language type is Chinese and the target language type is also Chinese, the first decoding result and the second decoding result of the speech frame corresponding to "I like" are both the text "I like".

In an optional embodiment of the present invention, the first decoding result and the second decoding result may include: time boundary information corresponding to the speech frame;

The replacing the second decoding result of the target speech frame with the first decoding result of the language type corresponding to the target speech frame may specifically include:

Step S31, from the second decoding result of the target speech frame, determine the replaced result; wherein, the replaced result coincides with the time boundary of the first decoding result of the language type corresponding to the target speech frame;

Step S32: Replace the replaced result with the first decoding result of the language type corresponding to the target speech frame.

In order to ensure that the second decoding result of the target speech frame can be accurately replaced with the first decoding result of the language type corresponding to the target speech frame, the first decoding result and the second decoding result in this embodiment of the present invention may include: the corresponding speech frame time boundary information.

For example, in the above example, for the second decoding result "I like love breaking", each word includes the time boundary information of the speech frame corresponding to the word. According to the time boundary information, from the second decoding result, Determine the replaced result, so that the replaced result coincides with the time boundary of the first decoding result of the language type corresponding to the target speech frame. According to the above example, the first decoding result corresponding to the language type of the target speech frame is " apple", if it is determined that the replaced result that coincides with the time boundary information of "apple" in the second decoding result "I like Aipo" is "Aipo", then the "Aipo" in "I like Aipo" can be replaced. " is replaced with "apple", and the decoded result after replacement is "I like apple".

In an optional embodiment of the present invention, the decoding network may specifically include: a general decoding network and a specialized decoding network; wherein, the general decoding network may include: a language model trained according to a general text corpus ; The professional decoding network may include: a language model obtained by training the text corpus in the preset field;

The decoding network corresponding to the language type of the speech frame to decode the speech frame to obtain the first decoding result of the speech frame may specifically include:

Step S41, respectively decode the voice frame according to the general decoding network and the professional decoding network, so as to obtain the first score of the voice frame corresponding to the general decoding network, and the voice frame corresponding to the professional decoding network. the second score of the decoding network;

Step S42: Use the decoding result with the highest score among the first score and the second score as the first decoding result of the speech frame.

In specific applications, the decoding network usually has a good decoding effect for the voice of the user's daily communication. However, for the voice in some professional fields, such as the voice in the medical professional field, it usually contains more medical professional vocabulary, such as " Aspirin", "Parkinson's disease", etc., will affect the decoding effect.

To solve the above problem, the decoding network in the embodiment of the present invention may include a general decoding network and a specialized decoding network. Among them, the general decoding network can be a general decoding network used in the user's daily communication process. The general decoding network can include: a language model trained according to a general text corpus. Therefore, the general decoding network can be used for most users' daily speech. identify. The professional decoding network can be a decoding network specially customized for the professional field. The professional decoding network can include: a language model trained according to the text corpus in the preset field; the preset field can be the medical field, the legal field, the computer field, etc. any field.

For example, in a medical seminar, the speaker may use a lot of mixed sentences in Chinese and English, and also use a large number of medical professional vocabulary, the embodiment of the present invention can recognize the speaker's speech as text in real time, and display it on the on the big screen for participants to watch.

Specifically, the speech of the speaker can be decoded frame by frame according to the general decoding network and the professional decoding network respectively, so as to obtain the first score of the speech frame corresponding to the general decoding network, and the first score of the professional decoding network for the speech frame. Two scores are obtained, and the decoding result with the highest score among the first score and the second score is used as the first decoding result of the speech frame.

It can be understood that the decoding network in this embodiment of the present invention may include a plurality of decoding networks corresponding to different language types, and the decoding network of each language type may further include a general decoding network and a professional decoding network corresponding to the language type. Therefore, the embodiment of the present invention can supplement or correct the decoding result of the general decoding network through the professional decoding network, and can improve the accuracy of decoding when the speech information contains vocabulary in the professional field.

It can be understood that the embodiment of the present invention does not limit the training method for training the multilingual acoustic model. In an optional embodiment of the present invention, each of the acoustic data of the at least two language types corresponds to at least two language types.

Specifically, in this embodiment of the present invention, mixed acoustic data including at least two language types can be collected to train a multilingual acoustic model, where the mixed acoustic data refers to each of the data corresponding to at least two language types. For example, the speech corresponding to "I like apple" may be a mixed acoustic data.

To train a multilingual acoustic model based on mixed acoustic data, it is necessary to merge similar pronunciation units in different types of languages to generate a pronunciation dictionary suitable for mixed languages. However, in the process of merging pronunciation units, it may bring some error. In addition, mixed acoustic data containing at least two language types is usually characterized by sparse data and difficult to collect, therefore, it will affect the accuracy of multilingual acoustic model recognition.

To solve the above problem, in an optional embodiment of the present invention, each of the acoustic data of the at least two language types corresponds to one language type.

Specifically, the embodiment of the present invention can separately collect monolingual acoustic data corresponding to at least two language types, and train a multilingual acoustic model according to a training data set composed of monolingual data corresponding to each language type. For example, the speech corresponding to "the weather is fine today" may be a monolingual acoustic data, and the speech corresponding to "What's the weather like today" may also be a monolingual acoustic data.

In an optional embodiment of the present invention, the training step of the multilingual acoustic model may specifically include:

Step S51, according to the collected acoustic data of at least two language types, respectively train the monolingual acoustic model corresponding to each language type;

Step S52, according to the single-language acoustic model, state annotating the acoustic data of the at least two language types respectively, wherein there is a corresponding relationship between the state and the language type;

Step S53: Train a multilingual acoustic model according to the marked data set composed of acoustic data of at least two language types.

Specifically, a single-language acoustic model NN1 corresponding to Chinese can be trained according to the collected Chinese acoustic data L1, wherein the language type corresponding to each data in L1 is Chinese. The number of HMM binding states for Chinese voice can be set to the number of nodes in the output layer of the NN1 network, such as M1. The output of the single-language acoustic model may include: state probabilities corresponding to one language type, that is, the state probabilities of the M1 nodes in the network output layer all correspond to the Chinese language type.

Similarly, a single-language acoustic model NN2 corresponding to English can be trained according to the collected English acoustic data L2, wherein the language type corresponding to each data in L2 is English. You can set the number of HMM binding states of English speech to the number of nodes in the output layer of the NN2 network, such as M2, and the state probabilities of M2 nodes all correspond to the English language type.

Then, according to the NN1 and NN2 obtained by training, the Chinese acoustic data L1 and the English acoustic data L2 are forcibly aligned, respectively, to label L1 and L2. Specifically, the state corresponding to the speech frame of each data in L1 may be determined by NN1, and the state corresponding to the speech frame of each data in L2 may be determined by NN2.

Finally, the labeled L1 and L2 are mixed together to obtain the labeled dataset (L1+L2) to train the multilingual acoustic model NN3. The output of the multilingual acoustic model may include: state probabilities corresponding to at least two language types. For example, the number of nodes in the output layer of the NN3 may be M1+M2, wherein the first M1 nodes may correspond to the Chinese HMM state, and the last M2 nodes may correspond to the English HMM state.

In the process of training a multilingual acoustic model in this embodiment of the present invention, monolingual acoustic data corresponding to each language type may be used to retain the pronunciation characteristics of each language type. Therefore, at the acoustic level, different language types may be distinguished to a certain extent. sex. In addition, in the process of collecting training data, the acoustic data of each language type is collected separately, which can avoid the problem of insufficient data caused by collecting mixed acoustic data of multiple language types, and thus can improve the accuracy of multilingual acoustic model recognition.

To sum up, in this embodiment of the present invention, a multilingual model can be obtained by training according to acoustic data of at least two language types. Through the multilingual acoustic model, the language type of the speech frame in the speech information can be determined. Therefore, the speech information contains multiple languages. In the case of different language types, the embodiment of the present invention can accurately distinguish the speech frames of different language types in the speech information, and can decode the speech frames according to the decoding network of the corresponding language type, so as to obtain the first decoding result of the speech frame, the The first decoding result is obtained by decoding according to the decoding network corresponding to the speech type of the speech frame, which can ensure the accuracy of decoding, and thus can improve the accuracy of speech recognition.

It should be noted that, for the sake of simple description, the method embodiments are described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequences, because According to embodiments of the present invention, certain steps may be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

Device embodiment

Referring to FIG. 2, a structural block diagram of an embodiment of a data processing apparatus of the present invention is shown, and the apparatus may specifically include:

The type determination module 201 is used to determine the language type of the speech frame in the speech information according to the multilingual acoustic model; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types;

A first decoding module 202, configured to decode the speech frame according to a decoding network corresponding to the language type of the speech frame, to obtain a first decoding result of the speech frame;

The result determination module 203 is configured to determine the recognition result corresponding to the voice information according to the first decoding result.

Optionally, the type determination module may specifically include:

The probability determination submodule is used to determine the posterior probability of the speech frame corresponding to each state according to the multilingual acoustic model; wherein, there is a corresponding relationship between the state and the language type;

a ratio determination submodule, configured to determine the probability ratio of the posterior probability of the speech frame corresponding to the state of each language type according to the posterior probability of the speech frame corresponding to each state and the language type corresponding to each state;

A type determination submodule, configured to determine the language type of the speech frame according to the probability ratio.

Optionally, the device may also include:

a target language determination module for determining a target language type from the at least two language types;

A second decoding module, configured to decode each speech frame in the speech information according to the decoding network corresponding to the target language type, to obtain a second decoding result of each speech frame;

The apparatus may also include:

A target frame determination module for determining a target speech frame from the speech frame of the speech information, and determining a second decoding result of the target speech frame; wherein, the language type of the target speech frame is a non-target language type ;

The first decoding module may specifically include:

The first decoding submodule is used to decode the target speech frame according to the decoding network corresponding to the language type of the target speech frame, to obtain the first decoding result of the target speech frame;

The result determination module may specifically include:

The first result determination submodule is used to replace the second decoding result of the target speech frame with the first decoding result of the corresponding language type of the target speech frame, and use the replaced second decoding result as the speech The identification result corresponding to the information.

Optionally, the first decoding result and the second decoding result include: time boundary information corresponding to the speech frame;

The first result determination sub-module may specifically include:

A result determination unit, configured to determine a replaced result from the second decoding result of the target speech frame; wherein, the replaced result coincides with the time boundary of the first decoding result of the language type corresponding to the target speech frame ;

A replacement unit, configured to replace the replaced result with the first decoding result of the language type corresponding to the target speech frame.

Optionally, the decoding network may specifically include: a general decoding network and a specialized decoding network; wherein, the general decoding network includes: a language model trained according to a general text corpus; the specialized decoding network includes: The language model obtained by training the text corpus in the preset domain;

The first decoding module may specifically include:

A score determination submodule, configured to decode the speech frame according to the general decoding network and the specialized decoding network respectively, so as to obtain the first score of the speech frame corresponding to the general decoding network, and the speech frame corresponding to the second score of the professional decoding network;

The second result determination submodule is configured to use the decoding result with the highest score among the first score and the second score as the first decoding result of the speech frame.

Optionally, the apparatus may further include: a model training module for training the multilingual acoustic model; the model training module may specifically include:

The first training submodule is used to separately train the monolingual acoustic model corresponding to each language type according to the collected acoustic data of at least two language types;

A state labeling submodule, configured to label the acoustic data of the at least two language types respectively according to the single language acoustic model, wherein the state and the language type have a corresponding relationship;

The second training sub-module is used for training a multilingual acoustic model according to the data set composed of the labeled acoustic data of at least two language types.

Optionally, each of the at least two language types of acoustic data corresponds to at least two language types; or, each of the at least two language types of acoustic data corresponds to one language type.

As for the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for related parts.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments may be referred to each other.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

Embodiments of the present invention provide an apparatus for data processing, including a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors The one or more programs include instructions for performing the following operations: determining language types of speech frames in speech information according to a multilingual acoustic model; wherein the multilingual acoustic model is based on acoustic data of at least two language types Training is obtained; according to the decoding network corresponding to the language type of the speech frame, the speech frame is decoded to obtain the first decoding result of the speech frame; according to the first decoding result, it is determined that the speech information corresponds to recognition result.

FIG. 3 is a block diagram of an apparatus 800 for data processing according to an exemplary embodiment. For example, apparatus 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, and the like.

3, the apparatus 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and communication component 816.

The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing element 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

Memory 804 is configured to store various types of data to support operation at device 800 . Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, and the like. Memory 804 may be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

Power supply assembly 806 provides power to the various components of device 800 . Power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 800 .

Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when device 800 is in operating modes, such as call mode, recording mode, and voice information processing mode. The received audio signal may be further stored in memory 804 or transmitted via communication component 816 . In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of device 800 . For example, the sensor assembly 814 can detect the open/closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect a change in the position of the device 800 or a component of the device 800 , the presence or absence of user contact with the device 800 , the orientation or acceleration/deceleration of the device 800 and the temperature change of the device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between apparatus 800 and other devices. Device 800 may access wireless networks based on communication standards, such as WiFi, 2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation is used to perform the above method.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory 804 including instructions, executable by the processor 820 of the apparatus 800 to perform the method described above. For example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

FIG. 4 is a schematic structural diagram of a server in some embodiments of the present invention. The server 1900 may vary greatly due to differences in configuration or performance, and may include one or more central processing units (CPU) 1922 (eg, one or more processors) and memory 1932, one or more A storage medium 1930 (eg, one or more mass storage devices) that stores applications 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 may be short-term storage or persistent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1922 may be configured to communicate with the storage medium 1930 to execute a series of instruction operations in the storage medium 1930 on the server 1900 .

Server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input/output interfaces 1958, one or more keyboards 1956, and/or, one or more operating systems 1941 , such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM and so on.

A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an apparatus (server or terminal), the apparatus enables the apparatus to execute the data processing method shown in FIG. 1 .

A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a device (server or terminal), enabling the device to execute a data processing method, the method comprising: according to a multilingual The acoustic model determines the language type of the speech frame in the speech information; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types; according to the decoding network corresponding to the language type of the speech frame, the The speech frame is decoded to obtain a first decoding result of the speech frame; and a recognition result corresponding to the speech information is determined according to the first decoding result.

The embodiment of the present invention discloses A1, a data processing method, comprising: determining language types of speech frames in speech information according to a multilingual acoustic model; wherein the multilingual acoustic model is acoustic data according to at least two language types trained;

According to the decoding network corresponding to the language type of the speech frame, the speech frame is decoded to obtain the first decoding result of the speech frame;

According to the first decoding result, the recognition result corresponding to the voice information is determined.

A2. The method according to A1, wherein determining the language type of the speech frame in the speech information according to the multilingual acoustic model, including:

Determine the posterior probability of each state corresponding to the speech frame according to the multilingual acoustic model; wherein, there is a corresponding relationship between the state and the language type;

According to the posterior probability of the speech frame corresponding to each state and the language type corresponding to each state, determine the probability ratio of the posterior probability of the speech frame corresponding to the state of each language type;

The language type of the speech frame is determined according to the probability ratio.

A3. The method according to A1, before determining the language type of the speech frame in the speech information according to the multilingual acoustic model, the method further includes:

determining a target language type from the at least two language types;

According to the decoding network corresponding to the target language type, each speech frame in the speech information is decoded to obtain the second decoding result of each speech frame;

After determining the language type of the speech frame in the speech information according to the multilingual acoustic model, the method further includes:

From the voice frame of the voice information, determine the target voice frame, and determine the second decoding result of the target voice frame; wherein, the language type of the target voice frame is a non-target language type;

Decoding the speech frame according to the decoding network corresponding to the language type of the speech frame to obtain the first decoding result of the speech frame, including:

According to the decoding network corresponding to the language type of the target speech frame, the target speech frame is decoded to obtain the first decoding result of the target speech frame;

The determining the recognition result corresponding to the voice information according to the first decoding result includes:

The second decoding result of the target speech frame is replaced with the first decoding result of the language type corresponding to the target speech frame, and the replaced second decoding result is used as the recognition result corresponding to the speech information.

A4. The method according to A3, wherein the first decoding result and the second decoding result include: time boundary information of the corresponding speech frame;

Described replacing the second decoding result of the target speech frame with the first decoding result of the language type corresponding to the target speech frame, including:

From the second decoding result of the target speech frame, determine the replaced result; wherein, the replaced result coincides with the time boundary of the first decoding result of the language type corresponding to the target speech frame;

The replaced result is replaced with the first decoding result of the language type corresponding to the target speech frame.

A5. The method according to A1, wherein the decoding network includes: a general decoding network and a specialized decoding network; wherein, the general decoding network includes: a language model trained according to a general text corpus; the specialized decoding network Including: the language model obtained by training the text corpus in the preset field;

Decoding the speech frame according to the decoding network corresponding to the language type of the speech frame to obtain the first decoding result of the speech frame, including:

The voice frame is decoded according to the general decoding network and the professional decoding network respectively, so as to obtain the first score of the voice frame corresponding to the general decoding network, and the voice frame corresponding to the professional decoding network. second score;

A decoding result with a higher score among the first score and the second score is used as the first decoding result of the speech frame.

A6. According to the method of A1, the training steps of the multilingual acoustic model include:

According to the collected acoustic data of at least two language types, respectively train the monolingual acoustic model corresponding to each language type;

According to the single-language acoustic model, state labeling is performed on the acoustic data of the at least two language types respectively, wherein there is a corresponding relationship between the states and the language types;

A multilingual acoustic model is trained according to a dataset consisting of annotated acoustic data of at least two language types.

A7. The method according to any one of A1 to A6, wherein each of the acoustic data in the at least two language types corresponds to at least two language types; or, in the acoustic data in the at least two language types Each data corresponds to a language type.

The embodiment of the present invention discloses B8, a data processing device, comprising:

a type determination module, used for determining the language type of the speech frame in the speech information according to the multilingual acoustic model; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types;

a first decoding module, configured to decode the speech frame according to a decoding network corresponding to the language type of the speech frame to obtain a first decoding result of the speech frame;

A result determination module, configured to determine a recognition result corresponding to the voice information according to the first decoding result.

B9. The device according to B8, wherein the type determination module includes:

The probability determination submodule is used to determine the posterior probability of the speech frame corresponding to each state according to the multilingual acoustic model; wherein, there is a corresponding relationship between the state and the language type;

a ratio determination submodule, configured to determine the probability ratio of the posterior probability of the speech frame corresponding to the state of each language type according to the posterior probability of the speech frame corresponding to each state and the language type corresponding to each state;

A type determination submodule, configured to determine the language type of the speech frame according to the probability ratio.

B10. The device according to B8, further comprising:

a target language determination module for determining a target language type from the at least two language types;

A second decoding module, configured to decode each speech frame in the speech information according to the decoding network corresponding to the target language type, to obtain a second decoding result of each speech frame;

The device also includes:

A target frame determination module for determining a target speech frame from the speech frame of the speech information, and determining a second decoding result of the target speech frame; wherein, the language type of the target speech frame is a non-target language type ;

The first decoding module includes:

The first decoding submodule is used to decode the target speech frame according to the decoding network corresponding to the language type of the target speech frame, to obtain the first decoding result of the target speech frame;

The result determination module includes:

The first result determination submodule is used to replace the second decoding result of the target speech frame with the first decoding result of the corresponding language type of the target speech frame, and use the replaced second decoding result as the speech The identification result corresponding to the information.

B11. The device according to B10, wherein the first decoding result and the second decoding result include: time boundary information corresponding to the speech frame;

The first result determination submodule includes:

A result determination unit, configured to determine a replaced result from the second decoding result of the target speech frame; wherein, the replaced result coincides with the time boundary of the first decoding result of the language type corresponding to the target speech frame ;

A replacement unit, configured to replace the replaced result with the first decoding result of the language type corresponding to the target speech frame.

B12. The device according to B8, wherein the decoding network includes: a general decoding network and a specialized decoding network; wherein, the general decoding network includes: a language model trained according to a general text corpus; the specialized decoding network Including: the language model obtained by training the text corpus in the preset field;

The first decoding module includes:

A score determination submodule, configured to decode the speech frame according to the general decoding network and the specialized decoding network respectively, so as to obtain the first score of the speech frame corresponding to the general decoding network, and the speech frame corresponding to the second score of the professional decoding network;

The second result determination submodule is configured to use the decoding result with the highest score among the first score and the second score as the first decoding result of the speech frame.

B13. The device according to B8, further comprising: a model training module for training the multilingual acoustic model; the model training module comprising:

The first training submodule is used to separately train the monolingual acoustic model corresponding to each language type according to the collected acoustic data of at least two language types;

A state labeling submodule, configured to label the acoustic data of the at least two language types respectively according to the single language acoustic model, wherein the state and the language type have a corresponding relationship;

The second training sub-module is used for training a multilingual acoustic model according to the data set composed of the labeled acoustic data of at least two language types.

B14. The apparatus according to any one of B8 to B13, each of the acoustic data in the at least two language types corresponds to at least two language types; or, in the acoustic data in the at least two language types Each data corresponds to a language type.

An embodiment of the present invention discloses C15, an apparatus for data processing, characterized in that it includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by a Or execution of the one or more programs by one or more processors includes instructions for:

Determine the language type of the speech frame in the speech information according to the multilingual acoustic model; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types;

According to the decoding network corresponding to the language type of the speech frame, the speech frame is decoded to obtain the first decoding result of the speech frame;

According to the first decoding result, the recognition result corresponding to the voice information is determined.

C16. The device according to C15, wherein determining the language type of the speech frame in the speech information according to the multilingual acoustic model, comprising:

Determine the posterior probability of each state corresponding to the speech frame according to the multilingual acoustic model; wherein, there is a corresponding relationship between the state and the language type;

According to the posterior probability of the speech frame corresponding to each state and the language type corresponding to each state, determine the probability ratio of the posterior probability of the speech frame corresponding to the state of each language type;

The language type of the speech frame is determined according to the probability ratio.

C17. The device of C15, further configured to execute the one or more programs by one or more processors comprising instructions for:

determining a target language type from the at least two language types;

According to the decoding network corresponding to the target language type, each speech frame in the speech information is decoded to obtain the second decoding result of each speech frame;

The device is also configured to execute, by one or more processors, the one or more programs including instructions for:

From the voice frame of the voice information, determine the target voice frame, and determine the second decoding result of the target voice frame; wherein, the language type of the target voice frame is a non-target language type;

Decoding the speech frame according to the decoding network corresponding to the language type of the speech frame to obtain the first decoding result of the speech frame, including:

According to the decoding network corresponding to the language type of the target speech frame, the target speech frame is decoded to obtain the first decoding result of the target speech frame;

The determining the recognition result corresponding to the voice information according to the first decoding result includes:

The second decoding result of the target speech frame is replaced with the first decoding result of the language type corresponding to the target speech frame, and the replaced second decoding result is used as the recognition result corresponding to the speech information.

C18. The device according to C17, wherein the first decoding result and the second decoding result include: time boundary information corresponding to the speech frame;

Described replacing the second decoding result of the target speech frame with the first decoding result of the language type corresponding to the target speech frame, including:

From the second decoding result of the target speech frame, determine the replaced result; wherein, the replaced result coincides with the time boundary of the first decoding result of the language type corresponding to the target speech frame;

The replaced result is replaced with the first decoding result of the language type corresponding to the target speech frame.

C19. The device according to C15, wherein the decoding network includes: a general decoding network and a specialized decoding network; wherein, the general decoding network includes: a language model trained according to a general text corpus; the specialized decoding network Including: the language model obtained by training the text corpus in the preset field;

Decoding the speech frame according to the decoding network corresponding to the language type of the speech frame to obtain the first decoding result of the speech frame, including:

The voice frame is decoded according to the general decoding network and the professional decoding network respectively, so as to obtain the first score of the voice frame corresponding to the general decoding network, and the voice frame corresponding to the professional decoding network. second score;

A decoding result with a higher score among the first score and the second score is used as the first decoding result of the speech frame.

C20. The device according to C15, wherein the training steps of the multilingual acoustic model include:

According to the collected acoustic data of at least two language types, separately train the monolingual acoustic model corresponding to each language type;

According to the single-language acoustic model, state labeling is performed on the acoustic data of the at least two language types respectively, wherein there is a corresponding relationship between the states and the language types;

A multilingual acoustic model is trained according to a dataset consisting of annotated acoustic data of at least two language types.

C21. The apparatus according to any one of C15 to C20, each of the acoustic data in the at least two language types corresponds to at least two language types; or, in the acoustic data in the at least two language types Each data corresponds to a language type.

The embodiment of the present invention discloses D22, a machine-readable medium on which instructions are stored, and when executed by one or more processors, cause an apparatus to execute the data processing method described in one or more of A1 to A7 .

Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. The present invention is intended to cover any modifications, uses or adaptations of the present invention that follow the general principles of the invention and include common knowledge or common technical means in the art not disclosed by this disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

It should be understood that the present invention is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

A data processing method, a data processing device, and a data processing device provided by the present invention have been described above in detail. In this paper, specific examples are used to illustrate the principles and implementations of the present invention. The description of the above embodiment is only used to help understand the method of the present invention and its core idea; meanwhile, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in specific embodiments and application scope, In conclusion, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. a data processing method, is characterized in that, described method comprises: Determine the language type of the speech frame in the speech information according to the multilingual acoustic model; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types; According to the decoding network corresponding to the language type of the speech frame, the speech frame is decoded to obtain the first decoding result of the speech frame; According to the first decoding result, the recognition result corresponding to the voice information is determined. 2. The method according to claim 1, wherein, determining the language type of the speech frame in the speech information according to the multilingual acoustic model, comprising: Determine the posterior probability of each state corresponding to the speech frame according to the multilingual acoustic model; wherein, there is a corresponding relationship between the state and the language type; According to the posterior probability of the speech frame corresponding to each state and the language type corresponding to each state, determine the probability ratio of the posterior probability of the speech frame corresponding to the state of each language type; The language type of the speech frame is determined according to the probability ratio. 3. The method according to claim 1, wherein, before determining the language type of the speech frame in the speech information according to the multilingual acoustic model, the method further comprises: determining a target language type from the at least two language types; According to the decoding network corresponding to the target language type, each speech frame in the speech information is decoded to obtain the second decoding result of each speech frame; After determining the language type of the speech frame in the speech information according to the multilingual acoustic model, the method further includes: From the voice frame of the voice information, determine the target voice frame, and determine the second decoding result of the target voice frame; wherein, the language type of the target voice frame is a non-target language type; Decoding the speech frame according to the decoding network corresponding to the language type of the speech frame to obtain the first decoding result of the speech frame, including: According to the decoding network corresponding to the language type of the target speech frame, the target speech frame is decoded to obtain the first decoding result of the target speech frame; The determining the recognition result corresponding to the voice information according to the first decoding result includes: The second decoding result of the target speech frame is replaced with the first decoding result of the language type corresponding to the target speech frame, and the replaced second decoding result is used as the recognition result corresponding to the speech information. 4. The method according to claim 3, wherein the first decoding result and the second decoding result comprise: time boundary information corresponding to the speech frame; Described replacing the second decoding result of the target speech frame with the first decoding result of the language type corresponding to the target speech frame, including: From the second decoding result of the target speech frame, determine the replaced result; wherein, the replaced result coincides with the time boundary of the first decoding result of the language type corresponding to the target speech frame; The replaced result is replaced with the first decoding result of the language type corresponding to the target speech frame. 5. The method according to claim 1, wherein the decoding network comprises: a general decoding network and a specialized decoding network; wherein, the general decoding network comprises: a language model trained according to general text corpus ; The professional decoding network includes: a language model obtained by training the text corpus in the preset field; Decoding the speech frame according to the decoding network corresponding to the language type of the speech frame to obtain the first decoding result of the speech frame, including: The voice frame is decoded according to the general decoding network and the professional decoding network respectively, so as to obtain the first score of the voice frame corresponding to the general decoding network, and the voice frame corresponding to the professional decoding network. second score; A decoding result with a higher score among the first score and the second score is used as the first decoding result of the speech frame. 6. The method according to claim 1, wherein the training step of the multilingual acoustic model comprises: According to the collected acoustic data of at least two language types, respectively train the monolingual acoustic model corresponding to each language type; According to the single-language acoustic model, state labeling is performed on the acoustic data of the at least two language types respectively, wherein there is a corresponding relationship between the states and the language types; A multilingual acoustic model is trained according to a dataset consisting of annotated acoustic data of at least two language types. 7. The method according to any one of claims 1 to 6, wherein each of the acoustic data of the at least two language types corresponds to at least two language types; or, the at least two languages Each of the types of acoustic data corresponds to a language type. 8. A data processing device, wherein the device comprises: a type determination module, used for determining the language type of the speech frame in the speech information according to the multilingual acoustic model; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types; a first decoding module, configured to decode the speech frame according to a decoding network corresponding to the language type of the speech frame to obtain a first decoding result of the speech frame; A result determination module, configured to determine a recognition result corresponding to the voice information according to the first decoding result. 9. An apparatus for data processing, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors The one or more programs include instructions for: Determine the language type of the speech frame in the speech information according to the multilingual acoustic model; wherein, the multilingual acoustic model is obtained by training according to the acoustic data of at least two language types; According to the decoding network corresponding to the language type of the speech frame, the speech frame is decoded to obtain the first decoding result of the speech frame; According to the first decoding result, the recognition result corresponding to the voice information is determined. 10. A machine-readable medium having stored thereon instructions which, when executed by one or more processors, cause an apparatus to perform a data processing method as claimed in one or more of claims 1 to 7.

Images