CN111816160A - Mandarin and Cantonese hybrid speech recognition model training method and system - Google Patents

Abstract

The invention discloses a method for training a Mandarin-Cantonese mixed speech recognition model, comprising: using mixed speech training samples of N languages to train a multi-task model. The last layer of is connected with N task neural network layers corresponding to N languages; network parameters of multiple shared network layers are transferred to the speech recognition model to be trained to complete the training of the speech recognition model to be trained. The embodiment of the present invention firstly uses mixed speech training samples of multiple languages to train a multi-task model, and then reuses the network parameters of the multi-task model by means of data migration, and trains a mixed speech recognition model of Putonghua and Cantonese based on the mixed modeling of Putonghua and Cantonese . It can solve the problem of mixed speech recognition in Mandarin and Cantonese, without the need to make major modifications to the original recognition service, and can use the existing achievements to reduce the cost of model training and service development.

Description

Mandarin and Cantonese hybrid speech recognition model training method and system

technical field

The invention relates to the technical field of speech recognition, in particular to a method and system for training a mixed speech recognition model in Mandarin and Cantonese.

Background technique

With the continuous development of mobile terminal equipment and speech recognition technology, there are some mixed speech recognition solutions in Mandarin and dialects. Such as iFLYTEK voice input method, Baidu voice input method, Sogou input method, Ali intelligent customer service, they all have the function of mixing Mandarin and dialect voice recognition.

The existing solutions are all algorithms based on the deep learning framework, using different acoustic modeling units according to their actual conditions, and supporting the simultaneous recognition of multiple languages through different acoustic training processes and algorithms.

There are two common solutions. One is to use a language classifier to determine which language the speech belongs to, and then input the speech into the corresponding speech recognition system for recognition, as shown in Figure 1. However, this method needs to introduce a language recognition module for language classification, so that the recognition result depends on the classification performance of the language recognition module. When the language recognition module and classifier are unstable, the recognition effect will be poor. The accuracy of speech recognition is determined based on the accuracy of language classification and the accumulation of the accuracy of speech recognition systems. Therefore, the accuracy of speech recognition is lower than that of a single speech recognition system. This solution is difficult to have strong robustness in various scenarios. Awesome. In addition, multiple sets of speech recognition systems need to be deployed on the server, and the engineering cost is high.

Another solution is to use a hybrid speech recognition method, which mixes modeling units in multiple languages, then mixes audio data and text data in different languages, and reuses the conventional training process for hybrid speech recognition; Dictionaries, training data and corpus texts in multiple languages are mixed, and then the conventional training process is reused for mixed speech recognition. This hybrid speech recognition method is easy to implement and the engineering cost is relatively low, but this method of speech recognition is to mix the data of multiple different languages for training. In reality, it is difficult to achieve balanced training data for each language. And different languages have differences in pronunciation, the amount of data is unbalanced or the selection is inappropriate, which will cause the phonemes of different languages to be unevenly distributed in the training set, so the trained recognition results will be biased towards languages with large amounts of data, such mixed speech. The performance of the recognition system will drop a lot compared to the performance of the speech recognition system of each language alone, the overall recognition rate is difficult to recognize each language well, and the performance of ASR (Automatic Speech Recognition) will lose a lot.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a method and system for training a Mandarin-Cantonese hybrid speech recognition model, and a Mandarin-Cantonese hybrid speech recognition method and system, which are used to solve at least one of the above-mentioned technical problems.

In a first aspect, an embodiment of the present invention provides a method for training a Mandarin-Cantonese hybrid speech recognition model, including:

A multi-task model is trained using mixed speech training samples in N languages, the multi-task model comprising a plurality of shared network layers and N corresponding to N languages connected to the last layer of the plurality of shared network layers a task neural network layer;

The network parameters of the multiple shared network layers are migrated to the speech recognition model to be trained to complete the training of the speech recognition model to be trained.

In a second aspect, an embodiment of the present invention provides a mixed speech recognition method for Mandarin and Cantonese, including: inputting the mixed speech of Mandarin and dialect into the speech recognition obtained by the training method of the mixed speech recognition model training method for Mandarin and Cantonese described in the embodiment of the present invention model for hybrid speech recognition.

In a third aspect, an embodiment of the present invention provides a Mandarin-Cantonese hybrid speech recognition model training system, including:

A multi-task model training module, used for training a multi-task model using mixed speech training samples of N languages, the multi-task model includes a plurality of shared network layers and is connected to the last layer of the plurality of shared network layers of N task neural network layers corresponding to N languages;

The speech recognition model training module is used for migrating the network parameters of the plurality of shared network layers to the speech recognition model to be trained, so as to complete the training of the speech recognition model to be trained.

In a fourth aspect, an embodiment of the present invention provides a Mandarin-Cantonese hybrid speech recognition system, including:

The speech recognition model is obtained by training using the Mandarin and Cantonese mixed speech recognition model training method described in the embodiment of the present invention;

The speech input module is used for inputting the mixed speech of Mandarin and Cantonese into the speech recognition model for mixed speech recognition.

A fifth aspect provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, The instructions are executed by the at least one processor to enable the at least one processor to perform any of the above-described mixed Mandarin and Cantonese speech recognition of the present invention.

In a sixth aspect, an embodiment of the present invention provides a storage medium, where one or more programs including execution instructions are stored in the storage medium, and the execution instructions can be executed by an electronic device (including but not limited to a computer, a server, or a network). equipment, etc.) to read and execute, so as to perform any one of the above-mentioned mixed speech recognition of Mandarin and Cantonese of the present invention.

The beneficial effect of the embodiment of the present invention is that: first, a multi-task model is trained by using mixed speech training samples of multiple languages, and then the network parameters of the multi-task model are reused by means of data migration. Cantonese hybrid speech recognition model. It can solve the problem of mixed speech recognition in Mandarin and Cantonese, and does not require major modifications to the original recognition service. It can use the existing achievements to reduce the cost of model training and service development.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

Fig. 1 is the scheme schematic diagram of using language classifier to carry out mixed speech recognition in the prior art;

2 is a flowchart of an embodiment of a method for training a Mandarin and Cantonese hybrid speech recognition model of the present invention;

3 is a schematic block diagram of an embodiment of the Mandarin and Cantonese hybrid speech recognition model training system of the present invention;

4 is a schematic block diagram of an embodiment of the Mandarin and Cantonese hybrid speech recognition system of the present invention;

Fig. 5 is the schematic diagram of the embodiment of the Mandarin and Cantonese mixed speech recognition model training method of the present invention;

FIG. 6 is a schematic diagram of another embodiment of the Mandarin and Cantonese hybrid speech recognition model training method of the present invention;

7 is a flow chart of an embodiment of the Mandarin and Cantonese hybrid speech recognition method of the present invention;

FIG. 8 is a schematic structural diagram of an embodiment of an electronic device of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict.

The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, elements, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

In the present invention, "module", "device", "system", etc. refer to relevant entities applied to a computer, such as hardware, a combination of hardware and software, software or software in execution, and the like. In detail, for example, an element may be, but is not limited to, a process running on a processor, a processor, an object, an executable element, a thread of execution, a program, and/or a computer. Also, an application program or script program running on the server, and the server can be a component. One or more elements may be in a process and/or thread of execution and an element may be localized on one computer and/or distributed between two or more computers and may be executed from various computer readable media . Elements may also pass through a signal having one or more data packets, for example, a signal from one interacting with another element in a local system, in a distributed system, and/or with data interacting with other systems through a network of the Internet local and/or remote processes to communicate.

Finally, it should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Furthermore, the terms "comprising" and "comprising" include not only those elements, but also other elements not expressly listed, or elements inherent to such a process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprises" does not preclude the presence of additional identical elements in a process, method, article, or device that includes the element.

FIG. 2 is a flowchart of an embodiment of a method for training a Mandarin-Cantonese hybrid speech recognition model of the present invention. The method in this embodiment includes:

S10. Use mixed speech training samples of N languages to train a multi-task model, where the multi-task model includes a plurality of shared network layers and a network corresponding to N languages connected to the last layer of the plurality of shared network layers The N task neural network layers;

S20. Migrate the network parameters of the multiple shared network layers to the speech recognition model to be trained, so as to complete the training of the speech recognition model to be trained.

The embodiment of the present invention firstly uses mixed speech training samples of multiple languages to train a multi-task model, and then reuses the network parameters of the multi-task model by means of data migration, and trains a mixed speech recognition model of Putonghua and Cantonese based on the mixed modeling of Putonghua and Cantonese . It can solve the problem of mixed speech recognition in Mandarin and Cantonese, and does not require major modifications to the original recognition service. It can use the existing achievements to reduce the cost of model training and service development.

The multi-language multi-task training method of the embodiment of the present invention enables the commonality between different languages to be learned through the multi-layer language sharing network layer parameters, and the individuality of different languages can be learned through the task-specific layer (for example, the task neural network layer). The output is used to learn, so that the trained network contains the pronunciation of various phonemes, and the model is more robust. In the form of transfer learning, the parameters of the trained model are transferred to the new model to help the training of the new model. In this way, the learned model parameters can be shared with the new model, thereby speeding up and optimizing the learning efficiency of the new model. Integrating the modeling units of Mandarin and Cantonese, through the sharing of modeling units with larger granularity, can avoid insufficient training of model parameters due to too few characters appearing in the training text.

In some embodiments, the training of the multi-task model using mixed speech training samples of N languages includes:

Train the network parameters of the N task neural network layers based on the N loss functions corresponding to the N task neural network layers;

The network parameters of the plurality of shared network layers are jointly trained at least based on the N loss functions corresponding to the N task neural network layers.

In some embodiments, the multi-task model further includes a language classification network layer connected to the last layer of the plurality of shared network layers;

The network parameters for jointly training the plurality of shared network layers at least based on the N loss functions corresponding to the N task neural network layers include:

The network parameters of the plurality of shared network layers are jointly trained based on the N loss functions corresponding to the N task neural network layers and the loss functions corresponding to the language classification network layers.

In some embodiments, jointly training the network parameters of the plurality of shared network layers based on the N loss functions corresponding to the N task neural network layers and the loss functions corresponding to the language classification network layers includes: based on the corresponding The network parameters of the plurality of shared network layers are trained on the weighted sum of the N loss functions of the N task neural network layers and the loss functions corresponding to the language classification network layers.

In some embodiments, the present invention also provides a mixed speech recognition method for Mandarin and Cantonese, comprising: inputting the mixed speech of Mandarin and dialect into the training method using the mixed speech recognition model training method for Mandarin and Cantonese described in any embodiment of the present invention The obtained speech recognition model is used for hybrid speech recognition.

As shown in FIG. 3 , it is a schematic block diagram of an embodiment of the Mandarin and Cantonese hybrid speech recognition model training system of the present invention. The system 300 includes:

The multi-task model training module 310 is used to train a multi-task model using mixed speech training samples of N languages, the multi-task model includes a plurality of shared network layers and the last layer of the plurality of shared network layers Connected N task neural network layers corresponding to N languages;

The speech recognition model training module 320 is configured to migrate the network parameters of the multiple shared network layers to the speech recognition model to be trained, so as to complete the training of the speech recognition model to be trained.

The embodiment of the present invention firstly uses mixed speech training samples of multiple languages to train a multi-task model, and then reuses the network parameters of the multi-task model by means of data migration, and trains a mixed speech recognition model of Putonghua and Cantonese based on the mixed modeling of Putonghua and Cantonese . It can solve the problem of mixed speech recognition in Mandarin and Cantonese, and does not require major modifications to the original recognition service. It can use the existing achievements to reduce the cost of model training and service development.

In some embodiments, the training of the multi-task model using mixed speech training samples of N languages includes:

Train the network parameters of the N task neural network layers based on the N loss functions corresponding to the N task neural network layers;

The network parameters of the plurality of shared network layers are jointly trained at least based on the N loss functions corresponding to the N task neural network layers.

In some embodiments, the multi-task model further includes a language classification network layer connected to the last layer of the plurality of shared network layers;

The network parameters for jointly training the plurality of shared network layers at least based on the N loss functions corresponding to the N task neural network layers include:

The network parameters of the plurality of shared network layers are jointly trained based on the N loss functions corresponding to the N task neural network layers and the loss functions corresponding to the language classification network layers.

In some embodiments, jointly training the network parameters of the plurality of shared network layers based on the N loss functions corresponding to the N task neural network layers and the loss functions corresponding to the language classification network layers includes: based on the corresponding The network parameters of the plurality of shared network layers are trained on the weighted sum of the N loss functions of the N task neural network layers and the loss functions corresponding to the language classification network layers.

As shown in FIG. 4, it is a schematic block diagram of an embodiment of the Mandarin and Cantonese hybrid speech recognition system of the present invention. The system 400 includes:

The speech recognition model 410 is obtained by training using the Mandarin and Cantonese mixed speech recognition model training method described in any embodiment of the present invention;

The speech input module 420 is used for inputting the mixed speech of Mandarin and Cantonese into the speech recognition model to perform mixed speech recognition.

In order to more intuitively reflect the technical contributions made by the present invention relative to the prior art, the following will further describe in detail with reference to specific embodiments.

It mainly includes three steps:

(1) Data preparation

In terms of data, the audio data and text data of Mandarin and Cantonese are mixed respectively, and the dictionaries are combined, and the modeling units are sorted out according to the above-mentioned sorting method of the modeling units; at the same time, considering that the dialect data is less, we use signal processing, real machine transcription , web crawling, etc. for data expansion;

Extract the features of the audio, use the FBANK feature, use a window with a frame length of 25ms and a frame shift of 10ms to divide the audio into frames, and extract 40-dimensional Fbank features from each frame to train the neural network;

We use N languages to train the neural network, so we need to prepare the corresponding features and labeled texts of the N languages, and then combine the features of the N languages and scramble them randomly to ensure the randomness of the input features of the training model.

(2) Multi-language multi-task training

We use N languages for multi-task training and supervised multi-task training. The training criteria can use frame-level CE (Cross Entropy) loss function or sequence-level CTC (ConnectionistTemporalClassification) loss function to train model parameters, According to the network structure of the example in Figure 5, each output represents a special training task for a language, and its output corresponds to the loss output of the language, so when the neural network has N language training tasks, there will be N languages at the same time. The loss output and a Language Id loss output.

FIG. 5 is a schematic diagram of an embodiment of the method for training a Mandarin and Cantonese mixed speech recognition model of the present invention. Among them, NN (Nerual Network) layer represents a layer of neural network layer, which can be commonly used DNN (Deep Neural Networks), LSTM (Long Short-Term Memory), FSMN (Feedforward Sequential Memory Networks) and so on.

At the same time, we will introduce the label Language ID (hereinafter referred to as LID) of each language, and use the CE criterion for training. The goal is to minimize the language type discrimination error in each frame, reduce the language domain classification error, and increase the network to many The robustness of different languages can reduce the impact of imbalanced data in various languages, and at the same time, it can speed up the convergence speed of the neural network.

Multilingual multitasking training process:

(a) First, randomly scramble the input data of N kinds of language features; according to the different training criteria and modeling units, prepare corresponding labels for each language, here we use frame-level phonemes for modeling, and get N kinds of The frame-level phoneme label of the language, and then all labels are merged and arranged according to the input order of the training features;

Here is how to get the label of the LID:

According to the prior information of each language category known in advance, encode different language categories according to different numbers, for example: for all sample data of language category 1, set it to 1 according to the length of its characteristic data, For all sample data of language type 2, set it to 2 according to the length of its feature data, and so on, you can get the label of the LID of each language, and then arrange it according to the order of training feature input;

At this point, we have prepared feature input data, phoneme labels for each language, and LID labels for each language, respectively.

(b) Here we will multiply each loss by a different scaling factor α and β according to the specific task situation, so as to avoid the final model parameter update biased towards some individual tasks:

Loss=α1·loss1(output1)+α2·loss2(output2)+…+αN.lossN(outputN)+β.loss(Language ID)

Use the loss (output) corresponding to each language to update its corresponding Task NN layer model parameters, and use all output losses to update the bottom shared model parameters, so that the bottom shared layer can learn the characteristics of each language, so that the trained The network can cover the pronunciation of a wider range of human phonemes. At the same time, this method collects various environments of different language training data sets. The parameters learned by a language in a certain environment can also improve the performance of other languages in the scene. Such network parameters are more robust from the point of view of training data robustness.

(3) Transfer learning initialization training

As shown in FIG. 6 , it is a schematic diagram of another embodiment of the method for training a mixed speech recognition model of Mandarin and Cantonese of the present invention.

The public layer parameters of the above-mentioned multi-language and multi-task training network are taken as the initialization model for the mixed recognition of Mandarin and Cantonese, and the above-mentioned trained model parameters are transferred to the new model in the form of transfer learning to help the training of the new model. In this way, the model parameters of multiple languages that have been learned can be shared with the new Mandarin and Cantonese (dialect) models, thereby speeding up and optimizing the learning efficiency of the new model.

Mandarin and Cantonese mixed training process:

(a) Data preparation: Different modeling units can be used for training according to different tasks. Here we use the method in 2.3 above to prepare the corresponding Mandarin and Cantonese character modeling units, and mix the feature data of the two languages. And randomly scrambled, the character labels prepared in two languages are arranged according to the characteristic data;

(b) Here we use the training criteria of CTC to update the model parameters, which is consistent with the traditional training method. There is only one network output, so the final trained model can support the simultaneous recognition of Mandarin and Cantonese.

We will combine the dictionaries of the two languages and the corpus texts to train the language model, so that during decoding, the acoustic model and the language model can recognize the two languages at the same time, and the hybrid recognition system has better performance than the previous single system. will lose.

The present invention uses N (N≥3, which can be Mandarin, Sichuan dialect, Cantonese, Shanghai dialect and foreign languages, etc.) languages through multi-language multi-task learning, and N languages are trained by multi-task learning. In this way, the commonality between different languages can be learned through multi-layer shared model parameters, and the individuality of different languages can be learned through the output layer of a separate language. In this way, the trained network includes the pronunciation of various phonemes, and the model is more robust. ;

In the form of transfer learning, the above trained model parameters are transferred to the new model to help the training of the new model. In this way, the learned model parameters can be shared with the new model, thereby accelerating and optimizing the learning efficiency of the new model;

Integrating the modeling units of Mandarin and Cantonese: We use the form of characters to model, and count the word frequency of each character in the training text corresponding to each character with the same pinyin pronunciation from the Mandarin and Cantonese dictionaries, respectively. These characters with a threshold value are modeled separately as a modeling unit, and the characters below a certain threshold are uniformly modeled with characters with high word frequency, so that the modeling units in the form of Mandarin and Cantonese characters can be obtained respectively, and then the Mandarin The same character units in the Cantonese modeling unit are merged and shared, and the different ones are kept separately, so that a set of modeling units mixed with Mandarin and Cantonese can be obtained. In this way, through the sharing of large-granularity modeling units, it is possible to avoid the problems caused by certain characters. There are too few cases in the training text and the model parameters are not sufficiently trained.

The model trained by this method not only satisfies the needs of recognizing two languages at the same time, but also does not need to change the framework of the kernel engineering, and its performance can also reach or even surpass the performance of a single language recognition system.

The present invention proposes a multi-language and multi-task training process and a processing method for mixing Mandarin and Cantonese modeling units, which can not only solve the mixed recognition problem of existing Mandarin and Cantonese, but also realize the mixed recognition problem of random speaking in any multiple languages. The original recognition service needs to be greatly modified, and the existing achievements can be reused, which greatly reduces the cost of model training and service development.

It should be noted that the present invention is not obvious and readily available to those skilled in the art. In fact, the inventor has at least adopted the following earlier versions of the solution in the process of realizing the present invention:

When decoding the same audio, use Mandarin and Cantonese recognition resources respectively, and then determine the final text recognition result according to the confidence or semantic analysis of the two recognition results. This method does not require complex model training, and is simple to implement. It only needs to do some post-processing in the recognition backend to achieve the goal. However, this implementation method is not universal enough, and there are still high engineering costs and waste of resources.

In the training of the acoustic model, the original modeling units of the two languages are directly merged and trained. This is also the easiest method to think of and the easiest to implement. However, this method does not share the modeling units of different languages. This results in a lot of performance loss in the end.

In fact, the present invention has at least the following beneficial effects relative to the prior art and various technical solutions tried by the inventor in the process of invention and creation:

(1) Multi-language multi-task training method, so that the commonality between different languages can be learned through multi-layer shared model parameters, and the personality of different languages can be learned through the output layer of a separate language, so the trained network contains The pronunciation of various phonemes, the model is more robust;

(2) In the form of transfer learning, the trained model parameters are transferred to the new model to help the training of the new model. In this way, the learned model parameters can be shared with the new model, thereby accelerating and optimizing the learning efficiency of the new model;

(3) Integrate the modeling units of Mandarin and Cantonese: Through the sharing of large-granularity modeling units, the situation that the model parameters are not sufficiently trained due to too few characters in the training text can be avoided.

As shown in FIG. 7 , an embodiment of the present invention provides a mixed speech recognition method for Mandarin and Cantonese, the method comprising:

S71. Use the mixed speech training samples of N languages to train the multi-task model, and obtain the parameter values of the multi-task model; the multi-task model has several language sharing network layers (the first to n-th shared layers), and the language sharing network N+1 parallel task-specific layers connected to the deepest layer of the layer, the language-sharing network layer is a neural network layer, and the task-specific layer is a neural network layer, where N≥3;

S72. Migrate the parameter values of the language sharing network layer to the Cantonese speech recognition model; the Cantonese speech recognition model has the same language sharing network layer as the multi-task training model, a general Cantonese recognition task-specific layer, and a general Cantonese recognition task-specific layer is the neural network layer, which is connected to the deepest layer of the language sharing network layer;

S73. Model a mixture of Mandarin and Cantonese, and train a Cantonese speech recognition model;

S74. Recognize the mixed speech of Mandarin and Cantonese based on the trained Cantonese speech recognition model.

The multi-task training model has several language-sharing network layers, and the depth (number of layers) of the language-sharing network layers can be set according to usage needs. The first shared layer receives input data, and the input data includes feature data extracted from multiple speech training samples of N languages, phoneme labels of each language, and labels of the language corresponding to each speech training sample (LanguageID, hereinafter referred to as LID). ).

In this embodiment, the N languages may be Mandarin, Sichuanese, Cantonese, Shanghainese, and foreign languages. Acquire audio in N languages, where each language corresponds to multiple audios, and acquire the labeled text corresponding to the audio, divide the audio into frames using a window with a frame length of 25ms and a frame shift of 10ms, and extract 40-dimensional FBANK features for each frame. Combine the features extracted from multiple audios in N languages, and randomly shuffle the order to generate training features, so as to form the training features corresponding to the mixed speech training samples of N languages, to ensure the randomness of the input features when training the model.

The phonemes of a language are the smallest phonetic units divided according to the natural attributes of speech. In this embodiment, for each language, frame-level phonemes are used, and frame-level phonemes are modeled. The modeling function can use conventional modeling in the field. function to get the frame-level phoneme labels for that language, and then merge all frame-level phoneme labels for that language. For the phoneme labels of N languages, they are arranged in the input order of the training features.

For a speech sample, since it is known in advance which language the speech sample corresponds to, that is, the prior information of each language category is known in advance, the speech samples are encoded according to different numbers, and the speech samples of the same language are It sets the same LID. For example, for all speech samples whose language type is 1, set their LID to 1 according to the length of their characteristic data; for all speech samples whose language type is 2, set their LID to 2 according to the length of their characteristic data, By analogy, the LID can be set for each speech sample, arranged according to the input order of the training features.

The multi-task training model has several language shared network layers (the first to nth shared layers), and the depth of the language shared network layers can be set by itself according to usage needs and computing performance. Each layer in the language shared network layer may adopt the same neural network structure, or may adopt different neural network structures.

As shown in Figure 5, the first shared layer (the shared layer can be a neural network layer) receives input data, the calculated data is input to the second shared layer, and the output is generated again after calculation, which is used as the next shared layer The input is input step by step until the deepest shared layer.

The language sharing network layer is adopted, so that the sharing between different languages can be learned by the language sharing network layer.

After the calculation of the deepest language sharing network layer, the output data of the deepest language sharing network layer is divided into N subsets according to the language category according to the prior language category information, and each subset corresponds to a Language output data.

There are N+1 parallel task-specific layers connected to the deepest language-sharing network layer; the task-specific layer is a neural network layer, and each of the N task-specific layers is trained on a subset, and each Each task-specific layer generates an output data, denoted as Output _i , where 1≤i≤N; the N+1 task-specific layer trains the output data of the deepest language sharing network layer, and the output result is the identified LIDs.

Each task-specific layer is a neural network layer, and each layer in the task-specific layer can use the same neural network structure or different neural network structures; it can use the same structure as a language shared network layer, or Adopt a structure that is different from the shared network layer for each language.

The task-specific layer is adopted, so that the personality of different languages can be learned through a separate task-specific layer, so that the trained network includes the pronunciation of various phonemes, and is more robust.

The language shared network layer and the task-specific layer are all trained in a supervised manner. The training criteria can use the frame-level CE (Cross Entropy) loss function or the sequence-level CTC (Connectionist Temporal Classification) loss function to train model parameters. Each output, that is, Output _i corresponds to a special training task of a language, and then calculate the loss output of Output _i corresponding to the language. In this embodiment, there are N language-specific training tasks, and N language loss outputs and one LID loss output are generated. Based on the loss output for N languages and the loss output for one LID, the total loss can be calculated.

In order to facilitate the understanding of the loss output of the LID, for example, the input mixed speech training samples of N languages include English, Sichuan, and Mandarin, and the LIDs are denoted as 1, 2, and 3, respectively. Since there is a certain deviation in the training accuracy of the N+1 task-specific layer, the LIDs corresponding to the speech recognized by the N+1 task-specific layer are 1, 2, and 4. Then according to the recognition result, the loss output of LID can be calculated.

In this embodiment, the N+1 task-specific layer is trained using the CE criterion. The goal is to minimize the language type discrimination error in each frame, reduce the language domain classification error, and increase the robustness of the network to many different languages. It can reduce the influence of the imbalance of various language data and speed up the convergence speed of the neural network.

In this embodiment, scaling factors α _i and β are configured for the loss according to specific task conditions, so as to avoid biasing towards some individual tasks when updating the parameters of each network layer.

The total loss is calculated as:

Loss=α ₁ ·loss ₁ (output ₁ )+α ₂ ·loss ₂ (output ₂ )+…+α _N ·loss _N (output _N )+β·loss(LID)

The loss output for each language is used to iteratively update the parameters of the task-specific layer corresponding to that language, and the total loss is used to iteratively update the parameters of the several language shared network layers. Through such a learning method, the language sharing network layer can learn the characteristics of each language, so that the trained language sharing network layer can cover a wider range of human phonemes pronunciation. At the same time, various environments of different language training sample sets are also collected. The parameters learned by a certain language in a certain environment can also improve the recognition performance of other languages in this scenario. Such network parameters are more robust.

Described S72, the parameter value of described language sharing network layer is migrated to Cantonese speech recognition model; Described Cantonese speech recognition model has the same language sharing network layer as described multi-task training model, a Cantonese recognition task exclusive layer, the Cantonese recognition task-specific layer is a neural network layer, which is connected to the deepest layer of the language shared network layer,

The Cantonese speech recognition model recognizes the mixed speech of Mandarin and Cantonese. As shown in FIG. 4 , the Cantonese speech recognition model has the same language sharing network layer as the multi-task training model, and also has a Cantonese recognition task-specific layer, which is a neural network layer. , connected to the deepest layer of the language-sharing network layer. The number of layers of the language sharing network layer and the structure of each layer in the Puyue speech recognition model are respectively the same as the number of layers of the language sharing network layer and the structure of the corresponding layers in the multi-task training model.

The Cantonese recognition task-specific layer structure may be the same as the shared network layer structure of a certain language, or may be different from the shared network layer structure of all languages.

Migrate the parameter values of the language sharing network layer in the trained multi-task training model to the Puyue speech recognition model as the initialization parameters of the language sharing network layer in the Puyue speech recognition model. Transferring the learned parameters in this way can speed up and optimize the learning efficiency of the Cantonese speech recognition model.

S73. Modeling a mixture of Mandarin and Cantonese, and train the Cantonese speech recognition model, including:

a. Obtain Mandarin and Cantonese audio and corresponding annotation text, use a window with a frame length of 25ms and a frame shift of 10ms to frame the audio, and extract 40-dimensional FBANK features for each frame; mix the features of Mandarin and Cantonese, and randomly type Further, if the dialect data in the training samples is small, the data can be expanded by means of signal processing, real machine transcription, and online crawling.

b. Integrate the modeling units of Mandarin and Cantonese, model the integrated Mandarin and Cantonese modeling units, and obtain the character labels of Mandarin and Cantonese, including:

The word frequency of each character with the same pronunciation in the training text is counted from the Mandarin and Cantonese dictionaries respectively, and the character with a word frequency greater than the preset threshold is regarded as a high-frequency character, and the high-frequency character is regarded as a modeling unit. The characters with the threshold are replaced by high-frequency words with the same pronunciation; the Mandarin character modeling unit and the Cantonese character modeling unit are obtained respectively;

For example, in the Mandarin dictionary, the pronunciations of "mou" and "mou" are the same, the word frequency of "mou" is greater than the preset threshold, and the word frequency of "mou" is lower than the preset threshold. "Mould" as a modeling unit.

In the Mandarin character modeling unit and the Cantonese character modeling unit, the modeling units with the same pronunciation are merged, and the modeling units with different pronunciations are kept separately;

For example, if there is a Mandarin modeling unit "mou" and a Cantonese modeling unit "no", the pronunciation of the Mandarin modeling unit "mou" and the Cantonese modeling unit "no" are both móu, then the Mandarin modeling unit "mou", Cantonese Modeling Unit "No" merged. This results in a set of modelling units that mix Mandarin and Cantonese. In this way, a larger granularity modeling unit is formed, which can avoid insufficient parameter training due to too few characters appearing in the training text.

Model the integrated Mandarin and Cantonese modeling units. The modeling function can use conventional modeling functions in the field to obtain the Mandarin and Cantonese character labels, and then arrange the Mandarin and Cantonese character labels according to the input order of the training features. .

c. The FBANK feature, Mandarin and Cantonese character labels in random order are used as input, and the training criterion of CTC is adopted to train the Cantonese speech recognition model, and the trained Cantonese speech recognition model is obtained.

In this embodiment, the iterative update of model parameters is performed using the training criteria of CTC, and there is only one network output, and the finally trained Cantonese speech recognition model can support the recognition of Mandarin and Cantonese.

The model trained by this method not only meets the needs of recognizing Mandarin and Cantonese, but also does not need to change the framework of kernel engineering, and its performance can reach or even exceed the performance of a single language recognition system.

Described S74, based on the trained Cantonese speech recognition model to recognize the mixed speech of Mandarin and Cantonese, including:

Input the audio mixed between Mandarin and Cantonese, recognize the mixed Mandarin and Cantonese speech based on the trained Cantonese speech recognition model, and output the corresponding text information.

Further, the trained Cantonese speech recognition model is an acoustic model capable of recognizing sounds, and in order to improve the accuracy of mixed speech recognition, a language model is trained, that is, the dictionary of Mandarin and Cantonese is merged, and the corpus used for training. Text merging to train language models. When decoding and recognizing the mixed audio of Mandarin and Cantonese, the acoustic model and the language model recognize the two languages at the same time, so that the mixed recognition system has no performance loss compared to a single system.

The solution of this embodiment is compared with the solution of using Mandarin and Cantonese recognition resources respectively when decoding the same audio, and then determining the final text recognition result according to the confidence or semantic analysis of the two recognition results. There is a waste of resources.

The solution in this embodiment has less performance loss than the solution in which the original modeling units of the two languages are directly combined and trained to obtain an acoustic model.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of actions combined, but those skilled in the art should know that the present invention is not limited by the described sequence of actions. As in accordance with the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

In some embodiments, embodiments of the present invention further provide a storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the program can perform the steps of the Mandarin and Cantonese mixed speech recognition method.

In some embodiments, embodiments of the present invention further provide an electronic device, which includes: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores data that can be accessed by the at least one processor. Instructions executed by a processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a Mandarin and Cantonese hybrid speech recognition method.

The Mandarin and Cantonese mixed speech recognition device of the above-mentioned embodiment of the present invention can be used to execute the Mandarin and Cantonese mixed speech recognition method of the present invention, and correspondingly achieve the above-mentioned embodiment of the present invention. The technical effect will not be repeated here. In the embodiment of the present invention, the relevant functional modules may be implemented by a hardware processor (hardware processor).

FIG. 8 is a schematic diagram of a hardware structure of an electronic device for a mixed speech recognition method for Mandarin and Cantonese provided by another embodiment of the present application. As shown in Figure 8, the device includes:

One or more processors 810 and a memory 820, one processor 810 is taken as an example in FIG. 8 .

The apparatus for performing the Mandarin and Cantonese mixed speech recognition method may further include: an input device 830 and an output device 840 .

The processor 810, the memory 820, the input device 830, and the output device 840 may be connected through a bus or in other ways, and the connection through a bus is taken as an example in FIG. 8 .

The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as the Mandarin and Cantonese hybrid speech recognition method in the embodiments of the present application. The corresponding program instruction/module. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 820, ie, implements the voice service method of the above method embodiment.

The memory 820 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the voice service device, and the like. Additionally, memory 820 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 820 may optionally include memory located remotely from processor 810 that may be connected to the voice service device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The input device 830 may receive input numerical or character information, and generate signals related to user settings and function control of the voice service device. The output device 840 may include a display device such as a display screen.

The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, perform the Mandarin and Cantonese hybrid speech recognition method in any of the above method embodiments.

The above product can execute the method provided by the embodiments of the present application, and has functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, reference may be made to the methods provided in the embodiments of this application.

The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

(1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions, and its main goal is to provide voice and data communication. Such terminals include: smart phones (eg iPhone), multimedia phones, feature phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has the characteristics of mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, such as iPads.

(3) Portable entertainment equipment: This type of equipment can display and play multimedia content. Such devices include: audio and video players (eg iPod), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

(4) Server: A device that provides computing services. The composition of the server includes a processor, a hard disk, a memory, a system bus, etc. The server is similar to a general computer architecture, but due to the need to provide highly reliable services, the processing power, stability , reliability, security, scalability, manageability and other aspects of high requirements.

(5) Other electronic devices with data interaction function.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence, or the parts that make contributions to related technologies, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic disks , optical disc, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions recorded in the foregoing embodiments, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (12)

1. A method for training a mandarin and cantonese mixed speech recognition model comprises the following steps:

training a multitask model by adopting a mixed voice training sample of N languages, wherein the multitask model comprises a plurality of shared network layers and N task neural network layers which are connected with the last layer of the shared network layers and correspond to the N languages;

and migrating the network parameters of the plurality of shared network layers to a speech recognition model to be trained so as to finish the training of the speech recognition model to be trained.

2. The method of claim 1, wherein training a multitask model with mixed speech training samples in N languages comprises:

training network parameters of the N task neural network layers based on N loss functions corresponding to the N task neural network layers;

collectively training network parameters of the plurality of shared network layers based on at least N loss functions corresponding to the N task neural network layers.

3. The method of claim 2, wherein the multitasking model further comprises a language classification network layer connected to a last layer of the plurality of shared network layers;

the co-training the network parameters of the plurality of shared network layers based at least on the N loss functions corresponding to the N task neural network layers comprises:

collectively training the network parameters of the plurality of shared network layers based on N loss functions corresponding to the N task neural network layers and a loss function corresponding to the language classification network layer.

4. The method of claim 3, wherein collectively training the network parameters of the plurality of shared network layers based on the N penalty functions corresponding to the N task neural network layers and the penalty functions corresponding to the language classification network layers comprises:

training network parameters of the plurality of shared network layers based on a weighted sum of the N loss functions corresponding to the N task neural network layers and the loss functions corresponding to the language classification network layers.

5. A mandarin and cantonese hybrid speech recognition method, comprising:

inputting mixed speech of Mandarin and dialect into the speech recognition model trained by the method of claims 1-4, and performing mixed speech recognition.

6. A mandarin and cantonese hybrid speech recognition model training system, comprising:

the multi-task model training module is used for training a multi-task model by adopting a mixed voice training sample of N languages, wherein the multi-task model comprises a plurality of shared network layers and N task neural network layers which are connected with the last layer of the shared network layers and correspond to the N languages;

and the speech recognition model training module is used for transferring the network parameters of the plurality of shared network layers to a speech recognition model to be trained so as to complete the training of the speech recognition model to be trained.

7. The system of claim 6, wherein the training of the multitask model with the mixed speech training samples in the N languages comprises:

training network parameters of the N task neural network layers based on N loss functions corresponding to the N task neural network layers;

collectively training network parameters of the plurality of shared network layers based on at least N loss functions corresponding to the N task neural network layers.

8. The system of claim 7, wherein the multitasking model further comprises a language classification network layer connected to a last layer of the plurality of shared network layers;

the co-training the network parameters of the plurality of shared network layers based at least on the N loss functions corresponding to the N task neural network layers comprises:

collectively training the network parameters of the plurality of shared network layers based on N loss functions corresponding to the N task neural network layers and a loss function corresponding to the language classification network layer.

9. The system of claim 8, wherein collectively training the network parameters of the plurality of shared network layers based on the N penalty functions corresponding to the N task neural network layers and the penalty functions corresponding to the language classification network layers comprises:

training network parameters of the plurality of shared network layers based on a weighted sum of the N loss functions corresponding to the N task neural network layers and the loss functions corresponding to the language classification network layers.

10. A mandarin and cantonese hybrid speech recognition system comprising:

a speech recognition model trained using the method of claims 1-4;

and the voice input module is used for inputting the mixed voice of the common Chinese and the Guangdong languages into the voice recognition model to perform mixed voice recognition.

11. An electronic device, comprising: at least one processor, and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method of any one of claims 1-5.

12. A storage medium on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 5.

Images