CN111816166A - Voice recognition method, apparatus, and computer-readable storage medium storing instructions - Google Patents

Abstract

A voice recognition method, apparatus, and computer-readable storage medium for storing instructions are provided. The sound recognition method includes: acquiring the time domain feature of the input audio; acquiring the frequency domain feature of the input audio; fusing the time domain feature of the input audio and the frequency domain feature of the input audio, features to perform voice recognition.

Description

Voice recognition method, apparatus, and computer-readable storage medium storing instructions

technical field

The present disclosure relates to the technical field of voice recognition, and in particular, to a voice recognition method and a voice recognition device.

Background technique

Sound recognition is a technology that analyzes the sound emitted by an object and compares it with the sound in a sound database to determine which object the object is. Voice recognition can have various applications, for example, it can be applied to speaker recognition, biometrics, gender/age recognition, etc. Speaker recognition is a biometric technology, also known as voiceprint recognition, which plays an important role in the field of speech processing because it can be widely used in biometric verification, authentication, and security. At present, the recognition effect of the traditional voice recognition scheme is relatively limited, and the effect of voice recognition needs to be further improved.

SUMMARY OF THE INVENTION

The embodiment of the present disclosure discloses a voice recognition method to improve the effect of voice recognition.

According to an aspect of the present disclosure, there is provided a sound recognition method, comprising: acquiring a time-domain feature of input audio; acquiring a frequency-domain feature of the input audio; Domain features are fused, and voice recognition is performed based on the fused features.

Optionally, the fusion of the time domain feature of the input audio and the frequency domain feature of the input audio may include: splicing and combining the time domain feature of the input audio and the frequency domain feature of the input audio. Transform processing to obtain the fused features.

Optionally, performing splicing and transformation processing on the time domain feature of the input audio and the frequency domain feature of the input audio, and obtaining the fused feature may include: combining the time domain feature of the input audio with the The frequency domain features of the input audio are spliced to obtain the spliced features; the two-layer fully connected layer transformation is performed on the spliced features to obtain the fused features.

Optionally, performing splicing and transformation processing on the time domain feature of the input audio and the frequency domain feature of the input audio, and obtaining the fused feature may include: performing a layer on the time domain feature of the input audio. The fully connected layer transforms to obtain the first transformation feature; the frequency domain feature of the input audio is transformed by one layer of the fully connected layer to obtain the second transformation feature; the first transformation feature and the second transformation feature are spliced , to obtain the spliced features; perform a layer of fully connected layer transformation on the spliced features to obtain the fused features.

Optionally, performing splicing and transformation processing on the time domain feature of the input audio and the frequency domain feature of the input audio, and obtaining the fused feature may include: performing two layers on the time domain feature of the input audio. Fully connected layer transformation to obtain a third transformation feature; performing two-layer fully connected layer transformation on the frequency domain features of the input audio to obtain a fourth transformation feature; splicing the third transformation feature and the fourth transformation feature , to obtain the fused features.

According to another aspect of the present disclosure, there is provided a sound recognition apparatus, comprising: a time-domain feature acquisition module configured to acquire time-domain features of input audio; a frequency-domain feature acquisition module configured to acquire frequency-domain features of the input audio Domain features; a sound recognition module configured to fuse the time domain features of the input audio and the frequency domain features of the input audio, and perform sound recognition based on the fused features.

Optionally, the sound recognition module may be configured to perform splicing and transformation processing on the time domain feature of the input audio and the frequency domain feature of the input audio to obtain the fused feature.

Optionally, the sound recognition module can be configured to: splicing the time domain feature of the input audio and the frequency domain feature of the input audio to obtain the spliced feature; perform two-layer full connection on the spliced feature. Layer transformation to obtain the fused features.

Optionally, the sound recognition module may be configured to: perform a layer of fully connected layer transformation on the time domain feature of the input audio to obtain a first transformation feature; perform a layer of fully connected layer on the frequency domain feature of the input audio transformation to obtain a second transformation feature; splicing the first transformation feature and the second transformation feature to obtain a spliced feature; performing a layer of fully connected layer transformation on the spliced feature to obtain the fusion later features.

Optionally, the sound recognition module can be configured to: perform two-layer fully connected layer transformation on the time domain features of the input audio to obtain a third transformation feature; perform two layers of fully connected layers on the frequency domain features of the input audio. Transform to obtain a fourth transformed feature; splicing the third transformed feature and the fourth transformed feature to obtain the fused feature.

According to another aspect of the present disclosure, there is provided a voice recognition device comprising a system of at least one computing device and at least one storage device storing computer instructions, the computer instructions running on the at least one computing device When the at least one computing device is caused to execute the voice recognition method according to the present disclosure.

According to another aspect of the present disclosure, there is provided a computer-readable storage medium storing instructions that, when executed on at least one computing device, cause the at least one computing device to perform a voice recognition method according to the present disclosure.

According to the voice recognition method and the voice recognition apparatus according to the exemplary embodiments of the present disclosure, the time domain information and the frequency domain information of the voice signal are used to jointly perform the voice recognition in a fusion manner, the time information and the frequency information of the voice signal are fully utilized, and the improvement is improved. The performance of voice recognition. For example, when the voice recognition method and the voice recognition apparatus according to the exemplary embodiments of the present disclosure are applied to speaker recognition, the voiceprint recognition is jointly performed by using the time domain information and the frequency domain information of the voice signal in a fusion manner, which fully utilizes the The time information and frequency information of the speech signal improve the performance of speaker recognition.

In addition, according to the voice recognition method and the voice recognition apparatus according to the exemplary embodiments of the present disclosure, the time domain feature and the frequency domain feature are transformed together into the classification feature space by the early fusion method, so the transformation process is to integrate the time domain feature and the frequency domain feature. The feature is performed, that is, the features of all two domains of the audio signal are considered more comprehensively, and therefore, a good sound recognition effect can be achieved.

Description of drawings

These and/or other aspects and advantages of the present disclosure will become apparent, and be more readily understood, from the following description of embodiments, taken in conjunction with the accompanying drawings, wherein:

Figures 1a and 1b show schematic diagrams of existing speaker recognition models.

FIG. 2 is a schematic diagram illustrating a Time-Frequency Network (TFN) model according to an exemplary embodiment of the present disclosure.

Figures 3a, 3b and 3c show schematic diagrams of a fusion manner according to an exemplary embodiment of the present disclosure.

FIG. 4 is a flowchart illustrating a voice recognition method according to an exemplary embodiment of the present disclosure.

FIG. 5 illustrates a block diagram of a voice recognition apparatus according to an exemplary embodiment of the present disclosure.

Figures 6a, 6b, 6c and 6d show schematic diagrams of a speaker recognition system.

Detailed ways

The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to aid in that understanding, but are to be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

It should be noted here that "at least one of several items" in the present disclosure all means including "any one of the several items", "a combination of any of the several items", The three categories of "the whole of the several items" are juxtaposed. For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "execute at least one of step 1 and step 2", which means the following three parallel situations: (1) execute step 1; (2) execute step 2; (3) execute step 1 and step 2.

In the field of voice recognition, for example, in speaker recognition applications, current speaker recognition models may include time domain models and frequency domain models according to different types of input data. As shown in Figures 1a and 1b, Figures 1a and 1b show schematic diagrams of existing speaker recognition models. As shown in Figure 1a, the temporal model uses the raw speech waveform (a temporal representation of the speech signal) as input, that is, the temporal model only utilizes the temporal information of the raw speech to perform speaker recognition. As shown in Figure 1b, the frequency-domain model uses the spectral signal (a frequency-domain representation of the speech signal) as input, that is, the frequency-domain model uses only the frequency-domain information of the speech information to perform speaker recognition. Therefore, none of the current speaker recognition models can achieve the best results. The frequency domain model and the time domain model are introduced in detail below.

frequency domain model

Before using Deep Neural Network (DNN), most speaker recognition methods used frequency domain features to classify speech signals, for example, Gaussian Mixture Model (GMM), i-vector of speech segmentation Feature representation. These methods are performed based on handcrafted frequency-domain features (eg, FilterBank (FBANK) or Mel-Frequency Cepstral Coefficient (MFCC)). With the widespread use of DNNs, DNNs are also designed to automatically extract frequency domain features for speaker recognition. However, these methods only deal with the frequency domain signal, while ignoring the time domain information.

For example, the gist of conventional frequency-domain speaker recognition methods may include one of the following: (1) Combining GMM supervectors with Support Vector Machine (SVM) and approximating based on the KL distance between the two GMM models method to derive linear kernels; (2) model speaker and channel variability and propose new low-dimensional global representations of speech, called unit vectors or i-vectors, which are the basis of most frequency-domain methods for speaker recognition ; (3) Based on i-vector, using pre-trained DNN to generate frame alignment, and improve the equal error rate by 30% compared with traditional systems; (4) By using frequency features with data augmentation (such as FBANK) to align DNNs Training is performed to introduce an x-vector to extract a fixed-length global vector. These methods all use the spectral signal of the speech signal as input, such as Mel Frequency Cepstral Coefficient (MFCC), Perceptual Linear Predictive (PLP) analysis, Linear Predictive Cepstral Coeficient (LPCC), etc. . Although these spectra also contain temporal information, due to time-frequency transformations such as Short-Time Fourier Transformation (STFT), the temporal resolution of the spectra is significantly reduced compared to the temporal resolution of the original speech signal. Specifically, a window-based time-frequency transform (such as a short-time Fourier transform) transforms a signal segment into the frequency domain using a window with a step size to produce time-frequency features, while the temporal resolution will drop from N to N/step (where // represents a rounding operation). Therefore, existing speaker recognition methods using spectrum as input cannot learn temporal features well, and thus cannot fully utilize temporal information.

time domain model

When Convolutional Neural Networks (CNNs) successfully solved large-scale image classification problems and demonstrated strong capabilities in high-dimensional data modeling, CNNs were used to solve speaker recognition problems directly in the time domain . In recent years, end-to-end models designed to extract features directly from raw speech waveforms have shown better performance than traditional methods that only use frequency-domain features. For example, the SincNet model has shown good performance in speaker recognition by designing the first layer filter as a learnable bandpass filter.

However, recent deep learning based methods using raw speech signals as input cannot learn frequency features well due to not applying frequency optimization or frequency transformation in the framework. Specifically, the deep neural network learns many small convolutional filters on the time axis, and the speech signal can be classified using the model. That is, in existing neural network-based speaker recognition models, filters are only learned from raw speech signals with only a time axis. Therefore, the frequency domain signal is ignored. These methods may include: (1) using an end-to-end approach to extract only the temporal information of the original speech to perform speaker recognition based on CNN, (2) combining CNN with Long Short Term Memory (LSTM) to learn from The raw speech signal extracts global vectors to perform speaker recognition, and (3) the SincNet model uses a learnable bandpass filter to replace the first layer of the CNN for better interoperability for improved performance. Here, although the SincNet model utilizes the frequency information of the speech signal using a learnable bandpass filter, it still only takes the original speech waveform as input, so it cannot fully utilize the frequency domain information.

However, for sound signal analysis, both frequency domain information and time domain information are very important, and the lack of information in either domain cannot achieve the best effect of sound signal analysis. In order to make full use of time-domain features and frequency-domain information, using shared or non-shared branches to learn time-domain feature representations and frequency-domain feature representations, the present disclosure proposes a novel Time-Frequency Network (TFN) that takes both raw audio waveform and spectrum as input )Model. Specifically, the TFN model designed by the present disclosure may include a time branch model for extracting the time domain information of the original audio waveform, a frequency domain branch model for extracting the frequency domain information of the spectral signal, and a fusion of the time domain information and the frequency domain information. To perform a fusion model of voice recognition, the output of the TFN model may be the predicted distribution of the person uttering the voice (eg, in the case of application to speaker recognition, the output of the TFN model may be the predicted distribution of the speaker). Hereinafter, a voice recognition method and a voice recognition apparatus according to an exemplary embodiment of the present disclosure will be described in detail with reference to FIGS. 2 to 6d .

FIG. 2 is a schematic diagram illustrating a TFN model according to an exemplary embodiment of the present disclosure.

Referring to FIG. 2 , the TFN model 200 may include three sub-models, ie, a time branch model 201 , a frequency domain branch model 202 and a fusion model 203 .

The temporal branching model 201 may be designed to extract temporal features from the raw audio waveform of the input audio. The temporal branching model 201 may be implemented using any available model that extracts temporal features of the input audio, and the present disclosure is not limited herein. For example, a multi-layer CNN or a Recurrent Neural Network (RNN) or the like can be used to implement the temporal branch model 201 to extract local features in the temporal domain from the original audio signal.

According to an exemplary embodiment of the present disclosure, the temporal branch model 201 can be designed as a SincNet model (or other common CNN models or RNN models). In this case, the first layer of the time branch model 201 is designed as a bandpass filter to model frequency features. Here, the band-pass filter can be expressed as the following formula (1).

g[n,f ₁ ,f ₂ ]=2f ₂ sinc(2πf ₂ n)-2f ₂ sinc(2πf ₁ n) (1)

Among them, g[] represents the output of the band-pass filter, n represents the size of the kernel of the band-pass filter, f ₁ represents the lower limit of the cut-off frequency, f ₂ represents the upper limit of the cut-off frequency, and sinc(x)=sinx/x.

By designing the filter of the first layer of the time branch model 201 as a band-pass filter, the time branch model 201 can have fewer parameters and better interpretability.

In addition, other layers of the temporal branch model 201 can be a typical one-dimensional convolutional layer (Conv), a batch normalization layer (BN) and an activation function layer (ReLU). After several convolutional layers, After the batch normalization layer and the activation function layer, the temporal branch model 201 can output temporal features.

The frequency domain branch model 202 may be designed to extract frequency domain features from the frequency spectrum of the input audio. According to an exemplary embodiment of the present disclosure, MFCC, PLP, LPCC, etc. may be used as a spectrum for extracting frequency-domain features. The frequency domain branch model 202 may also be implemented using any available model for extracting frequency domain features from spectral signals, which the present disclosure is not limited to herein. For example, the frequency-domain branching model 202 may be implemented using a one-dimensional or two-dimensional multilayer CNN to extract frequency-domain features from the spectral signal, or the frequency-domain branching model 202 may be implemented using any GMM, DNN, or RNN to extract frequency-domain features from the spectral signal. Domain features.

The fusion model 203 can be designed to fuse time-domain features and frequency-domain features to perform voice recognition based on the fused features, that is, to output a predicted distribution result. Here, fusion refers to transforming time-domain features and frequency-domain features together into a categorical feature space to obtain categorical features (ie, fused features). Therefore, the fusion processing includes feature splicing processing and transformation processing. Specifically, the fusion model 203 may perform splicing and transformation processing on the time domain features of the input audio and the frequency domain features of the input audio to obtain the fused features. For example, the fusion model 203 may include one feature stitching layer and multiple fully connected layers (FC layers). The feature stitching layer is used to stitch two feature vectors into one feature vector, and the FC layer is used to transform the feature vector. The present disclosure does not limit the number and arrangement of one feature splicing layer and multi-layer FC layers in the fusion model 203 .

According to an exemplary embodiment of the present disclosure, the fusion model 203 may include one feature concatenation layer and two FC layers. According to different transformation types, the fusion model 203 can have three different implementations, namely, early fusion, mid-phase fusion, and late fusion.

Figures 3a, 3b and 3c show schematic diagrams of a fusion manner according to an exemplary embodiment of the present disclosure.

As shown in FIG. 3a, the fusion model 203 may adopt an early fusion method. In the early fusion method, the feature splicing layer is set at the first layer, and the two FC layers are set at the second and third layers, respectively. Specifically, the fusion model 203 may first splicing together two local feature embeddings (ie, the time-domain features output by the time-branch model 201 and the frequency-domain features output by the frequency-domain branch model 202 ) at the first layer, to obtain the spliced The features (ie, global features) of (ie, fused features), voice recognition is performed based on the classification features of the input audio. For example, the classification feature of the input audio is subjected to softmax processing to obtain the predicted classification result (ie, the probability distribution value). The early fusion method first splices the time-domain features and the frequency-domain features, and then transforms the spliced features. Therefore, the transformation process is performed by integrating the time-domain features and the frequency-domain features, that is, it more comprehensively considers all the two aspects of the audio signal. Therefore, a good sound recognition effect can be achieved.

As shown in FIG. 3b, the fusion model 203 may adopt a mid-term fusion method. In the mid-term fusion method, one FC layer is set on the first layer, the feature splicing layer is set on the second layer, and the other FC layer is set on the third layer. Specifically, the fusion model 203 can firstly embed two local features at the first layer (ie, the time-domain features output by the time-branch model 201 and the frequency-domain features output by the frequency-domain branch model 202 ) through an FC layer, and then In the second layer, the features output by the two FC layers are spliced together, and finally the spliced global features are passed through an FC layer in the third layer to project the global features to the classification feature space to obtain the classification features of the input audio. (ie, fused features), voice recognition is performed based on the classification features of the input audio. For example, the classification feature of the input audio is subjected to softmax processing to obtain the predicted classification result (ie, the probability distribution value).

As shown in FIG. 3c, the fusion model 203 may adopt a late fusion method. In the late fusion method, two FC layers are set on the first and second layers respectively, and the feature splicing layer is set on the third layer. Specifically, the fusion model 204 can first embed two local features (ie, the time domain features output by the time branch model 201 and the frequency domain features output by the frequency domain branch model 202 ) at the first layer and the second layer respectively. FC layers are used to project two local features into the classification feature space to obtain the classification features of the time domain features and the classification features of the frequency domain features respectively, and then in the third layer, the classification features of the time domain features and the classification features of the frequency domain features are classified. The features are stitched together to stitch the categorical features in the two low-dimensional categorical feature spaces into categorical features in the high-dimensional categorical feature space, thereby obtaining a global categorical feature in the categorical feature space, and performing voice recognition according to the global categorical feature. For example, the global classification feature is subjected to softmax processing to obtain the predicted classification result (ie, the probability distribution value).

The fusion model 204 can use any of the above-mentioned fusion methods to perform the fusion of the time-domain features of the input audio and the frequency-domain features of the input audio, and can also use any other feasible fusion methods to perform the time-domain features of the input audio and the input audio. For the fusion of features, for example, the number of FC layers may not necessarily be two layers, but can also be a single layer, or more than three layers, and the feature splicing layer can also be set at any position among the multiple layers, or it can be The time-domain features of the audio and the frequency-domain features of the input audio are spliced to directly obtain the fused features to perform sound recognition.

FIG. 4 is a flowchart illustrating a voice recognition method according to an exemplary embodiment of the present disclosure.

Referring to FIG. 4, in step 401, time domain features of the input audio are acquired. Specifically, the time-domain features of the input audio can be obtained by extracting the time-domain features from the original audio waveform of the input audio. According to an exemplary embodiment of the present disclosure, the step of extracting temporal features from the original audio waveform of the input audio may be performed by the temporal branching model 201 . According to another exemplary embodiment of the present disclosure, the time-domain features of the input audio may be acquired from a local storage, a server, or the like. The time-domain feature of the input audio may also be acquired in any other feasible manner, and the present disclosure does not limit the acquisition approach and source.

According to an exemplary embodiment of the present disclosure, the temporal branching model 201 may be implemented using a multi-layer CNN or RNN or the like. According to an exemplary embodiment of the present disclosure, the temporal branching model 201 may be implemented using a SincNet model. Of course, the method of extracting temporal features is not limited to this, and any method can be used to extract temporal features, for example, any multi-layer CNN or RNN can be used to extract temporal features.

In step 402, frequency domain features of the input audio are acquired. Specifically, the frequency-domain features of the input audio can be obtained by performing a time-frequency transform (such as Fast Fourier Transform STFT) on the original audio signal of the input audio, and extracting frequency-domain features from a spectral signal obtained by the time-frequency transform. According to another exemplary embodiment of the present disclosure, the frequency domain features of the input audio may be acquired from a local storage or a server or the like. The frequency domain feature of the input audio can also be acquired in any other feasible manner, and the present disclosure does not limit the acquisition approach and source.

According to an exemplary embodiment of the present disclosure, MFCC, PLP, LPCC, etc. may be used as a spectrum for extracting frequency-domain features.

According to an exemplary embodiment of the present disclosure, the step of extracting frequency domain features from the spectral signal may be performed by the frequency domain branch model 202 . According to an exemplary embodiment of the present disclosure, the frequency domain branching model 202 may be implemented using a one-dimensional or two-dimensional multilayer CNN. Of course, the method of extracting frequency domain features is not limited to this, and any method can be used to extract frequency domain features, for example, any GMM or DNN can also be used to extract frequency domain features from spectral signals.

In addition, steps 401 and 402 may be performed sequentially, in reverse order, or in parallel, and the present disclosure does not limit the execution order of steps 401 and 402 .

In step 403, the time domain features of the input audio and the frequency domain features of the input audio are fused, and sound recognition is performed based on the fused features. According to an exemplary embodiment of the present disclosure, the fusion module 203 may perform the steps of fusing the time domain features of the input audio and the frequency domain features of the input audio, and performing the steps of sound recognition based on the fused features.

According to an exemplary embodiment of the present disclosure, splicing and transformation processing may be performed on the time-domain feature of the input audio and the frequency-domain feature of the input audio to obtain a fused feature. Here, after the time-domain features of the input audio and the frequency-domain features of the input audio are processed by concatenation and transformation, they can be projected into the categorical feature space, that is, transformed into categorical features (ie, fused features). A softmax process is performed on the classification features to obtain predicted classification results (ie, probability distribution values), thereby performing voice recognition.

According to an exemplary embodiment of the present disclosure, the time-domain feature of the input audio and the frequency-domain feature of the input audio may be fused through an early fusion method. In the early fusion method, the feature splicing layer is set at the first layer, and the two FC layers are set at the second and third layers, respectively. Specifically, the time-domain features of the input audio and the frequency-domain features of the input audio can be concatenated (e.g., at the first layer) to obtain concatenated features, and a two-layer FC layer transformation can be performed on the concatenated features (e.g., respectively in the second and third layers) to obtain the fused features. The early fusion method first splices the time-domain features and the frequency-domain features, and then transforms the spliced features. Therefore, the transformation process is performed by integrating the time-domain features and the frequency-domain features, that is, it more comprehensively considers all the two aspects of the audio signal. Therefore, a good sound recognition effect can be achieved.

According to an exemplary embodiment of the present disclosure, the time-domain feature of the input audio and the frequency-domain feature of the input audio may be fused by a mid-term fusion method. In the mid-term fusion method, one FC layer is set on the first layer, the feature splicing layer is set on the second layer, and the other FC layer is set on the third layer. Specifically, a layer of FC layer transformation can be performed on the time domain features of the input audio (for example, in the first layer) to obtain the first transformed feature, and a layer of FC layer transformation can be performed on the frequency domain features of the input audio (for example, in the first layer). The first layer), obtain the second transformation feature (wherein, the order of transformation of the time domain feature of the input audio and the transformation of the frequency domain feature of the input audio is not limited), splicing the first transformation feature and the second transformation feature (eg, at the second layer), get the concatenated features, and perform a layer of FC layer transformation (eg, at the third layer) on the concatenated features to obtain the fused features.

According to an exemplary embodiment of the present disclosure, the time-domain feature of the input audio and the frequency-domain feature of the input audio may be fused in a post-fusion manner. In the late fusion method, two FC layers are set on the first and second layers respectively, and the feature splicing layer is set on the third layer. Specifically, two layers of fully connected layer transformations (eg, at the first and second layers, respectively) may be performed on the time-domain features of the input audio to obtain a third transformed feature, and two layers of fully-connected layers may be performed on the frequency-domain features of the input audio. connecting layer transformations (eg, at the first and second layers, respectively) to obtain a fourth transformed feature (wherein the order of transforming the time-domain features of the input audio and transforming the frequency-domain features of the input audio is not limited), The third transformed feature and the fourth transformed feature are concatenated (eg, at the third layer) to obtain the fused feature.

Of course, the fusion method is not limited to the above method, and any other feasible fusion method can be used to transform the time domain feature of the input audio and the frequency domain feature of the input audio together into the classification feature space. For example, the number of FC layers may not necessarily be Two layers, it can also be a single layer, or more than three layers, the feature splicing layer can also be set at any position among the multiple layers, or it can be performed only by performing the time domain feature of the input audio and the frequency domain feature of the input audio. The fused features are directly obtained by splicing to perform sound recognition.

FIG. 5 illustrates a block diagram of a voice recognition apparatus according to an exemplary embodiment of the present disclosure.

5 , a sound recognition apparatus 500 according to an exemplary embodiment of the present disclosure may include a time-domain feature acquisition module 501 , a frequency-domain feature acquisition module 502 , and a sound recognition module 503 .

The temporal feature extraction module 501 can obtain temporal features of the input audio. Specifically, the time-domain feature acquisition module 501 may acquire the time-domain feature of the input audio by extracting the time-domain feature from the original audio waveform of the input audio. Alternatively, the time-domain feature acquisition module 501 may acquire the time-domain features of the input audio from a local storage or a server. The time-domain feature acquisition module 501 may also acquire the time-domain feature of the input audio in any other feasible manner, and the present disclosure does not limit the acquisition approach and source.

According to an exemplary embodiment of the present disclosure, the temporal feature acquisition module 501 may extract temporal features from the original audio waveform of the input audio through the temporal branching model 201 . According to an exemplary embodiment of the present disclosure, the temporal branching model 201 may be implemented using a multi-layer CNN or RNN or the like. According to an exemplary embodiment of the present disclosure, the temporal branching model 201 may be implemented using a SincNet model. Of course, the method of extracting temporal features is not limited to this, and any method can be used to extract temporal features, for example, any multi-layer CNN or RNN can be used to extract temporal features.

The frequency-domain feature acquisition module 502 may acquire frequency-domain features of the input audio. Specifically, the frequency-domain feature acquisition module 502 may obtain the input audio by performing a time-frequency transform (such as Fast Fourier Transform STFT) on the original audio signal of the input audio, and extracting frequency-domain features from the spectral signal obtained by the time-frequency transform frequency domain features. Alternatively, the frequency-domain feature acquisition module 502 may acquire the frequency-domain features of the input audio from a local storage, a server, or the like. The frequency-domain feature acquisition module 502 may also acquire the frequency-domain feature of the input audio in any other feasible manner, and the present disclosure does not limit the acquisition approach and source.

According to an exemplary embodiment of the present disclosure, MFCC, PLP, LPCC, etc. may be used as a spectrum for extracting frequency-domain features.

According to an exemplary embodiment of the present disclosure, the frequency domain feature acquisition module 502 may extract frequency domain features from the spectral signal through the frequency domain branch model 202 . According to an exemplary embodiment of the present disclosure, the frequency domain branching model 202 may be implemented using a one-dimensional or two-dimensional multilayer CNN. Of course, the method of extracting frequency domain features is not limited to this, and any method can be used to extract frequency domain features.

The sound recognition module 503 may fuse the time domain features of the input audio and the frequency domain features of the input audio, and perform sound recognition based on the fused features. According to an exemplary embodiment of the present disclosure, the voice recognition module 503 may perform voice recognition through the fusion module 203 .

According to an exemplary embodiment of the present disclosure, the sound recognition module 503 may perform splicing and transformation processing on the time-domain features of the input audio and the frequency-domain features of the input audio to obtain fused features. Here, after the time-domain features of the input audio and the frequency-domain features of the input audio are processed by concatenation and transformation, they can be projected into the categorical feature space, that is, transformed into categorical features (ie, fused features). The voice recognition module 503 may perform softmax processing on the classification features to obtain predicted classification results (ie, probability distribution values), thereby performing voice recognition.

According to an exemplary embodiment of the present disclosure, the sound recognition module 503 may fuse the time domain feature of the input audio and the frequency domain feature of the input audio through an early fusion method. In the early fusion method, the feature splicing layer is set at the first layer, and the two FC layers are set at the second and third layers, respectively. Specifically, the sound recognition module 503 can concatenate the time domain features of the input audio and the frequency domain features of the input audio (eg, in the first layer), obtain the concatenated features, and perform a two-layer FC layer on the concatenated features Transform (eg, at the second and third layers, respectively) to obtain the fused features. The early fusion method first splices the time-domain features and the frequency-domain features, and then transforms the spliced features. Therefore, the transformation process is performed by integrating the time-domain features and the frequency-domain features, that is, it more comprehensively considers all the two aspects of the audio signal. Therefore, a good sound recognition effect can be achieved.

According to an exemplary embodiment of the present disclosure, the voice recognition module 503 may fuse the time-domain features of the input audio and the frequency-domain features of the input audio through a mid-term fusion method. In the mid-term fusion method, one FC layer is set on the first layer, the feature splicing layer is set on the second layer, and the other FC layer is set on the third layer. Specifically, the voice recognition module 503 may perform a layer of FC layer transformation (for example, at the first layer) on the time domain features of the input audio to obtain the first transformation feature, and perform a layer of FC layer transformation on the frequency domain features of the input audio. (for example, at the first layer), obtain the second transformation feature (wherein, the order of transformation of the time domain feature of the input audio and the transformation of the frequency domain feature of the input audio is not limited), the first transformation feature and the second transformation The transformed features are spliced (eg, at the second layer) to obtain the spliced features, and a layer of FC layer transformation is performed on the spliced features (eg, at the third layer) to obtain the fused features.

According to an exemplary embodiment of the present disclosure, the sound recognition module 503 may fuse the time-domain features of the input audio and the frequency-domain features of the input audio through a post-fusion method. In the late fusion method, two FC layers are set on the first and second layers respectively, and the feature splicing layer is set on the third layer. Specifically, the voice recognition module 503 may perform two-layer fully connected layer transformation (eg, in the first layer and the second layer, respectively) on the time-domain features of the input audio to obtain a third transformation feature, which is used for the frequency-domain features of the input audio. Perform a two-layer fully connected layer transformation (for example, at the first layer and the second layer, respectively), and obtain a fourth transformation feature (wherein the order of transformation of the time-domain features of the input audio and the transformation of the frequency-domain features of the input audio is not the same. Restricted), concatenate the third transformed feature and the fourth transformed feature (for example, at the third layer) to obtain the fused feature.

Of course, the fusion method is not limited to the above method, and any other feasible fusion method can be used to transform the time domain feature of the input audio and the frequency domain feature of the input audio together into the classification feature space. For example, the number of FC layers may not necessarily be Two layers, it can also be a single layer, or more than three layers, the feature splicing layer can also be set at any position among the multiple layers, or it can be performed only by performing the time domain feature of the input audio and the frequency domain feature of the input audio. The fused features are directly obtained by splicing to perform sound recognition.

According to an exemplary embodiment of the present disclosure, the TFN model proposed in the present disclosure and the voice recognition method and voice recognition apparatus according to the present disclosure can be applied to speaker recognition. When the TFN model proposed by the present disclosure and the voice recognition method and voice recognition apparatus according to the present disclosure can be applied to speaker recognition, the input audio may be the speaker's voice.

Specifically, speaker identification may include speaker identification and speaker verification according to different output types. Speaker recognition is used to determine to which person in the registered population the input speech belongs and to output the index of the predicted person. Speaker verification is used to confirm whether the input speech was uttered by that person and output true or false. Speaker identification is a multi-class problem, and speaker verification is a binary classification problem. Multi-classification problems can be transformed into multiple binary classification problems. The voice recognition method and the voice recognition device according to the present disclosure can be applied to speaker identification and speaker verification.

Depending on whether the user needs to cooperate with the system, speaker identification names can include text-dependent speaker identification and text-independent speaker identification. The text-dependent speaker recognition system requires the user to speak specific content based on the interaction with the system, so that the system can avoid voice attacks after recording and playback, and provide better robustness. However, it requires user cooperation, which is limited in some applications, such as in a non-interactive scenario. Text-independent speaker recognition systems do not need to specify the content of the input speech, the system needs to identify the speaker from speech with unknown content, which is more difficult. Meanwhile, text-independent speaker recognition systems are more widely used due to their less interactive requirements. The voice recognition method and the voice recognition device according to the present disclosure can be applied to text-dependent speaker recognition, and can also be applied to text-independent speaker recognition.

Figures 6a, 6b, 6c and 6d show schematic diagrams of a speaker recognition system. 6a shows a schematic diagram of a text-dependent speaker recognition system, FIG. 6b shows a schematic diagram of a text-dependent speaker verification system, FIG. 6c shows a schematic diagram of a text-independent speaker recognition system, and FIG. 6d shows a text-independent speaker verification system Schematic diagram of the system.

Table 1 below shows the experimental data comparison results between the TFN model proposed in the present disclosure and the traditional SincNet model based on the TIMIT dataset and the LibriSpeech dataset.

[Table 1]

The TIMIT dataset includes 462 speakers and the LibriSpeech dataset includes 2484 speakers. Experiments are performed using models of a certain size by controlling the dimensionality of the categorical feature space. For the TIMIT dataset, the categorical feature space dimension of the model is 1024. For the LibriSpeech dataset, the categorical feature space dimension of the model is 2048. In the TFN model and the SincNet model proposed in the present disclosure, the bandpass filter data is set to 512 for the small model and 1024 for the large model. Furthermore, in the TFN model proposed in the present disclosure, MFCC is used as the spectrum for extracting frequency domain features. The classification error rate (CER) is used to evaluate the model performance, and the lower the CER, the better the model performance. As can be seen from Table 1, for the TIMIT dataset, the CER of the traditional SincNet model is 0.85%, and the CER of the TFN model proposed in this disclosure is 0.65%. For the LibriSpeech dataset, the CER of the traditional SincNet model is 0.96%, and the CER of the TFN model proposed in this disclosure is 0.32%. It can be seen that the TFN model proposed in the present disclosure exhibits better performance.

According to the voice recognition method and the voice recognition apparatus according to the exemplary embodiments of the present disclosure, the time domain information and the frequency domain information of the voice signal are used to jointly perform the voice recognition in a fusion manner, the time information and the frequency information of the voice signal are fully utilized, and the improvement is improved. The performance of voice recognition. For example, when the voice recognition method and the voice recognition apparatus according to the exemplary embodiments of the present disclosure are applied to speaker recognition, the voiceprint recognition is jointly performed by using the time domain information and the frequency domain information of the voice signal in a fusion manner, which makes full use of the The time information and frequency information of the speech signal improve the performance of speaker recognition.

In addition, according to the voice recognition method and the voice recognition apparatus according to the exemplary embodiments of the present disclosure, the time domain feature and the frequency domain feature are transformed together into the classification feature space by the early fusion method, so the transformation process is to integrate the time domain feature and the frequency domain feature. The feature is performed, that is, the features of all two domains of the audio signal are considered more comprehensively, and therefore, a good sound recognition effect can be achieved.

The voice recognition method and the voice recognition apparatus according to the exemplary embodiments of the present disclosure have been described above with reference to FIGS. 2 to 6d .

The various modules shown in FIG. 5 may be configured as software, hardware, firmware, or any combination of the foregoing to perform specific functions. For example, each module may correspond to a dedicated integrated circuit, may also correspond to pure software code, or may correspond to a module combining software and hardware. In addition, one or more functions implemented by the security monitoring module can also be uniformly performed by components in a physical entity device (eg, a processor, a client or a server, etc.).

Furthermore, the method described with reference to FIG. 4 may be implemented by programs (or instructions) recorded on a computer-readable storage medium. For example, in accordance with exemplary embodiments of the present disclosure, a computer-readable storage medium may be provided that stores instructions that, when executed on at least one computing device, cause the at least one computing device to execute a sound according to the present disclosure recognition methods.

The computer program in the above-mentioned computer-readable storage medium can run in an environment deployed in computer equipment such as a client, a host, an agent device, a server, etc. It should be noted that the computer program can also be used to perform additional steps in addition to the above-mentioned steps or More specific processing is performed when the above steps are performed, and the contents of these additional steps and further processing have been mentioned in the description of the related method with reference to FIG.

It should be noted that each module according to the exemplary embodiment of the present disclosure can completely rely on the running of the computer program to achieve corresponding functions, that is, each module corresponds to each step in the functional architecture of the computer program, so that the entire system can be implemented through a special software package (for example, the lib library) is called to implement the corresponding function.

On the other hand, each module shown in FIG. 5 can also be implemented by hardware, software, firmware, middleware, microcode or any combination thereof. When implemented in software, firmware, middleware, or microcode, program codes or code segments for performing corresponding operations may be stored in a computer-readable medium such as a storage medium, so that a processor can read and execute the corresponding program by reading code or code segment to perform the corresponding action.

For example, exemplary embodiments of the present disclosure may also be implemented as a computing device including a storage component and a processor, the storage component stores a computer-executable instruction set, and when the computer-executable instruction set is executed by the processor, executes the A voice recognition method according to an exemplary embodiment of the present disclosure.

Specifically, the computing device may be deployed in a server or a client, or may be deployed on a node device in a distributed network environment. Furthermore, the computing device may be a PC computer, a tablet device, a personal digital assistant, a smartphone, a web application, or other device capable of executing the set of instructions described above.

Here, the computing device does not have to be a single computing device, but can also be any set of devices or circuits capable of individually or jointly executing the above-mentioned instructions (or instruction sets). The computing device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces locally or remotely (eg, via wireless transmission).

In a computing device, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a special purpose processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.

Some operations described in the voice recognition method according to an exemplary embodiment of the present disclosure may be implemented by software, some operations may be implemented by hardware, and these operations may also be implemented by a combination of software and hardware.

The processor may execute instructions or code stored in one of the storage components, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transport protocol.

The memory component may be integrated with the processor, eg, RAM or flash memory arranged within an integrated circuit microprocessor or the like. Additionally, the storage components may include separate devices, such as external disk drives, storage arrays, or any other storage device that may be used by a database system. The storage component and the processor may be operatively coupled, or may communicate with each other, eg, through I/O ports, network connections, etc., to enable the processor to read files stored in the storage component.

In addition, the computing device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the computing device may be connected to each other via a bus and/or network.

The voice recognition method according to the exemplary embodiment of the present disclosure may be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams may be equally integrated into a single logical device or operate along non-precise boundaries.

Therefore, the voice recognition method described with reference to FIG. 4 may be implemented by a voice recognition device including at least one computing device and at least one storage device storing computer instructions.

According to an exemplary embodiment of the present disclosure, at least one computing device is a computing device for executing a voice recognition method according to an exemplary embodiment of the present disclosure, and a computer-executable instruction set is stored in the storage device. When executed by at least one computing device, the voice recognition method described with reference to FIG. 4 is executed.

Various exemplary embodiments of the present disclosure have been described above, and it should be understood that the above description is merely exemplary and not exhaustive, and the present disclosure is not limited to the disclosed exemplary embodiments. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of this disclosure. Therefore, the scope of protection of the present disclosure should be determined by the scope of the claims.

Claims (12)

1. A method of voice recognition, comprising:

acquiring time domain characteristics of input audio;

acquiring frequency domain characteristics of the input audio;

and fusing the time domain characteristics of the input audio and the frequency domain characteristics of the input audio, and executing sound identification based on the fused characteristics.

2. The sound recognition method of claim 1, wherein the fusing the time-domain features of the input audio and the frequency-domain features of the input audio comprises:

and splicing and transforming the time domain characteristics of the input audio and the frequency domain characteristics of the input audio to obtain the fused characteristics.

3. The sound recognition method of claim 2, wherein the splicing and transforming the time-domain features of the input audio and the frequency-domain features of the input audio to obtain the fused features comprises:

splicing the time domain characteristics of the input audio and the frequency domain characteristics of the input audio to obtain spliced characteristics;

and executing two-layer full-connection layer transformation on the spliced features to obtain the fused features.

4. The sound recognition method of claim 2, wherein the splicing and transforming the time-domain features of the input audio and the frequency-domain features of the input audio to obtain the fused features comprises:

performing a layer of full-link layer transform on the time domain features of the input audio to obtain first transform features;

performing a layer of full-link layer transform on the frequency domain characteristics of the input audio to obtain second transform characteristics;

splicing the first transformation characteristic and the second transformation characteristic to obtain a spliced characteristic;

and executing one-layer full-connection layer transformation on the spliced features to obtain the fused features.

5. The sound recognition method of claim 2, wherein the splicing and transforming the time-domain features of the input audio and the frequency-domain features of the input audio to obtain the fused features comprises:

performing two-layer full-link layer transformation on the time domain characteristics of the input audio to obtain third transformation characteristics;

performing two-layer full-connected layer transformation on the frequency domain characteristics of the input audio to obtain fourth transformation characteristics;

and splicing the third transformation characteristic and the fourth transformation characteristic to obtain the fused characteristic.

6. A voice recognition apparatus, comprising:

a time domain feature acquisition module configured to acquire a time domain feature of an input audio;

a frequency domain feature acquisition module configured to acquire a frequency domain feature of the input audio;

a voice recognition module configured to fuse the time-domain features of the input audio and the frequency-domain features of the input audio and perform voice recognition based on the fused features.

7. The voice recognition apparatus of claim 6, wherein the voice recognition module is configured to:

and splicing and transforming the time domain characteristics of the input audio and the frequency domain characteristics of the input audio to obtain the fused characteristics.

8. The voice recognition apparatus of claim 7, wherein the voice recognition module is configured to:

splicing the time domain characteristics of the input audio and the frequency domain characteristics of the input audio to obtain spliced characteristics;

and executing two-layer full-connection layer transformation on the spliced features to obtain the fused features.

9. The voice recognition apparatus of claim 7, wherein the voice recognition module is configured to:

performing a layer of full-link layer transform on the time domain features of the input audio to obtain first transform features;

performing one-layer full-connection layer transformation on the frequency domain characteristics of the input audio to obtain second transformation characteristics;

splicing the first transformation characteristic and the second transformation characteristic to obtain a spliced characteristic; and executing one-layer full-connection layer transformation on the spliced features to obtain the fused features.

10. The voice recognition apparatus of claim 7, wherein the voice recognition module is configured to:

performing two-layer full-link layer transformation on the time domain characteristics of the input audio to obtain third transformation characteristics;

performing two-layer full-connected layer transformation on the frequency domain characteristics of the input audio to obtain fourth transformation characteristics;

and splicing the third transformation characteristic and the fourth transformation characteristic to obtain the fused characteristic.

11. A voice recognition apparatus comprising at least one computing device and at least one memory device having computer instructions stored thereon, wherein the computer instructions, when executed by the at least one computing device, cause the at least one computing device to perform a voice recognition method as claimed in any one of claims 1 to 5.

12. A computer-readable storage medium having instructions stored thereon, which when executed on at least one computing device, cause the at least one computing device to perform a voice recognition method as claimed in any one of claims 1 to 5.

Images