CN111816215A - Voice endpoint detection model training and use method and device - Google Patents

Abstract

The invention discloses a method and a device for training and using a voice endpoint detection model, wherein the training method comprises the following steps: inputting training audio into a generalized voice endpoint detection model; detecting a plurality of audio events present in the training audio via the generalized speech endpoint detection model, wherein the plurality of audio events include a speak-by-person event, a silence event, and at least one noise event; obtaining the voice and non-voice distinguishing results of the plurality of audio events output by the generalized voice endpoint detection model; computing a loss function based on the audio event labels of the training audio and the output of the generalized speech endpoint detection model; and optimizing the generalized speech endpoint detection model by controlling the loss function. The scheme of the embodiment of the application can distinguish different types in the non-voice part, can improve the accuracy of classification, and is not easy to misjudge noise as the voice part.

Description

Method and device for training and using voice endpoint detection model

technical field

The invention belongs to the field of language models, and in particular relates to a method and device for training and using a voice endpoint detection model.

Background technique

In the related art, there are various types of speech endpoint detection models, including speech endpoint detection models that use thresholds such as short-term energy and zero-crossing rate as the criteria for distinguishing between speech and non-speech, and speech endpoints trained using discriminative models such as neural networks. Detector.

On the one hand, the threshold discrimination method mainly uses short-term energy, zero-crossing rate and other indicators as thresholds. After extracting acoustic features from the audio, these indicators are calculated for each frame or each segment, and then the audio is distinguished by whether the threshold is reached. Voice part and non-voice part.

On the other hand, the model discrimination method mainly uses a discriminative model such as a deep neural network, takes acoustic features as input, and then trains the hidden layer of the neural network, and finally outputs the posterior probability of whether each frame is speech or non-speech. Compare the magnitude of the posterior probability to judge speech or non-speech.

During the process of realizing the present application, the inventor found that the existing solution has at least the following defects:

The main defect of the related art is that speech and non-speech cannot be distinguished well in a noisy environment. Some noise-adaptive technical methods require a lot of labeled data for training to be robust to some specific noises, and these labeled data are often difficult to obtain, or the amount is not so large.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a method and device for training and using a voice endpoint detection model, which are used to solve at least one of the above technical problems.

In a first aspect, an embodiment of the present invention provides a method for training a voice endpoint detection model, including: inputting training audio into a voice endpoint detection model in a broad sense; There are multiple audio events, wherein the multiple audio events include a human speaking event, a silence event, and at least one noise event; acquiring the voice and the multiple audio events output by the voice endpoint detection model in the broad sense. Non-speech discrimination result; a loss function is calculated based on the audio event labeling of the training audio and the output of the speech endpoint detection model in a broad sense, wherein the audio event labeling includes pre-labeling the training audio with audio events ; and optimizing the generalized speech endpoint detection model by controlling the loss function.

In a second aspect, an embodiment of the present invention provides a method for using a voice endpoint detection model, including: detecting multiple audio events existing in the input audio through a voice endpoint detection model in a broad sense trained by the method according to the first aspect, Wherein, the plurality of audio events include a human speaking event, a silence event and at least one noise event; and a distinction result of speech and non-speech of the plurality of audio events output by the speech endpoint detection model in the broad sense is obtained.

In a third aspect, an embodiment of the present invention provides an apparatus for training a voice endpoint detection model, including: an input module configured to input training audio into a voice endpoint detection model in a broad sense; The voice endpoint detection model detects a plurality of audio events existing in the training audio, wherein the plurality of audio events include a human speaking event, a silence event and at least one noise event; an output module is configured to obtain the generalized The results of distinguishing speech and non-speech of the multiple audio events output by the speech endpoint detection model; the loss calculation module is configured to calculate the loss based on the audio event annotation of the training audio and the output of the speech endpoint detection model in a broad sense function, wherein the audio event labeling comprises pre-labeling the training audio with audio events; and an optimization module configured to optimize the speech endpoint detection model in a broad sense by controlling the loss function.

In a fourth aspect, an embodiment of the present invention provides an apparatus for using a voice endpoint detection model, comprising: a model processing module configured to detect an input audio through a voice endpoint detection model in a broad sense trained by the method according to claims 1-4 A plurality of audio events existing in the audio event, wherein the plurality of audio events include a human speech event, a silence event and at least one noise event; Discrimination results of speech and non-speech for multiple audio events.

A fifth aspect provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, The instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the voice endpoint detection model training and use method of any embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, and when the program is When the instructions are executed by a computer, the computer is made to execute the steps of the method for training and using the voice endpoint detection model according to any embodiment of the present invention.

The solution provided by the method and device of the present application uses the method of audio event detection to solve the problem of audio endpoint detection, which is relatively innovative. There is no similar work to solve this problem before. The biggest advantage of this approach is that It improves the detection performance in noisy environments, because it refines the original non-speech category, distinguishes silence and various types of noise, which can reduce the possibility of misjudging various noises as speech, the original speech In the endpoint detection model, all non-voices are classified into one category, but there are many different types of noise in this category, and their characteristics are completely different, so the similarity between categories will be relatively low during training, which will reduce the model's performance. Distinctive.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a flowchart of a method for training a voice endpoint detection model according to an embodiment of the present invention;

2 is a flowchart of a method for using a voice endpoint detection model according to an embodiment of the present invention;

3 is a flowchart of a model provided by an embodiment of the present invention;

Figure 4 shows the distribution of evaluation data between Aurora4 (dark) and DCASE18 (light) with respect to duration (left) and number of segments per utterance (right);

Figure 5 shows per-frame probability output for three sample segments with visual speech occurrences (boxes, grey);

6 is a block diagram of an apparatus for training a voice endpoint detection model according to an embodiment of the present invention;

7 is a block diagram of an apparatus for using a voice endpoint detection model according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to FIG. 1 , which shows a flowchart of an embodiment of the voice endpoint detection model training method of the present application. The voice endpoint detection model training method of the present embodiment can be applied to training the voice endpoint detection model. There is no limit to this.

As shown in Figure 1, in step 101, the training audio is input into the voice endpoint detection model in a broad sense;

In step 102, a plurality of audio events existing in the training audio are detected via the generalized speech endpoint detection model, wherein the plurality of audio events include a human speaking event, a silence event and at least one noise event;

In step 103, obtain the distinction result of speech and non-speech of the multiple audio events output by the speech endpoint detection model in the broad sense;

In step 104, a loss function is calculated based on the audio event labeling of the training audio and the output of the speech endpoint detection model in a broad sense, wherein the audio event labeling includes pre-labeling the training audio with audio events;

In step 105, the generalized speech endpoint detection model is optimized by controlling the loss function.

In some optional embodiments, the audio event annotations are paragraph-level annotations. Paragraph-level annotations are easier to obtain, and the use of paragraph-level annotations in audio event detection methods does not affect the annotation effect.

In some optional embodiments, before the inputting the training audio into the speech endpoint detection model in a broad sense, the method further comprises: extracting acoustic features of the training audio; and using a convolutional recurrent neural network The model is trained to classify the acoustic features.

In some optional embodiments, the detecting a plurality of audio events existing in the training audio via the speech endpoint detection model in the broad sense comprises: using audio event detection in the speech endpoint detection model in the broad sense to identify multiple audio events present in the training audio.

Please refer to FIG. 2 , which shows a method for using a voice endpoint detection model provided by an embodiment of the present application.

As shown in FIG. 2 , in step 201, a plurality of audio events existing in the input audio are detected via the speech endpoint detection model in a broad sense trained by the method according to claims 1-4, wherein the plurality of audio Events include human speaking events, silence events, and at least one noise event;

In step 202, a result of distinguishing speech and non-speech of the plurality of audio events output by the speech endpoint detection model in the broad sense is obtained.

In some optional embodiments, acquiring the speech and non-speech discrimination results of the plurality of audio events output by the speech endpoint detection model in the broad sense includes: adding a double threshold based on the plurality of audio events The processing method obtains the distinction between speech and non-speech.

In some optional embodiments, the post-processing method based on the multiple audio events plus a double threshold to obtain a distinction result between speech and non-speech includes: based on the identified multiple audio events, the person speaking event As a voice part, the silence event and the at least one noise event are regarded as a non-voice part; and a voice endpoint of the audio to be detected is determined based on the discrimination result of the voice part and the non-voice part.

The following describes some problems encountered by the inventor in the process of implementing the present invention and a specific embodiment of the finalized solution, so that those skilled in the art can better understand the solution of the present application.

Those skilled in the art generally want to solve these defects in the related art, and usually adopt some noise reduction methods, that is, perform various noise removal processing on the audio that needs to be judged, reduce the influence of noise, and then perform voice and noise reduction processing. The distinction of speech, or some methods of noise adaptation, is to use noisy data for training or extract some features of scene noise and add them to the model to assist in the distinction for some specific scenes.

In the process of implementing the embodiments of the present application, the inventor found that the above-mentioned defects in the related art are mainly caused by a relatively common assumption, that is, the vast majority of voice endpoint detection models are to distinguish between two categories, That is, speech and non-speech, speech is the part of people speaking, and then mute and various noises are classified into the non-speech category, but in fact, the characteristics of silent segments and various types of noise are different. into the same class will lead to a decrease in the discriminative performance of the model.

The method proposed in the embodiment of the present application is to use the method of audio event detection to help train the voice endpoint detection model, that is, to use audio event detection to detect multiple audio events in the audio. Events, as well as various noise events, such as bird calls, running water, etc., after this distinction is completed, the event of people speaking is regarded as the voice part, and other various events are regarded as the non-voice part. The advantage of this is that different types (silence and different types of noise) in the non-speech part are also distinguished, which can improve the accuracy of classification, and it is not easy to misjudge the noise as the speech part.

Figure 3 is the flow chart of our model, in which the gray part is the part of model training. First, the CRNN (Convolutional Recurrent Neural Network, Convolutional Recurrent Neural Network) model is used to train and classify the input acoustic features, and then our proposed GPVAD ( The General Purpose Voice Activity Detection (VAD in a broad sense) model uses the audio event detection model to identify audio events (speaking, cat meowing, door opening, etc.) existing in the input audio, which is different from the traditional voice endpoint detection model VAD In the training process of -C, the GPVAD model does not need frame-level annotations to calculate the loss function, but only needs paragraph-level annotations that are easier to obtain (that is, the sequence of 0, 1 whether each event occurs in a certain piece of audio) to calculate the paragraph-level loss function. Then in the inference and evaluation (test and evaluation) part, add the post-processing method of Double threshold (double threshold) to obtain the distinction between speech and non-speech. Among them, Label represents the label, Speech represents the speech, Noise represents the noise, Prediction represents the prediction, and truth represents the truth.

In the process of realizing this application, the inventor has also adopted some alternatives. One alternative is to use the same data and annotations to train the voice event detection model and the voice endpoint detection model, that is, both use frame-level annotations. Calculate the same loss function, which guarantees comparability between the two models. But the disadvantage is that frame-level annotations are not easy to obtain, especially for audio events, frame-level annotations are more difficult to obtain, and the problem we have to solve also includes too little data with complete frame-level annotations, so we did not use exactly the same. Instead, use paragraph-level annotations to train the audio event detection model and frame-level annotations to train the audio endpoint detection model.

The method proposed in the embodiment of the present application mainly uses the method of audio event detection to solve the problem of audio endpoint detection, which is relatively innovative. There is no similar work to solve this problem before. The biggest advantage of this method is that It improves the detection performance in noisy environments, because it refines the original non-speech category, distinguishes silence and various types of noise, which can reduce the possibility of misjudging various noises as speech, the original speech In the endpoint detection model, all non-voices are classified into one category, but there are many different types of noise in this category, and their characteristics are completely different, so the similarity between categories will be relatively low during training, which will reduce the model's performance. Distinctive.

The inventor's process of implementing the embodiments of the present application, as well as some experimental procedures and corresponding experimental data in the process are described below, so that those skilled in the art can better understand the technical solutions of the present application.

Traditional supervised speech endpoint detection techniques can work well in clean and noise-free specific environments. However, the performance will drop significantly in real noisy scenes. A possible bottleneck is that speech in real scenes usually has a lot of unpredictable noise, so frame-level prediction is difficult for traditional supervised speech endpoint detection models. We propose a general speech endpoint detection framework (GPVAD) that can be trained on noisy data more easily than semi-supervised training and requires only paragraph-level annotations. We propose two GPVAD models, GPV-F, a multi-classifier trained on the Audioset dataset containing 527 audio events, and GPV-B, which only distinguishes between speech and noise. We compare these two GPV models as well as the traditional CRNN-based speech endpoint detection model (VAD-C) on three different test sets (clean, synthetic noise, real scene). The results show that the detection performance of our proposed GPV-F model is comparable to the conventional VAD-C model on both clean and synthetic test sets. At the same time, in the test of real scenes, both the frame-level evaluation indicators and the paragraph-level evaluation indicators show that the results of the GPV-F model are much better than the traditional VAD-C model. In real scenarios, the relatively simple GPV-B model also achieves comparable performance to the VAD-C model.

1 Introduction

The main purpose of Voice Endpoint Detection (VAD) is to detect speech segments and distinguish them from non-speech, and it is a key component for tasks such as speech recognition, speaker identification, and speaker confirmation. Deep learning methods have been successfully applied to VAD. For VAD in complex environments, Neural Networks (NN) have been successful. Compared with traditional methods, Deep Neural Networks (DNN, Deep Neural Networks), especially Convolutional Neural Networks (CNN, Convolutional Neural Networks) provide improved modeling capabilities, while Recurrent (RNN) and Long Short-Term Memory (LSTM, Long Short TermMemory) network can better model long-term dependencies between sequence inputs. However, despite applying deep learning methods, NN-based VAD training still requires frame-level labels. Therefore, the exploited training data is usually performed in a controlled environment with or without other synthetic noise. This inevitably prevents the application of VAD in the real world, because speech in real scenes is often accompanied by countless unseen noises with different characteristics.

Therefore, this paper aims to propose a method for detecting speech outside a clean and noise-free noisy environment.

It should be noted that frame-level labels for real-life audio are relatively difficult to obtain due to the high cost of human labeling, and label prediction from hidden Markov models requires prior knowledge about the language used. The task of detecting speech components while enabling training on noisy data is related to Weakly Supervised Sound Event Detection (WSSED), which detects and localizes distinct sounds, including speech via paragraph-level supervision. Since the WSSED system is validated to be robust to noise and only requires paragraph-level labels, this work integrates the WSSED method to scale VAD to speech-in-the-wild scenes and relax the reliance on frame labels. Specifically, we investigate two questions: 1) Can the performance of current, multi-class WSSED models be comparable to DNN-based VADs; 2) Can paragraph-level training replace frame-level label training? Therefore, we introduce our framework, a generalized training framework for VAD (GPVAD, see Figure 1). In general, we refer to two distinct aspects: first, the framework is robust against noise and can be deployed in real life scenarios; second, the framework can be trained on unconstrained data, thereby Can learn on large amounts of unlabeled data such as noisy online videos.

The document is structured as follows: In Section 2, we briefly review related work on WSSED and how it can be used for VAD tasks in real-world settings. In Section 3, the GPVAD method is introduced. Furthermore, in Section 4, we introduce the experimental setup and provide implementation details. The results are presented in Section 5, and finally the conclusions are provided in Section 6.

2. Weakly supervised audio event detection

Since WSSED can detect speech well in noisy environments without frame-level labeling, we borrow this idea to implement VAD in real environments. Here, we present related work on Audio Event Detection (SED, SoundEvent Detection), which aims to classify (audio markers) and possibly locate multiple simultaneous sound events from a given audio segment. In this work, we mainly focus on Weakly Supervised SED (WSSED), a semi-supervised task that only has access to paragraph-level labels during training and requires classification and localization of specific events during evaluation . This weakly supervised approach enables training on noisy data with less demanding labelling methods. Recent advances in weakly supervised audio event detection, especially the Detection and Classification of Acoustic Scenes and Events (DCASE) challenge, have brought great progress in predicting accurate sound event boundaries and event labels. In particular, recent work has shown encouraging performance in the detection of short-term discontinuous events such as speech.

FIG. 3 shows the framework proposed by the embodiments of the present application. The CRNN architecture is utilized, while GPVAD is trained with paragraph-level labels, and VAD-C is trained with frame-level labels. Each Conv2d block represents a batch normalization followed by a 0-padded 2D convolution with a kernel size of 3 × 3, using ReLU with leakage as the activation function, and a negative slope of 0.1. The CNN output is fed to a bidirectional Gated Recurrent Unit (GRU) with 128 hidden units. The architecture downsamples the time dimension T by a factor of 4, and then upsamples it to match the time dimension of the original input. For GPV-F, the number of events E is set to 527, and for GPV-B and VAD-C, the number of events is set to 2. After post-processing, only output events (speech) are retained for final evaluation.

3 Using VAD in a noisy environment with WSSED

Traditionally, VADs for noisy scenes are modeled according to Equation (1). Suppose that the additive noise u can be filtered from the observed speech signal x to obtain a clear speech s.

x=s+u(1)

However, modeling u directly is tricky because each type of noise has its own characteristics. Therefore, we aim to understand the nature of s by looking at potentially L distinct non-speech events (u1...,uL). These events are not limited to background/foreground noise and can have different real world sounds (eg cats, music).

X={x1,...,xl,...,xL}

xl=(s,ul)(2)

Our method derives from multiple instance learning (MIL), which means that training set knowledge about a particular label is incomplete (e.g., speech is never directly observed). Here, we model the observed speech data X as "bags" containing passages where speech and any other possibly noisy background/foreground event label l ∈ {1,...,L} co-occur, Possible event labels L<E (equation (2)). Suffice it to say, our approach aims to refine the model's beliefs about speech signals in complex environmental scenarios. The advantage of this modeling approach is that it can be applied to both frame- and paragraph-level training. Therefore, our GPVAD relaxes these constraints by allowing training at the segment/full-sentence level, where each training segment contains at least one event of interest. We propose two different models: GPV-F, which outputs E=527 labels (L=405), and naive GPV-B, E=2, L=1. GPV-F can be seen as a full-fledged WSSED method that uses the most label supervision and is therefore more demanding on labels than GPV-B, which only requires knowledge about segments containing speech. However, GPV-F should be able to model each individual noise event instead of clustering all noises into one class (GPV-B), thus potentially enhancing performance in heavy noise scenarios. The two models were compared with a model trained at the frame level (aka VAD-C).

All models share a common backbone Convolutional Recurrent Neural Network (CRNN) method used in WSSED, which is robust to short-duration discrete events such as speech. The following modifications are made to the above method: 1. An upsampling operation is added to keep the temporal resolution of the model constant. 2. Use Lp pooling and set the default value (p=4) as it is beneficial for duration invariance estimation. Unlike VAD-C training, which can use frame-level labels, our GPVAD framework is divided into two distinct stages. During training, only labels for fragments/full sentences are accessible. Therefore, a temporal pooling function (equation (4)) is required. During inference, post-processing (Section 4.3) is required to convert probabilistic sequences into binary labels (absence/presence of events) and discard all predicted non-speech labels. The frame is shown in Figure 3.

4. Experiment

In our work, deep neural networks are implemented in PyTorch and front-end feature extraction utilizes librosa. Codes will be available online.

4.1 Dataset

Table 1: Training datasets for GPVAD (audio set) and VAD-C (Aurora4+), and three proposed test sets for clean, synthetic noise, and real scenes. Duration represents the approximate speaking time.

Figure 4 shows: The distribution of evaluation data between Aurora4 (dark) and DCASE18 (light) with respect to duration (left) and number of fragments per sentence (right). Color works best.

The datasets utilized in this work can be divided into training data parts (which differ between GPVAD and VAD methods) and evaluation data, which are shared by both methods. Our main GPVAD training dataset is the "balanced" set provided by the AudioSet corpus, which contains 21100/22160 (due to partial unavailability) 10-second Youtube audio clips grouped into 527 noisy event labels. Of the 21100 paragraphs available (58h), 5452 paragraphs (≈15h) were marked as containing speech, but were consistently side-by-side with L=405 other events (eg, dog barking). Regarding GPV-B, instead of treating speech as "noise", we replaced all 526 events in the balanced dataset, so XGPV-B={(s, unoise, ), unoise}. It is important to note that for the training of GPVB/V, speech is never observed alone.

Our VAD-C model was trained on the Aurora4 training set, which was extended on the 15h subset of Switchboard data to obtain our Aurora4+ training subset, which contains clean as well as synthetic noise data. The additive synthetic noise (Syn) is obtained from six different noise types (car, murmur, restaurant, street, airport, and train) with randomly selected SNRs between 10 and 20 dB for addition. All utilized datasets are described in Table 1. Three different evaluation scenarios are proposed. First, we test on a clean Aurora4 test set of up to 40 minutes. Second, we synthesize a noisy test set based on the clean Aurora4 test set by randomly adding noise from 100 noise types with SNR from 5db to 15db in steps of 1db. Finally, we merge the dev and test sets of the audio set of the DCASE18 challenge [10] itself to create evaluation data for our real-world scenarios. The DCASE18 data provides ten home environment event labels, we ignore all labels except speech, but report the number of instances where non-speech labels are present. Our DCASE18 evaluation set contains 596 utterances labelled "speech", 414 utterances (69%) contain another non-speech label, 114 utterances (20%) contain speech only and 68 utterances (11%) ) contains two or more non-speech tags.

As can be seen in Figure 4, the DCASE18 evaluation dataset differs from the Aurora4 dataset in the average duration of speech (1.49s vs. 3.31s) and the number of detected fragments within a sentence (3.87 vs. 2.08).

4.2 Evaluation indicators

Frame-level: For frame-level evaluation, we utilize frame-level macro/micro-average F1 score (F1-macro, F1-micro), area under the curve (AUC, Area Under the Curve) and frame error rate (FER, frame error rate) .

Paragraph level: For paragraph level evaluation, we use event-based F1-Score (Event-F1). Event F1 computes onset, offset and predicted labels overlap with ground truth and are therefore measures of temporal consistency. Based on the WSSED study, we set the t-collar to 200ms to allow for an error at the start of the prediction point and further allow a 20% error in duration between reference and prediction.

4.3 Settings

Regarding feature extraction, in this work, 64-dimensional log-Mel power spectrograms (LMS, log-Mel power spectrograms) are used for all experiments. Each LMS sample uses a Hann window, extracted by a 2048-point Fourier transform every 20ms, with a window size of 40ms. During training, all data is padded with 0s to the longest sample length in the batch, while during inference, a batch size of 1 will be used, meaning no padding.

之间的所有实验的训练准则是所有样本N的交叉熵方程式(3)。线性softmax(方程式(4))用作合并帧级别的时间合并层单个向量表示y(e)∈[0,1]E的概率yt(e)∈[0,1]。true outcome y and prediction

The training criterion for all experiments in between is the cross-entropy equation (3) for all samples N. Linear softmax (Equation (4)) is used as the temporal merging layer at the merging frame level. A single vector represents the probability yt(e) ∈ [0,1] of y(e) ∈ [0,1]E.

GPVAD: The available training data is split into 90% training balanced label data and 10% validation set. Due to the inherent label imbalance inherent in audio sets, sampling is required so that each batch contains evenly distributed segments within each label. Training uses Adam optimization with a starting learning rate of 1e-4, a batch size of 64, and is terminated after seven epochs if the metric does not decrease on the validation dataset.

VAD-C: VAD-C training uses a batch size of 20, while for padded frames, no loss function is computed (equation (3)). The learning rate is set to 1e-5, and SGD is used for model optimization. The training target labels are obtained by aligning from the Kaldi-trained ASRHMM model.

Post-processing During inference, post-processing is required to obtain hard labels from the sequence of class probabilities (yt(e)). Here we use dual threshold postprocessing which uses two thresholds

5 results

Our results are shown in Table 2. First, we provide evidence that our VAD-C model achieves comparable performance with other deep neural network approaches. Comparing VAD-C with GPV-B/F, it can be seen that VAD-C is indeed the best performing model given our metrics for clean and synthetic noisy datasets. However, evaluation on real datasets revealed something else. Here, VAD-C does not seem to be able to compete against the naive GPV-B method (AUC 87.87 vs 89.12, FER 21.92 vs 19.65), suggesting that VAD-C is more likely to misclassify speech in the presence of real noise. Furthermore, GPV-F outperforms VAD-C for every proposed metric in real-world situations. Our proposed GPV-F method can also be considered robust to noise, since its performance difference between synthetic noise and real scenes is small.

Even though the average performance of GPV-B is not as good as the other two methods, it should be noted that this is the lowest cost system, because the label data for GPV-B is inherently a binary problem of whether anyone hears any speech in a passage, which is A method that scales to big data cheaply. We conclude that GPVAD models trained using only paragraph-level labels are competitive with models trained on frame-level labels.

Quantitative results

To visualize model capability, three segments (one Aurora4Noisy, two DCASE18) were sampled from the test set, and the output probabilities for each frame are shown in Figure 5. In the synthetic Aurora4 test at the top, we can see that our GPVAD model is able to model short pauses between two speech segments that fail VAD-C, but both None of the GPVAD models can correctly estimate the end of the second speech segment. The center sample further demonstrates a typical VADC problem in real scenarios: for most speech, it cannot distinguish foreground events (Guitar here) and active speech. The bottom sample in particular exemplifies the problem: VAD-C starts to predict speech, and without speech, both GPVAD models are able to distinguish any background noise in speech. Note that the end of the bottom segment contains laughter, which VAD-C classifies as speech. In future work, we hope to further expand the scope of GPVAD training by leveraging larger training data (e.g., an unbalanced AudioSet).

Table 2: Best results obtained under each evaluation condition. Bold text indicates the best result for each dataset, underlined indicates the next best. Among them, AUC represents the area under the curve, and FER represents the frame error rate.

彩色效果最佳。图中，Speech表示语音。Figure 5: Per-frame probability output for three sample segments with visual speech occurrences (boxes, grey). (Top) contains clips from Aurora4 (B); (middle) contains a musician playing guitar (DCASE18); (bottom) contains someone talking about background noise (DCASE18). postprocessing threshold indicated

Color works best. In the figure, Speech represents speech.

6 Conclusion

Embodiments of the present invention introduce a robust VAD method for audio event detection using weak labels for training. Two GPVAD systems are studied: GPV-B, which trains only on binary speech and non-speech pairs, and GPV-F, which uses all 527AudioSet labels. Our test set thoroughly compares our proposed GPVAD method with traditional VAD using five different metrics. The results show that even if GPV-B is trained with paragraph-level labels, it can be used to detect speech without using clean frame-labeled training data. Furthermore, although GPV-B/F falls short for both clean noise and synthetic noise scenes for VAD-C, they excel in stable prediction for real scenes. Specifically, it can be seen that our proposed method performs very reliably on both synthetic noise and real-world noise datasets. Our best performing model, GPV-F, outperforms traditional supervised VAD methods by a large margin on real-world scenarios, with final absolute performance gains of 5.57% F1-macro, 6.45% F1micro, 3.93% AUC, 6.45% FER and 10.4% Event-F1.

Please refer to FIG. 6 , which shows a block diagram of an apparatus for training and using a voice endpoint detection model according to an embodiment of the present invention.

As shown in FIG. 6 , the voice endpoint detection model training apparatus 600 includes an input module 610 , a detection module 620 , an output module 630 , a loss calculation module 640 and an optimization module 650 .

Wherein, the input module 610 is configured to input the training audio into the speech endpoint detection model in a broad sense; the detection module 620 is configured to detect multiple audio events existing in the training audio via the speech endpoint detection model in the broad sense , wherein the multiple audio events include a human speaking event, a mute event, and at least one noise event; the output module 630 is configured to acquire the voice and the multiple audio events output by the voice endpoint detection model in the broad sense. non-speech discrimination result; the loss calculation module 640 is configured to calculate a loss function based on the audio event annotation of the training audio and the output of the speech endpoint detection model in the broad sense, wherein the audio event annotation includes pre- training audio for audio event labeling; and an optimization module 650 configured to optimize the generalized speech endpoint detection model by controlling the loss function.

Please refer to FIG. 7 , which shows an apparatus for using a voice endpoint detection model provided by an embodiment of the present invention.

As shown in FIG. 7 , the apparatus 700 for using a voice endpoint detection model includes a model processing module 710 and a distinguishing module 720 .

Wherein, the model processing module 710 is configured to detect multiple audio events existing in the input audio through the speech endpoint detection model in a broad sense trained according to the above method, wherein the multiple audio events include human speaking events, mute events and at least one noise event; and a distinguishing module 720, configured to obtain a distinguishing result of speech and non-speech of the plurality of audio events output by the speech endpoint detection model in the broad sense.

It should be understood that the modules recited in FIGS. 6 and 7 correspond to the various steps in the method described with reference to FIGS. 1 and 2 . Therefore, the operations and features described above with respect to the method and the corresponding technical effects are also applicable to the modules in FIG. 6 and FIG. 7 , and will not be repeated here.

It should be noted that the modules in the embodiments of the present application are not used to limit the solution of the present application, for example, the receiving module may be described as a module that receives a voice recognition request. In addition, the relevant functional modules may also be implemented by a hardware processor, for example, the receiving module may also be implemented by a processor, which will not be repeated here.

In other embodiments, embodiments of the present invention further provide a non-volatile computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions can execute the voice endpoint in any of the foregoing method embodiments Detect model training and usage methods;

As an embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are set to:

Input the training audio into the speech endpoint detection model in a broad sense;

Detecting, via the generalized speech endpoint detection model, a plurality of audio events present in the training audio, wherein the plurality of audio events include a human speaking event, a silence event, and at least one noise event;

Obtaining the discrimination results of speech and non-speech of the multiple audio events output by the speech endpoint detection model in the broad sense;

A loss function is calculated based on the audio event labeling of the training audio and the output of the speech endpoint detection model in a broad sense, wherein the audio event labeling includes pre-labeling the training audio for audio events;

The generalized speech endpoint detection model is optimized by controlling the loss function.

As another implementation manner, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are set to:

Detecting a plurality of audio events present in the input audio via a generalized speech endpoint detection model trained by the method according to the first aspect, wherein the plurality of audio events include a human speaking event, a silence event, and at least one noise event;

Obtaining a distinction result of speech and non-speech of the plurality of audio events output by the speech endpoint detection model in the broad sense.

The non-volatile computer-readable storage medium can include a stored program area and a stored data area, wherein the stored program area can store an operating system and an application program required by at least one function; Use the data created by the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-transitory computer-readable storage medium may optionally include memory located remotely from the processor that can be connected to the voice endpoint detection model training and use apparatus through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

An embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, the computer is made to execute the above Any speech endpoint detection model training and usage method.

FIG. 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As shown in FIG. 8 , the device includes: one or more processors 810 and a memory 820 . In FIG. 8 , one processor 810 is used as an example. The apparatus for training and using the voice endpoint detection model may further include: an input device 830 and an output device 840 . The processor 810, the memory 820, the input device 830, and the output device 840 may be connected through a bus or in other ways, and the connection through a bus is taken as an example in FIG. 8 . The memory 820 is the aforementioned non-volatile computer-readable storage medium. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 820, that is, to implement the voice endpoint detection model training and use methods of the above method embodiments. The input device 830 may receive input numerical or character information and generate key signal input related to voice endpoint detection model training and user settings and function control of the use device. The output device 840 may include a display device such as a display screen.

The above product can execute the method provided by the embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference may be made to the method provided by the embodiment of the present invention.

As an embodiment, the above-mentioned electronic equipment is applied to a voice endpoint detection model training device, including:

at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

Input the training audio into the speech endpoint detection model in a broad sense;

Detecting, via the generalized speech endpoint detection model, a plurality of audio events present in the training audio, wherein the plurality of audio events include a human speaking event, a silence event, and at least one noise event;

Obtaining the discrimination results of speech and non-speech of the multiple audio events output by the speech endpoint detection model in the broad sense;

A loss function is calculated based on the audio event labeling of the training audio and the output of the speech endpoint detection model in a broad sense, wherein the audio event labeling includes pre-labeling the training audio for audio events;

The generalized speech endpoint detection model is optimized by controlling the loss function.

As an embodiment, the above-mentioned electronic equipment is applied to a device for using a voice endpoint detection model, including:

at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

Detecting a plurality of audio events present in the input audio via a generalized speech endpoint detection model trained by the method according to the first aspect, wherein the plurality of audio events include a human speaking event, a silence event, and at least one noise event;

Obtaining a distinction result of speech and non-speech of the plurality of audio events output by the speech endpoint detection model in the broad sense.

The electronic devices in the embodiments of the present application exist in various forms, including but not limited to:

(1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions, and its main goal is to provide voice and data communication. Such terminals include: smart phones, multimedia phones, feature phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has the characteristics of mobile Internet access. Such terminals include: PDA, MID and UMPC devices.

(3) Portable entertainment equipment: This type of equipment can display and play multimedia content. Such devices include: audio and video players, handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

(4) Server: A device that provides computing services. The composition of the server includes a processor, a hard disk, a memory, a system bus, etc. The server is similar to a general computer architecture, but due to the need to provide highly reliable services, the processing power, stability , reliability, security, scalability, manageability and other aspects of high requirements.

(5) Other electronic devices with data interaction function.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place , or distributed to multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A speech endpoint detection model training method comprises the following steps:

inputting training audio into a generalized voice endpoint detection model;

detecting a plurality of audio events present in the training audio via the generalized speech endpoint detection model, wherein the plurality of audio events include a speak-by-person event, a silence event, and at least one noise event;

obtaining the voice and non-voice distinguishing results of the plurality of audio events output by the generalized voice endpoint detection model;

calculating a loss function based on the audio event label of the training audio and the output of the generalized voice endpoint detection model, wherein the audio event label comprises the label of an audio event of the training audio in advance;

optimizing the generalized speech endpoint detection model by controlling the loss function.

2. The method of claim 1, wherein the audio event annotation is a paragraph level annotation.

3. The method of claim 1, wherein prior to the inputting training audio into the generalized speech endpoint detection model, the method further comprises:

extracting acoustic features of the training audio species;

and training and classifying the acoustic features by using a convolution cyclic neural network model.

4. The method of claim 3, wherein the detecting, via the generalized speech endpoint detection model, a plurality of audio events present in the training audio comprises:

audio event detection in the generalized speech endpoint detection model is utilized to identify a plurality of audio events present in the training audio.

5. A method for using a voice endpoint detection model comprises the following steps:

detecting a plurality of audio events present in input audio via a generalized speech endpoint detection model trained according to the method of claims 1-4, wherein the plurality of audio events include a human speaking event, a silence event, and at least one noise event;

and acquiring a voice and non-voice distinguishing result of the plurality of audio events output by the generalized voice endpoint detection model.

6. The method of claim 5, wherein obtaining the speech and non-speech discrimination results for the plurality of audio events output by the generalized speech endpoint detection model comprises:

and obtaining a voice and non-voice distinguishing result based on the plurality of audio events and the dual-threshold post-processing method.

7. The method of claim 6, wherein the deriving a distinction between speech and non-speech based on the plurality of audio events plus a dual threshold post-processing method comprises:

identifying the human speaking event as a speech portion based on the identified plurality of audio events and the silence event and the at least one noise event as non-speech portions;

and determining the voice endpoint of the audio to be detected based on the distinguishing result of the voice part and the non-voice part.

8. A speech endpoint detection model training apparatus, comprising:

an input module configured to input training audio into a generalized speech endpoint detection model;

a detection module configured to detect a plurality of audio events present in the training audio via the generalized speech endpoint detection model, wherein the plurality of audio events include a human speaking event, a silence event, and at least one noise event;

an output module configured to obtain a result of distinguishing between speech and non-speech of the plurality of audio events output by the generalized speech endpoint detection model;

a loss calculation module configured to calculate a loss function based on an audio event label of the training audio and an output of the generalized speech endpoint detection model, wherein the audio event label includes a label of an audio event of the training audio in advance;

an optimization module configured to optimize the generalized speech endpoint detection model by controlling the loss function.

9. A speech endpoint detection model using apparatus, comprising:

a model processing module configured to detect a plurality of audio events present in input audio via a generalized speech endpoint detection model trained according to the method of claims 1-4, wherein the plurality of audio events include a human speaking event, a silence event, and at least one noise event;

and the distinguishing module is configured to acquire a distinguishing result of the voice and the non-voice of the plurality of audio events output by the generalized voice endpoint detection model.

10. An electronic device, comprising: at least one processor, and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method of any one of claims 1 to 7.

Images