CN111816216A - Voice activity detection method and device - Google Patents

Abstract

The invention discloses a voice activity detection method and a voice activity detection device, wherein the voice activity detection method comprises the following steps: carrying out voice activity detection of a frame level in a VAD system of an input frame level of audio to be detected to obtain first audio output by the VAD system of the frame level; and inputting the first audio into a sentence-level VAD system for sentence-level voice activity detection, acquiring a second audio output by the sentence-level VAD system, and performing subsequent processing on the second audio. By adding a sentence-level VAD system behind the existing frame-level VAD system, the audio which is judged to be voice by the previous system can be further judged in a whole sentence level, the misjudgment of the audio is reduced, the recall rate of non-voice sections is improved, and the resources of rear-end identification are further saved.

Description

Voice activity detection method and device

technical field

The invention belongs to the technical field of voice recognition, and in particular relates to a voice activity detection method and device.

Background technique

Voice activity detection (VAD), also known as speech activitydetectionorspeech detection, is a technique used in speech processing to detect the presence of speech signals. VAD technology is mainly used for speech coding and speech recognition. It simplifies voice processing and can also be used to remove non-voice segments during an audio session: encoding and transmission of silent packets can be avoided in IP telephony applications, saving computational time and bandwidth.

VAD technology enables a range of voice-based applications. Therefore, there is a family of VAD algorithms with different characteristics and delay times, sensitivity, accuracy and computational cost. Some VAD algorithms also provide further analysis, such as whether speech is voiced, unvoiced, or sustained. Voice activity detection is usually language-independent.

VAD technology was first used in time-assignment speech interpolation (TASI) systems.

Based on traditional acoustic features such as short-term energy, spectral energy, zero-crossing rate, etc., or based on features extracted by neural networks, voice activity detection is performed, and each frame of audio is judged whether it is voice or not. This method has good performance when the signal-to-noise ratio is high.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a voice activity detection method and device, which are used to solve at least one of the above technical problems.

In a first aspect, an embodiment of the present invention provides a voice activity detection method, comprising: inputting audio to be detected into a frame-level VAD system to perform frame-level voice activity detection, and obtaining the first audio output from the frame-level VAD system; Sentence-level voice activity detection is performed in the sentence-level VAD system for the first audio input, second audio output from the sentence-level VAD system is acquired, and subsequent processing is performed on the second audio.

In a second aspect, an embodiment of the present invention provides a voice activity detection device, including: a first input detection and output module configured to input audio to be detected into a frame-level VAD system to perform frame-level voice activity detection, and obtain the frame-level voice activity detection. The first audio output by the VAD system; the second input detection output module is configured to input the first audio into the sentence level VAD system to perform sentence level voice activity detection, and obtain the second audio output of the sentence level VAD system, and perform subsequent processing on the second audio.

In a third aspect, a computer program product is provided, the computer program product comprising a computer program stored on a non-volatile computer-readable storage medium, the computer program comprising program instructions that when executed by a computer , causing the computer to execute the steps of the voice activity detection method described in the first aspect.

In a fourth aspect, an embodiment of the present invention further provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores data that can be processed by the at least one processor processor-executed instructions, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the method of the first aspect.

By adding a sentence-level VAD system after the existing frame-level VAD system, the method provided by the embodiment of the present application can further perform sentence-level judgment on the audio that was judged as speech by the previous system, thereby reducing audio misjudgments , which improves the recall rate of non-speech segments and further saves resources for back-end recognition.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a flowchart of a voice activity detection method provided by an embodiment of the present invention;

2 is a flowchart of another voice activity detection method provided by an embodiment of the present invention;

3 is a framework diagram of a sentence-level voice activity detection system according to a specific embodiment of a solution for voice activity detection according to an embodiment of the present invention;

4 is a flow chart of classification for providing original audio according to a specific embodiment of a solution for voice activity detection according to an embodiment of the present invention;

FIG. 5 is a block diagram of a voice activity detection apparatus according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to FIG. 1 , which shows a flowchart of an embodiment of a voice activity detection method of the present invention.

As shown in Figure 1, in step 101, the audio to be detected is input into the frame-level VAD system for frame-level voice activity detection, and the first audio output from the frame-level VAD system is obtained;

In step 102, the first audio is input into the sentence-level VAD system for sentence-level voice activity detection, the second audio output from the sentence-level VAD system is acquired, and the second audio is subjected to subsequent processing.

In this embodiment, for step 101, the voice activity detection device inputs the audio to be detected into the frame-level VAD system to perform frame-level voice activity detection, and obtains the first audio output from the frame-level VAD system, wherein, in the frame-level VAD system VAD is based on the FSMN (Feed-forward Sequential Memory Network) model. The FSMN model uses the online FSMN model and the offline FSMN model. The online FSMN model only looks forward a few frames, not backward. , that is, only the historical information is used and the future information is not used. The offline FSMN model can not only look forward, but also look back a few frames. It uses the historical information and also uses the future information. After the audio is generated, the audio to be detected is sent to the frame-level VAD system for screening after feature processing, and the first audio that the frame-level VAD system determines to be speech is obtained.

For step 102, the voice activity detection device inputs the first audio into the sentence-level VAD system to perform sentence-level voice activity detection, obtains the second audio output from the sentence-level VAD system, and performs subsequent processing on the second audio, for example, frame The level VAD system determines that the first audio of the voice is extracted and processed first, and then passes through the offline FSMN layer and through the DNN (deep neural network) to give the final second audio.

In the solution of this embodiment, by adding a sentence-level VAD system after the existing frame-level VAD system, further sentence-level determination can be performed on the audio that the previous system determined to be speech, thereby reducing audio errors. The judgment improves the recall rate of non-speech segments, and further saves the resources of back-end recognition.

Please refer to FIG. 2 , which shows a flowchart of another voice activity detection method provided by an embodiment of the present invention. The flowchart is mainly for step 102 in the flowchart of FIG. 1 "input the first audio into the sentence level" Flowchart of the steps further defined in "Sentence-level Voice Activity Detection in a VAD System".

As shown in Figure 2, in step 201, the first audio frequency is divided into multiple audio segments, and the sentence level VAD system is utilized to perform voice activity detection on the multiple segments audio respectively;

In step 202, if it is detected that any one of the plurality of audios contains speech, the entire first audio is output.

In this embodiment, for step 201, the voice activity detection device divides the first audio that the frame-level VAD system determines as speech into multiple audio segments, and uses the sentence-level VAD system to perform voice activity detection on the multiple segments of audio, for example, a segment of audio is "*you* good** little *chi". Among them, * is the noise part, hello Xiaochi is the user's voice part, and the voice activity detection device divides it into multiple audio segments, for example, **you*, hao**, Xiaochi*, and then uses the sentence-level VAD system to respectively **You*, Hao**, and Xiaochi* perform voice activity detection, in which the audio can be divided into segments according to an exact time or length. If the length is particularly long, the audio can be divided into more segments, which is not limited in this application.

For step 202, if the voice activity detection device detects that any audio segment of the multiple audio segments contains speech, it outputs the entire segment of the first audio segment, for example, detects **you*, ok* that will be divided into audio segments *, Xiaochi* contains speech in any segment. For example, if the segment audio of "**you" is detected to contain speech, the detection of other segments will be stopped, and it will be directly determined that the entire audio segment contains speech. , and then output the entire audio.

In the solution of this embodiment, a whole piece of audio is divided into segments, and each segment is judged separately. As long as one segment is judged to be speech, it is considered that the piece of audio contains speech, and the rest of the audio can be omitted. segment judgment.

In some optional embodiments, the frame-level VAD system is used to judge whether each frame of audio in the audio to be detected is a speech frame, and output the first audio composed of the audio of the audio to be detected that is determined to be a speech frame; If it is determined that the audio to be detected does not contain a speech frame, no subsequent processing is performed.

In some optional embodiments, the sentence-level VAD system is used to determine whether the entire first audio sentence is speech, and if so, input the first audio into a speech-only recognition system for speech recognition; if not, No subsequent processing is performed.

In some optional embodiments, the sentence-level VAD system is a FSMN-based model, and the FSMN (Feed-forward Sequential Memory Network)-based model includes a feature extraction layer, multiple offline FSMNs layers and DNN layers.

In the method described in the above embodiment, the frame-level VAD system is also a FSMN-based model.

It should be noted that although steps 101, 102 and other numbers with a clear sequence are used in the above embodiments to limit the sequence of steps, in actual application scenarios, some steps can be executed in parallel, and some steps can be executed in parallel. The sequence is also not limited by the above numbers, which is not limited in this application, and will not be repeated here.

The following describes some problems encountered by the inventor in the process of implementing the present invention and a specific embodiment of the finalized solution, so that those skilled in the art can better understand the solution of the present application.

The inventors discovered the defects of these similar technologies in the process of realizing the present invention:

When the signal-to-noise ratio is low and the background noise is large, the system will easily determine non-speech segments as speech segments and send them to the back-end recognition system, resulting in a waste of resources.

The asr recognition system is generally based on the vad to control the input and output of the audio. Once the vad control is not good enough, the noise audio or other non-human voice audio will be sent to the recognition system, which will cause false trigger recognition, confusion or false triggering of the recognition results. Recognizing this action, especially in interactive scenarios, the whole system is constantly interrupting and waiting for customer input, which is very bad for the customer experience.

Each frame of audio is judged, the duration is too short, and the given result is too local, so it is impossible to accurately judge the entire sentence.

In the process of realizing the present invention, the inventor found why it is not easy to think of the reasons:

Usually, the method of increasing the threshold is adopted, which is simple and direct, or find counter-example data to train to reduce the probability of false triggering of the model.

The reason why I didn't think of this solution: I didn't take into account the separation of the solution from the existing vad system, and it has been improving on the existing frame-level system, so it is difficult to achieve substantial improvement.

The solutions of the embodiments of the present application solve the technical problems existing in the above-mentioned prior art through the following solutions:

By adding a sentence-level vad system after the existing frame-level vad system, the audio that was judged as speech by the previous system is further judged at the whole sentence-level, so as to reduce the flow of non-speech segment errors to the back-end recognition.

The technical innovation of the present invention:

Figure 3 is a block diagram of a sentence-level speech activity detection system.

Figure 4 is a classification flow diagram for providing raw audio.

For Figure 3, the input of the system is the audio that has been screened by the frame-level vad system (that is, the audio that the frame-level vad system determines as speech). The audio is extracted with features, passed through the offline fsmn layer, and then passed through the DNN to give the final result. .

For Figure 4, the flow chart shows the relationship between the sentence-level vad system and the frame-level vad system.

Alternative solutions discovered by the inventor in the process of realizing the present invention:

Using SVM and DNN as classifiers to classify audio, their advantages are that the system is easy to build and the classification speed is fast, but the disadvantage is that the model's expressive ability is not strong, and the performance on the test set is not good.

For the input of the system, the initial input is the entire audio, that is, an audio is only judged once whether it is a voice. The advantage of this method is that it is the best for the whole world. The real-time performance will bring serious lag. Therefore, it was later changed to segmented judgment, that is, the entire audio was divided into several segments, and each segment was judged separately, and as long as one segment was judged to be speech, the audio was considered to contain speech, and the remaining segments of the audio could be omitted. determination.

When doing frame-level vad, we are also based on the fsmn model, not only using the online fsmn model (that is, fsmn only looks forward a few frames, not backward, only the historical information is used, and the future information is not used) , and the offline fsmn model is used, (that is, fsmn can not only look forward, but also look back a few frames, using historical and future information), the offline fsmn model, although the delay is worse than the online one, but the performance Much better than online. Therefore, if we encounter a scenario where the delay is not considered, we can use the vad trained by the offline fsmn.

In the process of realizing the present invention, the inventor finds that a deeper effect is achieved:

The most direct effect of this solution is to reduce the misjudgment of audio and improve the recall rate of non-speech segments; further save the resources of back-end recognition; improve the robustness of the entire speech recognition system to noise; In the user experience, since there will be non-voice segments flowing to the recognition system before, the recognized results are often meaningless "um, ah" or NULL. In the voice interaction system, these misrecognitions are often interrupted by mistake. It will significantly reduce the user experience.

Please refer to FIG. 5 , which shows a block diagram of a voice activity detection apparatus provided by an embodiment of the present invention.

As shown in FIG. 5 , a first input detection output module 510 and a second input detection output module 520 are provided.

The first input detection and output module 510 is configured to input the audio to be detected in the frame-level VAD system to perform frame-level voice activity detection, and obtain the first audio output from the frame-level VAD system; the second input detection and output module 520 is configured to input the first audio into the sentence-level VAD system for sentence-level voice activity detection, obtain the second audio output by the sentence-level VAD system, and perform subsequent processing on the second audio.

It should be understood that the modules recited in FIG. 5 correspond to various steps in the method described with reference to FIGS. 1 and 2 . Therefore, the operations and features described above with respect to the method and the corresponding technical effects are also applicable to the modules in FIG. 5 , which will not be repeated here.

It is worth noting that the modules in the embodiments of the present application are not used to limit the solution of the present application. For example, the first input detection and output module can be described as inputting the audio to be detected into the frame-level VAD system to perform frame-level voice activity detection, The module for obtaining the first audio output from the frame-level VAD system, in addition, the relevant functional modules may also be implemented by a hardware processor, for example, the first input detection and output module may be implemented by a processor, which will not be repeated here.

In other embodiments, embodiments of the present invention further provide a non-volatile computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions can execute the voice activity in any of the foregoing method embodiments Detection method;

As an embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are set to:

The voice activity detection of frame level is carried out in the audio input frame level VAD system to be detected, and the first audio output of the frame level VAD system is obtained;

The first audio is input into the sentence-level VAD system for sentence-level voice activity detection, the second audio output from the sentence-level VAD system is acquired, and subsequent processing is performed on the second audio.

The non-volatile computer-readable storage medium may include a storage program area and a storage data area, wherein the storage program area may store an operating system and an application program required by at least one function; the storage data area may store the usage according to the voice activity detection device created data, etc. In addition, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-transitory computer-readable storage medium may optionally include memory located remotely from the processor, the remote memory being connectable to the voice activity detection device through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

An embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, the computer is made to execute the above Any voice activity detection method.

FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As shown in FIG. 6 , the device includes: one or more processors 610 and a memory 620 . One processor 610 is taken as an example in FIG. 6 . The apparatus for the voice activity detection method may further include: an input device 630 and an output device 640 . The processor 610, the memory 620, the input device 630, and the output device 640 may be connected by a bus or in other manners, and the connection by a bus is taken as an example in FIG. 6 . The memory 620 is the aforementioned non-volatile computer-readable storage medium. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, implementing the above method embodiments for the voice activity detection apparatus method. The input device 630 may receive input numerical or character information and generate key signal input related to user settings and function control for the voice activity detection device. The output device 640 may include a display device such as a display screen.

The above product can execute the method provided by the embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference may be made to the method provided by the embodiment of the present invention.

As an embodiment, the above-mentioned electronic equipment is applied to a voice activity detection device, including:

at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

The voice activity detection of frame level is carried out in the audio input frame level VAD system to be detected, and the first audio output of the frame level VAD system is obtained;

The first audio is input into the sentence-level VAD system for sentence-level voice activity detection, the second audio output from the sentence-level VAD system is acquired, and subsequent processing is performed on the second audio.

The electronic devices in the embodiments of the present application exist in various forms, including but not limited to:

(1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions, and its main goal is to provide voice and data communication. Such terminals include: smart phones, multimedia phones, feature phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has the characteristics of mobile Internet access. Such terminals include: PDA, MID and UMPC devices.

(3) Portable entertainment equipment: This type of equipment can display and play multimedia content. Such devices include: audio and video players, handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

(4) Server: A device that provides computing services. The composition of the server includes a processor, a hard disk, a memory, a system bus, etc. The server is similar to a general computer architecture, but due to the need to provide highly reliable services, the processing power, stability , reliability, security, scalability, manageability and other aspects of high requirements.

(5) Other electronic devices with data interaction function.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place , or distributed to multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A voice activity detection method, comprising: The voice activity detection of frame level is carried out in the audio input frame level VAD system to be detected, and the first audio output of the frame level VAD system is obtained; The first audio is input into the sentence-level VAD system for sentence-level voice activity detection, the second audio output from the sentence-level VAD system is acquired, and subsequent processing is performed on the second audio. 2. The method according to claim 1, wherein said inputting the first audio into a sentence-level VAD system for sentence-level speech activity detection comprises: The first audio frequency is divided into multi-segment audio, and the sentence-level VAD system is utilized to carry out voice activity detection to the multi-segment audio respectively; If it is detected that any one of the plurality of audios contains speech, the entire first audio is output. 3. The method according to claim 1, wherein, the frame level VAD system is used to judge whether each frame of audio in the audio to be detected is a speech frame, and output the audio that is determined to be the audio frame of the audio to be detected in the audio to be detected. first audio; If it is determined that the audio to be detected does not contain a speech frame, no subsequent processing is performed. 4. method according to claim 1, wherein, described sentence level VAD system is used for judging whether described first audio frequency whole sentence is speech, if so, then described first audio frequency input only speech recognition system carries out speech recognition ; If not, no subsequent processing is performed. 5. The method of any one of claims 1-4, wherein the sentence-level VAD system is a FSMN-based model comprising a feature extraction layer, a plurality of offline FSMN layers, and a DNN layer . 6. The method of claim 5, wherein the frame level VAD system is also a FSMN based model. 7. A voice activity detection device, comprising: The first input detection output module is configured to input the audio to be detected in the frame-level VAD system to perform frame-level voice activity detection, and obtain the first audio output of the frame-level VAD system; The second input detection and output module is configured to input the first audio into the sentence-level VAD system to perform sentence-level voice activity detection, obtain the second audio output from the sentence-level VAD system, and perform sentence-level voice activity detection on the second audio. Subsequent processing. 8. A voice activity detection device, further comprising: A segmentation detection module, configured to segment the first audio into multiple segments of audio, and utilize the sentence-level VAD system to perform voice activity detection on the multiple segments of audio respectively; The output module is configured to output the entire first audio segment if it is detected that any segment of audio in the multiple segments of audio contains speech. 9. A computer program product comprising a computer program stored on a non-volatile computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause all The computer performs the steps of the method of any one of claims 1 to 6. 10. An electronic device comprising: at least one processor, and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions Executed by the at least one processor to enable the at least one processor to perform the steps of the method of any of claims 1 to 6.

Images