TWI878975B - Speech recognition device and method - Google Patents

Abstract

A speech recognition device including a spoof defense module and a speaker verification module is provided. The spoof defense module includes a first deep learning model. The first deep learning model determines whether a first input speech is real speech or a spoof speech. When the first input speech is the real speech, the spoof defense module outputs a second input speech. The speaker verification module includes a second deep learning model. The second deep learning model processes the second input speech to produce a voice-point. The speaker verification module determines whether the first input speech is provided by a legitimate speaker according to the voice-point. When the first input speech is provided by a legitimate speaker, the speaker verification module performs a specific action. When the first input speech is not provided by a legitimate speaker, the speaker verification module does not perform the specific action.

Description

Speech recognition device and method

The present invention relates to a voice recognition device, and in particular to a voice recognition device with attack and defense capabilities.

For smart access control systems that focus on security, in order to achieve a higher level of security protection, it may be necessary to pass through multiple systems (such as facial, fingerprint and voice recognition) to enter the room. However, such a system causes huge power consumption and the cost will increase a lot.

Furthermore, with the rapid development of technology, some malicious attack methods have also made greater progress. For example, voice attacks include voice synthesis, pre-recorded playback, and even false audio generation. If the attack methods are not updated frequently, it is like being exposed to risks at all times.

An embodiment of the present invention provides a speech recognition device, including an attack defense module and a speaker verification module. The attack defense module is used to filter out a false voice, and includes a processing circuit and a prediction circuit. The processing circuit processes a first input voice to generate an audio signal. The prediction circuit receives the audio signal and processes the audio signal using a first deep learning model to determine whether the first input voice is a true voice or a false voice. When the first input voice is a true voice, the prediction circuit outputs a second input voice. The speaker verification module includes a feature extraction circuit, a comparison circuit, and a decision circuit. The feature acquisition circuit uses a second deep learning model to process the second input speech to generate a speech feature. The comparison circuit compares the speech feature with at least one preset feature to generate a comparison result. The decision circuit determines whether the comparison result is greater than a threshold value. When the comparison result is greater than the threshold value, the decision circuit performs a specific action. When the comparison result is not greater than the threshold value, the decision circuit does not perform a specific action.

The present invention also provides a speech recognition method, including receiving and processing an input speech to generate an audio signal; using a first deep learning model to process the audio signal to determine whether the input speech is a true speech or a false speech; when the input speech is a true speech, using a second deep learning model to process the input speech to generate a speech feature; comparing the speech feature with at least one preset feature to generate a comparison result; determining whether the comparison result is greater than a threshold value; when the comparison result is greater than the threshold value, performing a specific action; and when the comparison result is not greater than the threshold value, not performing the specific action.

The speech recognition method of the present invention can be implemented by the speech recognition device of the present invention, which is hardware or firmware that can execute specific functions, or can be recorded in a recording medium in the form of program code and implemented in combination with specific hardware. When the program code is loaded and executed by an electronic device, processor, computer or machine, the electronic device, processor, computer or machine becomes an attack defense module and a speaker verification module for implementing the present invention.

100: Voice recognition device

110: Attack and defense module

120: Speaker verification module

130: Receiving circuit

SP_IN1~SP_IN3: Input voice

SP_RL: Real Voice

SP_SF: False voice

ST: trigger signal

210: Processing circuit

220: Circuit estimation

ADS_1~ADS_3: audio signal

230, 350: Input and output ports

DA_1, DA_2: Update data

221, 311: Deep learning model

310: Feature acquisition circuit

330: Comparison circuit

340: Decision circuit

SP_EM: Voice features

S_CM: Comparison results

SP_PR: Default features

S611~S619: Steps

Figure 1 is a schematic diagram of the speech recognition device of the present invention.

Figure 2 is a schematic diagram of the attack defense module of the present invention.

Figure 3A is a schematic diagram of the speaker verification module of the present invention.

Figure 3B is another schematic diagram of the speaker verification module of the present invention.

Figure 4 is a schematic diagram of the attack defense operation of the present invention.

Figure 5 is a schematic diagram of the speaker verification operation of the present invention.

Figure 6 is a schematic diagram of the process of the speech recognition method of the present invention.

In order to make the purpose, features and advantages of the present invention more clearly understandable, the following examples are specifically cited and detailed descriptions are made in conjunction with the attached drawings. This invention specification provides different examples to illustrate the technical features of different implementations of the present invention. Among them, the configuration of each component in the embodiment is for illustrative purposes and is not used to limit the present invention. In addition, the partial repetition of the figure numbers in the embodiment is for the purpose of simplifying the description and does not mean the correlation between different embodiments.

FIG. 1 is a schematic diagram of a voice recognition device of the present invention. As shown in the figure, the voice recognition device 100 includes an attack defense module 110 and a speaker verification module 120. In a possible embodiment, the voice recognition device 100 is a door access voice recognition system. In this example, the voice recognition device 100 first determines whether an input voice SP_IN1 is a real speech. If the input voice SP_IN1 is not a real voice, for example, the input voice SP_IN1 is a spoof speech, the voice recognition device 100 ignores the input voice SP_IN1. If the input voice SP_IN1 is a real voice, the voice recognition device 100 then determines whether the input voice SP_IN1 matches the registered default voice. If yes, it means that the person who issued the input voice SP_IN1 is a legitimate person. Therefore, the voice recognition device 100 performs a specific action according to the input voice SP_IN1, such as opening the door. If the input voice SP_IN1 does not match the registered default voice, it means that the person who issued the input voice SP_IN1 is an illegal person. Therefore, the voice recognition device 100 does not perform a specific action. In some embodiments, although the voice recognition device 100 does not perform a specific action, the voice recognition device 100 performs a warning action, such as issuing a warning sound, to notify relevant personnel that someone is trying to enter the room.

In a possible embodiment, the voice recognition device 100 further includes a receiving circuit 130. The receiving circuit 130 receives an external voice. The external voice may be a true voice SP_RL or a false voice SP_SF. Regardless of whether the external voice is a true voice SP_RL or a false voice SP_SF, the receiving circuit 130 uses the external voice as an input voice SP_IN1 and provides the input voice SP_IN1 to the attack defense module 110. In some embodiments, the receiving circuit 130 has a microphone. The present invention does not limit the structure of the receiving circuit 130. In a possible embodiment, the receiving circuit 130 is a recorder.

The attack defense module 110 receives the input voice SP_IN1 and determines the type of the input voice SP_IN1. When the input voice SP_IN1 is a true voice, the attack defense module 110 provides an input voice SP_IN2 according to the input voice SP_IN1. In some embodiments, the attack defense module 110 directly uses the input voice SP_IN1 as the input voice SP_IN2. When the input voice SP_IN1 is a false voice, the attack defense module 110 stops providing the input voice SP_IN2.

In a possible embodiment, the real voice is the voice uttered by a living body (such as a user), and the false voice is the voice generated by a voice generator. The voice generator may generate a false voice using a voice synthesis method, a pre-recorded playback method, or a virtual audio generation method. A person with intentions may approach the receiving circuit 130 and then play the false voice. In this example, although the false voice is also uttered by a living body, it is not the voice spoken directly by the living body to the receiving circuit 130. Therefore, as long as the voice is not spoken by the living body to the receiving circuit 130, it is called a false voice. In other embodiments, the voice generator may capture part of the voice of the living body, rearrange it, or even use the upper and lower limits of human hearing to simulate an attacking voice (i.e., a false voice). Therefore, any voice that is played is called false voice.

In this embodiment, the attack defense module 110 has a first deep learning model. The parameters of the first deep learning model are provided by an external device. The external device performs training of real voice and false voice, and then writes the trained parameters into the first deep learning model of the attack defense module 110. Therefore, the attack defense module 110 has the ability to distinguish whether the input voice SP_IN1 is a false voice or a real voice. In this example, the external device may update the parameters of the first deep learning model regularly or from time to time, so that the voice recognition device 100 is more robust and more effective in resisting attacks from new false voices. When a new attack appears, the external device is trained based on the attack data, and then the parameters of the first deep learning model are updated using the trained parameters. The first deep learning model can then distinguish the new attack voice.

In some embodiments, the attack defense module 110 performs machine learning to distinguish advanced speech synthesis. The attack defense module 110 may train the attack speech generated by the false audio generator and the real human voice, and then write the parameters generated by the training into the first deep learning model, so that the real voice and the false voice can be distinguished, and malicious attack voices can be blocked.

The speaker verification module 120 receives the input voice SP_IN2 and determines whether the input voice SP_IN2 is the voice of a registered user. If the input voice SP_IN2 is the voice of a registered user, the speaker verification module 120 performs a specific action. In a possible embodiment, the speaker verification module 120 enables a trigger signal ST to unlock the door lock. If the input voice SP_IN2 is not the voice of a registered user, the speaker verification module 120 does not perform a specific action. For example, the speaker verification module 120 may not be able to trigger the signal ST. Therefore, the door lock continues to be locked. In other embodiments, the speaker verification module 120 may emit a warning sound to notify a specific person that someone is trying to enter the house.

In some embodiments, during a registration period, the speaker verification module 120 receives input voice SP_IN3. The input voice SP_IN3 is the voice of a legitimate person. The speaker verification module 120 captures the voice-point of the input voice SP_IN3 and uses the voice-point of the input voice SP_IN3 as a default feature. In this example, the speaker verification module 120 compares the voice-points of the input voices SP_IN2 and SP_IN3. When the voice-point of the input voice SP_IN2 is similar to the default feature, it indicates that the input voice SP_IN1 is uttered by a legitimate person. Therefore, the speaker verification module 120 performs a specific action, such as unlocking a door.

In a possible embodiment, the speaker verification module 120 has a second deep learning model. The parameters of the second deep learning model are provided by an external device. The external device uses multiple speech as training data to learn how to extract the speaker's voice features from the speech. The external device writes the trained parameters into the second deep learning model of the speaker verification module 120. Therefore, the second deep learning model has the ability to extract voice features. During a registration period, the second deep learning model extracts the voice features of at least one legitimate person and records the extracted voice features in a database. In some embodiments, when the second deep learning model captures the voice features of multiple legitimate persons, the database records the multiple voice features. Each voice feature corresponds to the voice of a legitimate person.

FIG. 2 is a schematic diagram of the attack defense module 110 of the present invention. The attack defense module 110 includes a processing circuit 210 and a prediction circuit 220. The processing circuit 210 processes the input speech SP_IN1 to generate an audio signal ADS_1. The present invention does not limit the structure of the processing circuit 210. In a possible embodiment, the processing circuit 210 has the ability to extract features. For example, the processing circuit 210 may extract the constant Q cepstral coefficient (CQCC) or the Mel cepstral coefficient (MFCC) of the input speech SP_IN1 to generate an extraction result. In this example, the processing circuit 210 uses the extraction result as the audio signal ADS_1.

The inference circuit 220 receives the audio signal ADS_1 and processes the audio signal ADS_1 using a deep learning model 221 to determine whether the input voice SP_IN1 is a true voice or a false voice. When the input voice SP_IN1 is a true voice, the inference circuit 220 outputs an input voice SP_IN2. In some embodiments, the inference circuit 220 uses the input voice SP_IN1 or the audio signal ADS_1 as the input voice SP_IN2. When the input voice SP_IN1 is a false voice, the inference circuit 220 does not provide the input voice SP_IN2. In a possible embodiment, the inference circuit 220 has a memory device for storing the deep learning model 221.

In some embodiments, the attack defense module 110 further includes an input/output port 230. The input/output port 230 is used to receive an update data DA_1. The update data DA_1 is provided by an external device. The external device trains a training data. The training data includes at least one real voice and at least one fake voice. After training, the external device generates at least one parameter and uses the parameter as the update data DA_1. During an update period, when the external device is coupled to the input/output port 230, the inference circuit 220 receives the update data DA_1 through the input/output port 230. The inference circuit 220 writes the update data DA_1 into the deep learning model 221 to update at least one parameter of the deep learning model 221.

The present invention does not limit the type of the input/output port 230. In one possible embodiment, the input/output port 230 is connected to an external device via a connection line. In another possible embodiment, the input/output port 230 is a wireless receiver that receives update data DA_1 from an external device.

FIG. 3A is a schematic diagram of the speaker verification module 120 of the present invention. As shown in the figure, the speaker verification module 120 includes a feature acquisition circuit 310, a comparison circuit 330, and a decision circuit 340. In this embodiment, the speaker verification module 120 is used to determine whether the input voice SP_IN2 from the attack defense module 110 is a legitimate voice. When the input voice SP_IN1 is a legitimate voice, the speaker verification module 120 performs a specific action, such as starting. When the input voice SP_IN1 is an illegal voice, the speaker verification module 120 does not perform a specific action, or performs a warning action, such as issuing a warning sound to notify the people in the room.

The feature acquisition circuit 310 processes the input speech SP_IN2 using a deep learning model 311 to generate a speech feature SP_EM. In one possible embodiment, the input speech SP_IN2 is the input speech SP_IN1. In this example, when the attack defense module 110 determines that the input speech SP_IN1 is a true speech, the attack defense module 110 directly uses the audio signal ADS_1 as the input speech SP_IN2. In this embodiment, the deep learning model 311 extracts the speaker's features (speaker embedding). In one possible embodiment, the feature acquisition circuit 310 has a memory device for storing the deep learning model 311.

The comparison circuit 330 compares the speech feature SP_EM with at least one preset feature SP_PR to generate a comparison result S_CM. In some embodiments, the comparison result S_CM indicates the similarity between the speech feature SP_EM and the preset feature SP_PR.

The decision circuit 340 determines whether the comparison result S_CM is greater than a threshold value. When the comparison result S_CM is greater than the threshold value, the decision circuit 340 performs a specific action, such as enabling the trigger signal ST. When the comparison result S_CM is not greater than the threshold value, the decision circuit 340 does not perform a specific action, such as not enabling the trigger signal ST.

In some embodiments, the speaker verification module 120 further includes a storage circuit 320. The storage circuit 320 is used to store the default feature SP_PR. In a possible embodiment, the default feature SP_PR is provided by the feature capture circuit 310. The present invention does not limit the number of the default features SP_PR. The storage circuit 320 may store more default features SP_PR. In this example, each default feature SP_PR represents a voice feature of a legitimate person (such as a family member). The present invention does not limit the type of the storage circuit 320. In a possible embodiment, the storage circuit 320 is a non-volatile memory.

In one possible embodiment, the parameters of the deep learning model 311 are obtained from an external device (not shown). The external device uses multiple speech sounds as training data to learn to extract speech features. The external device writes the trained parameters into the deep learning model 311. Therefore, the deep learning model 311 has the ability to extract speech features.

In other embodiments, the speaker verification module 120 further includes an input/output port 350. The input/output port 350 is used to receive an update data DA_2. When an external device is coupled to the input/output port 350, the speaker verification module 120 enters an update mode. In the update mode, the feature acquisition circuit 310 updates at least one parameter of the deep learning model 311 according to the update data DA_2. The present invention does not limit the type of the input/output port 350. In one possible embodiment, the input/output port 350 is connected to the external device through a connection line. In another possible embodiment, the input/output port 350 is a wireless receiver that receives the update data DA_2 from the external device.

FIG. 3B is another schematic diagram of the speaker verification module 120 of the present invention. FIG. 3B is similar to FIG. 3A, but the difference is that FIG. 3B has an additional processing circuit 360. The processing circuit 360 processes the input speech SP_IN2 to generate an audio signal ADS_2. Since the characteristics of the processing circuit 360 are similar to those of the processing circuit 210 in FIG. 2, it will not be described in detail.

The feature acquisition circuit 310 receives the audio signal ADS_2 and processes the audio signal ADS_2 using the deep learning model 311 to generate the speech feature SP_EM. In this embodiment, when the attack defense module 110 knows that the input speech SP_IN1 is a true speech, the attack defense module 110 uses the input speech SP_IN1 as the input speech SP_IN2.

In some embodiments, during an initial period (or registration period), the processing circuit 360 processes the input voice SP_IN3 to generate an audio signal ADS_3. In this example, the input voice SP_IN3 is the voice of a legitimate person. The deep learning model 311 captures the voice features of the audio signal ADS_3 and records the capture results (i.e., the default features SP_PR) in the storage circuit 320.

In a possible embodiment, a legitimate person speaks to the receiving circuit 130 of FIG. 1. The receiving circuit 130 provides the voice of the legitimate person (i.e., the input voice SP_IN3) to the processing circuit 360. In some embodiments, the receiving circuit 130 may receive the voices of five legitimate persons. Therefore, the storage circuit 320 records five preset features SP_PR, corresponding to the five legitimate persons respectively. In this example, the comparison circuit 330 compares the voice feature SP_EM with the five preset features SP_PR, and provides five comparison results to the decision circuit 340. The decision circuit 340 compares each comparison result with a valve value.

Figure 4 is a schematic diagram of the attack defense operation of the present invention. In a training phase, many different training data are provided. The training data are real voice and fake voice. Real voice is the sound made by a living body. Fake voice includes voice obtained using various methods, such as generating voice similar to the voice of a living body. In addition, there are recording and playback, capturing part of the voice, and using surround sound and stereo. Then, extract the features of the training data and perform voice training (such as machine learning). Through multiple voice trainings, at least one parameter can be generated. The parameter is then written into a first deep learning model.

In an inference phase, an inference data is received. In a possible embodiment, the inference data is an unknown voice. The unknown voice may come from a recorder. Then, the features of the inference data are extracted to generate a first extraction result. The first extraction result is processed using a first deep learning model to determine whether the inference data is a true voice or a false voice. In a possible embodiment, if the inference data is a true voice, it is verified whether the true voice is uttered by a legitimate person. If the inference data is a false voice, the false voice is not verified.

Figure 5 is a schematic diagram of the speaker verification operation of the present invention. In a training phase, a number of different training data are provided. In a possible embodiment, each training data is a real voice. Then, the features of the training data are extracted and voice training is performed. Through multiple voice trainings, at least one parameter can be generated. The parameter is then written into a second deep learning model.

In an enrollment phase, at least one enrollment data is received. Then, features of the enrollment data are extracted to generate a second extraction result. The second extraction result is processed using a second deep learning model to obtain voice features (or default features) of the enrollment data. In a possible embodiment, the voice features of the enrollment data are stored in a database.

In a verification phase, a verification data is received. In a possible embodiment, the verification data is an unknown true voice, such as the true voice determined by the first deep learning model in FIG. 4. Then, the features of the verification data are extracted to generate a third extraction result. The third extraction result is processed by the second deep learning model to generate an unknown voice feature. The unknown voice feature is compared with the preset features in the database to determine the similarity between the unknown voice feature and each preset feature. Then, it is determined whether the similarity between the unknown voice feature and each preset feature exceeds a threshold value. If so, it means that the verification data is a sound made by a legitimate person, so the verification data is accepted and a specific action is performed. If not, it means that the verification data is not the voice of a legitimate person, so the verification data is rejected and the specific action is not performed.

Figure 6 is a flow chart of the speech recognition method of the present invention. The speech recognition method of the present invention can exist through program code. When the program code is loaded and executed by a machine, the machine becomes a speech recognition device for implementing the present invention.

First, an input speech is received and processed to generate an audio signal (step S611). In a possible embodiment, step S611 extracts the constant Q inverse spectrum coefficient or the Mel inverse spectrum coefficient of the input speech, and then uses the extraction result as the audio signal.

Next, a first deep learning model is used to process the audio signal (step S612). In a possible embodiment, the parameters of the first deep learning model are provided by an external device. The external device performs training of real speech and fake speech, and then writes the trained parameters into the first deep learning model. Therefore, according to the output result of the first deep learning model, the ability of the input speech to be a fake speech or a real speech can be known.

Determine whether the first deep learning model classifies the input speech as real speech (step S613). When the input speech is not real speech, reject the false speech (step S614). When the input speech is real speech, use a second deep learning model to process the input speech to generate a speech feature (step S615).

Next, the voice feature is compared with at least one preset feature to generate a comparison result (step S616). In a possible embodiment, the comparison result is the similarity between the voice feature and the preset feature. Then, it is determined whether the comparison result is greater than a threshold value (step S617). When the comparison result is not greater than the threshold value, it indicates that the input voice is not the voice of a legitimate person. Therefore, no specific action is performed (step S618). When the comparison result is greater than the threshold value, it indicates that the input voice is the voice of a legitimate person. Therefore, a specific action is performed (step S619).

The present invention does not limit the type of specific action. In one possible embodiment, the specific action is to unlock the door. In another possible embodiment, when the comparison result is not greater than the valve value, it means that an illegal person attempts to enter the house, so step S618 emits a warning sound to remind the person in the house. In other embodiments, by adjusting the valve value, the strictness of voice recognition can be improved.

The speech recognition method of the present invention, or a specific form or part thereof, can exist in the form of program code. The program code can be stored in a physical medium, such as a floppy disk, an optical disk, a hard disk, or any other machine-readable (such as computer-readable) storage medium, or a computer program product that is not limited to an external form, wherein when the program code is loaded and executed by a machine, such as a computer, the machine becomes used to participate in the attack defense module and speaker verification module of the present invention. The program code can also be transmitted through some transmission media, such as wires or cables, optical fibers, or any transmission type, wherein when the program code is received, loaded and executed by a machine, such as a computer, the machine becomes used to participate in the attack defense module and speaker verification module of the present invention. When implemented on a general-purpose processing unit, the code combines with the processing unit to provide a unique device that operates like an application-specific logic circuit.

Unless otherwise defined, all terms (including technical and scientific terms) herein are generally understood by those with ordinary knowledge in the technical field to which the present invention belongs. In addition, unless expressly stated, the definition of a term in a general dictionary should be interpreted as consistent with the meaning in the article in the relevant technical field, and should not be interpreted as an ideal state or overly formal tone. Although terms such as "first" and "second" can be used to describe various components, these components should not be limited by these terms. These terms are only used to distinguish one component from another.

Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. Any person with ordinary knowledge in the relevant technical field may make some changes and modifications without departing from the spirit and scope of the present invention. For example, the system, device or method described in the embodiments of the present invention can be implemented in a physical embodiment of hardware, software or a combination of hardware and software. Therefore, the scope of protection of the present invention shall be determined by the scope of the attached patent application.

100: Voice recognition device 110: Attack defense module 120: Speaker verification module SP_IN1~SP_IN3: Input voice SP_RL: Real voice SP_SF: False voice 130: Receiving circuit ST: Trigger signal

Claims (6)

A speech recognition device includes: an attack defense module for filtering a false voice, and includes: a first processing circuit for processing a first input voice to generate a first audio signal; and a guessing circuit for receiving the first audio signal and processing the first audio signal using a first deep learning model to determine whether the first input voice is a true voice or the false voice. When the input voice is the real voice, the inference circuit outputs a second input voice; and a speaker verification module, including: a feature extraction circuit, using a second deep learning model to process the second input voice to generate a voice feature; a comparison circuit, comparing the voice feature with at least one preset feature to generate a comparison result; a second processing circuit, processing the second input voice, using to generate a second audio signal; and a decision circuit to determine whether the comparison result is greater than a threshold value. When the comparison result is greater than the threshold value, the decision circuit performs a specific action. When the comparison result is not greater than the threshold value, the decision circuit does not perform the specific action, wherein: when the first input voice is the true voice, the inference circuit uses the first input voice as the second input voice, The feature acquisition circuit receives the second audio signal and processes the second audio signal using the second deep learning model to generate the voice feature. During a registration period, the second processing circuit processes a third input voice to generate a third audio signal. The feature acquisition circuit receives the third audio signal and processes the third audio signal using the second deep learning model to generate the default feature. As in claim 1, the speech recognition device, wherein the speaker verification module further comprises: a storage circuit for storing the preset feature. As in claim 1, the speech recognition device, wherein the attack defense module further comprises: a first input/output port for receiving a first update data; wherein, during a first update period, the inference circuit updates at least one parameter of the first deep learning model according to the first update data. A speech recognition device as claimed in claim 1, wherein the speaker verification module further comprises: a second input/output port for receiving a second update data; wherein, during a second update period, the feature acquisition circuit updates at least one parameter of the second deep learning model according to the second update data. The voice recognition device of claim 1 further includes: a receiving circuit for receiving the first input voice and providing the first input voice to the attack defense module. A speech recognition method includes: receiving and processing a first input speech to generate a first audio signal; using a first deep learning model to process the first audio signal to determine whether the first input speech is a true speech or a false speech; when the first input speech is the true speech, using a second deep learning model to process the first input speech to generate a speech feature; comparing the speech feature with the first audio signal; A preset feature is provided to generate a comparison result; determine whether the comparison result is greater than a threshold value; when the comparison result is greater than the threshold value, perform a specific action; when the comparison result is not greater than the threshold value, do not perform the specific action; and during a registration period: process a second input voice to generate a second audio signal; use the second deep learning model to process the second audio signal to generate the preset feature.

Images