CN106157956A - The method and device of speech recognition - Google Patents

Abstract

The invention discloses a voice recognition method and device, wherein the method acquires the voice recognition information of the user's current voice, and acquires the auxiliary identification information of the voice recognition information based on the user's current state corresponding to the user's current voice; according to the voice recognition Information and auxiliary identification information determine the final recognition result of the user's current voice. The present invention solves the problem in the related art that the accuracy of voice recognition is not high due to the acquisition of the user's speech content only through the user's voice, thereby improving the accuracy of voice recognition.

Description

Speech recognition method and device

technical field

The present invention relates to the communication field, in particular, to a voice recognition method and device.

Background technique

With the development of computer and related software and hardware technology, speech recognition technology has been more and more applied in various fields, and its recognition rate is also constantly improving. Under specific conditions such as a quiet environment and standard pronunciation, the recognition rate of the current speech recognition input text system has reached more than 95%. Conventional speech recognition technology is relatively mature. For speech recognition of mobile terminals, the effect of speech recognition is limited because the speech quality is relatively poor compared with ordinary speech recognition scenarios. The poor voice quality here includes the following reasons, such as the background noise of the client, the voice collection device of the client, the noise of the communication device, the noise and interference of the communication line, and the accent or the use of a dialect, the speaker himself Slurred or unclear speech. All of these factors can contribute to poor speech recognition. Its recognition rate is affected by many factors, and no effective solution has been proposed for the problem of poor user experience caused by low speech recognition rate in related technologies. In the car or in the case of loud noise and non-standard pronunciation, the recognition rate will be greatly reduced, so that it cannot achieve the real practical purpose. Its correct recognition rate is low, affects precise control, and the effect is not ideal enough. If other methods can be used to assist the judgment to improve the accuracy of speech recognition, the practicability of speech recognition will be significantly improved.

The human language cognition process is a multi-channel perception process. In the process of daily communication between people, the content of other people's speech is perceived through the sound. In a noisy environment or when the other party's pronunciation is unclear, it is necessary to observe the changes in the mouth shape and expression of the other party in order to accurately understand what the other party is saying. The content of the speech. The current speech recognition system ignores the visual characteristics of language perception, and only uses a single auditory characteristic, so that the recognition rate of the existing speech recognition system is significantly reduced in a noisy environment or under the condition of many speakers, and the speech quality is reduced. The practicability and application range of identification are also limited.

No effective solution has been proposed for the problem in the related art that the accuracy of speech recognition is not high due to the acquisition of the user's speech content only through the user's voice.

Contents of the invention

The present invention provides a voice recognition method and device to at least solve the problem in the related art that the accuracy of voice recognition is not high due to the acquisition of the user's speech content only through the user's voice.

According to one aspect of the present invention, a voice recognition method is provided, including: acquiring voice recognition information of the user's current voice, and acquiring auxiliary identification information of the voice recognition information based on the user's current state corresponding to the user's current voice ; Determine the final recognition result of the user's current voice according to the voice recognition information and the auxiliary recognition information.

Further, determining the final recognition result of the user's current voice according to the voice recognition information and the auxiliary recognition information includes: acquiring one or more first candidate words corresponding to the user's current voice according to the voice recognition information; Obtain the vocabulary category corresponding to the user's current voice or one or more second candidate vocabulary according to the auxiliary identification information; determine the final vocabulary of the user's current voice according to the one or more first candidate vocabulary and the vocabulary type A recognition result; or, determining a final recognition result of the user's current voice according to the one or more first candidate words and the one or more second candidate words.

Further, determining the final recognition result of the user's current speech according to the one or more first candidate words and the vocabulary type includes: selecting a word that meets the vocabulary category from the one or more first candidate words The first specific vocabulary, using the first specific vocabulary as the final recognition result of the user's current voice.

Further, determining the final recognition result of the user's current voice according to the one or more first candidate words and the one or more second candidate words includes: selecting from the one or more second candidate words The second specific vocabulary with high similarity to the one or more first candidate vocabulary is used as the final recognition result of the user's current speech.

Further, obtaining the auxiliary recognition information of the speech recognition information based on the current state of the user corresponding to the current voice of the user includes: obtaining an image indicating the current state of the user; obtaining image feature information according to the image; The image feature information acquires a vocabulary category and/or one or more candidate vocabulary corresponding to the image feature information, and uses the vocabulary category and/or the one or more candidate vocabulary as the auxiliary identification information.

Further, obtaining the vocabulary category and/or one or more candidate vocabulary corresponding to the image feature information according to the image feature information includes: searching for a specific image with the highest similarity to the image feature information in a predetermined image library ; Obtain the vocabulary category or one or more candidate vocabulary corresponding to the specific image according to the preset correspondence between the image and the vocabulary category or one or more candidate vocabulary.

Further, the current user state includes at least one of the following: the user's lip movement state, the user's throat vibration state, the user's face movement state, and the user's gesture movement state.

Further, before acquiring the voice recognition information of the user's current voice, and acquiring the auxiliary recognition information of the voice recognition information based on the user's current state corresponding to the user's current voice includes: determining to determine the user's current voice based on the voice recognition information The accuracy rate of the final speech recognition result is less than a predetermined threshold.

According to another aspect of the present invention, there is provided a voice recognition device, the device comprising: an acquisition module, configured to acquire voice recognition information of the user's current voice, and acquire information based on the user's current state corresponding to the user's current voice Auxiliary identification information of the voice recognition information; a determination module configured to determine a final recognition result of the user's current voice according to the voice recognition information and the auxiliary identification information.

Further, the determining module includes: a first acquiring unit, configured to acquire one or more first candidate words corresponding to the user's current voice according to the speech recognition information; a second acquiring unit, configured to acquire The recognition information acquires the vocabulary category corresponding to the user's current voice or one or more second candidate vocabulary; a determination unit is configured to determine the user's current voice according to the one or more first candidate vocabulary and the vocabulary type A final recognition result; or, determining a final recognition result of the user's current voice according to the one or more first candidate words and the one or more second candidate words.

Further, the determining unit is further configured to select a first specific vocabulary conforming to the vocabulary category from the one or more first candidate vocabulary, and use the first specific vocabulary as the final recognition of the user's current voice result.

Further, the determining unit is further configured to select a second specific word with a high similarity with the one or more first candidate words from the one or more second candidate words, and select the second specific word As the final recognition result of the user's current voice.

Further, the acquiring module further includes: a third acquiring unit, configured to acquire an image indicating the current state of the user; a fourth acquiring unit, configured to acquire image feature information according to the image; a fifth acquiring unit, Obtaining the vocabulary category and/or one or more candidate vocabulary corresponding to the image feature information according to the image feature information, using the vocabulary category and/or the one or more candidate vocabulary as the auxiliary recognition information.

Further, the fifth acquisition unit further includes: a search subunit, configured to search for a specific image with the highest similarity to the image feature information in a predetermined image library; an acquisition subunit, configured to Vocabulary categories or correspondences of one or more candidate words, acquiring the vocabulary category or one or more candidate words corresponding to the specific image.

Further, the current user state includes at least one of the following: the user's lip movement state, the user's throat vibration state, the user's face movement state, and the user's gesture movement state.

Further, the device further includes: a judging module, configured to judge that the accuracy rate of the final recognition result of the user's current speech determined based on the speech recognition information is less than a predetermined threshold.

According to another aspect of the present invention, there is also provided a terminal, including a processor, the processor is used to acquire the voice recognition information of the user's current voice, and acquire the voice based on the user's current state corresponding to the user's current voice Auxiliary identification information of the identification information; determining a final recognition result of the user's current voice according to the voice recognition information and the auxiliary identification information.

Through the present invention, the voice recognition information of the user's current voice is acquired, and the auxiliary identification information of the voice recognition information is obtained based on the user's current state corresponding to the user's current voice; the final recognition result of the user's current voice is determined according to the voice recognition information and the auxiliary identification information . It solves the problem in the related art that the accuracy of speech recognition is not high due to the acquisition of the user's speech content only through the user's voice, thereby improving the accuracy of speech recognition.

Description of drawings

The accompanying drawings described here are used to provide a further understanding of the present invention and constitute a part of the application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention. In the attached picture:

Fig. 1 is the flowchart of the speech recognition method according to the embodiment of the present invention;

Fig. 2 is a structural block diagram of a speech recognition device according to an embodiment of the present invention;

Fig. 3 is a structural block diagram (1) of a speech recognition device according to an embodiment of the present invention;

Fig. 4 is the structural block diagram (2) of the speech recognition device according to the embodiment of the present invention;

Fig. 5 is a structural block diagram (3) of a speech recognition device according to an embodiment of the present invention;

Fig. 6 is a structural block diagram (4) of a speech recognition device according to an embodiment of the present invention;

7 is a flowchart of a voice recognition processing method according to an embodiment of the present invention;

FIG. 8 is a structural block diagram of a speech recognition processing device according to an embodiment of the present invention;

FIG. 9 is a flowchart of speech recognition processing according to an embodiment of the present invention.

detailed description

Hereinafter, the present invention will be described in detail with reference to the drawings and examples. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

A method for speech recognition is provided in the present embodiment, and Fig. 1 is a flowchart of a speech recognition method according to an embodiment of the present invention, as shown in Fig. 1, the process includes the following steps:

Step S102, acquiring voice recognition information of the user's current voice, and acquiring auxiliary identification information of the voice recognition information based on the user's current state corresponding to the user's current voice;

Step S104, determine the final recognition result of the user's current voice according to the voice recognition information and auxiliary recognition information.

Through the above steps, the voice recognition information of the user's current voice is obtained, and the state feature information of the user when the voice is uttered is obtained, and the state feature information of the user when the voice is uttered is used as auxiliary information for recognizing the current voice. Compared with the prior art The accuracy rate of voice recognition is low only through the user's current voice. The above steps solve the problem in the related art that the accuracy of voice recognition is not high due to the acquisition of the user's speech content only through the user's voice, thereby improving the accuracy of voice recognition. sex.

The above step S104 involves determining the final recognition result of the user's current voice according to the voice recognition information and auxiliary recognition information. In an optional embodiment, one or more first candidate words corresponding to the user's current voice are obtained according to the voice recognition information; Obtain the vocabulary category corresponding to the user's current voice or one or more second candidate vocabulary according to the auxiliary recognition information; determine the final recognition result of the user's current voice according to one or more first candidate vocabulary and the vocabulary type; or, according to one Or a plurality of first candidate words and one or more second candidate words determine the final recognition result of the user's current speech.

There are many ways to determine the final recognition result of the user's current voice according to one or more first candidate vocabulary and vocabulary types. In an optional embodiment, select the corresponding vocabulary category from one or more first candidate vocabulary The first specific vocabulary is used as the final recognition result of the user's current voice. In another optional embodiment, a second specific vocabulary with high similarity to one or more first candidate vocabulary is selected from one or more second candidate vocabulary, and the second specific vocabulary is used as the final recognition of the user's current voice result.

In the above process of determining the final recognition result of the user's current voice according to one or more first candidate words and one or more second candidate words, in an optional embodiment, first obtain the image, and then obtain image feature information according to the image, and then obtain the vocabulary category and/or one or more candidate vocabulary corresponding to the image feature information according to the image feature information, and the vocabulary category and/or the one or more The candidate words are used as the auxiliary identification information.

In an optional embodiment, the specific image with the highest similarity to the image feature information is searched in a predetermined image library, and according to the preset corresponding relationship between the image and the vocabulary category or one or more candidate vocabulary, the specific The word category corresponding to the image or one or more candidate words. Thus, the vocabulary category and/or one or more candidate vocabulary corresponding to the image feature information are obtained according to the image feature information.

The current status of the user may include various types, which will be illustrated below with an example. In an optional embodiment, the user's lip motion state, the user's throat vibration state, the user's face motion state, and the user's gesture motion state. The information included in the above-mentioned user's current state characteristics is only for illustration, and is not limited thereto. For example, in real life, what a speaker says can be recognized only by lip reading. Therefore, lip language is an important auxiliary factor for speech recognition.

In an optional embodiment, before obtaining the voice recognition information of the user's current voice and obtaining the auxiliary identification information of the voice recognition information based on the user's current state corresponding to the user's current voice, it is determined based on the voice recognition information that the user's current The accuracy rate of the final speech recognition result is less than a predetermined threshold.

This embodiment also provides a voice recognition device, which is used to implement the above embodiments and preferred implementation modes, and what has been described will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

Fig. 2 is a structural block diagram of a speech recognition device according to an embodiment of the present invention. As shown in Fig. 2 , the device includes: an acquisition module 22 for obtaining speech recognition information of the user's current speech, and based on the speech recognition information corresponding to the user's current speech The user's current state acquires the auxiliary recognition information of the speech recognition information; the determining module 24 is configured to determine the final recognition result of the user's current speech according to the speech recognition information and the auxiliary recognition information.

Fig. 3 is a structural block diagram (1) of a speech recognition device according to an embodiment of the present invention. As shown in Fig. 3 , the determination module 24 includes: a first obtaining unit 242, which is used to obtain the corresponding speech of the user's current speech according to the speech recognition information. One or more first candidate words; the second acquiring unit 244, configured to acquire the vocabulary category corresponding to the user's current voice or one or more second candidate words according to the auxiliary identification information; the determining unit 246, configured to Or a plurality of first candidate words and the vocabulary type determine the final recognition result of the user's current speech; or, determine the final recognition result of the user's current speech according to the one or more first candidate words and the one or more second candidate words recognition result.

Optionally, the determining unit 246 is further configured to select a first specific word matching the vocabulary category from the one or more first candidate words, and use the first specific word as a final recognition result of the user's current speech.

Optionally, the determining unit 246 is further configured to select a second specific word with a high similarity with the one or more first candidate words from the one or more second candidate words, and use the second specific word as the user's The final recognition result of the current speech.

FIG. 4 is a structural block diagram (2) of a speech recognition device according to an embodiment of the present invention. As shown in FIG. 4 , the acquiring module 22 further includes: a third acquiring unit 222, configured to acquire an image indicating the current state of the user; The fourth acquiring unit 224 is configured to acquire image feature information according to the image; the fifth acquiring unit 226 is configured to acquire the vocabulary category and/or one or more candidate vocabulary corresponding to the image feature information according to the image feature information, and The vocabulary category and/or the one or more candidate vocabulary serve as the auxiliary identification information.

Fig. 5 is the structural block diagram (3) of the speech recognition device according to the embodiment of the present invention, as shown in Fig. 5, the 5th acquisition unit 226 also comprises: search sub-unit 2262, is used for searching in predetermined image storehouse and this image The specific image with the highest similarity of feature information; the obtaining subunit 2264, configured to obtain the vocabulary category or one or more candidate vocabulary corresponding to the specific image according to the preset correspondence between the image and the vocabulary category or one or more candidate vocabulary vocabulary.

Optionally, the user's current state includes at least one of the following: the user's lip motion state, the user's throat vibration state, the user's face motion state, and the user's gesture motion state.

Fig. 6 is a structural block diagram (four) of a speech recognition device according to an embodiment of the present invention. As shown in Fig. 6, the device also includes: a determination module 26, which is used to determine the final recognition of the user's current voice based on the speech recognition information The accuracy of the result is less than a predetermined threshold.

According to another aspect of the present invention, there is also provided a terminal, including a processor, the processor is used to obtain the speech recognition information of the user's current speech, and obtain the speech recognition information based on the user's current state corresponding to the user's current speech Auxiliary identification information: determine the final recognition result of the user's current voice according to the voice recognition information and the auxiliary identification information.

It should be noted that each of the above-mentioned modules can be implemented by software or hardware. For the latter, it can be implemented in the following manner, but not limited to this: each of the above-mentioned modules is located in the same processor; or, each of the above-mentioned modules is located in the In the first processor, the second processor and the third processor . . .

Aiming at the above-mentioned problems existing in the related technologies, the following description will be made in conjunction with specific optional embodiments, and the above-mentioned optional embodiments and their optional implementation manners are combined in the following optional embodiments.

This optional embodiment provides a speech recognition processing method and device to solve the problem of poor user experience caused by low speech recognition rate in the related art. In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this optional embodiment is to provide an intelligent voice recognition method and device based on an auxiliary interactive mode. On the basis of voice recognition, lip shape recognition is used as a basic signal. , face recognition, gesture recognition, throat vibration recognition, etc., as auxiliary signals. Utilizing the advantages of each technology in its application field, learning from each other, each technology module is relatively independent and integrated with each other, greatly improving the recognition rate of speech processing. Preferably, the increase of auxiliary signal recognition can be determined by the speech recognition result. When the speech recognition result is less likely than Thresholds add auxiliary data. In line with the human language cognitive process is a multi-channel perception process. Let the terminal perceive the content of the speech based on the sound, and cooperate with the recognition of its mouth shape, facial changes, etc. to accurately understand the content of the speech.

According to one aspect of this optional embodiment, a voice recognition processing method is provided, on the basis of acquiring audio data through an audio sensor as a basic signal for voice recognition, and collecting moving images of a human body through a terminal device camera or an external sensor, Including gesture movement, facial movement, throat vibration, lip shape recognition, etc., which are analyzed by integrated image algorithms and motion processing chips, and used as auxiliary signals for voice recognition. The recognition results of basic signals and auxiliary signals are comprehensively processed by the terminal and corresponding operate. The auxiliary signal recognition result and the speech recognition basic signal result are accumulated and processed to form a unified recognition result, which plays an auxiliary role in speech recognition and improves the speech recognition rate.

Combining gesture movement, facial movement, throat vibration, and lip shape recognition, each method is organically combined through feature extraction, template training, template classification, and judgment processes. First, speech recognition is used as the basic signal for analysis and confirmation. Auxiliary signal for auxiliary judgment logic judgment sequence, effectively reducing the probability of recognition errors due to noise and external sound interference. In the process of auxiliary signal recognition, feature data is collected by sensors and cameras, feature data is extracted, and a series of matching judgments and recognition are performed with the preset template library data, and then compared with the corresponding recognition feature results to identify the The vocabulary of possible candidate words in the model lexicon.

Optionally, the above-mentioned lip shape recognition collects the lip shape image of the speaker through a camera, performs image processing on the lip shape image, extracts lip shape features dynamically in real time, and then uses a lip shape pattern recognition algorithm to determine the speech content. Using the combination of lip shape and lip color to judge the position of lips accurately. Appropriate lip-matching algorithms are used for recognition.

Optionally, the above-mentioned lip shape recognition extracts the features of the lip shape image from the preprocessed video data, and uses the features of the lip shape image to identify the current user's mouth shape change; detect the user's mouth movement to realize the lip shape recognition, and improve the recognition efficiency and accuracy. Classify the above-mentioned mouth motion feature maps to obtain classification information, and classify the above-mentioned mouth motion feature maps, and each feature type of mouth motion feature maps corresponds to several vocabulary categories. The information acquired by the above-mentioned lip shape recognition, after a series of processing such as denoising and analog-to-digital (A/D) conversion, is compared with the template library data preset in the image/speech recognition processing module, and the above-mentioned lip shape recognition information is compared The similarity with all pre-sampled mouth movement feature maps, read several vocabulary categories corresponding to the mouth movement feature map with the highest similarity.

Optionally, the above throat vibration recognition collects the speaker's throat vibration pattern through an external sensor, processes the vibration pattern, extracts the characteristics of the vibration pattern dynamically in real time, and then uses the vibration pattern pattern recognition algorithm to determine the speech content.

Optionally, before laryngeal vibration recognition is performed on the user, the user's laryngeal vibration motion feature map needs to be sampled first, and different throat vibration motion feature files are established for different users. When pre-sampling the user's throat vibration motion feature map, the user may sample the throat vibration motion feature map of a syllable, or sample the user's throat vibration motion feature map of a word. For voice events with different pronunciations, the laryngeal vibration movement is different. Since each voice event uttered by the user is related, after the recognition of the laryngeal vibration is completed, by using contextual error correction technology, the recognized laryngeal Vibration is verified to reduce the recognition error of the same type of laryngeal vibration motion feature map, and further improve the accuracy of laryngeal vibration recognition.

Optionally, the above laryngeal vibration recognition extracts the features of the laryngeal vibration image from the preprocessed vibration data, and uses the features of the laryngeal vibration image to identify the change of the laryngeal vibration of the current user; Vibration recognition improves recognition efficiency and accuracy. The above laryngeal vibration motion feature map is classified to obtain classification information, and the above laryngeal vibration motion feature map is classified, and the throat vibration motion feature of each feature type corresponds to several vocabulary categories. The acquired information of laryngeal vibration recognition is compared with the template library data preset in the image/speech recognition processing module, and the similarity between the above laryngeal vibration recognition information and all pre-sampled laryngeal vibration motion feature maps is compared, Several vocabulary categories corresponding to the laryngeal vibration motion feature map with the highest similarity are read.

The above-mentioned face recognition is used to extract the facial features of the user in the video data, and determine the identity and position of the user; when speaking, the facial muscles also correspond to different movement patterns. Identify the corresponding muscle action patterns in the computer, and then assist in the recognition of voice information.

According to an aspect of this optional embodiment, a speech recognition processing device is further provided, including: a basic signal module. Auxiliary signal module, signal processing module.

The basic signal module is a traditional speech recognition module. The above-mentioned speech recognition module is used to recognize the preprocessed audio data through the audio sensor; the recognition objects of the speech recognition module include speech recognition of isolated vocabulary and speech recognition of continuous large vocabulary , the former is mainly used to determine control instructions, and the latter is mainly used for text input. In the present invention, the recognition of isolated vocabulary is mainly used as an example for illustration, and the recognition of continuous large vocabulary adopts the same processing method.

Optionally, the audio sensor is a microphone array or a directional microphone. Due to the existence of various forms of noise interference in the environment, the existing audio acquisition methods based on ordinary microphones have the same sensitivity to user voice and environmental noise, and have no ability to distinguish voice from noise, so it is easy to cause the correctness of user voice recognition instructions. Decline. The above problems can be overcome by using a microphone array or a directional microphone, using sound source localization and voice enhancement algorithms to track the voice of the operating user and enhance its voice signal, suppressing the influence of ambient noise and human voice interference, and improving the system voice and audio input. The signal-to-noise ratio ensures the reliability of the data quality obtained by the back-end algorithm.

Auxiliary signal module, including front-end camera, audio sensor, throat vibration sensor; used to obtain video data, audio data and motion data;

Optionally, the throat vibration sensor is integrated into the wearable device, and the position is in contact with the user's throat to detect the voice vibration generated by the user. A temperature sensor is placed on the inside of the wearable device, and a temperature sensor is placed on the outside of the wearable device. The processor judges whether the wearable device is worn by the user by comparing the temperatures detected by the two sensors. When the wearable device is not worn, it will automatically enter the sleep mode to reduce the overall power consumption of the wearable device. The microprocessor will detect the state of the vibration sensor to judge and recognize the voice command sent by the user, and send the voice command to the device to be controlled through the Bluetooth device to execute the voice recognition command.

The signal processing unit includes a lip shape recognition module, a face recognition module, a vibration recognition module, a gesture recognition module, a speech recognition module and a sub-adjustment module; it is used to recognize the basic signal (voice signal) and the auxiliary signal, and select the basic signal as The main voice information, using the auxiliary signal as auxiliary voice information;

Use the basic signal (speech signal) as the basic signal for analysis and confirmation, and then the auxiliary signal for auxiliary judgment. In the specific recognition process, select several words with the highest probability scores obtained from speech signal recognition as candidate words , for generating a multi-level set of related words according to a predetermined vocabulary for each candidate word. The auxiliary speech information generated by the auxiliary signal is used to improve the scores of the candidate words in the speech recognition model and the related words in the related word set in the phonetic recognition model lexicon. After the basic signal and the auxiliary signal are all processed, the candidate word or related word with the highest score is selected as the recognition result.

The lip shape recognition module is used to extract the features of the lip shape image from the preprocessed video data, and use the lip shape information to identify the current user's mouth shape changes;

The above-mentioned face recognition module is used to extract the facial features of the user in the video data, determine the identity and location of the user, and identify the identity of different registered users is mainly conducive to the customization of the personalized operation of the entire device, such as different control rights Granted, the user's location information can be used to assist gesture recognition to determine the operating area of the user's hand and determine the orientation of the user when performing voice operations, so as to improve the audio input gain of the microphone user's orientation; when there are multiple possible users, this module It can identify the position of all faces, judge all user identities, and process them separately. Ask the user which user in the camera's field of view will be granted control;

The above gesture recognition module is used to extract the gesture information in the preprocessed video data, determine the hand shape, the movement trajectory of the hand, and the coordinate information of the hand in the image, and then track any hand shape, and the outline of the hand in the image For analysis, the user obtains the start and control rights of the entire terminal through specific gestures or actions.

Through an optional embodiment, the existing various forms of human-computer interaction technologies, including gesture recognition, throat vibration recognition, voice recognition, face recognition, lip shape recognition technology, etc., are integrated, and voice recognition is used as the basic signal. Use lip shape recognition, face recognition, gesture recognition, throat vibration recognition, etc. as auxiliary signals to adjust the score of speech recognition candidate words. Using the basic signal (speech signal) as the basic signal for analysis and confirmation, and then the auxiliary signal for auxiliary judgment, the logical judgment sequence takes advantage of the advantages of each technology in its application field, learns from each other, and each technology module is relatively independent and integrated with each other. The information identifies the current user's mouth shape changes, based on this to reduce the misjudgment rate when the user performs voice recognition operations, so as to ensure that voice operations can be recognized normally in noisy environments; the face recognition module recognizes the user's location information, which can It is used to assist gesture recognition to determine the operating area of the user's hand and determine the orientation of the user when performing voice operations, so as to increase the audio input gain of the microphone user's orientation. Thereby overcoming the influence of noise, significantly improving the speech recognition rate, and then converting the results into relevant instructions. It has done a good job of improving the stability of terminal voice recognition and the comfort of operation.

The steps shown in the flowcharts of the accompanying drawings may be performed in a user terminal such as a smartphone, a tablet computer, etc., and, although a logical order is shown in the flowcharts, in some cases, may be performed in a manner different from that shown here. Perform the steps shown or described in the order shown or described.

This embodiment provides a speech recognition processing method. FIG. 7 is a flowchart of a speech recognition processing method according to an embodiment of the present invention. As shown in FIG. 7, the process includes:

Step S702, using the voice information acquired by the audio sensor as a basic signal for recognition processing;

In step S704, lip shape recognition, face recognition, vibration recognition, and gesture recognition are used as auxiliary signals for recognition processing, and the recognition results of basic signals are sub-adjusted.

Speech recognition objects include speech recognition of isolated vocabulary and speech recognition of continuous large vocabulary. The former is mainly used to determine control instructions, and the latter is mainly used for text input. In this embodiment, the recognition of isolated vocabulary is taken as an example for illustration, and the recognition of continuous large vocabulary adopts the same processing method. Through the above steps, the basic signal (speech signal) is used as the basic signal for analysis and confirmation, and then the auxiliary signal is used for auxiliary judgment of the logical judgment sequence, and several words with the highest possibility scores obtained by speech signal recognition are selected as candidate words , for generating a multi-level set of related words according to a predetermined vocabulary for each candidate word. The candidate word category with the highest probability score generated by auxiliary signal recognition is used as auxiliary information, and several candidate words recognized by the basic signal are judged in turn. The score of the related words in the phonetic model lexicon. After the basic signal and the auxiliary signal are all processed, the candidate word or related word with the highest score is selected as the recognition result.

In the specific implementation process, lip shape recognition, face recognition, vibration recognition, and gesture recognition are used as auxiliary signals for recognition processing. Various recognition methods are independent of each other, and one or more recognition methods can be used as auxiliary signal input at the same time.

An apparatus is also provided in the embodiment, and the apparatus corresponds to the method in the foregoing embodiment, and what has been described will not be repeated here. The modules or units in the device may be codes stored in a memory or in a user terminal and executed by a processor, or may be implemented in other ways, which will not be exemplified here.

According to one aspect of the present invention, a speech recognition processing device is also provided. FIG. 8 is a structural block diagram of a speech recognition processing device according to an embodiment of the present invention. As shown in FIG. 8, the device includes:

The basic signal module, including an audio sensor, is a traditional speech recognition module, and the above-mentioned speech recognition module is used to recognize the preprocessed audio data through the audio sensor;

Auxiliary signal module, including front-end camera, throat vibration sensor; used to acquire video data, audio data and motion data, including lip shape recognition, face recognition, throat vibration recognition, gesture recognition, etc.;

Signal processing module, including lip shape recognition module, face recognition module, vibration recognition module, gesture recognition module, voice recognition module and sub-adjustment module; used to identify the basic signal (voice signal) and auxiliary signal, select the basic signal as For the main voice information, the auxiliary signal is used as auxiliary information for sub-adjustment;

The lip shape recognition module is used to extract the features of the lip shape image from the preprocessed video data, and use the lip shape information to identify the current user's mouth shape changes;

The above-mentioned face recognition module is used to extract the facial features of the user in the video data, determine the identity and location of the user, and identify the identity of different registered users is mainly conducive to the customization of the personalized operation of the entire device, such as different control rights the grant of;

The above gesture recognition module is used to extract the gesture information in the preprocessed video data, determine the hand shape, the movement trajectory of the hand, and the coordinate information of the hand in the image, and then track any hand shape, and the outline of the hand in the image For analysis, the user obtains the activation and control rights of the entire terminal through specific gestures or actions;

Fig. 9 is a flow chart of the speech recognition processing method according to the present invention, as shown in Fig. 9, the speech recognition method of this embodiment is as follows:

Step S902, the voice information obtained from the audio sensor, the video data and motion data obtained from the front-end camera and the throat vibration sensor, including information such as lip shape recognition, face recognition, throat vibration recognition, and gesture recognition;

Step S904, taking speech recognition of an isolated vocabulary as an example, identifying and confirming the speech signal as a basic signal, identifying the isolated vocabulary and obtaining several words with the highest probability as candidate words;

Step S906, collect motion images of the human body with the camera of the terminal device or an external sensor, including gesture motion, facial motion, throat vibration, lip shape recognition, etc. as auxiliary signals, analyze and confirm, and obtain the candidate word with the highest possible score category;

Step S908, judging several candidate words identified by the basic signal in turn, if they meet the category of candidate words identified by the auxiliary signal, then increase the score of the candidate word in the phonetic model lexicon;

Step S910, after the basic signal and the auxiliary signal are all processed, the candidate word with the highest score is selected as the recognition result.

The following uses a specific example to illustrate this optional embodiment. For example, by recognizing the owner's voice, the following results are obtained:

"Please (0.6) business card folder (0.9) call (0.9) browser (0.7), where the value in brackets is the possibility score value, representing the possibility, the greater the score, the greater the possibility. Select the possibility score The word with the highest value is the candidate word. For example, the following candidate words are selected: business card folder (0.9) call (0.9) as the speech recognition result.

Simultaneous gesture movement, facial movement, throat vibration, lip shape recognition, etc. are combined or only one or more of them are used as auxiliary signals for recognition, and the candidate word category with the highest possible score is obtained.

Sequentially judge the call (0.9) of the business card holder (0.9) recognized by the voice signal, and judge whether it matches the candidate word category recognized by the auxiliary signal. Assume that the business card holder meets the candidate word category. Then increase the possibility score of the contacts, for example update to contacts (1.0) call (0.9).

When the speech basic signal and auxiliary signal are all processed, the candidate word card folder (1.0) with the highest score is selected as the recognition result.

As an optional embodiment of this embodiment, a logical judgment sequence may be used in which the auxiliary signal is first identified to determine the category of the candidate words, and then the voice signal is used as the basic signal for analysis and confirmation. First, a combination of gesture movement, facial movement, throat vibration, lip shape recognition, etc., or only one or more of them are used as auxiliary signals for recognition. When multiple methods are used for recognition, the recognition of each method The results are accumulated and processed to obtain the candidate word category with the highest possible score. On this basis, combined with the speech recognition results, the word with the highest possible score is selected as the final recognition result. The scheme is described below with a specific example. For example, by recognizing the owner's voice, the following results are obtained:

"Please (0.6) name card folder (0.9) call (0.9) browser (0.7), where the value in brackets is the possibility score. Select the word with the highest possibility score as the candidate word, for example, select the following candidate word: Contacts (0.9) call (0.9) as a speech recognition result.

Simultaneous laryngeal vibration and lip shape recognition are combined as an auxiliary signal for recognition. Assume that the laryngeal vibration recognition is the first, and then judge the business card holder (0.9) and call (0.9) recognized by the basic signal to determine whether it meets the throat. Candidate word categories identified by vibration recognition. Assuming that the business card meets the category of throat vibration recognition, the likelihood score of the business card is increased, eg, updated to business card (1.0) call (0.9). On the basis of the last recognition result, the lip shape recognition judgment is continued, and the business card holder (1.0) and call (0.9) are judged in turn to judge whether it meets the candidate word category of lip shape recognition. Assuming that the name card holder meets the category of lip shape recognition, then increase the possibility score of the name card holder, for example, update to name card holder (1.1) call (0.9). The recognition results of the two methods are accumulated.

After the speech basic signal and auxiliary signal are all processed, select the candidate word card folder (1.1) with the highest score as the recognition result.

As an optional embodiment of this embodiment, the further screening process is completed through score adjustment, that is, the score of candidate words that meet auxiliary signal recognition can be increased, and the score of candidate words that do not meet auxiliary signal recognition can also be reduced. , when the basic signal and the auxiliary signal are all processed, the candidate word with the highest score is selected as the recognition result.

As an optional embodiment of this embodiment, it is optional for the user to use auxiliary information to confirm the recognition result added in order to improve the accuracy of the speech recognition, and the speech recognizer determines the recognition result according to the input speech. A likelihood measure is calculated for the above recognition results. If the possibility metric value is less than the threshold, the user is prompted whether to input auxiliary data or the identification of auxiliary data is automatically enabled. If the possibility metric is greater than the threshold, the user is prompted whether to turn off the auxiliary data or automatically turn off the recognition of the auxiliary data. The specific numerical value of the threshold is not limited, and is derived from empirical values or based on user experience.

Based on the speech recognition method improved in the above-mentioned embodiments, the existing various forms of human-computer interaction technologies, including gesture recognition, throat vibration recognition, speech recognition, face recognition, lip shape recognition technology, etc. are integrated, speech recognition As the basic signal, lip shape recognition, face recognition, gesture recognition, throat vibration recognition, etc. are used as auxiliary signals to adjust the points of speech recognition candidate words. Using the basic signal (speech signal) as the basic signal for analysis and confirmation, and then the auxiliary signal for auxiliary judgment, the logic judgment sequence is well done to improve the stability of terminal speech recognition and the comfort of operation.

In summary, through the speech recognition processing method and device provided by the present invention, on the basis of speech recognition, as the basic signal, lip shape recognition, face recognition, gesture recognition, throat vibration recognition, etc. are used as auxiliary signals. Signal. The problem of poor user experience caused by low voice recognition rate in the related art is solved. Taking advantage of the advantages of each technology in its application field, learning from each other, each technology module is relatively independent and integrated with each other, which greatly improves the recognition rate of speech processing.

In another embodiment, software is also provided, and the software is used to implement the technical solutions described in the above embodiments and preferred implementation manners.

In another embodiment, there is also provided a storage medium, in which the software is stored, the storage medium includes but not limited to: optical discs, floppy disks, hard disks, rewritable memories, and the like.

Obviously, those skilled in the art should understand that each module or each step of the above-mentioned present invention can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed in a network formed by multiple computing devices Alternatively, they may be implemented in program code executable by a computing device so that they may be stored in a storage device to be executed by a computing device, and in some cases in an order different from that shown here The steps shown or described are carried out, or they are separately fabricated into individual integrated circuit modules, or multiple modules or steps among them are fabricated into a single integrated circuit module for implementation. As such, the present invention is not limited to any specific combination of hardware and software.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (17)

1. A method for speech recognition, characterized in that, comprising: Acquiring voice recognition information of the user's current voice, and acquiring auxiliary identification information of the voice recognition information based on the user's current state corresponding to the user's current voice; A final recognition result of the user's current voice is determined according to the voice recognition information and the auxiliary recognition information. 2. The method according to claim 1, wherein determining the final recognition result of the user's current voice according to the voice recognition information and the auxiliary recognition information comprises: Obtain one or more first candidate words corresponding to the user's current voice according to the voice recognition information; Obtain the vocabulary category or one or more second candidate vocabulary corresponding to the user's current voice according to the auxiliary identification information; Determine the final recognition result of the user's current voice according to the one or more first candidate vocabulary and the vocabulary category; or, according to the one or more first candidate vocabulary and the one or more second candidate Vocabulary determines the final recognition result of the user's current speech. 3. The method according to claim 2, wherein determining the final recognition result of the user's current voice according to the one or more first candidate vocabulary and the vocabulary type comprises: Selecting a first specific word matching the vocabulary category from the one or more first candidate words, and using the first specific word as a final recognition result of the user's current speech. 4. The method according to claim 2, wherein determining the final recognition result of the user's current voice according to the one or more first candidate words and the one or more second candidate words comprises: Select a second specific vocabulary with a high similarity with the one or more first candidate vocabulary from the one or more second candidate vocabulary, and use the second specific vocabulary as the final recognition result of the user's current voice . 5. The method according to claim 1, wherein obtaining the auxiliary identification information of the voice recognition information based on the user's current state corresponding to the user's current voice comprises: obtaining an image indicative of the user's current status; Acquiring image feature information according to the image; The vocabulary category and/or one or more candidate vocabulary corresponding to the image feature information are acquired according to the image feature information, and the vocabulary category and/or the one or more candidate vocabulary are used as the auxiliary identification information. 6. The method according to claim 5, wherein obtaining the vocabulary category and/or one or more candidate vocabulary corresponding to the image feature information according to the image feature information comprises: Searching for a specific image with the highest similarity to the image feature information in a predetermined image library; According to the preset correspondence between the image and the vocabulary category or one or more candidate vocabulary, the vocabulary category or one or more candidate vocabulary corresponding to the specific image is acquired. 7. The method according to any one of claims 1 to 6, wherein the current state of the user includes at least one of the following: the user's lip movement state, the user's throat vibration state, The facial movement state of the user and the gesture movement state of the user. 8. The method according to any one of claims 1 to 7, wherein the speech recognition information of the user's current voice is obtained, and the speech recognition information of the speech recognition information is obtained based on the user's current state corresponding to the user's current speech. Secondary identifying information previously included: Determining that the accuracy rate of the final recognition result of the user's current voice determined based on the voice recognition information is less than a predetermined threshold. 9. A device for speech recognition, characterized in that the device comprises: An acquisition module, configured to acquire voice recognition information of the user's current voice, and acquire auxiliary identification information of the voice recognition information based on the user's current state corresponding to the user's current voice; A determining module, configured to determine a final recognition result of the user's current voice according to the voice recognition information and the auxiliary recognition information. 10. The device according to claim 9, wherein the determining module comprises: A first acquiring unit, configured to acquire one or more first candidate words corresponding to the user's current voice according to the voice recognition information; A second acquiring unit, configured to acquire the vocabulary category corresponding to the user's current voice or one or more second candidate vocabulary according to the auxiliary identification information; A determining unit, configured to determine the final recognition result of the user's current speech according to the one or more first candidate words and the vocabulary category; or, according to the one or more first candidate words and the one or A plurality of second candidate words determine the final recognition result of the user's current speech. 11. The device according to claim 10, wherein the determining unit is further configured to select a first specific vocabulary that conforms to the vocabulary category from the one or more first candidate vocabulary, and the second A specific word is used as the final recognition result of the user's current speech. 12. The device according to claim 10, wherein the determining unit is further configured to select, from the one or more second candidate words, those with high similarity to the one or more first candidate words The second specific vocabulary, using the second specific vocabulary as the final recognition result of the user's current voice. 13. The device according to claim 9, wherein the acquiring module further comprises: a third acquiring unit, configured to acquire an image indicating the current state of the user; a fourth acquiring unit, configured to acquire image feature information according to the image; A fifth acquiring unit, configured to acquire, according to the image feature information, a vocabulary category and/or one or more candidate vocabulary corresponding to the image feature information, and convert the vocabulary category and/or the one or more candidate vocabulary to as the auxiliary identification information. 14. The device according to claim 13, wherein the fifth acquiring unit further comprises: A search subunit, configured to search for a specific image with the highest similarity to the image feature information in a predetermined image library; The acquiring subunit is configured to acquire the vocabulary category or one or more candidate vocabulary corresponding to the specific image according to the preset correspondence between the image and the vocabulary category or one or more candidate vocabulary. 15. The device according to any one of claims 9 to 14, wherein the current state of the user includes at least one of the following: the user's lip movement state, the user's throat vibration state, The facial movement state of the user and the gesture movement state of the user. 16. The device according to any one of claims 9 to 15, further comprising: A judging module, configured to judge that the accuracy rate of the final recognition result of the user's current speech based on the speech recognition information is less than a predetermined threshold. 17. A terminal, comprising a processor, wherein the processor is configured to acquire voice recognition information of the user's current voice, and acquire auxiliary recognition of the voice recognition information based on the user's current state corresponding to the user's current voice information; determine the final recognition result of the user's current voice according to the voice recognition information and the auxiliary recognition information.