KR102755684B1 - Elecctronic apparatus and method for performing speech recognition based speech termination and speech certainty - Google Patents

Abstract

An electronic device for performing speech recognition based on utterance termination and utterance certainty according to the present disclosure comprises: a speech recognition module for receiving user speech data based on a user interface; a memory for storing at least one process for performing a user speech termination detection operation; and at least one processor for performing the user speech termination detection operation according to the process; wherein the at least one processor may be configured to output whether utterance has been terminated for input user speech data using a Voice Activity Detection (VAD) module and a Natural Language Understanding (NLU) processing module, and to receive the utterance termination status using the NLU processing module and an Automatic Speech Recognition Confidence Score (ASR) calculation module and output whether re-speech is requested.

Description

{ELECCTRONIC APPARATUS AND METHOD FOR PERFORMING SPEECH RECOGNITION BASED SPEECH TERMINATION AND SPEECH CERTAINTY}

The present disclosure relates to an electronic device and method for performing speech recognition, and more particularly, to an electronic device and method for performing speech recognition based on utterance termination and utterance certainty.

Recently, AICC (AI Contact Center) has been built in various fields to perform customer response tasks, etc. using AI models. In other words, in an environment where AI models and people converse, human voices are recognized, and scenarios in which AI models respond based on the recognition results are automatically performed.

When a human response to an AI model's query is input, the method of determining when the response is complete and the AI model should process the response is called Endpoint Detection. Endpoint Detection is a technology that performs real-time STT (Speech To Text) and detects the end point of a human response and divides the speech section. In the past, when a preset time elapsed after the end of the recognized voice, the elapsed time was determined as the end point of the speech.

However, in an environment where a user answers an AI model's query, there may be situations where the user's voice is noisy, such as when another person's voice is recognized, or the user has a gap in their voice and makes an additional answer, making the end of the utterance unclear. In such situations, the AI model cannot clearly determine the end of the user's utterance.

Even if noise is removed from the user's voice signal, whether the user has ended speech or has additional speech remaining is determined by subjective circumstances, making objective preprocessing difficult, and it is difficult for an AI model to objectively determine the end of speech.

In addition, even if it is determined whether the speech has ended, if the user's voice recognition is performed inaccurately, the AI model must request re-speech, and this cannot be performed together with the judgment of the end of speech, so it must be performed as a separate procedure, which is inconvenient.

Republic of Korea Public Notice 2012-0081639 A

In order to solve the problems of the related art, the embodiment disclosed in the present disclosure aims to provide an electronic device and method for performing speech recognition based on utterance termination and utterance certainty by combining a VAD (Voice Activity Detection) module, an NLU (Natural Language Understanding) processing module, and an ASR (Automatic Speech Recognition Confidence Score) calculation module to determine whether utterance has ended or whether a re-utterance is requested.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

An electronic device for performing speech recognition based on utterance termination and utterance certainty according to the present disclosure for achieving the above-described technical problem comprises: a speech recognition module for receiving user speech data based on a user interface; a memory storing at least one process for performing a user speech termination detection operation; and at least one processor for performing the user speech termination detection operation according to the process; wherein the at least one processor may be configured to output whether utterance has been terminated for input user speech data using a Voice Activity Detection (VAD) module and a Natural Language Understanding (NLU) processing module, and to receive the utterance termination status using the NLU processing module and an Automatic Speech Recognition Confidence Score (ASR) calculation module and output whether re-speech is requested.

According to the present disclosure for achieving the above-described technical problem, a speech recognition method based on utterance termination and utterance certainty is provided, which is performed by a computing device including a user interface-based speech recognition module, a memory, and at least one processor that performs a user utterance termination detection operation, wherein the method may include a step of outputting whether utterance has been terminated for input user voice data by using a Voice Activity Detection (VAD) module and a Natural Language Understanding (NLU) processing module; and a step of receiving whether the utterance has been terminated and outputting whether a re-utterance is requested by using the NLU processing module and an Automatic Speech Recognition Confidence Score (ASR) calculation module.

In addition, a computer program stored in a computer-readable recording medium for implementing the present disclosure may be further provided.

In addition, a computer-readable recording medium recording a computer program for implementing the present disclosure may be further provided.

According to the above-described problem solving means of the present disclosure, when performing STT by recognizing a user's voice in real time, it is possible to accurately determine whether the user's speech has ended, thereby providing an effect of improving voice recognition accuracy and performance.

In addition, the aforementioned problem solving means of the present disclosure provides an effect of inducing the AI module to quickly acquire an accurate user voice through a series of processes by determining whether re-speech is necessary as an answerable utterance.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

FIG. 1 is a block diagram briefly illustrating the configuration of an electronic device that performs speech recognition based on utterance termination and utterance certainty according to one embodiment of the present disclosure.
FIG. 2 is a block diagram briefly illustrating the configuration of a processor of an electronic device that performs speech recognition based on utterance termination and utterance certainty according to one embodiment of the present disclosure.
FIG. 3 is a conceptual diagram schematically illustrating a VAD module process of an electronic device that performs speech recognition based on utterance termination and utterance certainty according to one embodiment of the present disclosure.
FIG. 4 is a diagram illustrating a speech recognition result according to an electronic device that performs speech termination and speech certainty-based speech recognition according to one embodiment of the present disclosure.
FIG. 5 is a data transmission and reception diagram schematically illustrating a data transmission and reception process for an audio stream of an electronic device performing speech recognition based on utterance termination and utterance certainty according to one embodiment of the present disclosure.
FIG. 6 is a table illustrating an utterance end determination process of an electronic device that performs utterance end and utterance certainty-based voice recognition according to one embodiment of the present disclosure.
FIG. 7 is a table illustrating a re-ignition request process of an electronic device that performs speech termination and speech certainty-based speech recognition according to one embodiment of the present disclosure.
FIG. 8 is a flow chart illustrating a speech recognition method based on utterance termination and utterance certainty according to one embodiment of the present disclosure.

Throughout this disclosure, the same reference numerals refer to the same components. This disclosure does not describe all elements of the embodiments, and any content that is general in the technical field to which this disclosure belongs or that overlaps between the embodiments is omitted. The terms ‘part, module, element, block’ used in the specification can be implemented in software or hardware, and according to the embodiments, a plurality of ‘parts, modules, elements, blocks’ can be implemented as a single component, or a single ‘part, module, element, block’ can include a plurality of components.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only a direct connection but also an indirect connection, and an indirect connection includes a connection via a wireless communications network.

Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

Throughout the specification, when it is said that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where there is another element between the two elements.

The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

Singular expressions include plural expressions unless the context clearly indicates otherwise.

The identification codes in each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

In this specification, the 'device according to the present disclosure' includes all of various devices that can perform computational processing and provide results to a user. For example, the device according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may be in the form of any one of them.

Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

The above portable terminal may include, for example, all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted-device (HMD).

The function related to artificial intelligence according to the present disclosure is operated through a processor and a memory. The processor may be composed of one or more processors. At this time, one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control to process input data according to a predefined operation rule or artificial intelligence model stored in a memory. Alternatively, when one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

The function related to artificial intelligence according to the present disclosure is operated through a processor and a memory. The processor may be composed of one or more processors. At this time, one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control to process input data according to a predefined operation rule or artificial intelligence model stored in a memory. Alternatively, when one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned by using a plurality of learning data by a learning algorithm, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). Such learning may be performed in the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system. Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs a neural network operation through an operation between the operation result of the previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, the plurality of weights may be updated so that a loss value or a cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network.

According to an exemplary embodiment of the present disclosure, a processor can implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that imitates human biological neurons to enable a machine to learn. Artificial intelligence methodologies can be divided into supervised learning in which input data and output data are provided together as training data according to a learning method, so that an answer (output data) to a problem (input data) is determined, unsupervised learning in which only input data is provided without output data, so that an answer (output data) to a problem (input data) is not determined, and reinforcement learning in which a reward is given from an external environment whenever an action is taken in a current state, and learning is performed in a direction to maximize this reward. In addition, artificial intelligence methodologies can be categorized by architecture, which is the structure of the learning model. The architectures of widely used deep learning technologies can be categorized into convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, and generative adversarial networks (GANs).

The present device and system may include an artificial intelligence model. The artificial intelligence model may be one artificial intelligence model or may be implemented as multiple artificial intelligence models. The artificial intelligence model may be composed of a neural network (or an artificial neural network) and may include a statistical learning algorithm that mimics biological neurons in machine learning and cognitive science. A neural network may refer to a model in which artificial neurons (nodes) that form a network by combining synapses change the strength of the synapses through learning and have problem-solving capabilities. Neurons of a neural network may include a combination of weights or biases. A neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. A neural network that constitutes the device may infer a desired result (output) from an arbitrary input (input) by changing the weights of neurons through learning.

The processor can generate a neural network, train (or learn) a neural network, perform a calculation based on received input data, generate an information signal based on the result of the calculation, or retrain the neural network. The models of the neural network can include various types of models such as CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, etc., but are not limited thereto. The processor can include one or more processors for performing calculations according to the models of the neural network. For example, the neural network can include a deep It may include a neural network (Deep Neural Network).

Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), and LSTM. (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), Generative Adversarial Network (GAN), Liquid State Machine (LSM), Extreme Learning Machine (ELM), It can include ESN (Echo State Network), DRN (Deep Residual Network), DNC (Differentiable Neural Computer), NTM (Neural Turning Machine), CN (Capsule Network), KN (Kohonen Network) and AN (Attention Network). It will be appreciated by those skilled in the art that this may include any neural network, but is not limited thereto.

According to an exemplary embodiment of the present disclosure, the processor may be configured to perform a CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC/QA for natural language processing, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, ResNet for data intelligence, Anomaly Detection, Prediction, Time-Series Forecasting, Various artificial intelligence structures and algorithms such as Optimization, Recommendation, and Data Creation can be used, but are not limited thereto. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

FIG. 1 is a block diagram briefly illustrating the configuration of an electronic device (100) that performs speech recognition based on utterance termination and utterance certainty according to one embodiment of the present disclosure.

Referring to FIG. 1, an electronic device (100) according to the present disclosure may include an input/output module (110), a communication module (120), a memory (130), and a processor (140). Hereinafter, the electronic device (100) is an electronic device that performs speech recognition operations based on utterance termination and speech certainty, and is assumed to be implemented through an electronic device (100) that performs speech recognition operations based on utterance termination and speech certainty.

The input/output module (110) may be various interfaces or connection ports that receive user input or output information to the user. The input/output module (110) may be divided into an input module and an output module.

The input module receives user input from a user. The input module is for inputting image information (or signal), audio information (or signal), data, or information input from a user, and may include at least one camera, at least one microphone, and at least one user input unit. Voice data or image data collected from the input unit may be analyzed and processed as a user control command.

User input can take various forms, including key input, touch input, and voice input. Examples of input modules that can receive such user input include, in addition to traditional keypads, keyboards, and mice, touch sensors that detect the user's touch, microphones that receive voice signals, cameras that recognize gestures through image recognition, proximity sensors such as light sensors or infrared sensors that detect the approach of a user, motion sensors that recognize user movements through acceleration sensors or gyro sensors, and various other input means that detect or receive various forms of user input.

Here, the touch sensor may be implemented as a piezoelectric or electrostatic touch sensor that detects touch through a touch panel or touch film attached to the display panel, an optical touch sensor that detects touch in an optical manner, etc. In addition, the input module may be implemented in the form of an input interface (such as a USB port or PS/2 port) that connects an external input device that receives user input instead of a device that detects user input on its own.

The output module can output various information and provide it to the user. The output module is a comprehensive concept that includes a display that outputs images, a speaker (and/or an amplifier connected thereto) that outputs sounds, a haptic device that generates vibrations, and various other forms of output means. In addition, the output module can also be implemented in the form of a port-type output interface that connects the individual output means described above.

For example, an output module in the form of a display can display text, still images, and moving images. The display is a concept that broadly refers to an image display device including various forms of devices that can perform image output functions, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flat panel display (FPD), a transparent display, a curved display, a flexible display, a 3D display, a holographic display, a projector, and others. Such a display may be in the form of a touch display configured integrally with a touch sensor of the input module.

In other words, the input/output module (110) can receive user input or provide output to the user based on a user interface.

The communication module (120) can communicate with an external device. Therefore, the device can transmit and receive information with the external device through the communication module. For example, the device can communicate with the external device so that information stored and generated within the electric vehicle charging management system is shared using the communication module. The communication module (120) can include, for example, at least one of a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

Here, communication, that is, data transmission and reception, can be performed wired or wirelessly. To this end, the communication module may be configured as a wired communication module that connects to the Internet, etc. via a LAN (Local Area Network), a mobile communication module that connects to a mobile communication network via a mobile communication base station to transmit and receive data, a short-distance communication module that uses a WLAN (Wireless Local Area Network) series communication method such as Wi-Fi or a WPAN (Wireless Personal Area Network) series communication method such as Bluetooth or Zigbee, a satellite communication module that uses a GNSS (Global Navigation Satellite System) such as a GPS (Global Positioning System), or a combination thereof. The wireless communication technology used for communication may include NB-IoT (Narrowband Internet of Things) for low-power communication. At this time, for example, NB-IoT technology may be an example of LPWAN (Low Power Wide Area Network) technology, and may be implemented with standards such as LTE Cat (category) NB1 and/or LTE Cat NB2, and is not limited to the names described above. Additionally or alternatively, a wireless communication technology implemented in a wireless device according to various embodiments may perform communication based on LTE-M technology. At this time, for example, LTE-M technology may be an example of LPWAN technology, and may be called by various names such as eMTC (enhanced Machine Type Communication). For example, LTE-M technology may be implemented with at least one of various standards such as 1) LTE CAT 0, 2) LTE Cat M1, 3) LTE Cat M2, 4) LTE non-BL (non-Bandwidth Limited), 5) LTE-MTC, 6) LTE Machine Type Communication, and/or 7) LTE M, and is not limited to the names described above. Additionally or alternatively, the wireless communication technology implemented in the wireless device according to various embodiments may include at least one of ZigBee, Bluetooth, and Low Power Wide Area Network (LPWAN) considering low-power communication, and is not limited to the above-described names. For example, ZigBee technology can create PAN (personal area networks) related to small/low-power digital communication based on various standards such as IEEE 802.15.4, and may be called by various names.

The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard232), power line communication, or plain old telephone service (POTS).

The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a WiFi module and a Wireless broadband module.

The wireless communication module may include a wireless communication interface including an antenna and a transmitter for transmitting a signal. In addition, the wireless communication module may further include a signal conversion module for modulating a digital control signal output from the control unit through the wireless communication interface into an analog wireless signal according to the control of the control unit.

The wireless communication module may include a wireless communication interface including an antenna and a receiver for receiving a signal. In addition, the wireless communication module may further include a signal conversion module for demodulating an analog wireless signal received through the wireless communication interface into a digital control signal.

The short-range communication module is for short-range communication and can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

The location information module is a module for obtaining the location (or current location) of the electronic device (10) according to the present disclosure, and representative examples thereof include a GPS (Global Positioning System) module or a WiFi (Wireless Fidelity) module. For example, if a GPS module is utilized, the location of the electronic device can be obtained by using a signal sent from a GPS satellite. As another example, if a Wi-Fi module is utilized, the location of the electronic device (10) can be obtained based on information of a wireless AP (Wireless Access Point) that transmits or receives a wireless signal with the Wi-Fi module. If necessary, the location information module may perform any function of other modules of the communication module to obtain data regarding the location of the device as a substitute or in addition. The location information module is a module used to obtain the location (or current location) of the device, and is not limited to a module that directly calculates or obtains the location of the device.

The memory (130) can store various types of information. The memory can store data temporarily or semi-permanently. For example, the memory can store an operating program (OS: Operating System) for operating the first device and/or the second device, data for hosting a website, a program for generating Braille, or data regarding an application (e.g., a web application). In addition, the memory can store modules in the form of computer codes as described above.

Examples of memory (130) may include a hard disk drive (HDD), a solid state drive (SSD), flash memory, read-only memory (ROM), random access memory (RAM), etc. Such memory may be provided as a built-in type or a removable type.

The processor (140) controls the overall operation of the electronic device (100). To this end, the processor (140) can perform calculations and processing of various types of information and control the operation of components of the first device and/or the second device.

The processor (140) may be implemented as a computer or a similar device according to hardware, software, or a combination thereof. In terms of hardware, the processor (140) may be provided in the form of an electronic circuit that processes electrical signals to perform a control function, and in terms of software, the processor may be provided in the form of a program that drives a hardware processor. Meanwhile, unless otherwise specifically mentioned in the following description, the operations of the first device and/or the second device may be interpreted as being performed under the control of the processor (140). That is, the modules may be interpreted as controlling the processor (140) to perform the following operations on the first device and/or the second device.

The processor (140) may be implemented as a memory storing data for an algorithm for controlling the operation of components within the device or a program reproducing the algorithm, and at least one subprocessor (not shown) performing the above-described operation using the data stored in the memory. In this case, the memory and the processor may be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

In addition, the processor (140) may control one or more of the components described above in combination to implement various embodiments according to the present disclosure described in FIGS. 2 to 8 below on the device.

FIG. 2 is a block diagram briefly illustrating the configuration of a processor of an electronic device that performs speech recognition based on utterance termination and utterance certainty according to one embodiment of the present disclosure.

An electronic device (100) that performs speech recognition based on utterance termination and utterance certainty according to one embodiment of the present disclosure may include a speech recognition module that receives user voice data based on a user interface, a memory (130) that stores at least one process for performing a user utterance termination detection operation, and at least one processor (140) that performs the user utterance termination detection operation according to the process.

At least one processor according to one embodiment of the present disclosure may be configured to output whether or not utterance has been terminated for input user voice data using a Voice Activity Detection (VAD) module (141) and a Natural Language Understanding (NLU) processing module (142), and to receive whether or not utterance has been terminated and output whether or not a re-utterance request has been made using the NLU processing module (142) and an Automatic Speech Recognition (ASR) confidence score calculation module (143).

As illustrated in FIG. 2, at least one processor (140) can use the VAD module (141), the NUL processing module (142), and the ASR confidence score calculation module (143) to cause the endpoint determination module (144) to detect an endpoint of a speech section within an audio stream and perform speech recognition.

As an example, the VAD module (141) may receive an audio stream based on VAD technology, distinguish between an activated section in which a user's speech is performed and an activated section in which no voice is input and the section is deactivated by silence, and output a deactivation time (Te) at which deactivation occurs.

As illustrated in FIG. 3, the audio stream can measure whether time has elapsed from the activation interval end point (1412) in the VAD segment (1411) containing the user speech.

As an example, at least one processor (140) may cause the VAD module (141) to output a de-activation time (Te) from an activation interval end point (1412) of the user input data, and output different indexes according to an interval including the output de-activation time.

De-activate time (Te) refers to the trailing time during which silence is maintained from the activation section end point (1412) where the user's speech ends until the next user speech begins. In this case, silence refers to a section during which no user speech is input, and activation due to noise such as external noise or another person's speech may be included.

Referring to FIG. 3, when Te is X seconds, the index (Vs) output by the VAD module (141) may be different depending on the section including X seconds.

As an example, the sections may include a first section, a second section, and a third section, and the index (Vs) may increase in the order of the first section, the second section, and the third section.

For example, if X belongs to the first interval from 0 to T1, Vs = 1 can be output. If X belongs to the second interval from T1 to T2, Vs = 2 can be output. If X belongs to the interval greater than or equal to T2, Vs = 3 can be output. T2 can be greater than T1.

As an example, the VAD module (141) may further include a VAD neural network that inputs the user voice audio stream and outputs a VAD score.

The VAD neural network can perform voice recognition by calculating high VAD scores for sections that are judged to be human voices in preprocessed audio streams such as noise removal, and low VAD scores for sections that are judged to be noise or elements other than human voices. It can learn parameters that are judged to be human voices and calculate VAD scores based on similarity.

In other words, based on the VAD score output by the VAD neural network, the user speech section and the non-user speech section are distinguished, and the VAD module (141) detects the activation section end point (1412) from the distinguished VAD segment (1411) to determine the section to which the de-activation time (Te) belongs.

As one embodiment, at least one processor (140) may be configured to cause the NLU processing model (142) to determine truth or falsity of a current utterance sentence using a BERT-based encoder or a GPT-based decoder based on a previous utterance sentence of the user speech data.

It is possible to determine whether the speech recognition result calculated at any time is appropriate as a response to the utterance of the existing counterpart, the artificial intelligence model.

For example, if the answer uttered at any time for the previous utterance of the artificial intelligence model, “What would you like to order?” is “I,” the value output by the NLU processing model (142) is 0, and if it is “Coke,” the value output by the NLU processing model (142) can have a value near 1, such as 0.9.

In other words, the value output by the NLU processing model (142) is a value between 0 and 1, and a value below the reference value can be classified as false (F), and a value above the reference value can be classified as true (T).

As an example, the NLU processing model (142) can perform learning using a BERT-based encoder. The NLU processing model (142) including the BERT-based encoder can perform learning as a question-answering model to output the correct answer to a question based on the context. A learning dataset can be created to output the closest answer in terms of context to a previous utterance.

Alternatively, as an example, the NLU processing model (142) can perform training using a BERT-based encoder. The NLU processing model (142) including the BERT-based encoder can perform training to output possible answers instead of outputting the correct answer based on the context. The training dataset can be generated to output possible answers regardless of whether there is a contextual association so that it can determine true or false for the previous utterance.

As another example, as an example, the NLU processing model (142) can perform learning using a GPT-based decoder. The NLU processing model (142) including the GPT-based decoder can perform learning using a learning data set whose inference value is True during encoder learning.

For example, an encoder model can perform supervised learning on an input question and output an answer, and a decoder model can be trained by configuring the input question and the output answer as a learning dataset. The decoder model can measure the perplexity of the answer and infer whether it is true or false.

Perplexity is a measure of the quality of a probabilistic or statistical model in natural language processing. It measures the degree of uncertainty in sample values in a discrete probability distribution, and can output true or false by comparing perplexity with a reference value.

As an example, as illustrated in FIG. 4, an NLU processing model (142) including a BERT-based encoder can generate a learning dataset with an input data structure that attaches a <sos> label to the front of question sentence 1 (14) and a <sep> label to the front of answer sentence 2 (15).

For the question, “Please tell me what banking service you want” (16), you can learn an appropriate answer, such as “Loan consultation” (17), or you can learn a possible answer that is not related to appropriateness, such as “The subway is coming.”

As another example, an NLU processing model (142) including a GPT-based decoder can generate a learning dataset with an input data structure that attaches a <sos> label to the front of a question sentence 1 (14) and a <sep> label to the front of an answer sentence 2 (15).

You can learn appropriate answers to the question, “Please tell me what kind of banking service you want” (16), such as “Loan consultation” (17).

Accordingly, the NLU processing module (142) according to one embodiment of the present disclosure can output whether sentence 2 (15) is true or false (Ns).

As an example, the endpoint determination module (144) may receive the activation section and deactivation section of the VAD module (141) and the true or false (Ns) of the answer of the NLU processing module (142) and output whether the user's speech of the input audio stream has ended.

Additionally, as an embodiment, at least one processor (140) may be configured to cause the ASR confidence score calculation module (143) to output an ASR confidence score for a sentence in which the utterance has ended if the utterance end status is output as true.

If the ASR confidence score is below the reference value, the results of speech recognition cannot be trusted. The ASR confidence score is a probability model that calculates the probability of all texts for the input audio stream, and can generate probability values for sentences spoken by the user. The ASR confidence score can be calculated by quantifying the difference between the probability value of the sentence with the highest probability and the probability value of the sentence with the second highest probability.

For example, given an audio A, we can calculate the probability that it is “Hello” and the probability that it is “I had a nice meal”, and the sentence with the highest probability can have a high ASR confidence score.

As an example, the ASR confidence score value of the speech recognition result may be a value between 0 and 1, and the ASR confidence score may be classified into a plurality of sections to correspond to the number of sections of the VAD module (141).

As an example, the index (Cs) output by the ASR confidence score calculation module (143) may be different depending on the section in which the ASR confidence score is included.

As an example, the ASR reliability score may include a first reliability interval, a second reliability interval, and a third reliability interval, and the index (Cs) of the ASR reliability score may increase in the order of the first reliability interval, the second reliability interval, and the third reliability interval.

For example, if the ASR confidence score belongs to the first confidence interval from 0 to C1, Cs = 1 can be output. If the ASR confidence score belongs to the second confidence interval from C1 to C2, Cs = 2 can be output. If the ASR confidence score belongs to the third confidence interval from C2 to C3, Cs = 3 can be output. In this case, the values can increase in the order of C1, C2, and C3.

As an example, the endpoint determination module (144) may receive the truth or falsehood (Ns) of the answer of the NLU processing module (142) and the ASR confidence score of the ASR confidence score calculation module (143) to determine the certainty of the user's utterance in the input audio stream and output whether or not to ignore the utterance. In other words, it may output whether or not to request the user to speak again.

Accordingly, the endpoint determination module (144) can output a voice recognition result by determining values for whether the user's speech has ended (O_1) and whether the user requests re-speech (O_2).

Specifically, as illustrated in FIG. 5, parameters can be transmitted and received between the VAD module (141), the ASR confidence score calculation module (143), the NLU processing module (142), and the endpoint determination module (144) to output whether the user's speech has ended (O_1) and whether a re-speech request has been made (O_2).

The input audio stream (21) may include multiple user speech segments (11). When the speech segment (11) is activated, an activation parameter may be transmitted to the endpoint determination module (144), and when the speech segment (11) ends and is deactivated, a deactivate parameter may be transmitted to the endpoint determination module (144).

The ASR confidence score calculation module (143) transmits Cs to the endpoint determination module (144), and the ASR confidence score calculation module (143) transmits the speech recognition result to the NLU processing module (142), so that the NLU processing module (142) can transmit Ns to the endpoint determination module (144).

The endpoint determination module (144) can determine whether the user's speech is terminated (O_1) and whether a re-speech request is made (O_2) based on the above-described de-activate and Cs and Ns. If the utterance is terminated (O_1) is Y, the ASR confidence score calculation module (143) can output the voice recognition result and whether a re-speech request is made.

Alternatively, the endpoint determination module (144) can determine whether the user's speech is terminated (O_1) and whether a re-speech request is made (O_2) based on the de-activate and Cs and Ns. If the utterance is terminated (O_1) is N, the ASR confidence score calculation module (143) can output the voice recognition result and whether a re-speech request is made.

The above parameter transmission and reception process can be represented in tables (610) and (710) as shown in FIGS. 6 and 7.

At least one processor (140) according to one embodiment of the present disclosure may be configured to output whether the utterance is terminated as false in the first section regardless of whether the NLU processing model (142) is true or false, output whether the utterance is terminated as false in the second section depending on whether the NLU processing model (142) is true or false, and output whether the utterance is terminated as true in the third section regardless of whether the NLU processing model (142) is true or false.

As shown in Fig. 6, when the index (Vs) of the VAD module (141) is 1, the end of utterance (O_1) can be output as false (F) regardless of whether the NLU processing model (142) is true or false (Ns). If the de-activate time is too short, it is considered that the user has not completely ended the utterance, but has stopped speaking for a moment during the utterance.

When the index (Vs) of the VAD module (141) is 2, the end of utterance (O_1) can be output based on the true or false (Ns) of the NLU processing model (142). If the NLU processing model (142) determines that the answer is true (T), it can output that the utterance is ended (T), and if the answer is determined to be false (F), it can output that the utterance is not ended (F). If an appropriate answer is given, it is determined that the user's utterance is ended, and if an appropriate answer is not given, it is considered that the user's utterance is not ended and is continuing.

When the index (Vs) of the VAD module (141) is 3, the end of utterance (O_1) can be output as true (T) regardless of whether the NLU processing model (142) is true or false (Ns). If the de-activation time is sufficiently long, it is considered that the user has completely ended the utterance.

Accordingly, as shown in Table 610 of Fig. 6, the output values of the VAD module (141) and the NLU processing model (142) can be combined to determine whether the user's speech has ended.

At least one processor (140) according to one embodiment of the present disclosure may be configured to output whether a re-trigger request is made as false in the first confidence interval regardless of whether the NLU processing model (142) is true or false, output whether a re-trigger request is made depending on whether the NLU processing model (142) is true or false in the second confidence interval, and output whether a re-trigger request is made as true in the third confidence interval regardless of whether the NLU processing model (142) is true or false.

As illustrated in Fig. 7, if the ASR reliability score of the ASR reliability score calculation module (143) is 0.1 or less and the index (Cs) is 1, the re-speech request (O_2) can be output as false (F) regardless of the truth or falseness (Ns) of the NLU processing model (142). If the ASR reliability score is too low, a sentence with too low a voice recognition probability is output, so it is considered that there is noise in the external environment or the user's voice accuracy is low, and a re-speech request is necessarily performed.

If the ASR confidence score of the ASR confidence score calculation module (143) is 0.1 to 0.3 or less and the index (Cs) is 2, whether or not to request re-speech (O_2) can be output based on the true/false (Ns) of the NLU processing model (142). If the NLU processing model (142) determines that the answer is true (T), it can output that a re-speech request is unnecessary (T), and if it determines that the answer is false (F), it can output that a re-speech request scenario is possible (F). If a reliable and appropriate answer is provided, it is determined that user re-speech is unnecessary, and if a reliable and appropriate answer is not provided, it is considered that the voice recognition was inaccurate and a re-speech request is performed.

If the ASR confidence score of the ASR confidence score calculation module (143) exceeds 0.3 and the index (Cs) is 3, the re-speaking request (O_2) can be output as true (T) regardless of the true/false (Ns) of the NLU processing model (142). If a reliable answer is obtained as a result of the voice recognition, even if the answer is not appropriate, it is considered that the speech was made as the user intended and the re-speaking request is not processed.

Accordingly, as shown in Table 710 of Fig. 7, the output values of the NLU processing model (142) and the ASR confidence score calculation module (143) can be combined to determine whether a user re-requests a request.

As one embodiment, at least one processor (140) may be configured to proceed with a re-fire request scenario via a user interface if the re-fire request is output as false.

FIG. 8 is a flow chart illustrating a speech recognition method based on utterance termination and utterance certainty according to one embodiment of the present disclosure.

The speech recognition method based on utterance termination and utterance certainty according to the present disclosure can be performed by a computing device including a user interface-based speech recognition module, a memory (130), and at least one processor (140) that performs a user utterance termination detection operation.

As illustrated in FIG. 8, the method may include a step (S810) of outputting whether or not utterance has ended for input user voice data using the VAD module (141) and the NLU processing module (142), and a step (S820) of receiving the input of whether or not utterance has ended and outputting whether or not a re-utterance request is made using the NLU processing module (142) and the ASR confidence score calculation module (143).

By combining the output values of the VAD module (141), the NLU processing module (142), and the ASR confidence score calculation module (143), it is possible to obtain a voice recognition result based on the termination of utterance and the certainty of utterance by outputting whether the utterance has ended and whether a request for re-utterance is required for the user's utterance. The content that overlaps with the above-described content will be omitted for the sake of brevity of the specification.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing instructions executable by a computer. The instructions may be stored in the form of program codes, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, and optical data storage devices.

As described above, the disclosed embodiments have been described with reference to the attached drawings. Those skilled in the art to which the present disclosure pertains will understand that the present disclosure can be implemented in forms other than the disclosed embodiments without changing the technical idea or essential features of the present disclosure. The disclosed embodiments are exemplary and should not be construed as limiting.

100: Electronic devices
110: Input/output module
120: Communication Module
130: Memory
140: Processor
141: VAD module
142: NLU Processing Module
143: ASR Confidence Score Calculation Module
144: Endpoint Determination Module
1411: VAD segment
1412: Activate section end point

Claims (10)

delete A voice recognition module that receives user voice data based on a user interface;
A memory storing at least one process for performing a user utterance termination detection operation; and
At least one processor for performing the user speech termination detection operation according to the above process;
At least one processor of the above,
Using the VAD (Voice Activity Detection) module and the NLU (Natural Language Understanding) processing module, it outputs whether the user's speech has ended for the input user voice data.
Using the above NLU processing module and the ASR confidence score (Automatic Speech Recognition Confidence Score) calculation module, the input whether the utterance has ended or not is used and the output whether a re-utterance request has been made or not is output.
An electronic device that performs speech recognition based on utterance termination and utterance certainty, wherein the VAD module is configured to output a de-activation time from an end point of an activation section of the user voice data and output different indices according to a section including the output de-activation time. In the second paragraph,
The above section includes the first section, the second section and the third section,
An electronic device that performs speech recognition based on utterance termination and utterance certainty, characterized in that the index increases in the order of the first section, the second section, and the third section. In the third paragraph,
At least one processor of the above,
An electronic device that performs speech recognition based on utterance end and utterance certainty, wherein the NLU processing model is configured to determine the truth or falsity of a current utterance sentence using a BERT-based encoder or a GPT-based decoder based on a previous utterance sentence of the user speech data. In the fourth paragraph,
At least one processor of the above,
In the above first section, regardless of whether the NLU processing model is true or false, the output is false whether the utterance has ended.
In the second section above, whether the speech is ended or not is output based on the truth or falseness of the NLU processing model.
An electronic device that performs speech recognition based on utterance termination and utterance certainty, wherein the third section is configured to output whether the utterance has ended as true regardless of the truth or falsity of the NLU processing model. In clause 5,
At least one processor of the above,
An electronic device that performs speech recognition based on utterance termination and utterance certainty, wherein the ASR confidence score calculation module is configured to output an ASR confidence score for a sentence in which the utterance has been terminated if the above utterance termination is output as true. In Article 6,
The above ASR reliability score includes the first confidence interval, the second confidence interval, and the third confidence interval.
An electronic device that performs speech recognition based on utterance termination and utterance certainty, characterized in that the ASR confidence score increases in the order of the first confidence interval, the second confidence interval, and the third confidence interval. In Article 7,
At least one processor of the above,
In the above first confidence interval, regardless of whether the NLU processing model is true or false, the output is false whether a re-trigger request is made.
In the second confidence interval above, whether a re-request is made is output depending on whether the NLU processing model is true or false.
An electronic device that performs speech recognition based on utterance termination and utterance certainty, wherein the third confidence interval is configured to output whether a re-utterance request is true regardless of the truth or falsity of the NLU processing model. In Article 8,
At least one processor of the above,
An electronic device that performs speech recognition based on utterance termination and utterance certainty, configured to proceed with a re-trigger request scenario through a user interface if the above re-trigger request is output as false. A speech recognition method based on utterance termination and utterance certainty, performed by a computing device including a user interface-based speech recognition module, a memory and at least one processor performing a user utterance termination detection operation, the method comprising:
A step of outputting whether or not speech has ended for input user voice data using a VAD (Voice Activity Detection) module and an NLU (Natural Language Understanding) processing module;
A step of receiving an input of whether the utterance has ended and outputting whether a re-utterance request has been made using the above NLU processing module and the ASR confidence score (Automatic Speech Recognition Confidence Score) calculation module; and
A speech recognition method based on utterance termination and utterance certainty, comprising: a step in which the VAD module outputs a de-activation time from an end point of an activation section of the user voice data, and outputs different indices according to a section including the output de-activation time.

Images