KR102768071B1 - Electronic apparatus for performing actual speaker indication on spoken text and processing method thereof - Google Patents

Abstract

An electronic device for performing an actual speaker display operation for a spoken text according to the present disclosure includes: an input/output module for receiving user input or providing output to the user based on a user interface; a memory storing at least one process for performing an actual speaker display operation for a spoken text; and at least one processor for performing the actual speaker display operation for the spoken text according to the process; wherein the at least one processor is configured to convert multi-speaker voice data into text to obtain a plurality of spoken texts in which arbitrary utterance identifiers are matched, provide a question for confirming an actual speaker for the plurality of spoken texts to a user interface in a questionnaire UI or UX in a CAPTCHA format, receive a label for the actual speaker through the user interface and change the arbitrary utterance identifier to the labeled actual speaker, and change the content and configuration of the questionnaire UI or UX based on feedback through the user interface.

Description

{ELECTRONIC APPARATUS FOR PERFORMING ACTUAL SPEAKER INDICATION ON SPOKEN TEXT AND PROCESSING METHOD THEREOF}

The present disclosure relates to an electronic device for performing classification of spoken text and a processing method thereof, and more specifically, to an electronic device for performing an actual speaker indication operation for spoken text and a processing method thereof.

The real-time STT (Speech to Text) model is a model that converts the speaker's speech into text in real time and provides it. In order to implement a real-time STT service, technology is required to accurately transcribe the speaker's speech into text, and additionally, technology is required to distinguish who the speaker of the transcribed speech data is.

However, the recognition rate for performing accurate matching of the speaker's speech is relatively low, and there is a technical limitation that it is difficult to identify the speaker, especially when extracting the speech of multiple speakers into text.

If the separated utterance data is expressed as a simple utterance identifier, a single utterance identifier exposed in the list may actually correspond to multiple speakers, and a problem may occur where even a single speaker is expressed as multiple utterance identifiers.

Therefore, if utterance data is expressed only with utterance identifiers, it may be difficult to visually distinguish who spoke, and readability may decrease.

In the case of conventional patent documents, entity names are extracted from speech texts and, if the extracted entity names contain errors, corrected entity names are labeled in the speech data. However, this makes it difficult to distinguish actual speakers when three or more speakers are speaking, and is physically difficult to operate in conjunction with various interfaces.

Republic of Korea Patent Publication No. 10-2610360 B1 (2023.12.01)

In order to solve the conventional problems, the embodiment disclosed in the present disclosure aims to provide an electronic device and a processing method thereof that performs an operation of indicating the actual speaker for a plurality of spoken texts by providing a question for confirming the actual speaker for the spoken texts in a questionnaire UI or UX in the form of a CAPTCHA on a user interface.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

According to the present disclosure for achieving the above-described technical problem, an electronic device for performing an actual speaker display operation for a spoken text includes: an input/output module for receiving a user input or providing an output to the user based on a user interface; a memory storing at least one process for performing an actual speaker display operation for a spoken text; and at least one processor for performing the actual speaker display operation for the spoken text according to the process; wherein the at least one processor is configured to convert multi-speaker voice data into text to obtain a plurality of spoken texts in which arbitrary utterance identifiers are matched, provide a question for confirming an actual speaker for the plurality of spoken texts to a user interface in the form of a questionnaire UI or UX in a CAPTCHA format, receive a label for the actual speaker through the user interface to change the arbitrary utterance identifier to the labeled actual speaker, and change the content and configuration of the questionnaire UI or UX based on feedback through the user interface.

In order to achieve the above-described technical problem, a method for processing actual speaker display for spoken text is provided, which is performed by a computing device including an input/output module based on a user interface, a memory, and at least one processor for performing an actual speaker display operation for spoken text according to the present disclosure, the method may include the steps of: converting multi-speaker voice data into text to obtain a plurality of spoken texts in which arbitrary utterance identifiers are matched; providing a question for confirming an actual speaker for the plurality of spoken texts in a questionnaire UI or UX in a CAPTCHA format on a user interface; receiving a label for the actual speaker through the user interface and changing the arbitrary utterance identifier to the labeled actual speaker; and changing the content and configuration of the questionnaire UI or UX based on feedback through the user interface.

In addition, a computer program stored in a computer-readable recording medium for implementing the present disclosure may be further provided.

In addition, a computer-readable recording medium recording a computer program for implementing the present disclosure may be further provided.

According to the above-described problem solving means of the present disclosure, a question for confirming the actual speaker of a plurality of utterance texts is provided in a CAPTCHA format questionnaire UI or UX on a user interface, thereby providing an interface where a user can easily match the utterance identifier with the actual speaker.

In addition, according to the aforementioned problem solving means of the present disclosure, readability is improved by distinguishing utterances through the names of actual speakers in the list, and data processing costs are reduced by allowing users to directly perform data labeling, and branding is strengthened through user feedback.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

FIG. 1 is a block diagram briefly illustrating the configuration of an electronic device for performing an actual speaker display operation for spoken text according to one embodiment of the present disclosure.
FIG. 2 is a block diagram briefly illustrating a process of an electronic device for performing an actual speaker indication operation for spoken text according to one embodiment of the present disclosure.
FIG. 3 is a screen diagram schematically illustrating a user interface for performing an actual speaker display operation for spoken text according to one embodiment of the present disclosure.
FIGS. 4 and 5 are schematic diagrams illustrating a user interface of an electronic device for performing an actual speaker display operation for spoken text according to one embodiment of the present disclosure.
FIG. 6 is a block diagram schematically illustrating a feedback process of an electronic device for performing an actual speaker indication operation for spoken text according to one embodiment of the present disclosure.
FIG. 7 is a graph showing a cluster of embedding vectors of an electronic device for performing an actual speaker indication operation for spoken text according to one embodiment of the present disclosure.
FIG. 8 is a flow chart illustrating a processing method of an electronic device for performing an actual speaker display operation for spoken text according to one embodiment of the present disclosure.

Throughout this disclosure, the same reference numerals refer to the same components. This disclosure does not describe all elements of the embodiments, and any content that is general in the technical field to which this disclosure belongs or that overlaps between the embodiments is omitted. The terms ‘part, module, element, block’ used in the specification can be implemented in software or hardware, and according to the embodiments, a plurality of ‘parts, modules, elements, blocks’ can be implemented as a single component, or a single ‘part, module, element, block’ can include a plurality of components.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only a direct connection but also an indirect connection, and an indirect connection includes a connection via a wireless communications network.

Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

Throughout the specification, when it is said that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where there is another element between the two elements.

The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

Singular expressions include plural expressions unless the context clearly indicates otherwise.

The identification codes in each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

In this specification, the 'device according to the present disclosure' includes all of various devices that can perform computational processing and provide results to a user. For example, the device according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may be in the form of any one of them.

Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

The above portable terminal may include, for example, all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted-device (HMD).

The function related to artificial intelligence according to the present disclosure is operated through a processor and a memory. The processor may be composed of one or more processors. At this time, one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control to process input data according to a predefined operation rule or artificial intelligence model stored in a memory. Alternatively, when one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

The function related to artificial intelligence according to the present disclosure is operated through a processor and a memory. The processor may be composed of one or more processors. At this time, one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU, a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control to process input data according to a predefined operation rule or artificial intelligence model stored in a memory. Alternatively, when one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned by using a plurality of learning data by a learning algorithm, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). Such learning may be performed in the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and/or system. Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs a neural network operation through an operation between the operation result of the previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, the plurality of weights may be updated so that a loss value or a cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network.

According to an exemplary embodiment of the present disclosure, a processor can implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that imitates human biological neurons to enable a machine to learn. Artificial intelligence methodologies can be divided into supervised learning in which input data and output data are provided together as training data according to a learning method, so that an answer (output data) to a problem (input data) is determined, unsupervised learning in which only input data is provided without output data, so that an answer (output data) to a problem (input data) is not determined, and reinforcement learning in which a reward is given from an external environment whenever an action is taken in a current state, and learning is performed in a direction to maximize this reward. In addition, artificial intelligence methodologies can be categorized by architecture, which is the structure of the learning model. The architectures of widely used deep learning technologies can be categorized into convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, and generative adversarial networks (GANs).

The present device and system may include an artificial intelligence model. The artificial intelligence model may be one artificial intelligence model or may be implemented as multiple artificial intelligence models. The artificial intelligence model may be composed of a neural network (or an artificial neural network) and may include a statistical learning algorithm that mimics biological neurons in machine learning and cognitive science. A neural network may refer to a model in which artificial neurons (nodes) that form a network by combining synapses change the strength of the synapses through learning and have problem-solving capabilities. Neurons of a neural network may include a combination of weights or biases. A neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. A neural network that constitutes the device may infer a desired result (output) from an arbitrary input (input) by changing the weights of neurons through learning.

The processor can generate a neural network, train (or learn) a neural network, perform a calculation based on received input data, generate an information signal based on the result of the calculation, or retrain the neural network. The models of the neural network can include various types of models such as CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, etc., but are not limited thereto. The processor can include one or more processors for performing calculations according to the models of the neural network. For example, the neural network can include a deep It may include a neural network (Deep Neural Network).

Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), Generative Adversarial Network (GAN), Liquid State Machine (LSM), Extreme Learning Machine (ELM), It will be understood by those skilled in the art that any neural network may be included, including but not limited to an ESN (Echo State Network), a DRN (Deep Residual Network), a DNC (Differentiable Neural Computer), an NTM (Neural Turning Machine), a CN (Capsule Network), a KN (Kohonen Network), and an AN (Attention Network).

According to an exemplary embodiment of the present disclosure, the processor may be configured to perform a CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, R-CNN (Region with Convolution Neural Network), RPN (Region Proposal Network), RNN (Recurrent Neural Network), S-DNN (Stacking-based deep Neural Network), S-SDNN (State-Space Dynamic Neural Network), Deconvolution Network, DBN (Deep Belief Network), RBM (Restrcted Boltzman Machine), Fully Convolutional Network, LSTM (Long Short-Term Memory) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC/QA for natural language processing, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, ResNet for data intelligence, Anomaly Detection, Prediction, Time-Series Forecasting, Various artificial intelligence structures and algorithms such as Optimization, Recommendation, and Data Creation can be used, but are not limited thereto. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

FIG. 1 is a block diagram briefly illustrating the configuration of an electronic device (100) for performing an actual speaker display operation for spoken text according to one embodiment of the present disclosure.

Referring to FIG. 1, an electronic device (100) according to the present disclosure may include an input/output module (110), a communication module (120), a memory (130), and a processor (140). Hereinafter, the electronic device (100) is an electronic device that performs an actual speaker display operation for a spoken text, and is assumed to be implemented through an electronic device (100) that performs an actual speaker display operation for a spoken text.

The input/output module (110) may be various interfaces or connection ports that receive user input or output information to the user. The input/output module (110) may be divided into an input module and an output module.

The input module receives user input from a user. The input module is for inputting image information (or signal), audio information (or signal), data, or information input from a user, and may include at least one camera, at least one microphone, and at least one user input unit. Voice data or image data collected from the input unit may be analyzed and processed as a user control command.

User input can take various forms, including key input, touch input, and voice input. Examples of input modules that can receive such user input include, in addition to traditional keypads, keyboards, and mice, touch sensors that detect the user's touch, microphones that receive voice signals, cameras that recognize gestures through image recognition, proximity sensors such as light sensors or infrared sensors that detect the approach of a user, motion sensors that recognize user movements through acceleration sensors or gyro sensors, and various other input means that detect or receive various forms of user input.

Here, the touch sensor may be implemented as a piezoelectric or electrostatic touch sensor that detects touch through a touch panel or touch film attached to the display panel, an optical touch sensor that detects touch in an optical manner, etc. In addition, the input module may be implemented in the form of an input interface (such as a USB port or PS/2 port) that connects an external input device that receives user input instead of a device that detects user input on its own.

The output module can output various information and provide it to the user. The output module is a comprehensive concept that includes a display that outputs images, a speaker (and/or an amplifier connected thereto) that outputs sounds, a haptic device that generates vibrations, and various other forms of output means. In addition, the output module can also be implemented in the form of a port-type output interface that connects the individual output means described above.

For example, an output module in the form of a display can display text, still images, and moving images. The display is a concept that broadly refers to an image display device including various forms of devices that can perform image output functions, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flat panel display (FPD), a transparent display, a curved display, a flexible display, a 3D display, a holographic display, a projector, and others. Such a display may be in the form of a touch display configured integrally with a touch sensor of the input module.

In other words, the input/output module (110) can receive user input or provide output to the user based on a user interface.

The communication module (120) can communicate with an external device. Therefore, the device can transmit and receive information with the external device through the communication module. For example, the device can communicate with the external device so that information stored and generated within the electric vehicle charging management system is shared using the communication module. The communication module (120) can include, for example, at least one of a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

Here, communication, that is, data transmission and reception, can be performed wired or wirelessly. To this end, the communication module may be configured as a wired communication module that connects to the Internet, etc. via a LAN (Local Area Network), a mobile communication module that connects to a mobile communication network via a mobile communication base station to transmit and receive data, a short-distance communication module that uses a WLAN (Wireless Local Area Network) series communication method such as Wi-Fi or a WPAN (Wireless Personal Area Network) series communication method such as Bluetooth or Zigbee, a satellite communication module that uses a GNSS (Global Navigation Satellite System) such as a GPS (Global Positioning System), or a combination thereof. The wireless communication technology used for communication may include NB-IoT (Narrowband Internet of Things) for low-power communication. At this time, for example, NB-IoT technology may be an example of LPWAN (Low Power Wide Area Network) technology, and may be implemented with standards such as LTE Cat (category) NB1 and/or LTE Cat NB2, and is not limited to the names described above. Additionally or alternatively, a wireless communication technology implemented in a wireless device according to various embodiments may perform communication based on LTE-M technology. At this time, for example, LTE-M technology may be an example of LPWAN technology, and may be called by various names such as eMTC (enhanced Machine Type Communication). For example, LTE-M technology may be implemented with at least one of various standards such as 1) LTE CAT 0, 2) LTE Cat M1, 3) LTE Cat M2, 4) LTE non-BL (non-Bandwidth Limited), 5) LTE-MTC, 6) LTE Machine Type Communication, and/or 7) LTE M, and is not limited to the names described above. Additionally or alternatively, the wireless communication technology implemented in the wireless device according to various embodiments may include at least one of ZigBee, Bluetooth, and Low Power Wide Area Network (LPWAN) considering low-power communication, and is not limited to the above-described names. For example, ZigBee technology can create PAN (personal area networks) related to small/low-power digital communication based on various standards such as IEEE 802.15.4, and may be called by various names.

The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard232), power line communication, or plain old telephone service (POTS).

The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a WiFi module and a Wireless broadband module.

The wireless communication module may include a wireless communication interface including an antenna and a transmitter for transmitting a signal. In addition, the wireless communication module may further include a signal conversion module for modulating a digital control signal output from the control unit through the wireless communication interface into an analog wireless signal according to the control of the control unit.

The wireless communication module may include a wireless communication interface including an antenna and a receiver for receiving a signal. In addition, the wireless communication module may further include a signal conversion module for demodulating an analog wireless signal received through the wireless communication interface into a digital control signal.

The short-range communication module is for short-range communication and can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

The location information module is a module for obtaining the location (or current location) of the electronic device (10) according to the present disclosure, and representative examples thereof include a GPS (Global Positioning System) module or a WiFi (Wireless Fidelity) module. For example, if a GPS module is utilized, the location of the electronic device can be obtained by using a signal sent from a GPS satellite. As another example, if a Wi-Fi module is utilized, the location of the electronic device (10) can be obtained based on information of a wireless AP (Wireless Access Point) that transmits or receives a wireless signal with the Wi-Fi module. If necessary, the location information module may perform any function of other modules of the communication module to obtain data regarding the location of the device as a substitute or in addition. The location information module is a module used to obtain the location (or current location) of the device, and is not limited to a module that directly calculates or obtains the location of the device.

The memory (130) can store various types of information. The memory can store data temporarily or semi-permanently. For example, the memory can store an operating program (OS: Operating System) for operating the first device and/or the second device, data for hosting a website, a program for generating Braille, or data regarding an application (e.g., a web application). In addition, the memory can store modules in the form of computer codes as described above.

Examples of memory (130) may include a hard disk drive (HDD), a solid state drive (SSD), flash memory, read-only memory (ROM), random access memory (RAM), etc. Such memory may be provided as a built-in type or a removable type.

The processor (140) controls the overall operation of the electronic device (100). To this end, the processor (140) can perform calculations and processing of various types of information and control the operation of components of the first device and/or the second device.

The processor (140) may be implemented as a computer or a similar device according to hardware, software, or a combination thereof. In terms of hardware, the processor (140) may be provided in the form of an electronic circuit that processes electrical signals to perform a control function, and in terms of software, the processor may be provided in the form of a program that drives a hardware processor. Meanwhile, unless otherwise specifically mentioned in the following description, the operations of the first device and/or the second device may be interpreted as being performed under the control of the processor (140). That is, the modules may be interpreted as controlling the processor (140) to perform the following operations on the first device and/or the second device.

The processor (140) may be implemented as a memory storing data for an algorithm for controlling the operation of components within the device or a program reproducing the algorithm, and at least one subprocessor (not shown) performing the above-described operation using the data stored in the memory. In this case, the memory and the processor may be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

In addition, the processor (140) may control one or more of the components described above in combination to implement various embodiments according to the present disclosure described in FIGS. 2 to 8 below on the device.

An electronic device (100) for performing an actual speaker display operation for a spoken text according to the present disclosure may include an input/output module (110) for receiving a user input or providing an output to a user based on a user interface, a memory (130) in which at least one process for performing an actual speaker display operation for a spoken text is stored, and at least one processor (140) for performing an actual speaker display operation for the spoken text according to the process.

As one embodiment, at least one processor (140) may be configured to convert multi-speaker voice data into text to obtain a plurality of utterance texts in which arbitrary utterance identifiers are matched, provide a question for confirming an actual speaker for the plurality of utterance texts to a user interface in the form of a questionnaire UI or UX in a CAPTCHA format, receive a label for the actual speaker through the user interface to change the arbitrary utterance identifier to the labeled actual speaker, and change the content and configuration of the questionnaire UI or UX based on feedback through the user interface.

As illustrated in FIG. 2, an electronic device (100) according to one embodiment of the present disclosure can perform an actual speaker display operation for spoken text by performing a UI/UX providing step (S210), an input step (S220), a feedback step (S230), and a UI/UX re-provisioning step (S240).

In the UI/UX provision step (S210), at least one processor (140) can output UI/UX to the user interface so that a user can easily match a speech identifier with an actual speaker.

Provides a UI configuration that can query extracted speech text before providing UI/UX, and can provide, for example, a chat room that displays speech delimiters, speech content, speech duration, and audio of the speech duration. In the chat room, speech delimiters can be displayed for each speech content, and audio text can be provided to be played back separately for each speech content.

When entering a chat room, a survey UI/UX in the form of CAPTCHA can be displayed to guide users to match who each speaker is when entering the chat room, and the readability of the speech data can be improved.

The above captcha-type survey UI/UX can be implemented in various interfaces according to various question types and provided on the user interface. The specific survey UI/UX will be described later.

In the input stage (S220), an answer can be provided to the questionnaire UI/UX in the form of a captcha output to the user interface through the input/output module (110).

Through the input/output module (110), you can input whether the answer is correct or not, such as O/X, in the questionnaire UI/UX in the form of a captcha displayed on the user interface, or input an answer corresponding to the question type, such as the number of speakers included in the audio, or matching the actual speakers with the audio.

In the feedback stage (S230), the content or configuration of the survey UI/UX in the form of a captcha to be displayed on the user interface or the question type can be adjusted based on the operation in the user interface-based input/output module (110).

As one embodiment, at least one processor (140) may adjust an appearance weight downward to lower the appearance probability of a question measured in the user interface by considering at least one of a dwell time, a non-response rate, and an incorrect response rate of the user interface according to the type of question that verifies the actual speaker.

For example, if the residence time required to input a response to a questionnaire UI/UX in the input/output module (110) exceeds a preset threshold, a downward weight can be applied to lower the probability of appearance of the questionnaire UI/UX content or configuration, or question type.

Alternatively, if the non-response rate, which means entering the chat room directly without entering a response to the survey UI/UX in the input/output module (110), exceeds a preset threshold, a downward weight can be applied to lower the probability of the survey UI/UX content or structure, or question type appearing.

Alternatively, if an answer to a questionnaire UI/UX is input into the input/output module (110), but is judged to be an incorrect answer when compared to answers to other questionnaire UI/UX, or an incorrect answer is input in a re-examination of a pre-matched voice, and the incorrect answer rate exceeds a preset threshold, a downward weight can be applied to lower the probability of the questionnaire UI/UX content or structure, or question type appearing.

Therefore, based on the length of stay, non-response rate, and incorrect response rate for the survey UI/UX, the content or structure of the survey UI/UX, and the question types provided in the survey UI/UX can be judged to be difficult or impossible to answer, and the output probability can be lowered to keep the question difficulty level light. This is to encourage and guide user participation.

As an example, at least one processor (140) may collect audio fragments with a high error rate as re-learning data. In order to utilize the audio fragments with a high error rate as error data, the audio fragments with a high error rate may be configured as a learning data set so as to reflect the fact that the recognition rate is low in the case of a speaker with a specific speaking habit or in a poor environment such as noise.

In the feedback step (S230), at least one processor (140) can adjust the number of times a question is extracted from the UI/UX questionnaire depending on the reliability of the voice.

If the inverse of the distribution entropy within the embedding cluster containing the pre-matched real speaker's voice embedding vector is greater than a preset threshold, the weight from which the audio fragment of the real speaker is extracted can be adjusted downward.

By using an embedding vector that represents a voice as an n-dimensional vector, we can represent the voice as an n-dimensional vector and distinguish clusters by distinguishing adjacent vectors from distant vectors.

Whenever a new voice is input, it can be determined which cluster it belongs to, and for clusters where vectors are placed adjacently and densely packed, a high reliability can be calculated based on the inverse of the distribution entropy.

On the other hand, clusters where vectors are not adjacent and are sparsely distributed can produce low reliability according to the inverse of the distribution entropy.

Therefore, there is no need to perform feedback when extracting multiple embedding vectors of the speaker and calculating the distance between the embedding vectors, and if it is output as a highly reliable voice, there is no need to receive input for the voice in the survey UI/UX. Therefore, a downward weight can be given to the probability of displaying the survey UI/UX for the pre-matched voice in the user interface.

In cases where feedback is required when a voice is produced with low reliability, an upward weight can be given to the probability of displaying the survey UI/UX for the pre-matched voice in the user interface to receive additional input for the voice in the survey UI/UX.

In the UI/UX re-provision step (S240), the survey UI/UX output can be adjusted and provided to the user interface according to the weight adjustment in the feedback step (S230).

This allows us to encourage user participation by maintaining the existing UI/UX while keeping the difficulty of users' responses low.

Referring to FIGS. 3 to 5, one can check the UI/UX output to the user interface of an electronic device (100) according to one embodiment of the present disclosure.

As an example, the system includes a plurality of utterance text utterance delimiters (210), utterance content (220), a sustained utterance segment, and audio (230) of the sustained utterance segment, wherein the at least one processor (140) can randomly extract at least one audio fragment matching the type of the question from the audio of the sustained utterance segment and trim it to within a preset time interval.

Referring to FIG. 3, arbitrary indicators of participant 1, participant 2, and participant 3 can be assigned as utterance identifiers (210). The utterance content (220) of each utterance identifier (210) can be displayed under the utterance identifier (210), and an interface for playing audio (230) of a sustained utterance section corresponding to each utterance content (220) can be additionally arranged.

Here, attendee 1, attendee 2, and attendee 3 are extracted, but this is an exemplary embodiment. If there are 5 or more attendees, only 3 speakers can be randomly extracted and audio (230) of the continuous speaking section can be played to select the voice information of 'me'.

In other words, at least one processor (140) can play back the trimmed plurality of audio fragments and output a user interface for selecting an audio fragment corresponding to one's own voice.

As another embodiment, the at least one processor (140) may play back the trimmed audio fragment and output a user interface for inputting the corresponding actual speaker name.

For example, it is possible to provide an input interface that reproduces only one audio fragment (230) of a continuous speech segment or a trimmed audio fragment of the audio fragment (230) and receives the actual speaker of the reproduced audio fragment.

As another embodiment, the at least one processor (140) may play a plurality of audio fragments included in the same voice clustering as the stored own voice and output a user interface for receiving input on whether each audio fragment is correct or not.

For example, if there is a voice fragment that matches my name of a pre-registered user, an audio fragment with a high similarity to the voice that matches my name registered in advance can be extracted through voice clustering, played, and a user interface can be provided that receives an input of O or X for each audio fragment to determine whether it corresponds to my voice.

In other words, the question type of the survey UI/UX can be general matching by selecting the speaker in the form of whether it is someone's voice, selecting my voice, and confirming that it is my voice.

At this time, as an example, at least one processor (140) may randomly extract and provide audio fragments belonging to a chat room to which the user interface is connected. Rather than displaying a voice of a completely unknown third party on the user interface, the audio fragments may be provided so as to match the speakers of the audio fragments provided from multiple chat rooms in which the user has participated.

Meanwhile, as an example, at least one processor (140) may re-examine information entered in a question type of another survey UI/UX or re-examine pre-matched audio fragments to verify that they are from an actual speaker.

As one embodiment, at least one processor (140) may output a user interface that displays audio fragments within a preset distance from a pre-matched voice embedding vector of an actual speaker as a predicted actual speaker and receives an input for whether the result is correct or not.

Referring to FIG. 4, a pre-matched voice embedding vector (310) can be output and an audio fragment of a voice similar to the voice embedding vector (310) can be output. In addition, an answer interface (320) for inputting O or X can be placed side by side with each audio fragment of the similar voice so that it can be input whether each audio fragment matches the voice embedding vector (310).

At this time, in Fig. 4, it is indicated as my voice embedding vector (310), but a name previously matched with my voice embedding vector (310) may also be output as an expected name.

Alternatively, as another embodiment, at least one processor (140) may randomly select a plurality of pre-matched actual speakers and voices, shuffle them, list them, and output a user interface that matches corresponding pairs.

Rather than directly matching audio fragments of voices similar to the expected names of actual speakers, a UI can be provided that mixes audio fragments of expected names and voices of multiple actual speakers, lists them side by side, and finds corresponding pairs. The UI can be implemented in various forms, such as an interface where pairs are matched according to the order of selection, an interface where lines are drawn directly, or an interface where matching pairs are displayed separately by clicking.

For example, you could randomly select three audio fragments for voices under the condition that the names do not overlap, to reexamine the previously matched names and voices. You could list the names and audio fragments side by side, shuffling them, and provide an interface to find matching pairs.

Meanwhile, as an embodiment, at least one processor (140) may provide a user interface for checking the level of separation of utterance data, such as whether the number of utterance delimiters is the same as the number of actual speakers, and whether utterance content with different utterance delimiters mapped to different actual speakers is different.

As one embodiment, at least one processor (140) may extract the top audio fragment with the highest number of overlaps by an overlapping utterance detector (OSD) or extract the top audio fragment with the highest non-response rate or incorrect response rate in another question type and output a user interface for inputting the number of people appearing in the audio fragment.

For example, referring to Fig. 5, the question window asking, 'How many speakers appeared in the corresponding sound?' is an actual speaker selection interface (410), and the window answering 'N people' is a correct answer interface (420).

There are cases where overlapping and duplicated voices are output, and at least one processor (140) can extract the audio fragment with the highest number of overlapping or the highest upper audio fragment through an overlapping speech detector (OSD) and play the audio fragment in the actual speaker selection interface (410).

Alternatively, in question types of other survey UI/UX, audio fragments with high error rates or non-response rates may be considered as situations where noise is present, overlapping, duplicated voices are heard, or voices are similar, and the audio fragment with the highest error rate or non-response rate or the highest top audio fragment may be extracted and played in the actual speaker selection interface (410).

The number of actual speakers in the audio fragment played in the correct answer interface (420) can be entered.

As one embodiment, at least one processor (140) may randomly extract multiple audio fragments, each of which is assigned a different speaker identifier and the same actual speaker, concatenate them, and output a user interface for receiving an answer as to whether the concatenated audio fragments are the voice of one person.

Referring to Figure 5, the question window asking, “Is this a person’s voice?” is an actual speaker selection interface (430), and the window answering “O/X” is a correct answer interface (440).

In the spoken text, three audio fragments separated by different speaker identifiers and assigned the same actual speaker name can be randomly selected through the user interface. In other words, audio fragments separated by different speakers but assigned the same name can be extracted and concatenated to play the audio in the actual speaker selection interface (430).

Then, in the correct answer interface (440), if one person's voice is heard, you can input O, and if multiple people's voices are heard, you can input X.

The readability of utterance data can be improved by replacing randomly separated utterance identifiers with actual speakers through the survey UI or UX in the CAPTCHA format described above.

You can go through an input stage where you get actual speakers to input the survey UI or UX in the form of a captcha, and provide feedback to adjust the survey UI or UX in the form of a captcha.

As shown in Fig. 6, user responses to question types of the survey UI or UX can be collected based on the screen dwell time (51), question non-response rate (52), and question incorrect response rate (53) within the user interface.

Questions with low user response, in other words, questions in the survey UI or UX that have a long screen dwell time (51), a high question non-response rate (52), and a high question mis-response rate (53), can be subject to question appearance weight downward adjustment (510). This allows users to answer more quickly, and thus, survey UI or UX question types can be provided that have a short screen dwell time (51), a low question non-response rate (52), and a low question mis-response rate (53), thereby inducing more user participation.

Additionally, feedback can be performed to adjust the probability of extracting audio fragments based on the confidence of the embedding vector.

As shown in Fig. 7, multiple embedding vectors for actual speakers can be displayed on a graph. At this time, there may be clusters of embedding vectors that are sparsely and widely arranged, such as cluster (71), and clusters of embedding vectors that are densely arranged, such as cluster (72). The density can be calculated by calculating the inverse of the distribution entropy for cluster (71) and cluster (72), and the calculated density can be regarded as the reliability of the cluster.

Therefore, we can extract the embedding information of the audio fragment and sort them in order of proximity by calculating the distance from the embedding cluster information.

The cluster list is randomly searched, and when a cluster with a reliability level higher than a preset value appears, the audio fragment can be matched with the cluster and the audio fragment can be pre-matched.

For high-confidence audio fragments, no additional actions are required, and thus, downward weighting of the audio fragment extraction probability can be applied, while for low-confidence audio fragments, additional feedback actions are required, and thus, upward weighting of the audio fragment extraction probability can be applied.

Accordingly, the electronic device (100) according to one embodiment of the present disclosure can provide an interface in which a user can easily match an utterance identifier with an actual speaker by providing a question for confirming the actual speaker of a plurality of utterance texts in a questionnaire UI or UX in the form of a CAPTCHA on a user interface.

Additionally, readability can be improved by distinguishing utterances through the actual speaker's name in the list, data processing costs can be reduced by allowing users to directly label data, and branding can be strengthened through user feedback.

Meanwhile, FIG. 8 is a flow chart illustrating a processing method of an electronic device for performing an actual speaker display operation for spoken text according to one embodiment of the present disclosure.

As illustrated in FIG. 8, the processing method may include a step of converting multi-speaker voice data into text (S810), a step of providing a survey UI/UX in a CAPTCHA format (S820), a step of receiving labeling input for each speaker through a user interface (S830), and a step of changing the content and composition of the survey UI/UX based on feedback through the user interface (S840).

In the step of converting multi-talker voice data into text (S810), the multi-talker voice data can be converted into text to obtain multiple utterance texts in which arbitrary utterance delimiters are matched.

In the step (S820) of providing a survey UI/UX in CAPTCHA format, a question for confirming the actual speaker of the plurality of spoken texts can be provided on the user interface as a survey UI or UX in CAPTCHA format.

In the step (S830) of receiving a label input through a user interface for each speaker, a label for the actual speaker can be input through the user interface to change the arbitrary utterance identifier to the labeled actual speaker.

In the step (S840) of changing the content and configuration of the survey UI/UX based on feedback through the user interface, the question type of the survey UI/UX can be adjusted and randomly extracted audio fragments can be adjusted based on the feedback.

Any content that overlaps with the above will be omitted for the sake of brevity of the specification.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing instructions executable by a computer. The instructions may be stored in the form of program codes, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, and optical data storage devices.

As described above, the disclosed embodiments have been described with reference to the attached drawings. Those skilled in the art to which the present disclosure pertains will understand that the present disclosure can be implemented in forms other than the disclosed embodiments without changing the technical idea or essential features of the present disclosure. The disclosed embodiments are exemplary and should not be construed as limiting.

100: Electronic devices
110: Input/output module
120: Communication Module
130: Memory
140: Processor
210: Speech delimiter
220: Content of speech
230: Audio Player
240: Selection Interface
310: My voice embedding vector
320: Correct interface
410, 430: Actual speaker selection interface
420, 440: Correct interface

Claims (10)

Input/output modules that receive user input or provide output to the user based on the user interface;
A memory storing at least one process for performing an actual speaker display operation for the spoken text; and
At least one processor for performing an actual speaker display operation for the spoken text according to the above process;
At least one processor of the above,
Converting multi-speaker speech data into text to obtain multiple utterance texts with arbitrary utterance delimiters matched,
Provide a question to confirm the actual speaker of the above multiple utterance texts in the form of a CAPTCHA questionnaire UI or UX on the user interface,
Inputting a label for said actual speaker through said user interface and changing said arbitrary utterance identifier to said labeled actual speaker,
It is configured to change the content and composition of the survey UI or UX based on feedback through the above user interface.
Adjusting the appearance weight downwards to lower the appearance probability of the question measured in the user interface by considering at least one of the dwell time, non-response rate, and incorrect response rate of the user interface according to the type of question confirming the actual speaker,
The audio fragments with the high error rate are configured to be collected as retraining data.
An electronic device for performing actual speaker identification actions on spoken text. delete In the first paragraph,
The above multiple utterance texts include utterance delimiters, utterance content, utterance duration segments, and audio of the utterance duration segments,
At least one processor of the above,
Configured to randomly extract at least one audio fragment matching the type of the question from the audio of the above-mentioned continuous speech section and trim it to within a preset time interval.
An electronic device for performing actual speaker identification actions on spoken text. In the third paragraph,
At least one processor of the above,
Output a user interface that plays a single trimmed audio fragment and prompts for the corresponding actual speaker name;
Outputting a user interface for playing back the trimmed plurality of said audio fragments and selecting the audio fragment corresponding to one's own voice, and
A user interface configured to perform at least one of the following: playing a plurality of audio fragments included in the same voice clustering as the stored own voice, and receiving an input as to whether each audio fragment is correct or not;
An electronic device for performing actual speaker identification actions on spoken text. In the fourth paragraph,
At least one processor of the above,
configured to randomly extract and provide audio fragments belonging to a chat room to which the above user interface is connected;
An electronic device for performing actual speaker identification actions on spoken text. In the third paragraph,
At least one processor of the above,
It is configured to output a user interface that displays audio fragments within a preset distance from the pre-matched real speaker's voice embedding vector as if they were the predicted real speaker and asks for input whether the answer is correct.
An electronic device for performing actual speaker identification actions on spoken text. In the third paragraph,
At least one processor of the above,
It is configured to output a user interface that randomly selects multiple pre-matched real speakers and voices, mixes them, lists them, and matches the corresponding pairs.
An electronic device for performing actual speaker identification actions on spoken text. In the third paragraph,
At least one processor of the above,
Extract the top audio fragment with the highest number of overlaps by an overlapping utterance detector (OSD), or extract the top audio fragment with the highest non-response or incorrect response rate in other question types, and output a user interface that inputs the number of people appearing in the audio fragment, or
The above speech discriminator is configured to randomly extract multiple audio fragments designated as different and identical actual speakers, concatenate them, and output a user interface that inputs a correct answer as to whether the concatenated audio fragments are the voice of one person.
An electronic device for performing actual speaker identification actions on spoken text. In Article 8,
At least one processor of the above,
configured to adjust downward the weight from which audio fragments of the real speaker are extracted if the inverse of the distribution entropy within the embedding cluster containing the pre-matched real speaker's voice embedding vector is greater than a preset threshold.
An electronic device for performing actual speaker identification actions on spoken text. A method for processing actual speaker display for spoken text, performed by a computing device including a user interface-based input/output module, a memory, and at least one processor performing actual speaker display operations for spoken text, the method comprising:
A step of converting multi-speaker speech data into text to obtain multiple utterance texts with matched arbitrary utterance delimiters;
A step of providing a question for verifying the actual speaker of the above multiple utterance texts in the form of a CAPTCHA questionnaire UI or UX on a user interface;
A step of inputting a label for the actual speaker through the user interface and changing the arbitrary utterance identifier into the labeled actual speaker; and
A step of changing the content and composition of the survey UI or UX based on feedback through the above user interface; Including,
A step of downwardly adjusting the appearance weight to lower the appearance probability of the question measured in the user interface by considering at least one of the residence time, non-response rate, and incorrect response rate of the user interface according to the type of question confirming the actual speaker; further comprising;
A method for processing actual speaker indication for spoken text, further comprising: a step of collecting audio fragments with a high error rate as re-learning data;

Images