KR20250002082A - Method and apparatus for speech signal processing - Google Patents

Abstract

A method for processing a speech signal of a terminal is provided, including the steps of receiving a speech signal, detecting a personalized information section including personal information in the speech signal, transmitting the speech signal to a server, wherein the personal information included in the personalized information section is transformed into a form in which the content of words or sentences cannot be identified and transmitted, receiving a data processing result for the speech signal from the server, and reprocessing the speech signal using the data processing result received from the server, the personal information included in the personalized information section, and a personalized model which is a customized voice processing model for each user, thereby supplementing the data processing result received from the server, and providing the result of the data processing.

Description

{METHOD AND APPARATUS FOR SPEECH SIGNAL PROCESSING}

The present invention relates to a voice signal processing method and device, and more specifically, to a voice signal processing method and device capable of protecting personal information while utilizing a personalized model.

Speech recognition is a technology that converts a user's voice into text. Since this process is performed automatically, it is also called Automatic Speech Recognition (ASR). Recently, it has been widely spread as an interface technology to replace keyboard input in devices such as smartphones and TVs. Natural Language Understanding (NLU) is a technology that extracts the meaning of a user's speech from the recognition results of speech recognition. It does not simply recognize the user's voice, but performs a higher level of analysis of the user's voice to more accurately understand the meaning of the voice.

These speech recognition and language understanding systems can generally be divided into a client that inputs a speech signal and an ASR/NLU engine that performs speech recognition and language understanding from the speech signal. In order to increase the speed of speech signal processing, the two modules can be designed to be separated from each other. In this case, a device such as a smartphone or TV with limited processing and data storage capabilities can be configured as a client, and the speech recognition and language understanding engine can be configured as an independent server with high computational capabilities. The two modules are connected via a network. The device located close to the user performs the role of receiving a speech signal, and the server with a fast data processing speed performs the role of performing speech recognition and language understanding. Another configuration may include a configuration in which a speech recognition and language understanding engine is installed inside the device in addition to the server, so that the two speech recognition and language understanding engines cooperate with each other to perform speech recognition and language understanding.

One way to improve the performance of these speech recognition and language understanding systems is to collect data for each user and create a user-specific model. This user-specific model is called a personalized model, and this method is called personalized modeling. Since personalized models can create customized models for specific individuals, they generally have higher performance than general models created for an unspecified number of people.

However, when using personalized modeling, the user's personal information must be used to create a personalized model, and problems with information protection may arise during the transmission and processing of personal information. If encryption technology is applied to resolve this, the processing speed may slow down.

The disclosed embodiments provide a method and device for processing voice signals that can protect personal information while utilizing a personalized model.

Specifically, the disclosed embodiment provides a voice signal processing method and device that divides a personalized information section and a general information section into a terminal and a server for processing.

In addition, the disclosed embodiment provides a voice signal processing method and device in which a terminal reprocesses a voice signal processed by a server using a personalized model.

In addition, the disclosed embodiment provides a voice signal processing method and device using an ID-based personalized model.

A method for processing a speech signal of a terminal according to the disclosed embodiment includes: receiving a speech signal; detecting a personalized information section including personal information from the speech signal; performing data processing on a speech signal corresponding to the personalized information section from the speech signal using a personalized model generated based on the personal information; and receiving a result of data processing on a speech signal corresponding to a general information section other than the personalized information section from a server.

In addition, the method may further include a step of generating voice section information for the personalized information section and the general information section and transmitting the generated voice section information to the server.

In addition, the voice segment information may include segment marking information that marks at least one segment among the personalized information segment and the general information segment of the voice signal.

In addition, the method may further include a step of receiving voice section information for the personalized information section and the general information section from the server.

In addition, the result of data processing for the voice signal corresponding to the general information section received from the server may be the result of the server processing the voice signal corresponding to the general information section using a general model.

A voice signal processing method of a server according to the disclosed embodiment comprises the steps of: receiving a voice signal; detecting a personalized information section including personal information from the voice signal; performing data processing using a general model for a voice signal corresponding to a general information section other than the personalized information section from the voice signal; and transmitting a result of data processing for the voice signal corresponding to the general information section to a terminal.

In addition, the method may further include a step of generating voice section information for the personalized information section and the general information section and transmitting the same to the terminal.

In addition, the voice segment information may include segment marking information that marks at least one segment among the personalized information segment and the general information segment of the voice signal.

In addition, the method may further include a step of receiving voice section information for the personalized information section and the general information section from the terminal.

A method for processing a voice signal of a terminal according to another disclosed embodiment comprises the steps of: receiving a voice signal; receiving a result of data processing using a general model for the voice signal from a server; and processing data using a personalized model generated based on personal information and the result of the data processing for the voice signal.

In addition, the step of performing data processing using a personalized model generated based on the data processing result and personal information for the voice signal may include a step of performing data processing for a voice signal corresponding to a personalized information section including the personal information.

In addition, the method may further include a step of detecting the personalized information section from the voice signal.

In addition, the method may further include a step of receiving voice segment information for the personalized information segment and the general information segment from the server.

In addition, the voice segment information may include segment marking information that marks at least one segment among the personalized information segment and the general information segment of the voice signal.

Additionally, the personalized model may be at least one of a personalized speech recognition model, a personalized natural language understanding model, and a personalized lexical model.

A method for processing a voice signal of a terminal according to another disclosed embodiment includes: a step of mapping an ID to personal information to create a mapping table; a step of creating an ID-based personalized model using the mapping table; a step of transmitting the ID-based personalized model to a server; a step of receiving a result of data processing using the ID-based personalized model for a voice signal from the server; and a step of restoring the personal information corresponding to the ID using the result of the data processing and the mapping table.

In addition, the step of generating an ID-based personalized model using the mapping table may include a step characterized by including a step of representing an ID mapped to the personal information as an acoustic unit ID, which is an ID mapped to sound.

Additionally, the sound unit ID may be an ID mapped to the sound according to an agreement with the server.

In addition, the method may further include a step of creating a mapping table by mapping an ID to additional information generated from the personal information.

A voice signal processing method of a server according to another disclosed embodiment comprises the steps of: receiving an ID-based personalized model from a terminal; receiving a voice signal; performing data processing on the voice signal using the ID-based personalized model; and transmitting a result of the data processing to the terminal.

In addition, the step of processing data using the ID-based personalized model for the voice signal may include a step of indicating an ID mapped to personal information using an acoustic unit ID, which is an ID mapped to sound according to an agreement with the terminal.

A terminal according to the disclosed embodiment includes a receiving unit for receiving a voice; a communication unit for communicating with a server; and a control unit for receiving a voice signal, detecting a personalized information section including personal information from the voice signal, performing data processing on a voice signal corresponding to the personalized information section among the voice signals using a personalized model generated based on the personal information, and receiving a result of data processing on a voice signal corresponding to a general information section other than the personalized information section from the server.

In addition, the control unit can control to generate voice section information for the personalized information section and the general information section and transmit it to the server.

Additionally, the voice segment information may include segment marking information that marks at least one segment among the personalized information segment and the general information segment of the voice signal.

Additionally, the control unit can control to receive voice section information for the personalized information section and the general information section from the server.

In addition, the result of data processing for the voice signal corresponding to the general information section received from the server may be the result of the server processing the voice signal corresponding to the general information section using a general model.

A server according to the disclosed embodiment may include a receiving unit for receiving a voice; a communication unit for communicating with a terminal; and a control unit for receiving a voice signal, detecting a personalized information section including personal information from the voice signal, performing data processing on a voice signal corresponding to a general information section other than the personalized information section among the voice signals using a general model, and controlling transmission of a result of data processing on the voice signal corresponding to the general information section to the terminal.

In addition, the control unit can control to generate voice section information for the personalized information section and the general information section and transmit it to the terminal.

Additionally, the voice segment information may include segment marking information that marks at least one segment among the personalized information segment and the general information segment of the voice signal.

In addition, the control unit can control to receive voice section information for the personalized information section and the general information section from the terminal.

A terminal according to another disclosed embodiment includes a communication unit that performs communication with a server; and a control unit that receives a result of data processing using a general model for the voice signal from the server, and controls data processing using a personalized model generated based on the result of the data processing and personal information for the voice signal.

In addition, the control unit can control to perform data processing on a voice signal corresponding to a personalized information section including the personal information when performing data processing using a personalized model generated based on the data processing result and personal information for the voice signal.

Additionally, the control unit can control to detect the personalized information section among the voice signal.

Additionally, the control unit can control to receive voice section information for the personalized information section and the general information section from the server.

Additionally, the voice segment information may include segment marking information that marks at least one segment among the personalized information segment and the general information segment of the voice signal.

Additionally, the personalized model may be at least one of a personalized speech recognition model, a language understanding model, and a personalized lexical model.

A terminal according to another disclosed embodiment includes a receiving unit for receiving a voice signal; a communication unit for communicating with a server; and a control unit for generating a mapping table by matching an ID to personal information, generating an ID-based personalized model using the mapping table, transmitting the ID-based personalized model to the server, receiving a result of data processing using the ID-based personalized model for a voice signal from the server, and controlling the restoration of the personal information corresponding to the ID using the data processing result and the mapping table.

In addition, the control unit can control, when generating an ID-based personalized model using the mapping table, to display the ID mapped to the personal information as an acoustic unit ID, which is an ID mapped to sound.

Additionally, the sound unit ID may be an ID mapped to the sound according to an agreement with the server.

In addition, the control unit can control to generate the mapping table by mapping an ID to additional information generated from the personal information.

*

*A server according to another disclosed embodiment comprises: a receiving unit for receiving a voice signal; a communication unit for communicating with a terminal; and a control unit for receiving an ID-based personalized model from the terminal, receiving a voice signal, performing data processing on the voice signal using the ID-based personalized model, and transmitting the data processing result to the terminal.

In addition, the control unit can control to display an ID mapped to personal information using an audio unit ID, which is an ID mapped to sound according to an agreement with the terminal.

FIG. 1 is a block diagram showing the internal configuration of a terminal according to the disclosed embodiment.
FIG. 2 is a block diagram showing the internal configuration of a server according to the disclosed embodiment.
Figure 3 is a block diagram showing the internal configuration of the terminal illustrated in Figure 1 in more detail.
Figure 4 is a block diagram showing the internal configuration of the server illustrated in Figure 2 in more detail.
FIG. 5 is a flowchart showing a voice processing method of a terminal according to the disclosed embodiment.
FIG. 6 is a flowchart showing a voice processing method of a server according to the disclosed embodiment.
Figure 7 is a diagram explaining the personalized information section and the general information section.
FIG. 8 is a flowchart showing an example of a specific operation process of a terminal and a server according to the disclosed embodiment.
FIG. 9 is a flowchart showing a voice processing method of a terminal according to another disclosed embodiment.
FIG. 10 is a flowchart showing a voice processing method of a server according to another disclosed embodiment.
FIG. 11 is a flowchart showing an example of a specific operation process of a terminal and a server according to another disclosed embodiment.
FIG. 12 is a block diagram showing the internal configuration of a terminal according to another disclosed embodiment.
FIG. 13 is a block diagram illustrating in more detail the internal configuration of a server according to another disclosed embodiment.
Figure 14 is a block diagram showing the internal configuration of the terminal illustrated in Figure 12 in more detail.
Figure 15 is a block diagram showing the internal configuration of the terminal illustrated in Figure 13 in more detail.
FIG. 16 is a flowchart showing a voice processing method of a terminal according to another disclosed embodiment.
FIG. 17 is a flowchart showing a voice processing method of a server according to another disclosed embodiment.
Figure 18 is a diagram showing personal information.
Figure 19 is a diagram showing personal information by pronunciation symbol.
Figure 20 is a diagram showing a mapping table that maps personal information to an ID.
Figure 21 is a diagram showing a mapping table that maps pronunciation symbols of personal information to IDs.
Figure 22 is a diagram showing a personal information ID as a pronunciation symbol ID.
FIG. 23 is a flowchart showing an example of a specific operation process of a terminal and a server according to another disclosed embodiment.

The advantages and features of the disclosed embodiments, and the methods for achieving them, will become apparent with reference to the embodiments described below together with the accompanying drawings. However, the disclosed embodiments are not limited to the embodiments disclosed below, but may be implemented in various different forms, and the present embodiments are provided only to make the disclosure of the disclosed embodiments complete and to fully inform those skilled in the art of the scope of the invention to which the disclosed embodiments belong, and the disclosed embodiments are defined only by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element in between. Also, throughout the specification, when a part is said to "include" a certain component, this does not mean that other components are excluded, but rather that other components can be included, unless specifically stated otherwise. Also, the term "part" used throughout the specification means software, hardware components such as FPGAs or ASICs, and the "part" performs certain roles. However, the "part" is not limited to software or hardware. The "part" may be configured to be on an addressable storage medium and may be configured to execute one or more processors. Thus, as an example, a "part" may include components such as software components, object-oriented software components, class components, and task components, and processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and "parts" may be combined into fewer components and "parts" or further separated into additional components and "parts".

Below, with reference to the attached drawings, the disclosed embodiments are described in detail so that those with ordinary skill in the art to which the disclosed embodiments belong can easily practice them. However, the disclosed embodiments can be implemented in various different forms and are not limited to the embodiments described herein. In addition, in order to clearly describe the disclosed embodiments in the drawings, parts that are not related to the description are omitted.

The terms used in the disclosed embodiments are selected from commonly used terms that are as much as possible while considering the functions of the disclosed embodiments, but this may vary depending on the intention of engineers working in the field, precedents, the emergence of new technologies, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meanings thereof will be described in detail in the description of the relevant invention. Therefore, the terms used in the disclosed embodiments should be defined based on the meanings of the terms and the overall contents of the disclosed embodiments, rather than simply the names of the terms.

In this specification, a personalized information section means a section of a voice signal that includes personal information that can directly or indirectly identify each individual. For example, a section of a voice signal that includes a name stored in the phone book of a user terminal, a user's search history, user location information, etc. may correspond to a personalized information section.

In this specification, the general information section refers to a section of a voice signal that contains general information, not personal information, and refers to the remaining sections excluding the personalized information section. For example, a section that contains sentences with general meanings, such as “Call me” or “Have you eaten?”, may correspond to the general information section.

In this specification, a personalized model is a voice processing model that reflects individual characteristics and is a customized voice processing model for a specific individual.

In this specification, a general model is a general speech processing model, which is a speech processing model for speech processing of an unspecified person, not a specific individual.

FIG. 1 is a block diagram showing the internal configuration of a terminal according to the disclosed embodiment.

Referring to FIG. 1, a terminal (100) according to the disclosed embodiment includes a receiving unit (110), a communication unit (130), and a control unit (150).

The receiving unit (110) performs the role of receiving a voice signal. The receiving unit (110) may include various components such as a microphone unit, a USB interface unit, and a DVD interface unit. For example, if the receiving unit (110) includes a microphone unit, the terminal (100) may directly receive a user voice signal through the microphone unit. In addition, if the receiving unit (110) includes a USB interface unit, the terminal (100) may also receive a voice signal file from a USB. Furthermore, if a voice signal is received from an external device through the communication unit (130), the communication unit (130) may also perform the role of the receiving unit (110).

The communication unit (130) performs a role of communicating with an external device. The communication unit (130) can be connected to a network by wire or wirelessly to perform communication with an external device. According to the disclosed embodiment, the communication unit (130) can communicate with a server and transmit and receive data. For example, the communication unit (130) can include a short-range communication module, a mobile communication module, a wireless Internet module, a wired Internet module, etc. In addition, the communication unit (130) can include one or more components.

The control unit (150) controls the operation of the entire terminal (100) and can process a voice signal by controlling the receiving unit (110) and the communication unit (130). The control unit (150) can include a RAM that stores signals or data input from the outside of the terminal (100) or is used as a storage area corresponding to various tasks performed in the electronic device, a ROM that stores a control program for controlling peripheral devices, and a processor. The processor can be implemented as a SoC (System On Chip) that integrates a core (not shown) and a GPU (not shown). In addition, the processor can include a plurality of processors.

The control unit (150) according to the disclosed embodiment receives a voice signal through the receiving unit (110), detects a personalized information section including personal information from the received voice signal, performs data processing on the voice signal corresponding to the personalized information section using a personalized model generated based on the personal information, and controls the communication unit (130) to receive a result of data processing on the voice signal corresponding to the general information section, which is a section other than the personalized information section, from a server. At this time, the result of data processing on the voice signal corresponding to the general information section received from the server may be a result of the server processing the voice signal corresponding to the general information section using a general model. Since the server has high computational capabilities, fast data processing on the general information section is possible.

In addition, the control unit (150) can control to generate voice segment information for the personalized information segment and the general information segment and transmit it to the server through the communication unit (130), or can control to receive voice segment information for the personalized information segment and the general information segment from the server. Here, the voice segment information can include segment marking information that marks at least one segment among the personalized information segment and the general information segment in the voice signal.

Furthermore, the control unit (150) can control the output of the voice processing result to the user.

According to the disclosed embodiment, the terminal (100) performs data processing on a voice signal corresponding to a personalized information section using a personalized model generated based on personal information, and receives a result of data processing from a server on a voice signal corresponding to a general information section other than the personalized information section. Since the server performs data processing only on the general information section, it does not use a personalized model in the data processing process. Accordingly, no form of information transmission including personal information occurs between the server and the terminal (100), and accordingly, no personal information is stored on the server. As a result, according to the disclosed embodiment, it is possible to implement a voice processing system having higher performance and processing speed while fundamentally protecting personal information.

A control unit (150) according to another disclosed embodiment receives a voice signal through a receiving unit (110), receives a result of data processing using a general model for the voice signal from a server through a communication unit (130), and controls data processing using a personalized model generated based on the data processing result and personal information for the voice signal. In this case, since the server (200) has high computational capability, fast data processing is possible for the general information section. In addition, the personalized model may be at least one model from among a personalized voice recognition model, a language understanding model, and a personalized lexical model.

In addition, the control unit (150) may perform data processing on a voice signal corresponding to a personalized information section including personal information when performing data processing using a personalized model generated based on the data processing result and personal information for the voice signal. At this time, the control unit (150) may perform control to detect a personalized information section in the voice signal, and may perform control to receive voice section information on the personalized information section and the general information section from the server through the communication unit (130). Here, the voice section information may include section marking information that marks at least one section among the personalized information section and the general information section in the voice signal.

According to the disclosed embodiment, the terminal (100) receives the result of data processing using a general model for a voice signal from the server, and performs data processing again using a personalized model generated based on the data processing result and personal information for the received voice signal. That is, since voice processing using personalized modeling is performed only in the terminal (100), no form of information transmission including personal information occurs between the server and the terminal (100), and accordingly, no personal information is stored in the server. As a result, according to the disclosed embodiment, it is possible to implement a voice processing system having higher performance and processing speed while fundamentally protecting personal information.

FIG. 2 is a block diagram showing the internal configuration of a server according to the disclosed embodiment.

Referring to FIG. 2, a server (200) according to the disclosed embodiment includes a receiving unit (210), a communication unit (230), and a control unit (250).

The receiving unit (210) performs the role of receiving a voice signal. The receiving unit (210) may include components that can receive voice signals in various forms, such as a USB interface unit and a DVD interface unit. For example, if the receiving unit (210) includes a USB interface, the server (200) can receive a voice signal file from the USB. Furthermore, if a voice signal is received from an external device through the communication unit (230), the communication unit (230) may also perform the role of the receiving unit (210).

The communication unit (230) performs a role of communicating with an external device. The communication unit (230) can be connected to a network by wire or wirelessly to perform communication with an external device. According to the disclosed embodiment, the communication unit (230) can communicate with the terminal (100) and transmit and receive data. According to the disclosed embodiment, the communication unit (130) can communicate with a server and transmit and receive data. For example, the communication unit (130) can include a short-range communication module, a mobile communication module, a wireless Internet module, a wired Internet module, etc. In addition, the communication unit (130) can include one or more components.

The control unit (250) controls the operation of the entire server (200) and can process a voice signal by controlling the receiving unit (210) and the communication unit (230). The control unit (250) can include a RAM that stores signals or data input from the outside of the server (200) or is used as a storage area corresponding to various tasks performed in an electronic device, a ROM that stores a control program for controlling peripheral devices, and a processor. The processor can be implemented as a SoC (System On Chip) that integrates a core (not shown) and a GPU (not shown). In addition, the processor can include a plurality of processors.

The control unit (250) according to the disclosed embodiment receives a voice signal through the receiving unit (210), detects a personalized information section including personal information among the voice signal, performs data processing using a general model for a voice signal corresponding to a general information section other than the personalized information section among the voice signal, and transmits the result of the data processing for the voice signal corresponding to the general information section to the terminal (100) through the communication unit (230).

In addition, the control unit (250) generates voice section information for the personalized information section and the general information section through the communication unit (230) and transmits it to the terminal (100), or receives voice section information for the personalized information section and the general information section from the terminal (100). Here, the voice section information may include section marking information that marks at least one section among the personalized information section and the general information section in the voice signal.

According to the disclosed embodiment, the server (200) performs data processing only on a voice signal corresponding to a general information section, and therefore does not use a personalized model (171) in the data processing process. Accordingly, no form of information transmission including personal information occurs between the server and the terminal (100), and accordingly, no personal information is stored in the server. As a result, according to the disclosed embodiment, it is possible to implement a voice processing system having higher performance and processing speed while fundamentally protecting personal information.

A control unit (250) according to another disclosed embodiment receives a voice signal through a receiving unit (210), performs data processing on a voice signal corresponding to a general information section other than a personalized information section among the voice signals using a general model, and transmits the data processing result to a terminal through a communication unit (230). The control unit (250) can generate voice section information for a personalized voice section among the voice signals, and transmit the generated voice section information to the terminal (100). Here, the voice section information can include section marking information that marks at least one section among the personalized information section and the general information section among the voice signals.

According to the disclosed embodiment, the server (200) performs data processing using only a general model for the received voice signal, and therefore does not use a personalized model (171) in the data processing process. Accordingly, no form of information transmission including personal information occurs between the server and the terminal (100), and accordingly, no personal information is stored in the server. As a result, according to the disclosed embodiment, it is possible to implement a voice processing system having higher performance and processing speed while fundamentally protecting personal information.

Fig. 3 is a block diagram showing the internal configuration of the terminal illustrated in Fig. 1 in more detail. In Fig. 3, descriptions of components that overlap with those in Fig. 1 are omitted.

The control unit (150) may include a voice processing engine (151). According to the disclosed embodiment, the voice processing engine (151) may include an voice recognition engine (ASR engine) and a language understanding engine (NLU engine), and performs voice recognition and language understanding by data processing a received voice signal. At this time, the voice recognition engine and the language understanding engine may process the voice signal using a voice recognition model and a language understanding model, respectively.

A speech recognition model may include an acoustic model and a language model. The acoustic model is a model for a speech signal, and is generated statistically from a large amount of collected speech data. The language model is a grammatical model for user speech, and is usually obtained through statistical learning from a large amount of collected text data. The language understanding model is a model that represents the meaning of user speech (semantic model), and is obtained by statistical learning from a large amount of text data or by writing semantic understanding rules considering the usage scenario.

The terminal (100) may further include a storage unit (170). The storage unit (170) serves to store programs and data required for the operation of the terminal (100). The storage unit (170) may be composed of a volatile storage medium or a nonvolatile storage medium, and may also be composed of a combination of both storage media. The volatile storage medium may include a semiconductor memory such as RAM, DRAM, and SRAM, and the nonvolatile storage medium may include a hard disk and a flash NAND memory.

According to the disclosed embodiment, personal information (172) may be stored in the storage (170). Personal information (172) is information that can directly or indirectly identify each individual, and the type of data stored may vary depending on the type of terminal. For example, in the case of a mobile device, it may include contact information, a music list, the contents of short messages or history of sending and receiving, and web search history, and in the case of a TV, it may include personal playlists, etc.

In addition, a personalized model (171) may be stored in the storage unit (170). The personalized model (171) is a voice processing model that reflects individual characteristics generated using personal information. A personalized voice recognition model and/or a personalized language understanding model may be stored in the storage unit (170). When such a personalized voice recognition model and/or a personalized language understanding model is used, a voice processing system with higher performance can be implemented.

Fig. 4 is a block diagram showing the internal configuration of the server illustrated in Fig. 2 in more detail. In Fig. 4, descriptions of components that overlap with those in Fig. 2 are omitted.

The control unit (250) may include a voice processing engine (251). According to the disclosed embodiment, the voice processing engine (151) may include an voice recognition engine (ASR engine) and a language understanding engine (NLU engine), and performs voice recognition and language understanding by data processing a received voice signal. At this time, the voice recognition engine and the language understanding engine may process the voice signal using a voice recognition model and a language understanding model, respectively.

The server (200) may further include a storage unit (270). The storage unit (270) stores programs and data required for the operation of the server (200). The storage unit (270) may be configured as a volatile storage medium or a nonvolatile storage medium, and may also be configured as a combination of both storage media. The volatile storage medium may include a semiconductor memory such as RAM, DRAM, and SRAM, and the nonvolatile storage medium may include a hard disk and a flash NAND memory. According to the disclosed embodiment, a general model (271) may be stored in the storage unit (270). The general model (271) is a general voice processing model, and is a voice processing model for voice processing of an unspecified person, not a specific individual. The general model (271) configured with a large capacity provides high voice processing performance for various language expressions (large vocabulary) of the user in combination with the high computational capability of the server. Therefore, according to the disclosed embodiment, no form of personal information is stored in the storage unit (270), thereby providing high voice processing performance while fundamentally protecting personal information.

Below, the operation of the terminal (100) and server (200) will be described in more detail.

FIG. 5 is a flowchart showing a voice signal processing method of a terminal according to the disclosed embodiment.

First, in step 510, the terminal (100) receives a voice signal. The terminal (100) can receive a voice signal through various components. Receiving a voice signal through a microphone unit would be the most common form, but the voice signal can also be received through a USB interface unit or a DVD interface unit, etc. Furthermore, the voice signal can also be received through communication with an external device. According to the disclosed embodiment, the terminal (100) can transmit the voice signal received in this way to the server (200).

Thereafter, in step 520, the terminal (100) detects a personalized information section including personal information in the voice signal. The personalized information section refers to a section including personal information that can directly or indirectly identify each individual in the voice signal. For example, the terminal (100) can detect a section including a name stored in the phone book of the user terminal, a user's search history, user location information, etc. in the voice signal as a personalized information section by referring to the personal information (172) stored in the storage unit (170) of FIG. 3. This will be described with reference to FIG. 7.

Figure 7 is a diagram explaining the personalized information section and the general information section.

Referring to Figure 7, the sentence ‘Mr. Hong Gil-dong, please call Mr. Kim Gil-dong on the 10th floor’ can be divided into 9 sections. That is, it can be divided into Hong Gil-dong (701), Mr. (702), 10 (703), floor (704), Kim Gil-dong (705), Mr. (706), to (707), phone (708), and please (709). The criteria for division of these sections can be applied differently depending on the situation.

Here, assuming that Hong Gil-dong (701) is a word referring to a user and Kim Gil-dong (705) is a name stored in the phone book of the user terminal, the voice signal sections corresponding to Hong Gil-dong (701) and Kim Gil-dong (705) correspond to personalized sections (710). That is, Hong Gil-dong (701) and Kim Gil-dong (705) correspond to personal information, and the sections 701 and 705 including such personal information correspond to personalized information sections (710). The terminal (100) detects the personalized information section (710) including personal information among the voice signals in step 520.

The terminal (100) can detect a personalized information section through various methods. For example, a word not included in the dictionary can be determined as personal information, and a voice section including the word can be determined as a personalized information section. However, this method is only one example, and various personalized information section detection methods can be used.

In addition, the terminal (100) can generate voice segment information for the personalized information segment and the general information segment using the personalized information segment detected as described above and transmit the same to the server (200). Here, the voice segment information can include segment marking information that marks at least one segment among the personalized information segment and the general information segment in the voice signal. That is, the terminal (100) can mark the personalized information segment and/or the general information segment in the voice signal to indicate that it is a corresponding segment, and generate and transmit voice segment information including such segment marking information. The voice segment information transmitted to the server (200) in this way can be used by the server (200) to confirm and/or process the general information segment in the voice signal. The server (200) can confirm the remaining segments excluding the segments marked as the general information segment or the segments marked as the personalized information segment and perform data processing on the voice signal corresponding thereto.

Returning to the description of FIG. 5, at step 530, the terminal (100) performs data processing on a voice signal corresponding to a personalized information section using a personalized model (171) generated based on personal information. The personalized model (171) is a voice processing model that reflects individual characteristics and is a customized voice processing model for a specific individual. When performing data processing on a voice signal using the personalized model (171), more accurate voice processing is possible compared to a general model (271) created for an unspecified number of people. In addition, since the terminal (100) performs data processing on a voice signal corresponding to a personalized section directly using the personalized model (171), there is no need to transmit the personalized model (171) including personal information to an external device such as a server, and thus, personal information can be protected at the source.

Furthermore, at step 540, the terminal (100) receives from the server (200) the result of data processing for a voice signal corresponding to a general information section other than a personalized information section. The general information section is a section of the voice signal that includes general information other than personal information, and refers to the remaining sections excluding the personalized information section. For example, a section including sentences with general meanings such as ‘call me’ or ‘have you eaten?’ may correspond to the general information section. This will be explained with reference to Fig. 7.

As seen above, in Figure 7, the sentence ‘Mr. Hong Gil-dong, please call Mr. Kim Gil-dong on the 10th floor’ can be divided into nine sections.

Here, Mr. (702), 10 (703), floor (704), Mr. (706), to (707), phone (708), please (709) are general words, and the sections 702, 703, 704, 706, 707, 708 and 709 containing such general information correspond to the general information section (720).

At step 540, the terminal (100) does not directly process data for a voice signal corresponding to a general information section (720) including general information among voice signals, but receives the result of data processing from a server (200) that has higher computational capabilities than the terminal (100). Therefore, fast data processing is possible for a voice signal corresponding to a general information section (720).

The result of data processing for a voice signal corresponding to a general information section received from a server (200) may be a result of the server (200) processing a voice signal corresponding to a general information section using a general model (271). The general model (271) can provide high voice processing performance for various language expressions (vocabulary) of a user in combination with the high computational ability of the server (200). In addition, since the server (200) has high computational ability, fast data processing is possible for the general information section.

The terminal (100) may receive voice segment information for the personalized information segment and the general information segment from the server (200). Here, the voice segment information may include segment marking information that marks at least one segment among the personalized information segment and the general information segment in the voice signal. The terminal (100) may also detect the personalized information segment on its own and generate voice segment information. However, since the terminal (100) and the server (200) may perform data processing on the personalized information segment and/or the general information segment detected by themselves, if the personalized information segment detected by the terminal (100) and the personalized information segment detected by the server (200) are different from each other, segments that are missed in data processing may occur. Therefore, in order to perform data processing without omission for all segments, the terminal (100) and the server (200) may share voice segment information for the personalized information segment and the general information segment.

Furthermore, the terminal (100) can output the data processing result to the user.

*

*According to the disclosed embodiment, personal information (172) or a personalized model (171) is maintained in the terminal (100) without being transmitted to the server (200), and the terminal (100) performs voice processing using the personal information (172) or the personalized model (171), thereby fundamentally protecting personal information while implementing a voice processing system having higher performance and processing speed.

FIG. 6 is a flowchart showing a voice signal processing method of a server according to the disclosed embodiment.

First, in step 610, the server (200) receives a voice signal. The server (200) can receive a voice signal through various components. Receiving a voice signal from a terminal (100) would be the most common form, but the voice signal can also be received through a USB interface unit or a DVD interface unit, etc.

Thereafter, in step 620, the server (200) detects a personalized information section including personal information from the received voice signal. The server (200) may detect the personalized information section by interpreting the received voice signal, or may detect the personalized information section by receiving and parsing voice section information from the terminal (100). The server (200) may detect the personalized information section through various methods. For example, a word not included in the dictionary may be determined to be personal information, and a voice section including the word may be determined to be a personalized information section. However, this method is only one example, and various personalized information section detection methods may be used.

The personalized information section and general section have been explained above together with Fig. 7, and will not be explained again.

In addition, the server (200) can use the personalized information section detected as described above to generate voice section information for the personalized information section and the general information section and transmit the same to the terminal (100). Here, the voice section information can include section marking information that marks at least one section among the personalized information section and the general information section in the voice signal. That is, the server (200) can mark the personalized information section and/or the general information section in the voice signal to indicate that it is the corresponding section, and generate and transmit voice section information including such section marking information. The voice section information transmitted to the terminal (100) in this way can be used by the terminal (100) to confirm and/or process the personalized information section in the voice signal. The terminal (100) can confirm the remaining sections excluding the sections marked as the general information section or the sections marked as the personalized information section, and perform data processing on the voice signal corresponding thereto.

Next, at step 630, the server (200) performs data processing using the general model (271) for the voice signal corresponding to the general information section, which is a section other than the personalized information section among the voice signals. The general model (271) can provide high voice processing performance for various language expressions (vocabulary) of the user in combination with the high computational capability of the server. Furthermore, since the server (200) has high computational capability, fast data processing is possible for the general information section. In addition, since the server (200) does not process the voice signal corresponding to the personalized section including personal information, but performs data processing in the terminal (100) having the personalized model (171), personal information can be fundamentally protected.

Further, at step 640, the server (200) transmits the result of data processing for the voice signal corresponding to the general information section to the terminal.

The server (200) may receive voice segment information for the personalized information segment and the general information segment from the terminal (100). Here, the voice segment information may include segment marking information that marks at least one segment among the personalized information segment and the general information segment in the voice signal. The server (200) may also detect the personalized information segment on its own and generate voice segment information. However, since the terminal (100) and the server (200) may perform data processing on the personalized information segment and/or the general information segment detected by themselves, if the personalized information segment detected by the terminal (100) and the personalized information segment detected by the server (200) are different from each other, segments that are omitted from data processing may occur. Therefore, in order to perform data processing without omission for all segments, the terminal (100) and the server (200) may share voice segment information for the personalized information segment and the general information segment.

As a result, according to the disclosed embodiment, it is possible to implement a voice processing system having higher performance and processing speed while fundamentally protecting personal information.

FIG. 8 is a flowchart showing an example of a specific operation process of a terminal and a server according to the disclosed embodiment.

First, in step 805, the terminal (100) receives a voice signal. As described above, the terminal (100) can receive a voice signal through various components. Receiving a voice signal through a microphone unit would be the most common form, but the voice signal can also be received through a USB interface unit or a DVD interface unit. Furthermore, the voice signal can also be received through communication with an external device. Then, in step 810, the terminal (100) can transmit the received voice signal to the server (200).

The terminal (100) and the server (200) detect personalized voice sections for the voice signal in steps 815 and 820, respectively. The terminal (100) and the server (200) can detect personalized information sections through various methods. For example, a word not included in the dictionary can be determined to be personal information, and a voice section including the word can be determined to be a personalized information section. However, this method is only one example, and various personalized information section detection methods can be used.

The terminal (100) may proceed to step 825 to process voice data based on a personalized model (171) for the personalized voice section detected at step 815, and may proceed to step 830 to pass data processing and generate voice section information for general information sections other than the personalized voice section. The server (200) may proceed to step 835 to pass data processing and generate voice section information for the personalized voice section detected at step 820, and may proceed to step 830 to process voice data based on a general model (271) for general information sections other than the personalized voice section.

Through this process, the terminal (100) performs data processing using a personalization model (171) for a voice signal corresponding to a personalization section, and the server (200) performs data processing using a general model (271) for a voice signal corresponding to a general information section based on high computational capability, thereby implementing high voice processing performance while fundamentally protecting personal information.

Thereafter, the terminal (100) and the server (200) share the voice segment information and the data processing result in steps 845 and 850. That is, the server (200) transmits the voice data processing result and the voice segment information to the terminal (100) in step 845, and the terminal (100) transmits the voice segment information to the server (200). In Fig. 8, this process is illustrated as steps 845 and 850, but either the terminal (100) or the server (200) may transmit the data first during the sharing process.

As described above, the terminal (100) and the server (200) can each independently detect a personalized information section and generate voice section information. However, since the terminal (100) and the server (200) can each perform data processing on the personalized information section and/or general information section detected by themselves, if the personalized information section detected by the terminal (100) and the personalized information section detected by the server (200) are different from each other, sections that are omitted from data processing may occur. Therefore, in order to perform data processing without omission on all sections, the terminal (100) and the server (200) can share voice section information on the personalized information section and the general information section.

The terminal (100) determines whether data processing is completed for all voice signal sections at step 855, and if so, terminates the operation. If data processing is not completed at step 855, the terminal (100) proceeds to step 865, and can perform data processing using a personalized model (171) for unprocessed voice sections based on voice section information received from the server (200). The server (200) determines whether data processing is completed for all voice signal sections at step 860, and if so, terminates the operation. If data processing is not completed at step 860, the server (200) proceeds to step 870, and can perform data processing using a personalized model (171) for unprocessed voice sections based on voice section information received from the terminal (100).

As illustrated in FIG. 8, the voice signal processing steps 815 to 870 may be performed simultaneously and in parallel at the terminal (100) and the server (200), or the terminal (100) and the server (200) may perform voice processing by switching alternately.

When the terminal (100) and the server (200) perform voice processing by switching alternately, if the terminal (100) performs data processing based on the personalized model (171) for the voice signal corresponding to the personalized information section and then a voice signal corresponding to the general information section is output, if the voice section information is transmitted to the server (200), the server (200) may perform voice processing using the general model (271) starting from the voice signal that follows the voice signal that the terminal (100) processed. Afterwards, if the voice signal corresponding to the personalized information section is output again, if the data processing result up to that point and the voice section information are transmitted to the terminal (100), the terminal (100) may perform voice processing using the general model (271) starting from the voice signal that follows the voice signal that the server (200) processed.

Conversely, when voice processing is started in the server (200), the server (200) performs voice processing using the general model (271) for the voice signal corresponding to the general section, and when a voice signal corresponding to the personalized information section is output, if the data processing result up to this point and the voice section information are transmitted to the terminal (100), the terminal (100) can perform voice processing using the general model (271) starting from the voice signal following the voice signal processed by the server (200).

According to the disclosed embodiment, a voice processing system having higher performance and processing speed can be implemented while protecting personal information at the source.

FIG. 9 is a flowchart showing a voice processing method of a terminal according to another disclosed embodiment.

First, in step 910, the terminal (100) receives a voice signal. The terminal (100) can receive a voice signal through various components. Receiving a voice signal through a microphone unit would be the most common form, but the voice signal can also be received through a USB interface unit or a DVD interface unit, etc. Furthermore, the voice signal can also be received through communication with an external device. According to the disclosed embodiment, the terminal (100) can transmit the voice signal received in this way to the server (200).

Thereafter, at step 920, the terminal (100) receives the result of data processing for the voice signal using the general model (271) from the server (200). That is, the terminal (100) receives the result of data processing for the voice signal performed independently from the terminal (100) by the server (200). The general model (271) can provide high voice processing performance for various language expressions (vocabulary) of the user in combination with the high computational ability of the server (200). Furthermore, since the server (200) has high computational ability, fast data processing is possible for the general information section.

Next, at step 930, the terminal (100) performs data processing using the personalized model (171) generated based on personal information for the voice signal and the data processing result received from the server (200). According to the disclosed embodiment, the terminal (100) can perform data processing again for all or part of the voice signal using the personalized model (171) and the data processing result from the server (200). As described above, the server (200) performs data processing using the general model (271), and performs data processing again using the personalized model (171) for higher voice processing performance.

In this case, the terminal (100) can perform data processing on the voice signal corresponding to the personalized information section including personal information. That is, the server (200) uses high computational power to perform data processing on the entire voice signal without distinction between the general information section and the personalized information section using the general model (271). After that, the terminal (100) repeats data processing once again using the personalized model for the personalized information section where processing performance can be improved using the personal information (172). To this end, the terminal (100) can detect the personalized information section in the voice signal. The terminal (100) can detect the personalized information section using various methods. For example, a word not included in the dictionary can be determined as personal information, and a voice section including the word can be determined as a personalized information section. However, this method is only one example, and various personalized information section detection methods can be used.

Alternatively, the terminal (100) may receive voice segment information for the personalized information segment and the general information segment from the server (200). By receiving voice segment information generated by the server (200) with high computational capability, the data processing burden of the terminal (100) is reduced, thereby making the overall voice processing speed faster. Here, the voice segment information may include segment marking information that marks at least one segment among the personalized information segment and the general information segment in the voice signal.

The personalized information section and general information section have been explained above together with Fig. 7, and will not be explained again.

Here, the personalized model (171) may be at least one of a personalized speech recognition model, a personalized natural language understanding model, and a personalized lexical model.

When the terminal (100) performs data processing on a voice signal using a personalized voice recognition model, the terminal (100) can receive data processing results in the form of phonemes, pseudo-morphemes or words from the server (200), and can receive multi-pass processing results such as N-best hypothesis, lattice and confusion network and utilize them for data processing.

When the terminal (100) performs data processing on a voice signal using a language understanding model, the terminal (100) can receive data processing results in units of phonemes, pseudo-morphemes or words and information such as confidence scores in units of sentences or words from the server (200) and utilize them for data processing. In addition, the terminal can receive multi-pass processing results from the server (200) and utilize them for data processing.

When the terminal (100) performs data processing on a voice signal using a personalized lexical model, the terminal (100) can receive from the server (200) data processing results in units of phonemes, pseudomorphemes or words and information such as confidence scores in units of sentences or words and use them for data processing. In addition, the terminal (100) can perform data processing using the data processing results received from the server (200) and a personal word list, and in this case, a pronunciation dictionary can be used for phoneme-by-phone comparison between a hypothesis and personal words.

Regardless of the model used, more accurate voice processing is possible by utilizing personal information without the need for personal information to be transmitted to the server (200).

According to the disclosed embodiment, since voice processing using personalized modeling is performed only in the terminal (100), no form of information transmission including personal information occurs between the server (200) and the terminal (100), and accordingly, no personal information is stored in the server (200). Accordingly, it is possible to implement a voice processing system having higher performance and processing speed while fundamentally protecting personal information.

FIG. 10 is a flowchart showing a voice processing method of a server according to another disclosed embodiment.

First, in step 1010, the server (200) receives a voice signal. The server (200) can receive a voice signal through various components. Receiving a voice signal from a terminal (100) would be the most common form, but the voice signal can also be received through a USB interface unit or a DVD interface unit, etc.

After that, in step 1020, the server (200) performs data processing on the received voice signal using the general model (271). That is, the server (200) performs data processing on the voice signal independently from the terminal (100) using the general model (271). The general model (271) can provide high voice processing performance for various language expressions (vocabulary) of the user in combination with the high computational ability of the server (200). Furthermore, since the server (200) has high computational ability, fast data processing is possible.

Next, in step 1030, the server (200) transmits the data processing result to the terminal (100). In this case, the server (200) may generate voice segment information for the personalized information segment and the general information segment and transmit them together to the terminal (100). By generating voice segment information in a server (200) with high computational capability and transmitting it to the terminal (100), the data processing burden of the terminal (100) can be reduced, thereby making the overall voice processing speed faster. Here, the voice segment information may include segment marking information that marks at least one segment among the personalized information segment and the general information segment in the voice signal.

As a result, according to the disclosed embodiment, it is possible to implement a voice processing system having higher performance and processing speed while fundamentally protecting personal information.

FIG. 11 is a flowchart showing an example of a specific operation process of a terminal and a server according to another disclosed embodiment.

First, in step 1110, the terminal (100) receives a voice signal. As described above, the terminal (100) can receive a voice signal through various components. Receiving a voice signal through a microphone unit would be the most common form, but the voice signal can also be received through a USB interface unit or a DVD interface unit. Furthermore, the voice signal can also be received through communication with an external device. Then, in step 1120, the terminal (100) can transmit the received voice signal to the server (200).

At step 1130, the server (200) processes data using a general model (271) for the received voice signal. The general model (271) can provide high voice processing performance for various language expressions (vocabulary) of the user in combination with the high computational ability of the server (200). Furthermore, since the server (200) has high computational ability, fast data processing is possible.

At this time, the server (200) can generate voice section information for the personalized information section and the general information section. By generating voice section information in a server (200) with high computational capability, the data processing burden of the terminal (100) can be reduced, thereby making the overall voice processing speed faster.

Thereafter, at step 1140, the server (200) can transmit the data processing result and voice section information to the terminal (100). At step 1150, the terminal (100) can perform data processing using the personalized model (171) generated based on personal information for the voice signal and the received data processing result. That is, the terminal (100) can perform data processing again for the entire or part of the voice signal. In order to achieve higher voice processing performance, the data is processed again using the personalized model (171).

In this case, the terminal (100) can detect a personalized information section and perform data processing on a voice signal corresponding to the personalized information section. In addition, the terminal (100) can perform data processing on a voice signal using at least one model among a personalized voice recognition model, a language understanding model, and a personalized lexical model.

According to the disclosed embodiment, since voice processing using personalized modeling is performed only in the terminal (100), no form of information transmission including personal information occurs between the server (200) and the terminal (100), and accordingly, no personal information is stored in the server (200). Accordingly, it is possible to implement a voice processing system having higher performance and processing speed while fundamentally protecting personal information.

FIG. 12 is a block diagram showing the internal configuration of a terminal according to another disclosed embodiment.

Referring to FIG. 12, a terminal (1200) according to another disclosed embodiment includes a communication unit (1210) and a control unit (1230).

The communication unit (1210) performs a role of communicating with an external device. The communication unit (1210) may be connected to a network by wire or wirelessly to perform communication with an external device. According to the disclosed embodiment, the communication unit (1210) may communicate with a server and transmit and receive data. According to the disclosed embodiment, the communication unit (130) may communicate with a server and transmit and receive data. For example, the communication unit (130) may include a short-range communication module, a mobile communication module, a wireless Internet module, a wired Internet module, etc. In addition, the communication unit (130) may include one or more components.

The control unit (1230) controls the operation of the entire terminal (1200) and can process audio signals by controlling the communication unit (1210). The control unit (1230) can include a RAM that stores signals or data input from the outside of the terminal (1200) or is used as a storage area corresponding to various tasks performed in the electronic device, a ROM that stores a control program for controlling peripheral devices, and a processor. The processor can be implemented as a SoC (System On Chip) that integrates a core (not shown) and a GPU (not shown). In addition, the processor can include a plurality of processors.

The control unit (1230) according to the disclosed embodiment generates a mapping table by associating an ID with personal information, generates an ID-based personalized model by using the mapping table, transmits the ID-based personalized model to a server through the communication unit (1210), receives a result of data processing using the ID-based personalized model for a voice signal from the server, and controls to restore personal information corresponding to the ID by using the data processing result and the mapping table. At this time, the control unit (1230) may also generate a mapping table by associating an ID with additional information generated from the personal information. In this case, the control unit (1230) may control to receive a result of data processing using the ID-based personalized model for the additional information from the server, and restore the additional information corresponding to the ID by using the data processing result and the mapping table.

When the control unit (1230) creates an ID-based personalized model using the mapping table, the ID mapped to the personal information can be expressed as an acoustic unit ID, which is an ID mapped to an acoustic unit. Here, the acoustic unit ID can be an ID mapped to the acoustic unit according to an agreement with the server. The acoustic unit ID represents a specific part of a voice recognition model corresponding to a pronunciation symbol, and may not be mapped 1:1 with the pronunciation symbol.

According to the disclosed embodiment, personal information and additional information generated from the personal information are masked with an ID directly assigned by the terminal to generate a personalized model, so that even if the personalized model is exposed to the outside, it is difficult to restore the personal information masked with the ID, thereby protecting the personal information. In addition, by transmitting the personalized model to a server having high computational capability and performing data processing on the voice signal, a voice processing system having higher performance and processing speed can be implemented.

FIG. 13 is a block diagram illustrating in more detail the internal configuration of a server according to another disclosed embodiment.

Referring to FIG. 13, a server (1300) according to the disclosed embodiment includes a receiving unit (1310), a communication unit (1330), and a control unit (1350).

The receiving unit (1310) performs the role of receiving a voice signal. The receiving unit (1310) may include components that can receive voice signals in various forms, such as a USB interface unit and a DVD interface unit. For example, if the receiving unit (1310) includes a USB interface, the server (1300) can receive a voice signal file from the USB. Furthermore, if a voice signal is received from an external device through the communication unit (1330), the communication unit (1330) may also perform the role of the receiving unit (1310).

The communication unit (1330) performs a role of communicating with an external device. The communication unit (1330) may be connected to a network by wire or wirelessly to perform communication with an external device. According to the disclosed embodiment, the communication unit (1330) may communicate with the terminal (1200) and transmit and receive data. According to the disclosed embodiment, the communication unit (130) may communicate with a server and transmit and receive data. For example, the communication unit (130) may include a short-range communication module, a mobile communication module, a wireless Internet module, a wired Internet module, etc. In addition, the communication unit (130) may include one or more components.

The control unit (1350) controls the operation of the entire server (1300) and can process a voice signal by controlling the receiving unit (1310) and the communication unit (1330). The control unit (1350) can include a RAM that stores signals or data input from the outside of the server (1300) or is used as a storage area corresponding to various tasks performed in an electronic device, a ROM that stores a control program for controlling peripheral devices, and a processor. The processor can be implemented as a SoC (System On Chip) that integrates a core (not shown) and a GPU (not shown). In addition, the processor can include a plurality of processors.

The control unit (1350) according to the disclosed embodiment receives an ID-based personalized model from a terminal (1200) through a communication unit (1330), receives a voice signal through a receiving unit (1310), performs data processing on the voice signal using the ID-based personalized model, and controls the transmission of the data processing result to the terminal (1200) through the communication unit (1330).

In addition, the control unit (1350) can control the display of an ID mapped to personal information using an acoustic unit ID, which is an ID mapped to an acoustic unit according to an agreement with the terminal (1200), when processing data using an ID-based personalized model for a voice signal.

According to the disclosed embodiment, the server (1300) receives an ID-based personalized model from the terminal (1200) and processes data using the received ID-based personalized model, thereby making it difficult to restore personal information (1272) masked by the ID even if the personalized model is exposed to the outside, thereby protecting personal information (1272). In addition, by having the server (1300) with high computational capability perform data processing on a voice signal, a voice processing system with higher performance and processing speed can be implemented.

Fig. 14 is a block diagram showing the internal configuration of a terminal according to another disclosed embodiment in more detail. In Fig. 14, descriptions of components overlapping with Fig. 12 are omitted.

The control unit (1230) may include a personalized model generation unit (1231). According to the disclosed embodiment, the personalized model generation unit (1231) may generate a personalized model based on personal information (1272) included in the storage unit (1270). The personalized model generation unit (1231) may generate a mapping table (1273) by matching an ID to personal information (1272) or additional information generated from personal information (1272), and may generate an ID-based personalized model using the generated mapping table (1273).

The terminal (1200) may further include a receiving unit (1250). The receiving unit (1250) performs a role of receiving a voice signal. The receiving unit (1250) may include various components such as a microphone unit, a USB interface unit, and a DVD interface unit. For example, if the receiving unit (1250) includes a microphone unit, the terminal (1200) may directly receive a user voice signal through the microphone unit. In addition, if the receiving unit (1250) includes a USB interface unit, the terminal (1200) may also receive a voice signal file from the USB. Furthermore, if a voice signal is received from an external device through the communication unit (130), the communication unit (1210) may also perform the role of the receiving unit (1250).

The terminal (1200) may further include a storage unit (1270). The storage unit (1270) serves to store programs and data required for the operation of the terminal (1200). The storage unit (1270) may be composed of a volatile storage medium or a nonvolatile storage medium, and may also be composed of a combination of both storage media. The volatile storage medium may include a semiconductor memory such as RAM, DRAM, and SRAM, and the nonvolatile storage medium may include a hard disk and a flash NAND memory. According to the disclosed embodiment, the storage unit (170) may store personal information (1272) and a mapping table (1273).

Personal information (1272) is information that can directly or indirectly identify each individual, and the type of data stored may vary depending on the type of terminal. For example, in the case of a mobile device, it may include contact information, music list, content of short messages, history of sending and receiving, and web search history, and in the case of a TV, it may include personal playlists, etc.

The mapping table (1273) includes an ID corresponding to personal information (1272) or additional information generated from personal information (1272). The mapping table (1273) is used by the personalization model generation unit (1231) to generate an ID-based personalization model. In addition, the mapping table (1273) is also used in the process of restoring personal information (1272) or additional information corresponding to the ID.

Figure 15 is a block diagram showing the internal configuration of the server illustrated in Figure 13 in more detail.

The control unit (1350) may include a voice processing engine (1351). According to the disclosed embodiment, the voice processing engine (1351) may include a voice recognition engine and a language understanding engine, and performs voice recognition and language understanding by data processing a received voice signal. At this time, the voice recognition engine and the language understanding engine may process the voice signal using a voice recognition model and a language understanding model, respectively.

The server (1300) may further include a storage unit (1370). The storage unit (1370) serves to store programs and data required for the operation of the server (1300). The storage unit (1370) may be composed of a volatile storage medium or a nonvolatile storage medium, and may also be composed of a combination of both storage media. The volatile storage medium may include a semiconductor memory such as RAM, DRAM, and SRAM, and the nonvolatile storage medium may include a hard disk and a flash NAND memory.

According to the disclosed embodiment, a storage unit (1370) may store a personalized model (1372) and a general model (1373). The personalized model (1372) is a personalized model based on an ID received from a terminal (1200), and even if the personalized model is exposed to the outside, it is difficult to restore personal information (1272) masked by an ID, thereby protecting the personal information (1272). The general model (271) is a general voice processing model, and is a voice processing model for voice processing of an unspecified person, not a specific individual. The general model (271) configured in a large capacity can provide high voice processing performance for various language expressions (large vocabulary) of the user in combination with the high computational capability of the server.

Below, the operation of the terminal (1200) and server (1300) is described in more detail.

FIG. 16 is a flowchart showing a voice processing method of a terminal according to another disclosed embodiment.

First, the terminal (1200) generates a mapping table (1273) by matching an ID to personal information (1272) at step 1610. In this case, the mapping table (1273) may also be generated by matching an ID to additional information generated from the personal information (1272) together with the personal information (1272). Here, the additional information may include pronunciation symbols, pronunciation strings, etc. According to the disclosed embodiment, the terminal (1200) generates a pronunciation dictionary using a word list of the personal information (1272), and maps an ID to a pronunciation symbol and a word using the pronunciation dictionary. In this case, the terminal (1200) may assign an arbitrary ID. This will be described in more detail with reference to FIGS. 18 to 20.

Figure 18 is a diagram showing personal information.

Personal information (1272) is information that can directly or indirectly identify each individual, and includes contact information, music lists, content of short messages or history of sending and receiving text messages, web search history, playlists, etc. Referring to FIG. 18, various forms of personal information are illustrated, and it can be seen that personal information (1272) such as names stored in contact items, music lists or singers in music playlists, and search results are stored. According to the disclosed embodiment, the terminal (1200) can create a mapping table by mapping an ID to such personal information (1272). This will be described with reference to FIG. 19.

Figure 19 is a diagram showing a mapping table that maps personal information to an ID.

Referring to FIG. 19, the terminal (1200) generates a word mapping table (1910) by mapping the words Hong Gil-dong, Kim Gil-dong, Gangnam Style, TOXIC, Psy, Galaxy, and Note included in the personal information (1272) to IDs 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, and 0x07, respectively. Furthermore, the terminal (1200) may generate a mapping table (1273) by matching an ID to additional information generated from the personal information (1272) as well as the personal information (1272). Such additional information may include pronunciation symbols, pronunciation strings, etc. This will be described with reference to FIG. 20.

Figure 20 is a diagram showing personal information by pronunciation symbol.

According to the disclosed embodiment, the terminal (1200) can create a phonetic dictionary (1910) using a word list of personal information (1272). Referring to FIG. 20, Hong Gil-dong, a name stored in a contact among personal information (1272), is displayed as a phonetic symbol as ‘HH OW NX K IY T OW NX’. In addition, Gangnam Style included in the music list can be displayed as ‘K AA NX N A M ST AI L’. The terminal (1200) can create a phonetic dictionary (1910) by displaying personal information (1272) as phonetic symbols in this way. At this time, the terminal (1200) can use various forms of phonetic symbols. As described above, pronunciation can be displayed using alphabets, and in the case of English words, a phonetic dictionary (2010) can be created using English phonetic symbols. Referring to FIG. 20, TOXIC is ‘t ?: k s ? k’, Galaxy can display personal information (1272) in English pronunciation symbols, as indicated by ‘g ? l ? k s i ’. The terminal (1200) can create a mapping table (1273) by associating an ID not only to personal information (1272) but also to additional information generated from personal information (1272), such as pronunciation symbols.

Returning to the description of FIG. 16, the terminal (1200) generates an ID-based personalized model (1372) using the mapping table (1273) at step 1620. The ID-based personalized model (1372) is generated using the mapping table (1273) generated at step 1610, and personal information (1272) and additional information are masked as IDs. According to the disclosed embodiment, the terminal (1200) can express the ID mapped to the personal information (1272) and additional information as an acoustic unit ID by generating the personalized model. This will be described with reference to FIGS. 21 and 22.

Figure 21 is a diagram showing a mapping table that maps sound units to IDs.

The sound unit ID represents a specific part of a speech recognition model corresponding to a pronunciation symbol. Referring to FIG. 21, the terminal (100) generates an sound unit mapping table (2110) by mapping HH, OW, NX, K, IY, L, which are the pronunciation symbols included in each word, to IDs 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, ..., respectively. According to the disclosed embodiment, when mapping an ID to a sound, the terminal (1200) may map a specific ID to a specific sound according to an agreement with the server (1300). That is, the ID 0x101 mapped by the terminal (1200) to the sound unit HH may be an ID agreed upon in advance with the server (1300). Accordingly, when the server (1300) processes a voice signal as data, it can correspond a specific voice signal, i.e., a specific sound, to a specific ID agreed upon with the terminal (1200). The agreement process between the terminal (1200) and the server (1300) can be such that one of the terminals (1200) or the server (1300) can designate an ID mapped to a specific sound and notify the other party of it, or can map the ID to a pronunciation symbol while exchanging opinions. In Fig. 20, for convenience, the pronunciation symbol and the sound unit ID are mapped 1:1, but the sound unit ID may not be mapped 1:1 with the pronunciation symbol. For example, a sound that combines the pronunciation symbols HH and OW can be viewed as a single sound unit, and one sound unit ID can be assigned to HH OW.

Figure 22 is a diagram showing a personal information ID as an acoustic unit ID.

The terminal (1200) can map a word ID (2220) to personal information (2210), and map an acoustic unit ID (2230) to the word ID (2220) using the pronunciation symbol and acoustic model of the personal information (2210). The acoustic unit ID represents a specific part of a voice recognition model corresponding to a pronunciation symbol, and may not be mapped 1:1 with the pronunciation symbol. However, for convenience, it is assumed and explained herein that the acoustic unit ID and the pronunciation symbol are mapped 1:1.

Referring to FIG. 22, the terminal (1200) arbitrarily mapped the word ‘Hong Gil-dong’, which is personal information (1272), to an ID of 0x01. The word ‘Hong Gil-dong’ can be expressed by the pronunciation symbol ‘HH OW NX K IY T OW NX’, and each pronunciation symbol is mapped to the sound unit IDs 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, … agreed upon with the server (1300). Therefore, the ID 0x01 corresponding to ‘Hong Gil-dong’ can be expressed by the sound unit IDs 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, ….

According to the disclosed embodiment, the terminal (1200) can arbitrarily map an ID to a word and represent the word ID as an acoustic unit ID agreed upon with the server. Accordingly, personal information (1272) can be protected even if the personalized model is exposed to the outside by masking the personal information (1272) with an ID, and the server (1300) can perform data processing on a voice signal using the agreed acoustic unit ID.

Next, the terminal (1200) transmits the ID-based personalized model (1372) to the server at step 1630. As described above, the ID-based personalized model (1372) can be generated based on the word ID (2220) and the sound unit ID (2230) illustrated in FIG. 22. Accordingly, when the server (1300) receives a voice signal to be recognized, it can process the voice signal and output the word ID (2220) corresponding to the sound unit ID (2230) of the voice signal as a result. At this time, the mapping table (1273) is not transmitted to the server (1300) but is stored only in the terminal (1200), thereby protecting personal information (1272) even if the personalized model is exposed to the outside.

Thereafter, the terminal (1200) receives the result of data processing using the ID-based personalized model (1372) for the voice signal from the server (1300) at step 1640. For example, the result of data processing using the ID-based personalized model (1372) at the server may include a word ID (2220) as illustrated in FIG. 22.

Next, at step 1650, the terminal (1200) restores the personal information (1272) or additional information corresponding to the ID using the data processing result and the mapping table (1273) received from the server (1300). That is, when the terminal (1200) receives the word ID (2220) as shown in FIG. 22 from the server (1300) as the data processing result, the terminal (1200) can restore the personal information corresponding to the word ID (2220) using the stored word mapping table (1273). Referring to FIG. 20, the terminal (1200) can restore the ID 0x01 to ‘Hong Gil-dong.’ According to the disclosed embodiment, the terminal (1200) can complete data processing by restoring the personal information (1272) masked by the ID using the mapping table (1273). The terminal (1200) generates a personalized model (1372) to enable the voice processing system to have high performance, and actual data processing is performed in a server (1300) with high computing power, thereby enabling voice signals to be processed quickly.

Furthermore, the terminal (1200) can output the data processing result to the user.

As a result, according to the disclosed embodiment, personal information (1272) is transmitted to the server (1300) in a form in which the content of words or sentences cannot be identified, so that even if the personalized model is exposed externally, it is difficult to restore the personal information (1272) masked with an ID, thereby protecting the personal information (1272). In addition, by transmitting the personalized model to a server with high computational capability and performing data processing on the voice signal, a voice processing system with higher performance and processing speed can be implemented.

FIG. 17 is a flowchart showing a voice processing method of a server according to another disclosed embodiment.

First, the server (1300) receives an ID-based personalized model (1273) from the terminal (1200) at step 1710. The ID-based personalized model (1273) has a portion of personal information (1272) masked with an ID, so that even if the personalized model is exposed externally, it is difficult to restore the personal information (1272) masked with an ID, thereby protecting the personal information (1272).

After that, the server (1300) receives a voice signal at step 1720. The server (200) can receive a voice signal through various components. Receiving a voice signal from a terminal (100) would be the most common form, but the voice signal can also be received through a USB interface unit or a DVD interface unit, etc.

Next, the server (1300) processes data using an ID-based personalized model (1273) for the voice signal at step 1730. In this case, the server (1300) can control the terminal (1200) to indicate a personal information ID using an ID mapped to an acoustic unit according to a prior agreement. This will be described with reference to FIGS. 21 and 22.

Referring to FIG. 21, the server (1300) maps an ID to an acoustic unit according to an agreement with the terminal (1200). The server (1300) recognizes the voice signal ‘Hong Gil-dong’ as a set of continuous sounds. Accordingly, by assigning 0x101, which is an ID previously agreed upon with the terminal (1200), to the sound ‘HH’ and 0x102 to the sound ‘OW’, the voice signal ‘Hong Gil-dong’ can be expressed as a set of acoustic unit IDs, 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, …

The server (1300) can find and match a word ID corresponding to the set of pronunciation symbol IDs in the personalized model (1273) received from the terminal (1200). Referring to FIG. 22, the word ‘Hong Gil-dong’ is mapped with an ID of 0x01. Accordingly, the server (1300) can represent the voice signal ‘Hong Gil-dong’ as a word ID of 0x01 corresponding to the set of pronunciation symbol IDs of 0x101, 0x102, 0x103, 0x104, 0x105, 0x106, … The server (1300) can have high performance by receiving the personalized model (1372) from the terminal (1200) and performing data processing, and further, since the server (1300) has high computational capability, it can quickly process the voice signal.

Furthermore, the server (1300) transmits the data processing result to the terminal (1200) at step 1740.

As a result, according to the disclosed embodiment, personal information (1272) is transmitted to the server (1300) in a form that cannot be understood as the content of words or sentences, so that even if the personalized model is exposed externally, it is difficult to restore the personal information (1272) masked with an ID, thereby protecting the personal information (1272). In addition, by performing data processing on a voice signal using the personalized model in a server (1300) with high computational capabilities, a voice processing system with higher performance and processing speed can be implemented.

FIG. 23 is a flowchart showing an example of a specific operation process of a terminal and a server according to another disclosed embodiment.

First, in step 2310, the terminal (1200) creates a mapping table (1273) by matching an ID to personal information (1272). The terminal (1200) can create a pronunciation dictionary (1910) using the word list of the personal information (1272) and map the ID to the pronunciation symbol and word using the pronunciation dictionary (1910). In this case, the terminal (1200) can assign an arbitrary ID.

Next, in step 2320, the terminal (1200) generates an ID-based personalized model (1372) using the mapping table (1273), and transmits it to the server (1300) in step 2330. The server (1300) stores the ID-based personalized model (1372) received in step 2340 in the storage unit (1370). The ID-based personalized model (1372) can be generated based on the word ID (2220) and sound unit ID (2230) illustrated in FIG. 22.

Thereafter, the terminal (1200) receives a voice signal at step 2350 and transmits it to the server (1300) at step 2360. As described above, the terminal (100) can receive a voice signal through various components. Receiving a voice signal through a microphone unit would be the most common form, but the voice signal can also be received through a USB interface unit or a DVD interface unit, etc. Furthermore, the voice signal can also be received through communication with an external device.

The server (1300) processes data using an ID-based personalized model for the voice signal received at step 2370, and transmits the data processing result to the terminal (1200) at step 2380. In this case, the server (1300) may display an ID mapped to personal information (1272) or additional information using an ID mapped to an acoustic unit according to an agreement with the terminal (1200).

After that, at step 2390, the terminal (1200) restores personal information (1272) or additional information corresponding to the ID using the data processing result and the mapping table (1273).

According to the disclosed embodiment, the server (1300) receives an ID-based personalized model from the terminal (1200) and processes data using the received ID-based personalized model, thereby making it difficult to restore personal information (1272) masked by the ID even if the personalized model is exposed to the outside, thereby protecting personal information (1272). In addition, by having the server (1300) with high computational capability perform data processing on a voice signal, a voice processing system with higher performance and processing speed can be implemented.

Meanwhile, the above-described embodiments can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium.

The computer-readable storage medium includes a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.), an optical readable medium (e.g., CD-ROM, DVD, etc.), and a carrier wave storage medium (e.g., transmission via the Internet).

Although the embodiments have been described with reference to the above and the attached drawings, those skilled in the art to which the disclosed embodiments pertain will understand that the disclosed embodiments can be implemented in other specific forms without changing the technical idea or essential characteristics thereof. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

100: Terminal
110: Receiver
130: Communications Department
150: Control Unit
200: Server
210: Receiver
230: Communications Department
250: Control Unit

Claims (12)

A step of receiving a speech signal;
A step of detecting a personalized information section including personal information among the above voice signals;
A step of transmitting the above voice signal to a server, but transmitting the personal information included in the personalized information section in a form in which the content of the words or sentences cannot be identified;
A step of receiving a data processing result for the voice signal from the server; and
A step of supplementing the data processing result received from the server by reprocessing the voice signal using the personal information included in the personalized information section and the personalized model, which is a customized voice processing model for each user; and
A method for processing a voice signal of a terminal, comprising a step of providing a result of the above data processing.
In the first paragraph,
A step of detecting a general information section other than the above personalized information section;
A voice signal processing method of a terminal, characterized in that it further includes a step of generating voice section information for the personalized information section and the general information section and transmitting the generated voice section information to the server.
In the second paragraph,
The above voice section information is,
A voice signal processing method of a terminal, characterized in that it includes section marking information that marks at least one section among the personalized information section and the general information section among the above voice signals.
In the second paragraph,
The data processing result for the voice signal corresponding to the general information section received from the above server is as follows:
A voice signal processing method of a terminal, characterized in that the server processes a voice signal corresponding to the general information section using a general model.
In the first paragraph,
The step of processing data for the voice signal based on the data processing result received from the server is as follows:
A voice signal processing method of a terminal, comprising a step of data processing for the personalized information section of the voice signal using a personalized model generated based on the personal information.
In the fifth paragraph, the personalized model,
A method for processing a speech signal of a terminal, the terminal including at least one model among a personalized speech recognition model, a personalized natural language understanding model, and a personalized lexical model.
A receiver for receiving a voice signal;
A communication unit that performs communication with the server; and
Detecting a personalized information section containing personal information from the above voice signal,
The above voice signal is transmitted to the server, but the personal information included in the personalized information section is transmitted in a form in which the content of the words or sentences cannot be identified.
Receive the data processing result for the voice signal from the above server,
The data processing result received from the server is supplemented by reprocessing the voice signal using the personal information included in the personalized information section and the personalized model, which is a customized voice processing model for each user.
A terminal characterized by including a control unit that controls to provide the result of the above data processing.
In the seventh paragraph, the control unit,
Detecting a general information section other than the above personalized information section,
A terminal that generates voice section information for the above personalized information section and the above general information section and transmits it to the server.
In Article 8,
The above voice section information is,
A terminal characterized by including section marking information that marks at least one section among the personalized information section and the general information section among the above voice signals.
In Article 8,
The data processing result for the voice signal corresponding to the general information section received from the above server is as follows:
A terminal characterized in that the server processes a voice signal corresponding to the general information section using a general model.
In the seventh paragraph, the control unit,
A terminal characterized in that it processes data for the personalized information section of the voice signal using a personalized model generated based on the personal information.
In the 11th paragraph, the personalized model,
A terminal characterized by including at least one model among a personalized speech recognition model, a personalized natural language understanding model, and a personalized lexical model.