KR102755648B1 - Artficial intelligence-based multilingual interpretation system and service method - Google Patents

Abstract

The present invention relates to an artificial intelligence-based multilingual interpretation system and service method, and more specifically, to an artificial intelligence-based multilingual interpretation system and service method capable of providing a service in which a translated text and a translated voice can be output in a customized manner according to an output mode for each user. To this end, the artificial intelligence-based multilingual interpretation system includes a speech recognition unit that converts a target speech in a first language into a first output text using a deep learning-based language-specific speech recognition model, an automatic translation unit that translates the first output text into a second output text in a second language through an automatic translation algorithm, a speech synthesis unit that synthesizes a translated speech in a second language for the second output text based on speech features analyzed from the target speech through HIFI-GAN technology, and an integrated service unit that provides a service in which the second output text and the translated speech can be output in a customized manner according to an output mode for each user registered through a web-based interpretation service, wherein the web-based interpretation service is a webpage that can be accessed from each customer terminal through a network.

Description

{ARTFICIAL INTELLIGENCE-BASED MULTILINGUAL INTERPRETATION SYSTEM AND SERVICE METHOD}

The present invention relates to an artificial intelligence-based multilingual interpretation system and service method, and more specifically, to an artificial intelligence-based multilingual interpretation system and service method capable of providing a service in which translated text and translated voice can be output in a customized manner according to an output mode for each user.

With the development of transportation and communication, human and material exchanges between countries have become more active. Despite this expansion of human and material exchanges, different languages between countries are acting as a barrier to communication.

Translation is the conversion between letters in different languages to alleviate the inconvenience caused by different languages. And the spoken language translation system refers to the conversion between the sounds of different languages, and broadcast news interpretation is an example of this.

In particular, the discomfort between speakers in different languages at various conferences between countries is being resolved through simultaneous interpretation. Simultaneous interpretation, which was once the exclusive domain of simultaneous interpreters, is now being automatically interpreted by machines due to the advancement of technologies such as speech recognition, automatic translation, and speech synthesis.

At this time, automatic interpretation is the process and technology of converting the speech of a first language into the speech of a second language in both directions. Specifically, automatic interpretation refers to the process and technology of converting the speech of a first language into a second language through processes such as speech recognition and automatic translation, and outputting it as subtitles or making it heard through a speaker after voice synthesis.

The present invention aims to provide an artificial intelligence-based multilingual interpretation system and service method capable of outputting translated text and translated voice obtained through automatic interpretation in a customized manner according to the output mode for each user.

The present invention is intended to solve the above problems, and an object of the present invention is to provide an artificial intelligence-based multilingual interpretation system and service method capable of providing a service in which translated text and translated voice can be output in a customized manner according to the output mode of each user.

The above and other objects and advantages of the present invention will become apparent from the following description of preferred embodiments.

In order to achieve the above object, according to one embodiment of the present invention, an AI-based multilingual interpretation system includes a speech recognition unit which converts a target speech in a first language into a first output text using a deep learning-based language-specific speech recognition model, an automatic translation unit which translates the first output text into a second output text in a second language using an automatic translation algorithm, a speech synthesis unit which synthesizes a translation speech in a second language for the second output text based on speech features analyzed from the target speech using an end-to-end speech synthesis technology such as HIFI-GAN, and an integrated service unit which provides a service so that the second output text and the translation speech can be output in a customized manner according to an output mode for each user registered through a web-based interpretation service, wherein the web-based interpretation service is a webpage which can be accessed from each customer terminal through a network.

In order to achieve the above object, according to one embodiment of the present invention, a method for providing artificial intelligence-based multilingual interpretation includes the steps of: a voice recognition unit receiving a target voice in a first language recorded from a customer terminal through a web-based interpretation service; a voice recognition unit converting and processing the target voice into a first output text using a deep learning-based language-specific voice recognition model; an automatic translation unit translating and processing the first output text into a second output text in a second language using an automatic translation algorithm; a voice synthesis unit synthesizing a translation voice in a second language for the second output text based on voice features analyzed from the target voice through an end-to-end voice synthesis technology such as HIFI-GAN; and a service step of an integrated service unit enabling the second output text and the translation voice to be output in a customized manner to the customer terminal according to a user-specific output mode pre-registered through the interpretation service.

According to an embodiment of the present invention, by providing a service in which translated text and translated voice can be output in a customized manner according to the output mode of each user, more accurate understanding of simultaneous interpretation can be increased, while convenience can be improved.

FIG. 1 is a diagram schematically illustrating an artificial intelligence-based multilingual interpretation system (1000) according to an embodiment of the present invention.
Figures 2(A) to 2(D) are examples showing web-based interpretation services.
Fig. 3 is a block diagram showing an embodiment of the automatic translation unit (200) of Fig. 1.
Figure 4 is a block diagram showing one embodiment of the integrated service unit (400) of Figure 1.
Figure 5 is a block diagram specifically explaining the decision unit (420) of Figure 4.
Figure 6 is a flowchart schematically illustrating an artificial intelligence-based multilingual interpretation service method of the multilingual interpretation system (1000) of Figure 1.
Figure 7 is an operation process specifically showing the operation of the automatic translation unit (200) of Figure 3.
Figure 8 is an operation process specifically showing the operation according to one embodiment of the integrated service unit (400) of Figure 4.

Hereinafter, the present invention will be described in detail with reference to examples and drawings of the present invention. It will be apparent to those skilled in the art that these examples are merely presented as examples to more specifically explain the present invention, and that the scope of the present invention is not limited by these examples.

Additionally, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, and in case of conflict, the description in this specification, including definitions, shall prevail.

In order to clearly explain the invention proposed in the drawings, parts that are not related to the description have been omitted, and similar parts have been given similar drawing reference numerals throughout the specification. In addition, when a part is said to "include" a certain component, this does not mean that other components are excluded, but rather that other components can be further included, unless otherwise specifically stated. In addition, a "part" described in the specification means a single unit or block that performs a specific function.

The identification codes (1st, 2nd, etc.) for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than stated unless the context clearly indicates a specific order. That is, each step may be performed in the same order as stated, may be performed substantially simultaneously, or may be performed in the opposite order.

FIG. 1 is a diagram schematically showing an artificial intelligence-based multilingual interpretation system (1000) according to an embodiment of the present invention, and FIGS. 2(A) to 2(D) are examples showing web-based interpretation services.

Referring to FIGS. 1 to 2(D), an artificial intelligence-based multilingual interpretation system (1000) may include a voice recognition unit (100), an automatic translation unit (200), a voice synthesis unit (300), and an integrated service unit (400).

First, the voice recognition unit (100) can recognize the target voice of the first language and convert it into the first output text using a deep learning-based language-specific voice recognition model.

Here, the deep learning-based language-specific speech recognition model can be an end-to-end artificial neural network based on continuous speech to recognize voice, pronunciation, and content.

For example, a continuous speech-based end-to-end artificial neural network recognizes continuous speech as a sequence and converts it into text, so it is more accurate than existing speech recognition technology and can be used in various environments.

According to one embodiment, the voice recognition unit (100) can remove preset noise and distortion from a target voice of a first language, amplify the voice volume, and then normalize the voice frame and voice spectrum according to a preset preprocessing range.

Here, the preset preprocessing range may include preset sound size, frequency and time.

According to another embodiment, the voice recognition unit (100) may process pre-collected language-specific speech data, field-specific speech data, and symbol and number expression-related data into refined learning data for a deep learning-based language-specific speech recognition model through a preset refinement technique that removes noise, improves sound quality, and classifies the collected language-specific speech data, field-specific speech data, and symbol and number expression-related data. Here, the refined learning data may be labeled with language-specific tags.

According to another embodiment, the speech recognition unit (100) may perform a check for consistency between the target speech of the first language and the first output text, and based on the result of the check, non-matching spoken speech-to-text data may be registered in advance.

At this time, the voice recognition unit (100) can perform a check in advance for non-matching spoken voice-text data based on the first output text converted from the target voice of the first language.

According to another embodiment, the voice recognition unit (100) can correct semantic errors, grammatical and spelling errors, and unnecessary words in the first output text converted from the target voice of the first language by using entity recognition technology that recognizes location or situational text, technology that recognizes text tagged with a predetermined crisis situation level, and natural language processing technology based on situational keywords.

Next, the automatic translation unit (200) can translate the first output text into a second output text in a second language through an automatic translation algorithm.

Here, the automatic translation algorithm may be at least one of a preset self-attention transformer-based, pre-trained, end-to-end neural machine translation (NMT) algorithm to translate the first output text into the second output text.

At this time, at least one translation algorithm can learn in advance a multilingual domain-specific translation corpus, a civil affairs translation corpus, a tourism-related translation corpus, a transportation-related translation corpus, a road guidance-related translation corpus, a shopping-related translation corpus, a food-related translation corpus, and a terminology-related translation corpus.

According to one embodiment, when collecting each corpus data for translation to learn an automatic translation algorithm, the automatic translation unit (200) may filter and remove in advance sentences that are outside a preset length range, sentences that include a preset number of symbols, asymmetric sentences with asymmetry between the original text and the translated text, sentences with translated words, sentences with inconsistent expressions, and sentences with data bias (suffering).

According to another embodiment, the automatic translation unit (200) may tag each meaning of a polysemous word extracted from each corpus data for translation by sentence, adjust the corpus distribution ratio by field according to a preset ratio, and individually set the token for setting the word expression unit for the automatic translation algorithm, the vocab size for setting the number of words for learning, and the number of model learning times to preset values.

Next, the voice synthesis unit (300) can synthesize and output a translation voice of a second language for a second output text based on voice features analyzed from a target voice of a first language through an end-to-end voice synthesis technology such as HIFI-GAN.

Here, voice features may include pitch, volume, and timbre of the voice.

At this time, end-to-end voice synthesis technology such as HIFI-GAN can be a technology for generating realistic voices by inputting a Mel Spectrogram extracted from a target voice.

Next, the integrated service department (400) can provide a service in which the second output text and the translation voice can be customized for each customer terminal (10_1 to 10_N) according to the output mode of each registered user through a web-based interpretation service.

Here, the web-based interpretation service is a web page that can be accessed through a network from each customer terminal (10_1 to 10_N), as shown in FIGS. 2(A) to 2(D), and may include various program menu tools for providing multilingual interpretation services.

These web-based interpretation services can be implemented to be linked with call center servers, or can be implemented as two-way tablet kiosk devices for public consultation, smart device terminal applications, or PC programs. For example, web-based interpretation services can be implemented as applications and used for interpretation services in places frequently visited by foreigners, including community centers (district offices), administrative welfare centers, 119 safety centers, tourist information centers, health care centers, public health centers, pharmacies, general hospitals, general hospitals, hospitals, oriental medicine hospitals, and dental hospitals.

At this time, the user-specific output mode may include first and second individual output modes, a simultaneous output mode, and a sequential repetition learning mode. Specifically, the first individual output mode is a mode in which a translated voice is output and then a second output text is output, the second individual output mode is a mode in which the second output text is output and then a translated voice is output, the simultaneous output mode is a mode in which the translated voice and the second output text are output simultaneously, and the sequential repetition learning mode may be a mode in which the translated voice and the second output text are output sequentially and repeatedly.

According to one embodiment, the integrated service unit (400) may perform cross-translation verification on the second output text through a preset multilingual translation web service based on the dialogue importance identified from the first output text.

For example, in the case of daily conversation, the integrated service department (400) may skip cross-translation verification for the second output text through a preset multilingual translation web service as the conversation importance for the first output text is identified as lower in importance. In addition, in the case of hospital or emergency conversation, the integrated service department (400) may perform cross-translation verification for the second output text through a preset multilingual translation web service as the conversation importance for the first output text is identified as higher in importance.

According to another embodiment, the integrated service unit (400) may skip the operation of the voice synthesis unit (300) and output the corresponding second output text through the interpretation service webpage based on the hearing impairment level confirmed from the customer information transmitted from the customer terminal (10) through the web-based interpretation service.

According to another embodiment, the integrated service unit (400) may adjust the weight for the automatic translation algorithm based on the translation grade scores between the first and second output texts evaluated by experts.

According to another embodiment, the integrated service unit (400) may correct preset control variable values applied to an end-to-end speech synthesis technology such as HIFI-GAN based on a similarity score calculated through a preset similarity calculation formula between a target voice in a first language and a translated voice in a second language.

Hereinafter, the configuration of the present invention and the effects thereof will be described in more detail through specific examples and comparative examples. However, these examples are intended to explain the present invention more specifically, and the scope of the present invention is not limited to these examples.

Figure 3 is a block diagram showing an embodiment of the automatic translation unit (200) of Figure 1.

Referring to FIGS. 1 to 3, the automatic translation unit (200) may include first and second search units (210, 220) and a translation evaluation unit (230).

First, the first search unit (210) can search for a first word image corresponding to at least one feature word of the first language detected from the first output text through a web search.

Next, the second search unit (220) can search for a second word image corresponding to at least one feature word of the second language detected from the second output text through a web search.

Next, the translation evaluation unit (230) can evaluate the translation accuracy of the automatic translation algorithm based on whether there is a correspondence between the first and second word images.

According to an embodiment, when the translation accuracy for the automatic translation algorithm is evaluated to be higher than a preset value, the integrated service department (400) may tag the first and second word images as associated images in the second output text and provide the service.

Figure 4 is a block diagram showing one embodiment of the integrated service unit (400) of Figure 1.

Referring to FIGS. 1 to 4, the integrated service unit (400) may include an interpretation test unit (410), a decision unit (420), a gap time decision unit (430), and an output control unit (440).

First, the interpretation test department (410) provides an interpretation grade test to analyze the interpretation ability of a registered user to each customer terminal (10_1 to 10_N), and based on each response information received, the subtitle comprehension grade and voice comprehension grade for each user can be determined.

Next, the decision unit (420) can determine the user-specific output mode for the second output text and the translated voice as one of the first and second individual output modes, the simultaneous output mode, and the sequential repetition learning mode based on the subtitle understanding level and the voice understanding level for each user.

Next, the gap time determination unit (430) can determine the output gap time between the second output text and the translated voice based on the speech rate and speech amount confirmed from the target voice of the first language.

For example, if the firing speed is higher than the preset speed and the firing amount is higher than the standard value, the gap time determination unit (430) can determine the output gap time to be a certain period longer than the standard time.

Next, the output control unit (440) determines the output order for the second output text and the translated voice based on the output gap time and the user-specific output mode, and can output the second output text and the translated voice to each customer terminal (10_1 to 10_N) through a web-based interpretation service.

According to an embodiment, the output control unit (440) may forcibly switch one output mode to the first individual output mode based on a preset psychological state analyzed from the target voice of the first language.

For example, if the psychological state analyzed from the target voice of the first language is analyzed as a state of tension, the output control unit (440) can support the ability to identify the mood state of the translated voice in advance by forcibly switching one output mode to the first individual output mode.

Figure 5 is a block diagram specifically explaining the decision unit (420) of Figure 4.

Referring to FIGS. 4 and 5, the decision unit (420) may include first to fourth mode registration units (421 to 424).

First, the first mode registration unit (421) can register the user-specific output mode for the second output text and the translated voice as the first individual output mode when the subtitle understanding level is lower than the preset threshold level and the voice understanding level is higher than the preset threshold level.

Next, the second mode registration unit (422) can register the user-specific output mode for the second output text and the translated voice as the second individual output mode when the voice understanding level is lower than the preset threshold level and the subtitle understanding level is higher than the preset threshold level.

Next, the third mode registration unit (423) can register the user-specific output mode for the second output text and the translated voice as a simultaneous output mode when the voice understanding level and subtitle understanding level are equal to or higher than a preset threshold level.

Next, the fourth mode registration unit (424) can register the user-specific output mode for the second output text and the translated voice in a sequential repetition learning mode when the voice understanding level and subtitle understanding level are below a preset threshold level.

Figure 6 is a flowchart schematically illustrating an artificial intelligence-based multilingual interpretation service method of the multilingual interpretation system (1000) of Figure 1.

Referring to FIG. 1 and FIG. 6, first, at step S110, the voice recognition unit (100) can receive a target voice in a first language recorded through a web-based interpretation service from any one customer terminal (e.g., 10_1).

Then, at step S120, the speech recognition unit (100) can convert the target speech of the first language into the first output text using a deep learning-based language-specific speech recognition model.

At this time, at step S130, the automatic translation unit (200) can translate the first output text converted through the voice recognition unit (100) into a second output text in a second language through an automatic translation algorithm.

According to an embodiment, at step S130, the integrated service unit (400) may perform cross-translation verification on the second output text through a preset multilingual translation web service based on the dialogue importance identified from the first output text.

For example, if the conversation importance is identified as being higher than a certain level, the integrated service unit (400) can perform cross-translation verification for the second output text through a multilingual service. Also, if the conversation importance is identified as being lower than a certain level, the integrated service unit (400) can skip cross-translation verification for the second output text through a multilingual service of OpenAI's whisper, Amazon TTS, and Google ASR.

Then, at step S140, the voice synthesis unit (300) can synthesize a second language translation voice for the second output text based on voice features analyzed from the target voice through an end-to-end voice synthesis technology such as HIFI-GAN.

Thereafter, at step S150, the integrated service department (400) can provide a service to output the second output text and the translation voice in a customized manner to one customer terminal (e.g., 10_1) according to the output mode of each registered user through a web-based interpretation service.

Figure 7 is an operation process specifically showing the operation of the automatic translation unit (200) of Figure 3.

Referring to FIGS. 3, 6, and 7, first, in step S210, the automatic translation unit (200) can search for a first word image corresponding to at least one first feature word detected from the first output text through a web search.

Then, at step S220, the automatic translation unit (200) can search for a second word image corresponding to at least one first feature word detected from the second output text through a web search.

Thereafter, the automatic translation unit (200) can evaluate the translation accuracy for the automatic translation algorithm based on whether each output probability value corresponds as the first and second word images are applied to the image classification model.

Figure 8 is an operation process specifically showing the operation according to one embodiment of the integrated service unit (400) of Figure 4.

Referring to FIGS. 4 and 8, first, at step S310, the integrated service department (400) provides an interpretation level test for analyzing the user's interpretation ability to one customer terminal (e.g., 10_1), and based on the response information received, the subtitle understanding level and voice understanding level for the user can be determined.

Then, at step S320, the integrated service unit (400) can determine a user-specific output mode for the second output text and the translated voice based on the subtitle understanding level and the voice understanding level.

Specifically, when the subtitle comprehension level is lower than the preset threshold level and the voice comprehension level is higher than the preset threshold level, the integrated service unit (400) can register the user-specific output mode as the first individual output mode.

In addition, if the voice understanding level is lower than the preset threshold level and the subtitle understanding level is higher than the preset threshold level, the integrated service department (400) can register the user-specific output mode as the second individual output mode.

In addition, if the voice understanding level and the subtitle understanding level are higher than the preset threshold level, the integrated service department (400) can register the user-specific output mode as a simultaneous output mode.

In addition, if the voice understanding level and subtitle understanding level are below the preset threshold level, the integrated service department (400) can register the user-specific output mode as a sequential repetition learning mode.

At this time, in step S330, the integrated service unit (400) can determine the output gap time between the second output text and the translated voice based on the speech rate and speech amount confirmed from the target voice.

Then, at step S340, the integrated service department (400) can determine the output order for the second output text and the translated voice based on the output gap time and the user-specific output mode and output them through a web-based interpretation service.

Furthermore, a computer-readable recording medium including a program for executing an artificial intelligence-based multilingual interpretation service according to an embodiment of the present invention can be implemented.

At this time, the AI-based multilingual interpretation service can be implemented to include a step in which a voice recognition unit receives a target voice in a first language recorded from a customer terminal through a web-based interpretation service, a step in which the voice recognition unit converts and processes the target voice in the first language into a first output text using a deep learning-based language-specific voice recognition model, a step in which an automatic translation unit translates and processes the first output text into a second output text in a second language using an automatic translation algorithm, a step in which a voice synthesis unit synthesizes and outputs a translation voice in a second language for the second output text based on voice features analyzed from the target voice through an end-to-end voice synthesis technology such as HIFI-GAN, and a step in which an integrated service unit provides a service so that the second output text and the translation voice can be output in a customized manner according to a pre-registered user-specific output mode through a web-based interpretation service.

Although this specification describes only a few examples among various embodiments performed by the inventors of the present invention, the technical idea of the present invention is not limited or restricted thereto, and can be modified and implemented in various ways by those skilled in the art.

10_1~10_N: Each customer terminal
100: Voice recognition unit
200: Machine translation department
300: Voice synthesis section
400: Integrated Services Department
1000: AI-based multilingual interpretation system

Claims (9)

A speech recognition unit that converts and processes a target speech of a first language into a first output text using a deep learning-based language-specific speech recognition model;
An automatic translation unit that translates the first output text into a second output text in a second language using an automatic translation algorithm;
A speech synthesis unit that synthesizes a second language translation speech for the second output text based on speech features analyzed from the target speech of the first language through an end-to-end speech synthesis technology such as HIFI-GAN; and
Includes an integrated service section that provides a service to output the second output text and the translated voice in a customized manner according to the output mode for each registered user through a web-based interpretation service.
The above web-based interpretation service is a web page that can be accessed through a network from each customer terminal.
The above user-specific output modes include first and second individual output modes, simultaneous output mode, and sequential repetition learning mode.
The above first individual output mode is a mode that outputs the translated voice and then outputs the second output text.
The above second individual output mode is a mode that outputs the second output text and then outputs the translated voice.
The above simultaneous output mode is a mode that outputs the translation voice and the second output text simultaneously.
The above sequential repetition learning mode is a mode that sequentially repeats and outputs the translated voice and the second output text.
The automatic translation unit includes a first search unit that searches for a first word image corresponding to at least one first feature word detected from the first output text through a web search;
A second search unit that searches for a second word image corresponding to at least one first feature word detected from the second output text through a web search; and
The first and second word images above It includes a translation evaluation unit that evaluates the translation accuracy of the automatic translation algorithm based on whether or not the liver corresponds,
The above integrated service department provides a service by tagging the first and second word images as related images to the second output text when the translation accuracy of the automatic translation algorithm is evaluated to be higher than a preset value.
The above integrated service department provides an interpretation level test to each customer terminal, and based on each response information received, an interpretation test department determines the subtitle comprehension level and voice comprehension level for each user;
A decision unit that determines the output mode for each user based on the subtitle comprehension level and the voice comprehension level;
A gap time determination unit that determines an output gap time between the second output text and the translated voice based on the speech rate and speech amount identified from the target voice; and
Based on the output gap time and the user-specific output mode, an output control unit is included that determines the output order for the second output text and the translated voice and outputs them through the interpretation service.
The above decision unit comprises a first mode registration unit that registers the user-specific output mode as a first individual output mode when the subtitle understanding level is lower than a preset threshold level and the voice understanding level is higher than the preset threshold level;
A second mode registration unit that registers the user-specific output mode as a second individual output mode when the above-mentioned voice understanding level is lower than a preset threshold level and the above-mentioned subtitle understanding level is higher than the preset threshold level;
A third mode registration unit that registers the user-specific output mode as a simultaneous output mode when the above voice understanding level and the above subtitle understanding level are equal to or higher than a preset threshold level; and
If the above voice understanding level and the above subtitle understanding level are below a preset threshold level, a fourth mode registration unit is included that registers the user-specific output mode as a sequential repetition learning mode.
The above output control unit forcibly switches the user-specific output mode to the first individual output mode based on the preset psychological state analyzed from the target voice,
The above integrated service department selectively performs cross-translation verification on the second output text through a preset multilingual translation web service based on the dialogue importance identified from the first output text.
An artificial intelligence-based multilingual interpretation system, wherein the speech recognition unit performs a check for consistency between the target speech of the first language and the first output text, and based on the results of the check, pre-registers non-matching speech-to-text data, and performs a check for non-matching speech-to-text data when converting the target speech of the first language into the first output text.













delete delete delete delete delete delete delete delete