KR102487847B1 - System and method for providing call service for the hearing impaired - Google Patents

Abstract

The present invention relates to a system and method for providing a call service for the hearing impaired and, more particularly, to a system and method for providing a call service for the hearing impaired that can provide a call service between the hearing impaired and non-disabled people by utilizing functions such as text-to-speech conversion, voice-to-text conversion, and motion-text conversion while visualizing or expressing an emotional state estimated from the voice or facial expression of a speaker, thereby enabling clear communication between callers.

Description

System and method for providing call service for the hearing impaired

The present invention relates to a system and method for providing a call service for the hearing impaired, and more particularly, provides a call service between the hearing impaired and non-disabled people by utilizing functions such as text-to-speech conversion, voice-to-text conversion, and motion-text conversion, and at the same time The present invention relates to a system and method for providing a call service for the hearing impaired that enables clear communication between callers by visualizing or expressing an emotional state estimated from a speaker's voice or facial expression.

Although the quality of life is improving with the development of information and communication, it is difficult for the hearing-impaired and speech-impaired people to access information and communication due to the lack of devices for the disabled, especially communication devices for the hearing-impaired or speech-impaired in daily life or public facilities. are doing

In some TV broadcasts, a sign language expert interprets the broadcasting voice into sign language for the hearing impaired and displays the contents at the bottom of one side of the TV broadcasting screen. However, it is a reality that it is very difficult for the hearing impaired to carry out a personal call using a communication device such as a smart phone.

Although an application that can use the sign language interpretation function for calls by the hearing impaired has been developed, this service is performed only for user devices that have the same application installed, so it is difficult to apply it to call connections between non-disabled people and people with disabilities. There is a difficult problem.

Recently, Text To Speech (TTS) technology that converts text to voice, Speech To Text (STT) technology that converts voice to text, and Motion Text that recognizes sign language motion and converts it into text As the Motion To Text (MTT) technology develops, various studies are being conducted to enable a call or video call between the hearing impaired and non-disabled people using these technologies. For example, Republic of Korea Patent Registration No. 10-2212298 discloses an artificial intelligence-based video communication platform system between non-disabled people and hearing-impaired people and its operation method.

The prior art uses an artificial intelligence relay server unit including the above-described TTS conversion module, STT conversion module, MTT conversion module, translation module, etc. to use a first communication terminal used by a non-disabled person and a second communication terminal used by a hearing impaired person. Although there is a technical feature in enabling video communication between communication terminals, since the same type of audio output is made for each user performing video communication regardless of the content of the communication, the emotions of the users cannot be properly conveyed. There are downsides.

In other words, in the case of a call using a communication terminal, in order to accurately convey each other's intentions, it is not only necessary to convey what the speaker wants to say through voice or text, but also to express the emotions of the speaker or the atmosphere inherent in the content to be said. It is important to express, but a system that conveys the emotions of the hearing impaired to the other person or conveys the emotions of the other person to the hearing impaired has not yet been developed.

1. Republic of Korea Patent Registration No. 10-2212298 (registered on January 29, 2021)

The present invention has been devised to solve the problems of the prior art as described above, and an object of the present invention is to visualize or categorize the emotional state estimated from the speaker's voice or expression, or to communicate it to the other party through voice. It is an object of the present invention to provide a system and method for providing a call service for the hearing-impaired so that communication between callers can be made more clearly.

In addition, the present invention enables the communication weak, represented by the hearing impaired, to communicate with the other party more easily and conveniently using a communication terminal, while clearly conveying their intention with emotion, as well as to communicate the other party's intention Another object is to provide a system and method for providing a call service for the hearing-impaired to be clearly identified.

The present invention for achieving the above objects,

In the system for providing a call service for the hearing impaired, including a first communication terminal possessed by a hearing impaired person, a second communication terminal possessed by a non-disabled person, and a relay server for relaying a call between the first and second communication terminals, The relay server includes a communication module for wireless communication with the first and second communication terminals, a voice generation module for generating a voice of a hearing-impaired person using a signal input from the first communication terminal, and the first communication terminal. and a conversion module for converting information received from the second communication terminal into a form recognizable by the other party, and an emotion display module for displaying emotional information of the speaker using the information received from the first and second communication terminals. It is characterized by including.

At this time, the voice generation module includes a voice extraction unit that primarily generates a unique voice of the hearing-impaired person using body information of the hearing-impaired person input through the first communication terminal, and a hearing-impaired person input through the first communication terminal. Characterized in that it comprises a voice correction unit for correcting the unique voice of the hearing-impaired person using peripheral information including the age of the person and family voice, and a voice selector for selecting the voice of the hearing-impaired person using the first communication terminal. do.

In addition, the emotion display module includes a voice display unit that controls the pitch and volume of the voice of the hearing-impaired person converted by the conversion module to display the emotion of the hearing-impaired person, and a voice display unit that is converted by the conversion module and transmitted to the first communication terminal. It is characterized in that it includes a text display unit that displays the emotion of the non-disabled person by giving a predetermined color to the text of the disabled person.

The emotion display module may further include an avatar display unit that expresses the emotion information displayed by the voice display unit or the text display unit by means of an avatar.

And, the relay server is characterized in that it further comprises a learning module for performing artificial intelligence learning for the algorithm used in the conversion module and the emotion display module based on deep learning.

In addition, the relay server includes a first database for storing information of hearing-impaired people using the first communication terminal, a second database for storing voice data capable of expressing the voice of hearing-impaired people, and artificial intelligence in the learning module. It is characterized in that it includes a third database for storing data that can be used for intelligent learning and learning results learned in the learning module, and a fourth database for storing conversation contents between the first and second communication terminals for a predetermined period. .

In addition, the relay server is an automatic transmission module that automatically transmits the input signal to the other party when the user's signal input using the first and second communication terminals is stopped for a predetermined time or a sentence capable of conveying meaning is completed. It is characterized in that it further comprises.

On the other hand, the method for providing a call service for the hearing impaired according to the present invention,

Call using a call service providing system for the hearing impaired including a first communication terminal possessed by a hearing impaired person, a second communication terminal possessed by a non-disabled person, and a relay server relaying a call between the first and second communication terminals. In the service providing method, a preliminary preparation step of registering information of the hearing impaired in a relay server using the first communication terminal, and a call with the other party by accessing the relay server using the first or second communication terminal. It includes a requesting call request step, and a call connection step in which a hearing-impaired person and a non-disabled person each having a first and second communication terminal carry out a call by relaying by the relay server, wherein the call connection step comprises: A signal conversion step of converting the signal transmitted from the first and second communication terminals into a signal recognizable by the other party in the relay server, and the signal transmitted from the first and second communication terminals or the signal converted in the signal conversion step characterized in that it comprises an emotion display step of displaying the speaker's emotion information.

At this time, the preliminary preparation step is a unique voice generation step of generating a unique voice of a hearing impaired person using information of a hearing impaired person input to the relay server, and based on deep learning, in the signal conversion step and emotion display step It is characterized in that it includes a learning step of performing artificial intelligence learning for the used algorithm.

In addition, the unique voice generation step includes a voice extraction step of extracting the unique voice of the hearing-impaired person using body information of the hearing-impaired person, and the unique voice extracted using surrounding information including the hearing-impaired person's age and family voice. It is characterized in that it includes a voice correction step of correcting and a voice selection step of selecting the final unique voice of the hearing-impaired person using the first communication terminal.

In addition, the signal conversion step includes an MTT conversion step of converting the sign language included in the video of the hearing impaired input from the first communication terminal into a text signal, and converting the voice signal of the non-disabled person input from the second communication terminal into a text signal. A TTS conversion step of converting the text signal converted in the MTT conversion step into a voice signal, and a TTM conversion step of converting the text signal converted in the STT conversion step into a sign language. to be

In addition, the emotion display step includes a voice display step of displaying the emotion of the hearing impaired person by adjusting the pitch and volume of the hearing-impaired person's unique voice converted in the signal conversion step and transmitted from the relay server to the second communication terminal; A text display step of displaying emotional information of a speaker in a text signal transmitted to the first or second communication terminal, and the expression and gesture of the avatar of the hearing impaired or non-disabled person selected by the first or second communication terminal. It is characterized in that it includes an avatar display step of displaying an emotional state.

And, in the call connection step, when the user's signal input using the first and second communication terminals is stopped for more than a predetermined time or a sentence capable of conveying meaning is completed, the input signal is automatically transmitted to the other party. .

According to the present invention, the communication weak represented by the hearing impaired can more easily and conveniently communicate with the other party using the communication terminal, and the conversation between the hearing impaired and non-disabled people, who have difficulty in direct communication with each other, can be freely conducted. In addition, the emotional state estimated from the speaker's voice or facial expression can be visualized or typified, or conveyed to the other party through voice along with the contents of the conversation, so that communication between callers can be more clearly achieved. have

1 is a diagram conceptually showing a system for providing a call service for the hearing impaired according to the present invention.
2 is a diagram conceptually showing a detailed configuration of a relay server in the present invention shown in FIG. 1;
3 is a diagram showing a method of providing a call service for the hearing-impaired in stages according to the present invention.

Hereinafter, preferred embodiments of a system and method for providing a call service for the hearing impaired according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram conceptually showing a call service providing system for the hearing impaired according to the present invention, FIG. 2 is a diagram conceptually showing the detailed configuration of a relay server in the present invention shown in FIG. 1, and FIG. 3 is a diagram showing the present invention It is a diagram showing a method of providing a call service for the hearing impaired step by step according to the present invention.

The present invention provides a call service between a hearing impaired and a non-disabled person by utilizing functions such as text-to-speech conversion, voice-to-text conversion, and motion-text conversion, and at the same time, visualizes the emotional state estimated from the voice or facial expression of the speaker or converts the voice It relates to a system and method for providing a call service for the hearing impaired that enables clear communication between callers by enabling communication between callers through communication. As shown, it may include a first communication terminal 100, a second communication terminal 200 and a relay server 300.

First, the first communication terminal 100 is used by people who have difficulty in general communication, such as people with hearing impairments who have poor language skills due to hearing impairments or people with language disabilities who have difficulty in fully speaking (hereinafter collectively referred to as 'hearing-impaired people'). A normal smartphone, notebook computer, tablet PC, or the like can be used as a terminal having a call function.

At this time, the first communication terminal 100 has functions such as a camera, a webcam, and a microphone, and can obtain video or audio from various external devices, as well as wired/wireless communication from the relay server 300 to be described later. It is possible to receive various signals and information such as video, audio, text, etc.

In addition, the first communication terminal 100 may be provided with various known auxiliary devices that enable the hearing impaired to use them more conveniently.

Next, the second communication terminal 200 is a terminal having a communication function possessed by people who can make general calls, that is, non-disabled people, and may have the same configuration as the first communication terminal 100 described above. to omit

Next, the relay server 300 plays a role of relaying a call between the first and second communication terminals 100 and 200, and provides video and audio between the first communication terminal 100 and the second communication terminal 200. And the text signal can be relayed so that it can be transmitted and received in real time.

In more detail, the relay server 300 may include a communication module 310, a voice generation module 320, a conversion module 330, and an emotion display module 340. First, the communication module 310 is a configuration for performing a wireless communication process with the first and second communication terminals 100 and 200, and various well-known wireless communication networks such as the Internet, WIFI, and LTE may be utilized.

Next, the voice generating module 320 is a component for generating a voice of a hearing-impaired person having the first communication terminal 100, and uses signals input or transmitted from the first communication terminal 100 to provide information to the hearing-impaired person. By generating an appropriate unique voice, the intention of the hearing impaired can be more clearly conveyed to non-disabled people equipped with the second communication terminal 200 .

That is, as shown in FIG. 2, the voice generation module 320 may include a voice extraction unit 322, a voice correction unit 324, and a sound selection unit. The voice extraction unit 322 is the first communication unit. As a component that serves to primarily generate the unique voice of the hearing-impaired people using the terminal 100, the body information of the hearing-impaired people input through the first communication terminal 100, that is, body information such as height, weight, and gender The unique voice of the hearing-impaired person can be primarily generated by using information and physique information such as the upper body of the hearing-impaired person.

Next, the voice correction unit 324 is a component for correcting the unique voice of the hearing-impaired person primarily generated by the voice extraction unit 322, and the surroundings of the hearing-impaired person input through the first communication terminal 100. The unique voice of the hearing-impaired person can be more accurately generated by correcting the height and timbre of the unique voice generated by the voice extraction unit 322 using information, that is, surrounding information such as age, family voice of the same gender, and smoking status. can

Next, the voice selection unit 326 is configured to finally select the unique voice of the hearing-impaired person, and the family or surroundings of the hearing-impaired person uses the first communication terminal 100 to select the voice extraction unit 322 or It is possible to select the unique voice of the hearing-impaired person generated by the voice correction unit 324 .

In addition, when it is difficult to input information of the hearing-impaired person using the first communication terminal 100, that is, body information or surrounding information, it is impossible to generate a unique voice of the hearing-impaired person. The communication terminal 100 may be configured to select one of various voice data stored in the second database 364 of the relay server 300 as the unique voice of the hearing impaired person using the communication terminal 100 .

Next, the conversion module 330 is a component for converting information received from the first and second communication terminals 100 and 200 into a form recognizable by the other party, and is input from the first communication terminal 100 to the hearing impaired. The conversation contents of the non-disabled people who use the second communication terminal 200 are converted into a form recognizable, and at the same time, the conversation contents of the non-disabled people input from the second communication terminal 200 are converted into the first communication terminal 100 can be converted into a form recognizable by the hearing impaired.

More specifically, the conversion module 330 may include an STT conversion unit 332, an MTT conversion unit 334, a TTS conversion unit 336, and a TTM conversion unit 338. First, the STT (Speech To Text) conversion unit is a component for converting a voice signal into text, and can mainly play a role of converting a voice signal of a non-disabled person input through the second communication terminal 200 into a text that can be recognized by a hearing impaired person. .

Next, the MTT (Motion To Text) converter is a component for recognizing a user's motion, that is, a motion, and converting it into text, and the sign language included in the video signal of the hearing impaired input through the first communication terminal 100. It can mainly play a role in converting text into text that non-disabled people can recognize.

Next, the TTS (Text To Speech) conversion unit is a component for converting text into a voice signal, and converts it into a voice signal when a hearing-impaired person inputs the contents of a call as text using the first communication terminal 100, or , It can serve to convert the text converted by the MTT conversion unit 334 into a voice signal.

At this time, as a voice signal converted by the TTS conversion unit 336, the unique voice of the hearing-impaired person generated by the above-described voice generation module 320 can be used as a matter of course.

Next, the TTM (Text To Motion) conversion unit is a component for converting text into motion, that is, a sign language that can be recognized by the hearing impaired, and the text converted by the STT conversion unit 332 or the second communication terminal ( 200) can play a role of converting the input text into a sign language.

As described above, the signal format conversion technology using the conversion module 330 including the STT conversion unit 332, the MTT conversion unit 334, the TTS conversion unit 336, and the TTM conversion unit 338 is already known. Since it is a common technology, a detailed description of this will be omitted.

In addition, the text, voice signal, and sign language converted by the conversion module 330 as described above are transmitted to the first or second communication terminals 100 and 200 by the communication module 310 so that the hearing impaired or non-disabled people can communicate with each other. It makes it easy for doctors to find out.

Next, the emotion display module 340 is configured to display emotion information of the speaker, that is, the hearing impaired and non-disabled people, using the information received from the first and second communication terminals 100 and 200, such emotion The display module 340 is a key component of the present invention to more clearly convey the intention of the speaker, that is, the content to be said to the other party.

More specifically, the emotion display module 340 may include a voice display unit 342, a text display unit 344, and an avatar display unit 346. First, the voice display unit 342 displays the hearing impaired person's emotions. It is a configuration for transmitting to the second communication terminal 200 used by non-disabled people, and the emotions of the hearing-impaired person can be displayed by adjusting the pitch and volume of the hearing-impaired voice converted by the conversion module 330.

That is, the sign language of the hearing impaired input from the first communication terminal 100 is converted into a text form by the MTT conversion unit 334 of the conversion module 330, and the converted text or the first communication terminal 100 The character (text) directly input by the hearing impaired is converted into a voice signal by the TTS conversion unit 336 of the conversion module 330, and the converted voice signal is converted into a hearing impaired person generated by the voice generation module 320. After applying the unique voice of the hearing impaired, by adjusting the pitch and volume of the hearing impaired person's unique voice through the voice display unit 342, the story the hearing impaired person wants to say as well as the emotional state at the time of the call through the second communication terminal 200 It can be passed on to non-disabled people.

Next, the text display unit 344 is configured to display emotion information in text transmitted to the hearing-impaired or non-disabled person, and mainly displays emotion information in the text transmitted to the first communication terminal 100 so that the hearing-impaired person cannot It is possible to check the emotional state of the disabled person, but when the hearing impaired person directly inputs a character (text) using the first communication terminal 100, it may be configured to display emotion information in the input text.

That is, the voice signal of a non-disabled person input from the second communication terminal 200 is converted into text by the STT conversion unit 332 of the conversion module 330, and the text display unit 344 converts the converted text to a preset value. Emotion information may be displayed by a method of assigning a color, a method of adjusting the size of text, and a method of assigning a preset color to a background screen excluding text.

For example, after setting yellow as the pitch of the voice or the middle value of the voice volume, setting the voice to be higher or louder as the color goes to red, and setting the voice as lower or quieter as the color goes to blue, the pitch and volume of the voice according to the color of the text It can be configured to display the emotional information of the speaker by allowing the user to determine the color, or by separately setting and displaying colors representing emotions such as joy, sadness, calmness, surprise, displeasure, happiness, anger, excitement, and the like.

Emotion information display using the text display unit 344 according to the above methods allows the hearing impaired or non-disabled person to directly set the color and font size of the text or background screen when text is input using the first or second communication terminal 100 or 200. It can also be implemented by a method of selection.

Next, the avatar display unit 346 is configured to deliver the speaker's emotions to the other party using avatars with various facial expressions and gestures stored in the relay server 300, and the hearing impaired or non-disabled person has You can select your own avatar using the first or second communication terminal 100 or 200, and the avatar display unit 346 can play a role of displaying the speaker's emotional state using the facial expression and gesture of the selected avatar. .

At this time, the display of the speaker's emotional state by the avatar display unit 346 may be displayed overlapping the display of emotional information by the above-described voice display unit 342 or text display unit 344, and the voice display unit 342 or text When the emotional state or emotional information of the speaker is recognized by the display unit 344, the avatar display unit 346 selects the facial expression or gesture of the avatar suitable for the recognized patient's emotional state or emotional information, together with voice or text. By transmitting to the first or second communication terminal 100 or 200, the other party can be configured to more accurately recognize the speaker's emotional state or emotional information.

Emotion information display using the emotion display module 340 as described above is performed by learning in the learning module 350 to be described later, and the emotion display module 340 analyzes signals input from the first or second communication terminals 100 and 200. It can be configured to automatically evaluate and display the emotional information of the hearing-impaired or non-disabled person through, but if necessary, the hearing-impaired person or non-disabled person uses the first or second communication terminal (100,200) possessed by the hearing-impaired person. It can also be configured to select one of the various emotional states, such as how to select the volume and pitch of the voice transmitted to the other party, how to select the text or the color of the text or the size of the background color on which the text is displayed And a method of selecting an avatar's expression or gesture may be used.

In addition, the emotion display module 340 may have an emotion display filtering function for adjusting the level of conflict that may occur during conversation using the first and second communication terminals 100 and 200, which is the emotion display module ( 340), if the emotions such as anger or anger among the hearing-impaired or non-disabled person's emotional information that can be recognized using the above-mentioned information exceed the preset level, the emotional level conveyed to the other party is reduced to below the preset level and displayed. can be used

That is, even during conversations between ordinary people, each other's feelings become increasingly intense due to minor misunderstandings, and later, the volume and tone of voices increase considerably or the use of profanity such as profanity frequently occurs. Since hearing-impaired people and non-disabled people, who are subject to the application, do not directly receive what they want to talk about, the possibility of misunderstanding during conversation and the resulting intensification of emotions may increase.

Therefore, the emotion display module 340 displays the emotions of the hearing impaired and non-disabled people and delivers them to the other party, but when the emotions such as anger or anger toward the other party are determined to be above a certain level, compared to the evaluated emotion level. It can be configured to indicate to the other party the level of mitigated emotion.

For example, when the emotions such as anger or anger of the hearing-impaired person evaluated by the emotion display module 340 exceed a preset level, the voice display unit 342 displays a unique voice of the hearing-impaired person compared to the emotion level evaluated. The tone or volume may be lowered and transmitted to the second communication terminal 200 .

In addition, when the emotion such as anger or anger of the hearing impaired or non-disabled person evaluated by the emotion display module 340 exceeds a preset level, the text display unit 344 sets the text to a lower level than the emotion level evaluated. The color may be displayed or the text may be reduced in size and transmitted to the first or second communication terminals 100 and 200 .

At this time, although not shown, the relay server 300 is additionally provided with a profanity detection module, so that when a profanity or profanity is input from the first or second communication terminal 100 or 200, it is removed and then the conversation content is transmitted to the other party. may be configured.

In addition, since it is desirable to faithfully convey the emotions of the caller to the other party unless conflicts between them escalate during the conversation, the emotion display module 340 measures emotions such as anger or anger when evaluating the emotional level of the hearing impaired or non-disabled people. It may be configured to determine whether the target of expression is the other party, a third party, or another incident, and to convey to the other party as it is without mitigating emotions such as anger or anger when it is determined that the expression is not an expression of emotion toward the other party.

In addition, when an emotion display filtering event by the emotion display module 340 occurs, the relay server 300 completes a call using the first and second communication terminals 100 and 200, and then first and second communication The terminals 100 and 200 may be notified that emotion display filtering has been performed for a predetermined period of time.

At this time, the relay server 300 transmits the contents of the conversation during the time during which the emotion display filtering was performed to the communication terminal of the other party, that is, the person who caused the emotion such as anger or anger, as a text signal, or the first and second communication By transmitting both to the terminals 100 and 200, it can be helpful to look back on conversations between each other and not repeat misunderstandings or mistakes.

Meanwhile, the relay server 300 may further include a learning module 350. The learning module 350 is based on deep learning and is used in the conversion module 330 and the emotion display module 340. By performing artificial intelligence learning on algorithms, it can play a role in improving the accuracy of communication between the hearing impaired and the non-disabled.

In more detail, the learning module 350 may include a transformation learning unit 352 and an emotion learning unit 354. As a configuration to improve accuracy, it can be configured so that the communication contents of the hearing-impaired person can be more accurately delivered to the non-disabled person through learning using information mainly input from the first communication terminal 100.

That is, the conversion learning unit 352 builds big data on sign language languages of the hearing impaired included in the video signal input through the first communication terminal 100, and performs learning based on the built big data. Through this, it is possible to learn the conversion contents of the MTT conversion unit 334 that converts the sign language of the hearing impaired into text and the conversion contents of the TTM conversion unit 338 that converts the text into a sign language language.

Next, the emotion learning unit 354 is a component for learning the emotion display contents made through the emotion display module 340, similarly to the video signal and the audio signal input through the first and second communication terminals 100 and 200. Emotion information displayed through the emotion display module 340, that is, mainly the voice display unit 342 and the text display unit 344, can be learned through learning based on the big data.

That is, the emotion learning unit 354 can learn emotion information of the hearing-impaired person included in the sign language through the sign language and expression of the hearing-impaired person included in the video signal input through the first communication terminal 100, , Emotional information of the non-disabled person can be learned using information such as the expression of the non-disabled person included in the video signal input through the second communication terminal 200 or the tone, pitch, and size of the voice signal.

The results learned by the conversion learning unit 352 are transmitted to the conversion module 330 and can be used for conversion in the MTT conversion unit 334 and the TTM conversion unit 338, and similarly, the emotion learning unit 354 The results learned from are transmitted to the emotion display module 340 and can be used to display emotion information by the voice display unit 342, the text display unit 344, and the avatar display unit 346, and these learning results are continuously or periodically updated. It can be.

Meanwhile, the relay server 300 according to the present invention may further include a database unit 360 for storing information necessary for providing a call service between the first and second communication terminals 100 and 200, the database unit The contents stored in 360 can be used for the purpose of improving the quality of communication service between the hearing impaired and the non-disabled.

In more detail, the database unit 360 may include first to fourth databases 362 , 364 , 366 , and 368 . First, the first database 362 is for hearing impaired people using the first communication terminal 100 . It is a structure for storing information.

That is, in the first database 362, the body information and surrounding information of the hearing-impaired person used in the voice generation module 320 as well as the finally generated unique voice information of the hearing-impaired person and avatar information selected by the hearing-impaired person are stored in the first database 362. can be stored

Next, the second database 364 is a configuration for storing various voice data that can be used as a unique voice of the hearing impaired, and various emotional states for each basic voice, that is, joy, Voices capable of expressing various emotional states such as sadness, anger, and excitement may be additionally stored.

Accordingly, the unique voice of the hearing-impaired person can be generated in the voice generation module 320 using various voice data stored in the second database 364, and the hearing-impaired person using the first communication terminal 100 A family member or a neighbor can select a voice to be used as a unique voice of a hearing impaired person from among various voice data stored in the second database 364 .

Next, the third database 366 is a configuration for storing data that can be used for artificial intelligence learning in the learning module 350 and learning results learned in the learning module 350, and can be used by the hearing impaired. Various sign language data and various facial expression data of the deaf and non-disabled people that can be input through the first and second communication terminals 100 and 200 are stored and learned in the learning module 350, that is, based on deep learning. It can be configured to be used for artificial intelligence learning for the algorithms used in the conversion module 330 and the emotion display module 340.

In addition, the learning results of the learning module 350 may also be stored in the third database 366, and similarly, the data stored in the third database 366 may be updated in real time or periodically.

Next, the fourth database 368 serves to store the contents of the call between the first and second communication terminals 100 and 200 for a certain period of time, and the contents stored in the fourth database 368 are the relay server ( 300) can be used for quality analysis of the call service provided.

At this time, a security solution may be built in the fourth database 368 to protect personal information, and when using the data stored in the fourth database 368 for quality analysis of call service, prior permission is required from the caller. Of course you should get it.

In addition, data stored in the fourth database 368 may be set to be automatically discarded after a predetermined period of time.

On the other hand, the relay server 300 according to the present invention may further include a translation module 370. The translation module 370 is a configuration that can be used when a hearing-impaired person or a non-disabled person is a foreigner who cannot speak Korean. As a result, the text transmitted to the first or second communication terminals 100 and 200 may be translated into a Korean language or a foreign language and transmitted.

That is, when the hearing-impaired person using the first communication terminal 100 is a foreigner, the voice signal input from the second communication terminal 200 is converted into text using the STT conversion unit 332 of the conversion module 330. After that, the converted text is translated into a foreign language through the translation module 370 and transmitted to the first communication terminal 100, and the text signal input from the first communication terminal 100 is translated into Korean through the translation module 370. It can be translated into and transmitted to the second communication terminal 200, or the text translated into Korean can be converted into a unique voice of the hearing impaired through the TTS conversion unit 336 and transmitted to the second communication terminal 200.

In addition, when the non-disabled person using the second communication terminal 200 is a foreigner, the text signal input from the first communication terminal 100 is translated into a foreign language through the translation module 370, and the second communication terminal 200 ), or the sign language input from the first communication terminal 100 is converted into text through the MTT conversion unit 334 and then translated into a foreign language through the translation module 370 to the second communication terminal 200. The foreign language voice signal input through the second communication terminal 200 is converted into text using the STT conversion unit 332, and then the converted text is translated into Korean through the translation module 370 to control the language. It can be transmitted to one communication terminal (100).

At this time, even when a text signal is transmitted to the first or second communication terminal 100 or 200 after being translated by the translation module 370 as described above, the emotion information displayed by the text display unit 344 among the emotion display modules 340 may be transmitted together, and the text signal transmitted to the first or second communication terminals 100 and 200 may be configured to display both the Korean language and the foreign language so that the content the speaker wants to say is conveyed more accurately together with emotions.

In addition, when foreign language voice data is included in the voice data stored in the second database 364, the foreign language voice unique to the hearing impaired may be generated and transmitted to non-disabled foreigners using the second communication terminal 200.

Next, the relay server 300 may further include an automatic transmission module 380. The automatic transmission module 380 automatically transmits signals input through the first and second communication terminals 100 and 200 without additional events. It is a configuration to be able to transmit to the other party.

That is, in the conventional case, in order to transmit a text signal, that is, a text, to the other party, an additional event such as pressing a send button such as send or enter had to be executed. In consideration of this, the signal input through the first communication terminal 100 may be automatically transmitted to the other party without an additional event.

For example, the automatic transmission module 380, when the input of a text signal by a hearing impaired person or a non-disabled person through the first and second communication terminals 100 and 200 is stopped for more than a predetermined time, such as about 3 to 5 seconds, input It may be configured to automatically transmit the received signal to the other party.

At this time, when the input of the text signal is stopped for a predetermined period, the automatic transmission module 380 automatically transmits the input signal to the other party after checking whether the sentence being input is a sentence in which meaning can be grasped or a completed sentence. It may also be configured to transmit.

In addition, the automatic transmission module 380 converts the previously transmitted sign language into a conversion module ( 330), it can be configured to be converted into a voice signal or a text signal and automatically transmitted to the second communication terminal 200, and as shown in FIG. 2, it can be provided inside the communication module 310.

On the other hand, the method for providing a call service for the hearing-impaired according to the present invention relates to a method for providing a call service between a hearing-impaired person and a non-disabled person using the call service providing system 10 for the hearing-impaired person as described above, As shown in FIG. 3, the configuration may largely include a preliminary preparation step (S100), a call request step (S200), and a call connection step (S300).

First, the preliminary preparation step (S100) is a process for a user to pre-register his or her information in the relay server 300 in order to use the call service providing system 10 for the hearing impaired according to the present invention, and is an information input step. (S110), a unique voice generation step (S120) and a learning step (S130) may be included.

More specifically, in the information input step (S110), the hearing-impaired person connects to the relay server 300 using the first communication terminal 100 possessed by the person with hearing loss, and accesses his/her own information, that is, body information such as height, weight, and gender. This is a process of inputting physique information such as the upper body of the hearing impaired person, age, family voice of the same gender, and surrounding information such as whether or not they smoked. It can be used to determine the form of a signal transmitted from the relay server 300 to the first communication terminal 100 and generation of emotion information of the hearing impaired to be transmitted to the person.

Next, the unique voice generation step (S120) is a process for generating a unique voice of the hearing impaired to be transmitted from the relay server 300 to the second communication terminal 200 used by non-disabled people. The first communication terminal 100 The sign language of the hearing-impaired person included in the text signal or video signal transmitted to the relay server 300 may be converted into a unique voice signal of the hearing-impaired person and transmitted to the second communication terminal 200 .

That is, in the unique voice generation step (S120), hearing is heard by the voice generation module 320 provided in the relay server 300 using the information of the hearing-impaired person input to the relay server 300 in the information input step (S110). It is possible to generate a unique voice of a disabled person, and the detailed process may include a voice extraction step (S122), a voice correction step (S124), and a voice selection step (S126).

First, in the voice extraction step (S122), the hearing-impaired person's unique voice is used by using the body information such as height, weight, and gender of the hearing-impaired person input in the information input step (S110) and physique information such as the hearing-impaired person's upper body. Can be generated primarily. In the voice extraction unit 322 of the voice generation module 320 provided in the relay server 300, among various voice data previously stored in the second database 364, Voice data matching body information can be extracted.

Next, the voice correction step (S124) is a process for correcting the voice data extracted in the voice extraction step (S122), and the voice correction unit 324 of the voice generation module 320 in the information input step (S110) The pitch, timbre, etc. of the unique voice primarily extracted in the voice extraction step (S122) can be corrected using peripheral information such as the age of the hearing-impaired person, the family voice of the same gender, and smoking status.

Next, the voice selection step (S126) is a process for finally selecting the unique voice of the hearing-impaired person. By accessing the voice selection unit 326, it is possible to finally select the unique voice of the hearing-impaired person generated by the voice extraction unit 322 or the voice correction unit 324.

Next, the learning step (S130) is based on deep learning and performs artificial intelligence learning for the algorithm used in the signal conversion step (S310) and the emotion display step (S330) to be described later, thereby making a call between the hearing impaired and the non-disabled As a process to improve the accuracy of content delivery, the conversion learning step (S132) for learning the algorithm used in the signal conversion step (S310) and the emotion learning step for learning the algorithm used in the emotion display step (S330) ( S134) may be included.

The configuration and learning contents used in the conversion learning step (S132) and the emotion learning step (S134) are the same as the description of the learning module 350 of the above-mentioned call service providing system 10 for the hearing impaired. A detailed description will be omitted.

Next, in the call request step (S200), the hearing-impaired or non-disabled person connects to the relay server 300 using the first or second communication terminal (100, 200), and the phone number, dedicated code, ID, etc. If the same identifiable information is entered, the relay server 300 transmits a call request signal to the other party's second or first communication terminal (200, 100), and when the other party accepts the call request, the first and second communication terminals Various known methods such as making a call between (100 and 200) can be used.

Next, the call connection step (S300) is a process in which a call is made between a hearing-impaired person and a non-disabled person having the first and second communication terminals 100 and 200, respectively, by the relay of the relay server 300, and the signal conversion step (S310), a translation step (S320), and an emotion display step (S330) may be included.

First, the signal conversion step (S310) is a process for converting signals input from the first and second communication terminals 100 and 200 into signals that the other party can recognize in the relay server 300, the first communication terminal ( 100) can be converted into a voice signal or text signal that can be recognized by the non-disabled person, and the voice signal of the non-disabled person input from the second communication terminal 200 can be recognized by the hearing-impaired person. It can be converted into a text signal or sign language that can be read.

That is, the signal conversion step (S310) may include an MTT conversion step (S312), an STT conversion step (S314), a TTS conversion step (S316) and a TTM conversion step (S318), wherein the MTT conversion step (S312) In , the sign language included in the image of the hearing impaired input from the first communication terminal 100 can be converted into a text signal that can be recognized by the non-disabled person, and in the STT conversion step (S314), the second communication terminal 200 ) can be converted into a text signal that can be recognized by the hearing impaired.

In addition, in the TTS conversion step (S316), the text signal mainly converted in the MTT conversion step (S312) can be converted into a voice signal that can be recognized by non-disabled people. In this case, the voice signal is a unique voice generation step ( The unique voice of the hearing-impaired person generated in S120) may be used.

In addition, in the TTM conversion step (S318), the text signal converted in the STT conversion step (S314) can be converted into a sign language that can be recognized by the hearing impaired.

The signal conversion step (S310) of the above process is performed by the conversion module 330 provided in the relay server 300, and more detailed information on each conversion step is provided in the above-described call service providing system 10 for the hearing impaired. Since it is the same as the description of the conversion module 330 of ), it will be omitted.

Next, in the translation step (S320), the hearing impaired using the first communication terminal 100 or the non-disabled person using the second communication terminal 200 is a foreigner through the first or second communication terminals 100 and 200. As a process for translating an input signal into a language recognizable by the other party, as described above, the translation module 370 provided in the relay server 300 is transmitted to the first or second communication terminals 100 and 200. The text to be translated into Korean or a foreign language may be transmitted, or a unique foreign language voice of the hearing impaired may be generated and transmitted to non-disabled foreigners using the second communication terminal 200 .

Next, the emotion display step (S330) is a process for displaying the emotion information of the speaker, that is, the hearing impaired and non-disabled people, using the information received from the first and second communication terminals 100 and 200, and the relaying In the emotion display module 340 provided in the server 300, the speaker's emotion is transmitted to the signal transmitted from the first and second communication terminals 100 and 200 or the signal converted in the signal conversion step (S310), that is, the voice signal or the text signal. By allowing information to be displayed, even one's emotional information can be delivered to the other party in the course of a call, so that the other party can more clearly convey what they want to say.

That is, the emotion display step (S330) includes a voice display step (S332) for including emotion information in the unique voice of the hearing impaired transmitted from the relay server 300 to the second communication terminal 200, and the relay server 300 ) to the first or second communication terminal (100,200) to include the emotional information of the non-disabled person or the hearing impaired in the text signal (S334) and various facial expressions and gestures stored in the relay server 300 An avatar display step (S336) of allowing the speaker's emotions to be conveyed to the other party using the avatars taken may be included.

The above voice display step (S332), text display step (S334), and avatar display step (S336) are the emotion display module 340 provided in the relay server 300 of the above-described call service providing system 10 for the hearing impaired. ) of the voice display unit 342, the text display unit 344, and the avatar display unit 346, and since the contents performed in each step are the same as those described above, a detailed description thereof will be omitted.

In addition, although not shown, the emotion display step (S330) includes the emotion display filtering step, as described above, when the emotion such as anger or anger toward the other party is higher than a predetermined level during conversation with each other, the emotion The display module 340 can alleviate this and transmit it to the other party, and after the call is completed, the fact that the emotion display filtering was performed and the conversation record for the time during which the emotion display filtering was performed are transmitted from the relay server 300 to the first and/or Alternatively, it can be transmitted to the second communication terminal (100,200).

On the other hand, in the call connection step (S300), when a hearing impaired or non-disabled person transmits a text signal to the other party using the first or second communication terminal (100, 200), the input signal is transmitted by pressing a send button such as send or enter. It can be automatically transmitted to the other party without additional events such as pressing. As described above, in the automatic transmission module 380 provided in the relay server 300, the hearing impaired or the hearing impaired using the first and second communication terminals 100 and 200 When the non-disabled person's text signal input is stopped for a predetermined time or more or when a sentence capable of conveying meaning is completed, the input signal may be automatically transmitted to the other party.

Therefore, according to the system and method for providing a call service for the hearing impaired according to the present invention as described above, the communication weak represented by the hearing impaired can more easily and conveniently communicate with the other party using the communication terminals 100 and 200, and can communicate with each other. Not only can conversations between hearing-impaired people and non-disabled people, who have difficulty in direct communication between them, be freely conducted, but also visualize or categorize the emotional state estimated from the speaker's voice or expression, or display the conversation content and information to the other person through voice. By enabling communication together, communication between callers can be made more clearly, and the like has various advantages.

The foregoing embodiments have been described with respect to the most preferred examples of the present invention, but are not limited to the above embodiments, and in addition to hearing impaired people, communication weak people such as the elderly who have difficulty using the communication terminals 100 and 200, and people with speech disabilities and non-disabled people. Various modifications are possible within a range that does not deviate from the technical idea of the present invention, such as being applicable to calls between people or between communication weak people.

The present invention relates to a system and method for providing a call service for the hearing impaired, and more particularly, provides a call service between the hearing impaired and non-disabled people by utilizing functions such as text-to-speech conversion, voice-to-text conversion, and motion-text conversion, and at the same time The present invention relates to a system and method for providing a call service for the hearing impaired that enables clear communication between callers by visualizing or expressing an emotional state estimated from a speaker's voice or facial expression.

10: call service providing system 100: first communication terminal
200: second communication terminal 300: relay server
310: communication module 320: voice generation module
322: voice extraction unit 324: voice correction unit
326: voice selection unit 330: conversion module
332: STT conversion unit 334: MTT conversion unit
336: TTS conversion unit 338: TTM conversion unit
340: emotion display module 342: voice display unit
344: text display unit 346: avatar display unit
350: learning module 352: conversion learning unit
354: emotion learning unit 360: database unit
362: first database 364: second database
366: third database 368: fourth database
370: translation module 380: automatic transmission module
S100: Preparatory step S110: Information input step
S120: unique voice generation step S122: voice extraction step
S124: voice correction step S126: voice selection step
S130: learning step S132: conversion learning step
S134: Emotion Learning Step S200: Call Request Step
S300: call connection step S310: signal conversion step
S312: MTT conversion step S314: STT conversion step
S316: TTS conversion step S318: TTM conversion step
S320: Translation step S330: Emotion display step
S332: voice display step S334: text display step
S336: Avatar display step

Claims (13)

In the system for providing a call service for the hearing impaired, including a first communication terminal possessed by a hearing impaired person, a second communication terminal possessed by a non-disabled person, and a relay server for relaying a call between the first and second communication terminals,
The relay server,
A communication module for wireless communication with the first and second communication terminals;
A voice generation module for generating a voice of a hearing-impaired person using a signal input from the first communication terminal;
A conversion module for converting the information received from the first and second communication terminals into a form recognizable by the other party;
An emotion display module for displaying emotional information of a speaker using information received from the first and second communication terminals;
The emotion display module,
A voice display unit that displays the emotions of the hearing impaired by adjusting the pitch and volume of the hearing impaired voice converted by the conversion module;
A text display unit for displaying the emotion of the non-disabled person by giving a predetermined color to the text of the non-disabled person converted by the conversion module and transmitted to the first communication terminal,
When the emotion including anger or anger toward the other party is determined to be above a predetermined level, the emotion display module mitigates the emotion level transmitted to the other party through the voice display unit or the text display unit to a predetermined level or less, and displays the emotion. Equipped with a display filtering function;
When an emotion display filtering event by the emotion display module occurs, the relay server notifies the first and second communication terminals that emotion display filtering has been performed for a certain period of time after the call is completed. Call service delivery system.
According to claim 1,
The voice generation module,
A voice extraction unit that primarily generates a unique voice of the hearing-impaired person using the body information of the hearing-impaired person input through the first communication terminal;
A voice correction unit correcting the unique voice of the hearing-impaired person using peripheral information including the hearing-impaired person's age and family voice input through the first communication terminal; and
A system for providing a call service for the hearing-impaired, characterized in that it comprises a voice selection unit for enabling the hearing-impaired to select a voice using the first communication terminal.
delete According to claim 1,
The emotion display module,
The system for providing a call service for the hearing-impaired further comprising an avatar display unit expressing the emotion information displayed by the voice display unit or the text display unit by an avatar.
According to claim 1,
The relay server,
A system for providing a call service for the hearing impaired further comprising a learning module for performing artificial intelligence learning on the algorithm used in the conversion module and the emotion display module based on deep learning.
According to claim 5,
The relay server,
A first database for storing information of hearing-impaired people using the first communication terminal;
A second database for storing voice data capable of expressing the voice of the hearing impaired;
A third database for storing data that can be used for artificial intelligence learning in the learning module and learning results learned in the learning module, and
A call service providing system for the hearing-impaired, comprising a fourth database for storing conversation contents between the first and second communication terminals for a predetermined period of time.
According to claim 1,
The relay server,
Further comprising an automatic transmission module that automatically transmits the input signal to the other party when the user's signal input using the first and second communication terminals is stopped for a predetermined time or when a sentence capable of conveying meaning is completed A call service providing system for the hearing impaired.
Call using a call service providing system for the hearing impaired including a first communication terminal possessed by a hearing impaired person, a second communication terminal possessed by a non-disabled person, and a relay server relaying a call between the first and second communication terminals. In the service provision method,
A preliminary preparation step of registering information of the hearing impaired in a relay server using the first communication terminal;
A call request step of accessing a relay server using the first or second communication terminal and requesting a call with the other party;
A call connection step in which a hearing-impaired person and a non-disabled person each having first and second communication terminals conduct a call by relaying by the relay server,
In the call connection step,
A signal conversion step of converting the signal transmitted from the first and second communication terminals into a signal that the other party can recognize in the relay server;
And an emotion display step of displaying the speaker's emotion information on the signal transmitted from the first and second communication terminals or the signal converted in the signal conversion step,
The emotion display step,
A voice display step of displaying the emotions of the hearing-impaired person by adjusting the pitch and volume of the hearing-impaired person's unique voice converted in the signal conversion step and transmitted from the relay server to the second communication terminal;
A text display step of displaying emotional information of a speaker in a text signal transmitted from a relay server to a first or second communication terminal;
Avatar display step of displaying the emotional state of the speaker using the expression and gesture of the avatar of the hearing impaired or non-disabled person selected by the first or second communication terminal, and
Including an emotion display filtering step of mitigating and displaying the emotion level transmitted to the other party to a predetermined level or less when the emotion including anger or anger towards the other party during the conversation is determined to be above a predetermined level,
When the emotion display filtering step is performed, the relay server notifies the first and second communication terminals that the emotion display filtering has been performed for a predetermined time after the call is completed.
According to claim 8,
In the preparatory stage,
A unique voice generation step of generating a unique voice of the hearing-impaired person using information of the hearing-impaired person input to the relay server;
A method for providing a call service for the hearing impaired, comprising a learning step of performing artificial intelligence learning on the algorithm used in the signal conversion step and the emotion display step based on deep learning.
According to claim 9,
The unique voice generation step,
A voice extraction step of extracting a unique voice of the hearing-impaired person using body information of the hearing-impaired person;
A voice correction step of correcting the extracted unique voice using peripheral information including the age of the hearing impaired and the voice of the family, and
and a voice selection step of selecting a final unique voice of the hearing impaired using the first communication terminal.
According to claim 8,
The signal conversion step,
An MTT conversion step of converting the sign language included in the image of the hearing impaired inputted from the first communication terminal into a text signal;
An STT conversion step of converting the voice signal of a non-disabled person input from the second communication terminal into a text signal;
A TTS conversion step of converting the text signal converted in the MTT conversion step into a voice signal; and
and a TTM conversion step of converting the text signal converted in the STT conversion step into a sign language.
delete According to claim 8,
In the call connection step, when the user's signal input using the first and second communication terminals is stopped for more than a predetermined time or a sentence capable of conveying meaning is completed, the input signal is automatically transmitted to the other party. How to provide call service for use.

Images