WO2019237429A1 - Method, apparatus and system for assisting communication, and augmented reality glasses - Google Patents

Abstract

Provided in the embodiments of the present invention are a method, apparatus and system for assisting communication, and augmented reality glasses, belonging to the technical field of augmented reality. The method comprises: receiving speech of at least one sound source; based on the received speech of each of the at least one sound source, determining the orientation of each of the at least one sound source; identifying the speech of each of the at least one sound source to convert the speech of each of the at least one sound source to characters; and displaying the orientation of each of the at least one sound source and the characters converted from the speech. The apparatus comprises a speech receiving module, a determination module, a speech identification module and a display module. The system comprises the device mentioned above and a client. The augmented reality glasses comprise the apparatus mentioned above. The present application can allow the hearing impaired to accurately grasp speech content emitted from a sound source and the orientation of the sound source and allow the hearing impaired to conveniently perform communication and exchange.

Description

Method, device and system for assisted communication and augmented reality glasses Technical field

The present invention relates to the field of augmented reality technology, and in particular, to a method, device, and system for assisting communication, and augmented reality glasses.

Background technique

Augmented Reality (AR) technology is a technology that calculates the position and angle of an image in real time, superimposes the corresponding image, video, and 3D model on the image, and then fuses the virtual world with the real world. The AR client can combine real-time image recognition of the offline environment of the user with the pictures stored directly in its local image recognition, and according to the pre-configured display of the identified offline targets in the real scene The effect is enhanced to display the corresponding display data. With the development of technology, the application of augmented reality technology is widespread, but for the hearing impaired, augmented reality technology has not helped them well.

At present, there are two main ways for hearing impaired people and hearing people to communicate: sign language interpreters or hearing aids. However, these two communication channels have certain problems for the hearing impaired, especially under the environment of multi-person communication.

Summary of the Invention

An object of the present invention is to provide a method, a device, and a system for assisting communication, and augmented reality glasses, which can enable a hearing impaired person to accurately grasp the content of a voice emitted by a sound source and the position of the sound source.

To achieve the above object, one aspect of the present invention provides a method for assisting a hearing impaired person to communicate, the method comprising: receiving a voice of at least one sound source; and based on each of the received at least one sound source , Determine the position of each sound source in the at least one sound source; identify the sound of each sound source in the at least one sound source to convert the sound of each sound source in the at least one sound source into Text; and text that translates the position and speech of each of the at least one sound source.

Optionally, the method further includes: receiving text; converting the received text into speech; and playing the converted speech.

Optionally, the form of displaying the orientation of each sound source of the at least one sound source and the text converted by the voice is any one of the following: using a preset foreground color and a preset background color for display and using a preset foreground color The orientation and text corresponding to different sound sources are displayed by changing colors alternately with the preset background color, wherein the preset foreground color and the preset background color are different colors.

Optionally, the preset foreground color is white and the preset background color is green; or the preset foreground color is green and the preset background color is white.

Optionally, the method further comprises: determining location information of the hearing impaired person; and sending the location information to a mobile terminal and / or client, so that the mobile terminal and / or client obtains the location in real time information.

Optionally, before sending the location information to a mobile terminal and / or client, the method further includes: receiving a setting for a contact, wherein the mobile terminal and / or client is a contact with the selected one A mobile terminal and / or client corresponding to a person.

Accordingly, another aspect of the present invention provides a device for assisting a hearing impaired person to communicate. The device includes: a voice receiving module for receiving voice of at least one sound source; and a determining module for receiving information based on all received signals. The speech of each sound source in the at least one sound source is described, and the position of each sound source in the at least one sound source is determined; the speech recognition module is configured to identify the sound of each sound source in the at least one sound source, and Convert the speech of each sound source in the at least one sound source into text; and a display module for displaying the position of each sound source in the at least one sound source and the text converted by the speech.

Optionally, the device further includes: a text receiving module for receiving text; a text conversion module for converting the received text into speech; and a voice playback module for playing the converted speech.

Optionally, the display module is a near-eye display.

Optionally, the near-eye display is a see-through near-eye display.

Optionally, the form of displaying the orientation of each sound source of the at least one sound source and the text converted by the voice is any one of the following: using a preset foreground color and a preset background color for display and using a preset foreground color The orientation and text corresponding to different sound sources are displayed by changing colors alternately with the preset background color, wherein the preset foreground color and the preset background color are different colors.

Optionally, the preset foreground color is white and the preset background color is green; or the preset foreground color is green and the preset background color is white.

Optionally, the device further includes: a positioning module for determining position information of the hearing impaired; and a communication module for sending the position information to a mobile terminal and / or a client, so that the mobile terminal And / or the client obtains the location information in real time.

Optionally, the device further includes: a contact setting module configured to receive the setting of the contact before the communication module sends the location information to the mobile terminal and / or the client, wherein the mobile terminal And / or the client is a mobile terminal and / or client corresponding to the selected contact.

In addition, another aspect of the present invention provides an augmented reality glasses, which includes the above-mentioned device.

In addition, another aspect of the present invention also provides a system for assisting the hearing impaired to communicate, the system includes: the above-mentioned device; and a client.

In addition, another aspect of the present invention also provides a machine-readable storage medium, where the machine-readable storage medium stores instructions, and the instructions are used to make the method of the machine described above.

Through the above technical solution, displaying the translated text of each sound source of at least one sound source enables a hearing impaired person to understand the content of the sound of each sound source; displaying the position of each sound source of at least one sound source, achieving A simple and intuitive way to indicate where the voice of the hearing impaired occurs. In this way, the hearing-impaired person can obtain the positional awareness similar to ordinary people while watching the "subtitles" to understand the voice information being heard, thereby enabling the hearing-impaired person to understand the content of the voice emitted by each sound source. Clearly and intuitively understand the position of each sound source, so that the hearing impaired can accurately grasp the content of the sound emitted by each sound source and the position of the sound source, which is convenient for the hearing impaired to communicate and communicate with others. In particular, in an environment where there are multiple sound sources, accurately grasping the content of the sound emitted by each sound source and the position of the sound source is very helpful for the communication of hearing impaired people with others.

Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed implementation sections.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are used to provide a further understanding of the embodiments of the present invention, and constitute a part of the description. Together with the following specific implementations, the drawings are used to explain the embodiments of the present invention, but not to limit the embodiments of the present invention. In the drawings:

FIG. 1 is a flowchart of a method for assisting a hearing-impaired person to communicate according to an embodiment of the present invention; FIG.

FIG. 2 is an exemplary diagram using arrows to indicate directions according to another embodiment of the present invention; FIG.

FIG. 3 is an exemplary diagram of an orientation provided by another embodiment of the present invention; FIG.

FIG. 4 is an exemplary diagram showing the position of a sound source and the text converted by speech provided by another embodiment of the present invention; FIG.

FIG. 5 is an exemplary diagram showing the positions of multiple sound sources and text converted by voice according to another embodiment of the present invention; FIG.

6 is a flowchart of a method for assisting a hearing impaired person to communicate according to another embodiment of the present invention; and

FIG. 7 is a structural block diagram of a device for assisting a hearing-impaired person to communicate according to another embodiment of the present invention.

Reference Signs

1 voice receiving module 2 determining module

3 speech recognition module 4 display module

detailed description

The specific implementations of the embodiments of the present invention will be described in detail below with reference to the drawings. It should be understood that the specific implementation manners described herein are only used to illustrate and explain the embodiments of the present invention, and are not intended to limit the embodiments of the present invention.

An aspect of an embodiment of the present invention provides a method for assisting a hearing impaired person to communicate. FIG. 1 is a flowchart of a method for assisting a hearing impaired person to communicate according to an embodiment of the present invention. As shown in FIG. 1, the method includes the following steps.

In step S10, the voice of at least one sound source is received.

In step S11, the position of each sound source in the at least one sound source is determined based on the speech of each sound source in the received at least one sound source.

Wherein, determining the position of the sound source may be based on the time when the voice emitted from the sound source is received. For example, a voice receiving module for receiving voice of at least one sound source includes a plurality of voice acquisition modules, the plurality of voice acquisition modules are disposed at different positions, and the plurality of voice acquisition modules receives voices from the same sound source. Time is different. For each sound source in the at least one sound source, the position of the sound source is determined according to the difference in the time when the speech arrives at the multiple speech collection modules, that is, according to the time difference between the time when the speech reaches the multiple speech collection modules. Optionally, in the embodiment of the present invention, the voice collection module may be a microphone, and the voice receiving module may be a microphone array. For example, the microphone array may include 2, 4, 6, 7, or 8 microphones.

Optionally, in the embodiment of the present invention, the reference point of the azimuth may be set according to the actual situation, for example, it may be the position where the voice receiving module is located. Specifically, it may be any one of the plurality of voice acquisition modules, or may be an intermediate position of the plurality of voice acquisition modules. In addition, when the speech receiving module is worn by the hearing-impaired person or is not far from the hearing-impaired person, the speech-receiving module is used as the reference point, which is actually the hearing-impaired person as the reference point. In this way, the hearing-impaired person can Azimuth knows where the sound source is relative to itself.

Optionally, in the embodiment of the present invention, the orientation may include a direction and / or a distance. Optionally, in the embodiment of the present invention, an arrow may be used to indicate the direction. The arrow is located in an area delimited by a circle. The starting point of the arrow is the origin of the circle. The origin is equivalent to the position of the hearing impaired and the arrow deviates. The vertical axis passing through the circle is at an angle, as shown in FIG. 2, and the vertical dotted line in the figure is the vertical axis passing through the circle. In addition, the horizontal axis of the circle is used as a reference. As shown by the horizontal dashed line in FIG. 2, when the arrow is located above the horizontal axis, it means that the sound source is in front of the hearing impaired; when the arrow is located below the horizontal axis, Indicates that the sound source is behind the hearing impaired. For example, taking the direction example shown in 2 as an example, the sound source indicated by the arrow is in front of the hearing impaired person. In addition, the way in which the direction of the sound source is indicated by arrows can also be interpreted as the direction is indicated by a clock. The circle represents the dial, and the vertical axis located in the upper half of the horizontal axis of the circle indicates the 12 o'clock direction. According to the angle at which the arrow deviates from 12 o'clock, it is determined that the sound source is about the clock direction. Taking the direction example shown in Figure 2 as an example, the sound source indicated by the arrow is about 10 o'clock. In addition, in a case where the azimuth includes a direction and a distance, the azimuth can be expressed using an example as shown in FIG. 3. It should be noted that the position of the display distance can be set according to the actual situation, and there is no limitation on this. In addition, in a case where the azimuth includes only the distance, only the distance may be displayed. In particular, when it is determined that the sound source comes from the hearing impaired person based on the received voice, when an arrow is used to indicate the direction of the sound source, "O" or "●" is displayed at the center of the circle to indicate the direction of the sound source. In addition, in the embodiment of the present invention, text may also be used to describe the orientation. For example, taking the orientation shown in FIG. 3 as an example, the text "direction is ten o'clock and distance is 50 cm" may be displayed.

In step S12, the speech of each sound source in the at least one sound source is identified to convert the speech of each sound source in the at least one sound source into text. For example, speech recognition technology is used to convert speech into text.

In step S13, the position of each of the at least one sound source and the text converted by the voice are displayed. An example showing the position of each sound source and the text converted by voice can be shown in FIG. 4. In addition, when displaying the speech-transformed text of a certain sound source, if all the text cannot be displayed in one line, it can be displayed in a new line or scrolled.

In addition, it should be noted that FIG. 4 only shows by way of example the position of the area where the orientation and the text converted by voice are displayed. The position of the two display areas can be selected according to the actual situation. For the two display areas, The position is not limited. In addition, when displaying the position of each of the at least one sound source and the text converted by speech, the example shown in FIG. 4 may be used for display. In addition, when displaying the positions of multiple sound sources and the text converted by voice, the sound sources may be displayed in a manner of being arranged one above the other in sequence, as shown in FIG. 5. In addition, the sound sources may be displayed in the order of the left and right of the sound sources, or displayed in other arrangements, which is not limited.

Optionally, in the embodiment of the present invention, there may be many ways to display the orientation and text. For example, the preset foreground color and the preset background color are used for display, and the preset foreground color and the preset background color are different colors. For example, the preset foreground color is white and the preset background color is black to display white characters on a black background; or the preset foreground color is black and the preset background color is white to display black characters on a white background. For another example, the preset foreground color is white, the preset background color is green, and the white text on a green background is displayed; or the preset foreground color is green, the preset background color is white, and the green text on a white background is displayed. In this way, the user can distinguish the text more clearly. For example, the way to display the orientation and text may also be to display the orientation and text corresponding to different sound sources by alternately changing the color of the preset foreground color and the preset background color, that is, the preset foreground color and the preset background color are different. Color, according to the order of received speech, when the sound source corresponding to the received adjacent speech is a different sound source, the preset foreground color and the preset background color are alternately changed; when the sound source corresponding to the received adjacent speech is the same For sound sources, the preset foreground color and preset background color do not change colors. The colors of the preset foreground color and the preset background color can be limited according to actual conditions. For example, the preset foreground color is white, the preset background color is black, and the white text on a black background is displayed; or the preset foreground color is black. , The preset background color is white, and black characters on a white background are displayed; for example, the preset foreground color is white, the preset background color is green, and white characters on a green background are displayed; or the preset foreground color is green, and the preset background color is white Displays green text on a white background. The following is an example of using the preset foreground color to be white and the preset background color to be green to alternately change the preset foreground color and the preset background color to display the orientation and text corresponding to different sound sources. If a certain voice (named as the first voice, the name is only for narrative purposes, without limitation) corresponds to the first sound source, and the text converted by the first voice and the orientation of the first sound source are displayed in white on a green background; according to In the order of receiving voices, the sound source corresponding to the next voice of the first voice (named second voice) and the first sound source are different sound sources (the sound source corresponding to the second voice is named second sound source). The text converted by the second voice and the position of the second sound source are in green on a white background. According to the order of the received voice, the sound source corresponding to the next voice of the second voice (named the third voice) is the second sound source. When displaying the text converted from the third voice and the orientation of the second sound source, the green text on a white background is still used. According to the order of the received voice, the sound source corresponding to the next voice of the third voice (named the fourth voice) and the second voice The sound source is a different sound source (the fourth sound source is named the third sound source, and the third sound source may be the first sound source or other sound sources, as long as it is not the second sound source), The text of the fourth voice conversion and the third sound source are displayed Azimuth using green and white, so, the cycle continues until all of the received display information corresponding to the voice is completed, wherein the information corresponding to the speech sound source and text orientation including voice corresponding to speech conversion.

Display the voice-transformed text of each sound source in at least one sound source so that the hearing impaired person can understand the content of the speech of each sound source; display the position of each sound source in at least one sound source, in a simple and intuitive way Prompt the sounding position of the hearing impaired. In this way, the hearing-impaired person can obtain the positional awareness similar to ordinary people while watching the "subtitles" to understand the voice information being heard, thereby enabling the hearing-impaired person to understand the content of the voice emitted by each sound source. Clearly and intuitively understand the position of each sound source, so that the hearing impaired can accurately grasp the content of the sound emitted by each sound source and the position of the sound source, which is convenient for the hearing impaired to communicate and communicate with others. In particular, in an environment where there are multiple sound sources, accurately grasping the content of the sound emitted by each sound source and the position of the sound source is very helpful for the communication of hearing impaired people with others. In addition, with the method described in the embodiment of the present invention, the user's operation experience is extremely light, and he / she can “hear” information beyond his ability without operating a technical system at all. In addition, it should be noted that the method for assisting the hearing-impaired person provided by the embodiment of the present invention can be applied not only to the hearing-impaired person, but also to ordinary people.

FIG. 6 is a flowchart of a method for assisting a hearing impaired person to communicate according to another embodiment of the present invention. The difference from the method shown in FIG. 1 is that the method shown in FIG. 6 further includes the following content.

In step S64, a character is received. Among them, there are many ways for the hearing impaired to enter text. For example, connecting a keyboard enables a hearing impaired person to enter text through the keyboard. For example, by connecting an interactive interface, a hearing impaired person can enter text through the interactive interface. In addition, you can connect to the client, and the hearing impaired can enter text through the client. Optionally, the client may be a mobile APP.

In step S65, the received text is converted into speech, for example, text-to-speech conversion is achieved by using TTS technology.

In step S66, the converted voice is played.

In this way, when the hearing-impaired person does not have the pronunciation ability or the pronunciation ability is limited, the hearing-impaired person can input text to express his meaning and communicate with others.

It should be noted that steps S64 to S66 may also be performed before steps S60 to S63, and there is no restriction on this.

Optionally, in the embodiment of the present invention, before the voice of at least one sound source is received and / or the text is received, the method for assisting a hearing impaired person to communicate may further include the following content: The language of the text and / or the language of the speech into which the text is converted. In this embodiment, the “hearing impaired person” may not be a person with limited hearing ability, but may be a “first-view equal hearing impaired person” who does not understand the language of others Of people who are different in their language are "second-view parity hearing impaired." Set the first language used by the “first-view hearing-impaired person” to convert the speech received by at least one sound source into words expressed in the first language. The text understands what others are talking to. Set a second language used by others who communicate with the “second-view hearing-impaired person”, and convert the text entered by the “second-view hearing-impaired person” into a voice expressed in a second language. The voice understands what "as a second hearing impaired" means. In this way, communication between the “persons with hearing impairment” and others is realized.

Optionally, in the embodiment of the present invention, the method for assisting the hearing impaired to communicate may further include the following: determining the position information of the hearing impaired; and sending the position information to the mobile terminal and / or the client to The mobile terminal and / or the client can obtain the location information in real time. In this way, the contact person related to the hearing impaired person can obtain the position information of the hearing impaired person in real time to confirm whether he is safe, and can find him as soon as possible when a situation occurs. Among them, in the embodiment of the present invention, the position information of the hearing-impaired person can be determined in real time through GPS positioning technology.

Optionally, in the embodiment of the present invention, before sending the location information to the mobile terminal and / or the client, the method further includes: receiving a setting of a contact, wherein the mobile terminal and / or the client are connected with the selected The mobile terminal and / or client corresponding to the specified contact. When there are many contacts related to the hearing impaired, in different situations, some contacts can appear beside the hearing impaired in time to help the hearing impaired, thereby, When sending the hearing-impaired person's location information, the hearing-impaired person's location information can be sent directly to the contacts who can appear in time, so that when the hearing-impaired person has difficulty, he can reach the hearing-impaired person's location as soon as possible, helping the hearing-impaired person to solve the problem . In addition, the correspondence relationship between the hearing-impaired persons and the mobile terminals and / or clients used by them can be set in advance.

In addition, in the embodiment of the present invention, the method for assisting the hearing impaired to communicate may further include the following: according to the order of receiving voices, recording the positions and characters corresponding to each sound source, and comparing the positions and corresponding positions of each sound source The text is stored locally or in the cloud to further help the hearing impaired to remember and share afterwards.

Accordingly, another aspect of the embodiments of the present invention provides a device for assisting a hearing impaired person to communicate. FIG. 7 is a device for assisting people with hearing impairment to communicate according to another embodiment of the present invention. As shown in FIG. 7, the device includes a voice receiving module 1, a determining module 2, a voice recognition module 3, and a display module 4. The voice receiving module 1 is configured to receive voice of at least one sound source. The determining module 2 is configured to determine the position of each sound source in the at least one sound source based on the received speech of each sound source in the at least one sound source. The voice recognition module 3 is used for recognizing the voice of each of the at least one sound source to convert the voice of each of the at least one sound source into text. The display module 4 is configured to display the position of each sound source and the text converted by the voice in at least one sound source.

Display the voice-transformed text of each sound source in at least one sound source so that the hearing impaired person can understand the content of the speech of each sound source; display the position of each sound source in at least one sound source, in a simple and intuitive way Prompt the sounding position of the hearing impaired. In this way, the hearing-impaired person can obtain the positional awareness similar to ordinary people while watching the "subtitles" to understand the voice information being heard, thereby enabling the hearing-impaired person to understand the content of the voice emitted by each sound source. Clearly and intuitively understand the position of each sound source, so that the hearing impaired can accurately grasp the content of the sound emitted by each sound source and the position of the sound source, which is convenient for the hearing impaired to communicate and communicate with others. In particular, in an environment where there are multiple sound sources, accurately grasping the content of the sound emitted by each sound source and the position of the sound source is very helpful for the communication of hearing impaired people with others. In addition, it should be noted that the method for assisting the hearing impaired to provide communication provided by the embodiment of the present invention can be applied not only to the hearing impaired, but also to ordinary people.

Optionally, in the embodiment of the present invention, the device for assisting the hearing impaired to communicate further includes a text receiving module for receiving text; a text conversion module for converting the received text into speech; and speech A playback module for playing the converted voice.

Optionally, in the embodiment of the present invention, the display module may be a near-eye display. The distance between the near-eye display and the eyeball may be less than 2 cm. In addition, the near-eye display may include a see-through near-eye display or a non-see-through near-eye display. In this way, the position of each sound source and the text converted by the voice are presented in front of the eyes. Preferably, in the embodiment of the present invention, the display module may be a see-through near-eye display. In this way, while not affecting the hearing-impaired person's observation of other things, the hearing-impaired person can understand the position of each sound source and the text converted by the voice by watching "subtitles".

Optionally, in the embodiment of the present invention, the device further includes: a positioning module for determining the location information of the hearing impaired; and a communication module for sending the location information to the mobile terminal and / or the client so that the mobile The terminal and / or the client obtains the location information in real time.

Optionally, in the embodiment of the present invention, the device further includes: a contact setting module, configured to receive the setting of the contact before the communication module sends the location information to the mobile terminal and / or the client, The mobile terminal and / or client is a mobile terminal and / or client corresponding to the selected contact.

In addition, in the embodiment of the present invention, the device for assisting a hearing-impaired person to communicate further includes a storage module. The storage module is used to record the position and text corresponding to each sound source according to the order of receiving voices, to further help the hearing impaired to remember and share afterwards. Wherein, the storage module records the orientation and text corresponding to each sound source, which may be storing the orientation and text corresponding to each sound source on the local end or in the cloud.

The specific working principle and benefits of the device for assisting the hearing impaired to provide communication provided by the embodiment of the present invention are similar to the specific working principle and benefits of the method for assisting the hearing impaired to provide communication provided by the embodiment of the present invention. More details.

In addition, another aspect of the embodiments of the present invention provides a system for assisting a hearing impaired person to communicate. The system includes: the device described in the foregoing embodiment and a client. The client can receive text input by the user; and / or can receive location information of the hearing impaired.

In addition, another aspect of the embodiments of the present invention provides an augmented reality glasses. The augmented reality glasses include the devices described in the above embodiments.

The augmented reality glasses include an electronic circuit system that supports the operation of the device described in the foregoing embodiment. The electronic circuit system includes a power source, a processor, a network connection module, and a voice receiving module, a text receiving module, and a voice playing module. In addition, the electronic circuit system may further include an externally visible human-machine interface module and buttons and / or a touch control panel. The processor includes the determination module, the speech recognition module, and the text conversion module described in the foregoing embodiments. The human-machine interface module includes a display module. The processor can also perform offline speech recognition locally, or online speech recognition in the cloud via a network connection.

Optionally, in the embodiment of the present invention, the touch control panel, the buttons, and / or the voice receiving module may be provided on the glasses or glasses accessories of the augmented reality glasses, for example, on the temples, frames, or lenses. Optionally, in the embodiment of the present invention, the voice receiving module may be disposed on the frame, on the same temple or on a different temple, or at a position close to the ears (both ears or monaural), to reach the pole. Try to fit the ear. For example, when the voice receiving module is a microphone array and the microphone sub-array includes two microphones, the two microphones are respectively disposed on two frames, or are disposed on different positions of the same temple, or are respectively disposed on two On the temples. When the number of microphones included in the microphone array is greater than two, a plurality of microphones may also be respectively arranged on the frame and / or the temple according to the actual situation. In addition, when using a microphone array, there is a difference in the time and intensity of speech reaching each microphone in the microphone array. By calculating the difference, a clearer sound that is more convenient to process can be obtained. In addition, compared to using a single microphone or a noise reduction microphone, the use of a microphone array is of great significance. Using a microphone array does not require the distance of the sound source from the voice receiving module. In addition, the use of the microphone array can adapt to various distances and can meet the requirements in most communication scenarios. The distance refers to the distance between the sound source and the microphone array. For example, it can meet the requirements of the following communication scenarios: two people talk individually, the distance between the sound source and the voice receiving module is between 50cm and 1m; in a multi-person group conversation, the sound source is between the distance of 1m and 2m from the voice receiving module; conference , The distance between the sound source and the voice receiving module is 3m; in class, the distance between the sound source and the voice receiving module is 3m to 5m, and so on.

In addition, in the case where the display module is a near-eye display, the orientation of each sound source and the text converted by the voice are presented in front of the eyes. Among them, the near-eye display may be transparent or non-transparent. Further, in the case that the near-eye display is a see-through near-eye display, it does not affect the hearing-impaired person's observation of the display scene, and through the graphical instructions superimposed on the real scene, the hearing-impaired person can see each The position of a sound source and the text converted by the voice makes the hearing impaired person watching "subtitles" to understand the voice information heard and obtain the position perception similar to ordinary people. In addition, in order to avoid distraction of the hearing impaired, the near-eye display may be a monochrome display, which uses a preset background color and a preset foreground color to display the orientation and text corresponding to the sound source. In addition, the near-eye display can also be a color display. The background color and foreground color are alternately displayed to display the orientation and text corresponding to different sound sources. For specific conversion methods, refer to the content described in the foregoing embodiment. In this way, it can also be fully avoided. Hearing-impaired people are distracted, allowing them to focus on the content itself; at the same time, they allow normal hearing exchanges without the discomfort of being interrupted and needing to change their focus.

In addition, another aspect of the embodiments of the present invention further provides a machine-readable storage medium, where the machine-readable storage medium stores instructions, and the instructions are used to cause a machine to execute the method described in the foregoing embodiments.

To sum up, displaying the translated text of each sound source in at least one sound source enables the hearing impaired person to understand the content of each sound source's speech; displaying the position of each sound source in at least one sound source, achieving A simple and intuitive way to indicate where the voice of the hearing impaired occurs. In this way, the hearing-impaired person can obtain the positional awareness similar to ordinary people while watching the "subtitles" to understand the voice information being heard, thereby enabling the hearing-impaired person to understand the content of the voice emitted by each sound source. Clearly and intuitively understand the position of each sound source, so that the hearing impaired can accurately grasp the content of the sound emitted by each sound source and the position of the sound source, which is convenient for the hearing impaired to communicate and communicate with others. In particular, in an environment where there are multiple sound sources, accurately grasping the content of the sound emitted by each sound source and the position of the sound source is very helpful for the communication of hearing impaired people with others. The text input by the hearing impaired is converted into speech and the converted speech is played. In this way, when the hearing impaired has no pronunciation ability or the pronunciation ability is limited, the hearing impaired person can express his meaning by entering text and communicate with others. In addition, the received speech is converted into words expressed in the language used by the "deaf hearing impaired" and / or the words entered by the "deaf hearing impaired" are converted into others who communicate with the "deaf The speech expressed in the language used thus realizes the communication between the "persons with hearing impairment" and others.

The optional implementations of the embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the foregoing implementations. Within the scope of the technical concept of the embodiments of the present invention, the embodiments of the present invention may be implemented. The technical solution of the present invention performs various simple modifications, and these simple modifications all belong to the protection scope of the embodiments of the present invention.

In addition, it should be noted that the specific technical features described in the foregoing specific embodiments can be combined in any suitable manner without conflict. In order to avoid unnecessary repetition, the embodiments of the present invention do not separately describe various possible combinations.

Those skilled in the art can understand that all or part of the steps in the method of the above embodiments can be completed by a program instructing related hardware. The program is stored in a storage medium and includes a number of instructions to enable a microcontroller, a chip, or a processor. (processor) executes all or part of the steps of the method described in each embodiment of the present application. The foregoing storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks and other media that can store program codes .

In addition, various combinations of the embodiments of the present invention can also be arbitrarily combined, as long as it does not violate the idea of the embodiment of the present invention, it should also be regarded as the content disclosed in the embodiment of the present invention.

Claims (17)

A method for assisting people with hearing impairment to communicate, which is characterized in that the method includes: Receiving voice from at least one sound source; Determining the position of each sound source in the at least one sound source based on the received speech of each sound source in the at least one sound source; Identifying the speech of each of the at least one sound source to convert the speech of each of the at least one sound source into text; and Display the position of each sound source in the at least one sound source and the text converted by the speech. The method according to claim 1, further comprising: Receive text Translate received text into speech; and Play the converted voice. The method according to claim 1 or 2, wherein the form of displaying the position of each sound source of the at least one sound source and the text converted by the voice is any of the following: using a preset foreground color and a preset The background color is set to display and the orientation and text corresponding to different sound sources are displayed by alternately changing the colors of the preset foreground color and the preset background color. The preset foreground color and the preset background color are different colors. The method according to claim 3, wherein: The preset foreground color is white, and the preset background color is green; or The preset foreground color is green, and the preset background color is white. The method according to claim 1 or 2, further comprising: Determining the location information of the hearing impaired; and Sending the location information to a mobile terminal and / or a client, so that the mobile terminal and / or the client obtains the location information in real time. The method according to claim 5, wherein before sending the location information to a mobile terminal and / or a client, the method further comprises: receiving a setting for a contact, wherein the mobile terminal and / or The client is a mobile terminal and / or client corresponding to the selected contact. A device for assisting people with hearing impairment to communicate, which is characterized in that the device includes: A voice receiving module, configured to receive voice of at least one sound source; A determining module, configured to determine the position of each sound source in the at least one sound source based on the received speech of each sound source in the at least one sound source; A voice recognition module, configured to recognize the voice of each of the at least one sound source to convert the voice of each of the at least one sound source into text; and The display module is configured to display the position of each sound source in the at least one sound source and the text converted by the voice. The apparatus according to claim 7, further comprising: Text receiving module for receiving text; A text conversion module for converting received text into speech; and The voice playback module is used to play the converted voice. The device according to claim 7, wherein the display module is a near-eye display. The device according to claim 9, wherein the near-eye display is a see-through near-eye display. The device according to any one of claims 7 to 10, wherein the form of displaying the position of each of the at least one sound source and the text converted by the voice is any one of the following: The foreground color and the preset background color are displayed, and the orientation and text corresponding to different sound sources are displayed by alternately changing the colors of the preset foreground color and the preset background color. Different colors. The device according to claim 11, wherein: The preset foreground color is white, and the preset background color is green; or The preset foreground color is green, and the preset background color is white. The device according to any one of claims 7 to 10, wherein the device further comprises: A positioning module for determining position information of the hearing impaired person; and A communication module is configured to send the location information to a mobile terminal and / or a client, so that the mobile terminal and / or the client obtains the location information in real time. The apparatus according to claim 13, further comprising: A contact setting module, configured to receive a setting of a contact before the communication module sends the location information to a mobile terminal and / or a client, wherein the mobile terminal and / or the client The mobile terminal and / or client corresponding to the specified contact. An augmented reality glasses, characterized in that the augmented reality glasses include the device according to any one of claims 7-14. A system for assisting the hearing impaired to communicate, characterized in that the system includes: The device of any one of claims 7-14; and Client. A machine-readable storage medium has instructions stored on the machine-readable storage medium, which are used to cause a machine to perform the method according to any one of claims 1-6.

Images