JP2004110615A - Sign Language Interpretation System - Google Patents

Abstract

A sign language interpreting system is provided that enables a deaf person using different sign languages or a deaf person and a listener to have a smooth conversation.
A sign language interpreting system according to the present invention transmits and receives a moving image and audio data between at least one terminal device used by a deaf person or a listener and the terminal device via a communication medium. The sign language interpreting center 3 includes a server 4 and a plurality of sign language interpreting devices 5. A plurality of sign language interpreters capable of interpreting a plurality of sign language languages are stationed at the sign language interpreting center 3, and if it is necessary to have a conversation between different deaf people or between deaf people and listeners, a moving image of the sign language is transmitted to the terminal. Since the sign language interpreter interprets the sign language by transmitting it from the device 1 to the sign language interpreting device 5 and transmitting the result to the terminal device 1 by voice or sign language, the persons who cannot understand the sign language are comfortable and almost uncomfortable. You can talk in real time.
[Selection diagram] Fig. 1

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a sign language interpreting system for assisting conversation between deaf people who handle different kinds of sign languages or conversation between deaf people and listeners.
[0002]
[Prior art]
Sign language widely used in Japan can be roughly classified into two types. One is called Japanese Sign Language, which is traditionally handed down from generation to generation by Deaf people. This Japanese sign language is a completely different language from Japanese. The other is called Japanese-language sign language, which attempts to express Japanese in sign language. Japanese-language sign language has a shorter history and is more modern than Japanese sign language.
[0003]
While Japanese sign language is often used by innate deaf people, Japanese-language sign language is often used by acquired deaf people.
[0004]
As described above, it is the same in foreign countries that a plurality of types of sign language coexist in Japan. For example, in the United States, there are American sign language and English-language sign language.
[0005]
[Problems to be solved by the invention]
Mastering Japanese Sign Language is as difficult as mastering a foreign language because Japanese Sign Language has a different grammar than Japanese. In this regard, Japanese-language sign language can be spoken as soon as words are mastered.
[0006]
Deaf people who speak Japanese Sign Language can easily understand Japanese-language if they can understand Japanese, but many acquired Deaf people cannot understand Japanese-language sign language. For this reason, interpreters may be needed even in conversations between Japanese sign language speakers and Japanese sign language speakers.
[0007]
Also, what is taught as sign language in current Japanese society is often Japanese-language sign language, and even the existence of two types of sign language is not fully recognized. Although it is difficult to master Japanese sign language, children whose listeners are deaf parents are in an environment where they can master both Japanese sign language and Japanese, and can understand both Japanese sign language and Japanese sign language. Highly likely to be an interpreter.
[0008]
The present invention has been made in view of such a point, and an object of the present invention is to provide a sign language interpreting system that enables smooth conversation between deaf people using different sign languages or between deaf people and listeners. To provide.
[0009]
[Means for Solving the Problems]
In order to solve the above-mentioned problem, the present invention provides a center having a plurality of sign language interpreters used by a plurality of sign language interpreters, and at least one unit for communicating voice and moving image data between the centers. A first image capturing unit that captures a facial expression and hand movements of a deaf person as a moving image, a first voice capturing unit that captures a speaker's voice, A first communication unit for transmitting a moving image and a voice to the center; a first display processing unit for displaying a moving image transmitted from the sign language interpreting device on a first display device; Each of the devices includes a second display processing unit that displays a moving image transmitted from the terminal device on a second display device, and sign language information included in the moving image displayed on the second display device. Through A second voice processing unit that captures the voice obtained as a result, a second image capture unit that captures the facial expression of the sign language interpreter and hand movements as a moving image, and transmits the captured voice and moving image to the terminal device. And a second communication unit.
[0010]
According to the present invention, voice and video data are transmitted and received between a terminal device and a sign language interpreting device, so that deaf people who handle mutually different sign language, or deaf people and listeners, can have a conversation in almost real time. I do.
[0011]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, the sign language interpreting system according to the present invention will be specifically described with reference to the drawings.
[0012]
FIG. 1 is a block diagram showing the overall configuration of an embodiment of the sign language interpreting system according to the present invention. The sign language interpreting system shown in FIG. 1 is a sign language interpreting center for transmitting and receiving a moving image and audio data between at least one terminal device 1 used by a deaf person or a listener and the terminal device 1 via a communication medium 2. 3 is provided.
[0013]
Although the type of the communication medium 2 is not particularly limited, the communication medium 2 capable of stably transmitting a moving image (streaming image) having a resolution of 350,000 pixels or more at 20 frames / second or more is desirable. Specific examples of this type of communication medium 2 include FTTH and ADSL.
[0014]
The sign language interpreting system of FIG. 1 is intended to hold conversations between deaf people who handle different sign languages, or between deaf people and listeners.
[0015]
The sign language interpreting center 3 of FIG. 1 includes a server 4 and a plurality of sign language interpreting devices 5. The sign language interpreter 5 is used by a sign language interpreter, and is provided in a plurality so as to be able to quickly cope with a plurality of types of sign language, so that a plurality of sign language interpreters can simultaneously respond to a plurality of deaf people. .
[0016]
The terminal device 1 includes a portable device that can be easily carried by a deaf person or a listener and a stationary device that is installed in a place where a listener needs to talk with a deaf person (such as a government office or a hospital). is there.
[0017]
For example, when a conversation is performed between deaf persons, the conversation is performed via the sign language interpreting center 3 using the portable terminal devices 1 possessed by each deaf person. Also, when it becomes necessary for a listener to have a conversation with a Deaf person at a government office or the like, the Deaf person is guided to a place where the stationary terminal device 1 is located, and the listener uses the terminal device 1 to interpret the sign language. Conversation with Deaf people through Center 3.
[0018]
FIG. 2 is a perspective view illustrating an example of the portable terminal device 1, and FIG. 3 is a perspective view illustrating an example of the stationary terminal device 1. These terminal devices 1 include an imaging unit 6 (for example, a CCD camera) capable of capturing the facial expression and hand movements of a deaf person performing sign language as a moving image, and a moving image and voice transmitted from the sign language interpreting center 3. And a keyboard 8 for transmitting and receiving character information and control information to and from the sign language interpreting center 3.
[0019]
Deaf people are deaf and do not notice incoming calls when they ring. Therefore, in the present embodiment, the deaf person is provided with a wristwatch-type vibrator 10 as shown in FIGS. When an incoming signal arrives at the terminal device 1 from the sign language interpreting center 3, the incoming signal transmitting unit in the terminal device 1 drives the vibrator 10 wirelessly. Further, not only the vibrator 10 may notify the incoming call, but also the lamp of the terminal device 1 may be turned on.
[0020]
FIG. 4 is a detailed block diagram illustrating an example of the internal configuration of the terminal device 1. As illustrated, the terminal device 1 includes a communication processing unit 21 that controls data communication with the sign language interpreting center 3, a demodulation unit 22 that demodulates data received from the sign language interpreting center 3, and a transmission to the sign language interpreting center 3. Modulation unit 23 that modulates data, incoming call detection unit 24 that detects an incoming call from sign language interpreting center 3, incoming call signal transmitting unit 25 that notifies vibrator 10 of the incoming call, and data received from sign language interpreting center 3. An audio processing unit 26 for processing audio data, a speaker 8 for reproducing audio, an image processing unit 28 for processing image data included in data received from the sign language interpreting center 3, and a display unit for displaying images 7, a warning detection unit 29 that detects a warning signal from the sign language interpreting center 3, a frame drop detection unit 30 that detects a frame drop of a moving image, a facial expression and hand of a deaf person An image capturing unit 6 for capturing motion, an image processing unit 31 for processing a captured image, a microphone 32 for capturing the voice of a deaf or a listener, a voice processing unit 33 for processing the captured voice, and selection of a sign language And a warning output unit 35 that warns the sign language interpreting apparatus 5 of a missing frame.
[0021]
The speaker 8 and the microphone 32 may use the receiver provided on the telephone as it is, or may be provided separately from the receiver.
[0022]
FIG. 5 is a detailed block diagram showing an example of the internal configuration of the sign language interpreting apparatus 5 in the sign language interpreting center 3. As illustrated, the sign language interpreting apparatus 5 includes a communication processing unit 41 that controls data communication with the terminal device 1, a demodulation unit 42 that demodulates data received from the terminal device 1, and a transmission data to the terminal device 1. A modulating unit 43 for modulating a signal, a sound processing unit 44 for processing sound data included in data received from the terminal device 1, a speaker 45 for reproducing sound, and image data included in data received from the terminal device 1. An image processing unit 46 for performing the above processing, a display unit 47 for displaying an image, a warning detection unit 48 for detecting a warning signal from the terminal device 1, a dropped frame detection unit 49 for detecting a dropped frame of a moving image, A sign language detection unit 50 for detecting the sign language selected by the terminal device 1, an imaging unit 51 for capturing the facial expression and hand movements of the sign language interpreter, an image processing unit 52 for processing the captured image, and a sign language Interpreter's voice A microphone 53, an audio processor 54 for processing the audio captured, and the warning output unit 55 to alert the frame dropping with respect to the sign language interpretation system 5, the Komu Ri.
[0023]
A deaf person or a listener using the terminal device 1 selects a sign language by the sign language selection unit 34 before starting sign language. The sign language selection unit 34 selects a sign language input by a deaf person or a listener by operating buttons or the keyboard 9 provided on the terminal device 1. The selected sign language is sent to the sign language interpreting center 3 and detected by the sign language detecting unit 50 in the sign language interpreting apparatus 5.
[0024]
The sign language interpreting center 3 has a plurality of sign language translators who interpret each of the plurality of sign language languages, and selects a sign language translator capable of interpreting the sign language selected by the terminal device 1. When the sign language is selected by the terminal device 1 and a sign language interpreter who can interpret the sign language is not available immediately, it is desirable to reply to the terminal device 1 to that effect.
[0025]
Sign language translators who are resident at the Sign Language Interpretation Center 3 can interpret Japanese sign language and Japanese-language sign language, as well as foreigners who are deaf. It is desirable to have a resident.
[0026]
Also, instead of having the sign language interpreter resident at the sign language interpreter center 3, the sign language interpreters who are connected to the sign language interpreter 3 via the network are connected to each other via a network. An interpreter may be provided. In this case, the sign language interpreting center 3 may send the moving image of the deaf person to the sign language interpreter, and the sign language interpreter may perform the interpretation, or the sign language interpreter may directly communicate with the deaf person according to the instruction from the sign language interpreting center 3. The moving image may be received.
[0027]
As a result, a large number of sign language interpreters capable of interpreting various sign language languages can be networked, and an interpreting service can be simultaneously provided to a large number of deaf people, thereby improving the utility value.
[0028]
The terminal device 1 and the sign language interpreting center 3 perform data communication using the high-speed communication medium 2. However, communication traffic occurs due to time zones and the like, and high-speed data communication may not be performed if conditions are poor. In this case, the moving image representing the sign language may be dropped, and the sign language interpreter or deaf person may not understand the content.
[0029]
In order to deal with such a problem, in the present embodiment, in both the terminal device 1 and the sign language interpreting device 5, dropped frame detection units 30, 49, warning output units 35, 55, and warning detection units 29, 48. Are provided. When the dropped frames are detected by the dropped frames detecting units 30 and 49, a warning signal is transmitted from the warning output units 35 and 55. Further, when a warning signal is detected by the warning detection units 29 and 48, a warning is issued to a deaf person or a sign language interpreter that sign language has occurred, and the user is prompted to reduce the speed of the sign language. As a specific warning method, for example, a dedicated lamp provided in the terminal device 1 is turned on or blinks. Alternatively, it may be displayed on the display device 7 of the terminal device 1 that the dropped frame has occurred.
[0030]
If the speed of the sign language is reduced, no frame drop occurs even if the number of frames per second is reduced, so that the number of frames per second can be reduced, thereby reducing the data amount of the moving image.
[0031]
As another method for coping with dropped frames, a background image irrelevant to sign language may be deleted to reduce the data amount of the moving image. The images required for sign language are the facial expression of the signer and the movement of the hands, and the other background images are not so important for understanding sign language. Therefore, the background image is cut (more specifically, the background image is set to a solid color) to reduce the amount of moving image data.
[0032]
As another method for coping with dropped frames, the number of pixels of a moving image may be reduced, or the screen size may be reduced to reduce the amount of data. However, if the number of pixels is reduced, the image becomes coarse, and if the screen size is reduced, the screen becomes difficult to see, and in any case, it becomes difficult to grasp facial expressions and hand movements. Therefore, it is not desirable to extremely reduce the number of pixels or reduce the screen size.
[0033]
If the above-described countermeasures against dropped frames are taken, the amount of information transmitted per unit time naturally decreases. Even if the amount of information transmission is reduced, there is no particular financial problem in an environment of constant connection, but in an environment where charges are made according to the communication time of an ISDN line or the like, the communication time becomes longer if dropped frames occur. Therefore, the user's financial burden increases. For this reason, in such an environment, it is desirable to charge according to the actually transmitted information transmission amount instead of the communication time. For example, a charging system that charges according to the amount of packets employed in mobile phones and the like is desirable.
[0034]
By the way, as a method of transmitting moving image and audio data, there is a possibility that a high transmission rate can be obtained, but a best effort type transmission method where the transmission rate is not guaranteed, and a minimum transmission rate instead of limiting the maximum transmission rate However, in the present embodiment, since a broadband line is used and measures against dropped frames are taken, it is desirable to adopt a best-effort transmission method if possible.
[0035]
When a deaf person who speaks different sign languages speaks via the sign language interpreting center 3, the deaf person's telephone number, IP address, etc. (hereinafter collectively referred to as identification information) are input each time the conversation is made. The burden on deaf people is great. Therefore, it is desirable that the identification information of the deaf person be registered in the sign language interpreting center 3 collectively, so that each deaf person can acquire the identification information of another deaf person from the sign language interpreting center 3.
[0036]
FIG. 6 is a block diagram of the server 4 showing an example in which the server 4 in the sign language interpreting center 3 manages identification information of a plurality of deaf people. The server 4 of FIG. 6 includes an identification information storage unit 61 for storing identification information of a deaf person, an identification information updating unit 62 for updating identification information of a deaf person, and identification information for providing identification information of a deaf person who has made a request. And a providing unit 63.
[0037]
When the deaf person A has a conversation with another deaf person B via the sign language interpreting center 3, the name and the like of the deaf person B are transmitted to the sign language interpreting center 3, and the server 4 in the sign language interpreting center 3 becomes deaf person. The identification information of B is provided to the deaf person A, or the deaf person A and the deaf person B are automatically connected to a network.
[0038]
Deaf people do not speak well, so there is a general tendency that they are not good at remembering others' names. Therefore, when the information registered in the identification information storage unit 61 of the server 4 is provided to the deaf person, as shown in FIG. It is desirable to provide the character information (for example, the name of the area where the person lives, a nickname, a hobby, etc.) as a set to the deaf person who made the search.
[0039]
Thereby, the deaf person can visually grasp the partner with whom he or she wants to have a conversation with the still image and the character information, and can easily remember the partner of the call.
[0040]
Also, when a deaf person searches for information registered in the identification information storage unit 61, it is desirable to be able to search not only by name but also by area name or nickname.
[0041]
For example, when the deaf person A calls the deaf person B for the purpose of having a conversation with the deaf person B, the screen of the terminal device 1 of the deaf person B is provided with character information as shown in FIG. A still image of the deaf person A is displayed. Here, when the deaf person B presses a specific button of the terminal device 1, a still image of the deaf person B is transmitted to the deaf person A. Thereby, the deaf person A can visually confirm that the deaf person B has responded.
[0042]
In the above-described embodiment, an example has been described in which deaf people who handle different sign language languages or a deaf person and a listener have a conversation via the sign language interpreting center 3. It can also be used for conversations. In this case, the sign language interpreting center 3 is not used. However, compared to the conventional videophone system, this system uses a broadband line and takes measures against dropped frames. It is as easy to use as you are, and you can have a conversation without discomfort.
[0043]
As described above, in the present embodiment, it is necessary to have a plurality of sign language interpreters who can interpret a plurality of sign language languages resident at the sign language interpreting center 3 and have a conversation between different deaf persons or between deaf persons and listeners. When this occurs, a sign language moving image is transmitted from the terminal device 1 to the sign language interpreting device 5, and the sign language interpreter interprets the sign language, and transmits the result to the terminal device 1 by voice or sign language, so that the sign language cannot be understood. It is possible for a person to have a conversation in almost real time without any discomfort.
[0044]
In particular, according to the present embodiment, communication between a deaf person or between a deaf person and a listener can be made more closely, and discrimination against a deaf person and lack of communication due to a difference in sign language may be eliminated. Is big.
[0045]
Also, when the data communication speed between the terminal device 1 and the sign language interpreting device 5 is slowed down for some reason and a frame drop occurs, the fact is notified to the transmitting side of the moving image. By reducing the number of frames per second at a low speed, the data amount of the moving image can be reduced.
[0046]
In the above-described embodiment, the communication between the terminal device 1 and the sign language interpreting device 5 may be performed via the Internet, or may be performed via a dedicated line of a telephone company or a provider.
[0047]
In addition, the terminal device 1 itself does not necessarily have to have a communication function with the sign language interpreting device 5. For example, a mobile phone is connected to the terminal device 1, and data communication with the sign language interpreting device 5 is performed via the mobile phone. May go. Alternatively, if the terminal device 1 has a wireless LAN function, data communication with the sign language interpreting device 5 may be performed via a router or a wireless access point.
[0048]
【The invention's effect】
As described in detail above, according to the present invention, since voice and video data can be transmitted and received between the terminal device and the sign language interpreting device, deaf people who handle mutually different sign language, or deaf people And the listeners can have a conversation in almost real time. In particular, by having a plurality of sign language interpreters capable of interpreting various sign language languages resident at the center, it is possible to perform interpretation with deaf people who handle various sign language languages.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an entire configuration of an embodiment of a sign language interpreting system according to the present invention.
FIG. 2 is a perspective view showing an example of a portable terminal device.
FIG. 3 is a perspective view showing an example of a stationary terminal device.
FIG. 4 is a detailed block diagram showing an example of an internal configuration of the terminal device.
FIG. 5 is a detailed block diagram showing an example of an internal configuration of a sign language interpreting apparatus in a sign language interpreting center.
FIG. 6 is a block diagram of a server showing an example in which a server in a sign language interpreting center manages identification information of a plurality of deaf people.
FIG. 7 is a diagram illustrating an example in which a registered still image of the upper body of a deaf person and character information characteristic of the deaf person are combined and provided to a deaf person who has performed a search.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Terminal device 2 Communication medium 3 Sign language interpreting center 4 Server 5 Sign language interpreting device 6,51 Imaging unit 7,47 Display unit 8,45 Speaker 9 Keyboard 10 Vibrator 21,41 Communication processing unit 22,42 Demodulation unit 23,43 Modulation unit 24 incoming call detection unit 25 incoming signal transmission unit 26, 33, 44, 54 voice processing unit 28, 31, 46, 52 image processing unit 29, 48 warning detection unit 30, 49 dropped frame detection unit 32, 53 microphone 34 sign language selection Department

Claims (11)

A center having a plurality of sign language interpreters used by a plurality of sign language interpreters,
And at least one terminal device that performs communication of voice and moving image data with the center,
The terminal device,
A first image capturing unit that captures a facial expression and hand movements of a deaf person as a moving image;
A first voice capturing unit for capturing the voice of the speaker;
A first communication unit for transmitting the captured moving image and audio to the center;
A first display processing unit for displaying a moving image transmitted from the sign language interpreting device on a first display device,
Each of the plurality of sign language interpreters,
A second display processing unit for displaying a moving image transmitted from the terminal device on a second display device;
A second voice processing unit that captures a voice obtained by the sign language interpreter interpreting the sign language information included in the moving image projected on the second display device;
A second image capturing unit that captures the facial expression and hand movement of the sign language interpreter as a moving image;
A second communication unit for transmitting the captured voice and moving image to the terminal device. The terminal device,
A first frame skip detection unit that detects whether a transmission speed of a moving image transmitted from the second communication unit is equal to or lower than a predetermined speed limit;
A first warning unit configured to generate a warning signal to the sign language interpreting device that has transmitted a moving image when the first frame skip detection unit detects that the speed has become equal to or less than the speed limit,
Each of the plurality of sign language interpreters,
A second frame skip detection unit that detects whether a transmission speed of a moving image transmitted from the first communication unit has fallen below a predetermined speed limit;
A second warning unit configured to generate a warning signal to the terminal device that has transmitted a moving image when the second frame skip detection unit detects that the speed has fallen below the speed limit. Item 1. The sign language interpreting system according to Item 1. Upon receiving the warning signal from the second warning unit, the terminal device instructs a deaf person to reduce the speed of sign language,
The sign language interpreting system according to claim 2, wherein the sign language interpreting device, upon receiving the warning signal from the first warning unit, instructs the sign language interpreter to reduce the speed of the sign language. When the terminal device receives the warning signal from the second warning unit, according to the transmission speed of the moving image, reduces the number of frames per second of the moving image transmitted to the sign language interpreter,
The sign language interpreter, upon receiving a warning signal from the first warning unit, reduces the number of frames per second of the moving image transmitted to the terminal device according to the transmission speed of the moving image. The sign language interpreting system according to claim 2 or 3, wherein The terminal device, upon receiving the warning signal from the second warning unit, simplifies the background image other than the facial expression of the deaf person and the hand movement included in the moving image transmitted to the sign language interpreting device. Reduce the amount of video data,
Upon receiving the warning signal from the first warning unit, the sign language interpreter simplifies the background image other than the facial expression and hand movements of the sign language interpreter included in the moving image transmitted to the terminal device. The sign language interpreting system according to claim 3 or 4, wherein the amount of data of the moving image is reduced by performing the operation. Upon receiving the warning signal from the second warning unit, the terminal device includes a background image other than the facial expression of the deaf person and the hand movement included in the moving image transmitted to the sign language interpreting device in a plain monochrome color. Set,
When receiving the warning signal from the first warning unit, the sign language interpreting apparatus converts the background image other than the facial expression of the sign language interpreter's face and the hand movement included in the moving image transmitted to the terminal apparatus into a solid color image. The sign language interpreting system according to claim 5, wherein the setting is set to: The terminal device, when receiving the warning signal from the second warning unit, by reducing the resolution of the moving image, to reduce the number of frames per second of the moving image transmitted to the sign language interpreter,
When receiving the warning signal from the first warning unit, the sign language interpreter reduces the resolution of the moving image, thereby reducing the number of frames per second of the moving image transmitted to the terminal device. The sign language interpreting system according to claim 2, wherein The terminal device according to any one of claims 1 to 7, wherein the terminal device has an incoming notification unit that notifies an incoming notification to an incoming notification device held by a deaf person when receiving a moving image or a voice from the sign language interpreting device. Sign language interpretation system described in Crab. The incoming call notification device is a wristwatch-type vibrator having a wireless function,
The sign language interpreting system according to claim 8, wherein the incoming call notification unit notifies the incoming call by vibrating the vibrator by a wireless signal. The center is
A communication information storage unit that stores information necessary for communication with each of the terminal devices;
A communication information updating unit that updates information stored in the communication information storage unit;
The sign language interpreting system according to any one of claims 1 to 9, further comprising: a communication information providing unit that provides information stored in the communication information storage unit to the terminal device as needed. The communication information storage unit, for each of a plurality of deaf people, stores a still image of the deaf person, name and characteristic identification information,
The communication information providing unit, in response to an instruction from the terminal device, a still image of any deaf person stored in the communication information storage unit, providing the terminal device with the name and characteristic identification information,
The first display processing unit of the terminal device displays a still image, a name, and characteristic identification information of a deaf person provided from the communication information providing unit on the same screen of the first display device. The sign language interpreting system according to claim 10.

Images