JP2015076774A - Communication support system, communication support method, and communication support program - Google Patents

Abstract

A communication support system, a communication support method, and a communication support program for supporting efficient communication using a computer terminal are provided. A control unit 21 of a support server 20 acquires a conversation person's motion data and executes a sign language recognition process using the motion data. Then, the control unit 21 outputs the sign language recognition result to the user terminals 10 and 15. Moreover, the control part 21 acquires the voice of a conversation person, and performs a speech recognition process. Then, the control unit 21 outputs the voice recognition result to the user terminals 10 and 15. Further, the control unit 21 specifies related words of the speech recognition result, performs weighting of the related words, and adjusts the sign language recognition process using the weights. [Selection] Figure 1

Description

  The present invention relates to a communication support system, a communication support method, and a communication support program for supporting communication performed using a computer terminal.

  An explanation support system for efficiently explaining a document using a computer terminal has been studied (for example, see Patent Document 1). In the technique described in this document, screen synchronization processing of two tablet terminals is executed.

  In addition, a technique for communicating with a visually impaired person or a hearing impaired person has been studied (see, for example, Patent Document 2). In the technique described in this document, a sender uses his / her mobile phone to input an image such as sign language, a character such as a keyboard operation, or a sound from a microphone. Then, the video information and voice information, the video information is converted into “sign language → character”, the voice information is converted into “voice → character”, and the voice information is converted into “character → voice”. Send to. The receiver selects video information, character information, or audio information and performs communication.

  In addition, it supports smooth communication between hearing-impaired and hearing-impaired people, texts sign language that realizes a barrier-free society in real time, and has nothing to do with anything and touches it with a computer using body movements and voices. Techniques for dialogue have also been studied (for example, see Non-Patent Document 1).

Japanese Patent Laying-Open No. 2013-25608 (first page, FIG. 1) Japanese Patent Laying-Open No. 2004-248022 (first page, FIG. 1)

Mizuho Information & Research Institute, Chiba University, Mizuho Information & Research Institute and Chiba University, Joint Development of “Sign Language Recognition System”, [online], September 4, 2013, Mizuho Information & Research Institute Home Page, [September 9, 2013 Day search], Internet <http://www.mizuho-ir.co.jp/company/release/2013/shuwa0904.html>

  As described above, sign language recognition technology has been studied to realize a barrier-free environment. However, when a predetermined input is recognized and converted into text, accurate recognition may be difficult depending on the recognition processing method and environment. For example, when a plurality of text candidates (synonyms) are specified for a specific sign language action or voice by the recognition process, there is a possibility that smooth communication according to the purpose of the conversation cannot be performed. Moreover, if only the recognized text is viewed, the state of the other party cannot be grasped, and accurate communication may not be possible.

  The present invention has been made to solve the above problems, and provides a communication support system, a communication support method, and a communication support program for supporting efficient communication using a computer terminal. is there.

  (1) A communication support system that solves the above problems includes a first acquisition unit that acquires first input information, a second acquisition unit that acquires second input information, and the first and second input information. A control unit that generates a recognition result obtained by converting the text into a text, and an output unit that outputs the recognition result. And the said control part is based on the 1st recognition process part which performs a text conversion based on the 1st input information acquired in the said 1st acquisition part, and the 2nd input information acquired in the said 2nd acquisition part. A recognition adjustment that adjusts the text conversion of the other recognition processing unit based on the recognition result of at least one of the first recognition processing unit and the second recognition processing unit that performs text conversion A part. According to this configuration, communication using different recognition methods can be realized. Furthermore, it is possible to support the recognition process of the other by using the recognition result of one.

  (2) The communication support system weights a word candidate included in a recognition result in a recognition processing unit having a high recognition rate among the first and second recognition processing units, and based on the weighting, It is preferable to adjust the text conversion of the recognition processing unit. Since common topics are used in communication, word candidates are weighted based on recognition results with a high recognition rate (correctly recognized rate), and recognition processing with a low recognition rate can be supported.

  (3) It is preferable that the communication support system calculates a recognition rate for each recognition processing unit according to the correction of the recognition result, and identifies a recognition processing unit with a high recognition rate based on the recognition rate. Thereby, it is possible to identify a recognition processing unit having a high recognition rate according to the correction status, and to support recognition processing having a low recognition rate. Therefore, even when the relative recognition rate in the first and second recognition processing units changes depending on the situation, the recognition process can be supported according to the situation.

  (4) The communication support system acquires a face image together with at least one of the first and second input information, and outputs the face image in association with a recognition result corresponding to the input information. It is preferable to output to the part. Thereby, it is possible to communicate while confirming the other party's facial expression.

  (5) It is preferable that the communication support system outputs the recognition result so that the face image is not superimposed on the output region. In the output unit, communication can be performed while confirming the facial expression together with the recognition result.

  According to the present invention, efficient communication can be supported using a computer terminal.

Explanatory drawing of the communication assistance system of embodiment of this invention. It is explanatory drawing of the process sequence of this embodiment, Comprising: (a) is a sign language corresponding | compatible process, (b) is explanatory drawing of an audio | voice corresponding | compatible process. It is explanatory drawing of the process sequence of this embodiment, Comprising: (a) is a recognition adjustment process, (b) is explanatory drawing of a sign language recognition process. Explanatory drawing of the display screen in the user terminal of this embodiment. It is explanatory drawing of the process sequence of other embodiment, Comprising: (a) is a sign language corresponding | compatible process, (b) is an audio | voice corresponding | compatible process, (c) is explanatory drawing of a recognition adjustment process. Explanatory drawing of the process sequence of other embodiment. Explanatory drawing of the process sequence of other embodiment. Explanatory drawing of the process sequence of other embodiment.

<First Embodiment>
An embodiment of a communication support system embodying the present invention will be described below with reference to FIGS. In the present embodiment, a case will be described in which a customer at a store makes a request for a procedure or the like at a counter of a financial institution and a person in charge of the counter responds. Here, it is assumed that the customer at the store communicates by sign language and the person in charge at the window communicates by utterance. Then, communication is supported by combining two recognition techniques (sign language recognition and voice recognition) in the first input information (sign language operation) and the second input information (voice). Specifically, assuming that the recognition rate (correct recognition rate) is higher in speech recognition than in sign language recognition, sign language recognition is supported using speech recognition results.

  As shown in FIG. 1, in this embodiment, user terminals 10 and 15 (output unit) installed at a counter of a financial institution are used. Furthermore, the user terminals 10 and 15 are connected to the support server 20 via a network.

  The user terminal 10 is a computer terminal (tablet terminal) used by a contact person in charge of a customer visit request, and the user terminal 15 is a computer terminal (tablet terminal) used by a customer visit. The user terminal 10 and the user terminal 15 communicate with the support server 20 by using wireless LAN communication or the like. Note that the communication method is not limited to wireless communication, and wired communication can also be used.

The user terminals 10 and 15 include a control unit and a touch panel display.
The touch panel display functions as input / output means, outputs information on the display, and when a touch on the display surface is detected, specifies the touch position (coordinates) and performs various operation processes (pointing process, key input process, etc.) To do. For example, when writing a handwriting, handwriting input is performed on the touch panel display.

  The support server 20 is a computer system for supporting communication using the user terminals 10 and 15. The support server 20 includes a control unit 21, a sign language recognition dictionary 22, a voice recognition dictionary 23, and a related word storage unit 24. Furthermore, a camera 31 and a microphone 32 are connected to the support server 20.

In the sign language recognition dictionary 22 (first recognition dictionary), data related to words is recorded with respect to motion patterns (features) used in sign language.
In the speech recognition dictionary 23 (second recognition dictionary), data relating to words is recorded with respect to speech patterns (features) used in utterance.
In the related word storage unit 24, a plurality of mutually related words are associated and registered as a group. Here, words belonging to the same group are treated as related words.

The camera 31 as the first acquisition unit functions as a photographing unit. Here, the camera 31 generates the photographed customer's face and motion (motion including depth information) as motion data. This action includes sign language actions such as upper arm movements and mouth movements, and includes information for identifying the sign language content.
The microphone 32 as the second acquisition unit functions as sound collection means. Here, the microphone 32 collects the voice of the person in charge of the window.

  The control unit 21 includes control means (CPU, RAM, ROM, etc.), and performs processes (each process such as a sign language recognition stage, a voice recognition stage, a recognition adjustment stage, a terminal control stage, and a writing support stage) to be described later. By executing the communication support program for that purpose, the control unit 21 functions as a sign language recognition unit 211, a speech recognition unit 212, a recognition adjustment unit 213, a terminal control unit 214, and a writing support unit 215, as shown in FIG. .

  A sign language recognition unit 211 serving as a first recognition processing unit identifies a sign language operation based on body image patterns of a mouth, an arm, a hand, and the like in a customer's operation photographed by the camera 31. Then, the sign language recognition unit 211 calculates a feature quantity of the movement, and executes a sign language text conversion process using the sign language recognition dictionary 22.

  The voice recognition unit 212 as the second recognition processing unit calculates the feature amount of the voice of the person in charge of the window collected by the microphone 32. Then, the speech recognition unit 212 executes speech text conversion processing using the speech recognition dictionary 23 based on the feature amount.

  The recognition adjustment unit 213 executes processing for adjusting a recognition method using text generated by sign language recognition and text generated by voice recognition. The recognition adjustment unit 213 includes a weighting memory. In this weighting memory, related words of the speech-recognized word are stored in association with the frequency (weighting). The weighting memory is reset when the customer service counter for one procedure request is terminated.

The terminal control unit 214 executes processing for controlling display and input on the touch panel display of the user terminals 10 and 15. In the present embodiment, text generated by sign language recognition or voice recognition and a character image in writing are displayed on the touch panel display, or information touched from the touch panel display is acquired.
The writing support unit 215 executes processing for generating a drawn character image based on the touch input trajectory acquired from the user terminals 10 and 15.

  Next, the operation in this communication support system will be described with reference to FIGS. In the following, description will be given in the order of sign language handling processing, voice handling processing, recognition adjustment processing, sign language recognition processing, and output processing.

(Sign language handling processing)
First, referring to FIG. 2A, a sign language handling process for a customer at the store will be described.
Here, the control unit 21 of the support server 20 executes a photographing process (step S1-1). Specifically, the sign language recognition unit 211 of the control unit 21 uses the camera 31 to capture the motion of the customer visiting the store and generate motion data, and acquires the motion data from the camera 31.

  Next, the control unit 21 of the support server 20 executes a determination process as to whether or not a writing input has been made (step S1-2). Specifically, the sign language recognition unit 211 of the control unit 21 determines whether or not touch input has been performed on the touch panel display of the user terminal 15. And when a continuous locus | trajectory is drawn by the touch input in a touchscreen display, it determines with a writing input.

When it determines with a writing input (in the case of "YES" in step S1-2), the control part 21 of the support server 20 performs a locus | trajectory identification process (step S1-3). Specifically, the sign language recognition unit 211 of the control unit 21 takes over the process to the writing support unit 215. In this case, the writing support unit 215 obtains a trajectory touch-input on the touch panel display from the user terminal 15.
This trajectory is output to the touch panel displays of the user terminals 10 and 15 as will be described later (step S1-7).

  On the other hand, when it is determined that there is no touch input on the touch panel display and it is not a writing input (in the case of “NO” in step S1-2), the control unit 21 of the support server 20 executes a sign language recognition process (step S1-4). ). Specifically, the sign language recognition unit 211 of the control unit 21 specifies word candidates using the sign language recognition dictionary 22 based on the motion pattern included in the motion data acquired from the camera 31. This process will be described later with reference to FIG.

  Next, the control unit 21 of the support server 20 executes word candidate output processing (step S1-5). Specifically, the sign language recognition unit 211 of the control unit 21 outputs the word candidates specified using the sign language recognition dictionary 22 to the touch panel display of the user terminal 15.

  Next, the control unit 21 of the support server 20 executes word candidate selection processing (step S1-6). Specifically, the customer confirms the word candidates displayed on the touch panel display of the user terminal 15. If there is an error in the word recognized in sign language, this word is selected on the touch panel display. In this case, the sign language recognition unit 211 of the control unit 21 outputs a list of other word candidates similar to the operation pattern to the touch panel display of the user terminal 15. Then, the sign language recognition word is determined by selecting the correct word on the touch panel display of the user terminal 15. If there is no mistake in the sign-recognized word, the sign-recognized word is confirmed by leaving it alone.

  Next, the control part 21 of the support server 20 performs an output process (step S1-7). Specifically, the terminal control unit 214 of the control unit 21 outputs the voice recognition result to the touch panel display of the user terminals 10 and 15. Details of the content output to the touch panel display will be described later with reference to FIG.

(Audio processing)
Next, the voice correspondence processing for the person in charge of the window will be described with reference to FIG. This process is performed in parallel with the sign language handling process shown in FIG.
Here, the control unit 21 of the support server 20 executes a voice acquisition process (step S2-1). Specifically, the voice recognition unit 212 of the control unit 21 acquires the voice of the person in charge using the microphone 32.

  Next, the control part 21 of the support server 20 performs a speech recognition process (step S2-2). Specifically, the voice recognition unit 212 of the control unit 21 calculates the feature amount of the voice collected by the microphone 32. Then, the voice recognition unit 212 acquires a word associated with this feature amount from the voice recognition dictionary 23. Here, the voice pattern having the feature quantity closest to the voice is specified, and the word associated with the voice pattern is specified as the word candidate.

  Next, the control unit 21 of the support server 20 executes word candidate output processing (step S2-3). Specifically, the voice recognition unit 212 of the control unit 21 outputs the word candidates specified using the voice recognition dictionary 23 to the touch panel display of the user terminal 10.

  Next, the control unit 21 of the support server 20 executes a word candidate selection process (step S2-4). Specifically, the contact person confirms the word candidates displayed on the touch panel display of the user terminal 10. If there is an error in the speech-recognized word, this word is selected on the touch panel display. In this case, the voice recognition unit 212 of the control unit 21 outputs a list of other word candidates similar to the voice pattern to the touch panel display of the user terminal 10. Then, by selecting the correct word on the touch panel display of the user terminal 10, the speech recognition word is confirmed. If there is no mistake in the speech-recognized word, the speech-recognized word is determined by leaving it as it is.

Next, the control unit 21 of the support server 20 executes recognition adjustment processing (step S2-5). This process will be described later with reference to FIG.
Next, the control part 21 of the support server 20 performs an output process (step S2-6). Specifically, the terminal control unit 214 of the control unit 21 outputs the voice recognition result to the touch panel display of the user terminals 10 and 15. Details of the content output to the touch panel display will be described later with reference to FIG.

(Recognition adjustment process)
Next, the recognition adjustment process (step S2-5) will be described with reference to FIG.
Here, the control unit 21 of the support server 20 executes related word identification processing (step S3-1). Specifically, the recognition adjustment unit 213 of the control unit 21 acquires the word specified by the voice recognition process from the voice recognition unit 212. Then, the recognition adjustment unit 213 acquires the related word of the word that has been voice-recognized from the related word storage unit 24.

  Next, the control part 21 of the support server 20 performs a weighting process (step S3-2). Specifically, the recognition adjustment unit 213 of the control unit 21 checks whether or not the related word acquired from the related word storage unit 24 is recorded in the weighting memory. If the acquired related word is not recorded in the weighting memory, the frequency “1” is recorded together with the related word. On the other hand, if the acquired related word is already recorded in the weighting memory, the frequency “1” associated with the related word is added.

(Sign language recognition processing)
Next, sign language recognition processing (step S1-4) will be described with reference to FIG.
Here, the control unit 21 of the support server 20 executes a movement specifying process for the mouth and the forearm (step S4-1). Specifically, the sign language recognition unit 211 of the control unit 21 specifies the mouth area and the upper arm part area using the body image pattern in the motion data acquired from the camera 31. Then, the sign language recognition unit 211 identifies the movement of the mouth or upper arm (the action of sign language) identified in the motion data.

  Next, the control unit 21 of the support server 20 executes a feature amount extraction process (step S4-2). Specifically, the sign language recognition unit 211 of the control unit 21 calculates a feature amount related to the direction and magnitude of the movement based on the movement of the mouth and upper arm (the sign language movement).

  Next, the control part 21 of the support server 20 performs a dictionary comparison process (step S4-3). Specifically, the sign language recognition unit 211 of the control unit 21 compares the calculated feature amount of motion with the motion pattern recorded in the sign language recognition dictionary 22 to calculate the degree of coincidence. Here, all words corresponding to an operation pattern having a matching degree equal to or higher than a reference value are specified as word candidates.

  Next, the control part 21 of the support server 20 performs a meaning estimation process (step S4-4). Specifically, the sign language recognition unit 211 of the control unit 21 performs matching with related words recorded in the weighting memory of the recognition adjustment unit 213. Then, the sign language recognition unit 211 identifies word candidates having a high degree of coincidence between the frequency of words recorded as related words in the weighting memory and the motion pattern.

(Output processing)
Next, output processing (steps S1-7 and S2-6) will be described with reference to FIG. Here, a display screen output to the touch panel display of the user terminal 10 used by the person in charge of the window will be described.

  Specifically, the control unit 21 of the support server 20 specifies the customer's face image in the motion data acquired from the camera 31. Then, the terminal control unit 214 of the control unit 21 outputs the face image 500 to the touch panel display of the user terminal 10. In this case, it is displayed large so that the customer's facial expression can be understood.

  Further, the terminal control unit 214 of the control unit 21 displays the sign language recognition results 510 and 511 in the sign language recognition unit 211 in the order of time passage. Specifically, in FIG. 4, the latest sign language recognition result 511 is displayed on the lower left of the touch panel display, and the old sign language recognition result 510 is moved upward.

Furthermore, the terminal control unit 214 of the control unit 21 displays the speech recognition results 520 and 521 in the speech recognition unit 212 in the order of passage of time. Specifically, in FIG. 4, the latest speech recognition result 521 is displayed on the lower left of the touch panel display, and the old speech recognition result 520 is moved upward. In this case, the terminal control unit 214 outputs the recognition result so as not to be superimposed on the area where the face image is output on the touch panel display.
Note that only the sign language recognition results 510 and 511 and the speech recognition results 520 and 521 are output to the touch panel display of the user terminal 15 used by the customer at the store.

As described above, according to the present embodiment, the following effects can be obtained.
(1) In the embodiment described above, the control unit 21 of the support server 20 executes the shooting process (step S1-1), the sign language recognition process (step S1-4), and the output process (step S1-7). Further, the control unit 21 of the support server 20 executes a voice acquisition process (step S2-1), a voice recognition process (step S2-2), and an output process (step S2-6). As a result, even when speech cannot be heard or when sign language is not possible, communication can be performed through text. Therefore, even when the communication partner and the communication method are different, communication can be smoothly realized.

  (2) In the above embodiment, the control unit 21 of the support server 20 executes recognition adjustment processing (step S2-5). Here, the control unit 21 of the support server 20 executes related word identification processing (step S3-1) and weighting processing (step S3-2). Thereby, when the speech recognition rate is higher than the sign language recognition rate, the sign language recognition process can be supported using the speech recognition result.

  (3) In the above embodiment, when it is determined that the input is a handwriting input (in the case of “YES” in step S1-2), the control unit 21 of the support server 20 executes a locus specifying process (step S1-3). Thereby, when sign language recognition processing is difficult, it can switch to writing and can communicate.

  (4) In the above embodiment, the control unit 21 of the support server 20 identifies the customer's face image in the motion data acquired from the camera 31. Then, the terminal control unit 214 of the control unit 21 outputs the face image 500 to the touch panel display of the user terminal 10. Thereby, communication can be aimed at, confirming a visitor's facial expression.

<Second Embodiment>
Next, a second embodiment will be described with reference to FIG. In the first embodiment, assuming that the speech recognition rate is higher than the sign language recognition rate, the sign language recognition processing is supported using the recognition result of the speech recognition processing. In the second embodiment, the recognition process (priority recognition method) used for support is determined according to the recognition rate of each recognition process, and detailed description of similar parts is omitted.
Here, in the first stage of conversation (a stage where the priority recognition method is not determined), the sign language recognition process and the voice recognition process are performed independently. Then, the priority recognition method is determined according to the progress of the conversation.

(Sign language handling processing)
First, the sign language handling process will be described with reference to FIG.
Here, the control unit 21 of the support server 20 executes the photographing process similarly to step S1-1 (step S5-1).

Next, the control part 21 of the support server 20 performs the determination process about whether it is a handwriting input similarly to step S1-2 (step S5-2).
When it is determined that the input is a handwriting input (in the case of “YES” in step S5-2), the control unit 21 of the support server 20 executes a trajectory specifying process as in step S1-3 (step S5-3).

  On the other hand, when it is determined that there is no touch input on the touch panel display and it is not a writing input (in the case of “NO” in step S5-2), the control unit 21 of the support server 20 executes a sign language recognition process (step S5-4). ). Here, when the priority recognition method is not determined, or when the sign language recognition is the priority recognition method, the sign language recognition unit 211 of the control unit 21 has the feature amount of the motion pattern recorded in the sign language recognition dictionary 22. Word candidates are identified in order from the closest. On the other hand, when the speech recognition is the priority recognition method, the sign language recognition unit 211 of the control unit 21 performs the frequency and operation of the words recorded as related words in the weighting memory, similarly to step S1-4. A word candidate having a high degree of matching with the pattern is specified.

  Next, similarly to steps S1-5 to S1-7, the control unit 21 of the support server 20 performs word candidate output processing (step S5-5), word candidate selection processing (step S5-6), and output processing. (Step S5-7) is executed.

(Audio processing)
Next, the voice correspondence process will be described with reference to FIG.
Here, the control part 21 of the support server 20 performs a voice acquisition process similarly to step S2-1 (step S6-1).

  Next, the control unit 21 of the support server 20 executes a voice recognition process (step S6-2). Here, when the priority recognition method is not determined or when the voice recognition is the priority recognition method, the voice recognition unit 212 of the control unit 21 performs the feature amount of the voice pattern recorded in the voice recognition dictionary 23. Word candidates are identified in order from the closest. On the other hand, when sign language recognition is a priority recognition method, the speech recognition unit 212 of the control unit 21 performs matching with related words recorded in the weighting memory of the recognition adjustment unit 213. Then, the speech recognition unit 212 identifies word candidates having a high degree of coincidence between the frequency of words recorded as related words in the weighting memory and the action pattern.

  Next, similarly to steps S2-3, S2-4, and S2-6, the control unit 21 of the support server 20 outputs a word candidate (step S6-3) and a word candidate selection process (step S6-4). ) And output processing (step S6-5).

(Recognition adjustment process)
Next, the recognition adjustment process will be described with reference to FIG. Here, the priority recognition method used in steps S5-6 and S6-4 is determined. This process is executed when the progress of the conversation has passed the reference time point. For example, as the reference time, a time when a predetermined number of words is converted in both recognition methods, a time when a predetermined time elapses, or the like can be used.
Here, the control unit 21 of the support server 20 performs recognition rate comparison processing (step S7-1). Specifically, the recognition adjustment unit 213 of the control unit 21 compares the recognition rates in sign language recognition and speech recognition. In the present embodiment, the recognition adjustment unit 213 uses, as a recognition rate, the ratio at which the word candidates that are output first are not corrected in the recognition processing up to the reference time.

  Next, the control unit 21 of the support server 20 executes a priority recognition method determination process based on the recognition rate (step S7-2). Specifically, the recognition adjustment unit 213 of the control unit 21 specifies a recognition method having a high recognition rate as the priority recognition method. Then, the recognition adjustment unit 213 notifies the sign language recognition unit 211 and the voice recognition unit 212 of which recognition method is prioritized. In response to this notification, as described above, sign language recognition processing (step S5-4) and voice recognition processing (step S6-2) are executed.

As described above, according to the present embodiment, the following effects can be obtained.
(5) In the above embodiment, the control unit 21 of the support server 20 executes the recognition rate comparison process (step S7-1) and the priority recognition method determination process (step S7-2) based on the recognition rate. Depending on the communication environment, the relative height of the recognition rate may be reversed. For example, when the surroundings are noisy, voice communication is difficult. Also in this case, the recognition rate can be evaluated according to the correction status, and the recognition process with the higher recognition rate can be used to support the other recognition process.

In addition, you may change each said embodiment into the following aspects.
In each of the above embodiments, it is used for communication support at a financial institution's window. The subject of the present invention is not limited to this, and can be applied to a mechanism that uses a plurality of types of recognition processing to improve the recognition rate in cooperation.

  In the second embodiment, the priority recognition method is determined based on the recognition rate. Instead, weighting may be performed using related words of words recognized by both recognition methods. In this case, the control unit 21 of the support server 20 executes recognition adjustment processing.

This recognition adjustment process will be described with reference to FIG.
Here, the control part 21 of the support server 20 performs a matching process (step S8-1). Specifically, the recognition adjustment unit 213 of the control unit 21 extracts the related word of the word selected in the sign language recognition process and the related word of the word selected in the speech recognition process from the related word storage unit 24. . And the recognition adjustment part 213 matches the related word extracted in each recognition system.

Next, the control part 21 of the support server 20 performs a weighting process based on matching (step S8-2). Specifically, the recognition adjustment unit 213 of the control unit 21 records related words common in the matching result in association with high weighting (high frequency) in the weighting memory.
Thereby, a plurality of recognition methods can be utilized to perform an accurate recognition process, and smooth communication can be realized.

  In each of the above embodiments, the user's face image, sign language recognition result, and voice recognition result are output to the user terminal 10. The user terminals 10 and 15 are not limited to tablet terminals, and desktop terminals and notebook terminals can also be used. Moreover, it is also possible to use a prompter that outputs the recognition result to the inclined half mirror as the display means. A head mounted display can also be used.

  In each of the above embodiments, the sign language recognition result and the speech recognition result are output to the user terminals 10 and 15. Here, the display form may be changed based on the recognition result. For example, in the voice handling process, a communication support process for outputting options corresponding to the voice recognition result may be performed.

The communication support process will be described with reference to FIG.
First, the control unit 21 of the support server 20 executes a voice correspondence process shown in FIG. 2B (step S9-1).

  Next, the control part 21 of the support server 20 performs the determination process about whether a speech recognition result is a question sentence (step S9-2). Specifically, the terminal control unit 214 of the control unit 21 determines whether the voice recognition result is questionable. For example, if a character string (final particle) representing a question such as “Is it?” Or “Is it?” Is detected at the end of the sentence, it is determined as a question sentence.

  When it determines with it not being a question sentence (in the case of "NO" in step S9-2), the control part 21 of the support server 20 performs a normal output process (step S9-3). Here, the speech recognition result is output as it is.

  On the other hand, when it determines with a question sentence (in the case of "YES" in step S9-2), the control part 21 of the assistance server 20 performs the output process of an option (step S9-4). Specifically, the terminal control unit 214 of the control unit 21 outputs options such as “Yes” and “No” along with the voice recognition result to the touch panel display of the user terminal 15 of the customer at the store.

Next, the control unit 21 of the support server 20 executes an option selection result output process (step S9-5). Specifically, when the terminal control unit 214 of the control unit 21 detects selection of an option on the touch panel display of the customer terminal 15 of the customer at the store, the selection result is output to the user terminals 10 and 15.
Thus, efficient communication can be realized using the options.

  In each of the above embodiments, the sign language recognition result and the speech recognition result are output to the user terminals 10 and 15. In addition to this, a communication support process for outputting a related service menu to the user terminal 15 according to the recognition result may be performed. In this case, the support server 20 is further provided with a service menu storage unit. The service menu storage unit stores data related to related words that may be recognized in sign language or speech for service menu items.

This communication support process will be described with reference to FIG.
First, the control unit 21 of the support server 20 executes related menu extraction processing according to the recognized word (step S10-1). Specifically, the recognition adjustment unit 213 of the control unit 21 extracts, from the service menu storage unit, a service menu item in which a sign language recognized or voice recognized word is registered as a related word.

  Next, the control unit 21 of the support server 20 executes a related menu weighting process (step S10-2). Specifically, the recognition adjustment unit 213 of the control unit 21 counts the number of related words recognized in sign language or speech for each service menu item recorded in the service menu storage unit. Then, the recognition adjustment unit 213 performs weighting on the service menu items of related words having a large number of recognitions.

Next, the control part 21 of the support server 20 performs a display change process according to weighting (step S10-3). Specifically, the recognition adjustment unit 213 of the control unit 21 preferentially displays a service menu item having a large weight on the touch panel display of the user terminal 15. For example, it is displayed in a form that can be easily selected.
Thus, efficient communication can be realized using the service menu item.

In each of the above embodiments, the writing support unit 215 of the control unit 21 of the support server 20 executes a process of generating a drawn character image based on the touch input trajectory acquired from the user terminals 10 and 15. Instead of this, the writing support unit 215 may execute an OCR process for converting the writing into text based on the touch input trajectory acquired from the user terminals 10 and 15. In this case, the support server 20 is provided with an OCR dictionary in which data relating to words according to the trajectory pattern is recorded. Thereby, it is possible to confirm handwritten characters by text.
Furthermore, sign language recognition and voice recognition may be supported based on the OCR recognition result. In this case, based on the OCR recognition result, a related word specifying process used in the sign language recognition process or the voice recognition process is performed.

  In each of the above embodiments, the control unit 21 of the support server 20 executes word candidate output processing (steps S1-5 and S5-5) and word candidate selection processing (steps S1-6 and S5-6). . Here, the sign language recognition unit 211 outputs the word candidates to the touch panel display of the user terminal 15, and the customer confirms the word candidates displayed on the touch panel display of the user terminal 15. If there is an error in the word recognized in sign language, this word is selected on the touch panel display. In addition, the sign language recognition unit 211 may output the word candidates to the touch panel display of the user terminal 10 so that the person in charge in charge can check the word candidates. And control part 21 of support server 20 performs selection processing of a word candidate by selection in user terminal 10 (Steps S1-6, S5-6). Thereby, since the sign language recognition can be advanced based on the judgment of the person in charge of the counter, it is possible to reduce the work load of the visiting customer.

  In each of the above embodiments, the face image 500 of the customer at the store is output to the touch panel display of the user terminal 10 in the output process. In addition to this, the face image of the person in charge of the window taken with another camera may be output to the touch panel display of the user terminal 15.

  DESCRIPTION OF SYMBOLS 10,15 ... User terminal, 20 ... Support server, 21 ... Control part, 211 ... Sign language recognition part, 212 ... Speech recognition part, 213 ... Recognition adjustment part, 214 ... Terminal control part, 215 ... Handwriting support part, 22 ... Sign language Recognition dictionary, 23 ... voice recognition dictionary, 24 ... related word storage unit, 31 ... camera, 32 ... microphone.

Claims (7)

A first acquisition unit for acquiring first input information;
A second acquisition unit for acquiring second input information;
A control unit that generates a recognition result obtained by text-converting the first and second input information;
A communication support system comprising an output unit for outputting the recognition result,
The controller is
A first recognition processing unit that performs text conversion based on the first input information acquired in the first acquisition unit;
A second recognition processing unit that performs text conversion based on the second input information acquired by the second acquisition unit;
A communication support system comprising: a recognition adjustment unit that adjusts text conversion of the other recognition processing unit based on a recognition result of at least one of the first and second recognition processing units. Of the first and second recognition processing units, weight the word candidates included in the recognition result in the recognition processing unit with a high recognition rate,
The communication support system according to claim 1, wherein the text conversion of the other recognition processing unit is adjusted based on the weighting. For each recognition processing unit, calculate the recognition rate according to the correction of the recognition result,
The communication support system according to claim 1, wherein a recognition processing unit having a high recognition rate is specified based on the recognition rate.   A face image is acquired together with at least one of the first input information and the second input information, and the face image is output to the output unit in association with a recognition result corresponding to the input information. The communication support system according to any one of claims 1 to 3.   The communication support system according to claim 4, wherein the recognition result is output so that the face image is not superimposed on the output region. A first acquisition unit for acquiring first input information;
A second acquisition unit for acquiring second input information;
A control unit that generates a recognition result obtained by text-converting the first and second input information;
A communication support system using a communication support system including an output unit that outputs the recognition result,
The controller is
A first recognition process for performing text conversion based on the first input information acquired by the first acquisition unit;
A second recognition process for performing text conversion based on the second input information acquired in the second acquisition unit;
And a recognition adjustment process for adjusting text conversion of the other recognition process based on a recognition result of at least one of the first and second recognition processes. A first acquisition unit for acquiring first input information;
A second acquisition unit for acquiring second input information;
A control unit that generates a recognition result obtained by text-converting the first and second input information;
A communication support system using a communication support system including an output unit that outputs the recognition result,
The control unit
A first recognition processing unit that performs text conversion based on the first input information acquired by the first acquisition unit;
A second recognition processing unit for performing text conversion based on the second input information acquired in the second acquisition unit;
A communication support program that functions as a recognition adjustment unit that adjusts text conversion of the other recognition processing unit based on a recognition result of at least one of the first and second recognition processing units.