JP6179036B2 - Input support apparatus, input support method, and program - Google Patents

Description

  The present invention relates to a technique for predicting a character string that a user intends to input.

  Conventionally, in a mobile phone equipped with a touch panel and PDA (Personal Digital Assistants), character input is performed with a software keyboard. In character input, for a definite and correct input character, prediction such as forward match or partial match is often performed, and the kana / kanji sentence is presented, or the subsequent characters are presented in a supplemented form.

  On the other hand, when the user operates the touch panel, a user may press a specific button on the software keyboard, but an unintended button may react. For this reason, conventionally, there has been proposed a technique for presenting a corrected character based on a probability and a score with respect to a character input by the user by mistake.

  Patent Documents 1 and 2 disclose an apparatus that estimates the priority of a word input by a user from the input coordinates of the user on the touch panel and the positional relationship of buttons corresponding to each character on the software keyboard. . In the technique described in Patent Document 1, as shown in FIG. 5, the pressed candidate character is extracted from the positional relationship between the input coordinates and the button area corresponding to each character, and a score is given. Next, as shown in FIG. 6A and FIG. 6B, character strings are created for all combinations by taking out one character from each character group extracted with continuous coordinate input, and words are extracted from the character strings. Extract what can be. Finally, the priority of each word is calculated from the score of each character constituting the word and the score calculated from the frequency of use of the word, and presented in the order of priority.

  The technique described in Patent Literature 2 is based on the product of the probability calculated from the probability density function of touch coordinates for each character and the language probability based on the language model using “n-gram”. An occurrence probability value for each character according to whether the occurrence probability is obtained, and candidate words are presented in descending order of occurrence probability. By this method, even if there is an error in touch input, it is possible to display a user-desired word among candidate words.

  The technique described in Patent Document 3 acquires line-of-sight information from information obtained from an image sensor and a depth sensor, performs calibration, and operates a pointer on a screen displayed two-dimensionally.

JP 2006-005655 A JP 2012-248153 A JP 2009-054101 A

  However, the techniques described in Patent Document 1 and Patent Document 2 predict and display a plurality of user-desired word candidates using input coordinates and a language model. The same character string as the displayed character string is displayed, and the word converted from the character string (kanji-kana mixed text) and the word converted from the character string with a different character string displayed are mixed. May be displayed.

  Here, when the user is inputting characters while checking the terminal screen, there are few input errors, and words that are converted from the actually input character string are presented or forward matching is used. By complementing, it is considered that the possibility that the character string desired by the user is displayed is increased.

  Conversely, when the user is inputting characters by blind touch, there are many input errors, and by displaying a large number of conversion candidates in consideration of touch errors, the user's desired character string may be displayed. It is thought to be higher.

  The technique described in Patent Document 3 determines the position indicated by the line of sight based on the line-of-sight information and operates as a pointer. However, when inputting characters, it is necessary to look at the software keyboard displayed on the screen. is there. Further, if a touch panel is mounted on the display, touch input is possible, but information on whether or not the user is looking at the screen is not taken into consideration. Therefore, in patent document 3, it can be said that it is not assumed that a conversion candidate is changed depending on whether the user is performing a blind touch.

  The present invention has been made in view of such circumstances, and improves the operability during character input by changing the displayed character candidates based on the degree of attention to the screen while the user is operating the terminal. It is an object to provide an input support apparatus, an input support method, and a program capable of presenting a highly accurate word or word string.

  (1) In order to achieve the above object, the present invention takes the following measures. That is, the input support device according to the present invention is an input support device that predicts a character string that a user intends to input, and includes a display that displays at least a character, a keyboard that receives character input by the user, and a user for the device body A sensing device that detects the operation status of the user, an attention level estimation unit that estimates a user's level of attention to the display based on the detected user operation status, characters input from the keyboard, and the estimated A prediction unit that predicts a character string as a candidate based on the degree of attention of the user with respect to the display, and an output unit that outputs the character string as the predicted candidate are provided.

  As described above, the user's attention level with respect to the display is estimated based on the detected user operation state, and the character string to be a candidate is determined based on the characters input from the keyboard and the estimated user's attention level with respect to the display. Therefore, it is possible to improve the operability at the time of character input and to present a highly accurate word or word string.

  (2) In the input support device of the present invention, when the degree of attention is high, the prediction unit increases the probability of occurrence of a character string including a character input by the user while the degree of attention is low. Is characterized in that the occurrence probability of a character string including a character input by a user is lowered.

  In this way, when the degree of attention of the user's display is high, it is considered that there are few input errors, so the occurrence probability of the character string including the character input by the user is increased. On the other hand, when the degree of attention to the user's display is low, there is a possibility that blind touch input is being performed, and it is considered that there are many input errors, so the occurrence probability of a character string including characters input by the user is lowered. . As a result, it is possible to improve the operability at the time of character input and to present a highly accurate word or word string.

  (3) Further, in the input support device of the present invention, the attention level estimation unit estimates the attention level for each character, and the prediction unit determines that the character having a high attention level when the attention level for each character is high. The occurrence probability of a character with a low degree of attention is lowered when the attention degree for each character is low.

  In this way, when the attention level for each character of the user is high, it is considered that there are few input errors, so the occurrence probability of a character with high attention level is increased. On the other hand, when the attention level for each character of the user is low, there is a possibility that a blind touch input is being performed, and it is considered that there are many input errors, so the occurrence probability of a character with low attention level is reduced. As a result, it is possible to improve the operability at the time of character input and to present a highly accurate word or word string.

  (4) In the input support device of the present invention, the keyboard is a software keyboard displayed on a touch panel, and detects coordinate information of a user's finger that touches the touch panel, and the prediction unit detects the detection A candidate character string is predicted based on the coordinate information, a language model that formulates the relationship between words, and the degree of attention.

  Thus, based on the detected coordinate information, the language model that formulates the relationship between words and the attention level, the candidate character strings are predicted, so that the operability at the time of character input by the touch panel is improved, It becomes possible to present highly accurate words or word strings.

  (5) In the input support device according to the present invention, the attention level estimation unit may determine a time, frequency, a user operation feature amount, or a user operation amount based on the detected user operation state. The degree of attention is estimated based on at least one of user input coordinate information for the keyboard.

  As described above, based on the detected user operation state, the degree of attention is determined based on at least one of the time and frequency with which the user views the display, the feature amount of the user operation, or the user input coordinate information with respect to the keyboard. Since the estimation is performed, it becomes possible to predict a candidate character string depending on whether the user is inputting characters while looking at the display or is inputting characters by blind touch. As a result, it is possible to improve the operability when inputting characters using the touch panel and present a highly accurate word or word string.

  (6) Further, in the input support device according to the present invention, the output unit outputs the predicted character string as character information to the display or as voice information to a speaker.

  In this way, the predicted character string is output as character information to the display or output as voice information to the speaker, so that the user can recognize the predicted character string and determine whether to confirm it. In particular, when outputting as voice information, since so-called talkback can be performed, the user can recognize a character string predicted by hearing. This makes it possible to present a highly accurate word or word string.

  (7) Further, the input support method of the present invention is an input support method for predicting a character string that the user intends to input, and at least the step of displaying characters on the display and the input of characters by the user from the keyboard are accepted. A step of detecting an operation state of the user with respect to the apparatus main body by the sensing device, a step of estimating a degree of attention of the user with respect to the display based on the detected operation state of the user, and an input from the keyboard The method includes at least a step of predicting a character string to be a candidate based on a character and a degree of attention of the user with respect to the estimated display, and a step of outputting the character string to be the predicted candidate.

  As described above, the user's attention level with respect to the display is estimated based on the detected user operation state, and the character string to be a candidate is determined based on the characters input from the keyboard and the estimated user's attention level with respect to the display. Therefore, it is possible to improve the operability at the time of character input and to present a highly accurate word or word string.

  (8) The program of the present invention is a program for an input support apparatus that predicts a character string that the user is going to input, and accepts at least character display processing and character input by the user from the keyboard. Processing, processing for detecting a user's operation status with respect to the apparatus main body with a sensing device, processing for estimating a user's attention level with respect to the display based on the detected user's operation status, and input from the keyboard A computer executes a series of processes including a process for predicting a character string as a candidate and a process for outputting the character string as a predicted candidate based on the user's degree of attention to the character and the estimated display. It is characterized by that.

  As described above, the user's attention level with respect to the display is estimated based on the detected user operation state, and the character string to be a candidate is determined based on the characters input from the keyboard and the estimated user's attention level with respect to the display. Therefore, it is possible to improve the operability at the time of character input and to present a highly accurate word or word string.

  According to the present invention, the user's attention level to the display is estimated based on the detected user operation status, and the candidate is based on the characters input from the keyboard and the estimated user's attention level to the display. Since the character string is predicted, it is possible to improve the operability at the time of character input and to present a highly accurate word or word string.

It is a block diagram which shows schematic structure of the input assistance apparatus which concerns on this embodiment. It is a figure which shows the external appearance of information portable terminal A1. It is a flowchart which shows operation | movement of the input assistance apparatus which concerns on 1st Embodiment. It is a flowchart which shows operation | movement of the input assistance apparatus which concerns on 2nd Embodiment. It is a figure which shows the score assigned to the input coordinate and the button. It is a figure which shows the score example of a touch coordinate. It is a figure which shows the example of score calculation.

  An input support apparatus according to an embodiment of the present invention recognizes a user's operation status with a sensing device and improves the operability of text input using a touch panel, and pays attention to a user's display based on data acquired from the sensing device. Estimate the degree. Then, the character candidates / areas to be displayed are changed according to the degree of attention.

FIG. 1 is a block diagram illustrating a schematic configuration of the input support apparatus according to the present embodiment. In FIG. 1, an input unit 1 is an interface for a user to input data. For example, it comprises a touch panel display and a keyboard displayed on the touch panel display. The two-dimensional information obtained from the touch panel display is grasped as coordinates X _t = (x _t , y _t ). In the case of three-dimensional information, it is grasped as coordinates X _t = (x _t , y _t , z _t ). The input character display unit 3 displays characters that are being input by the user and characters that have been determined by user operations. The dictionary storage unit 5 stores a combination of a word and a character string constituting the word.

  The character string prediction unit 7 calculates the score value of the word registered in the word dictionary of the dictionary storage unit 5 based on the coordinate information obtained from the input unit 1 and the language model in which the relationship between words is formulated. Calculate and predict the top N character strings. Typical language models include an N-gram model, a hidden Markov model, and a maximum entropy model.

  The kana-kanji conversion unit 9 predictively converts and complements candidate words (words / phrases / sentences) from a character string that is input and displayed on the display by a forward match, partial match, or backward match. This candidate word may be a word, a clause, or a sentence. Based on the character string acquired from the kana-kanji conversion unit 9, the character string complementing unit 11 performs a process of complementing the character string by forward matching, partial matching, backward matching, or the like. The candidate kana / kanji display unit 13 displays the word output from the character string complementing unit 11.

  The sensing device 15 includes an image sensor and a depth sensor, and performs line-of-sight recognition. The display attention level estimation unit 17 estimates the user's display attention level based on the data input from the sensing device 15. The candidate kana / kanji display changing unit 19 changes the ratio of the candidate character strings according to the display attention level input from the display attention level estimation unit 17. The voice output unit 21 outputs candidate character strings as voice. The main control unit 22 controls the operation of each component.

(First embodiment)
In the present embodiment, an information portable terminal equipped with a touch panel will be described as an example of the information portable terminal. In the present embodiment, as an example of obtaining the display attention degree O, line-of-sight recognition including an image sensor and a depth sensor is employed. The line-of-sight recognition in the present embodiment can be acquired as a pointer indicating X _t = (x _t , y _t ) on the display.

The processing by the display attention level estimation unit 17 is as follows. That is, the display attention degree O means an index of how much attention a user has on the character input area of the display when the user inputs characters. Enter a certain character W _t _ _1, and the time to enter the n-th character and T _n, display attention L is, in T _n, and the character input area of the display, the other region ( The number of times (including outside the display) num, the time when the display was viewed, and the pointer _Xt time series function indicated by the line of sight can be expressed as follows. Not only these values, the average of X _t, dispersion, use other calculation values can be.
O = α (time / T _n ) * β (1-exp (-a · num)) * γ (exp (-b · RMS (σx, σy))) (0 ≦ O ≦ 1)
However, a, b, α, β, γ are coefficients (0 <a, b, 0 <α, β, γ <1), and RMS represents a mean square.

  The display attention degree O is transmitted to the candidate word display changing unit. Although the display attention level is calculated for a plurality of character inputs here, the display attention level for each character may be calculated.

  Processing by the character string prediction unit 7 is as follows. That is, the character string predicting unit 7 predicts the input actual character string and other similar character strings. As this method, as described in Patent Document 1, a score is assigned to a character close to the touch position, and the scores are compared. As described in Patent Document 2, probability is used. It is possible to take a technique. In this embodiment, a method using a touch model and an n-gram character chain probability is adopted.

In the touch model, first, a probability distribution function Pw _i (x, y) with a two-dimensional coordinate as a variable is defined in advance for each character. Pw _i (x, y) is a function indicating the probability of occurrence of the character w _i with respect to the coordinates (x, y). An example of this function is a mixed Gaussian distribution.

The n-gram character chain probability is the probability of displaying the character w _i after the characters w _i-1 ,…, w ₁ are input, and P (w _i | w _i-1 ,…, w ₁ ) = ΠP (w _i | w _i−1 ,…, w ₁ )).

P (w _i | (x, y)) = P w _i (x, y) P (w _i | w _i-1 ,…, w ₁ ) is the character occurrence probability for that character. A character string occurrence probability is obtained by accumulating the character occurrence probabilities. Actually, if all the character strings are calculated, the calculation amount and memory used become enormous.

  FIG. 2 is a diagram showing an overview of the portable information terminal A1. The information portable terminal A1 includes an input character display area A2, a candidate word display area A3, and a toggle keyboard A4 on the touch panel. For example, when “Tomorrow” (“Ah”, “Sa”, “Sa”, “Ta”) is entered with a toggle keyboard as shown in FIG. “Akita”, “Kakashi”, etc. are calculated as candidates, and a probability is given to each. These pieces of information are transmitted to the kana-kanji conversion unit 9.

  The process of the kana-kanji conversion unit 9 is as follows. That is, kana-kanji conversion includes heuristic kana-kanji conversion and statistical kana-kanji conversion. In this embodiment, the “statistical least cost method”, which is one of the latter, is adopted. Thus, the actually input character string and a plurality of character strings (kana strings) acquired from the character string predicting unit are converted to kana-kanji characters, and the candidates are transmitted to the character string complementing unit 11.

  The processing of the character string complementing unit 11 is as follows. That is, the character string complementing unit 11 performs processing for complementing the character string by forward matching, partial matching, backward matching, and the like based on the character string acquired from the kana-kanji conversion unit 9. For example, if the input is “Tomorrow”, the output is “Tomorrow is”.

  Next, the process of the candidate kana / kanji display changing unit 19 based on the display attention level will be described. Here, for example, as described above, when the character string related to the actually input character string is “Ashita”, kana-kanji conversion / conversion is performed for other character strings predicted by the character string prediction unit 7. Processing to change the ratio, area size, color scheme, etc. for displaying the result of character string completion is performed. Examples of parameters processed by the candidate kana / kanji display changing unit 19 include the number (region) / position of word display predicted from each character string predicting unit 7.

  When the display attention degree O is high, the ratio and display area of the kana-kanji conversion result (this is referred to as “result 1”) regarding the actually input character string is increased. On the other hand, when the display attention degree O is low, the ratio of the kana-kanji conversion result (referred to as “result 2”) related to the character string (kana string) acquired from the character string prediction unit is decreased. The function of the ratio of the number of candidate displays can be, for example, (the ratio r1 of the number of candidate displays of result 1) = 1−μO. This is displayed on the candidate kana / kanji display unit 13.

FIG. 3 is a flowchart showing the operation of the input support apparatus according to the first embodiment. First, it is determined whether there is an input while waiting for an input from the user (step S1). If there is no input, the determination in step S1 is repeated. If there is an input, the input coordinates (x1, y1) are detected (step S2). Next, initialization is performed to set “k = 1, Pw _i ⁰ = 1” (step S3). Next, the character occurrence probability is calculated for all the characters w _i (step S4), and the cumulative probability “Pw _i ^k = Pw _i ^k−1 × Pw _i ((x _k , y _k ))” is calculated (step S5). Next, n word candidates are extracted (step S6), and kana-kanji conversion and character string interpolation are performed (step S7).

  On the other hand, if there is an input from the user in step S1, information is acquired from the sensing device (step S10), and the display attention level is estimated (step S11). Based on the display attention degree, the character string candidate is changed (step S8). Next, the changed character string candidate is displayed as a word candidate (step S9).

Next, it is determined whether or not there is an input (step S12). If there is no input, the determination in step S12 is repeated. If there is an input, the input coordinates (x _{k + 1} , y _{k + 1} ) are detected (step S13). Next, it is determined whether (x _{k + 1} , y _{k + 1} ) is a word candidate display area or a keyboard area (step S14). Step S16) and the process proceeds to Step S4. On the other hand, when it is a word display area in step S14, a character string is displayed on the input character display part 3 (step S15), and it transfers to step S1.

(Second Embodiment)
As described above, in the first embodiment, the display attention level is reflected only on the change of the character string candidate. However, in the second embodiment, the display attention level O _i is calculated for each character. Using the calculated display attention degree O _i , the probability for each character calculated by the character string prediction unit 7 is weighted as shown in the following equation.

P (w _i | (x, y)) = ξO _i * P w _i (x, y) P (w _i | w _i-1 ,…, w ₁ )
Thereby, the occurrence probability of the character when the display attention degree is high is increased.

FIG. 4 is a flowchart showing the operation of the input support apparatus according to the second embodiment. In step T5, weighting is performed using the display attention degree o _i (step T11). Other steps are the same as those in the first embodiment.

(Third embodiment)
In the second embodiment, the display attention degree o _i is calculated for each character and used for character weighting. In the third embodiment, O _i and P w _i (x, y) P ( If w _i | w _i−1 ,..., w ₁ ) exceeds a certain threshold, the character w _i input at that time is determined. Thereby, it is possible to reduce the calculation amount and the memory usage.

  In the first to third embodiments, the character color and highlight can be changed as the display method of the input character display area / candidate character area based on the display attention level.

(Fourth embodiment)
In the fourth embodiment, talkback is performed by speech synthesis. In the first to third embodiments, the candidate kana / kanji display changing unit 19 changes the candidate kana / kanji display and then outputs the candidate kana / kanji display unit 13. On the other hand, in the fourth embodiment, the speech output unit 21 performs talkback in descending order of probability from the probability of the character string calculated by the character string prediction unit 7. The content of the talkback is either reading in the order of high probability (kana), kana-kanji conversion result, or complement result.

(About sensing devices)
In the first to fourth embodiments, the line-of-sight recognition is used as the sensing device, but the display attention degree may be calculated using a sensor such as a proximity sensor, acceleration, gyroscope, and angular velocity. As the calculation of the display attention level, there is a method of collecting data of a situation that is being viewed and a situation that is not, performing machine learning / pattern recognition using the data, and using the reliability / score value. The pattern recognition may be a general method such as SVM, Bayes, or HMM.

  As described above, according to the present embodiment, the displayed character candidates are changed based on the degree of attention to the screen during the user's terminal operation, so that highly accurate words or word strings can be presented. It becomes possible.

DESCRIPTION OF SYMBOLS 1 Input part 3 Input character display part 5 Dictionary storage part 7 Character string prediction part 9 Kana kanji conversion part 11 Character string complement part 13 Candidate Kana kanji display part 15 Sensing device 17 Display attention estimation part 19 Candidate kana kanji display change part 21 Voice output unit 23 Main control unit A1 Information portable terminal A2 Input character display area A3 Candidate word display area A4 Toggle keyboard

Claims (7)

An input support device that predicts a character string that a user is about to input,
A touch panel display that displays at least characters,
A software keyboard that is displayed on the touch panel display and accepts input of characters by a user;
A sensing device for detecting a user's operation status on the apparatus main body;
A degree-of-interest estimation unit that estimates the degree of attention of the user with respect to the software keyboard based on the detected user operation state;
A prediction unit based on the user's attention, predicts a character string as a candidate for the software keyboard that is character and the estimated input from the software keyboard,
And an output unit that outputs a character string as the predicted candidate.   When the degree of attention is high, the prediction unit increases the occurrence probability of the character string including the character input by the user. On the other hand, when the degree of attention is low, the prediction unit increases the probability of the character string including the character input by the user. 2. The input support apparatus according to claim 1, wherein the occurrence probability is lowered. The attention level estimation unit estimates the attention level for each character,
The predicting unit increases the probability of occurrence of a character with a high degree of attention when the degree of attention for each character is high, while decreasing the probability of occurrence of a character with a low degree of attention when the degree of attention for each character is low. The input support apparatus according to claim 1, wherein: The software keyboard detects coordinate information of a user's finger that has touched the touch panel display ,
The prediction unit predicts a candidate character string based on the detected coordinate information, a language model that formulates a relationship between words, and the degree of attention. 4. The input support device according to any one of 3. The attention degree estimation unit based on the operation status of the detected user, time for the user to visually the software keyboard, the frequency, at least one user input coordinate information for the feature amount or the software keyboard operation of the user The input support apparatus according to any one of claims 1 to 4, wherein the attention degree is estimated based on one of the two. An input support method for predicting a character string that a user is about to input,
Displaying at least characters on the touch panel display; and
Receiving a character input by a user from a software keyboard displayed on the touch panel display ;
Detecting a user's operation status on the apparatus main body with a sensing device;
Estimating a user's attention to the software keyboard based on the detected user operation status;
A step of based on the user's attention, predicts a character string as a candidate for the software keyboard that is character and the estimated input from the software keyboard,
Outputting at least the character string as the predicted candidate. A program of an input support device that predicts a character string that a user is going to input,
Processing to display at least characters on the touch panel display;
A process of accepting a character input by a user from a software keyboard displayed on the touch panel display ;
A process for detecting a user's operation status with respect to the apparatus main body with a sensing device;
A process of estimating the degree of attention of the user with respect to the software keyboard based on the detected operation status of the user;
A process of the software based on the user's attention for the character and the estimated software keyboard input from a keyboard, predicts a character string as a candidate,
A program for causing a computer to execute a series of processes of outputting a character string as the predicted candidate.

Images