WO2025189038A1 - Method of proficient typing using a limited number of classes - Google Patents

Abstract

Systems and methods for proficient typing using a limited number of classes are disclosed. In one embodiment, capturing keystrokes and returning predictive text on a display includes creating a dictionary of words and representative keypresses associated with each, capturing a series of keypresses of keys on a keyboard, where at least one key represents assignments of more than two characters, for each captured keypress, generating probabilities for the likelihood each of a plurality of words from the dictionary to be next by providing a language model with all previous text in the series up to but not including the last token, storing the generated probabilities with the word in the dictionary, applying a filter to the dictionary to obtain a subset of words, and displaying the subset of words on a display, obtaining input that selects a word from the subset, and indicating the selected word as committed.

Description

METHOD OF PROFICIENT TYPING USING A LIMITED NUMBER OF CLASSES

STATEMENT OF FEDERALLY SPONSORED RESEARCH

[0001] This invention was made with government support under NS 122037 awarded by the National Institutes of Health. The government has certain rights in the invention.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The current application claims priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application Serial No. 63/562,051, entitled “Method of Proficient Typing Using a Limited Number of Classes”, filed March 6, 2024 to Kao et al. The disclosure of U.S. Provisional Patent Application Serial No. 63/562,051 is hereby incorporated herein by reference in its entirety.

FIELD OF THE INVENTION

[0003] The present invention relates to computing interfaces and more specifically to proficient typing using limited keys and predictive techniques.

BACKGROUND OF THE INVENTION

[0004] Communication is an integral part of daily life and is becoming an increasing necessity in today’s society. We spend a significant amount of time on our mobile phones, laptops, and companies are continuing to push the boundaries on simplifying typing to improve user experience. In the realm of typing, efforts have been made to use smaller keyboards with less than 26 keys, for example, T9 typing. However, such typing schemes are outdated due to impracticality and difficulty of use. With predictive text and various attempts at disambiguation methods, the user is forced to scroll/toggle through a list of predictions after each word. These disambiguation methods also struggle with time and memory constraints, making them hard to use in a real-time application. Such inconveniences have caused these methodologies to become largely obsolete. Furthermore, layouts for keyboards smaller than 9 keys are confusing for users, resulting in another barrier in user experience. For such condensed keyboards to become viable solutions, the accuracy of the most likely word must be near perfect to minimize the number of times the user has to scroll/toggle to select their desired word. SUMMARY OF THE INVENTION

[0005] Systems and methods for proficient typing using a limited number of classes are disclosed. In one embodiment, a method for capturing keystrokes and returning predictive text on a display includes creating at least one dictionary of words, tokens, and word sequences and representative keypresses associated with each, capturing a series of keypresses of keys on a representation of a keyboard, where the keyboard comprises a plurality of keys and at least one key represents assignments of more than two letters, for each captured keypress, generating probabilities for the likelihood each of a plurality of words and word sequences from the at least one dictionary to be next by providing a language model with all previous text in the series up to but not including the last token in the series, storing the generated probabilities with the corresponding word or word sequence in the at least one dictionary, applying at least one filter to the at least one dictionary to obtain a subset of words and word sequences, and displaying the filtered subset of words and word sequences on a display, obtaining input that selects a word or word sequence from the displayed subset, and indicating the selected word or word sequence as committed on the display.

[0006] Another embodiment includes resetting the probabilities in the at least one dictionary.

[0007] Yet another embodiment includes completing the series of keypresses and starting a new series of keypresses and generated probabilities.

[0008] Still another embodiment includes displaying each keypress on the display as it is captured. [0009] In another embodiment, the language model is a recurrent neural network.

[0010] In yet another embodiment, the language model is a Receptance Weighted Key Value (RWKV) algorithm.

[0011] In still another embodiment, the language model is trained in English.

[0012] In yet another embodiment again, the representation of a keyboard is a computer keyboard. [0013] In another embodiment, the representation of a keyboard is a virtual keyboard on a surface. [0014] In yet another embodiment, the representation of a keyboard is a touch screen graphical interface.

[0015] In still another embodiment, the at least one dictionary includes a dictionary of words and a dictionary of word sequences.

[0016] In yet another embodiment again, the at least one dictionary includes a dictionary of both words and word sequences.

[0017] In yet another embodiment again, the probability of a word sequence is calculated as

[0019] where (t_l, ..., t n) is the sequence of tokens that comprise the sequence of words, and t_0 is the start-of-sequence token used to condition the language model.

[0020] In another embodiment, the at least one filter includes Boolean logic.

[0021] In yet another embodiment, the at least one filter includes words and word sequences that differ by one character from captured series.

[0022] In still another embodiment, the at least one filter includes words and word sequences that differ in keypresses of one keypress proximity from the captured keypress.

[0023] In yet another embodiment again, the at least one filter includes words and word sequences that differ in keypresses of up to four keypress proximity from the captured keypress.

[0024] In another embodiment, the at least one filter ranks the subset of words and word sequences by highest probability.

[0025] In yet another embodiment, the keyboard includes a key configured for obtaining input that selects a word or word sequence.

[0026] In still another embodiment, the at least one dictionary includes punctuation characters.

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Fig. 1 illustrates a system for predictive typing in accordance with embodiments of the invention.

[0028] Fig. 2 illustrates an arrangement of alphabet to key assignments in accordance with an embodiment of the invention.

[0029] Fig. 3 illustrates an arrangement of alphabet to key assignments in accordance with an embodiment of the invention.

[0030] Fig. 4 illustrates entries in a dictionary in accordance with an embodiment of the invention. [0031] Fig. 5 illustrates a process for typing using a limited number of classes in accordance with embodiments of the invention.

DETAILED DISCLOSURE OF THE INVENTION

[0032] Systems and methods for typing using a limited number of classes in accordance with embodiments of the invention incorporate prior word context into predictions of the next word. Many embodiments of the invention utilize recurrent neural networks, which allow for predictions to incorporate semantic information from prior words into the prediction of the next word. The neural network assigns a probability to each word in a dictionary as the likelihood of that word occurring as the next word in the sentence, and a subset of these outputs are organized by this probability score in an interface for the user. In certain embodiments, after each character is input by the user, the outputs are displayed. This technique is shown to work in real-time up to typing speeds of 120 wpm, and it is shown to have near-perfect top-1 accuracy in the 9 finger example. This approach also shows that this near-perfect accuracy extends to even more condensed keyboards with as few as 5 keys. This approach also allows for commas, periods, and capitalization of words, and can contain a large set of words in the English language that encapsulates all words that would be used in a natural typing context.

[0033] Systems and methods for typing in accordance with embodiments of the invention can be used in a wide set of general tasks, including every-day typing. A specific example of a use case to highlight would be in the field of brain computer interfaces. In this field, a large goal is to get a machine to be able to decode the intentions of paralyzed or amputated patients. However, achieving good control over many different actions proves to be a difficult task. The approach discussed here may be used to allow paralyzed or amputated patients to type with the same efficiency as natural typing on a 26 key keyboard while only having to execute relatively few actions. Similarly, for any person who finds difficulty in typing with 26 keys or has less proficient control of their fingers, this approach would allow them to reduce the number of keys while still achieving the same efficiency as natural typing.

[0034] Systems for typing in accordance with embodiments of the invention include a set of physical or virtual keys (where at least one key includes multiple letter/character assignments), a user interface (that can display typed words and suggestions), a dictionary of word probabilities, and a language model trained to provide probabilities of word occurrence. In some embodiments, the dictionary of word probabilities also includes word sequence probabilities. In some embodiments, a dictionary of word sequence probabilities is in addition to the dictionary of word probabilities. Each component will be discussed in greater detail further below.

[0035] A computing system that may be utilized in accordance with embodiments of the invention is illustrated in Fig. 1. The computing system 10 includes a user computer 18, a portable device 20, and a server 14, which may communicate over network 16. The user computer 18 or portable device 20 can include a reduced keyset as described here and a user interface. They may also include a dictionary of word probabilities and language model, or may communicate with server 14 when server 14 includes the dictionary of word probabilities and/or the language model.

[0036] In some embodiments of the invention, letters are mapped to keys or fingers as follows (and as illustrated in Fig. 2):

[0037] (1) Left pinky: q, a, z

[0038] (2) Left ring: w, s, x

[0039] (3) Left middle: e, d, c

[0040] (4) Left index: r, t, f, g, v, b

[0041] (5) Right index: y, u, h, j, n, m

[0042] (6) Right middle: i, k,

[0043] (7) Right ring: o, 1, p,

[0044] (8) Right thumb: space

[0045] (9) Right pinky: toggle/scroll

[0046] Left thumb (optional): backspace

[0047] This arrangement follows the format of a QWERTY keyboard and is intended for familiarity and ease of use for a user. The mapping of letters to fingers can be flexible. If the user instead preferred an alphabetic ordering of the keys, near-perfect performance can still be achieved. This arrangement is also flexible to the choice of which finger maps to which keys, the example above is chosen for convenience in illustration.

[0048] In additional embodiments of the invention, letters are mapped to keys or fingers as follows (and as illustrated in Fig. 3):

[0049] (1) Left pinky: a, b, c, d, e, f, g

[0050] (2) Left ring: h, i, j, k, 1, m, n, o

[0051] (3) Left middle: p, q, r, s, t,

[0052] (4) Left index: u, v, w, x, y, z,

[0053] (5) Right thumb: space

[0054] (6) Right index: toggle

[0055] Left thumb (optional): backspace

[0056] This arrangement is the format of alphabetically ordering the keys. The mapping of letters to fingers is flexible. If instead the user preferred a different ordering of the keys, similar performance can still be achieved. This arrangement is also flexible to the choice of which finger maps to which keys, the example above is chosen for convenience in illustration. An optimal arrangement can be utilized to further improve performance.

[0057] The above examples illustrate possible key configurations for condensed keyboards of variable size that are less than the typical number of ten fingers and thumbs of a user. Further embodiments can extend to keyboards of all sizes. Similarly, as stated above, for condensed keyboards of any size, similar performance can be achieved for most evenly-distributed (approximately the same number of keys per finger/key) configurations of the letters.

[0058] In additional embodiments of the invention, the arrangement of keys is optimized to reduce the frequency of incorrect suggestions. The optimization is performed by maximizing (minimizing) a score function. In some embodiments, the optimization procedure is performed with iterative sampling, and sampling probabilities may or may not change at each iteration. One example of a score function to be minimized is the frequency of words having the same corresponding key sequence. In some embodiments, the score function incorporates the representations of words within a language model.

[0059] As mentioned above, a keyboard can be any representation of a keyboard that a user can perceive and interact with, physical or virtual. For example, a keyboard can be a traditional computer-style keyboard, a keyboard displayed on a touch screen graphical interface, a virtual keyboard created by projecting light onto a surface and using sensors, a virtual keyboard displayed in a heads-up display (HUD) or virtual reality (VR) or augmented reality (AR) display, and so on. One skilled in the art will recognize that the concept of a representation of a keyboard can be extended to many types of keyboards. Many keyboards may be suitable in embodiments of the invention when at least one key may have multiple characters/letters mapped to it.

Processes for Typing Using a Reduced Keyset

[0060] Processes in accordance with embodiments of the invention can include receiving keypresses via a variety of mechanisms (e.g., on a user interface or physical keyboard), generating word suggestions, and presenting word suggestions on the user interface. In many embodiments of the invention, the set of keys are less than a typical keyboard, for example, where keys can represent multiple letters. In further embodiments, there are less than ten keys. Arrangements of keysets in various embodiments can include those discussed further above. A process for typing using a reduced keyset in accordance with embodiments of the invention is illustrated in Fig. 5. [0061] The process 500 can include creating (512) a dictionary of words and tokens (word portions) for a given set or arrangement of keys (e.g., one of the key arrangements discussed above). Each word or token in the dictionary is associated with a sequence of keys that “spell” that word/token and a reserved space for a probability for the word/token. In some embodiments of the invention, the process can create a sequence dictionary containing all possible sequences of words in the word dictionary and their associated probabilities. Alternatively, the sequences of words and probabilities can be incorporated into the word dictionary. In various embodiments, the dictionary can take the form of a database or table. A portion of a dictionary as an example is provided in Fig. 4.

[0062] The process includes receiving (514) a keystroke of one of the keys in the set of keys. The dictionary is updated (516) upon receiving the keypress by running a language model to determine probabilities that each word is the next word, or each word sequence is next, in the sentence being typed. In several embodiments of the invention, the language model is provided as input all the text (words) that is considered up to, but not including, the last token in the sequence. In some embodiments, word sequences are no longer considered if their probability is lower than a threshold, e.g., the kth highest sequence probability in the sequence dictionary.

[0063] In several embodiments of the invention, an Receptance Weighted Key Value (RWKV) algorithm utilized in inference mode, passing in the previous hidden state as the prior context. This allows for immense memory and time optimizations, and shows the massive benefits of using recurrency. For each word sequence, one forward pass of RWKV is run for each token in the word sequence, assigning probabilities to each token used in the tokenizer for the large language model. Conditional probabilities for each token given all preceding tokens in previously typed text and in the sequence are multiplied to yield a single probability value for each sequence. As characters are inputted, the current sequence of fingers in the word is compared against all sequences of tokens, and the tokens that match the sequence of fingers are selected. The algorithm can be any language model that predicts next tokens, and can be specifically trained for each user to maximize typing rate.

[0064] Similarly, for the specific task the user would like to perform, the language model can be fine-tuned to their preferences. If the user would like to communicate in a language other than English, whether it be Spanish or a coding language, the corresponding fine-tuned language model can be chosen. [0065] In some embodiments of the invention, the total language model probability of a sequence can be calculated as:

[0066] where (t_l, ..., t n) is the sequence of tokens that comprise the sequence of words, and t_0 is the start-of-sequence token used to condition the model and which cannot be typed. The probability function can be calculated by the RWKV model.

[0067] From this individual tokens p(t_i|t_O, ..., t {i-1 }) can be presented, or words or sequences scored by the above probability function. After selection of m-1 tokens, then consider

[0068] Although log(p(t_l,...,t_n) can also be used as equivalent.

[0069] The words and sequences of words in the dictionaries are fdtered (518) for presentation to the user. One or more fdters, in combination or alternatives (using any various boolean logic), can be applied to identify a subset of the words that match the keys that have been typed in the current word. Filters can include, but are not limited to, an exact match of the keys to the associated keypresses of a word (including number of keypresses), autocorrection or autosuggestion to include inexact matches that have keypresses to some proximity of the associated keypresses, and/or extended matches that match but have a higher number of associated keypresses than the keys that have been currently received.

[0070] Before or after the above filtering, the words and sequences of words in the dictionaries can also be filtered to limit the number of words to present to the user to a predetermined number (e g., four or five) for practicality. The words can be limited by top ranking of the highest probabilities.

[0071] The filtered set of words and/or word sequences is presented to the user on the user interface for selection. The user can select (520) one of the words or word sequences to be entered as the intended next word of the sentence. The process can repeat when additional keystrokes are received to add characters to the current word sequence. After input is received to select a word or word sequence, that word or word sequence is indicated on the display as committed, that is selected by the user. After commitment of the word or word sequence, a new word or word sequence may begin with repeating the process above. I several embodiments of the invention, the probabilities may be maintained over an entire series of words cumulatively with subsequent words, including the previously committed words and word sequences, while starting a new set of probabilities for the text after the part of the series that was committed.

[0072] In some embodiments of the invention, the dictionaries are reset after commitment of the word or sequence. In alternative embodiments of the invention, the dictionaries can be reset when starting typing of a new word in the sentence. Further embodiments of the invention can include variations of typing style, e.g., typing multiple words without typing spaces and typing only parts of multiple words before selection.

[0073] While a specific process is described above with respect to Fig. 5, one skilled in the art will recognize that any of a variety of processes may be utilized in accordance with embodiments of the invention.

[0074] While the above description contains many specific embodiments of the invention, these should not be construed as limitations on the scope of the invention, but rather as an example of one embodiment thereof. It is therefore to be understood that the present invention may be practiced otherwise than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive.

Claims

WHAT IS CLAIMED IS:

1. A method for capturing keystrokes and returning predictive text on a display, the method comprising: creating at least one dictionary of words, tokens, and word sequences and representative keypresses associated with each; capturing a series of keypresses of keys on a representation of a keyboard, where the representation of a keyboard comprises a plurality of keys and at least one key represents assignments of more than two letters; for each captured keypress: generating probabilities for the likelihood each of a plurality of words and word sequences from the at least one dictionary to be next by providing a language model with all previous text in the series up to but not including the last token in the series; storing the generated probabilities with the corresponding word or word sequence in the at least one dictionary; applying at least one filter to the at least one dictionary to obtain a subset of words and word sequences; and displaying the filtered subset of words and word sequences on a display; obtaining input that selects a word or word sequence from the displayed subset; and indicating the selected word or word sequence as committed on the display.

2 The method of claim 1, further comprising resetting the probabilities in the at least one dictionary.

3. The method of claim 1, further comprising completing the series of keypresses and starting a new series of keypresses and generated probabilities.

4. The method of claim 1, further comprising displaying each keypress on the display as it is captured.

5. The method of claim 1, where the language model is a recurrent neural network.

6. The method of claim 1, where the language model is a Receptance Weighted Key Value (RWKV) algorithm.

7. The method of claim 1, where the language model is trained in English.

8. The method of claim 1, where the representation of a keyboard is a computer keyboard.

9. The method of claim 1, where the representation of a keyboard is a virtual keyboard on a surface.

10. The method of claim 1, where the representation of a keyboard is a touch screen graphical interface.

11. The method of claim 1, where the at least one dictionary includes a dictionary of words and a dictionary of word sequences.

12. The method of claim 1, where the at least one dictionary includes a dictionary of both words and word sequences.

13. The method of claim 1, wherein the probability of a word sequence is calculated as where (t_l, ..., t n) is the sequence of tokens that comprise the sequence of words, and t_0 is the start-of-sequence token used to condition the language model.

14. The method of claim 1, where the at least one filter includes Boolean logic.

15. The method of claim 1, where the at least one filter includes words and word sequences that differ by one character from captured series.

16. The method of claim 1, where the at least one filter includes words and word sequences that differ in keypresses of one keypress proximity from the captured keypress.

17. The method of claim 1, where the at least one filter includes words and word sequences that differ in keypresses of up to four keypress proximity from the captured keypress.

18. The method of claim 1, where the at least one filter ranks the subset of words and word sequences by highest probability.

19. The method of claim 1, where the keyboard includes a key configured for obtaining input that selects a word or word sequence.

20. The method of claim 1, where the at least one dictionary includes punctuation characters.