JP2009527041A - System and method for entering data into a computing system - Google Patents

Abstract

Systems and methods are provided that allow data entry into a computing system. The system is functionally connected to the image capture device and is configured to set a map of input keys or the entire keyboard layout based on the acquired image (s) collected by the image capture device. Includes controller. The system can collect images of user actions and / or gestures within a selected field of view, and can process these images to identify and execute commands according to the actions.
[Selection figure] None

Description

The present disclosure relates generally to the field of data input devices. More particularly, the present disclosure relates to a method for facilitating multilingual data entry and to a system that utilizes the multilingual data entry method.

A data input device with a key, collectively referred to herein as a “keyboard”, for inputting information into an electronic system such as a personal computer (PC), a mobile phone, a palm computer, an aircraft computer, etc. For example, using an alphanumeric keyboard, touchpad or touch screen, it has not changed much since the PC was invented.

The keyboard is for the input of data such as text and other types of characters (collectively referred to below as "symbols" or "key labels"), and also for controlling the operation of the computer As designed, almost all computers are equipped with a keyboard as the main form of interaction between the user and the computer. Physically, a computer keyboard is an array of rectangular or nearly rectangular buttons or “keys”. Keyboards typically have one or more symbols carved, printed or otherwise marked on each key, and in most cases each key press is entered into a computer and often Corresponds to a single symbol displayed on a computer display screen. However, to generate several symbols, it is necessary to press and hold several keys simultaneously or sequentially. Other keys can generate actions when pressed, and other actions are available by pressing multiple action keys simultaneously.

There are many different keyboard layouts (key placement and symbol assignment on the keys). Because different people may need to use different sets of symbols, a need for different keyboard layouts arises, typically because they read and write in different languages. Depending on the application, the number of keys on the keyboard typically varies from a standard 101 to 104 Windows keyboard to 130 keys with several programmable keys. There are also small variants with fewer than 90 keys, which are usually found on laptops or on space-constrained desktop computers. On most English computers and typewriter keyboards, the key layout of the most common keyboard layouts today is called the QWERTY design after the first six characters found in the top row of character keyboards.

Most information or data is typically typed or keyed using a keyboard with fixed key functionality. Particularly with respect to PCs, dual language keyboards are usually second language alphabet symbols that are marked on the keyboard keys along with the English alphabet, or by switching between the dual language functions of each key typically achieved by software instructions. , With a replaceable layout that allows the language to be changed. With a standard keyboard, the user can usually see up to three symbols on each depressed key that define different languages and options provided by different software and system operating methods. A computer mouse can also work independently to select options such as menus and for graphic command entry.

Standard keyboards suffer from several disadvantages and limitations. For example, a standard keyboard typically includes function keys, for example with symbols F1 to F12, which hide functions defined by separate instructions. Often, the user must learn and store these hidden functions, or recall their meaning from a lookup table, a “help” directory, or from other sources. However, this type of keyboard is limited by the number of keys and hence the key function. Typically, a non-professional typist frequently moves his line of sight between a keyboard placed on a desk and a PC monitor screen, usually placed higher in front of the desk, to follow typing actions. There must be. Especially for dual language keyboards, frequent eye movements and frequent non-feedback use of the “Alt + Shift” and “Caps Lock” functions lead to typing errors. With the advent of widespread Internet usage, PC users need to remember increasingly complex input commands to use a conventional keyboard, or more “hidden” functions of each key.

U.S. Patent No. 5,637,086, by the same inventor as the present disclosure, describes a method and system for creating an input device with a virtual input environment and a variable key display. However, Patent Document 1 does not teach the use of hand gestures to control the virtual keyboard layout, but also teaches that the appearance of a virtual hand can depend on the virtual keyboard layout used. Absent. In addition, U.S. Pat. No. 6,057,077 does not evaluate previous commands to predict current or future commands or user requests.

For some languages, such as those mentioned above, eg Russian and Hebrew, hardware keyboards typically have a second language alphabet symbol etched into the keyboard keys along with the English alphabet. In some countries where the spoken language has a large alphabet and / or many characters (eg, more than 50 characters), the keyboard is generally in English because not all the characters of the other language can be displayed on the physical keyboard Only the characters are included. This situation often creates enormous problems for users of this type of language to perform data entry tasks in this type of language.

An example of a country where this problem is exacerbated is India, which is a multilingual country with 22 characters prescribed by the Constitution and 10 different characters. Eighteen Indian languages, as defined by the Constitution, are mentioned with their letters in parentheses as follows: Hindi (devanagari), Konkani (devanagari), Marathi (devanagari), Nepali (devanagari), Sanskrit (devanagari), Sind (devanagari / urdu), Kashmiri (devanagari / urdu), assam (Assam), Manipur (Manipur), Bengali (Bengali) Characters), Oriya (Ouriya), Gujarati (Gujarati), Punjabi (Gurmuky), Telugu (Telugu), Kannada (Kannada), Tamil (Tamil), Malayaram (Malaylam) ) And Urdu (Urdu characters). Indian characters typically have 12-15 vowels, 35-40 consonants and a few diacritics. Besides this, for each vowel there is a corresponding modifier symbol, and for each consonant there is a corresponding pure consonant form (called half-letter). This makes the entire set of symbols required to input this type of language more than what a normal keyboard can accommodate. In India, for example, in order to attempt to provide a solution to the lack of a keyboard, different Indian language word processors are each distributed with a hard copy “map” that points to the Indian characters hidden behind the English keys. Nearly 50 hardcopy maps are available among different vendors of Indian word processors, but hardware manufacturers generally do not supply keyboards with an Indian language layout. More than 95 percent of Indians are generally deprived of English-based information technology.
US Patent 6,611,253

The following embodiments and aspects thereof are described in conjunction with the figures in connection with systems, tools and methods, which are meant to be exemplary and illustrative and do not limit the scope. In various embodiments, one or more of the above problems have been reduced or eliminated, while other embodiments are directed to other advantages or improvements.

In accordance with some embodiments of the present disclosure, systems and methods for entering data into a computing system are provided. The system disclosed herein sets and displays a map of the input keys (multiple keys) or the entire (or partial) layout of the keyboard based on the signals associated with the acquired images. And a controller that is operatively connected to the controller and configured to provide signals to or associated with or associated with the acquired image. . The system can collect and identify, recognize, or interpret one or more gestures, for example, gestures generated or performed by a user's hand, finger or other body part, And a command can be executed according to this gesture. Thus, by “identifying, recognizing or interpreting”, it is meant to relate (by the controller) a given gesture or combination of gestures to a particular command. The system includes a virtual keyboard and a monitor or display screen that optionally displays the virtual hand, optionally simulating the position (multiple positions) and / or movement (multiple movements) of the user's physical hand Can do.

In some embodiments, the meaning of keys (multiple keys) or key labels (multiple labels) on the virtual keyboard is dynamically updated (changed) according to user commands. Instead, all or only part of the virtual keyboard layout is dynamically updated (changed) according to user commands. For example, the virtual keyboard is dynamically updated (changed) according to the position (multiple positions) and / or action (multiple actions) of the user's hand.

In some implementations, the system may provide the user's previous command, the predicted key (s) that are subsequently required on the virtual keyboard, and / or the predicted layout that is subsequently required for the virtual keyboard. An evaluation and predictability software application (configured by the controller) that evaluates, predicts, or otherwise determines based on may be included. In some implementations, hand movements or other gestures can perform mouse type navigation.

As part of this disclosure, a method is provided for entering data into a computing system. In some embodiments, the method obtains an image (multiple images) of a part of a user's body and a physical keyboard, and a key (multiple keys) based on the obtained image (multiple images). Setting and displaying a mapping of. The method further includes processing and interpreting a signal relating to or associated with the acquired image (s) and allowing input of commands and / or symbols selected according to the signal. it can. The method further includes the step of using a keyboard identification function to identify a physical keyboard key located within the FOV of the image capture device, and the hand position and / or hand movement relative to the physical keyboard. Processing at least one image of the user's hand for measurement and the position (s) of the at least one hand on a virtual keyboard on a corresponding display screen, for example on a computer display or on a computer display object. Position) motion (multiple motions) displaying.

In some embodiments, the method dynamically updates key labels on the virtual keyboard in response to the processed image and / or the entire keyboard layout of the virtual keyboard in response to the processed image. Or it can include a step of dynamically updating a part.

In some embodiments, the method may be interpreted as a hand movement intended to perform mouse type navigation and / or user input commands and / or data using a virtual mouse. Other body movements can be included. The term “other body movements” refers to, for example, hand movements, head movements, eye movements, mouth movements, or other types of movements that can direct user commands and / or data entry.

In addition to the illustrative aspects and embodiments described above, further aspects and embodiments will become apparent by reference to the drawings and by a review of the following detailed description.

Although several exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain modifications, substitutions, additions and subcombinations thereof. Accordingly, it is intended that the following appended claims and the following introduced claims include all such modifications, substitutions, additions and subcombinations, as well as their true To be construed as being within the spirit and scope of

Unless otherwise stated, what is clearly understood from the following description is to utilize terms such as “processing”, “calculation”, “calculation”, “measurement”, etc. throughout this discussion of the specification. A computer system memory and register or other such information storage, transmission or display device that represents the physical, e.g. electronic quantity of data in the registers and / or memory of the computing system. Refers to the operation and / or process of a computer or computing system or similar electronic computing device that manipulates and / or translates into other data that is similarly represented as an internal physical quantity.

The platforms, processes, and displays displayed herein are not inherently related to any particular computer or other apparatus. Various general purpose computing systems and networked devices can be used with programs in accordance with the teachings herein, or it may prove convenient to configure a more specialized device to perform the desired method. can do. The desired configuration for a variety of these systems will appear from the description below. In addition, embodiments are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the present disclosure as described herein.

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of various embodiments. However, it will be understood by one skilled in the art that the described embodiments may be practiced without these specific details.

As used herein, the term “gesture” includes at least actions (multiple actions) and / or signals (multiple signals) and / or instructions (multiple instructions) performed by the body part (s) of the person operating the keyboard. And / or signs (multiple signs) and / or instructions (multiple instructions) and / or requests (multiple requests), and the like. What is meant herein by "command" is to change the meaning or interpretation of a selected key (assign or reset a symbol) using a gesture or a sequence or combination of gestures, or Instructing, requesting, or instructing a computer to change the meaning or interpretation of the entire keyboard layout according to this gesture or sequence or combination of gestures. In this regard, the meaning or interpretation of the key (the way in which the computer interprets the key when pressed down) is the symbol that is physically marked on the key (initial or default symbol) or specified at a given moment. One of the different symbols assigned (or reset) to the key by the computer according to the command

The embodiments described herein can facilitate the improvement of human-machine interaction (HCI) problems, particularly associated with, but not exclusively, languages with many symbols. This type of implementation increases the speed of data entry by providing an easy-to-operate data entry environment, provides a complete solution for linguistic data entry tasks, and substantially reduces the number of typographical errors. Improve the usability of language word processors.

With reference to FIG. 1A, an exemplary data input system in accordance with some embodiments of the present disclosure is depicted in part. Data input system 100 includes at least one image capture or capture device, such as image capture device 110, which is located on computer display monitor 120, for example, a digital camera, video camera, PC camera, webcam, etc. be able to. Of course, the position (multiple positions), location (multiple locations), movement (multiple movements) and gestures (multiple gestures) of the user's hand (both hands) or of the user's body are clearly visible in that regard. That is, assuming that they appear in the FOV of the image capture device 110, the image capture devices 110 can be placed at different locations. The data input system 100 is further a signal associated with or functionally associated with the image capture device 110 and generated by the image capture device 110 and output to the controller, the signal being a gesture (multiple gestures). Or a map of keys (multiple keys) or an entire keyboard layout based on signals that represent the images (multiple images) in the FOV of the image capture device 110 that contain them and that contain them A controller (not shown) configured to set may be included. “Set a map” as used herein generally refers to an allocation (or a specific set of symbols for each particular key) that depends on the gesture (multiple gestures) of a particular symbol (several symbols) for a particular key. Allocation). More particularly, the mapping of keys (multiple keys) may be done by a user, such as a user at 131 and 132, such as a user where only a (real) hand placed on the physical keyboard 130 is shown. It may include changing the symbolic meaning assigned to the key (s) according to an associated action or gesture. The controller is an integral part of or embedded in a computer (PC, laptop, etc.) that takes input signals from a keyboard such as keyboard 130 and operates a display screen such as display screen 120. Can be incorporated, added, or added.

A user (not shown) can move his hand 131 and / or 132 with respect to or relative to the physical keyboard 130 from one position to another while being acquired by the image capture device 110. Signals relating to the images of the hands 131 and 132 are continuously or intermittently transferred to the data input system 100 for processing and interpretation. Data input system 100 relates to acquired images to identify gestures (multiple gestures) and / or actions (multiple actions) made by a user with his hands (both hands) or other body parts (multiple parts). The signal can be processed and interpreted, and commands can be executed according to or in conjunction with this gesture (multiple gestures) and / or action (multiple actions). The physical keyboard 130 can be a standard keyboard (with symbols marked on it), a blank keyboard (keyboard with no markings on the keys), a paper keyboard (eg, a keyboard with any number of keys). ), Touchpads, keypads, virtual keyboards (flat bare surfaces such as tables and boards), and the like. The data input system 100 can also take advantage of a word application (multiple applications) that is suitable for processing the language (s) used (eg, English, Hindi and German).

In some implementations, the controller of the data input system 100 can operate on a digital signal processing (“DSP”) technology and a computer screen such as the computer screen 120 to process the images collected by the image capture device 110. In addition, a simulation technique for displaying a corresponding virtual keyboard such as the virtual keyboard 140 can be utilized. In some of these embodiments, the number, size, and spacing of keys on the virtual keyboard 140 may be substantially similar to those of the physical keyboard 130 to facilitate user orientation. it can. According to other aspects, either the number or size of keys on the virtual keyboard 140 or the spacing can differ from those of the physical keyboard 130. An image that is acquired or collected by the controller of the data input system 100 as symbols (symbols) or meanings assigned to keys (several keys) and / or symbols or meanings assigned to the entire layout of the virtual keyboard 140 ( Multiple images) can be varied according to corresponding user gestures or actions that can be identified, recognized or interpreted by the controller of the data input system 100.

In some embodiments of the present disclosure, the controller of the data input system 100 can utilize simulation techniques to create and process virtual hands (both hands) and use the virtual hands (both hands) for the user. It can appear and move on the display screen according to the position, location and movement of its (real physical) hands (both hands). For example, virtual hands 121 and 122 are shown in FIG. 1A reflecting user hands 131 and 132, respectively. Virtual keyboard 140 and / or virtual hands 121 and / or 122 can be similarly or differently scaled to facilitate ease of data entry.

To enter data in the required language by utilizing the data entry system 100, the user can place his hand as shown at 131 and 132 and within the FOV of the image capture device 110 In addition, a gesture or a series or combination of gestures associated with the requested language can be made. The gesture or sequence or combination of gestures is then recognized or interpreted by the controller of the data input system 100 as being associated with the requested language. In response to recognizing or interpreting a gesture or a sequence or combination of gestures, the controller of the data input system 100 may assign the symbol (s) that make up the requested language to the selected key on the virtual keyboard 140. And a layout of the virtual keyboard 140 with the currently requested and assigned symbol (s) can be displayed. It can be said that the controller of the data input system 100 sets the keyboard layout map corresponding to the requested language. Once the requested language is set by the controller of the data entry system 100 in response to a user command, the user observes the keys on the virtual keyboard 140 and moves his hand across the physical keyboard 130. Data can be input to the data input system 100 by moving (131 or 132) or the specific finger (a plurality of fingers).

The user has reached each virtual hand (121 or 122) or its corresponding virtual finger (s) near the next key on the virtual key 140 to be depressed and the virtual hand 121 Or his hand or its specific finger (s) can be moved until 122 fingers overlap the key. The user can then depress the key on the physical keyboard 130 below the finger corresponding to or associated with the virtual finger that overlaps the requested key on the virtual keyboard 140. The above procedure can be repeated as many times as necessary to enter additional symbols. If the user desires to change or set to a different language, the user can pose a gesture or a series or combination of gestures corresponding to the different languages. Each time the controller of the data entry system 100 is asked to set a different language to set a different map of keys, the controller can enable the corresponding word application / processor. For example, if the controller of the data entry system 100 is asked to change from French to English, then the controller can disable the French word application / processor and enable the English application / processor. Can do. The physical keyboard 130 is functionally connected to the controller of the data input system 100 or to a computer and transfers to the controller signal representative of default symbols or functions associated with keys in the physical keyboard 130 within the computer. There is a controller for the data input system 100 to do this. Nevertheless, the controller of the data input system 100 is configured or configured to interpret signals transferred to it from the physical keyboard 130 according to the current mapping settings.

Data input system 100 has several advantages over prior art solutions. For example, a user entering data into the system does not need to move his / her line of sight back and forth between a physical keyboard (eg, keyboard 130) and a screen (eg, display screen 120) that displays the resulting typed input. Instead, the user looks at the virtual keyboard (eg, virtual keyboard 140) and the virtual hand (eg, virtual hands 121 and 122) is positioned at his (real) hand (eg, hands 131 and 132) ( All you have to do is look at moving, arranged in correlation with multiple positions) and movement (multiple movements).

In general, a symbol or function can be assigned to a key according to the language, mode or function requested by the user (by performing a corresponding action or gesture), so that a given key, it is assigned to a new language Once assigned, when depressed by the user, the mode or function is interpreted by the controller of the data entry system 100 in a different manner. The controller of the data entry system 100 can change the appearance of the virtual layout (such as the virtual layout 140) in response to commands provided by the user or paused. For example, the controller may change the keyboard architecture or key arrangement according to the desired application (multiple applications), for example by changing the number of keys, key dimensions, spacing and / or placement on the virtual keyboard. it can. According to some embodiments, the layout of the virtual keyboard is independent of whether the keyboard is real (with actual labels marked on each key), blank or paper keyboard. It can be changed according to a real-time simulation of the positioning and movement of the user's hand over the user's hand and physical keyboard. Another advantage of data entry system 100 is that the same physical keyboard (eg, physical keyboard 130) has as many sets of symbols (each of which belongs to a different language) as the number of available word applications / processors. It can be used to input.

In some embodiments, by collecting images of the position of the user's hand (both hands) along with keyboard key entries (buttons to be depressed), the controller of the data entry system 100 can It can detect the position and / or movement of the user's hand and finger at any given moment and in real time, and display them by displaying the virtual finger at the appropriate position on the keys of the virtual keyboard 140 Can be imitated. This allows the user to observe the placement and movement of his (female) hands on the monitor 120, thereby allowing the user to place his (female) fingers on any key on the physical keyboard 130. Is given at any given moment, regardless of the selected language, before pressing a key and without having to look down on a physical keyboard (eg, keyboard 130). If a real finger (eg, finger of hand 131) moves across the physical keyboard and presses down or touches an area above the physical keyboard, such as physical keyboard 130, the controller A virtual hand (eg, hands 121 and 122) can be imitated with the movement and shape of a finger with respect to the virtual keyboard.

In other embodiments, the data entry system 100 can allow processing of motion images of one or two hands, as well as other bodies. For example, in some embodiments, the data input system 100 collects and processes head movements, eye movements, mouth movements or other movements to direct user commands, signals and / or data input. be able to. If necessary, an additional image capture device, such as image capture device 110, may be used when different parts of the user's body cannot be placed within the FOV of a single image capture device. it can. In this type of case, each image capture device can be spatially installed to capture images associated with different body parts.

Referring now to FIG. 1B, an exemplary virtual keyboard and virtual hand are shown in accordance with some embodiments of the present disclosure. The virtual keyboard 150 displayed and shown on the computer screen 155 can be increased or decreased, adjusted, configured, or modified as needed or required by the controller of the data entry system. The controller can also change the position of the virtual keyboard 150 relative to the screen 155. The virtual keyboard 150 displays the current Indian language keyboard setup or layout, depending on the language / character set used (eg from 50 options) or for function keys or configured keyboards. It is shown that it can be defined by a defined macro. When a physical finger moves from one key on a physical keyboard (such as physical keyboard 130 in FIG. 1A) to another key and the latter key is depressed, the corresponding diagram change is virtual. 1 on the virtual keyboard 150 (as shown at 160 in FIG. 1B) that can be made with respect to the keyboard 150 and mimics or reflects the movement of the user's hand to the appropriate (desired) physical key It leads to the movement of hands (both hands) from one place to another. In some embodiments, the controller of the data entry system may change the appearance of the user's hand 160 according to a user-issued direct command or according to the result of such a command. What is meant by “according to a user-issued direct command” (for the latter embodiment) is to depress a key on the physical keyboard that functions to change the appearance of the virtual hand (both hands); On the other hand, what is meant by “according to the result of this kind of command” (with respect to the latter embodiment) is that the language to the controller (by posing a gesture or action as described above). Issuing commands to change and at the same time changing the appearance of the virtual hand (by the controller) according to the language used. Changing the appearance of the virtual hand can be, for example, to make the virtual hand (both hands) transparent or partially transparent to allow the user to view substantially all areas of the virtual keyboard 150. Can be included. A transparent virtual hand 160 is shown in FIG. 1B overlaid on the virtual keyboard 150. Text 151 is shown consisting of Hindi symbols according to the Hindi language currently assigned to virtual keyboard layout 150.

Referring now to FIG. 1C, three examples of virtual hands overlaid on different virtual keyboards are shown according to some embodiments of the present disclosure. A data input system, such as data input system 100 of FIG. 1A, may allow a user to change the labeling and / or layout of a virtual keyboard, such as virtual keyboard 150. If the user wishes to write a document using the Indian language (Hindi language with Devanagari characters when represented in FIG. 1C), the “KA” + “HALANT” combination (“K” in Windows®) + “D” key), he / she can change from the first to the second layout (eg, keyboard layout 170 from keyboard layout to 171) and a specific key on the virtual keyboard Or the entire appearance can thus change the language and / or mode to be used and / or the associated keys and / or layout or map for the characters (eg “KA's when represented in FIG. 1C HALANT format ").

In other embodiments, the user can change the keyboard mode or function, for example, he / she can switch between his (female) key language, symbols, characters, graphics and others on the virtual keyboard. Switching can be done by pointing the appropriate gesture with the hand / finger. Moreover, the appearance and / or degree of transparency of the virtual hands (both hands) can vary according to the actual keyboard used. For example, virtual hand 172 is shown to be less transparent compared to virtual hand 173 because they are each associated with a different keyboard layout (respectively to keyboards 170 and 171). Virtual keyboard 175 is shown having only six selected symbols (collectively indicated as 174). Virtual keyboard 175 can be shown as translucent.

Referring now to FIG. 1D, it represents an example of a signal or gesture, in which a user instructs a data input system, such as data input system 100, to change language, mode, input data, change function For example, an image capture device such as image capture device 110 can be paused. For example, the user can use the object and / or his (woman) left hand to create a selected signal / gesture that can be collected by the image capture device and that is preferably mapped to a virtual keyboard key. it can. FIG. 1D represents ten exemplary hand gestures, each of which is assigned a unique hand gesture number, for example. Each hand gesture number can be associated with a specific command or action performed or taken by the controller of the data entry system. For example, hand gesture number 5, shown at 182, associated with hand gesture description 181, may, for example, map a virtual keyboard layout or mapping (eg, the layout of virtual keyboard 150 in FIG. 1B) from one language to another. Can be instructed to instruct or signal the controller of the data input system. Subsequently, the user moves his hand (both hands) / finger over the physical keyboard to create a corresponding virtual hand (both hands) that moves in relation to the modified virtual keyboard. A modified virtual layout can be used to enter (type) characters or symbols in other languages.

Thus, identifying a signal, command, instruction, etc. by a data input system, such as data input system 100, first causes a hand gesture or signal (by an image capture device such as image capture device 110 of FIG. 1A) to be received. Performed by identifying or recognizing and then interpreting hand gestures into corresponding hand gesture numbers and using hand gesture numbers as described above. Can.

The data input system 100 of FIG. 1A may be manually or automatically performed by a user (by displaying a gesture or action corresponding to the image capture device 110), for example after pressing a “Reset” button on a physical keyboard. Directed by receiving a signal or following a selected command. Any number and type of hand gestures, signals and / or actions and / or other suitable signals and / or gestures and / or actions made by body parts and / or objects and others may be Can be used as a command. For example, left hand and / or right hand positions and / or movements can be collected, as well as facial movements, head movements, finger movements, shoulder movements or used by the user to command commands. Other suitable actions that can be collected. In this way, this data entry system allows a user to change the layout, mode, function, etc. of a virtual keyboard such as virtual keyboard 150 of FIG. 1B with a minimum number of keystrokes or other actions. May be.

In one example, the user can type one or more keys to make the gesture associated with the selected layout and subsequently enter the required data. This kind of data entry, which may require several key entries to change the layout, keys and others and usually reach the required layout, is a combination of gesture and type input for the selected key Can be implemented by applying For example, if the user wants to enter a Hindi character with more characters compared to the keys on the physical keyboard, the user can change the keyboard key label and / or layout on the virtual keyboard, for example. To change, commands can be entered using gestures or signals. This change in the virtual keyboard can cause the necessary characters to be displayed on the virtual keyboard so that a minimum number of keystrokes are required to enter the selected key. Thus, only one keystroke is required to enter any selected character from a set of language characters with many different characters. Other operations and / or combinations of operations are similarly performed.

According to some embodiments, the data entry system allows the controller of the data entry system to subsequently determine the expected keys and / or keyboard layout (map) that can be requested or desired by the user. Evaluation and predictability applications to help can be included. This predictability depends on the user's previous command (multiple commands), for example, commands previously issued using hand gestures (multiple gestures), movements (multiple movements), mouse movements (multiple movements), key entries and others , Based on the evaluation. For example, if a language with many characters is currently in use, the word application of the currently used language can interpret two or more specific key combinations that are equal to the selected character entry. The predictive application automatically updates other related keys to complete possible functions resulting from the first key combination by various other keys, for example after hitting the first one of the combination keys Can do. For example, hitting “A” and then various other keys will enter a selection of commands, at which time “A” will act as a kind of function key. If the user enters “A”, the virtual keyboard can be immediately changed to display all relevant commands or keys that can be entered in combination with “A”. In this way, the user does not need to memorize or use a physical table to find key combinations, but rather the related combinations are dynamically updated on the virtual keyboard in real time. Can do.

Referring now to FIG. 1E, two comparative approach tables are illustrated according to some embodiments of the present disclosure. Table 190 represents a number of Indian symbols and each English character to be entered to obtain this Indian symbol. According to the exemplary table 190, an Indian language character must be typed (input or keyed) accordingly, a single or 2, 3, 4, or 5 combination of English characters or signatures accordingly. It can be represented, used or obtained using conventional methods. For example, character 193 is obtained by entering the character “s” (194), and character 195 is obtained by entering the character combination “s / t / r” (196). Thus, five keystrokes are necessary to obtain the character 195.

Table 191 represents a method for obtaining the same Indian language characters (indicated by 197) by entering only one English character in combination with the corresponding hand gesture (one strike implementation). For example, the character 193 is obtained by entering the character “s” (198), as before (194), and without using any hand gestures. Because using a single character (194 or 198) is simple enough. However, instead of using five keystrokes (196) to obtain the character 195, only one character (eg, the letter “s”) is combined with the corresponding hand gesture 199 (in this example, hand gesture 3). Can be entered.

Referring now to FIG. 1F, some examples of mappings corresponding to the Indian language characters shown in FIG. 1E are schematically illustrated and described. FIG. 1F is described in connection with FIG. 1E. In the example, the letter “S” was entered according to this example without any hand gesture, so the initial or default English letter “S” is assigned symbolically (at 184) to the Indian language letter 183. (In 185 in both figures, “hand gesture” equals “none”). In another example, according to this example along with hand gestures, the letter “S” was entered, so the initial or default English letter “S” is symbolically (at 186) assigned to the Indian language letter 187 and shown. (At 188 in both figures, “hand gesture” equals “none”).

With reference now to FIG. 2A, a series of operations or processes are schematically illustrated and can be implemented to operate a data entry system. In block 200, the user can set up or initialize data entry system software on his (female) computer. Once the software is running and functioning, a calibration screen can be shown, indicating that the system has started or has begun operation. At block 205, the user can place a real (physical) keyboard within the field of view of the camera. At block 210, the data entry system can use, for example, a keyboard identification function to identify keyboard positions and keys, and then the keyboard is identified and the data entry system is ready to operate. The user can be notified.

At block 215, the user can place his (female) hands (both hands) within the field of view of the image capture device. At block 220, the data entry system can collect and process hand (both hands) images, and then the data entry system can notify the user that the hand (both hands) identification is complete. . At block 225, the user can run the word processing application according to the actual language used. At block 230, the data entry system may display a virtual keyboard on the display of the data entry system along with a default set of keys according to the selected word processing application. The user enters commands into the real keyboard as needed to enter data in the selected language, paying attention to the associated movement of his (female) virtual hands (both hands) on the virtual keyboard. Can be typed. A virtual keyboard represents a virtual finger moving actively to depress a selected key, thereby indicating the actual entry of the command.

At block 235, if the user desires to change the keyboard mode, language, input commands, and others, the user can make a selected signal using one or two hands (or other body parts). This signal (multiple signals) can be selected from a set of preset signals or actions. At block 240, the data entry system can collect and process hand signals and enter the required user command (s) or data. At block 245, keys and / or layouts on the virtual keyboard are entered by entering, for example, function keys, key combinations, mouse actions or commands, key entry (multiple entries) and mouse action (multiple actions) combinations, and others. , Can be changed according to user commands (multiple commands). Any combination of the above steps can be implemented. In addition, other steps or sequence of steps can be used instead of and / or in addition to previously specified steps.

In other embodiments, the method can allow processing of one or two hands, as well as other body movements. For example, in some embodiments, the method can collect and process head movements, eye movements, mouth movements or other commands that direct user commands, signals and / or data input.

Referring now to FIG. 2B, the general layout and functionality of a data entry system is illustrated and described schematically in accordance with some embodiments. At block 260, the image capture device 290 may collect gestures made or taken by a user (not shown) hand 292. At block 265, the collected gesture can be identified, for example, as gesture number 3 in FIG. 1D. At block 270, the virtual keyboard (291) can be changed to display layout number 3 (for example) corresponding to gesture number 3. Thus, at block 275, the word processor can change to mode 3 operation (change language). At block 280, the user can depress the key (s) on the physical keyboard 293. At block 285, the controller (not shown) of the data entry system 201 determines the position (multiple positions) and movement (multiple movements) of the virtual hand 294 using the data entry system 201 to the user's physical. Simulates the user's fingers (multiple fingers) (denoted by 294) that hit the corresponding key (multiple keys) on the virtual keyboard 291 in substantial association with the position (multiple positions) and movement (multiple movements) of the hand 292 Can be displayed. Other steps, a sequence of steps, can be used. The computer 286 is operatively connected to a physical keyboard 293 from which it can receive a signal representing a depressed key, and in particular to a display screen 295 where it transfers a virtual hand and virtual keyboard image. ing.

Referring now to FIG. 3, a mouse-like implementation is described with the figure according to some embodiments of the present disclosure. As part of this disclosure, a virtual hand can simulate or mimic mouse-like navigation, a user can enter data into the computer (304) using the virtual mouse, and A graphic application can be run. Computer 304 includes a controller (not shown) for data 306. At block 300, the image capture device 301 can collect user actions or body parts, such as a hand (shown at 302), which can be moved to perform mouse-type navigation. At block 305, the direction of hand movement is observed and transferred to the computer 304, for example, in the XY plane. Additionally or alternatively, at block 310, user actions or gestures (multiple gestures) may be collected by the image capture device 301. At block 315, the collected image (multiple images) of the gesture (multiple gestures) may be processed to enter user commands (multiple commands) and / or data. At block 320, commands and / or data and others can be entered into the computer 304, which can accordingly navigate, for example, on the display 303, change modes and / or functions, and Entered by command and executed by others.

Computer screen (eg, display screen 295 of FIG. 2B or 303 of FIG. 3), physical keyboard (eg, physical keyboard 293 of FIG. 2B) and image capture device (eg, image capture device 290 of FIG. 2B or 301 of FIG. 3) Is any suitable conventional display screen, physical keyboard or image capture device. The computer 286 of FIG. 2B or the computer 304 of FIG. 3 is captured to determine the actions and gestures it makes or poses by the user in addition to its normal hardware and software components. If we assume that we include hardware and software applications that enable the necessary virtual reality to analyze the captured images and to generate and process the virtual keyboard and virtual hands (both hands) in general, Any suitable conventional computer.

The above description of various embodiments has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. It should be understood by those skilled in the art that many modifications, variations, substitutions, modifications, and equivalents are possible in light of the above teaching. Accordingly, it is intended that the appended claims be construed to include all modifications, substitutions, additions and subcombinations, which are also within the true spirit and scope thereof. .

Briefly, exemplary embodiments are illustrated in the referenced figures. It is intended that the embodiments and figures disclosed herein are to be considered illustrative rather than limiting. However, the disclosure of both the structure and method of operation, along with its objects, functions and advantages, can best be understood with reference to the following detailed description when read in conjunction with the accompanying drawings, in which: ,

FIG. 6 is a diagram illustrating a virtual keyboard according to some embodiments of the present disclosure. FIG. 6 is a diagram illustrating a virtual keyboard according to some embodiments of the present disclosure.

FIG. 3 is an example of a keyboard diagram with a limited number of Hindi characters that can be utilized as a physical and / or virtual keyboard, according to some embodiments.

FIG. 4 illustrates an exemplary set of graph diagrams of various finger-based signals or gestures that can be used to direct commands and / or data for input in accordance with some implementations.

4 illustrates an example of mapping input keys based on signals associated with acquired images and an example of input key maps based on signals associated with acquired images in accordance with some implementations of the present disclosure. 4 illustrates an example of mapping input keys based on signals associated with acquired images and an example of input key maps based on signals associated with acquired images in accordance with some implementations of the present disclosure.

2 shows an exemplary flow diagram for operating a data entry system in accordance with some embodiments of the present disclosure.

Fig. 4 schematically illustrates the general layout and functionality of a data entry system according to some embodiments of the present disclosure. and

Fig. 4 schematically illustrates the general layout and functionality of a data entry system according to another embodiment of the present disclosure.

It will be understood that for simplicity and clarity of explanation, the elements shown in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the drawings to indicate corresponding or analogous elements throughout the consecutive drawings.

Explanation of symbols

100 Data Input System 110 Image Capture Device 120 Computer Display Monitor 121 Virtual Hand 122 Virtual Hand 130 Physical Keyboard 131 User Hand 132 User Hand 140 Virtual Keyboard 150 Virtual Keyboard 155 Computer Screen 160 User Hand 170 Keyboard Layout 171 Keyboard Layout 172 Virtual hand 173 Virtual hand 175 Virtual keyboard 181 Gesture description 183 Character 187 Character 190 Table 191 Table 193 Character 194 Character 195 Character 196 Keystroke 197 Character 198 Character 199 Hand gesture 201 Data input system 286 Computer 291 Virtual keyboard 292 User's physical hand 293 physical keyboard 294 user's finger 295 display screen
301 image capture device 302 user's hand 303 display 304 computer 306 data

Claims (34)

A data entry system,
An image capture device for generating an output signal associated with a captured image (s) of a part of a user's body and a physical keyboard;
A controller configured to receive and output the output signal and to set and display a map of input keys (multiple keys) based on the captured image (multiple images). The system of claim 1, wherein the map of input keys (multiple keys) is part of a virtual keyboard. The part (s) of the user's body is the user's hands (both hands), and the controller simulates the user's hands (both hands) placed or moved relative to the physical keyboard The system according to claim 2, further displaying a virtual hand (both hands). The system of claim 1, wherein the controller is further configured to interpret the captured image (s) for a user gesture as a unique command. The system of claim 4, wherein the controller is further configured to change a key (multiple keys) on the virtual keyboard according to a user gesture (multiple gestures). The system of claim 4, wherein the controller is further configured to change the layout of the virtual keyboard according to a user gesture (multiple gestures). The system according to claim 3, wherein the controller is further configured to change an appearance of the virtual hand (both hands). The system of claim 1, wherein the physical keyboard is selected from the group of a keyboard consisting of a blank keyboard, a paper keyboard, a touchpad, a keypad, a virtual keyboard and a flat bare surface. The controller is configured to determine and display a subsequent expected key required on the virtual keyboard based on a user's previous command (s). Item 5. The system according to Item 4. The controller is further configured to determine and display an expected keyboard layout required subsequently on the virtual keyboard based on a user's previous command (s). The system according to claim 4. The system of claim 4, wherein the gesture is performed by a user's hand. The gesture is one or more user actions selected from the group consisting of actions performed to perform mouse-type navigation and actions executed to perform commands. 4. The system according to 4. The gesture is performed by one or more body parts of a user selected from the group comprising hand movements, head movements, eye movements, mouth movements or other movements that direct user commands; The system according to claim 4. The system according to claim 3, wherein the virtual hands (both hands) are displayed on the virtual keyboard in a transparent manner. The system of claim 3, wherein the appearance of the virtual hand depends on the virtual keyboard layout used. The controller is further configured to predict a next key (multi-key) label (multi-label) and a next virtual keyboard layout by evaluating a previous command (multi-command). The system according to claim 4. The system of claim 4, wherein the gesture is used in combination with a depressed key of the physical keyboard to set and display a corresponding mapping. A method of entering data into a computing system, the method comprising: capturing a portion of a user's body and an image of a physical keyboard (multiple images) and inputting on a virtual keyboard based on the captured image (multiple images) Setting and displaying a mapping of keys (multiple keys). The method of claim 18, wherein the image (multiple images) relates to a gesture made by the user to be interpreted as a command. 20. The method of claim 19, wherein the gesture relates to an action (multiple actions) selected from the group consisting of hand movements, head movements, eye movements and mouth movements. 20. The method of claim 19, wherein key labels (multiple labels) change on the virtual keyboard in response to the gesture. The method of claim 19, wherein the layout of the virtual keyboard changes in response to the gesture. Furthermore, capturing the user's hands (both hands) and displaying a virtual hand (both hands) that simulates the user's hands (both hands) placed or moved relative to the physical keyboard The method of claim 18 comprising: The method of claim 23, wherein characters are entered into the computing system using a single keystroke of the physical keyboard. The method of claim 21, wherein the hand movement (s) simulates a virtual mouse to perform mouse type navigation. The method according to claim 23, wherein the virtual hands (both hands) are displayed on the virtual keyboard in a transparent manner. 24. The method of claim 23, wherein the appearance of the virtual hand depends on the virtual keyboard layout used. 20. The method of claim 19, further comprising evaluating a previous command (multiple commands) to predict a next key (multi-key) label (multiple labels) and a next virtual keyboard layout. The method of claim 19, wherein the gesture is used in combination with a depressed key on the physical keyboard to set and display a corresponding mapping. A method of entering data into a computing system comprising:
Displaying a virtual hand (both hands) that is correlated with an actual user hand (both hands) and overlaid on a virtual keyboard that is representative of a physical keyboard;
Setting a map to a key on the virtual keyboard in response to a gesture made by a user's hand. The method of claim 30, wherein the layout of the virtual keyboard changes in response to gestures made by actual user hands. The method according to claim 30, wherein the virtual hands (both hands) are displayed on the virtual keyboard in a transparent manner. The method according to claim 30, wherein the appearance of the virtual hands (both hands) depends on the virtual keyboard layout used. The method of claim 30, wherein the gesture is used in combination with a depressed key on the physical keyboard.

Images