CN105830011A - User interface for overlapping handwritten text input - Google Patents

Abstract

A "stroke cleaner" composes handwritten messages from handwritten strokes representing overlapping letters or portions of letter segments drawn on a touch screen device or touch sensitive surface. These overlapping strokes are automatically cleaned up and then segmented and combined into one or more letters, words or phrases. Advantageously, segmentation and composition are performed without user gestures, timeouts, or other input for delimiting characters in words, and without using handwriting-based recognition techniques to guide cleanup and composition of overlapping strokes to form characters. In other words, the user draws multiple overlapping strokes. These strokes are then automatically segmented and combined into one or more corresponding characters. Recognition of the resulting character is then performed. Further, the segmentation and combination is performed in real-time, thereby enabling the resulting characters to be presented in real-time in a user interface window. A related drawing mode enables a drawing to be input in combination with handwritten characters.

Description

User interface for overlapping handwritten text input

Background technique

Traditionally, text is entered using a hardware keyboard (eg, 12 phone keys or a QWERTY keyboard) or a software keyboard (eg, keys are graphical buttons displayed on a touch screen display). The main problem with this approach is the small size of the keys on handheld devices, making them relatively difficult to type. Typically, the keyboard is divided into sections (letters vs. numbers vs. special characters). This allows the user to switch between sections and only have one section displayed on the screen at a time. This allows for larger buttons per section, but switching causes overhead for the user. Also, as the screen size of mobile devices decreases, users tend to make more mistakes when touching intended characters on the touch screen, thereby reducing overall accuracy. Thus, there is a perception that typing on a smaller screen offers no particular value in terms of speed and/or accuracy compared to more natural user input (eg, handwriting recognition).

Thus, in the case of small screens, handwriting sometimes offers more benefits than typing for personal communications. In addition, handwritten notes often contain subtle personal cues by virtue of handwriting, as well as drawings that cannot be conveyed through typed text. Many of today's communication devices (eg, smartphones, tablet computers, etc.) have touchscreen-based systems that are capable of recording handwriting entered on the device's touchscreen. Several business applications employ handwriting for note-taking and messaging. Unfortunately, these types of handwriting applications typically require a stylus, or a surface large enough for a finger to write with sufficient precision. This is problematic for the small screens of mobile devices.

For example, one conventional handwriting technology provides a mobile phone-based application in which a user writes characters while the phone is in landscape mode. A user's leftmost touch of the screen (or wait for a timeout) accepts the writing and clears the screen for more input. Unfortunately, this approach requires the user to spatially organize her input so that an "accept and clear" type action is triggered and thus has the right amount of space around each character, since the characters are created by placing the and trailing blanks are included to be concatenated. Additionally, the landscape orientation of such applications necessitates frequent rotation of the device. Another similar application uses a two-finger swipe gesture to manually arrange predefined ink segments on the screen, forming individual characters.

In an attempt to address the limited space available for writing with fingers on smaller screens, various automatic segmentation techniques have been implemented for use in mobile text entry scenarios. Such a system allows letters to be written on top of each other, and these letters are then separated by post-processing. For example, so-called overlap recognition systems typically work by evaluating overlapping strokes to identify intended characters, and then using this information to segment the strokes. Unfortunately, this means that these systems use a form of text recognition or handwriting recognition, which is performed in conjunction with stroke segmentation as a guide or aid in segmenting the strokes, with the resulting complexity and computational Increased overhead requirements. One way to avoid the character segmentation problem is to use a custom single-stroke alphabet, where each individual stroke represents a different character. Such techniques may allow for eyes-free input, but require the user to learn a new set of gestures (ie, different individual strokes for each individual character) to enter text.

Another method of finger input handwriting allows a user to write overlapping characters through a fabric on a touch surface. This approach typically recognizes single-stroke letters (eg, "c" or "l") without further user action. However, to segment overlapping strokes for more complex characters such as "k", "t", "f" and "x", this method looks for a second specific stroke. In particular, if a "<", "—", "\" or "/" gesture is encountered, the method simply fuses that stroke with a previously entered stroke to produce a character. In other words, the method either recognizes individual strokes as characters, or looks for one or more special identifier strokes or gestures that indicate that consecutive strokes are to be merged. Without recognizing that type of special character, the system cannot segment multiple overlapping strokes to determine what character the user intended.

Contents of the invention

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. Furthermore, while certain shortcomings of the prior art may be noted or discussed herein, the intent is not to limit claimed subject matter to implementations that may address or address any or all of these prior art shortcomings.

In general, a "Stroke Untangler" as described herein provides various techniques for using fingers or writing on touch screens, or other real or virtual writing surfaces (collectively referred to herein as "writing pads") Implement tools to compose handwritten messages. Handwritten strokes representing overlapping letters or partial letter segments are first drawn on the writing pad. These overlapping strokes are then automatically untangled, segmented and synthesized into one or more letters, words or phrases. Advantageously, this untangling, segmentation, and synthesis is performed without requiring user gestures (or predefined segmentations), timeouts, or other input to delimit characters in words. Furthermore, in contrast to existing segmentation techniques, the untangling and segmentation process is performed without using text or handwriting recognition techniques as a guide for untangling and segmenting overlapping strokes input by the user. However, in various embodiments, recognition of the resulting characters is then performed using various handwriting recognition (HWR) techniques after untangling and segmentation have been performed.

In view of the foregoing disclosure, it should be apparent that the stroke untangler described herein provides various techniques for composing handwritten messages from multiple overlapping strokes, without the need for predefined gestures or strokes, and without the use of HWR-based techniques As a guide to untangle, segment and combine multiple overlapping strokes to form characters. In addition to the benefits just described, other advantages of the stroke untangler will be readily understood from the following detailed description when taken in conjunction with the accompanying drawings.

Description of drawings

Certain features, aspects and advantages of the claimed subject matter will be better understood from the ensuing description, appended claims and drawings. In the attached picture:

Figure 1 illustrates the use of the "Stroke Untangler" to automatically untangle, segment, and synthesize multiple overlapping strokes to form characters, words, and phrases, as described herein.

2 provides an exemplary architectural flow diagram illustrating program modules for implementing various embodiments of a stroke untangler, as described herein.

Figure 3 shows an exemplary Stroke Untangler user interface based on a bracelet or wrist watch form factor showing display of untangled, segmented and synthesized strokes, display of HWR based recognition candidates, search suggested Display, and fade stroke traces to provide visual feedback to the user, as described in this article.

4 illustrates an exemplary stroke untangler user interface showing the display of untangled, segmented, and synthesized strokes, and the corresponding HWR-based machine-readable text and drawing input.

Figure 5 illustrates an exemplary stroke untangler user interface showing the display of untangled, segmented and synthesized strokes, corresponding HWR-based machine-readable text, search suggestions, and pair stroke segmentation errors combined with the display of the user corrections made.

6 illustrates an exemplary state diagram used in performing stroke segmentation for untangling and compositing multiple user strokes to form one or more characters, as described herein.

7 illustrates an exemplary stroke untangler user interface showing a stroke-based predictive keyboard on a wristwatch-type device, where untangled, segmented, and synthesized strokes are used to display a limited set of keys to for user selection, as described in this article.

8 shows an example of a ring form factor based writing pane with a touchscreen or touch-sensitive surface on which overlapping strokes can be entered for processing by the Stroke Untangler, as described herein. describe.

9 illustrates the use of a handwriting recognition model for processing broken characters formed from broken strokes (e.g., unfinished strokes caused by crossing the edge of a writing panel while inputting overlapping strokes), as described herein. describe.

FIG. 10 provides a general flowchart illustrating an exemplary method for implementing various embodiments of the Stroke Untangler, as described herein.

11 is an overall system diagram depicting a simplified general-purpose computing device with reduced computing and I/O capabilities for implementing various embodiments of the Stroke Untangler, as described herein.

detailed description

In the following description of embodiments of the claimed subject matter, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustrations specific embodiments in which the claimed subject matter may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the presently claimed subject matter.

1.0 Introduction:

In general, "Stroke Untangler" as described herein provides various techniques for composing handwritten messages with fingers or writing implements on a touch screen or other real or virtual writing surface. Handwritten strokes representing overlapping letters or partial letter segments are first drawn on the touchscreen device or touch-sensitive surface. These overlapping strokes are then automatically untangled, segmented and combined or synthesized to one or more letters, words or phrases. Furthermore, segmentation and combining are performed in real-time, thereby enabling real-time presentation of the resulting characters in a user interface window. In various embodiments, after segmentation has been completed, text recognition is performed on the resulting characters using various handwriting recognition (HWR) techniques.

In other words, the stroke untangler allows a user to continuously or periodically draw multiple overlapping strokes on a touch screen or touch-sensitive surface. These strokes are then automatically segmented and combined into one or more corresponding characters. Advantageously, segmentation and combination are performed without requiring user gestures (or predefined segmentations), timeouts, or other inputs to delimit characters in words. However, in various embodiments, the Stroke Untangler enables various control gestures to implement functions such as backspace to delete characters, spaces between words, and clear-type commands to clear entire words or phrases. Note that in various embodiments, the user does not enter spaces between words, but subsequent HWR processing automatically determines word breaks and then inserts spaces between characters when appropriate.

Furthermore, in contrast to existing HWR-based segmentation techniques for segmenting strokes, the Stroke Untangler segments strokes by dividing overlapping strokes into k-stroke characters without using HWR-based segmentation techniques. This enables the stroke untangler to untangle and render handwritten messages while preserving the user's personal handwriting in a manner that maintains an aesthetic quality not found in typical recognition systems. Furthermore, these handwritten messages are inherently in a format that can be presented to various HWR systems for further processing. Note that an associated drawing mode enables drawing input in conjunction with handwritten characters.

Advantageously, stroke segmentation is based on a trained segmentation model without using HWR-based techniques as a guide to the segmentation process, significantly reducing computational overhead. Furthermore, the method facilitates unusual words or expressive spellings (eg, "whoooah") because the techniques enabled by the Stroke Untangler do not require the use of dictionaries or the like when untangling and segmenting strokes. Furthermore, character or text recognition following the untangling, segmentation and rendering process performed by the Stroke Untangler has been observed using ) significantly reduces computational overhead, thus facilitating the initial untangling and segmentation process.

In various embodiments, the Stroke Untangler also provides an "ink display" (i.e., a displayed representation of a user-entered stroke) of a complete word, which updates in real time as strokes are untangled, segmented, and combined to form characters , thereby creating a very tight feedback loop with the user. In other words, the display of the current stroke entered by the user utilizes the stroke untangling and segmentation algorithm (see Section 2.3) which periodically updates the resulting character groupings to update the displayed characters in real time. As noted above, in various embodiments, text recognition or handwriting recognition is then performed on those characters after the untangling, segmentation, and synthesis processes. This capability is further leveraged in various embodiments by providing a graphical user interface (UI) where the resulting character is displayed adjacent to one or more recognition candidates available for user selection as a reference to Input to one or more other applications (eg, text editors, search engines, etc.).

For example, FIG. 1 illustrates the use of a stroke untangler implemented via a user interface of a handheld computing device to automatically untangle, segment, and synthesize multiple overlapping strokes to form characters and words. Note that Figure 1 is not intended to represent all features, features or embodiments of the Stroke Untangler, but is intended simply as a basic visual example to introduce some general concepts described throughout this document.

In particular, FIG. 1 shows a handheld computing device 100 having a touch screen 110 . A plurality of overlapping strokes 120 (represented by a plurality of overlapping dashed curves displayed on the touch screen 110 ) are input by fingers 120 of the user's hand. In the embodiment shown in FIG. 1, the stroke untangler untangles and segments overlapping strokes 120 and synthesizes these strokes to form a set of human-readable characters 140, which appear to spell the word "pizza." Such a set of human-readable characters 140 is displayed in a character input window 150 on the touch screen 110 .

Note that direct input on the touch screen is not required in various embodiments. Such strokes can be performed, for example, on the touchpad of a laptop computer, or on the back of a mobile device with a touch-sensitive surface. Furthermore, such strokes can be made on one device (eg, watch, touch-sensitive bracelet) while being sorted out (and presented to the user) on another device (eg, desktop computer, electronic whiteboard, etc.). Other examples include various The TouchCover or TouchCover2 input device associated with the device. Such devices include various combinations of capacitance-sensing, pressure-sensing, and proximity-sensing technologies. Such a device may be adapted (via various software or driver modifications) to receive strokes entered by tapping on or over the top of one or more pressure-sensitive or proximity-sensitive keys, rather than just sensing on-screen or touch strokes on the board.

Furthermore, in the embodiment shown in FIG. 1 , the stroke untangler performs automatic handwriting recognition on a set of human-readable characters 140 that was passed to represent the text "pizza" as machine-readable text Recognized by the HWR component of the Stroke Untangler. This recognized text is then automatically provided as input to the search engine, which in turn returns a set of user-selectable search suggestions in suggestion window 160 (eg, "pizzahut," "pizzahutmenu," and "pizzarecipe").

Note that in various embodiments, HWR is not executed until an entire word is entered (determined by the user entering a space or word separator after a stroke entry). In this case, the HWR performs word-by-word processing of human-readable characters after each individual word is entered. In a related embodiment, recognition is performed on each word or part of words as soon as each stroke is added to the character, even when the word is not yet complete. The resulting part identification is often helpful to the user. For example, if a user writes "pizz" and part of the recognition has gone wrong, e.g., the system returns "riff", there is a good chance that the word is not written in a way that would be recognizable based on HWR processing of the corresponding characters, so even in It is also not recognized correctly when completed. In this case, the user may choose to start over with the word (eg, delete one or more characters and re-enter those characters).

It should also be noted that in the embodiment shown in FIG. 1 , the user interface of the stroke untangler includes buttons (170, 180, and 190) or the like, for switching between text mode and drawing mode, and for sending The set of human-readable characters 140 or corresponding recognized text serves as input to another input device or application (eg, search, email, text editing application, etc.).

As a final note, because a touchscreen is typically capable of receiving input across its entire surface, the Stroke Untangler allows user-entered strokes 130 to be entered on any part of the touchscreen, even in other user interface windows (e.g., the character entry window 150 and the suggestion window 150). window 160). Advantageously, this increases the space available for a user to enter individual character fields. However, portions of the touch screen may be excluded from stroke input if desired. Also, note that a device's touch sensitivity may extend beyond the boundaries of its visible screen, such as a touch-sensitive bezel on a watch or mobile device, which allows a user's strokes to be detected even when the strokes may extend beyond the boundaries of the touchscreen. Perceived completely.

Examples of additional features and embodiments of the Stroke Untangler include, but are not limited to, various UI elements, such as presenting fast-fade ink trails to provide visual feedback on the path a user's finger takes across the touchscreen while entering a stroke. Another example of a UI element provided by a stroke untangler is when a user moves her finger across a touchscreen while entering a stroke, displaying untangled and segmented "ink" (i.e., a visual representation of the stroke) on the writing panel. ).

In another embodiment, various recognition candidates (similar to the search candidates shown with respect to FIG. 1 ) are provided relative to the recognition of the user's handwriting. In various embodiments, a user may accept a recognition candidate at any time by tapping or otherwise selecting one of the candidates. Recognition candidates may also include predictions, for example, the system may include the word "pizza" in the recognition candidates even though the user has only written "piz" so far. A copy of the top candidate (topcamdidate) is shown in the application's edit or text entry control or window until a candidate has been selected. When a candidate is selected, it is sent to the application's edit control (possibly replacing the previously displayed top candidate). Selecting a candidate may insert a space into the application edit control after the recognized word such that subsequent user-entered words or characters are separated by previous characters or words.

In other UI embodiments of the stroke untangler, various control gestures are provided via one or more swipe gestures relative to the touch screen. Exemplary control gestures include deleting characters, deleting words, adding spaces between characters or words, switching drawing input modes, and the like. Note that the specific control gestures discussed throughout this document are not intended to limit the specific types of control gestures, the number of control gestures, or the swipes or multi-touches used to activate the control gestures available by various embodiments of the Stroke Untangler. The direction of the input.

1.1 System overview:

As mentioned above, the "Stroke Untangler" provides techniques for composing handwritten messages from multiple overlapping strokes without requiring predetermined gestures or strokes, and without using HWR-based techniques for untangling, segmentation, and compositing Multiple overlapping strokes to form the leading character. The above process is shown by the overall system diagram in FIG. 2 . In particular, the system diagram of FIG. 2 shows the interrelationships between program modules for implementing various embodiments of the Stroke Untangler, as described herein. Furthermore, while the system diagram of FIG. 2 shows a high-level view of various embodiments of a stroke untangler, FIG. 2 is not intended to provide an exhaustive or complete list of every possible embodiment of a stroke untangler. Graphically, as described throughout this document.

Additionally, it should be noted that any blocks represented by broken or dashed lines in FIG. 2 and the interconnections between the blocks represent alternative embodiments of the stroke untangler described herein, and as described below, any or all of these alternative Embodiments may be used in conjunction with other alternative embodiments described throughout this document.

Generally, as shown in FIG. 2 , the process enabled by the Stroke Untangler is performed by using a writing panel 200 (e.g., a touch screen, touch-sensitive surface, or other real or virtual writing surface capable of tracking a user's handwriting) to enter letters representing overlaps or Multiple handwritten strokes of part of the letter segment to start the operation. In various embodiments, the stroke display module 205 optionally presents a faded trace of each user stroke on the touchscreen immediately following the user's finger in contact with the touchscreen. This provides the user with visual feedback showing each stroke as it is being typed.

As each stroke is received, the control gesture module 210 evaluates the stroke to distinguish it from strokes representing a swipe or other multi-touch input representing a control gesture. Examples of control gestures include, but are not limited to: swipe gestures to delete the last stroke (e.g., swipe left, across the left edge of the writing pad), delete the last word (e.g., swipe up, across the top edge of the writing pad), Mark the end of a word (eg, swipe right, across the right edge of the writing panel), and toggle between text and drawing mode (eg, swipe down, across the bottom edge of the writing panel). Note that examples of control gestures are discussed in more detail in Section 2.2.2. Each stroke and any control gestures are then stored or cached in the stroke context module 215 .

The stroke context module 215 then passes the cached strokes and any associated control gestures to the stroke untangling and segmentation module 220 . In general, the stroke untangling and segmentation module 220 untangling, segmenting, and compositing overlapping strokes to form one or more characters, taking into account any control gestures. As mentioned above, these overlapping strokes are automatically untangled, segmented and synthesized without requiring pre-defined gestures or strokes and without using HWR-based techniques as a guide. Exemplary embodiments of segmentation algorithms used in this untangling and segmentation process are discussed in more detail below in Section 2.3. Furthermore, the segmentation algorithm is operated in conjunction with one or more trained segmentation models, as discussed in more detail later in Section 2.4 of this document. Note that segmentation is used to combine multiple strokes into individual characters. In other words, segmentation forms groups of strokes representing individual characters.

Once the strokes have been untangled, segmented, and synthesized by the stroke untangling and segmentation module 220, the character rendering module 225 will render the synthesized strokes to produce human-readable characters representing one or more letters, words, and/or phrases. Character display module 230 then displays a representation of the rendered characters on the touch screen used to enter the overlapping strokes, or on some other display device associated with writing panel 200 . It should also be noted that the rendered characters produced by the character rendering module 225 may be used for a variety of purposes, including but not limited to handwriting-based text editing applications where the rendered characters are passed to another user as a human-readable image of the text. A device or application.

Once the character display module 230 has displayed the rendered characters, the user can see the results of the untangling, segmentation, and compositing performed by the stroke untangler. Thus, if there are any errors in this untangling, segmentation and synthesis process, the user optionally corrects these errors via the character correction module 235 (which receives user input for correcting the errors). Note that correction of errors includes, but is not limited to: correcting compositional errors relative to untangled and segmented strokes (e.g., moving one or more strokes from one character to a previous or subsequent character), deleting characters, words or phrases, insert or add characters, words or phrases, etc. In various embodiments, the Stroke Untangler provides a "tap-to-correct" user interface (see Section 2.2.6), where the user simply taps, touches, or otherwise selects a displayed Incorrectly synthesized characters. Once selected, the stroke untangling and segmentation module 220 automatically corrects the composition of one or more characters taking into account information that one or more strokes of the selected character were incorrectly composed.

The character rendering module 225 also passes the human readable characters resulting from the synthesis process to the handwriting recognition module 240 . The handwriting recognition module 240 then processes these rendered characters using various HWR techniques to recognize letters, words, etc., and generate corresponding machine-readable text. Note that further discussion of the recognition of the resulting overlapping handwriting is provided later in Section 2.6.

The machine-readable text produced as output by handwriting recognition module 240 is then made available as text input in any of a variety of applications. For example, in various embodiments, the recognition candidates module 245 provides one or more auto-complete or recognition suggestions for user selection. The user selects one of these suggestions for providing the text corresponding to the selected suggestion as input to any application (search engine, messaging application, word processor, password entry window, etc.) that receives input from the output of handwriting recognition module 240 . For example, auto-completed suggestions or recognition candidates may include predictions, eg, suggesting the word "pizza" when the user has only handwritten characters corresponding to "piz". Additionally, recognition candidates may be derived from one or more language models or dictionaries based on partial or complete user input. Additionally, search candidates may be retrieved from various search engines based on partial or complete user input. Note that any or all of these suggestions or search candidates are used in conjunction with various embodiments of the Stroke Untangler.

In other embodiments, once the rendered characters have been recognized, the handwriting recognition module 240 passes samples of the corresponding rendered characters, words, and/or phrases to the learned character sample database 250 . By populating the database with samples of each user's own handwriting, these samples can be used for various purposes. For example, when one or more auto-completed suggestions are selected using the recognize candidates module 245, the character display module 230 may retrieve corresponding samples of the rendered characters from the learned character sample database 250 to update the display of the rendered characters to correspond to user selections. The result is that the automatically rendered letters, words and/or phrases are displayed by the character display module in the user's own handwriting. Thus, like the rendered characters generated by the character rendering module 225, the characters generated using the learned character sample database 250 can be used for a variety of purposes, including but not limited to handwriting-based text editing applications in which the rendered characters are used as A human-readable image of the text is passed to another device or application.

It should be understood that the discussion in the preceding paragraphs regarding individual user data in the learned character sample database 250 is not intended to limit the database to only the user's own handwriting. For example, a corpus of handwriting samples from multiple users is used for the initial training of the segmentation model described above. In various embodiments, the corpus may be augmented with online models with more data from other users or from specific users of the device. Alternatively, it is assumed that the learned character sample database 250 is used to assist in rendering disentangled characters in the user's own handwriting, and in cases where the database contains samples from other users, can also be used to utilize samples of other users' handwriting Generate character output.

Note that correction of any errors summarized above, or insertion or deletion of characters, words or phrases may be performed at any time. Any such corrections or changes will result in corresponding real-time changes in the display of characters rendered by character display module 230, corresponding real-time changes in the output of handwriting recognition module 240, and in the output of optional recognition candidates module 245. Corresponds to real-time changes.

2.0 Stroke Untangler operation details:

The various embodiments of the stroke untangler are implemented using the program modules described above. As mentioned above, the Stroke Untangler provides techniques for composing handwritten messages from multiple overlapping strokes without requiring pre-defined gestures or strokes, and without using HWR-based techniques for untangling, segmentation, and compositing Multiple overlapping strokes to form the leading character. Subsequent sections provide a detailed discussion of the operation of various embodiments of the Stroke Untangler, and an exemplary method of implementing the program modules described in Section 1 in connection with FIG. 1 . In particular, subsequent sections provide examples and operational details of various embodiments of the Stroke Untangler, including:

·Operation overview of Stroke Untangler;

Exemplary user interface for interacting with the Stroke Untangler;

Segmentation algorithms for untangling and synthesizing user strokes;

· Train the segmentation model for use by the segmentation algorithm;

Correction of stroke segmentation errors;

· Overlapping handwriting recognition; and

• Additional embodiments and considerations.

2.1 Operation overview:

As noted above, the stroke untangler-based process described herein provides various techniques for composing handwritten messages from multiple overlapping strokes without requiring pre-defined gestures or strokes, and without using HWR-based techniques as Guide to untangling, splitting, and compositing multiple overlapping strokes to form characters. Once the untangling, segmentation, and synthesis processes have been completed, the corresponding human-readable characters are passed to the HWR process, which evaluates these characters to generate corresponding machine-readable text. The resulting machine-readable text is then made available to a variety of applications and devices.

The capabilities summarized above provide a variety of advantages. For example, as mentioned above, stroke segmentation is based on the trained segmentation model without using HWR-based techniques as a guide, significantly reducing the computational overhead. Furthermore, the method facilitates unusual words or expressive spellings, since dictionaries etc. are not used when untangling, segmenting and synthesizing strokes. Furthermore, the character or text recognition following the untangling, segmentation and synthesis process significantly reduces computational overhead compared to existing systems (which operate with text or handwriting recognition techniques), thereby facilitating the initial untangling and segmentation process. Additionally, the ability to continuously enter multiple overlapping strokes without waiting for a timeout or entering character separators increases the rate at which users can enter characters. Another advantage is that the ability to untangle, segment and combine overlapping strokes means that the Stroke Untangler is well suited for use with writing tablets of any size, including, for example, small wristwatches or ring-shaped ones. Touch screen based devices.

2.2 Example of user interface:

Stroke untanglers can be implemented in a variety of form factors including, but not limited to, smartphones, tablets, wrist-watch-type devices with or without touch screens, ring-based devices that can be worn on the user's fingers, or with real or virtual Any other device with a writing pad to track overlapping strokes. Given the wide range of devices that can implement a stroke untangler, there are many ways to present a user interface that employs some or all of the functionality described herein. Additionally, some of the different ways in which the user interface can be presented are related to the screen sizes available on the device. For example, a wristwatch-type device with a relatively small screen may display various components (or a different number of components) of the user interface differently than a larger tablet computer or smartphone. As such, it should be understood that the various user interface examples described throughout this document are provided for purposes of discussion and explanation only, and that the examples discussed herein are not intended to limit the user interface of the Stroke Untangler to the described of various forms.

Typically, the user interface used to implement the various features of the stroke untangler includes a writing panel for entering overlapping strokes and various visual feedback mechanisms. These visual feedback mechanisms allow the user to see and interact with the untangled, segmented, and synthesized human-readable character sets produced in response to user input strokes. Additionally, these visual feedback mechanisms allow users to see and interact with machine-readable text (including various combinations of recognition candidates, search suggestions, etc.) resulting from HWR-based processing of human-readable character sets. Both the set of human-readable characters and the corresponding machine-readable text are displayed in various combinations and used in a variety of ways.

Additionally, various control gestures are provided to implement various embodiments of the user interface. Examples of control gestures include, but are not limited to: adding spaces between words, deleting one or more characters, deleting one or more words, correcting untangling, segmentation, and synthesis errors, and the like. As discussed in greater detail herein, these control gestures include, in various embodiments, stroke-based control gestures. However, other control gestures may also be used in various embodiments of the Stroke Untangler user interface, including but not limited to: voice-based control gestures, touch-based real or virtual device buttons, device motion-based controls, multi- Controls with touch or combined gestures, and more.

2.2.1 Writing panel:

As mentioned above, the stroke untangler begins to operate by receiving input corresponding to one or more overlapping strokes. Accordingly, existing or contemplated techniques for tracking such strokes are operable for use by a stroke untangler. For example, such techniques include, but are not limited to, using a touch screen, touch-sensitive surface, drawing tablet, or other real or virtual surface that can track a finger (or stylus). For purposes of illustration and discussion, all such surfaces (real or virtual) will generally be referred to herein as writing pads.

Furthermore, since writing enabling implements (e.g., stylus, pen, etc.) can be tracked in a manner similar to that of a finger, the subsequent discussion will generally relate to creating strokes with fingers, with the understanding that the subsequent discussion applies equally to various methods of creating strokes. Any kind of technique or tool. It should also be noted that various techniques for tracking strokes with respect to arbitrary surfaces and even in-air are discussed in more detail below in Section 2.4

In general, the writing pad covers part or all of the real or virtual surface on which the user enters overlapping strokes. In operation, the user can use her usual writing fonts, use uppercase or lowercase letters, numbers, punctuation, special characters, etc., to write strokes on the writing panel. Depending on the size of the writing pad and how large the user's writing is, the writing pad may allow the user to write one or more adjacent characters, each character comprising multiple overlapping strokes. Users can place and space overlapping strokes of characters horizontally and/or vertically on the writing pad without limitation. In fact, in the case of a touch screen, overlapping strokes entered by the user may encroach on various user interfaces or input windows or other display areas of the touch screen. Note that examples of these situations are discussed above with respect to FIG. 1 .

2.2.2 Control Gestures:

As described above, any of a wide range of control gestures can be implemented for interacting with the Stroke Untangler's user interface. It was observed in the tested embodiments that, using relatively small computing resources, it was easy to distinguish a swipe gesture designed to cross the outer boundary of the writing panel from a stroke intended to utilize an input character. This allows Stroke Untangler to avoid accidental activation of various control gestures. Other embodiments, such as utilizing two-finger swipes or other multi-touch gestures or device motions to differentiate control gestures, are also provided in various implementations of the stroke untangler.

For example, in various embodiments, the Stroke Untangler provides a four-way control system based on swipes that continue across the physical edges of the touchscreen. In proven embodiments, these swipe or control gestures include a "backspace" gesture for deleting previous strokes or characters by swiping across the left edge of the writing panel. Similarly, a swipe across the right edge of the writing panel inserts a space (equivalent to tapping the current leading recognition candidate) for automatically starting a new character or word that is separate from the previously entered character or word. Swiping up across the top edge of the writing panel clears all ink and/or candidates from the screen. Finally, a downward swipe across the lower edge of the writing panel toggles the drawing mode to allow the user to enter an image via multiple strokes without compositing the strokes into individual characters.

2.2.3 Display screen:

The user interface of the stroke untangler includes a display screen, eg, a touch screen of the device implementing the stroke untangler, or some other screen in communication with the stroke untangler. Typically, the display screen displays one or more lines of handwritten text that are untangled, segmented, and synthesized based on overlapping strokes entered by the user. On every touch event (including stroke entry, stroke or character deletion via control gestures, entering a space between characters, etc.), update the rendering of the current stroke (the stroke being drawn on the writing panel) on the writing panel .

In other words, strokes are untangled, segmented and synthesized from one or more previous strokes to form a character, or segmented and used to start a new character, as a real-time process. This allows the user to have visual validation of her input as it is being made. As a result, the display screen shows the current state of the characters forming the user input evolving in real time as the user writes on the writing panel.

Note that as shown in FIG. 1 , in various embodiments, the synthesized characters (e.g., set of human-readable characters 140 ) are scaled (i.e., reduced in size) to fit within a virtual bounding box disposed within character input window 150 , so that these characters can be displayed on the screen in a manner that allows other elements of the user interface to be implemented. Note that in various embodiments, instead of scaling each character individually to fit into the above bounding box. In other words, in various embodiments, all strokes are scaled with the same scaling factor. Alternatively, the obtained human-readable characters are scaled to fit into a bounding box.

When laying out a segmented word (i.e., one or more segmented characters each comprising one or more overlapping strokes generated from one or more overlapping strokes) for display, the Stroke Untangler automatically separates the strokes from character to character Split or group, and align corresponding characters (or bounding boxes for each character) horizontally, separated by small spaces (eg, about one stroke width). Wider horizontal spaces may be inserted at the end of each word to improve the readability of human-readable character sets. Character positions are generally not adjusted vertically, however in various embodiments, various alignment or layout settings allow characters to be positioned in any desired manner (e.g., horizontally, vertically, diagonally, along any curve, increased or reduced intervals, etc.) are adjusted relative to each other.

In various embodiments, the stroke untangler maintains relative stroke positions within each character from the writing panel. In some cases, the assumptions behind this simple scheme do not hold, for example if the user drops the device halfway through a symbol and tries to end later. In this case, the Stroke Untangler automatically determines a more likely location for the stroke based on the trained segmentation model used in the untangling and segmentation process.

2.2.4 Visual feedback:

Visual feedback (rendering ink strokes) helps users check which part of the letter (or word) they are currently writing. In fact, however, writing characters on top of each other creates complications that users are not normally interested in viewing. In other words, rendering all strokes on the display and retaining them for a period of time would mean rendering multiple characters jumbled on top of each other. Accordingly, various embodiments of the Stroke Untangler user interface handle this issue in different ways, some or all of which may be selected or adjusted via a configuration menu or the like associated with the Stroke Untangler user interface.

For example, in one embodiment, as described above, strokes are simply not shown except for scaled versions of those strokes used to form the human-readable character set. In the case of a touch screen, in various embodiments, the Stroke Untangler provides optional visual feedback on the writing pad in the form of faded stroke traces that track the user's strokes across the touch screen. In other embodiments, overlapping strokes are displayed until the segmentation performed by the stroke untangler begins a new character, at which point the previous stroke is cleared and the new overlapping stroke is displayed by the stroke untangler. Again, any or all of these patterns may be selected and used in various implementations of the user interface.

As mentioned above, the Stroke Untangler utilizes various HWR techniques to process human-readable character sets created from overlapping strokes to create machine-readable text. Note that handwriting recognition techniques are well known to those skilled in the art and will not be discussed in detail here.

Regarding various user interface aspects of the HWR process, in various embodiments, the Stroke Untangler displays the current handwritten word or character adjacent to the recognition candidate (or at any other location on the display screen). An example of this embodiment is shown in Figure 3, along with several additional user interface elements. For example, in response to untangling and segmenting overlapping strokes of user input, touchscreen-based computing device 300 (in this case, presented as a bracelet- or wrist-based form factor) displays human-readable character set 310 (which in this case The following shows a rendering of the spelled word "bill"). As described above, the untangling, segmentation, synthesis, and rendering of these human-readable characters is performed in real-time as the user enters strokes.

Adjacent to the set of human-readable characters 310 is a set of one or more user-selectable HWR recognition candidates 320 . In the illustrated embodiment, the recognition candidates 320 are ordered from left to right from most likely to least likely candidates such that the most likely candidates are adjacent to the corresponding set of human-readable characters 310 . User selection of one of these recognition candidates 320 acts to inform the stroke untangler of the correct text, and also makes the corresponding machine-readable text available to one or more additional applications.

For example, in the user interface embodiment shown in Figure 3, the most likely recognition candidate 320 (in this case the text "bill" (or whatever other recognition candidate the user selected)) is also provided to the search engine's Input field 330 . As with many search engines, when input is provided to the input field 330 by the stroke untangler, but before the search is actually performed, a set or list of user-selectable search candidates 340 based on partial user input is provided. When the user selects any of these search candidates 340, or when the user selects the search button 350, etc., a search is performed, with the results displayed in place of the search candidates (or any other portion of the display screen). It should also be noted that FIG. 3 shows an example of a faded stroke trail 360 that tracks the user's current stroke input as it is untangled, segmented, synthesized, and rendered to create a human-readable character set 310. Stroke input.

In other words, the exemplary user interface of FIG. 3 shows the display of untangled and segmented user strokes as a contiguous set of one or more HWR recognition candidate human-readable characters. This user interface example also shows a search engine input window with search suggestions based on the top HWR recognition candidate or one of the user-selected HWR recognition candidates. Updates to the set of human readable characters occur in real-time, with corresponding real-time updates to HWR recognition candidates and corresponding real-time updates to search suggestions. It has been observed that the real-time visual feedback provided to the user as a result of these updates allows the user to quickly perform searches based on handwritten text entered by the user without using a real or virtual keyboard. It should also be noted that a similar user interface may be used to provide input to applications other than search engines (eg, messaging applications, word processors, or any other application that utilizes machine-readable text input).

One advantage of using these multi-level visual feedback of user-selectable recognition candidates and search suggestions is that even if there are errors in the untangling, segmentation and synthesis of strokes or in the HWR-based recognition of the resulting characters, it is possible to One of the candidates or suggestions corresponds to the intended user input. As such, this type of user interface has been observed to reduce overall handwriting input errors by providing the user with multiple options for selecting the correct text.

2.2.5 Drawing mode:

Various embodiments of the Stroke Untangler user interface allow users to add small drawings to text. Drawings differ from characters in that stroke segmentation is turned off during their compositing. To compose a drawing, the user brings the interface into drawing mode by using the aforementioned swipe control gesture, by toggling the drawing button, or by using any other desired control gesture.

In contrast to generating characters by performing untangling and segmentation processes from overlapping strokes, in painting mode, paintings are typically custom creations requiring potentially multiple strokes and a magnified view during compositing. As such, in various embodiments, the stroke untangler user interface provides a visual indication to alert the user that the system is in painting mode. For example, in various embodiments (eg, button 180 of FIG. 1 ), selecting or otherwise highlighting the drawing mode toggle button indicates that the system is in drawing mode, even if there are no strokes on the writing pad. Note that the delete gesture can also be used in drawing mode. To end the drawing, the user either toggles the drawing mode button, or makes an end-of-word gesture. When displaying a message, treat the drawing as a word, ie, separate it from adjacent words or drawings by white space. There is no limit to the number of drawings in a message.

In a proven embodiment of the stroke untangler, each stroke includes a bit indicating whether it was entered in drawing mode or text mode, so that subsequent editing of the drawing or text can be performed, and thus when performing For the recognition of human readable character sets, the HWR-based recognition process can skip the drawing. For example, suppose the user "backspaces" through multiple strokes, and eventually deletes the last stroke of the painting. At this point, the Stroke Untangler UI automatically switches to painting mode and displays all strokes of said painting on the writing panel, allowing the entire painting to be edited. Similarly, if the user "backspaces" through the entire drawing, the stroke untangler user interface reverts to text entry mode (ie, captures overlapping strokes of the user-entered character).

4 provides an illustration of the combined use of character and drawing input via another example of a stroke untangler user interface. In particular, as described above, in various embodiments, the stroke untangler user interface enables a user to enter a finger paint image with overlapping strokes used to enter a character. For example, device 400 on the left side of FIG. 4 shows the result of untangling and segmenting overlapping strokes 410 as a set of human-readable characters 420 appearing to spell the word "thequickbrownfox." Additionally, processing the set of human-readable characters 420 based on the HWR results in the display of corresponding machine-readable text 430, as shown. Note that in various embodiments, the set of human-readable characters 420 and the corresponding machine-readable text 430 are both user-selectable and user-editable.

Additionally, device 405 on the right side of FIG. 4 shows the result of adding a drawing 415 to a previously entered set of human-readable characters 420 . In this case, since the drawing 415 is subsequently added to the input of the human readable character set 420, the drawing is then scaled to fit the human readable character set and presented in combination with these characters (see element 425).

It should also be noted that in the example shown, the drawing is also presented in combination with corresponding machine-readable text (see element 435). Advantageously, in this case, the machine-readable text and drawing can be saved for later use or presented as input to any other application, eg a search engine, which then performs a joint search on the machine-readable text and images. Alternatively, an image-based search may be performed using the drawing 415 as input.

2.2.6 Tap to correct:

As with any system that interprets natural user input, there are cases where the output does not match the user's intent. In cases where the stroke untangler uses segmentation algorithms, this error causes characters to render fully or partially on top of each other, or break into pieces. Thus, in various embodiments, the stroke untangler user interface provides various techniques that allow a user to correct segmentation and synthesis errors of the human-readable character set produced by the untangling and segmentation process.

For example, suppose a user wants to write "clean", but the untangling process mistakenly splits the strokes to produce "dean". Then, when the user simply taps or otherwise selects the "d", the segmentation will automatically correct to separate the strokes forming the "d" character into separate "c" and "l" characters.

Figure 5 shows an example of a Stroke Untangler user interface that includes a tap-to-correct control gesture that enables automatic correction of incorrectly segmented strokes into corrected human-readable ones by compositing these strokes with corresponding automatic corrections collection of characters.

For example, as shown in FIG. 5, a plurality of overlapping strokes 500 entered on a touch screen 510 of the device are automatically untangled, segmented, and synthesized to form a set of human-readable characters 520, which appears to spell the word "hellowotd." Also adjacent to the human-readable character combination 520 is the corresponding result of the HWR-based processing, which results in the display of the machine-readable character 530 spelling "hellowotd". One interesting feature of this exemplary user interface is that search suggestion window 540 provides suggested search terms based on machine-readable characters 530 .

In this case, the search suggestions returned by the search engine are actually corrected for the intended user input. In particular, device 515 on the right side of FIG. 5 shows the result of a user selecting (e.g., the user "tapping" the displayed character) a human-readable character representing a character in the set of characters represented by the spelling "wotd." "t". In this example, the stroke untangler automatically re-segments the strokes, displays the resulting corrected human-readable characters 525 and corresponding machine-readable characters 535 on the display screen, and highlights the corrected segmentation and corresponding characters ( See element 555).

Note that the display of the highlighted corrected segmentation and corresponding characters (see element 555) may be presented for a predetermined period of time or until canceled by the user. Additionally, as shown in this example, in various embodiments, the corrected segmentation and corresponding characters may be presented in a larger view highlighted (see element 555) to allow these characters to be displayed in case the user wishes to further edit Presented to the user in a more easily editable format.

Note that in the example of Figure 5, when the user input is corrected to the original intent of "helloworld", the displayed search suggestion remains the same. This is not always the case for every segmentation error, but it is the case when a very high percentage of segmentation errors occur. In this way, even if errors occur in untangling, segmenting, and synthesizing strokes when forming a human-readable character set, the user can quickly input or select an intended input.

Note that exemplary techniques for correcting segmentation errors are discussed in detail in Section 2.5.

2.3 Segmentation algorithm:

Typically, the stroke untangler user interface allows the user to mark word boundaries with a space gesture or any other control gesture that marks the end of a word. Strokes within words are automatically segmented into characters using techniques described below in connection with the stroke segmentation presented in FIG. 6 (which are used to implement various embodiments of the stroke untangling and segmentation model 220 of FIG. 1 ).

In particular, FIG. 6 shows an exemplary stroke segmentation map 600 . The stroke segmentation graph 600 includes a plurality of nodes 610-650 (each node represents a corresponding segmentation model, see the segmentation model discussed in Section 2.4). In the example shown, the stroke segmentation map 600 can be used to represent handwritten characters consisting of 1-4 strokes, since this number of strokes is usually sufficient for most characters. However, it should be understood that any number of strokes can be split by increasing the number of nodes in the split graph. In any case, in the example of FIG. 6, an arbitrary path from node 4 (i.e., element 650) to node 0 (i.e., element 610) represents a stroke segmentation, where the score of such segmentation is the traversed edge The sum of weights (traversededgeweight) s _ik , and where the visited node marks the end of characters.

In an exemplary embodiment, the segmentation algorithm of Note Untangler is used to segment any word or character comprising N strokes. For each stroke i=1,...,N, four scores _sik ,k=1,...,4 can be calculated, where each score describes how likely it is that stroke i is the last stroke in a k-stroke character. Figure 6 depicts a segmentation graph where N=4 and includes N+1 nodes; one node for each stroke i=1,...,N plus an extra node 610 labeled 0, where the graph 600 includes nodes return to the directed edges of their (up to) four predecessors. An edge from node i to node ik corresponds to grouping strokes (i−k+1),...,i into a character and is weighted by the corresponding score _sik . A path from node N to node 0 represents the segmentation of N strokes into characters. Predicting the most likely split amounts to locating the path with the maximum sum of edge weights. Those skilled in the art will appreciate that this is a shortest path problem that can be efficiently solved using dynamic programming in O(N) time and space.

2.4 Segmentation model:

In general, a segmentation algorithm utilizes one of several trained segmentation models for segmenting strokes. These trained segmentation models are used by the segmentation algorithm to compute the character ending score as _sik = fk(v _i ), where v _i is the feature vector describing stroke _i and fk is the probability that the stroke is the last of the _k -stroke characters statistical model. In other words, these models determine when a character's stroke ends, and subsequent strokes are then segmented into the next character. The character-by-character segmentation of multiple overlapping strokes is then synthesized into a corresponding human-readable character, as discussed throughout this document.

In various embodiments, the feature vector _vi includes a plurality of values, including stroke input's bounding box location, dimension, area, aspect ratio, stroke length, multi-bin slope histogram, at stroke start and end the slope of , and so on. The feature vectors may also hold similar values in both directions (ie, previous and subsequent adjacent strokes) for temporally adjacent strokes. For example, feature vectors can hold similar values in both directions (i-2),...,( _i +2) for two temporally adjacent strokes, if such strokes exist (inputs lost in vi Can be set to some default value, such as -1, to indicate that no value exists for a particular input). Furthermore, bounding box features for stroke inputs may be employed to determine the union and intersection of adjacent bounding boxes for stroke inputs. Note that these bounding boxes for stroke input on a touch screen should not be confused with the optional bounding boxes used to scale human-readable characters rendered from the untangled and segmented strokes discussed above in Section 2.2.3.

In a proven embodiment of the stroke untangler, a boosted decision stump is employed to learn the end of the character segmentation model: For example, the sum of n step functions _hjk over the individual dimension of _vi . Note that in this validated embodiment, the model is trained discriminatively using the end of k-stroke character strokes in the training data as positive examples and the remaining strokes as negative examples. Additionally, it should be understood that other computer learning or modeling techniques may be used to construct the end of the character segmentation model used by the untangling algorithm, and the use of lifting decision stubs is described for explanatory purposes only.

2.4.1 Model training:

Training a segmentation model for the nodes of a segmentation graph can be achieved using a variety of techniques. For example, in the validated embodiment, manually input stroke data was recorded for known stages, the data was manually marked with character boundaries, and used to train four segmentation models (for the case of four node segmentation graphs). However, this type of manual data entry can be slow and less accurate unless large amounts of data are entered.

Thus, in a validated embodiment, utilizing a publicly available dataset of thousands of handwritten English characters observed in typical online messaging systems comprising a corpus of writers and most frequent words from a large population, Generate significantly more training data by automatically imitating strokes. Note that other languages and character sets, such as Chinese, Brahmi, etc., can be used to generate language-specific segmentation models.

To mimic words entered on the writing pad, random words are taken from the messaging system corpus. Then, for each character, retrieve a set of ink strokes from the handwriting dataset. The positions and sizes of the simulated strokes were randomly shuffled such that their mean and variance matched data from the manually created (but real) dataset discussed previously. The relative stroke positions within the character are not random, as it is assumed that the relative stroke positions are usually saved within the character when the user is entering strokes on the writing pad. These techniques are used to generate very large numbers of strokes for training segmentation models. In a validated embodiment, the model is trained with each of the n=256 stubs (ie, with the boosted decision stubs described above). Note that more or fewer stubs can be employed in this process, depending on the available training data and on the desired model fidelity.

2.5 Correct stroke segmentation errors:

As discussed above in connection with Figure 5, errors can sometimes occur when untangling and segmenting strokes, with the result that one or more strokes intended to be human-readable characters are rendered wholly or partially on top of each other, or in incorrect Way to break into pieces. Stroke Untangler offers various techniques for resolving this error.

In general, when a user indicates (eg, "tap to correct," or otherwise selects or specifies) that one or more characters are incorrectly displayed or rendered, the Stroke Untangler operates by determining one or more of the Automatically re-segment the selected character or characters by selecting the most likely alternative segmentation for the selected and adjacent strokes.

In particular, the stroke untangler identifies the closest sequence of one or more characters to a tap or selection location with respect to one or more characters that were not correctly displayed due to segmentation errors. The strokes associated with the characters of the word are then classified into three categories, including:

Category 1: Strokes to be segmented by changes. In a proven embodiment, strokes around 5mm from the user's tap location are placed into this category because the user is expected to actually tap or select something she sees as a segmentation error, but this distance will depend on depending on various factors, such as the displayed size of characters;

Category 2: strokes that may change segmentation. In general, these are one or more strokes that are in the immediate vicinity (in either direction, as above) of a user-selected character or stroke in Category 1 where the segmentation will be changed; and

Category 3: Strokes not in category 1 and category 2. The division of these strokes does not change until the user subsequently makes a selection, or unless they are next to a subsequently selected stroke.

Given the above classes of strokes that may or may not change (e.g., classes 1 and 2) segment, using the above segmentation algorithm, compute the most likely alternative segmentation for some threshold (e.g., 32) while keeping strokes in class 3 constant. This can be achieved by removing from the segmentation map (see Fig. 6) edges that would result in an alternative segmentation of strokes in category 3, and determining a score up to a threshold number of remaining paths. Note that, depending on the number of strokes being re-segmented, the possible total number of alternative paths may be less than a threshold number that places an upper limit on the number of paths considered.

In various embodiments, given the set of most likely alternative segmentations discussed above, the stroke untangler automatically selects a single most likely alternative segmentation, defined as a feasible path that satisfies the following criteria:

1. The selected alternative segmentation changes at least one stroke that falls into Category 1; and

2. The selected alternative segmentation has minimal impact on strokes that fall into category 2.

The above rules facilitate partial corrections, but also allow for larger-area changes when partial changes require changes over larger areas (ie, possible changes to adjacent note divisions). Furthermore, utilizing this rule enables the stroke untangler to automatically correct the segmentation in real time when the user taps or selects a character with a segmentation error. In various alternative embodiments, the stroke untangler instead presents a display corresponding to two of the plurality of most likely alternative segmentations for user selection.

2.6 Overlapping handwriting recognition:

As described throughout this document, the Stroke Untangler utilizes automatic untangling and segmentation of overlapping strokes to perform handwriting recognition of corresponding human-readable characters. In other words, the stroke untangler provides an overlapping HWR process that "untangles" the characters representing overlapping ink by first grouping or segmenting overlapping strokes into characters and then performing subsequent HWR processing on these human-readable characters. strokes.

In contrast to techniques that integrate untangling logic with handwriting recognition in a way that requires the use of various HWR processing techniques as an aid to perform segmentation, the Stroke Untangler implements untangling as a separate dedicated process without using HWR-based techniques as an aid or guide. Perform untangling in case of performing split. In other words, the preprocessing strokes only provide for grouping or segmentation of ink strokes into human readable components. The grouped or segmented characters are then sent to a standard (non-overlapping) HWR process, which operates to return machine-readable text corresponding to the human-readable characters.

There are several advantages to the stroke segmentation process performed by the Stroke Untangler. For example, untangling and segmentation is simpler than full identification, and thus is generally faster (and lower power) than existing HWR-based untangling and segmentation techniques. This makes it well suited for real-time feedback for presenting entire handwritten words on the display while the user is entering strokes. Furthermore, the stroke segmentation process described above can utilize any existing HWR system, which is generally a highly optimized process that cannot be easily modified to handle overlapping inputs.

2.6.1 Identify Candidates:

As described above, the stroke untangler provides various user-selectable word predictions and "semantic" recognition candidates in response to handwritten characters entered by the user. In particular, when the results from the HWR algorithm are fed to an application like a search algorithm, the latter can access a larger set of corrections or predictions than the HWR algorithm alone.

For example, if a user types "facebock" into a search engine, typically the search engine will respond with "facebook" as a search candidate. In this case, the Stroke Untangler can directly display the more detailed corrections and predictions anywhere on the candidate window or display screen as user-selectable input. Note that an example of this concept is described above in connection with FIG. 5 . In various embodiments, once the user has selected a particular recognition candidate, the input window is cleared and ready for the next character input.

Word prediction is another technique supported by Stroke Untangler. For example, in various embodiments, the stroke untangler uses a statistical language model or the like to display the most likely completion for the current word, eg, when a user writes "faceb", the candidate list will contain "facebook". This allows the user to complete word input with a single tap, resulting in a faster input speed.

Note that because the Stroke Untangler displays handwritten characters adjacent to or close to recognition candidates, prediction candidates, suggestion candidates, and/or correction candidates (or any combination of any or all of these) (i.e. generated human-readable characters), so the user can still access what she actually wrote by hand. This has been observed to help the user relate her input relative to the various identification candidates presented by the Stroke Untangler. This becomes increasingly useful when using more complex predictive algorithms or language models, and when the relationship between actual written input and candidates becomes more abstract. For example, in an extreme case, if the user writes "lowes", the Stroke Untangler may display "homedepot", since both relate to home improvement storage.

2.6.2 Out-of-vocabulary words or strings:

Often, users want to enter text that is not actual words. Examples include: passwords, unusual words not found in dictionaries (including words in another language), or expressive spellings (for example, "whoooah"), etc. Thus, in various embodiments of the stroke untangler, these types of out-of-vocabulary words or strings are allowed, via the HWR process, by initiating a recognition mode in which it is recommended to include pure untangling and synthesis as An option (eg, turn off spellcheck or word recognition for at least one option) to allow users to enter new words (eg, "wooooah") or passwords with a mix of special characters, numbers and letters (eg, "&4KrPqwS"), etc. . Stroke Untangler can also achieve similar results by using the HWR system to perform character-by-character recognition without providing spelling or word corrections or suggestions. Note that in some embodiments, such results may be represented as alternatives in the prediction list, rather than forcing the user to switch to a different recognition mode.

2.7 Additional Examples and Considerations:

As discussed above, stroke untanglers can be implemented in a wide range of form factors, for example, smartphones, tablet computers, wristwatch-type devices with touchscreens or touch-sensitive surfaces, or real or virtual writing devices with overlapping strokes that can be tracked Any other device on the panel. Furthermore, the machine-readable text produced as output following the HWR process is made available as text input for any of a variety of applications.

Thus, given the wide range of devices that can implement a Stroke Untangler, and the ability of a Stroke Untangler to deliver machine-readable text to any number of applications, there are multiple ways to present an application that employs some or all of the functionality described herein. User Interface. Furthermore, some of the different ways in which the user interface can be presented are related to the size of the display screen available on the device.

With these considerations in mind, several additional examples of some of the many embodiments and uses of the various features and capabilities of the Stroke Untangler are described below. It should be understood that these examples are not intended to limit the scope of any of the user interface embodiments or capabilities of the Stroke Untangler, and that such examples are provided for explanatory purposes only.

2.7.1 Automatically complete user handwriting samples:

As described throughout this document, a stroke untangler generates untangled and segmented characters and words in a human-readable format from overlapping strokes input by a user. As such, these human readable characters and words are the user's personal handwriting. Thus, after the HWR process, there is a direct mapping between individual characters and words of the corresponding machine-readable text of the user's personal handwriting.

Thus, in various embodiments, the stroke untangler constructs a database or library (eg, element 250 of FIG. 2 ) of actual samples of user handwriting and strokes corresponding to characters and words of machine-readable text. This database or library can then be used for a wide range of purposes.

For example, when selecting from various recognition candidates, search suggestions, or autocomplete suggestions, the Stroke Untangler can retrieve corresponding character samples from the user's library of character samples to update the display of rendered characters to correspond to the user's choose. The result is the display of letters, words and/or phrases automatically rendered in the user's personal handwriting by the character display module. Note that the Stroke Untangler is also allowed to correct misspellings in user-entered text while maintaining the user's personal handwriting for the corresponding human-readable characters.

Other exemplary uses include, but are not limited to: handwriting-based text editing applications, where a user types or enters text using any desired text (or speech) input technology, and corresponding character samples from a library of samples are then used as human-readable images of the text passed to another device or application. Note that corresponding machine-readable text may be communicated to other devices or applications along with human-readable images of such text. another example

2.7.2 Clarification of overall words:

Typically, the stroke untangler's segmentation algorithms and segmentation models are trained to support not just one character at a time, but two or more characters or even whole words. Thus, if the writing panel is large enough, or if the user can write small enough, two or more letter strokes or even whole words can be entered at a time, and the stroke untangler will operate to untangle, segment, synthesize and recognize these multiple characters as a combined operation.

2.7.3 Search engine integration:

As described throughout this document, the Stroke Untangler automatically unscrambles and synthesizes user input strokes to form characters representing letters, words or phrases. In various embodiments, text recognition is then performed on these characters and provided automatically as input to a search engine or the like. As with many search engines, in various embodiments, an autocomplete function is used to provide search suggestions based in part on user input.

For example, as shown in FIG. 1 , as described above, user strokes that are untangled to produce characters subsequently recognized as the text "pizza" are used as part of the search input for auto-completion purposes. In response, the search engine returns a number of search suggestions, which in the example of FIG. 1 include "pizzahut," "pizzahutmenu," and "pizzahutcoupons." Users can then simply select any of these autocomplete suggestions to perform a search. Alternatively, if the user does not want to accept either of the autocomplete suggestions, she can perform a search based only on the term "pizza", or simply continue typing additional strokes to provide additional character input (e.g., additional strokes to enter to represent the word "recipe" characters). It should also be noted that, as discussed in connection with FIG. 4, in various embodiments, when a user enters a drawing, the stroke untangler performs a search based on the image or performs a combined image and text search.

2.7.4 Overlapping strokes in the air or on any surface:

Various techniques for tracking finger movements are known to those skilled in the art, for example, Cameras, RF-based tracking systems, laser-based tracking systems, etc. Such techniques and devices are adapted in various embodiments to track a user's fingers (or the tip of a writing implement) while the user is making overlapping stroke motions in the air or on any surface (eg, a desktop computer or tabletop).

Furthermore, as mentioned above, in various embodiments, direct input on the touch screen is not required. Such strokes can be performed, for example, on the touchpad of a laptop computer, or on the back of a mobile device with a touch-sensitive surface. Furthermore, such strokes can be made on one device (eg, watch, touch-sensitive bracelet) while being sorted out (and presented to the user) on another device (eg, desktop computer, electronic whiteboard, etc.). Other examples include various The "Touch Cover" or "Touch Cover 2" input device associated with the device. Such devices include various combinations of capacitance-sensing, pressure-sensing, and proximity-sensing technologies. Such a device may be adapted (via various software or driver modifications) to receive strokes entered by tapping on top of or over one or more pressure-sensitive or proximity-sensitive keys, rather than just sensing on-screen or touchpad strokes on.

In other words, the UserStroke Untangler is able to track overlapping finger strokes against any real or virtual surface using various tracking techniques. As strokes are entered directly to a touch screen or other touch-sensitive surface, the User Stroke Untangler then automatically unscrambles and synthesizes these strokes to form letters, words, or phrases, using some or all of the various techniques described in this document. The resulting human-readable characters are then processed using various HWR techniques to provide machine-readable text for use in any desired application, as discussed throughout this document.

It should also be noted that in various embodiments, an inertial tracking device is worn on one or more fingers. The motion of these fingers is then tracked via an inertial tracking device as they move to create overlapping strokes in the air or on an arbitrary surface. The resulting motion is then treated by the stroke untangler as input overlapping strokes processed as discussed throughout this document.

2.7.5 Consider user handwriting samples for security:

As noted above, in various embodiments, the stroke untangler collects learned character samples of the user based on actual samples of the user's handwriting. Therefore, these samples can be used for a wide range of purposes, eg, handwriting auto-completion techniques discussed in Section 2.7.1. Another interesting use of these learned character samples is to consider whether the handwritten characters entered by the user are close enough to the examples of those characters stored in the learned character database. To this end, various similarity measures can be calculated, for example, existing image comparison or matching techniques. Then, where the handwritten characters closely match the stored character samples, the stroke untangler can accept those handwritten characters as valid image-based password input.

Furthermore, this processing can be used as a precursor to the processing of these characters by the HWR described above. For example, when entering passwords, etc. by untangling, segmenting, and compositing strokes, the corresponding machine-readable text will not be passed as a password entry attempt until the characters closely match the user's handwriting. The result is a two-level authorization process that requires entering only a password. In other words, even if the recognized text provided the correct password, the password would be rejected when entered in a different user's handwriting.

2.7.6 Stroke-based predictive keyboard:

In various embodiments, the stroke untangler utilizes techniques to process overlapping strokes for selectively populating a dynamic keyboard for the user's next input. In other words, as shown in FIG. 7, the dynamic keyboard feature of the Stroke Untangler UI automatically fills in adjacent positive input strokes based on the untangling, segmentation, synthesis and subsequent recognition of the resulting characters which are updated with each input stroke. A small subset of the optional keys at .

For example, FIG. 7 shows a wrist-wrist-type device 720 , worn in this example on the user's left hand 730 , which provides a writing surface for stroke input with the fingers of the user's right hand 710 . Furthermore, as shown in FIG. 7, paint strokes in the approximate shape of circle 700 will be recognized by the HWR processing the stroke as the beginning of "o", "p", "d", "g", etc. Accordingly, a dynamic keyboard including one or more user-selectable keys 740 provides the most likely match for the user-input stroke and is displayed relative to the user stroke 700 .

Then, as more strokes are entered, the set of keys comprising the dynamic keyboard may change (by letter, character, or overall number and/or size of keys displayed) until the user selects a key, or until HWR processing automatically determines that more than characters Some threshold of match likelihood. The machine-readable text resulting from user selection of one or more keys of the dynamic keyboard may be used for any desired purpose. It should also be noted that keys may be sized according to their likelihood of matching a user stroke, with more likely matches being displayed larger than less likely matches.

2.7.7 Remote HWR processing of human readable characters:

Typically, the above-described untangling and segmentation techniques operate in real-time using relatively small amounts of computing resources. However, HWR-based techniques generally require relatively more computing resources for subsequent processing of the resulting human-readable characters. Thus, in various embodiments, the stroke untangler uses a remote server or cloud-based system to perform HWR processing of human-readable characters sent from the user device. In other words, in various embodiments, HWR-based processing is performed by a remote service while untangling is performed locally.

In this case, an image of the human-readable character (or in various embodiments, a 2D trajectory or any other mathematical representation of a curve describing a stroke) is sent to the remote service in real-time as untangling occurs. The remote service then periodically sends back one or more recognition candidates derived from one or more dictionaries, language models, search engine-based search suggestions, and the like. In other words, a locally cleared ink display is provided in real time as the user enters strokes, while the remote HWR system provides periodic updates. In a proven embodiment, periodic HWR-based updates are returned to the user device at approximately 500 ms intervals, while partial stroke segmentation and display is presented to the user in real time.

An advantage of such embodiments is that they allow computationally expensive tasks (eg, processing character images based on HWR) to be performed remotely, while minimizing the computational burden and corresponding power expenditure on the user equipment. This is especially advantageous in the case of small devices, such as a watch-based form factor (see Figure 7), or some other small wearable or sticky touchscreen based device with limited computing power and limited power resources, but with communication capabilities to send and receive data.

2.7.8 Sketch-based auto-completion:

As noted above, in various embodiments, the Stroke Untangler includes a drawing mode that allows a user to enter a sketch without untangling strokes (see FIG. 4 ). HWR processes these paintings to return character combinations corresponding to common text-icon autocompletion scenarios. For example, if a user enters a sketch with a stroke representing a colon adjacent to a closing bracket character (ie, ":)") in various embodiments, the Stroke Untangler autocompletes with smiley ascii type characters. Also note that in text input mode, entering these same character strokes can also be used for auto-completion or return to a corresponding smiley face icon, etc. Similar character-based auto-completion or suggestions based on any desired relationship between characters and icons are supported by various embodiments of the Stroke Untangler.

Additionally, as described above, in various embodiments, the Stroke Untangler performs searches based on user-input drawings or sketches. In this case, this search is used to return one or more clip-type images or photos that can be entered automatically or by user choice.

2.7.9 Other form factors for stroke untangling:

As mentioned above, stroke untanglers can be implemented in a wide range of form factors, including but not limited to: smartphones, tablet computers, wrist-wrist-style devices with or without touch screens, ring-based devices that can be worn on the user's finger , or any other device with a real or virtual writing pad capable of tracking overlapping strokes. For example, FIG. 8 shows an example of a writing panel 800 having a touch screen or touch-sensitive surface on which a user may input overlapping strokes 810 for processing by a stroke untangler, as described herein.

2.7.10 Automatic kerning of handwritten characters:

In various embodiments, the stroke untangler uses one or both of negative kerning and positive kerning to adjust between one or more human-readable characters rendered on the display device while the user is entering strokes. interval.

In general, kerning is the process of adjusting the spacing between characters of a proportional font, usually to achieve a visually pleasing result. Kerning adjusts the space between individual character forms, while tracking (letter spacing) adjusts the spacing of character ranges consistently. In digital typography, kerning is usually applied as a number on a letter-pair by which the default character spacing should be increased or decreased: a positive value indicates an increase, a negative value indicates a decrease.

Negative kerning is often used to align capital letters, such as "T," "V," "W," and "Y," with some other capital letter, such as "A," Fits to some lowercase letters, for example, the combination "Ta", "Te" and "To". Negative kerning is also sometimes used to fit periods or commas close to various letters or combinations of letters. Examples of other combinations using negative kerning include, but are not limited to: "FA", "LT", and "LY", and letters such as "A", "L", and "h" followed by quotation marks. Note that sufficiently large negative kerning can also cause characters to partially overlap, for example, when an 'o' is placed very close to a 'T', the top of the 'T' can be directly above at least part of the letter 'o', as in The kerned word "To".

Instead, kerning is often used in conjunction with special characters and punctuation (for example, a lowercase "f" followed by a closing parenthesis or quotation mark) to increase the space between characters. Depending on the font, some small kerning can also be used to emphasize letters and for pairs, for example, "Bo", "Dw", and "TY".

In various embodiments, the stroke untangler uses any of a variety of kerning techniques, with or without scaling human-readable characters based on bounding boxes, to build characters, words, and / or sentences. It has been observed that the resulting kerned human readable characters have a more visually pleasing appearance relative to the non-kerned version of the same human readable characters.

2.7.11 Identification model of broken characters:

When the writing surface is relatively small, or when the user enters relatively large strokes, one or more overlapping strokes entered by the user may span the edge of the writing surface during input such that the finger or writing implement is no longer in contact with the writing surface Sensitive surface contact. In this case, when the stroke untangler determines that the stroke is not a control gesture, the stroke is inherently broken into two or more fragments, each fragment is interpreted by the stroke untangler as a separate stroke for untangling , segment and composite these strokes. The result of untangling, segmenting and compositing these strokes is to render partial or broken characters for HWR processing. Therefore, to improve handwriting recognition in this situation, in various embodiments, one or more HWR processing models are trained based on broken characters, wherein for training purposes, one or more of the upper, lower, left, and right edges of the character are Multiple eliminated.

For example, as shown in FIG. 9, part of the upper and lower parts of the handwritten letter "g" 900 is removed, as shown by the dashed lines at the upper and lower parts of the letter, and provided as the letter "g" when training the HWR model. Positive example. Then, when the user enters overlapping strokes 910 that span the upper and lower edges of writing panel 920 , the result is several broken overlapping strokes 930 . The stroke untangler then untangles, segments and synthesizes these strokes to produce ostensibly human readable characters 940 . When the human-readable characters 940 are then passed to the HWR processing module, which includes one or more models trained on broken characters (e.g., the handwritten letter "g" 900 with parts of its upper and lower parts removed) , the result is the extremely readable letter "g" 950.

2.7.12 Post-processing of strokes:

In various embodiments, each overlapping stroke is processed before or after the untangling, segmentation, and compositing processes to self-generate multiple visual effects. For example, the individual curves that form each stroke are filtered or modified using various image processing techniques to create effects including but not limited to:

· Each stroke is rendered to appear to be drawn with a pencil;

· Each stroke is rendered to appear to have been drawn with various types of pens;

Renders each stroke to appear to have been painted with various brush types and sizes;

· Render each stroke in one or more selected colors;

· Render each stroke with any of a variety of fill patterns;

Render each stroke with any of a variety of shading effects;

• Smooth each stroke using a spline-based representation of each stroke or any other desired smoothing or filtering technique;

Rendering of composite strokes or human-readable characters onto any desired background (e.g., parchment, image, etc.);

·etc.

Operation summary of 3.0 stroke sorting out:

The process described above in connection with FIGS. 1 to 9 , with further consideration of the detailed description provided above in Sections 1 and 2, is illustrated in the operational flow diagram of FIG. 10 . In particular, FIG. 10 provides an exemplary operational flow diagram summarizing some operations of various embodiments of a stroke untangler. Note that FIG. 10 is not intended to be an exhaustive representation of all of the various embodiments of the stroke untangler described herein, and the embodiments represented in FIG. 10 are provided for purposes of explanation only.

Furthermore, it should be understood that the blocks and interconnections between the blocks represented by broken or dashed lines in FIG. 10 represent alternative or alternative embodiments of the stroke untangler described herein, such alternative or Any or all of the alternative embodiments may be combined with other alternative embodiments described throughout this document.

In general, as shown in FIG. 10 , the Stroke Untangler begins by receiving (1000) a multi-touch gesture representing one or more characters, one or more control strokes, and/or one or more other multi-touch gestures. overlapping strokes (eg, 1005). In various embodiments, the stroke untangler optionally presents (1010) the faded traces of the strokes on the touch screen used to input the strokes. The stroke untangler then evaluates (1015) each stroke to determine if the stroke is a control gesture, e.g., delete last stroke gesture, delete last word gesture, mark end of word or insert space gesture, toggle drawing mode gesture, etc. .

The stroke untangler then uses (1020) the aforementioned segmentation algorithm and the trained segmentation pattern (1025), taking into account any control gestures or other multi-touch gestures and user corrections to untangle, segment and synthesize one or more strokes to form one or more characters. Once the segmentation process has been completed, the stroke untangler renders (1030) the synthesized strokes to produce one or more human-readable characters (1035), e.g. Window along with an optional drawing or sketch to provide real-time visual feedback to the user. As discussed above, in various embodiments, the stroke untangler also receives (1040) user input to correct untangling and synthesis errors in any character (eg, utilizing the aforementioned "tap to correct" process).

The Stroke Untangler then performs (1045) handwriting recognition (HWR) of the human readable characters (1035) and outputs the corresponding machine readable text. The Stroke Untangler then optionally provides (1050) an output of this recognized text, word or phrase as input to various applications such as search engines, messaging, word processing, password entry, gaming Wait. Finally, in various embodiments, the stroke untangler receives user input (1055) to select from one or more autocomplete suggestions, recognition candidates, and/or search suggestions.

4.0 Exemplary operating environment:

The stroke untangler described herein is operable in a variety of general purpose or special purpose computing system environments or configurations. Figure 11 shows a simplified example of a general-purpose computer system that can implement various embodiments and elements of a stroke untangler as described herein. It should be understood that any blocks represented by broken or dashed lines in FIG. 11 represent alternative embodiments of simplified computing devices, and that any or all of these alternative embodiments, as described below, may be substituted with others described throughout this document. Example combinations.

For example, FIG. 11 shows a generalized system diagram showing a simplified computing device 1100 . Examples of such devices that may operate with the Stroke Untangler include, but are not limited to: portable electronic devices, wearable computing devices, handheld computing devices, laptop or mobile computers, communication devices (e.g., cellular phones, smartphones, and PDS ), multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, microcomputers, audio or video media players, handheld remote control devices, and more. It should also be noted that, as described above, the Stroke Untangler can be implemented with any touch screen or touch-sensitive surface that communicates or otherwise couples with a wide range of electronic devices or objects.

To allow a device to implement a stroke untangler, computing device 1100 should have sufficient computing power and system memory to support basic computing operations. Additionally, computing device 1100 includes one or more sensors 1105 including, but not limited to, accelerometers, cameras, capacitive sensors, proximity sensors, microphones, multispectral sensors, and the like. Additionally, computing device 1100 may also include optional system firmware 1125 (or other firmware or processor-accessible memory or storage device) for implementing various embodiments of the stroke untangler.

As shown in FIG. 11 , the computing capabilities of computing device 1100 are typically represented by one or more processing units 1110 , and may also include one or more GPUs 1115 , either or both of which are in communication with system memory 1120 . Note that the processing unit 1110 of the computing device 1100 may be a special-purpose microprocessor, such as a DSP, VLIW, or other microcontroller, or may be a conventional CPU with one or more processing chips, including a multi-core CPU based Dedicated GPU cores.

Additionally, simplified computing device 1100 may also include other components, such as communication interface 1130 . The simplified computing device 1100 may also include one or more conventional computer input devices 1140, or a combination of such devices (e.g., touch screen, touch-sensitive surface, pointing device, keyboard, audio input device, voice or voice-based input and control devices, video input devices, tactile input devices, devices for receiving wired or wireless data transmissions, etc.). The simplified computing device 1100 may also include other optional components, such as one or more conventional computer output devices 1150 (e.g., a display device 1155, an audio output device, a video output device, a device for sending wired or wireless data transmissions, Wait). Note that typical communication interface 1130, input device 1140, output device 1150, and storage device 1160 for a general-purpose computer are known to those skilled in the art and will not be described in detail herein.

Simplified computing device 1100 may also include a variety of computer-readable media. Computer readable media can be any available media that can be accessed via storage device 1160 and includes removable (1170) and/or non-removable (1180) volatile and nonvolatile media for storing information , such as computer readable or computer executable instructions, data structures, program modules or other data. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage medium means a tangible computer or machine readable medium or storage device such as DVD, CD, floppy disk, tape drive, hard drive, optical drive, solid state memory device, RAM, ROM, EEPROM, flash memory or other memory technology, Magnetic tape cartridges, tapes, magnetic disk storage devices or other magnetic storage devices, or any other device that can be used to store desired information and can be accessed by one or more computing devices.

Storing information may also be accomplished by encoding one or more modulated data signals or carrier waves, or other transmission media or communication protocols, using any of a variety of the aforementioned communication media, and including any wired or wireless information delivery mechanism, such as Computer-readable or computer-executable instructions, data structures, program modules, etc. Note that the terms "modulated data signal" or "carrier" generally refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media includes wired media such as a wired network or direct-wire connection that carry one or more modulated data signals, and wireless media such as acoustic, RF, infrared, laser, and other wireless media for transmitting and/or receiving One or more modulated data signals or carriers. Any combination of the above should also be included within the scope of communication media.

Retention of information, such as a computer-readable Read or computer-executable instructions, data structures, program modules, etc. Note that the terms "modulated data signal" or "carrier" generally refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media includes wired media such as a wired network or direct-wire connection that carry one or more modulated data signals, and wireless media such as acoustic, RF, infrared, laser, and other wireless media for transmitting and/or receiving One or more modulated data signals or carriers. Combinations of any of the above should also be included within the scope of communication media.

In addition, the various components implementing the stroke untangler described herein may be stored, received, sent, or read from any desired combination of computer-executable instructions or other data instructions in the form of computer- or machine-readable media or the storage device's own communication media. Some or all of the software, programs and/or computer program products of the embodiments or a portion thereof.

Finally, the stroke untanglers described herein may also be further described in the general context of computer-executable instructions (eg, program modules) being executed by a computing device. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Embodiments described herein can also be practiced in distributed computing environments where tasks are performed by one or more remote processing devices or in the cloud with one or more devices linked through one or more communications networks. In a distributed computing environment, program modules may be located in both local and remote computer storage media including media storage devices. Furthermore, the foregoing instructions may be implemented in whole or in part as hardware logic circuitry, which may or may not include a processor.

The foregoing description of the Stroke Untangler has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precision disclosed. Many modifications and variations are possible in light of the above teachings. Furthermore, it should be noted that any or all of the aforementioned alternative embodiments may be used in any desired combination to form further hybrid embodiments of the stroke untangler. It is intended that the scope of the invention be limited not by this detailed description, but rather by the appended claims. Although the inventive subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the inventive subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims (10)

1. the computer inputted for receiving text realizes a process, including:

Utilizing computer to perform process action, described process action is used for:

Receive multiple overlapping strokes；

In to stroke or human readable character before arbitrary execution identification based on person's handwriting, automatically split and synthesize described overlapping strokes to form one or more human readable characters；And

After having formed one or more human readable character, perform the identification based on person's handwriting to described human readable character to generate corresponding machine-readable text.

Computer the most according to claim 1 realizes process, and wherein, described identification based on person's handwriting performs on the basis of character by character.

Computer the most according to claim 1 realizes process, and wherein, described identification based on person's handwriting performs on the basis of complete word.

Computer the most according to claim 1 realizes process, and wherein, when described stroke is transfused to, user interface shows the segmentation automatically to described overlapping strokes and the scaled expression of synthesis in real time.

Computer the most according to claim 1 realizes process, and wherein, when each stroke is transfused to, what user interface showed this stroke in real time fades out vestige.

Computer the most according to claim 1 realizes process, and wherein, user interface shows the one or more identification candidates that can select corresponding with described machine-readable text.

Computer the most according to claim 6 realizes process, and wherein, described identification candidate includes suggestion based on dictionary and the combination in any of suggestion based on search.

Computer the most according to claim 1 realizes process, and wherein, user interface includes painting mode, and described painting mode allows input by the most divided and synthesis to form the sketch that the stroke of human readable character is formed.

9. for receiving a system for text input, including:

Writing panel, for receiving the input of multiple overlapping strokes；

For before any one in stroke or human readable character is performed identification based on person's handwriting, split and synthesize described overlapping strokes to form the equipment of one or more human readable characters；And

For performing the identification based on person's handwriting to described human readable character to generate the equipment of corresponding machine-readable text.

10. wherein storage has a computer-readable medium for the computer executable instructions for receiving text input, and described instruction makes calculating equipment perform method, and described method includes:

Receive multiple overlapping strokes on the touchscreen；

In to stroke or human readable character before any identification based on person's handwriting of arbitrary execution, split and synthesize described overlapping strokes to form one or more human readable characters；And

Perform the identification based on person's handwriting to described human readable character to generate corresponding machine-readable text.