DE3752294T2 - Computer system with handwritten keyboardless data entry - Google Patents

Description

The present invention relates generally to a method and apparatus for keyboardless input for a computer, and in combination with a central processing unit, to a computer system with keyboardless input. The present invention may be particularly embodied in an information storage, manipulation and transmission device in which text, data, computer commands and functions may be entered by writing alphanumeric or other characters and symbols by hand with a pen-like stylus on an input/output (I/O) screen. In a preferred embodiment, the I/O screen includes a transparent touch screen mounted above a substantially flat output display. The present invention may be embodied in a preferred embodiment in a stand-alone computer system or in a peripheral device for a host computer.

There are now vast amounts of information and sophisticated application software available on conventional keyboard computers. We have recognized that the usefulness of this information and application software could be greatly increased if it were possible to enter text and data and edit application software by directly typing in the normal manner on a flat display. There is thus a need to extend the usefulness of computer technology to use by non-keyboard oriented individuals. One way to address this problem is to use a keyboardless input system, such as a touch screen. However, with conventional input devices of this type, accurate and detailed input within a real-time, high-resolution frame in a manner that is familiar and comfortable to the user is not easy.

One known type of keyboardless input system uses handwriting recognition to enter handwritten text characters, as described, for example, in U.S. Patent No. US-A-3,487,731 (Frank).

U.S. Patent No. 4,491,960 to Brown describes a handwritten symbol recognition system in which an array of pixels in the form of a raster line scan is converted into segment-oriented lists that are filtered and compressed to produce topological features that are then analyzed using a logic tree decision mechanism.

U.S. Patent No. 4,262,281 to Buckle et al. discloses a handwriting recognition device. The disclosed design is intended for use with a host computer and does not employ coincident display technology.

U.S. Patent No. 4,475,239 to Van Raamsdonk discloses a text editing device. The '239 patent prefers the use of paper as the medium for entering editing functions and requires a conventional keyboard for entering text.

Wang's U.S. Patent No. 4,521,909 shows a two-plane structure detection system. The system is intended for use with an optical instrument.

U.S. Patent No. 4,520,357 to Castlebeery et al. discloses an electroscopic information display and input system using a writing pen. The disclosed design does not claim to have the speed or accuracy that would allow recognition of handwritten characters.

US Patent No. 4,318,096 to Pepper teaches the use of a conductive pen. The '096 patent relates to graphic design and allows the change of line width and intensity by applying pressure to the pen, with the results displayed on a conventional CRT screen. U.S. Patent 4,055,726 to Turner and U.S. Patent No. 4,055,726 to Turner et al. disclose two methods for electronic position sensing through the use of a probe.

A prior art handwritten character recognition program is described in an article by P. G. Moulton et al entitled "Graphical input/output for the Character Recognition Program" published in Marconi Review, Vol. 34, No. 181, 1971, Great Baddow, UK, pages 113 to 120. The program is run by a computer coupled to a black and white cathode ray tube (CRT) display on which a bright rectangle (called a character matrix) is displayed. A user can use a bright pen to draw handwritten characters in the bright rectangle. When the drawing is complete, an "END" command is entered and the handwritten character is presented as data to the recognition program, which attempts to convert the data into an appropriate font symbol and displays the result on the display below the character matrix.

Various prior art user interface techniques are described in an article by D. Canfield Smith et al entitled "Designing the Star User Interface" published in BYTE, April 1992, by BYTE Publications Inc. The Star system was a multi-function system that combined document creation, data processing and electronic filing, mailing and printing. It was controlled with a mouse for selecting text and for creating, scrolling and deleting windows depending on the position of a mouse-driven cursor on a display when a button on the mouse was pressed. For the A keyboard was used to enter data.

The present invention is defined in the accompanying independent claims, to which reference should now be made. Advantageous features of the invention are set out in the dependent claims.

In a preferred embodiment, the invention, which is described in more detail below with reference to the drawings, can be embodied in a unique keyboardless computer system having the ability to recognize and display handwritten symbols or graphic characters and to cause the computer to display font symbols or graphic symbols and to perform editing functions or commands according to editing symbols quickly, easily and at a reasonable cost.

This keyboardless computer system preferably includes a computer case with a flat display panel on which a user can "write" with a pen, and has the ability to recognize symbols handwritten on the panel with the pen and convert them into displayed font symbols and/or perform editing functions with editing symbols, all with a minimum of technical complexity for the user. Once the desired information and application software are loaded onto the keyboardless portable computer, the information and software can be used and manipulated without requiring any skill or knowledge of modern computers or other data sources.

The ease of use of the device's input technology advantageously increases the computer's usefulness for keyboard-oriented individuals.

Due to its portability, it can also be used in applications and environments where the use of portable keyboard computers is cumbersome, difficult or even impossible. For example, a variety of of blank, fully or partially completed forms stored in the portable computer memory. In a hospital, "sheets" of patient data can be stored in the portable computer memory, accessed by a nurse on rounds and relevant data such as blood pressure, temperature, etc. then manually entered using a pen. These corrected or enhanced forms can then be downloaded to a central computer memory.

In this preferred embodiment, the invention thus advantageously provides a portable computer system that is lightweight, reliable, accurate, inexpensive, and can be used while standing or walking.

One embodiment of the present invention includes a transparent input screen. As the user writes alphanumeric or other characters or symbols (for example, handwritten symbols or graphic characters as described below) on the input screen, each character or symbol is represented as a chain of dots emulating a written pen-on-paper input. When the discrete alphanumeric and other characters or symbols are complete, they are translated into computer text or computer commands or functions which, as the case may be, can be displayed on a display screen at a location preferably below the area on the input screen where they were entered and/or executed. The embodiment also includes a structure recognition algorithm which allows the translation of any written characters or symbols, such as ideograms and scientific symbols, into computer text. In a particular, presently preferred In one embodiment, a keyboardless computer is configured as an editing and display device, comprising a transparent touch screen and associated electronics placed over a display screen of 80 columns and 25 rows or larger; a pen for entering data; a microprocessor and storage means; artificial intelligence/pattern recognition software and editing software; and a battery power system; and other I/O means.

As used herein, the term "handwritten symbols" includes "graphic characters" or "handwritten characters" or the like, as well as any symbols that can be handwritten, drawn, sketched or written and have a communicative effect. For example, but not limited to, alphanumeric or other characters or symbols, numbers, letters, "kanji" (Japanese ideograms) or other language symbols, editing symbols, and technical, scientific, architectural and mathematical symbols are handwritten symbols. Other examples of handwritten symbols include freehand drawings or signatures or any other such written, drawn or sketched information that is uniquely configured by a particular scribe. "Handwritten symbols" may also include editing symbols and "computer commands" and functions (defined below).

As used herein, the term "font symbols" includes "graphic symbols" and means computer-generated symbols displayed in a predetermined font format. For example, but not limited to, alphanumeric symbols may be font symbols and may be displayed in numerous font formats. Japanese or Chinese "ideograms" may also be font symbols, as may technical, scientific, mathematical, architectural, or other such characters. Other examples of font symbols include any shape that can be stored and displayed by a computer, such as a drawing of a car or a house.

As used herein, the term "editing symbol" is a command symbol that is written as a symbol or graphic character (such as a caret, a horizontal line, a short vertical line, a long vertical line, etc.) and, when recognized, has the purpose of causing the computer to perform a specific editing function or command (defined below), such as inserting text (caret), deleting text (horizontal line), deleting a letter (short vertical line), or moving a margin (long vertical line), to name just a few representative examples.

"Editing Function" means any computer-generated text editing operation or command for changing text, such as, but not limited to, inserting text, deleting text, moving text, and substituting text. Some primary editing functions are listed near the end of the description.

A preferred embodiment of a keyboardless input computer system is described in more detail below. It includes a transparent input screen that generates position information when touched with a pen, and a display screen physically mounted below the input screen so that a displayed character is visible below the input screen. The system includes a computer programmed to compile the position information into strokes to calculate stroke characteristics and then compare the stroke characteristics with those stored in a database to to recognize the symbol drawn with the pen. Main features of the system are as follows: (A) transparent position sensing subsystem; (B) underlying display on which a plot of the sensed positions is presented and characters or symbols are shown; (C) means for converting sensed positions first to plotted points and then to recognized characters or symbols; and (D) means for "learning" to associate sensed input positions with a character or symbol. The apparatus may be housed in a portable unit measuring up to about 400 mm by 400 mm by 100 mm (16" x 16" x 4") and weighing up to about 7 kilograms (15 lbs) with a self-contained power supply.

The following is an overview of certain relevant technologies suitable for use in carrying out the invention.

A variety of positioning technologies can be used to meet the requirements of position sensing input. These requirements are essentially accuracy, resolution and speed. The technologies include: mechanical, electrostatic, electromagnetic, acoustic, optical and inertial. The desired application of this system is to be as close as possible to writing with a pen or pencil on paper. One problem is proximity: a pen only leaves a mark on paper when contact is actually made. Many of these technologies require an additional "pen contact" sensor, which is cumbersome to use with many commercially available pens. Another problem is writing angle: a pen leaves the same mark regardless of the writing angle. In many of these technologies, the position detector is offset from the pen tip, so the pen angle can cause erroneous displacements. Beyond these general problems, each technology has numerous advantages and disadvantages with respect to (1) the pen: size, weight, shape, and whether it needs to be powered and/or wired, and (2) the writing surface: transparency, smoothness, "grip," and whether or not physical contact is necessary (as opposed to pressure transmitted through overlying transparencies or paper).

The requirements for the position sensing input technology are accuracy (point to point), resolution (absolute position) and speed (points per unit time) to sufficiently define the written stroke for recognition analysis. For the recognition devices and methods currently in use, as described below, the current minimum requirements are: an accuracy of 0.125 mm (0.005 inches), a resolution of 0.375 mm (0.015 inches) and a speed of 150 points per second. At this accuracy, a 6 mm (1/4 inch) high writing line can be produced with more than 10 raw input points along a stroke of a small letter. The resolution provides positioning of the symbol to an accuracy within two pixels for a current display of 640 pixels on 225 mm (9 inches). The speed allows about 50 raw input points for a single letter written quickly (1/3 second).

Short description of the drawings

The preferred embodiment of the invention is described in more detail below by way of example with reference to the drawings. In the drawings:

Fig. 1 is a schematic system block diagram of an embodiment of the present invention;

Fig. 2 is a perspective view of the housing which houses the operating elements of the system;

Fig. 2A is an enlarged portion of Fig. 2 with parts removed to show the positional relationship between the touch input screen and the display screen;

Fig. 3 a schematic diagram of input screen, pen and associated electronics:

Fig. 4 is an overall schematic system block diagram of the apparatus of a keyboardless input computer system according to the present invention;

Fig. 5 is a schematic block diagram describing the movement of data within the system as it is modified by handwritten characters and commands;

Fig. 6 is an overall system block diagram describing the hierarchy of software used to operate the system;

Fig. 7 is a generalized block diagram of the character and structure recognition algorithm;

Fig. 8A and 8B together show a detailed block diagram of the structure detection algorithm;

Fig. 9 is a schematic block diagram of the stroke characterization subroutine;

Fig. 10 is a top view of a screen illustrating the "initialization" of the handwritten symbol database;

Figures 11A through 11I are a series of plan views of screens describing the operation of a text editing system;

Figures 12A through 12G are a series of plan views of screens describing the operation of a data entry system;

Fig. 13 a generalized block diagram of the Linus Editor.

Detailed description of the preferred embodiments:

In the figures where equal digits have equal 1, there is shown an overall block diagram of a portable keyboardless handwriting input computer system 10. The complete computer system is enclosed in a housing 12, graphically represented by the dashed line, and includes a conventional, general-purpose digital microcomputer 14, described in more detail below. Input information is entered into the microcomputer 14 by "writing" with pen 19 on the writing or input screen 18. The pen 16 (Fig. 2) is connected to the computer of the system 10 by line 17 (Fig. 2). As pen 16 is "writing" on the input screen 18, a plurality of locating signals representative of a plurality of corresponding position coordinates are transmitted to the microcomputer 14. The microcomputer 14 is programmed in accordance with a computer program described below to recognize the chain of locating signals and to store these signals in a computer memory. The programmed microcomputer 14 also outputs a corresponding plurality of display signals to a display screen 20. Both input screen 18 and display screen 20 are described in more detail below.

Fig. 2 shows a perspective view of the keyboardless computer system 10 incorporating the features of the present invention. The keyboardless computer system 10 is contained within the housing 12 which is a rectangular, closed box with a sloped surface 22 having a multi-line solid state display area 24. The input screen 18 is shown in Fig. 2A as being positioned above the display screen 20. In this example, the display screen 20 displays a plurality of horizontal lines 25 having the following indicia:

Surname

Address

Handwritten input is made above each line 25. The distance or space between two lines 25, denoted by 26, is used by the system to normalize all distances, and the lines 25 themselves serve as the reference axis or baseline.

Below the display area 24 on the surface 22 is a key input section 26 consisting of a plurality of "soft keys" 28. The soft keys 28 can be programmed by the operator for any purpose, such as entering computer commands. Commands for soft keys 28 include "save," "recall," and "delete." In addition, soft keys 28 can be used to switch between different programs or between modes (e.g., data entry mode and edit mode). However, soft keys 28 are optional and are used to supplement the input by handwritten entry. The pen 16 used to write input data and commands in the display area 24 is also used to activate the selected soft key 28. An ON-OFF switch 30 is located on the side of the housing 12 next to the soft keys 28. A data output or peripheral connector 31 is located on the upper right side of the housing 12.

The input screen 18 may be a conventional resistive touch screen in which a voltage is applied to the edges of the screen and a pen detects the voltage at the touched locations. The writing surface is made of a transparent material, usually glass, coated with a thin, uniform conductive layer (currently vacuum deposited indium tin oxide). Vertical bus bars or conductive strips (not shown) are placed along both sides are used to apply the reference voltage to determine the "X" coordinates of the pen position, and horizontal bus bars or conductive strips (not shown) are used along the bottom and top to apply the reference voltage to determine the "Y" coordinates of the pen position. In this embodiment, the pen 16 is merely an electrical probe which, when in direct contact with the conductive layer, detects the local voltage at the point of contact, which varies depending on the distance from the conductive strips or bus bars. With the zero point at the voltage application point, the X,Y coordinates are inversely proportional to the voltage applied. The pen 16 must make good contact to minimize additive resistance which would lower the detected voltage and thus add an erroneous distance increment. In a presently preferred embodiment, a soft graphite tip is used. The voltage is passed from the pen through a lead, such as lead 17 in Fig. 2, to an A/D converter for use in the calculations described below. The pen may be a charged "pen" as described herein, a light pen as commonly known in the art, or any other handheld device capable of displaying handwritten symbols on a screen.

An example of a conventional electrostatic screen is disclosed in the aforementioned patent 4,318,096 to Pepper. This resistive screen has the advantage of minimizing the interference caused by the user touching the screen with his hand.

The horizontal and vertical position detection is carried out by alternately switching the voltage applied to a conductive layer between the pairs of horizontal and vertical busbars through an interface and multiplexer controlled by a microcomputer or microcontroller. In a commercially available input or touch screen, the busbars are divided into a series of short strips with diodes to prevent the horizontal strips from shorting vertical strips and vice versa. This technique is used in a commercially available touch screen by Touch Technologies of Annapolis, Md. and Elographics of Oak Ridge, Tenn.

Referring to Fig. 3, an alternative embodiment of a novel low power position sensing input screen 33 will now be described in detail. Input screen 33 also serves to determine an X,Y position on an electrically resistive plate 34. A stylus 35, which includes a voltage source such as battery 36 or a voltage supplied to stylus 35 from an external source such as the system power supply, is used to touch screen 34 and apply a voltage to the touched position. When the touched position is charged by stylus 35 with a positive voltage with respect to a plurality of plate measurement points 37, the voltage at these points varies with distance from the stylus position, such as position X₁, Y₁ at 38. These voltages are measured sequentially in the X and Y directions by conventional means such as disclosed in the aforementioned prior art patents. In Fig. 3, these means include a conventional interface/multiplexer 42. A conventional A/D converter 43 converts the detected voltages into a digital signal. A microcontroller 44 receives the digital signal, performs standard checks to ensure that the numerical value of the signal is "valid" (e.g., within the possible range of voltages), and then converts the voltages in the into X and Y distances in the manner described herein. Microcontroller 44 is conventional but could be replaced by a system computer. Microcontroller 44 provides a digital signal representative of the X and Y distances to measurement point 38 to an output port 46. Port 46 may be a conventional RS 232 port. Alternatively, microcontroller 44 could translate point X₁, Y₁ to any other reference point, such as a point on baseline 25 (Fig. 2).

As long as the pen 35 does not make contact at position 38 or at any other position on plate 34, no current flows and power consumption is minimal. A sample measurement of the voltage at the measuring points can be made by using sawtooth voltage at the positioning point and measuring the time when the measuring point voltage exceeds a predetermined counter voltage.

There are several options for the input or touch screens 18 and 33: the resistive plate 34 or its equivalent for screen 18 may be transparent or translucent, and the position point may be made with a stylus or a user's finger or a connection point of an overlapping conductive screen (such as the commercially available touch screens from Touch Technology, Annapolis, MD.). The input screens 18 and 33 may be a physical solid surface that is transparent or translucent and may be made of glass or plastic such as Mylar. The surface may be coated with a conductive/resistive substance such as indium tin oxide. Other physical surfaces may use the transmission of sound or electromagnetic radiation from the touched position to a reference point(s), the distance being determined by the time delay or phase shift. Alternatively, the input screens 18 and 33 may have a ethereal or geometric surface defined by an electromagnetic, optical or acoustic field.

Position determination can be carried out by electrical contact closure through ohmic, capacitive or inductive coupling, remote detection through acoustic, electrical or magnetic fields or with light (UV, IR or microwave) scanning.

The low power position sensing input arrangement has the following advantages over other such displays: 1) minimal power requirements in standby mode, 2) elimination of distortions that are a result of opposing parallel busbars in conventional touch displays, and 3) using a ramp voltage eliminates the need for an A/D chip, which is a major cost factor in state-of-the-art touch display technology.

The coefficient of friction of the screen 18 is preferably chosen to be "rough" enough to provide some resistance to the movement of the pen 16 on the screen. If the screen were too smooth, the pen would slide too easily and would therefore be difficult to control.

Reference is now made to Figure 4 which is an overall system block diagram of the essential electronic circuitry used in the preferred embodiment of the present invention. Microcomputer 14 includes a microprocessor 50 which is coupled to a plurality of other electronic elements via data path or bus 52. A read only memory (ROM) 54 programmed with the operating and application programs and a battery powered random access memory (RAM) 56 are connected to bus 52 for bidirectional data flow. Microprocessor 50 may be a conventional single chip eight-bit or sixteen-bit The ROM 54 may be a device responsible for executing the fixed control programs stored in the ROM 54 and for receiving control programs from the other electronic elements and providing control signals to them over the bus 52. The microprocessor 50 may be the commercially available Type 280 (from Zilog Microcomputers of Cupertino, Calif.), Type 8088 (from Intel Corp. of Santa Clara, Calif.), or any other similar or more powerful microprocessor. The ROM 54 may be a Type 2564 or a Type 4764, both manufactured by Texas Instruments of Dallas, Texas. The storage capacity of the RAM 56 is determined in part by the size of the application programs, operating system, and database. As will be explained below, the RAM 56 may be a static SRAM or a dynamic DRAM. The primary requirements of the RAM 56 are that it have sufficient storage capacity and that it require a minimum of input power.

A battery 58, such as a lithium battery, provides power to make the memory of RAM 56 non-volatile for extended periods of time. A battery pack 60, which accommodates known rechargeable battery types, is used to provide the various voltage levels required by the other electronic elements of microcomputer 14.

Alternatively, the storage function of the RAM 56 may be performed by a non-volatile device that does not require energy to maintain storage, such as an electronically erasable or reprogrammable memory (EEPROM), or by devices that use magnetic bubbles or capacitance. State-of-the-art diskettes or tapes may also be used for mass storage. Suitable magnetic bubble storage devices include the types 7110 and 7114, each with a storage capacity of 1 Megabit and 4 Megabit, both manufactured by Intel Corp. It is also possible to use a single integrated circuit chip containing the microprocessor 50, at least a portion of the ROM 54 and at least a portion of the RAM 56.

Also connected to bus 52 is an EIA RS-232 serial interface 62 which provides a means for input and output of data. Data is provided to bus 52 (usually to RAM 56) by coupling an external data source to RS-232 port 62 directly to microprocessor 50 and other elements of microcomputer 14. Offloading of data from RAM 56 may also be performed by microprocessor 50 to an external computer, another data acquisition device, a mass data storage device (e.g., floppy and hard disk drives), or an electronic telecommunications system. Similarly, data may be transferred through port 62 to a printer (not shown) from interconnect bus 52.

Pen 16 is used to write on input screen 18 and to cause X,Y coordinate information to be generated by conventional electronic touch screen interface circuitry. The coordinate information is transmitted over bus 52 and is used by system 10 for control. Solid state display 20, which consists of a multi-line display - illustratively, 80 columns and 25 rows - is coupled to bus 52 through display interface 66. The basic requirements for the display are that it be substantially flat and sufficiently thin for use in the present invention. The display may be of any of the following types: scanning displays such as cathode ray tubes, projection displays such as rear view projectors, light emitting dot matrix displays (e.g. electroluminescence or plasma discharge) and light-blocking dot matrix displays (e.g., liquid crystal displays, solid state PLTZ or magneto-optical displays). In addition, it is preferred that the display be compatible with the input screen 18 in size, configuration and transparency and that both be energy-saving types.

The X,Y coordinates are entered into the keyboardless computer 14 via the input screen interface electronics 64 and transmitted via bus 52 to the microprocessor 50 which executes programs stored in ROM 54 and RAM 56.

The number of points (i.e., sets of X,Y coordinates) used to define each handwritten symbol and the speed at which points are identified are important for practical use. It is desirable to use at least about 4 points per millimeter (100 points per inch) and at least about 100 points per second to define handwritten symbols. Note that the more points identified per unit length, the greater the accuracy of the system in identifying handwritten symbols - however, as more points are identified, the speed of identification decreases and more computer memory is required. Consequently, a balance must be achieved based on the size (available memory and processing capability) of the computer system and the requirements for speed of response and accuracy. For most purposes, standard values in the range between about 4 dots per mm (100 dots per inch) and 100 dots per second and about 8 dots per mm (200 dots per inch) and 200 dots per second are suitable.

It should also be noted that the higher the precision of the system in identifying the X,Y coordinates of each point, the lower the number of points that need to be identified per unit length per second to accurately identify handwritten symbols. In contrast, the lower the precision, the more points are needed.

Dot resolution is needed to place dots where they are intended, for example to write an edit symbol exactly between two characters. Ideally, resolution to within a single display pixel is desirable. In operation, however, resolution to within two displayed pixels is sufficient for a display with 640 pixels in a 225 mm (nine inch) horizontal scan line.

When switch 30 (Fig. 2) is in the "on" position, the basic display mode is activated and the microcomputer 14 (Fig. 4) is programmed by the operating system, and a menu appears on the display screen 20 (Fig. 1). The menu presents various software options. A primary software function, editing, operates in a manner similar to conventional word processing software, except that handwritten characters, symbols and commands are interpreted by the system as if they had been entered via a conventional keyboard. The system is able to learn the editing symbols used by a particular typist for functions such as indenting, inserting, deleting, moving and reformatting and translates such symbols into digital command functions. When required, soft keys 28 (Fig. 2), activated by touching these areas on the input screen with the pen 16, function like conventional hard function keys on a computer keyboard.

The system is particularly designed for use as an interactive screen editor or word processor. After a writer has edited a document through (for example) touching the displayed name of an existing file with the pen or by writing the file name on the screen, all the usual editing functions can be performed by pen input. If the user wishes to change a displayed character or symbol he can simply write over the displayed character or symbol and, as described below, the pattern recognition algorithm will convert the written entry into computer text. For example, the editing software allows text to be removed by simply drawing a line through it and a conventional caret can be used to change the mode of operation to insert mode. In insert mode the display screen 20 provides additional space for the entry of handwritten characters or symbols which are inserted after the point in the text where a caret was written. Text can be moved easily by placing brackets or other user-defined separators around a displayed phrase or word and writing a caret or other user-defined symbol in the area of the text where the user wants that text material to appear. New margins can be set by dragging vertical lines along the side of the displayed text at the location where the new margins are to appear.

The basic editor software also allows new documents to be created simply by entering handwritten symbols on the screen. All documents can be saved, modified and transferred in the same way as these functions are performed on a conventional word processor, with the difference that these functions are performed using handwritten editing symbols on the (optional) screen or by touching the soft keys with the pen. The composite text produced and stored in this way can then be downloaded via RS 232 port 62 (Fig. 4) to another computer, a similar device, an external data acquisition device or a recording device, to a printer or via a telecommunications system.

In addition to these important modes of operation, a number of additional elements and features contribute to the usefulness of the system. A conventional alphanumeric keyboard (not shown), comprising a full set of keys, can be connected to a conventional keyboard interface (not shown) to support the entry of alphanumeric characters. An AC/DC connector can also be used in applications where portability is not necessary and where the performance requirements of display technologies such as plasma displays and electroluminescent displays must be met.

In actual use, the keyboardless computer can operate in any application or environment in which handwritten input converted to computer text is useful or necessary. To illustrate, the device can operate as a new generation of word processors or be used in such areas as sales, nursing, inventory control, surveys, claims settlement, to name just a few of its many uses, or as a general learning and testing device in education. Since the structure recognition software can learn and convert languages that are not composed of a small or limited set of alphanumeric characters (e.g., Japanese, Korean, and Chinese) into computer text, it is particularly useful for word processing and telecommunications in those languages.

In practice, it is particularly desirable to use a single computer screen to display any source form, font symbol, or other parameter to be edited, and to create a nearby "window" of white space in which to write, display, and identify handwritten symbols, and in which to display font symbols that match the handwritten symbols. In this way, the user can view the text being edited and the proposed insertion or change without much (if any) head and eye movement. This is illustrated in Figures 11A through 11D. This feature (proximity on a screen between text to be edited and the window in which the new text is to be handwritten) is very important for easy, fast, convenient use of the system.

In a preferred embodiment of this invention, the system "learns" the handwriting of a particular user prior to actual use. For example, when using the Latin alphabet, the twenty-six letters of the alphabet and the numbers 0 through 9 would be entered into the database. Punctuation marks such as periods, commas, question marks, colons, semicolons, hyphens, and the like could also be inserted. There is virtually no limit to the handwritten symbols that can be recognized and stored in the database. Of course, the computer must store an appropriate matrix of font symbols for converting the handwritten symbols. Various sets of font symbols could be created and stored in the computer's read-only memory, such as ROM chip 54. For example, when using the English language, a chip could store a (or more) font(s) of numbers and letters, appropriate punctuation marks, and appropriate mathematical symbols. Other chips might store font symbols for the Arabic, Cyrillic, or Greek alphabets, for Japanese, Chinese, or Korean "konji," symbols for use by architects or engineers, or chemical symbols (e.g., benzene rings, and the like).

Figure 10 shows one of a series of learning screens that prompt the user to enter the numbers 0 to 4. The computer attempts to compare the written numbers with the existing database (if one exists). If no match is found, because no database exists, or the match with an existing database is weak, then the character is added to the database. This learning process continues until all alphanumeric (or other) characters and symbols to be used have been entered into the database. The system is capable of storing multiple stroke characterization databases for systems used by more than one user. Having a unique stroke characterization database for each user has the additional advantage of making the angle of writing irrelevant. Consequently, the system is adaptable to all handwriting styles and can be used by both right-handed and left-handed people. A feature may desirably be included in the device to accommodate left-handed and right-handed users. This feature is a receptacle (not shown) for the pen connector on either side of the housing 12 so that the pen 16 can be connected on the left side for left-handed users and on the right side for right-handed users.

Fig. 10 also shows an example of the use of "Soft keys". In addition to the input line, a selection of soft keys appears. Each soft key corresponds to a function that can be performed by the system. To perform the function, the user simply touches the indicated point with the stylus. The soft key then appears in reverse and the selected function is performed. Soft keys have many advantages over traditional function keys. Here are some of the more important ones: the user no longer needs to remember which function key performs which function; function key label strips are no longer required; different soft keys can be made available (displayed and activated) at different points within a program.

Figures 11A through 11I demonstrate some of the simplifications in word processing made possible by the use of this invention. In Figure 11A, a normal screen of text is shown. The user of the keyboardless input system decides that additional information needs to be added and draws an insertion symbol (e.g., a caret) on the screen at the desired location. A data entry "window" appears (Figure 11B). The text is typed in as handwritten symbols (Figure 11C), matched (converted to font symbols) (Figure 11D), and finally inserted (Figure 11E). The operator changes his mind and draws a horizontal line through the new text material (Figure 11F). It is immediately deleted (Figure 11G). Next, the operator decides that a larger right margin would be more appropriate for the text. It draws a vertical line on the screen (Fig. 11H) and the border is automatically changed (Fig. 11I).

Fig. 13, described below, shows a generalized block diagram of the editing process. Figures 12A-12G illustrate how a blank form can be used for a hospital patient. The system user first calls up the correct blank form (Fig. 12A). This can be done, for example, by touching an appropriate soft key. The area where the information, in this case a pulse observation, is to be inserted is touched with the stylus (Fig. 12B). After the desired location is highlighted, a "window" appears directly below the space where the observation is to be entered (Fig. 12C). The desired insertion text is then handwritten in the window, as shown in Fig. 12C. As also shown in Fig. 12C, the window area does not cover the area of the form where the text is to be inserted. The nurse then touches the voting box with the stylus, which is highlighted (Fig. 12D). The software matches the handwritten entry with the appropriate font symbols and displays the result (Fig. 12E). In Fig. 12E, the result is displayed in reverse font. If the vote is successful, the "Insert" block is touched (Fig. 12F) and the new observation value is added to the patient's record (Fig. 12G). As shown in Figs. 12B, 12F and 12G, the area of the window in which the handwritten entry is made is larger than the area of the form occupied by the corresponding font symbols that are inserted into the form at the end of the insert operation. This feature is also shown in Figs. 10 and 11C, 11D and 11E.

This mechanism is of course applicable to a wide range of "blank forms" where data is inserted or corrected in a form. For example, it could be used to correct or update financial information in a spreadsheet program Other information may be recorded in the same way.

Using a black background and white letters for the newly entered font symbols makes it easier to check the accuracy of the characters entered. While this is a preferred, it is not a required feature, and a white background with black letters would also be acceptable.

The ability to create a window and enter data on the same screen, in physical proximity to the text being edited or the space for data to be entered, is an important feature of the device, as it makes the device easy and quick to use. The user's gaze can focus on the space into which the data is being inserted, and the ability to display handwritten symbols and the corresponding font symbols simultaneously means that errors, when the system "misreads" a handwritten symbol, can be easily detected and then quickly and easily corrected.

Referring first to Fig. 5, the overall operation and function of the pattern recognition software is described. When the operating system invokes the pattern recognition program, the program starts in terminal 75 where a series of variables and counters are initialized. The software then proceeds to decision diamond box 76 where the program determines whether the pen 16 (Fig. 2) is in contact with the input screen 18 (Fig. 2A). The system provides a "pen contact" signal as shown in processing box 78 as well as the X,Y coordinate voltages as locating signals as described above. The microcomputer 14 (Fig. 4) converts the X,Y coordinate locating signals into dash characteristics; the conversion may also be performed by a separate microcomputer such as microcontroller 44. When a pin contact signal is received, the software proceeds to processing box 80 where the individual location signals are combined into "dashes," with a dash being defined as the point location signals produced between a "pin contact" signal and a "no pin contact" signal.

The system then calculates a transform for each point, as described below, transforming the point coordinates from the X,Y Cartesian coordinate system to a relational coordinate system. Next, the software goes to processing box 82 where it compares the stroke to previously entered strokes stored in a database and determines whether the stroke is represented by a symbol in the database. If a match is found (if the font symbol represented by the strokes is recognized), as shown in decision diamond box 84, then microprocessor 50 (Fig. 4) causes the symbol to be sent to display screen 20 (Fig. 4) as shown in processing box 86. If no match is found, the microprocessor 50 (Fig. 4) displays a message as shown in processing box 88 requesting further pen input on the input screen 18 (Fig. 4) by flashing either an entry that is close to a match or a non-recognition symbol.

As previously mentioned, the software compares the stroke characteristics of each handwritten symbol with data entries previously stored in a database. In a preferred embodiment, the database is organized into sections of characters or symbols according to the number of strokes required for the character or symbol. In each section, the entries are initially arranged haphazardly, but after use the most frequently used entries "rise" to the top of the database as described herein. It should be noted that each user has their own particular style of writing a handwritten symbol and that each handwritten symbol may have a number of different variations.

For example, many people write the lowercase letter "h" with a single stroke. They place the pen on the writing pad at a point where they want to place the top of the letter and draw a vertical line down to the baseline; then, without lifting the pen from the paper, they go back up to the center of the previously drawn vertical line, then right and down to the baseline where the pen is lifted from the paper. In contrast, the same people may write the uppercase letter "H" with two strokes. They draw the left vertical line and the horizontal line as they did for the lowercase letter "h," then lift the pen from the pad and then draw the right vertical line. Appendix I lists the stroke data point data for these two letters as the data is stored in memory after being generated by an embodiment of the present invention.

As shown in Appendix I, the letter "h" as written by a user at a given time has a stroke (ns = I) of 20 points (np = 20) and the following x- and y-coordinate characteristics for the minimum, mean and maximum norm values (1/80th of a line width): -17 and -6, respectively; 0 and 18; and 19 and 60. The values in the first vertical column are the point-to-point slopes normalized to 360º/256. The values in the second vertical column are the average vertical Point-to-point positions above the baseline, normalized to 1/80 of the line width. A typical line width is about 1 mm (0.04 in).

A software hierarchy of programs is shown in Fig. 6. At the top is an operating system (see box 90) which oversees the overall operation of the computer system 10 (Fig. 1). The application programs shown in boxes 92 and 94, which are stored in RAM 56 (Fig. 4) and ROM 54 (Fig. 4), can be executed by the microprocessor 50 (Fig. 4) under control of the operating systems. When a handwritten character is required or indicated by an interrupt, the handwriting recognition software 96 is called. A first subroutine (see box 98) encodes the X,Y coordinates into strokes. The characteristics of the strokes are then defined by a subroutine 100, after which the strokes are compared with a database loaded from the ROM 54 (Fig. 4) into the RAM 56 (Fig. 4). The comparison is made via a subroutine 102. If the operating system is in "learn" mode, the database is updated with the new stroke data and symbols (see box 104). In a similar manner, a previously stored document can be edited with application program 92 using the edit function 94, which is called by the operator who enters the instructions using subroutines 98, 100 and 102 of the handwriting recognition program 92.

Referring to Fig. 7, the operating system 90 (Fig. 6) executes the handwriting character recognition software 96 (Fig. 6) by accepting as input the X,Y coordinate points shown in box 110 of the position of the pen 16 (Fig. 2) on the input screen 18 (Fig. 2) and encoding these points into strokes (see box 112). The program then characterizes the strokes with some descriptive set that takes into account, for example, the length, curvature, inclination and position of the stroke (see box 114). In box 116, the characterized stroke or sequence of strokes is then compared with those in the database and the best match is sought [*1]. If a sufficiently close match is found, the character in box 118 is identified and the database entry is swapped with the entry above it as shown in box 120. In this way, the most frequently identified characters "rise" to the top of the database and overall system performance, which is measured by the time required to find a match, is increased. If no match is found, the user can add to the bottom of the database as shown in box 122.

8 and 9, there is shown a flow chart of the computer program for recognizing a particular stroke sequence. The computer program starts at terminal 150 and proceeds to process the X,Y voltages from [*2] processing box 152, which voltages have been converted to a digital signal. The program then proceeds to decision box 154 where the program determines whether the stylus or pen 16 (Fig. 2) is not in contact with the input screen 18. This determination is made when both the X voltage and the Y voltage are zero. If the program determines that there is no pen contact, it determines that the stroke has been completed and the program proceeds to decision box 156. At decision box 156, the program determines whether there are fewer than three dots in the stroke; if so, the program proceeds to decision box 158. At decision box 158, the program determines whether there are zero dots in the stroke. If there are zero points in the stroke, the program loops back to the beginning of processing box 152 where another set of points is read. If the point counter (incremented in processing box 164) indicates that more than zero points are present, the program jumps to processing box 172. In processing box 172, the stroke is identified as a point (dot), and its height above the baseline (HABL) is calculated in processing box 173. From processing box 173, the program jumps to processing box 171.

However, if the pin contact signal is received, then the program jumps to processing box 160 where the voltages are scaled to determine the coordinate point using the following formulas:

X = a₁v₁ + b₁

y = a₂v₂ + b₂

The constants a₁ and b₁ are scaling parameters that are determined by calibrating the input area of the respective display.

Once the voltages are scaled, the program goes to decision diamond box 162 where the program determines if it is a bad point. It does this by comparing the distance between points and eliminating a point if the distance is too great (currently a scale is greater than 2.5 mm or 0.10 inches). On the other hand, a point is also eliminated if the points are too close together. Currently, points are thinned out if they are within about 0.4 mm (0.015 inches).

The comparison problem that exists for the first point is solved by determining if a point is the first point after pin contact is made and then using that point only to check the next point, which is accepted provided that that point is within the maximum distance of 2.5 mm (0.10 inches).

If the determined distance between points is outside the two criteria, the program drops the point and jumps back to the beginning of the processing box 152 to create another pair of to read coordinate point voltages.

If, on the other hand, the points are within the criteria, then the program proceeds to processing box 164 where a point counter is incremented to keep track of the score. This number is used in decision diamond box 156 as mentioned above. The program then proceeds to processing box 166 where the points are smoothed using any of a number of formulas. Smoothing is used to minimize digitizing noise, noise due to uneven hand movement, and electronic noise. The simplest smoothing technique is a multiple point average, which results in the following calculation of new points (xj', yj'):

xj' = xi/(n 2 - n 1 + 1)

And similarly for yj, smoothed over points n₁ - n₂.

Another simple method is called the running weighted average method, which uses the following formula:

xj' = αxj + (1 - α)xj' - 1

Alpha is a weighting constant that is usually positive (and less than one) and was used at 0.25. Summations were made on the basis of n2 minus n1 equals one. A third method involves a so-called spline fit, which uses the following formula:

xj' = (xj-1 + 4 xj + xj+1)

Any of the above methods can be applied either before or after filtering. Filtering is done to reduce the number of input points and to space data so that difference and/or angle calculations can be made within acceptable statistical error limits. found that a simple process of thinning a sequence of points by excluding the acceptance of subsequent points within a specified distance from previously accepted points is an effective filter.

From processing box 166, the program passes to processing box 168 where the point is stored in a matrix that is incremented for each new point since the last pin contact signal. Thus, an addressable matrix of points is created for each sequence of points obtained between a pin contact signal and a no-pin contact signal. This sequence of points is called a stroke. From processing box 168, the program loops back to the beginning of processing box 152 where another point is obtained until a no-pin contact signal terminates the stroke.

Decision diamond box 156 determines whether there are fewer than three dots in a stroke. If there are three or more dots in a stroke, the stroke is by definition a line and not a dot. If there are three or more dots in a stroke, the program jumps to subroutine box 170. In subroutine box 170, which is described in more detail below with reference to Fig. 9, the stroke is characterized by its slope and baseline height.

From the foregoing, it will be seen that the segmentation of the chain of coordinate points to form a stroke is based primarily on determining when the pen 16 is "up", ie not in contact with the surface of the input screen 18. Alternatively, a chain of points can be segmented to form strokes based on other considerations. For example, they can be segmented based on changes in a locally calculated curvature or on a large local curvature. The local curvature is determined from the Change in distance along the input coordinates divided by the change in slope. This gives the radius of curvature. If the radius of curvature changes rapidly with respect to the distance along the input coordinates, or if the radius is too small, then one segmentation stroke is considered to have ended and a new stroke is beginning. Other segmentation techniques may look at the relative maximum and minimum in one or both coordinates and/or the curve transitions in the coordinates. However, these latter two methods have been found to be less effective.

By characterizing a stroke, the sequence of coordinates defining the stroke or segment is reduced to a set of characteristics that are unique, generalized, and minimal.

Uniqueness refers to the two circumstances that the same characteristics are generated by the same coordinates and that the characteristics are sufficient to regenerate an approximation of the original coordinate sequence. The term "generalized" means that the characterization is invariant under such transformations such that the symbols are invariant (e.g., translation and scaling or stretching or slight skew). Scaling of all distances is achieved by taking a ratio of the distance to a write input line width.

The minimum set of segment characteristics has the following features:

(1) Stroke position: one or more of:

Center of gravity/average, extreme range, or start and end points, determined relative to the writing input line, previous strokes, or character perimeter or center;

(2) The line shape is characterized by one or more of the following parameters: average slope, change in slope (a measure of average curvature) and/or change in curvature, by the sequence of slopes over specific length segments or over fracture lengths, or by a rough description of linear direction or circular closure and opening direction;

(3) and the stroke length, which is characterized by the distance along the curve and/or the extreme region along the coordinate system.

In one embodiment of the present invention, positioning by centroid, extremum, and start and end coordinates has been successfully used. The stroke shape is encoded as a sequence of slopes and vertical positions (relative to the stroke centroid). The stroke length can be approximately calculated using the number of filtered points. Alternatively, the average curvature can be encoded in total slope change (together with length), change from start to end slope, or adjusting the slope angle versus length curve to the rate of change of the slope angle. Other characteristics that could be used include the location of relative coordinate extremes, curve transition, return points, and stroke direction. A specific method for determining the unique characteristics is explained below.

1. The numerical values of the criteria are determined for each stroke of the handwritten symbol.

2. The database values for each stroke of the previously learned handwritten symbol are determined and subtracted from the newly determined values.

3. The absolute values of each difference are scaled to make each of the five measurements somewhat equivalent to the others, like lengths scaled to height between lines.

4. The five values determined in this way are added together.

5. A predetermined threshold is used as a "goodness of mouth" test for recognition - if the value is too high, font symbols are rarely recognized, if the value is too low, font symbols are misidentified. Initially, thresholds of around 1000 are used, then adjusted to around 100 for improved recognition. If the threshold is exceeded, the comparison is discarded and an error message is generated and displayed.

6. The database is searched for a minimum numerical difference. If the minimum difference is below the acceptable detection threshold, then, as appropriate, the appropriate font icon is displayed on the screen or the command is executed.

It has also been found that the preferred classification of a stroke is continuous and not strongly separated. For example, it is desirable to determine a slope by angle in 256 directions rather than 8. Other non-continuous classifications may include bars/arcs/hooks, number and termination of return points, or horizontal or vertical strokes.

From subroutine 170, the program proceeds to processing box 171 where an individual stroke and one or more preceding strokes are compared to a database entry stored in RAM 56 (Fig. 4).

This comparison begins with three elimination questions that the program asks in the decision diamond boxes 174, 176 and 178. If the database entry is removed, the program always goes to a processing box 180, where the address of the next data entry is received and from where the program loops back to the beginning of the processing box 171. In the decision diamond box 174, the first elimination question asks if the number of strokes is different. If the number of strokes is the same, the program proceeds to decision diamond 176 where the average height above the baseline (HABL) is calculated and compared to the HABL of the data entry. The entry is removed if the difference in the average HABLs is greater than half the height of the entry line. If the question in decision diamond 176 is answered no, the program proceeds to decision diamond 178 where the number of points per stroke is compared, with the database entry being removed if the difference in scores is greater than ten. This determination is different from that in decision diamond 174 because it only concerns the number of points for each stroke. However, in decision diamond 174, certain letters such as the capital letters "E" and "A" have more than one stroke per letter.

If a data entry in the decision diamond box 178 is not removed, the program proceeds to the processing box 182 where the program calculates a benchmark to be used to determine the closeness of the match between the selected entry in the database and the drawn stroke. A currently preferred benchmark is the sum of the absolute values of the differences between the stroke values and the database entry values of:

a) distances or lengths in units of 1/80 of the line height (e.g. room 26, Fig. 2); and

b) Slopes in units of 1/256 of 360º over all points along the diagonal of the comparison matrix.

Alternatively, dynamic programming techniques can be used to optimize the comparison by Off-diagonal elements can also be used.

From processing box 182, the program proceeds to decision diamond 186 where a match is determined. In reality, a match is determined by applying an arbitrary benchmark (maximum allowable variance) which is the sum of absolute values of the differences between the characterization of the input stroke and that of the stored database entry. In processing box 183, the lower of the current benchmarks and the previous lower benchmark are stored as the best match. The program then proceeds to decision diamond 184 where it is determined whether the current entry is the last database entry. If not, the program jumps to processing box 180 where the next entry is selected. If it is the last entry, the program goes to the decision diamond box 185 [*3], where a match is determined on the basis that the benchmark is below a predetermined threshold. This threshold is set by the user based on experience with the system.

If no match is obtained, the program jumps to decision diamond 188 where it is determined whether all strokes have been checked. After checking the last stroke, the current stroke is compared in sequence with a previous stroke with all two-stroke entries. As with the comparison with all one-stroke dictionary entries, the best comparison match for all strokes entered corresponds to the recognized symbol or sequence of symbols.

However, if the last line has been read and still no match is found, the program proceeds to the processing box 190 where the user is asked the question displayed on the display screen 20 whether a new font symbol is to be added to the database. The user's response is used in the decision diamond box 192. Either the stroke sequence is added to the database in the processing box 194 and the program jumps back to the beginning of the processing box 152, or the program jumps directly to the beginning of the processing box 152.

If, however, a match is found in the decision diamond box 186, then the program moves to processing box 195 where the program rearranges the database by exchanging the serial location of the matched entry with the entry above it. The program then moves to processing box 196 where it sets the point counter and the increment counter to zero. Next, the program moves to processing box 198 where the matched and characterized stroke(s) is/are displayed by the computer as the identified font symbol. This display is located at the location where the entry was made on the input screen 18 (Fig. 2).

From processing box 198, the program passes to processing box 200, where the program can act on any command it has interpreted. An alternative characterization of the stroke uses the points themselves rather than length, circumference, curvature and position.

This stroke characterization will now be described in more detail with reference to Fig. 9. The stroke characterization subroutine 170 essentially performs a mathematical transformation of each point on a point-to-point basis to transform the points from a Cartesian X,Y coordinate system to one in which the coordinates are the normalized slope of each point and the normalized height of each point above the baseline (HABL).

In subroutine 170, the point-to-point slope is first calculated in processing box 220 and then the height of each point above the baseline in processing box 222. The slope and HABL of each point are then normalized to 1/256 of 2 pi and 1/80 of the width of the input line, respectively, in processing box 224. From processing box 224, the system proceeds to processing box 226 where the calculated normalized values for each point are stored in an addressable array. The subroutine then returns to the program via terminal 228.

When a comparison is made between each stroke and the stored values, the comparison is made using the normalized point slope and point height above the baseline. As previously mentioned, a match is determined using an arbitrary yardstick that is the sum of absolute values of the differences between the written stroke and the stored stroke or index stroke. The system learns by adding new strokes to the index database. As the database is populated, font symbols that are rarely used are replaced with new entries.

In a practical embodiment of the present invention, the algorithm successfully identified upper and lower case letters and numbers when they were written separately. For handwritten symbols that are written continuously, direct extrapolation would require sequentially searching a database for one-, two-, three-, etc. stroke symbols and finding the best match. After identifying a stroke match, a "new" letter is provisionally recognized, but the next few strokes are analyzed to see if they modify the previous symbol for a better match. For example, two strokes that are written as "one dash" and changed to the capital letter "H" if a crossbar was identified.

The system design shown in Figures 7 through 9 could easily be coded into almost any computer language by a person of general skill in the art of computer programming. The source code records for an application program making use of the disclosed invention are provided in Appendix 11. The software in Appendix 11 is written in Microsoft Basic, a common computer language available for virtually all microcomputers and operating systems. The program is a complete text editing demonstration showing the improvements over traditional word processing systems.

Program lines 2600 to 4000 contain the character recognition subroutine, which contains the software code necessary to obtain the X and Y coordinates. This program section corresponds to Fig. 8.

Program lines 2600 to 2699 form a subroutine designed to obtain the X and Y coordinates of a given point. This code corresponds to boxes 152, 154, 160, 162, 164, 166 and 168 of Fig. 8.

Program lines 3000 to 3339 form a point and line analysis and characterization routine corresponding to boxes 156, 158, 170, 172 and 173 of Fig. 8.

The program lines 3700 to 3790 form a subroutine intended for a comparison of the analyzed strokes with a stroke database. These program lines correspond to boxes 171 to 184 of Fig. 8.

Program lines 3810 to 3980 form a subroutine designed to learn a new character. This code corresponds to boxes 186, 188, 190, 192 and 194 of Fig. 8.

Program lines 3060 to 3273 form a subroutine intended for stroke characterization purposes. This section of the code corresponds to Fig. 9.

Program lines 3060 to 3095 are used to calculate point-to-point slopes and correspond to box 220.

Program lines 3056, 3241 and 3262 are used to calculate a height above the baseline (HABL) and correspond to box 222 of Fig. 9.

Program lines 3253, 3270-3273 are used to normalize point height and slope and correspond to box 224 of Fig. 9.

Program line 3253 is used to store the height above the baseline and corresponds to box 226 of Fig. 9.

The preceding program can be stored in the memory of a microprocessor-type microcomputer and requires about 25 kB of machine memory, so the application of the program does not require a large amount of expensive memory space and the execution of the program is relatively fast. If the program is written in a language other than Basic that requires less memory space, such as assembly language, the program can be made smaller.

Boxes 195 to 200 in Fig. 8 appear in logical places throughout the code.

A list of variables of the relevant code section is given in Annex III.

Fig. 13 shows a flow chart for the editing software ("Editor") demonstrated in Figs. 11A to 11I and described above. As soon as the editor is loaded into the system (box 229), the system takes control of the screen again. The system then goes in the normal manner described above to detect and display points (box 230), convert the points to strokes (box 231), characterize each stroke (box 232), and attempt to match the stroke(s) with the database (box 233). In processing box 234, the system sends each handwritten symbol to the editor to interpret and execute a command if necessary. At decision diamond box 235, the editor determines whether the handwritten symbol is an editing symbol or a font symbol. If it is determined that the character is an editing symbol, the editor proceeds to processing box 236 where it determines which editing symbol was entered and performs the editing function. If it is determined that the character is not an editing symbol, the alphanumeric character corresponding to the handwritten entry is displayed in processing box 237. In an alternative configuration of the editor, font symbols are only accepted when the editor is in "insert mode". This structure ensures that each font symbol is verified before it is added to a document.

The editor uses a selection of icons designed to make system editing similar to traditional pencil and paper editing, but far more efficient. These primary editing functions include, but are not limited to, the following:

DELETE symbol - "- ". A horizontal line drawn through one or more characters. The editor removes the underlying characters and reformats the text.

CHANGE MARGINS Symbol - "". A vertical line longer than the height of a line on the display. The editor changes the margin according to the specified position and reformats the text.

INSERT symbol - " ". A caret drawn at the point where text is to be inserted. The Editor displays an input writing line (Fig. 118) and, if recognized, inserts the input into the text.

MARK TEXT Symbols - "< " and "> ". A less-than and greater-than symbol drawn at the beginning and end of a text block, respectively. The marked text is displayed in reverse and special block functions can then be performed.

DELETE SELECTED TEXT - A delete symbol drawn in selected text deletes the selected text and performs reformatting.

MOVE SELECTED TEXT - An insertion symbol drawn anywhere within the text moves the selected text to the specified location, deletes it from its original position, and reformats the text.

REPLACE SELECTED TEXT - An insertion symbol drawn into the selected text displays an input line and replaces the selected text with the entered text.

The editing symbols described above can be changed to specific editing symbols preferred by the respective user, so that the editor can be customized and new users do not have to learn unfamiliar editing symbols.

Further modifications and improvements to the current system will be obvious to those skilled in the art. For example, the general characteristics of each font symbol could be extracted and organized into a synthetic symbol. The characteristics of the synthetic symbol could then be exaggerated to maximize their difference from all other synthetic symbols. This would a very compact, optimal database would be created. On the other hand, a database created with the preferred embodiment of the invention described, for example, usually results in two or three different characterizations for each symbol.

The system has numerous useful applications, which are almost unlimited. The most obvious applications are text editing and the filling and modification of forms. Some of the many other, perhaps not so obvious, applications are: writing in languages that use large numbers of symbols, such as Japanese or Chinese; writing in Arabic and similar languages, which consist of a limited number of complex symbols; writing chemical equations, including those involving organic compounds; writing musical notation (a "window" with five parallel lines can be provided for musical applications); writing symbols and codes for graphical data processing, including transfer of graphical data to a worksheet; in education, where given questions are presented on screens and the answers are written in longhand; in mathematics teaching, where numbers are manually inserted into equations and the equation analyzed to determine the result resulting from the use of those numbers; in CAD/CAM applications that involve symbols, geometric shapes and the like.

We have described a particularly preferred embodiment of a keyboardless computer on which ordinary computer functions are performed by writing in the normal manner with a pen-like pen on an input screen located directly above a flat panel display. The illustrated keyboardless computer is ideally designed for use by non-keyboard-oriented individuals, by keyboard-oriented individuals for whom the usefulness of the computer is increased, and configured in various environments and applications where keyboard input is cumbersome or not possible. Computer information and application software can be loaded into a portable device for later viewing, editing of text and data, and addition of new text and data in a normal handwriting mode. Thereafter, the user can transfer this computer text to another computer, similar device, external electronic storage device, hard copy printer, or through a telecommunications system. The computer is capable of recognizing handwritten symbols with a high degree of accuracy and of "learning" individual handwriting styles. The data and commands can be entered using a pen. ANNEX I ANNEX II

ANNEX III

A1 - Angle of inclination between point M and M1

A2 - previous angle

DA1 - Change in angle (A1-A2)

SEND - Flag for end of line

ZX - double-precision X-position summation according to arc length weighting

ZY - double-precision Y-position summation according to arc length weighting

ZA - double-precision angle summation according to arc length weighting

ZDA - double-precision change in angle summation

ZL - double precision length

NSP - Number of points in the line, numerator

NSTRN - Number of strokes

XX - Maximum X

XN - Minimum X

YX - Maximum Y

YN - Minimum Y

M1 - Pointer to previous point

M2 - (1) Minimum number of points for which an adjustment to 2it angles must be made

M3 - (2) Minimum number of points for which the change in Rinkel change must be calculated

NLONG - Counter (not in use)

M - Pointer to considered point

NP - Number of points

X - matrix of the X coordinate

Y - matrix of the Y coordinate

DX - Change at X

DY - Change in Y

ADX - Absolute value of DX

ADY - Absolute value of DY

DS - pseudo arc length ADX + ADY

AY - Calibration multiplier for vertical (Y) direction, e.g. AY ' O mean coordinate system true downward

ADA - running average change in angle

DA2 - previous change in angle

ASTRT - Matrix for starting angle of line

SZL - single-precision line length

SNO - matrix of the number of the 1st point for each line

LHTS - Height between lines divided by 5 in display coordinates used as a measure

SL - Matrix of line lengths sealed at height between lines

AEND - Matrix of angles at the end of the lines

SC - Matrix of angle changes for each stroke

AZA - Average angle through line

SX - Matrix of the average centroid X coordinate for each stroke

SY - Matrix of the average centroid Y coordinate for each stroke

SA - Matrix of the average angle (slope) coordinate for each stroke

SXX - Matrix of the maximum X coordinate for each stroke

SXN - Matrix of the minimum X coordinate for each stroke

SYN - Matrix of the minimum Y coordinate for each stroke

SYX - Matrix of the maximum Y coordinate for each stroke

DBUG - Diagnostic flag for printing

INPUT - String of input bytes to be rejected

XP, YP - input coordinates

XPA - Average 2 points in X direction

YPA - Average 2 points in Y direction

XPO - initial X-average at downward swing

YPO - initial Y-average at downward swing

I - Display image element corresponding to X coordinate input

J - Display image element corresponding to Y coordinate input

N - Counter for input points

MM - Counter for timing of upward swing

NU - Counter for timing of upward swing In addition to the foregoing, it will be apparent that aspects of the invention may also be embodied in the following embodiments, which initially include:

an input/output (I/O) device for sending input signals to an exploitation device representative of a graphic character generated by a user and for receiving output signals from the exploitation device and providing a visual representation of the output signals, said I/O device comprising:

(a) display means for providing a visual display of a graphic character in response to output signals provided by the exploitation device;

(b) an input screen for generating a series of input signals while a graphic character or symbol is being sketched by a user, said input screen comprising a translucent input screen having upper and lower surfaces, said lower surface being positioned relative to said display surface so that the visual display generated by said display means is visible from said upper surface of said input screen; and

(c) means for determining the identity of said graphic character and for sending output signals representative thereof to said display means.

Preferably, in such an I/O device, said display means comprises a display area on which the graphic character is generated; and

the lower surface of said input screen is located in close proximity above said display surface so that the visual display produced by said display means is visible from the top of said input screen.

In such an I/O device, it is also preferred that said input screen comprises a substantially flat plate, the surface of said input screen being the surface of said plate.

Preferably, such an I/O device may further comprise a pointing means operable by a user to cooperate with the surface of said input screen such that said input screen generates said series of input signals when said pointing means is in a predetermined position with respect to said disk surface and/or said input screen may generate an input signal when said pointing means is in contact with said input screen surface.

In addition, this I/O device can

(a) said plate has electrical characteristics such that the voltage at any point thereon varies in a known manner with distance from the location at which a voltage is applied thereto;

(b) said input screen further includes means for selectively, sequentially applying a voltage across one Cartesian axis of said plate and then across the other Cartesian axis of said plate; and

(c) said pointing means is elongated and has a tip at one end thereof and includes a voltage sensing means for providing a signal representative of the voltage present where said tip contacts said input screen plate.

The I/O device of this embodiment may also include the following features:

(a) said plate has electrical characteristics such that the voltage at each any point thereon varies in a known manner with the distance from the place where a voltage is applied to it;

(b) said input screen further includes means for selectively, sequentially sampling - at an initial point thereon - the voltage across one Cartesian axis of said plate and then across the other Cartesian axis of said plate; and

(c) said pointing means is elongated and has a tip at one end thereof and includes means for applying a known voltage to said tip; said sensing means detecting the voltages at said output point when said tip contacts said input screen plate and outputting a signal representative of the detected voltages.

In the I/O device of this embodiment, said display screen preferably has a display area on which graphic symbols can be generated and a key input area with which command signals for the processing device can be generated when contact is made with the corresponding area above on said input screen.

According to a further aspect, the invention can also be embodied by a handwriting character recognition device comprising:

(a) a display means having a display surface for providing a visual display of graphical characters in response to a plurality of display signals;

(b) a writing surface for receiving graphic characters written by hand by a user, the writing surface being in close proximity to said display means and consisting of a substantially translucent surface having associated position coordinates;

(c) a pointing means operable by a user in conjunction with said writing surface to generate writing signals in cooperation with said writing surface;

(d) position detecting means for converting said writing signals into location signals representative of the position coordinates of said pointing means with respect to said writing surface;

(e) processing means for receiving and storing said location signals as they are produced, and for generating said display signals so that said display means can graphically display the graphic character after it has been produced by said pointing means; said processing means comprising character recognition means for comparing said generated location signals representative of a graphic character with a plurality of stored database signals to identify said graphic character.

In the handwriting character recognition device, said writing surface preferably comprises a substantially flat plate which generates a signal when said pointing means is in contact therewith.

The embodiment may include: means for converting said position coordinates into relational coordinates and also means for dividing a chain of location signals into sections representative of a stroke; and means for calculating stroke characteristics which can be compared with stroke characteristics of the database signals.

The processing means of the handwriting character recognition device of this embodiment preferably comprises a means for updating said stored Database signals with the above generated location signals if no correct graphic character is identified.

A method according to the invention can be embodied by a method for recognizing a handwritten symbol comprising the following steps:

(a) generating a chain of point location signals by moving a pen around locations proximate to a writing surface which generates a signal representative of said location of said pen;

(b) dividing a chain of locating signals into sections representative of a line;

(c) Calculating line characteristics;

(d) comparing said calculated stroke characteristics with stroke characteristics previously stored in a database; and

(e) Determine the best comparison and indicate whether the best comparison is good enough for a match.

A further embodiment of the invention provides an apparatus for recognizing handwritten symbols and displaying handwritten symbols and font symbols on a screen, comprising:

(a) a visual display screen having graphical capability to display font symbols and execute predetermined commands;

(b) a hand-held means for writing or drawing hand-written symbols on or over the said screen;

(c) a means for displaying a true representation of the handwritten symbols on said screen as they are being created;

(d) digitizing means for detecting the position of said hand-held means and converting it into a series of electrical signals that define the position, size and shape of each handwritten symbol;

(e) means for comparing predetermined characteristics of each digitized handwritten symbol with a database of predetermined characteristics of font symbols to assess the identity of each font symbol;

(f) means for converting the handwritten symbol into a predetermined font symbol or command and for displaying the font symbol on the screen in close proximity to the area of the screen into which the handwritten symbol was originally entered or for executing the command, as the case may be.

This embodiment preferably includes means for displaying predetermined fonts or symbols in the form of text or forms on said screen so that the handwritten symbols can be used to enter information to complete a form or to edit predetermined text, and can preferably be embodied in a portable unit measuring up to about 400 mm by 400 mm by 100 mm, weighing up to about 7 kilograms, and is arranged to include a self-contained power supply.

The apparatus of the embodiment further preferably comprises means for displaying predetermined symbols in a font and command-representing symbols on said screen to enable handwritten symbols representing each predetermined symbol to be manually defined on the screen, and means for identifying the manually defined symbol with the corresponding predetermined symbol in said database.

Preferably, the apparatus of the embodiment also comprises means for modifying said database if a handwritten symbol causes an incorrect font symbol to be displayed on the screen or an incorrect command to be executed, as the case may be, and/or means for creating a window of blank space on said screen in response to a predetermined command for the input of handwritten symbols into said window.

In the device of the embodiment, advantageously there may be areas on said screen which are touch sensitive and perform predetermined functions, and/or the screen may be substantially flat and designed to be used in a substantially horizontal position.

The device of the embodiment may also advantageously include soft keys on said screen and means for executing operational commands in response to touching any of said soft keys.

Another method aspect of the invention may be included in a microprocessor process for recognizing, converting and displaying handwritten symbols and executing commands, comprising the following steps:

(a) creating a personalized database for each individual user of the process by having the user first enter a handwritten symbol for each character corresponding to a font to be displayed or a command to be executed;

(b) determining a unique set of characteristics to characterize each of these symbols, and storing the set in the database;

(c) writing or drawing handwritten symbols on a computer screen using a pen;

(d) digitizing each handwritten symbol to identify the X,Y coordinates of a plurality of points that define the symbol;

(e) processing the digitized characteristics of each handwritten symbol to determine the predetermined characteristics of the symbol;

(f) searching the database for a "hit", a perfect or closest match to the characteristics;

(g) Displaying the font symbol or executing the command that is most closely associated with the "hit" characteristics.

This process may advantageously include the following additional steps:

(a) determining whether the font symbol displayed or the command executed is incorrect; and

(b) if an error exists, re-entering a handwritten symbol for the desired font symbol or command to modify the database, and/or it may include the following additional step:

(a) displaying the handwritten symbol on the screen substantially simultaneously as it is being written or drawn.

In the process of this embodiment, steps (b) and (c) may preferably include determining (i) the length, (11) the average slope, (iii) the centroid height above the baseline, (iv) the curvature (rate of change in slope), and (v) comparing the location of the centroid of each stroke with the centroid of the handwritten symbol.

Step (f) of the process may, if necessary, include:

(a) Determine the value of each of the five characteristics;

(b) subtracting the database value of the characteristic from the newly measured value;

(c) scaling the absolute values of each measurement to make each of the five measurements reasonably equivalent to the others;

(d) adding all five values;

(e) if a predetermined threshold is exceeded, discard the comparison and generate an error message; and

(f) Search the database for a numerical "hit" or, if this is not feasible, for the closest numerical match between the database value and the newly entered value.

In the process of design, the error rate is advantageously 5% or less and/or the program for step (f) occupies less than 6 kB of machine memory.

If necessary, said handwritten symbols in the inventive process include editing symbols, and said process includes recognizing the editing symbols drawn on the screen and executing the represented editing functions.

If this process further includes displaying text in font symbol format on the screen, the process may preferably include the following steps:

(a) creating a window on the screen near, but not covering, an area of the text to be edited;

(b) entering and displaying said handwritten symbols in said window(s);

(c) Display the font symbols corresponding to the handwritten symbols, in close proximity to the handwritten symbols mentioned.

The process according to the invention can advantageously also include the following steps:

(a) generating one or more soft keys on said screen to cause the computer to perform operational functions; and

(b) Touch one or more of the soft keys mentioned to perform the corresponding functions.

Furthermore, this process may include the step of determining the X,Y coordinates of about 4 to 8 points per mm and about 100 to 200 points per second in characterizing said handwritten symbols.

Claims (16)

1. A method for entering one or more symbols into a computer system, comprising the following steps: generating, in response to a predetermined command, a window in at least a portion of a display area (24) of a display screen (18, 20) for input of the one or more handwritten symbols; using a handheld means (16) to write or draw the one or more handwritten symbols in the window; Digitizing signals representative of the one or more handwritten symbols; Entering the digitized signals into the computer system (14). 2. The method of claim 1, further comprising displaying the image of the or each handwritten symbol(s) on the screen during writing. 3. The method of claim 1 or 2, wherein the one or more handwritten symbols comprise a number, a letter, a kanji symbol, a user-defined symbol, a command or editing symbol, a graphic image or drawing, or the like. 4. A method according to any one of claims 1 to 3, wherein at least one of the one or more symbols is identified as corresponding to a font symbol and the corresponding font symbol is entered into the computer system. 5. Method according to one of claims 1 to 4, in which at least two handwritten symbols are entered into the window. 6. A method according to any one of claims 1 to 5, wherein the predetermined command in response to which the window is generated is entered by a user using the handheld means as a handwritten symbol. 7. A device for recognizing a handwritten symbol, comprising: a display screen (18, 20); a handheld means (16) for use by a user for writing or drawing a handwritten symbol on or over at least a portion of the display screen(s); a digitizer (18, 64) for digitizing the handwritten symbol; a processor (14) coupled to the digitizer for determining the identity of the handwritten symbol; and means coupled to the display screen for, in response to a predetermined command, creating a window in at least a portion of a display area of the display screen for input of one or more handwritten symbols for recognition or otherwise. 8. Apparatus according to claim 7, comprising a display means (14) coupled to the digitizer and to the display screen for displaying on the display screen an image of the or all handwritten symbol(s) during writing. 9. Apparatus according to claim 7 or 8, wherein the window on the display screen is used as: (i) a window for input of one or more handwritten symbols during a first period of time after execution of the predetermined command; and (ii) a display area for displaying predetermined text during the other display period. 10. Device according to one of claims 7 to 9, in which the window is free of text and other information. 11. Device according to one of claims 7 to 10, in which the window can have different shapes and sizes. 12. Apparatus according to any one of claims 7 to 11, wherein the one or more handwritten symbols comprise a number, a letter, a kanji symbol, a user-defined symbol, a command or editing symbol, a graphic image or drawing, or the like. 13. Apparatus according to any one of claims 7 to 12, wherein a document to be edited is displayed by inserting a symbol on the display screen, the window is created near the portion of the displayed document where the symbol is to be inserted, and the symbol is entered by writing or drawing in the window by a user with the handheld means. 14. Device according to one of claims 7 to 13, in which at least two handwritten symbols are entered into the window area. 15. Apparatus according to any one of claims 7 to 14, wherein a soft key can be configured as an area in a display area of the display screen in which, when a user selects a point in the soft key area with the handheld means, a predetermined command is executed. 16. Apparatus according to any one of claims 7 to 15, wherein the predetermined command in response to which a window is generated is entered by a user as a handwritten symbol.