CN1310122C - Device for tracking the position of a writing implement - Google Patents

Abstract

An apparatus for tracking writing instrument position is disclosed, comprising: two or more sensors, the sensors comprising a plurality of pixels and being configured to receive sensed light from a writing instrument on the two-dimensional writing surface and to generate a sequence of signals representative of the sensed light for use in determining a position of the writing instrument on the two-dimensional writing surface, at least one of the sensors being characterized by an instability in data output by the plurality of pixels of the sensor; an optical device configured to enhance the light intensity of light received from the writing instrument; and a processor configured to receive the signals and determine a partial center of gravity output by a sub-array of the plurality of pixels, the sub-array centered on a pixel closest to a sub-pixel location of the light from the writing instrument, for increasing a stability of the calculated sub-pixel location of the light from the writing instrument.

Description

Device for tracking the position of a writing implement

This application claims U.S. Provisional Patent Application No. 60/161752, filed October 27, 1999, entitled Electronic Portable Penthat Captures Handwriting and Drawing with IR Sensing PSD Beacon System, and filed April 10, 2000, entitled Using Handwritten Information U.S. Provisional Application No. 60/195491 and U.S. Provisional Application No. 60/230,912, filed September 13, 2000, entitled Electronic Pen with a Keyboard Template, and filed August 18, 1999, entitled Electronic Portable The benefit of continuation-in-part of US Patent Application No. 09/376,837 for Pen Apparatus and Method is hereby incorporated by reference in its entirety.

technical field

The present invention relates to tracking the movement of a writing instrument.

Background technique

By tracking the movement of a pen, for example, as the pen is being used to write or draw on paper, it is possible to electronically capture and reproduce what is being written or drawn. It is also possible to track the movement of those nibs that leave no marks on a writing surface.

In some proposed approaches, the surface on which the pen is moved may have a pixel or other array of sensing locations, each of which responds when the pen is at that location.

In other techniques, this pen tracking is done entirely through electronics embedded in the pen tip. In some designs, a moving pen communicates with a static sensor independent of the pen and uses a triangulation algorithm to track that movement.

Contents of the invention

In general, in one aspect, the invention features a method comprising transmitting light from a moving writing implement as an indication of the writing implement's position and path on a two-dimensional writing surface; sensing the light with two or more sensors and generating a sequence of signals representative of the sensed light; and applying a technique for reducing the effect of changes in light intensity along a third direction relative to changes in the generated signal .

Implementations of the invention may include one or more of the following features. The technique may be based on optical properties programmed to enhance the uniformity of the sensor's signal response. The lens can be a spherical lens or an aspheric lens. The sensor can be an array of photosensitive pixel elements or an analog sensor. The technique may be based on algorithmic processing of the generated signal. The algorithmic processing may include linearization of the sensor signal response according to parameters associated with the writing instrument. This technique can be implemented in digital hardware or in analog circuitry. In addition to reducing the size effect, this algorithm technology can also reduce the influence of changes in light intensity. The signal may be grouped into frames, and the signal processing techniques may include multi-frame processing to remove noise. The light transmitted by the mobile writing instrument can be modulated at a frequency related to the rate at which signals are generated by the sensor, and the sensor signal can be shunted at the modulation frequency. Opposite gains may be applied to each of the shunted signals according to the on or off state of the light corresponding to the signal transmitted from the writing implement. Frame rate can vary. This demultiplexed signal can be integrated in time in the future. The light transmitted by the writing instrument may include a short pulse applied at the modulation frequency, a phase-locked loop determines the modulation frequency based on the signal from the sensor, and during the duration of the short pulse, the The sensor signal is sampled when the phase-locked loop is triggered. The characteristics of the transmitted light can be used to synchronize between the writing instrument and the sensor. The transmitted light may comprise low frequency modulation periods and high frequency modulation pulse trains, and the sensor signal associated with the high frequency pulse trains may be used to lock to a modulation clock.

In general, in another aspect, the invention features a method that includes transmitting light from a moving writing implement in a time-varying directional pattern, using two or more The light is sensed by sensors at two or more different positions of the writing implement spatially separated, and the position of the writing implement is determined by detecting a phase difference between signals measured by the two or more sensors.

According to an aspect of the present invention, there is provided a device for tracking the position of a writing instrument, comprising: two or more sensors, the sensors comprising a plurality of pixels, and configured to receive signals from writing on a two-dimensional writing surface Sensing light of the implement, and generating a sequence of signals representative of the sensed light, for determining the position of the writing implement on a two-dimensional writing surface, at least one sensor characterized by instability of data output by a plurality of pixels of the sensor the optics configured to enhance the light intensity of light received from the writing instrument; and the processor configured to receive the signal and determine a partial center of gravity output by the sub-array of pixels closest to The pixel centered at the sub-pixel position of the ray from the writing implement is used to increase the stability of the calculated sub-pixel position of the ray from the writing implement.

Implementations of the invention may include one or more of the following features. The time-varying directional pattern may comprise a rotational pattern relative to the X-Y plane on which the writing implement moves. The signal for light emitted in the direction of the positive X axis is in phase quadrature with the signal for light emitted in the direction of the Y longitudinal axis.

In general, in another aspect, the invention features apparatus including sensors configured to receive light from a writing implement being moved on an X-Y writing surface, the light having a direction perpendicular to the The intensity variation in the direction of the Z axis of the writing surface, and the optics, are configured to enhance the light intensity of the light received from the writing instrument.

Implementations of the invention may include one or more of the following features. The optical device may be a spherical lens or an aspheric lens. The optics may comprise a single spherical lens and the lens, together with the corresponding sensor, is configured to enhance the light intensity of light received at a larger angle or distance or at an unfavorable location of the writing instrument. The optics may include a special lens configured to enhance the light intensity of the light received from a location on the X-Y surface other than a predetermined location. The optics may include two cylindrical lenses, one close to the sensor for projecting light horizontally onto the sensor while the other is positioned to collect light in the direction of the Z axis and the other lens has a A lens curved shape. The algorithmic processing may enhance the immunity of the signal to changes in the intensity of the received light caused by the tilt of the writing instrument or the distance from the writing instrument. This process can determine the integral power of the overall signal distribution on the sensor and calculate the position of a sub-pixel based on half of the integral power position. The process may use a polynomial approximation to the signal distribution and calculate a subpixel position with the approximate maximum position. The calibration process can generate parameters for use in conjunction with the sensor characteristics.

The calibration parameters can correct for manufacturing imperfections of the optics and the sensor, and the algorithmic process can use an upright triangulation technique to determine a position of the writing instrument. The calibration parameters can correct for manufacturing imperfections of the optics and sensor, and the algorithmic process can use a polynomial progression to determine the writing instrument's position when coefficients in the polynomial are determined during the calibration process.

In general, in another aspect, the invention features a method comprising receiving light from a moving writing implement with an array of sensing elements of a sensor, sequentially reading the sensing elements, to generate a signal sequence indicative of light sensed by the array of elements, and to reset the element after each element is read and before at least some other elements in the array are read.

Implementations of the invention may include one or more of the following features. The array can include a CMOS or CCD position sensor. All elements are read before all elements are reset.

In general, in another aspect, the invention features a method that includes determining a sequence of three-dimensional positions of the moving writing implement from a signal.

In general, in another aspect, the invention features a writing implement combination including an elongated housing configured to be hand-held, a light source within the housing, and a A lens configured to receive light from the light source and transmit the light to a light sensor spatially separated from the writing implement via a vacuum path, the lens configured to be semi-reflective.

In general, in another aspect of the invention, the light source comprises an array of light sources arranged around an axis of the writing instrument and configured to emit light in a direction perpendicular to the axis.

Implementations of the invention may include one or more of the following features. The lens may be configured to internally reflect and concentrate the light and emit the light by reflection from a reflective outer surface of the lens. The lens may include a cylindrical body having an upper surface that receives the light and a bottom toroidal surface that reflects the light to the light sensor. The reflective outer surface may comprise a tapered surface at a defined angle of 45 degrees to the longitudinal axis of the writing instrument. The light source in the nib may comprise LEDs arranged in a circle.

In general, in another aspect, the invention features a device configured to turn the light source on and off in response to a user applying pressure from the writing implement to a writing surface, the light source being configured to , so that the amount of pressure required to trigger the device is not so great as to interrupt the normal writing movement of the writing instrument on the writing surface.

Implementations of the invention may include one or more of the following features. The writing implement may comprise a ballpoint pen holder having a writing point, and the device may be positioned at the end of the holder remote from the writing point. The device can be a switch or a pressure sensor.

In general, in another aspect, the invention features a holder having a receptacle for receiving at least a portion of the writing implement for storing the writing implement, the writing implement and the holder comprising respective elements , enabling wireless transmission of a signal associated with the movement of the writing instrument and tracking of the writing movement based on the signal.

In the practice of the invention, the holder may be a pen cap, and may include a clip configured to attach the holder to a stack of papers or a notebook. The holder may include at least two light sensors and a processor that processes signals from the light sensors to determine a sequence of positions of the writing implement. The holder may include a socket for holding the writing instrument and enabling recharging of the battery in the writing instrument.

In general, in another aspect, the invention features an element enabling wireless transmission of a signal associated with movement of the writing instrument and tracking of the writing movement based on the signal, the element being embedded in a cellular A phone, a PDA, a WEB personal digital assistant, or a tablet with paper clips.

In general, in another aspect, the invention features a holder having a mechanism for attaching the holder to a writing mat in an orientation enabling the element to communicate with the wireless transmission use. The clip mechanism may include a switch that activates a function of the processor in the holder when the clip mechanism is operated.

In general, in another aspect, the invention features a holder including a receptacle for the writing implement and an associated recharging circuit for recharging the writing implement while in the receptacle. The battery is recharged.

In general, in another aspect, the invention features a CMOS sensor adapted to receive light associated with movement of a writing implement and adapted to provide an indication of light received relative to a known direction. The angle of the signal, and a lens, are positioned so as to direct the received light toward the CMOS array.

In the practice of the present invention, the lens may be optimized to collect light from the region where the writing instrument's motion occurs. The lens can be a field lens or a Fresnel lens. The lens system may be configured to collect light in a direction perpendicular to the plane of motion of the writing instrument and project the light onto the sensor in a direction parallel to the plane of motion.

In general, in another aspect, the invention features calibration by positioning a writing implement at successive positions on a writing surface at which light received from the writing implement at the successive positions generates a signal at a sensor, And determining a calibration parameter for the writing implement for calibrating a process for determining a position of the writing implement as it moves.

In an implementation of the invention, the calibration parameters may include coefficients used in a polynomial series used as part of the position determination process.

In the practice of the present invention, the locations are not on a regular rectangular grid.

In general, in another aspect, the invention features (1) identifying a location on a writing surface corresponding to an input element to be entered into an electronic device, the writing surface being non-electronic and independent of The electronic device, (2) uses a writing implement to touch selected ones of the identified positions corresponding to the input element to be entered, and (3) senses the position at which the writing implement is clicked touch and input the corresponding data into the electronic device.

In an implementation of the invention, the writing surface comprises a sheet of paper and the input element comprises characters or commands of a language printed on the writing surface.

In general, in another aspect, the invention features moving a writing implement on a non-electronic writing surface to indicate a path, and remotely sensing the path and generating information for inputting the path into a A signal in an electronic device independent of the writing surface.

In general, in another aspect, the invention features modulating light at a predetermined frequency transmitted from a moving writing implement to a photosensitive element spatially separated from the writing implement, and using a A phase-locked loop associated with the sensor to lock the phase of the modulated light.

In general, in another aspect, the invention features circuitry for tracking the circuitry of a writing instrument by using wireless signal transmission between the writing instrument and a static element, the static element including a host processor and A separate pre-processor is coupled for at least performing data acquisition regarding the movement of the writing instrument, and the main processor is coupled for performing at least data communication regarding the tracking.

In the practice of the present invention, the pre-processor can also be connected to perform user interface functions and sub-pixel data storage, and the main processor can also be connected to perform background cancellation and sub-pixel calculations.

In general, in another aspect, the invention features a light reflective element configured to reflect light received from outside the writing implement to a sensor for tracking movement of the writing implement.

In the implementation of the present invention, the reflective element can reflect light to the sensor when the writing instrument is being used for writing, and prohibit the reflective element from reflecting light to the sensor when the writing instrument is not being used for writing. sensor.

In general, in another aspect, the invention features an array of photosensitive elements having a photosensitive pixel element that receives light from a writing implement that is moving and determines the light received from the writing implement in the array. The position where the ray has maximum intensity is determined with sub-pixel precision.

In an implementation of the invention, the sub-pixel position is determined by determining the integer pixel position proximate to the sub-pixel position, and finding a partial centroid of a sub-array centered at the integer pixel position.

In general, in another aspect, the invention features indicating locations on a non-electronic surface corresponding to inputs to an electronic device, and detecting the locations to input them into the electronic device.

In general, in another aspect, the invention features a clip for securing paper on which the writing implement is to be moved to the sensor.

In the practice of the invention, the mechanism may be part of a tablet or a notebook with means for gripping the paper, and the grip may include a mechanism to enable a user to grip or release the paper. The clip may include an activation button and a spring and a lever operated by the button. The lever can be configured to rotate in response to the button. The button can be configured to be pushed or pulled. Other advantages and features will be apparent from the following description and claims.

Description of drawings

Figure 1 illustrates pen tracking.

Figure 2 shows a pen.

Figure 3 shows the lens in a pen.

Figure 4 shows the lens in a pen.

Figures 5 and 6 show light reflections in a pen.

Figure 7 shows a tracking method.

Figure 8 shows a pen.

Figure 9 shows a pen.

Figure 10 shows a holder.

Figure 11 shows a lens in front of a sensor.

Figure 12 shows a holder.

Figure 13 shows a holder.

Figures 14 and 15 show one half of a holder.

Figure 16 shows the other half of the holder.

Figure 17 shows a field of view.

Fig. 18 shows a block circuit diagram.

Figure 19 shows a state diagram.

Fig. 20 shows a timing chart.

Fig. 21 shows a sequence diagram.

Figure 22 shows the tracked geometry.

Fig. 23 shows a circuit diagram.

Fig. 24 shows a sequence diagram.

Figures 25 and 26 illustrate a rotating beam technique.

Figure 27 shows a channel diagram.

Figure 28 shows a channel loop.

Figure 29 shows a paper keyboard.

Figure 30 shows a spherical lens.

Fig. 31 shows an aspheric lens.

Figure 32 shows a two-lens solution.

Figure 33 shows a clip.

Figure 34 shows a clip.

Figure 35 shows a sliding belt clip.

Figure 36 shows two views of a clip.

Figure 37 shows two views of a clip.

We describe an electronic wireless pen that, in addition to its normal function of leaving a visible trace on the writing surface, emits infrared (IR) light that is picked up by an external IR sensor to measure the pen relative to the sensor. s position. The sensor is a CMOS or CCD linear or two-dimensional array, position sensitive detector (PSD) or other photosensitive detector. The sensor can be clipped to the edge of the writing surface, providing a reproduction of the writing on those pages. The pen position is determined by mapping the sensor readings to the actual XY position of the pen tip on the paper.

The electronic input device looks like a conventional pen with a holder.

Users use it just like any ordinary pen to write on paper, notebook or any other flat surface. It is used to capture handwritten text or drawings. The pen stores all of its movements during its use by recording the sensor's measurements into its memory. The pen then downloads it to a computer, personal digital assistant, laptop or cellular phone. Then, when the handwriting appears on a page, it is automatically reproduced based on the sensor information.

As shown in FIG. 1, a pen or other writing implement 10 that leaves visible marks 12 of writing or drawing in a conventional manner on a piece of paper or other writing surface 14 may also have a source 16 that emits infrared light (IR) 18 for automatically tracking the motion of the pen. The light is detected by infrared sensors 20, 22 at a nearby location, for example near the edge 23 of the paper and held stationary relative to the pen.

The sensor emits a sequence of signals indicative of the position (eg, angle 24) of the pen on the writing surface at which light from the pen is received at each instant of successive measurement times. The circuitry associated with the sensor uses an algorithm for processing the orientation information (and the known distance 26 between the sensors) to determine the continuous position of the pen as it moves across the writing surface. The algorithm can use a mathematical model to convert the sensor's pixel signals to positions on the writing surface. The algorithm may be a quasi-triangulation algorithm using calibration parameters (the distance from the lens to the sensor and the horizontal offset between the centers of their refractive indices), or it may be a polynomial approximation.

The tracked motion of the pen can be used to recognize handwriting created with the pen or capture pictures created with the pen or for a variety of other applications. The tracked athletic information can be sent to a local PC or to a central computer via a personal digital assistant, a portable computer or a cellular phone for central storage and processing.

Track light sources with 2D or 1D sensors

The problem of tracking XY from a pen can be formalized as follows.

The pen carries a limited light source close to the tip of the pen. The source emits light rays whose intensity at the test point depends on the XYZ position of the test point relative to the source.

A multiplexer is located at another location. It captures some portion of the light emitted by the source on the stylus. The strength delivered to the different channels varies according to the XYZ position of the channel input relative to the position of the source. The intensity data is sufficient to calculate the XYZ position of the source relative to the detector. Intensity data is also affected by noise including source instabilities, detector noise, and other categories.

We are only interested in obtaining an XY plot from the pen. In practice, all three coordinates will vary due to thickness irregularities on the writing surface and changes in the slope of the pen during writing.

Along with noise, this will cause complex variations in the channel readout data.

There are different ways of processing such a signal: weighted average (centroid), median calculation, thresholding, etc. Most of them consider noise cancellation and also consider Z-motion of the source as noise.

Our goal is to establish properties of the detected signal that are invariant to Z-motion of the pen as well as certain noise sources.

For this purpose we introduce an aperture between a detector and the source. For example the aperture may comprise a lens. From this we obtain a spatially limited signal. This means that there is a set of closed detector channel devices simultaneously excited by signal and noise (partly in the case of a linear array detector). This group is surrounded by channels excited only by noise. In the absence of an aperture, all channels are excited by both signal and noise.

After creating such a signal we establish a certain point (eg maximum) and define a processing window around that point in a way that extends beyond the spatially clipped signal. A cumulative distribution function of the data inside the processing window is then calculated against the channel number. The projection of half-sized points of these functions onto the channel number yields this invariant property. When the channel number is an integer, the invariant property can be fractional.

There are basically two types of detectors, two-dimensional or one-dimensional. Each detector can be a two-channel detector such as a PSD, or a "multi-channel" detector such as a CMOS and CCD device. A PSD detector has two output signals whose ratio defines the relative position of the incident light spot. CMOS and CCD detectors have many pixels. Each pixel defines a specific space on the detector and has an analog output. These analog outputs may be digitized for subsequent processing in firmware or software, or may be processed by analog devices. Algorithms used in software can also be implemented in hardware by the same method.

a calibration process

It is not necessary to obtain a linear response of the detector to pen motion, as has been suggested in other known methods.

If simple triangulation is used to interpret a geophone readout as a pen XY plot position, a linear response will be required

There is a clear dependency between the device map position of a pen and the readout data of the left and right detectors (L and R), as follows:

X=Fx(L,R);

Y=Fy(L,R). ;

These functions can be expressed as polynomial series.

The coefficients in these polynomials can be determined during the calibration process.

During the calibration process, the pen is placed in different known XY positions on the paper, and the readout data of the two detectors is obtained and stored for future processing. After a sufficient number of points have been accumulated, the coefficients in (1) are calculated using ordinary linear algebra methods.

We know that system (1) is nonlinear in nature. It is important to position the calibration points in such a way that the necessary resolution can be obtained over the entire writing area. We do not locate calibration points at nodes in a regular rectangular grid. Instead, we use a mathematical model that matches a calibration grid to the particular nonlinear nature of our geophone.

Another Calibration Process

If the XY position of a pen is calculated using simple triangulation from the geophone data, we intentionally introduce an error into the geometry of our geophone in order to account for the non-linear nature discussed above.

By our design, we know exactly the refractive index of the lens in our detector and the distance between the lens and the sensor. Also it has been demonstrated that by varying these values in the triangulation calculation one can effectively compensate for the non-linear nature of the geophone.

To obtain the rms values for the index of refraction and the distance between the lens and the sensor, we run another calibration process. Calibration points at different XY plot locations are required for proper resolution on the writing area. The locations of the calibration points for these interrupted triangulations are obtained by a mathematical model.

Pen

As shown in Figure 2, in one example, the infrared source in the pen may be an LED 13 that emits infrared light 15 at the nib 17 when pressure is applied during writing. In this example, the LED source 13 consists of a ring 19 of LEDs arranged around the longitudinal axis 21 of the pen (only two LEDs are shown).

Light from the LEDs is projected down the nib and into the acrylic body/lens 18 that acts as a light guide. The acrylic lens scatters and transmits received light so that light emitted by the pen is transmitted along an optical path in all directions around the pen.

As shown in Figure 3, the tube 18 is polished and reflective to concentrate the light 502 from inside the LEDs 19 without allowing the light to escape in sideways directions. The bottom of the tube is polished with a 45 degree conical face 504 at the bottom of the tube. A reflective cylindrical housing 506 helps to confine and induce mixing of the light emitted by the LEDs. A cone 508 supports the light pipe. Direct light down in the tube is reflected by the tapered face 504 and transmitted into the air at all angles around the pen.

Figures 5 and 6 illustrate side and top views of the internal reflection of light rays in the light pipe. Most of the light from the LEDs travels the length of the pen's housing and is reflected toward the sensor at a 90 degree angle. Other rays find their way out of the pen at angles other than 90 degrees.

As shown in Figure 2, the light emitted by the pen is confined in a vertical space 11 close to the writing surface 13, so that as much light as possible can reach the sensor (not shown), which is also placed on the writing surface within a small distance.

Other configurations with different light pipe and lens shapes can be used, including the one shown in Figure 4, which allow for better coupling between the LED and the light pipe, and more efficient light splitting and direct the light onto the reflective surface located at the bottom of the tube.

The pen ( FIG. 2 ) in this example comprises a ballpoint pen barrel 23 terminating in a writing tip 25 . When the user applies pressure to the writing tip during writing, a pressure switch 26 sends out a signal which can be used to turn on the LEDs and which can also be used to trigger the function of an electrical circuit 28 mounted on the pen. Circuit 28 and LED 19 are powered by a battery 30. These elements are all kept in one housing 15 .

As shown in FIG. 7 , the position of the pen in an x-y coordinate system 40 parallel to the writing surface is determined by the two angles a and β sensed by the two sensors 20, 22 and the known distance 26 between the sensors. Sure.

In another example, as shown in Figure 8, the pen consists of three miniature AAAs stored on the back of the pen (for better weight distribution, the battery is moved closer to the tip and the circuitry is placed on the back). Nickel-cadmium rechargeable battery 51 supplies power.

The battery powers the electronics 28 directly without any DC to DC converter. The power is only delivered when the pressure switch 26 is activated. The light activated switch moves only a short distance (eg 0.25mm). The switch is preloaded by a spring mechanism to minimize cartridge travel, which should not exceed 0.008-0.010 inches.

A pressure sensor may be an effective way to match the pressure on the cartridge to the activation of the LED, as many popular switches have an activation pressure of 20 to 30 grams above the required level.

The electronics board 28 located behind the battery generates modulated pulses at about 100 Hz and 50% duty cycle for the infrared LEDs, and pen on and pen off signals at 1 to 10 kHz pulse trains for sleep mode.

The light emitted by the pen is visible in all directions, so that the pen can be used in any direction in the hand. The closer the emitted light is to the nib, the less errors due to pen-to-paper angle variations will be and the more accurate the nib will be tracked. The LED light should be in the infrared range away from the visible spectrum so that ambient light from the sun and lighting fixtures does not unduly interfere with the infrared emission and is also invisible to the human eye.

The infrared source of the pen and the sensor in the holder are oriented so that when the pen tilts and rotates during normal writing or drawing, its infrared beam can reach the sensor.

Figure 9 shows a more detailed isometric view of the partially assembled pen

pen holder

As shown in Figures 1 and 10, the sensor can be placed in a typical pen cap 70 in which the pen can be kept when not in use.

When the pen is in use, the pen is uncapped and the cap is placed in a fixed position near the writing surface and near the pen. In some examples, the sensor is a linear CMOS array (eg, a 1024 pixel array from Photo Vision Systems, LLC (PVS) (PO 509, Cortland, NY 13045) (part number LIS1024) can be used). Other linear CMOS sensors from PVS or other companies with the same or different number of pixels can also be used. The analog output of each sensor is a sequence of 1024 analog signals, one signal from each sensor pixel.

As shown in Figure 10, the holder may include a clip 62 for attaching the holder to the edge of a stack of papers or a notebook.

When writing starts (eg, the pen starts emitting light), the processor is woken up from a sleep mode (described below) using a third sensor in the form of a photodiode 56 located in the middle of the holder.

The third sensor signal can also be used to synchronize the circuitry in the sensor system with the circuitry in the pen. All three sensors are covered by infrared filter windows facing the writing surface.

As shown in FIG. 11 , in one example, the front surface 100 of each primary sensor has a vertical height 104 of 125 microns and a distance 106 from the front surface 108 of the lens 110 of four millimeters. The FOV 112 is 10 degrees. The stylus 114 directs infrared light into the FOV when the stylus is on the paper 116 .

As shown in Figure 17, the two sensors 88.90 are placed 100mm apart. Each sensor has a field of view (FOV) 94 , 96 centered on a FOV axis 195 , 197 . The axes of the FOV are not parallel but are tilted in at an angle 199 in order to increase the amount of overlap of the FOV. The FOV of each sensor has a width of 150° in the horizontal (xy) plane and a height of +/- 5° in the vertical plane.

The FOV does not cover certain locations 101 on the writing surface 15 close to the edge 303 of the paper, and the FOV is set so that the dead area 98 does not extend beyond the holder 25mm.

In another example of a pen holder, as shown in FIG. 12, the sensor 117 and the lens 119 are mounted on a support mount 121 having an infrared filter 123 on the front. The mount is mounted on a printed circuit board 125 and housed in a housing 127 . The centers of the two main sensors are separated by 100mm.

Light from the stylus is collected by two sensors to identify the linear position of a modulated light source within a defined range (8.5 x 11 inches). The linear position can be calculated using triangulation, lookup tables, polynomial approximations, or any combination of these methods.

The sensor is a planar, linear multi-pixel sensor. Different pixels of each sensor are illuminated when the light source is at different positions within the field. As the light source moves across the field, the corresponding motion of the light passing through the pixels of the sensor may not be linear, but this lack of linearity can be resolved because the linear position can be correlated mathematically and optically with knowledge from this is calculated by combining the calibration data of a pair of sensors.

Rather than looking for a linear response from the sensor, we try to optimize the light falling on the sensor within the writing area. Correct reproduction of writing is obtained by using parameters stored from previous calibration procedures. In some implementations (in the polynomial example shown below, the number of parameters may be larger and provided in a single file), the system uses only four parameters, which are sent by each pen to a host or server for processing and linearize these data. The specific calibration parameters for a pen are stored in the pen's memory during production inspection and calibration. The parameters can also be stored on a server or on a personal computer used by the user rather than being transferred from the pen during download.

Each sensor has a lens or a set of lenses attached to it. The objectives of the optical system are to optimize the efficiency of light delivery, cover the entire field of view, provide a consistent signal response across the field, and make the optical system as small and inexpensive as possible. This goal is achieved in part through the following steps:

As shown in FIG. 30 , a spherical lens 753 is used to focus the light onto the sensor 755 . The focal plane has the shape of a semicircle. The distance 759 from the lens to the sensor is optimized, while other optical and mechanical properties of the lens include focal length, diameter, thickness and material.

As shown in Figure 31, an aspheric lens 760 can be designed to have a focal point on the sensor as the light source moves around the periphery of the field, where the total energy supplied to the sensor by the light source is at its weakest. Thus, the aspheric lens is designed to have a focal plane that coincides with the plane of the sensor only for points on the periphery of the page. Points within the page will not be in focus, but the amount falling on the sensor will be significantly larger (closer to the sensor or better angle), and the signal stronger.

As shown in FIG. 32 (including a top view above and a side view below), two vertical cylindrical lenses 770, 771 can be used instead of one lens. The sensor distance limits the focal length of the lens on the horizontal axis. Therefore the lens diameter must be small and the lens must be located close to the sensor. On the vertical axis, the lens may be farther from the sensor, so the diameter may be larger. This larger diameter allows more light to be collected from the light source. The first cylinder (near the lens) focuses the light along the horizontal axis. The lens can be spherical since the horizontal axis spot diameter is not very important and does not vary much in our given field. The second cylinder has energy in the direction of the vertical axis. It is important to focus as much light as possible on this sensor in the vertical direction. For this to be true, the ray must travel an equal distance from the cylinder to the sensor for any angle. To accomplish this, the second cylinder should be curved into an aspherical shape. Either of the two cylindrical lenses can be a Fresnel lens to save space.

In a more detailed example of the pen holder shown in FIGS. The base 80 contains a clip 62 (not shown). When a clip button 86 is depressed, paper can be inserted between the clip 62 and the bottom of the pen holder. When the button is released, the clip grips the paper. The pen clip holds a stack of 7mm thick paper or a standard notepad 83. The clip places the pen holder on the paper so that the side 87 facing the pen is vertical, inclined no more than +/- 1°, thereby ensuring that the sensor receives data from the pen when the pen is used to write on a surface. infrared.

The holder can also just sit on top of a paper or notebook without the use of clips.

In the holder shown in Figures 13-16, the two sensors are mounted behind the IR filter windows 89, 91 with the photodiode 93 in between. An "ink well" 95 can receive the nib 97 for temporary storage, while a tube 99 provides a place to store the pen. The pen can be fully inserted into the tube and the battery in the pen can be recharged during storage.

Various mechanisms for operating the clip are possible including the example shown in US Patent Application Serial No. 09/376,837, filed August 18,1999.

In one approach, the clip mechanism shown in Figure 33 is used. The figure shows the steps in operating the mechanism, as follows.

Step 1: Agency is not activated.

Step 2: When the button 780 is pressed against a spring 782 , the latter releases the mount 733 and lets the hinge spring 784 open the clip 785 .

Step 3: Insert paper 786 between the clip and housing 787 of the pen holder.

Step 4: The lever 785 (the clip) bears against the paper, and the spring 788, which is significantly weaker than the hinge spring, gives way and begins to sag. The clip rotates about a point of rotation 789 .

Step 5: The clip presses the paper to the bottom of the holder. The spring 788 is depressed and both levers are lowered according to the amount of paper inserted.

A clip button can be used to lower the clip by converting horizontal movement to vertical motion with a lever. That is, the button presses a lever that rotates to convert horizontal motion to vertical motion, thereby lowering the clip.

This mechanism is shown in Figure 34. The clip has two longitudinal slide bars 790 , 791 connected to a horizontal bar 792 . There is a spring 793 between the horizontal bar and the housing of the holder. There are two longitudinal guides 794, 795 for sliding the longitudinal strip up and down.

In the bottom side view of the figure, when the button is not depressed, the spring is all the way up and the clip is pressed against the barrel housing (left side of the figure). When the button is pressed (right side of the figure), the spring is depressed and the clip (see the front view now) drops between the sliding devices. After paper is inserted and the button is released, the spring pushes the clip up and the mechanism catches the paper between the clip and the barrel housing.

The button moves a lever along multiple axes to lower the clip.

The button activates a lever that rotates and moves linearly to lower the clip.

In another version, the operator presses on a slide (button) 901 as shown in FIG. 36 . The slide button touches the center of the two top spring bars 903, 905, moving the fulcrums down in the same direction as the button/plate. The bottom terminal of the lever is secured around the swivel pins 907, 909 so that the bottom terminal moves down approximately twice the distance of the button. The top terminal of the lever is provided with prongs 911, 913 which are seated in guide slots 915, 917 and two small protrusions bent up perpendicularly from the bottom clip. The net result is a vertical downward movement of the clip, which is approximately twice the distance the button/plate travels.

Alternatively, as shown in Figure 35, the slide plate 930 may be placed horizontally and with the aid of a rigid elastic band, to achieve the desired result of a horizontal pushing movement opposite the vertical movement of the button.

As shown in Figure 37, another method can be obtained by a pulling movement of the button 951, if the force acting on the two levers 953, 955 in the mechanism moves from the middle of the lever to the bottom terminal, and The fulcrum of the lever moves to a point located 1/3 of the distance from the top terminal to the bottom terminal. This will provide the necessary mechanical advantage in order to maintain the 2:1 ratio of the distance the clip travels to the travel of the button.

In both mechanisms, this positioning of the point of application of force on the lever relative to the center of rotation can be used to increase the mechanical displacement of the clip.

Pen Holder Circuit

A block circuit diagram of the supporter is shown in FIG. 18 . An ASIC 205 is powered by a battery 511 , or by an AC voltage converter 513 , or by connecting a USB to a host 211 . The two CMOS sensors 201, 203 have outputs connected 165, 167 to ASICs via operational amplifiers 169, 171 via a multiplex 180 and a 12 bit A-D converter.

The analog output of this CMOS sensor undergoes signal processing in the form of offset cancellation and automatic gain control. The signal-to-noise ratio for this processing requires the use of a 5v power supply for the CMOS sensor and all analog signal processing circuits. Certain analog-to-digital converters operate with a 2.5v reference, and the signal from the CMOS sensor can be scaled down by a factor of 2 using a resistor divider.

The ASIC may be model Clarity 2B from Sound Vision, located in Framingham MA) and based on an ARM7 core. The ASIC firmware implements data acquisition, data storage, file system, I/O services (LEDs and switches), RS232 and USB communications, power regulation for idle and sleep modes, optical calibration and test modes.

The multiplexer enables the A-D converter 182 to alternate between the two CMOS arrays 201 and 203 in order to minimize the time difference between the two sensors. The clock frequency of the A-D converter is 1.2MHz. Each CMOS sensor is clocked at 600khz. Data acquisition uses the direct memory access (DMA) device of the application specific integrated circuit ASIC.

A wake-up input of the ASIC is driven by a phase locked loop 513 which receives the input signal from the photodiode. The photodiode is driven by modulated light from the pen.

The ASIC is clocked by a 48 MHz crystal and a clock distributor 517. I/O features are provided via a USB port 211 and an RS232/IrDA port 209 . Firmware and data are processed in SDRAM 207 and stored in a flash memory 519. An optional LCD 172 may be provided for a user display.

USB provides two modes of data transfer from the support to the personal computer: batch and real-time. Real-time transport is interrupt driven and used as a keyboard and mouse replacement application.

A dual function transceiver is used to implement both RS232 and IrDA communications. The RS232 communication is used as a dial-up connection to the server via cellular phone.

The support can support different kinds of external connections, including USB, serial, parallel, IrDA, Bluetooth (Bluetooth), firewall or any kind of communication port. When power is lost, the support has an automatic power supply capability if it is connected to any external device. It also has the ability to power itself down when an external device is disconnected. There are two connections to this support:

A connection is an external memory type of connection. Such a connection is made with a computer or other device, called a host, capable of displaying graphics and having an adequate user interface. When connected to a host device, the support acts as an external memory device. A user of the host device can browse the file system through the supporter, copy, view and edit files previously collected by the supporter. Software residing in the host is capable of converting, displaying, printing, and editing files stored on the support on the host screen, or copied from the support to the host. When connected to the host, the pen can also be used as a real-time input device.

Another type of connection is using a portable network or modem enabled device, such as a cellular phone. When such a connection is detected, the pen holder automatically initiates an e-mail or fax transmission of all previously collected data.

The optional LCD display notifies the user of the pen's status, such as connections and downloads via Internet-ready cellular phones where communications are not reliable. The display screen can be mounted on top of the holder. If an LCD is used, the LED may not be needed.

A single tri-color (green, yellow, red) LED 170 (see Figure 18) indicates normal collection of written data, download to a personal computer via the cellular phone, and monitoring of battery and memory status.

Pen, clip and inkwell switches 141, 143 and 145 are used to control the ASIC, and a reset switch 147 is used to reset the ASIC.

Figure 19 shows a state diagram of the inventive operating states. Shaded blocks indicate study status. Bright blocks indicate intermediate states. In other cases, the figure identifies the manner in which multi-color LEDs are used to indicate the operational status.

On power up, this green light blinks for the same amount of time when there are pages in memory. When the memory is empty, the green light is off at power up, and stops flashing 30 seconds or soon after if the user starts writing.

During writing, the LED is light green when data acquisition is taking place correctly. The green light goes out due to a fault detection triggered, for example, by blocking light, by leaving the pen off the writing surface, or by a dead battery.

Low battery status is indicated by a flashing yellow light when no writing is taking place. While writing, however, the yellow and light green lights flash intermittently if the battery capacity is low.

The memory is almost full indicated by double flashing of the yellow light when writing is not taking place, and almost full by intermittent flashing of yellow and light green lights when writing is taking place.

After a successful download, the download status is indicated by a bright green light (whether the pen is in or out of the holder or inkwell can be started independently). Flashing green indicates a download is in progress. A red light flashes when there is no active download service or the download signal is weak. The red light is doubly blinking for an Internet problem such as when a server is down. A triple flashing red light indicates that communication error settings include incorrect user ID or server address. This entails returning a code from the server to the pen after the data from the pen is unsuccessfully matched with the data recorded in the database.

Battery recharge status is indicated by a continuous green light after a successful recharge and by a multiple flashing green light when the support is plugged into the AC rectifier and charging. A signal from the battery monitoring circuit combined with a fast-charge signal from a charger (high when not charging) can identify the state, whether charging in progress or trickle charging.

The pen can be used during charging. If the pen is removed from the inkwell and used during charging, the yellow light is replaced with all the normal lights as described above.

Writing to the flashing flash state is indicated by a continuous yellow light.

All faults are reset by activating either of the two switches of the pen or of the inkwell described hereinafter. The only exception is when the download is successful and the user starts writing. The constant bright green light will then transition to a light green light.

In sleep mode, all fault indications, low memory capacity and low battery capacity continue in the normal way. All download failures remain.

If the ASIC needs to indicate a low storage capacity or low battery capacity condition during power up, the power up indication takes priority. The fault indication is then displayed after a timeout of 30 seconds. If the ASIC needs to indicate a low memory capacity or low battery capacity condition during a download, the download indication takes precedence. After resetting the download status, this fault indication is displayed.

Of the four holders that are turned on, the clip switch 141 indicates in a manner that the clip is opening and closing to inform the circuit that the user is starting a page change. The pen switch 143 indicates when the pen is in the holder or out of the holder. An inkwell switch 145 indicates when the pen is in the inkwell or out of the inkwell. The reset switch 147 is hidden using a paper clip accessible through a hole in the bottom.

The pen switch and the inkwell switch indicate when the pen is in the holder or the inkwell and remove power to the data acquisition and storage electronics when the pen is in the holder or the inkwell. The pen switch also opens a new file (or page) when activated, while the inkwell switch does not.

The clip switch indicates when the clip is activated, and also a page change and the start of a new file (each page is a file).

If the software freezes, the reset switch resets the ASIC. The switch is normally turned on as follows:

Pen Opens when the pen is out of the holder.

Inkwell Opens when the pen is outside the inkwell.

The clip opens when the clip button is released.

Reset Opens when the switch is depressed.

The support also includes a small connector for USB and RS232 interfaces and an antenna for using Bluetooth or other wireless technologies. The USB and RS232 connectors are also connected to the wake-up power circuit so that the pen holder can power itself when the cable is plugged into the mini-connector.

Angle signals generated by the sensor are processed by the ASIC and stored in flash memory for transmission to other devices such as cellular phones, PDAs and PCs (not shown), where they are used for handwriting recognition or capture picture. For example, the transmission may be performed using USB, RS232, IrDA or Bluetooth protocols.

File system

The structure of the flash memory is a FAT (File Allocation Table) compatible file system, in which each file represents a page of handwritten information. Each file has a 12-character unique name, including a 3-character extension and a separating "dot".

Data file creation

When a user puts the pen into writing mode by taking the pen out of the holder, or by pressing the form holder button when the pen is already in writing mode, a new file is created and subsequent writing is saved in a new file. If the user does not actually perform more writing after creating a new file, the newly created file is deleted and the same file name is reused the next time the pen enters writing mode.

During data acquisition, uncompressed data is stored in a temporary buffer in SDRAM and compressed by a data store job before being stored in a file in flash memory. Each page is stored in a separate file. The previous page is compressed before starting to capture the new page.

data file format

We use a binary compression format based on variable-rate Huffman coding with cubic approximation. Such a format includes encoded data coordinates and timestamps.

Before being compressed, the file has the following format:

The file is structured in four byte segments. Each segment corresponds to a pixel or a timestamp. Each pixel has a zero most significant bit (MSB) and consists of two 15-bit numbers, which are the sub-coordinates of the corresponding CMOS sensor. The timestamp is identified by the most significant bit of one and can store the full date and time of the next pixel (called the full timestamp), or it can store an incremental counter of pixels since the last full timestamp.

Every file starts with this full timestamp. A timestamp incremented by 1 is inserted at the end of each written stroke. Since all pixels are scanned uniformly in time, such a combination of time stamps enables efficient recovery of the entire history of the handwriting in future processing.

data download

When the support is connected to a personal computer using a USB cable, the personal computer automatically recognizes the support as a PC compatible USB device, and through the personal computer file system extension, the contents of the support file system becomes visible to the personal computer. The user can browse through it and use a stroke viewing program to watch the file.

When an RS232 cable is connected between the stand and, for example, a cellular phone, the stand automatically powers itself and starts transferring data files from the stand's memory to the phone. Infrared data transmission to the phone is also possible.

The data is sent to the server in compressed form and stored there until requested to be sent to a recipient. It is then decompressed and converted to one of the following formats: .GIF, .pdf, .gif, .ps for e-mail or fax communications.

Sensor Signal Preprocessing

In some examples, a pre-processor (not shown) may be used for background removal and storage to flash memory, while the ASIC processor performs all communication and I/O functions. The preprocessor can be implemented in the form of a programmable device such as a PLD, FPGA or digital application specific integrated circuit ASIC or a DSP. In this example, based on the pen's LED modulation frequency recovered by the phase locked loop (PLL), a frequency multiplication operation is performed to generate a high frequency pixel clock and a clock for the pre-processor.

The second processor may be a processor of another portable device, such as a cellular phone or PDA.

data collection

Target coordinates were acquired in successive sampling spaces 10 milliseconds apart in order to capture sufficient writing motion at a typical rate of 5 cm per second for a resolution of 0.5 mm. The ASIC operates as a master, generating the clock and all necessary signals for the sensor.

The sensor in the holder uses the pixel clock from the ASIC. A frame signal is generated by each sensor and read back into the ASIC. Therefore, in some implementations, the LED pulsing from the pen and the signal acquisition performed on the holder are not synchronized. In other examples, the data acquisition is synchronized with the modulation frequency of the pen. Synchronization effectively improves angular resolution.

In each sampling period, the position coordinates of the pen are obtained, data are captured from both sensors. One version of the background cancellation algorithm (asynchronous to the pen) requires capturing three adjacent frames at each sensor. An additional frame is used by the ASIC architecture for sensor reset.

To minimize the misalignment of the two sensors, the multi-channel data acquisition is alternated between the two sensors for each pixel.

Operating the A-D converter with a maximum sampling rate of 1.2MHz and alternating between the two sensors allows pixel sampling frequencies up to 600kHz. Each CMOS array has 1024+4 pixels, which results in a frame rate of about 600Hz. More pixel exposure and a correspondingly better signal-to-noise ratio can be obtained using a slow rate of 300Hz.

Each sensor operates in a mode in which each pixel is reset after being read into the A/D converter.

The infrared LED duty cycle is 50% of the three frame period. For this duty cycle, the LED frequency cannot exceed 200Hz.

For the purpose of eliminating background noise and low frequency crosstalk without synchronization, three data frames of 1024 pixels are required, as described below.

In addition to the main analog output, each CMOS also issues an END_FRAME signal. By each CMOS, the acquisition period for each of the three consecutive frames of data begins with the END_FRAME signal, which coincides with the last pixel of the frame. Each A-D conversion occurs on the falling edge of the PIXEL_CLOCK pulse. The total number of points is basically (1024+4)*3, where 1024 is the length of the CMOS array, 4 is the number of clock pulses between the END_FRAME signal and the start of the next frame, and 3 is needed to implement background compensation The number of consecutive frames.

From the waveforms obtained, the ASIC extracts three arrays, each equivalent to 1024 pixels. The array must be positioned correctly so that the ith cell in each of them corresponds to the ith pixel of the CMOS.

Let us call these arrays A1, A2 and A3. Background compensation is based on the fact that the LEDs in the pen are modulated at a frequency equal to 1/3 of the frame rate and a duty cycle of 50%. To implement background compensation, the following calculations are performed cell-wise on the array: A12=abs(A1-A2); A23=abs(A2-A3); A13=abs(A1-A3). The arrays A12, A13 and A23 are then summed in elements to form a new array, called A.

The array A is 1024 elements long and carries beam information with background cancellation.

To reliably remove large peaks that appear in the pixel waveform during the END_FRAME pulse, subarrays shorter than 1024 units can be extracted, for example three 1020 unit long subarrays starting at pixels 3, 1032 and 2061 ( based on 0).

The readout data of the two sensors are digitized simultaneously (or quasi-simultaneously with multiple channels when only one A-D converter is used).

Find peaks along a CMOS array at sub-pixel resolution

Determining the angle at which light is received by each sensor depends on the pixel location along the array of that sensor for determining peak light intensity. The algorithm for finding peak positions at sub-pixel resolution uses two parameters: T, the intensity threshold in volts, and W, the window width in pixels. Typical values for these parameters are T=0.1V and W=15.

As an initial step, look up the peak and its index in the array, call them Amax and M. If either of the two Amax values (corresponding to the two sensors) is less than T, the point is discarded. In this case, the LED is considered to be off with the pen not touching the paper. If M<W/2 or M>(1024-W/2), the point is discarded because it is too close to the edge of the field of view.

Extract a subarray from A that is W cells long starting at cell M-W/2. Find its partial centroid as follows: Create an array (call it S) of running sums of cells of the extracted subarrays. Get the value of its last element. Divide it by 2. Use a linear insertion/lookup table to find the partial subscript of the value's position in S. Add M-W/2 to this value. This will be the subscript of the part of the beam center of gravity in the original 1024 element array. Reverse its sign and add 512 (for a 1024-unit long A, or 510 for a 510-unit long A). The result P is the partial position (in pixels) of the beam relative to the axis of the sensor.

The use of sub-pixel algorithms allows to increase this pixel resolution by a factor of 8 to 10.

Calculates the angle of the light source relative to the sensor axis

As a result of the previous calculation we have the angular coordinates (in pixels) of each sensor's beam. We'll call them Pleft and Pright (the sensor from the point of view of the stylus). We recalculate Ps in radians based on the sensor geometry. In one example, the pixel pitch L=7.77 microns, the distance from the lens to the CMOS is D=4800 microns (common), and the refractive index of the lens material is N (1.5 for glass, 1.4 for plastics, and SF6 is 1.8). The parameters, Distance D, Refractive Index N, and Horizontal Offset and Off, will be adjusted using the calibration data used to correct the written image.

This angle (in radians) is then calculated as F=arcsin(N*sin(arctan((P*L)/D))).

As shown in Figure 22, the following parameters are required to calculate the position of this light source in the Cartesian coordinate system:

Sensor argument (inclination) C (radian), typically 30/57

Reference B, distance between sensors (mm); typically 150

Left sensor: Kleft=tan(C-Fleft)

Right sensor: Kright=tan(C+Fright)

X(mm)＝B*Kright/(Kleft+Kright);

Y(mm)＝Kleft*X

criteria for accepting a point as a valid point

Store points as coordinate pairs (X,Y). When storing points into memory, coordinates are stored contiguously, except in the following cases:

If the signal is found to be below the threshold (as described above) a flag (a pair of single values) is written to the memory, eg (NaN, NaN), which will later indicate that the pen was raised (NaN stands for no number, as defined in the IEEE algorithm standard). Then, no new data points are added to the file until the signal is detected again. This method allows the pen to tell the playback program exactly where the restored track line breaks.

If the signal is valid, but the pen position has not changed significantly compared to the previous position, no new data points are added to the memory, but unlike the no-signal case, no marker is written to the memory. Calculate the size of this moving square as (X1-X0) ² +(Y1-Y0) ² . A useful typical value for this moving square is 0.04 square millimeters.

Timestamps are not included in a file because this information is not needed to recover the pen's trace.

Coordinates are stored in a temporary buffer and only compressed before being stored in flash memory. Each page is stored in a separate file. Therefore, no end-of-page marker is required.

A full timestamp will be inserted before the first valid pixel. Every time the pen is lifted off the paper, all other timestamps on a page (document) will be incremented and inserted. No matter how long the pen was off the paper only a time stamp is inserted.

sleep mode

When the pen is taken out from the holder or the inkwell for writing, the ASIC turns on in sleep mode and waits until a light signal from the pen is detected.

When the support is woken up and detects that writing has been interrupted for a predetermined period of time, the support returns to a power saving sleep mode. The ASIC enters sleep mode by reducing its normal 48MHz clock frequency to 750kHz. The SDRAM refresh rate also varies accordingly in order to maintain data integrity.

When the pen is within the holder or the inkwell, the holder is almost completely powered off. The RS232 receiver and USB supervisory circuits consume negligible holding current. When the circuit is connected to a cellular phone via a cable or to a personal computer via a USB cable, once the active level of RS232 or USB is detected, the circuit wakes up and powers up the rest of the electronics when the pen is in the holder When inside, the pen holder is completely disconnected.

In sleep mode, the only function of the cradle electronics is to wait for a WAKEUP input from the photodiode and associated phase locked loop circuitry indicating that the stylus is activated. In sleep mode, the stylus consumes little power between periods when it checks the photodiode.

During writing, the pen transmits IR pulses that have been modulated. The pulse is detected at the support, causing the phase locked loop to wake up the processor and start normal acquisition mode once the ASIC switches to the 48MHz system clock.

Phase Locked Loop (PLL)

When the modulated IR light from the pen is detected, the pen LED's modulation clock (represented by the 1 kHz pulse train in the output light) uses the PLL circuit 132 to extract the modulation frequency tuned to the IR light.

All acquired data is initially stored in SDRAM 134 by DMA. The refresh rate of the SDRAM remains constant from acquisition mode to sleep mode. The storage requirement is 1 Mbyte for 50 pages of compressed or 10 pages of uncompressed data. The 5:1 compression algorithm must have fast and computationally simple encoding without limitations on decoding.

Data collected during writing is initially stored in SDRAM. When the pen is returned to the inkwell or the pen, or the page feed switch 136 is activated, the ASIC writes all data from the SDRAM to the flash memory 138. It takes only a short time to write a full page of handwritten text data into flash memory. The transfer is indicated for the user by lighting a yellow LED 140 on the support.

The 8Mbit flash memory stores compressed files representing a maximum of 50 pages of handwritten text. This compression algorithm allows at least 6 to 1 compression without visible text distortion.

Power supply for this support

The stand is powered at 3.0V by two AA NiMH batteries connected in series. The pen's three nickel-cadmium batteries are trickle charged when the pen is in the inkwell or holder. The pen's battery has a large capacity and is rarely fully charged. The trickle charge is sufficient to maintain the battery charge. A dedicated mode is provided when both the pen and the pen holder are in the charger to charge all batteries including the pen battery with the full charge current.

Battery life is the average amount of ten handwritten pages or one week without data compression for storage in memory. An average user can write 2 characters per second, which is 120 characters/minute, which is 7200 characters/hour. A typical handwritten page is about 700 characters. In order to produce ten pages, the battery must be active for 5 hours.

When connected to a USB port, the support can draw power from the USB host. The charge on the battery is maintained at a high enough level to enable the circuit before transitioning to USB power. Power from the USB connector is provided only after the ASIC establishes communication over the USB link and notifies the PC at the other end of the USB link that the connection is high power. In response, the personal computer provides power up to 0.5A. The battery charge current is set at 0.4A and monitored to convert the charger to trickle charge.

The pen holder circuit is activated when the pen is removed from the holder or inkwell. Certain support circuits such as the RS232 driver and the wake-up power circuit draw power directly from this battery.

The other circuits draw power from a 3.3v supply generated from a battery pack voltage of 2 to 3 volts through an on-board switching regulator. When connected to a USB link, the 3.3v is generated from the USB power.

Generate 5v for this analog circuit from this 3.3v supply.

Synchronization of pen and support

Synchronization of the pen and the pen receiver can result in better signal resolution and correspondingly better written resolution and angular resolution.

As shown in Figure 20, for synchronization, the pen generates a periodic pulse train 401 of higher frequency pulses, such as 1-10 kHz pulses (we show some timing diagrams as a suggestion), which can be easily detected by a phase locked loop out. The phase locked loop PLL detects not only the actual modulation clock but also its phase, which can generate a signal for starting data acquisition and synchronizing it with the pen LED.

As shown in FIG. 21 , the control signals LED_ON and LED_OFF trigger signal acquisition. In this case, only two frames are needed for background cancellation, one for the IR signal when the pen LED is on and one for the IR signal when the LED is off.

For a CMOS sensor, a shutter mode is provided to reset all pixels simultaneously.

Having only two frames per sample increases sample rate and resolution, and allows the processor to enter idle mode between samples to save power.

The use of 2-dimensional CMOS arrays

Suppliers to manufacturers of digital sensors produce small, power-efficient sensors and sensors with image processing circuitry that can be integrated into pen-on-paper or 3D pen-on-paper applications.

Three-dimensional positioning of the light spot can use two two-dimensional photoelectric arrays. The projection of a light point on two planes defines a single point in 3D space. When a 3D position track is enabled, the movement of an IR pointer pen can control a 3D object on a personal computer screen. As the pen moves in space, it drags or rotates the object in any direction.

Slave mode (SLAVE MODE)

In other implementations, using the ASIC-based ARM7 in a slave mode, the DMA can handle data acquisition but the vertical sync signal is provided by the pen light detection circuit (PLL).

Two alternate analog channels

Two separate channels can be used for analog signal processing and A-D conversion. Such an implementation can use more economical components that do not require fast settling times, bandwidths, and slew rates.

frame change replacement

CMOS sensors have a limited dynamic range. Although an adjustable electronic gain can be used simultaneously by both CMOSs after the CMOS output, this solution may not be ideal for two reasons.

First, when the pen is moving over an area of the paper, the signals for the two CMOSs may differ in magnitude, so changing the gain of both may fix one signal while reducing the other to an unacceptable level. In order to get the proper signal on this page, it is useful to have a separate gain for each CMOS. Second, using an electronic gain does nothing to prevent actual CMOS saturation, which is unavoidable for the area the sensor must cover.

The gain of the CMOS can be changed by changing the irradiance of each CMOS independently. As shown in FIG. 21 , the pen transmission rate remains at 100Hz, while the CMOS frame rate 601 changes between 300Hz, 600Hz and 1200Hz. At 300Hz, the background cancellation is directly forward. For 600Hz, the algorithm uses every other frame (frames 1, 3 and 5). For 1200Hz, the algorithm uses every fourth frame (frames 1, 5 and 9). The pixel rates are 300kHz, 600kHz and 1.2MHz. Changing the frame rate can be done by an ASIC without any other hardware.

Each CMOS can be connected directly to its own ADC, or both can be routed to an ADC which will be able to handle 1.5 Msamples/sec and have A 4V reference voltage. The analog-to-digital converter ADC may then be connected to a digital multiplexer to feed the signal into the application specific integrated circuit ASIC.

Method-based PSD

Instead of a CMOS array, it is also possible to use two PSDs to detect the infrared light from the pen. Each PSD determines the line of sight between the pen and PSD and the angle between the page. The two angles determined by the two PSDs and the distance between the PSDs are sufficient to calculate the position of the pen tip.

Even with an IR filter, ambient light will introduce errors in PSD position measurements. To reduce this error, the pen's infrared light is modulated to generate pulses at a modulation frequency with a 50% duty cycle, as described above.

This PSD signal converted to angles for triangulation can be distinguished using two analog techniques.

In a method called synchronous demodulation and used in testing electronic devices, the incident synchronous light pulses are split at the optical modulation frequency and the opposite gains (+1 and - respectively, respectively) are applied depending on whether the LED is on or off. 1) Applied to those signals. This subtracts out background noise. The signal is then integrated on the one hand using a time constant that varies in response to the signal, while the noise is balanced on the other hand. In one example, the modulation frequency can be 3 kHz and the pulse amplitude can be ILEDpeak=Xma.

A method for determining the use of a sample and hold technique. The shape of the optical signal has a duty cycle of 50% at a modulation frequency of 3kHz and, as before, a significantly intense short pulse applied to the modulation frequency. The modulation frequency is determined by a phase locked loop PLL and used to trigger the sample and hold circuit, while the bright light pulse is actually sampled. The pulse amplitude is ILEDpeak=Xma and the pulse duration T=Yusec.

PSDs sense and measure the position of light on their photosensitive surfaces with extreme precision. They are cheap and require negligible power consumption. The implementation of the PSD is also simpler than that of CMOS.

As shown in Figure 23, current-to-voltage conversion is performed on each of the four channels, two for each PSD. The four analog signals are subjected to low-frequency filtering 605 , synchronous detection 607 , integration 609 and digital conversion 611 through a microcontroller 613 (12-bit analog-to-digital converter A/D). A-D conversion is performed at 100Hz sampling rate. The processor is activated when the pen makes a mark on the paper. The processor performs signal acquisition and periodically stores to flash memory. Fig. 24 shows a system sequence diagram. The microcontroller enters idle mode when no traces are made, and enters sleep mode after an arbitrary period of time.

The microcontroller is woken up from idle mode or sleep mode by an interrupt or polling (TBD) of one of the following inputs: one of the four analog channels, when the signal at its modulation frequency exceeds a certain Threshold; an interrupt from a USB port when activity is detected from a host; one of its buttons is pressed.

The processor writes data from RAM to flash memory when the RAM becomes full and/or reaches a flash memory page boundary condition. If the acquisition continues and the page is full, the microcontroller starts writing to the flash memory. However, most of the flash operations should be performed during idle cycles when there are no writes.

Each PSD has two dual channels of analog signal processing. Each channel has a current-to-voltage converter whose output is AC-coupled to a first gain amplifier. This signal is shunted with the modulated frequency of the pulse generation IR LED (on the pen), currently 1kHz.

This chopper has a gain of +1 when the LED emits light. When there is no light, the gain is -1, so the signal is demodulated synchronously. The final stage is an integrator whose output is close to DC. More precisely, it is a sawtooth waveform due to the charging and discharging of the integrating capacitor in the amplifier's feedback.

The analog-to-digital converter A/D, or a PC-based DAQ or an analog-to-digital converter A/D of the microcontroller, samples the output synchronously at the modulation frequency at given time intervals in order to eliminate Error due to sawtooth waveform.

In order to use all 12 bits of the A/D resolution of the analog-to-digital converter, a dynamic change of the converter's reference voltage is used. The microcontroller always reads the ADC channel with the highest range and then divides it in half until the range is optimal for the signal.

The chopper amplifier uses a profile of the modulation frequency detected on each channel (four channels in total) with an analog circuit. The signal is acquired after the second gain stage, processed for signal transition detection, and then the recovered modulated pulse is passed through an OR gate to drive the chopper amplifier analog switch to change the gain between +1 and -1.

Using the phase shift of the photodiode, the rotating tip

As shown in Figure 25, if a rotating light source 617 is used at the tip of the pen, the phase difference between the signals on the three photodiodes 619, 620, 621 on the holder can be measured to find the pen position.

The rotating light on the nib can be implemented using a number (e.g. eight) of LEDs 623 time-triggered at spaced intervals by T/N, where T is the entire period of the LED cycle and N (e.g. 8) is the number of LEDs .

The signal source is at a certain position on an X-Y plane. The two signal detectors 619, 620 are located at two other fixed positions on the same plane. If the signal source has a radiation pattern such that the signal emitting light along the positive X-axis is in phase quadrature (rotated in space at the signal frequency) with respect to the signal emitting along the Y-axis, the propagation delay is equal to the signal period If the ratio is negligible, then the angle A1 formed by the two intersection lines 637, 639 drawn from the detector to the source will be the same as the phase difference between the signals measured at the detector.

If a third fixed position detector 621 is added, a second angle A2 will be formed because the three lines cross at the signal source. Likewise, the angle A2 between the lines at the point of intersection will be the same as the phase difference of the signal measured between the detectors. By applying some basic trigonometry, the position of the signal source in the X-Y plane can be found by the known fixed position of the detector and measuring the phase difference of the detector's signal. This calculation becomes insignificant if the three detectors are arranged along a straight line with equal distances between them.

Referring to Figure 26, the angles B and A are calculated from the angles measured by the sensors, "a" and "b" as follows:

There is: a/A＝d/R (1)

and b/B=d/R (2)

and B+A+b+a=180°(3), according to the basic geometric theorem,

We get: B/A=b/a, (4),

and correspondingly B=A×b/a (5)

A＝B×a/b (6);

Now substituting (3) into (5) and (6) we get:

A×b/a+A+b+a=180° (7) and

B×a/b+B+b+a=180°(8);

We use them to solve A and B:

A＝a×(180°-b-a)/(a+b) (9)

B＝b×(180°-b-a)/(a+b) (10)

The rotating light on the nib can be achieved using a number (e.g. eight) of LEDs 623 time-triggered at spaced intervals by T/N, where T is the entire period of the LED cycle and N is the number of LEDs. For example, eight light emitting diodes (LEDs) may be arranged in an outwardly pointing circle, spaced 45 degrees apart and driven by a signal oscillator with a 45 degree phase difference between adjacent LEDs.

As shown in Figures 27 and 28, the three detectors 641, which may be positive-inverted (PIN) diode photodetectors, drive a signal-processing chain 642 consisting of a transimpedance amplifier 643 and a high-gain limiter 645 for Remove any amplitude modulation in the detected signal.

Phase detection can be accomplished with two edge triggered one bit up and down counter type phase detectors 649, two binary counters and a clock running a number of decades at the signal frequency. If the counters are wired so that they count up by 1 for each clock cycle for which one sensor's phase is ahead of the other, and count down for each clock cycle for which one sensor's phase is lagging and set a third counter to continuously count up by 1, a microprocessor can periodically read and reset all counters by reading the continuously running counter pair from the phase detector ( Driven by) the reading of these two counters is scaled proportionally. This number is the phase difference (in slope units) between the three sensors and is the same as the angle between the intersections of the lines from the sensor to the source. Computing the position of the source relative to the sensor is then a trivial task.

Pen Light Activated Switch Refill

Different pen light activation methods can be used, including conductive rubber, pressure-sensitive materials, or strain gauges.

Pressure sensitive materials allow for a varying pressure threshold and coordination between the switching point and the ink flow. This will prevent data loss when the ink makes a trail but the pen is not active. For example, most ballpoint refills release ink at 20 to 30gf +/- 30%, while a current switch activates at 50 to 100gf and +/40gf, making reliable coordination of ink flow and data acquisition impossible. Special refills can be designed to prevent ink flow below 50gf so that current chip switches can be used.

Pen Optics Refill

Another method can be used for emitting light from the nib. Fiber optics can be used to collect light from an LED and emit the light in a 360° manner around the tip. Individual LED chips can be placed around the nib and emit light through a semi-reflective lens/window such that 50% of the light is emitted and the other 50% is reflected internally to mix with additional light, ultimately producing uniform 360° lighting. Light from a single LED can be mixed using a dedicated ring to redistribute them for consistency.

passive pen refill

If the IR light source is placed next to the sensor, the pen can be completely passive. A reflective surface will be provided near or at the nib. The sensor will see the reflection of infrared light from the nib and calculate the angle as described above.

The nib must only be reflective when pressure is applied to the paper and the ink forms a trail. Otherwise there will also be erroneous traces in digital form when there is no corresponding trace on paper.

Activation of the reflective mechanism can be mechanical or electrical. In a mechanical implementation, pressure on the nib will open a sheath and expose reflective surfaces around the nib. In an electrical implementation, pressure on the stylus will activate liquid crystals or other electro-optic technology that causes the material to reflect light. Reflected light from other objects such as fingernails and rings can be resolved by using polarized infrared light.

passive pen holder

Instead, the holder may have two mirrors, and the pen both emits light and receives reflected light. The sensing element on the pen can be a 2D PSD or CMOS array. The pen won't be omnidirectional if a flat 2D sensor is used, but it can form a custom circular 2D sensor that will have 360° coverage.

Keyboard and mouse replacement architecture

As mentioned above, the pen can be used to replace standard PC input devices such as mouse and keyboard.

When used as a keyboard or mouse replacement, paper, plastic or other flat surfaces can have a printed keyboard pattern used as keyboard and mouse pads for, for example, personal computers, laptop computers and cellular phones.

Today, when entering data into a personal computer, portable computer, or cellular phone, users are mostly limited to keyboards, keypads, or finger touch input on a screen. This is efficient and convenient when the keyboard is full size, but not when used in portable devices such as palmtop computers and cellular phones. Cellular telephone keypads are effective for dialing telephone numbers, but require too many keystrokes in an attempt to generate ASCII letters and symbols, making any type of data entry a very tedious and time-consuming process. Touch input on the screen of a handheld device requires the user to use a single writing style such as "graffiti" (rough carving) in order to minimize the amount of handwriting recognition the device requires, or requires the user to display a virtual keyboard on the screen Tap on. Both finger touch methods often produce incorrect inputs, limiting the functionality of the device.

Electronic pens can be used to enter text characters as well as record handwritten images and lines through modes that provide a highly reliable method. All a user needs to type with a pen is a piece of paper or any other surface (with or without a printed pattern of a keyboard).

The electronic pen together with the spatial transcription capability of the keyboard template is used to replace the mouse or keyboard.

The paper keyboard can be of various sizes according to user's needs. Sizes can vary from 8 ^1/2 x 11 sheets to _the size of a cell phone cover. The user first selects his desired keyboard size and then calibrates by touching the pen to specified characters on the keyboard. To type a message, the user touches the pen tip on the appropriate key. When the pen touches a square area of the paper corresponding to a certain letter, the position of the pen tip is calculated and the designated letter is determined. The method allows the user to generate text on a computer with an electronic pen without relying on any handwriting recognition software.

This approach is an improvement over current embedded keypads, software keyboards, or on-screen finger touch input methods on ever-shrinking personal devices. The paper keyboard enables users to type messages on hand-held and portable phones more quickly and reliably than alternative methods. The paper can also be used in other modes for recording drawn and handwritten notes and images. When complete, the paper keyboard can be discarded or folded for future use.

In addition to characters, the keyboard can also contain shortcut keys and function keys, enabling more efficient interaction with the small device. Shortcut keys minimize the number of keystrokes required to activate a command. Shortcut keys can be customized according to the type of input device.

The keyboard can also include a section that acts as a paper mouse pad. By using the spatial transcription capability of the electronic pen, the user can move within a designated square space on the paper to subsequently move a point on the screen of a device. Thus, the paper mouse is used as an alternative to buttons and finger touches on the screen, as a device for navigating on the screen of a hand-held or cellular phone.

This paper keyboard also has the flexibility to enter foreign characters. Keyboards can be created for several languages such as Japanese, Korean, Spanish, French and Russian. If a different alphabet is desired, the user can simply print out a new keyboard.

To type a message, the user touches the pen tip 701 on the appropriate key 703 printed on a paper keyboard 705 as shown in FIG. 29 . The location where the pen is touched is tracked by the tracker 707, converted to text and sent to a portable device such as a cellular phone 711 or a palmtop computer 709.

After use, the paper keyboard can be folded or discarded.

In other implementations, there is no printed paper for the keyboard; instead, when the pen is used to write on any surface, the movement of the pen is tracked via handwriting recognition to derive text, commands and drawings.

In one implementation of the method, the mouse and keyboard can continue to be used, with the pen used as a replacement. The pen can operate in a "pointing device" (mouse) mode or a "character input" (keyboard) mode. This mode can be selected by a dedicated hardware switch or button on the pen holder or pen, or by sending a command from the personal computer to the holder.

In mouse mode, pen operation is indistinguishable from operation of a second (USB) mouse. It is a relative positioning pointing device that moves the cursor on the screen. In keyboard mode, pen input may be received through a specially designed application that uses an active character recognizer to convert graphic input (strokes) into characters. Other applications are unaware of the pen's presence and continue to use regular (legacy) keyboard operations.

In another approach, the pen is the only input device to the system. In this case, a software driver stack is modified to provide system-wide keyboard functionality. Mouse mode operation has no effect and is consistent with the first described method. When operating in keyboard mode, pen input is recognized by the active handwriting recognition software embedded in the keyboard filter driver, and then passed to the system input queue in a manner similar to traditional keyboard input.

This second approach requires a platform availability model and (likely) modification of some system component, such as the basic input/output system (BIOS).

Both approaches increase the human element as well as usability issues. Specifically, there are two basic approaches to handwriting recognition: discrete (each time a single character) and continuous (each time a word, phrase or page). In the former case, the user must continuously rely on computer screen output for feedback. This can be inflexible because looking at the computer screen for feedback must often interrupt the handwriting process. In the latter case, the user has to look at the screen only occasionally, when a written unit (word or phrase) is completed and corrects it as needed.

Switching from mouse to pen input mode can be performed by retractable pen refill action. While the new supplement is inside the pen (the pen cannot write), it is used as a mouse. When writing is activated, the pen acts as a keyboard.

Other embodiments are within the scope of the following claims.

The holder need not include an ink well as described above, but can be any kind of device that can accommodate the sensor. The holder can be a simple pen cap, as shown before, or it can be any other type of device, whether it fits snugly or covers the pen, and whether or not it includes a clip. For example, the holder can be incorporated into a clipboard or a notebook.

The light in the pen can be an optical fiber that delivers the light to the tip of the pen and sends it in a disc-like pattern in all directions around the pen.

Claims (8)

1. A device for tracking the position of a writing instrument comprising: two or more sensors comprising a plurality of pixels configured to receive sensed light from a writing implement on a two-dimensional writing surface, and generating a sequence of signals representative of sensed light for determining a position of a writing implement on a two-dimensional writing surface, the at least one sensor being characterized by instability of data output by a plurality of pixels of the sensor; optics configured to enhance the light intensity of light received from the writing instrument; and a processor configured to receive the signal and determine a partial centroid output by a sub-array of pixels centered on the pixel closest to the sub-pixel location of the light from the writing instrument for increasing the calculated Stabilization of subpixel position for rays from writing instruments. 2. The device of claim 1, wherein the processor is further configured to enhance the sensitivity of the signal to changes in intensity of received light caused by tilting of the writing instrument or distance from the writing instrument Immunity. 3. The apparatus of claim 2, wherein enhancing the noise immunity of the signal includes determining an integer power of the overall signal distribution over the sensor. 4. The apparatus of claim 2, wherein enhancing the noise immunity of the signal comprises calculating a subpixel position based on half of the integer power position. 5. The apparatus of claim 2, wherein enhancing the noise immunity of the signal comprises calculating a subpixel position with an approximate maximum position using a polynomial approximation to the signal distribution. 6. The device according to claim 1, wherein said sensor comprises: light sensors separated by a distance; generating the sequence signal includes determining the direction in which the light was received relative to a reference direction, and Provides a signal indicating direction. 7. The device of claim 1, wherein at least one of the two sensors comprises a CMOS or CCD array. 8. The device of claim 1, wherein the processor is configured to, on the writing surface, convert the calculated sub-pixel positions to coordinates of the writing implement.

Images