HK1112760A - Tracking motion of a writing instrument - Google Patents

Description

Apparatus for tracking position of writing implement

The application is a divisional application of an invention patent application with the application date of 2000, 10 and 27, and the application number of "00814981. X", and the name of the invention of "equipment for tracking the position of a writing tool".

This application claims the benefit of a partial continuation of the application entitled Electronic Portable Pen with needles handles and Drawing with IR Sensing PSD Beacon System, U.S. provisional patent application No. 60/161752, No. 10/2000, No. 60/195491, No. 13/9/2000, No. 60/230,912, No. Electronic Pen with a Keyboard Template, filed 1999, No. 8/18, and No. 09/376,837, No. 6/18, filed 1999, No. 27/10, incorporated herein by reference in its entirety.

Technical Field

The present invention relates to an apparatus for tracking the position of a writing instrument.

Background

By tracking the motion of a pen, for example when writing or drawing on paper with the pen, the content being written or drawn can be captured and reproduced electronically. The movement of those nibs that do not mark a writing surface may also be tracked.

In some proposed methods, the surface over which the pen is moved may have an array of pixels or other sensing locations, each of which responds when the pen is at that location.

In other techniques, the pen tracking is accomplished entirely by electronics mounted on the pen tip. In some designs, a moving pen communicates with a static sensor that is independent of the pen and uses triangulation algorithms to track the motion.

Disclosure of Invention

In general, in one aspect, the invention features a method that includes transmitting light from a moving writing instrument as an indication of the position and path of the writing instrument on a two-dimensional writing surface; sensing the light with two or more sensors and generating a sequence of signals representative of the sensed light; and applying a technique for reducing the effect of variations in light intensity along a third direction relative to the generated signal.

Implementations of the invention may include one or more of the following features. The technique may be based on optical characteristics set to enhance the consistency of the signal response of the sensor. The lens may be a spherical lens or an aspherical lens. The sensor may be an array of light sensitive pixel elements or an analog sensor. The technique may be based on algorithmic processing of the generated signals. The algorithmic processing may include linearization of the sensor signal response based on parameters associated with the writing instrument. The techniques may be implemented in digital hardware or in analog circuitry. The algorithmic technique may reduce the effect of variations in light intensity in addition to the size effect. The signals may be grouped by frame and the signal processing technique may include multi-frame processing to eliminate noise. The light transmitted by the moving writing instrument may be modulated at a frequency related to the rate at which the signal is generated by the sensor, and the sensor signal may be branched off at the modulation frequency. An opposite gain may be applied to each of the split signals depending on the on or off state of the light corresponding to the signal transmitted from the writing instrument. The frame rate may vary. The split signal may be integrated over time in the future. The light transmitted by the writing instrument may include a strong short pulse imposed on the modulation frequency, a phase locked loop determines the modulation frequency from the sensor signal, and the sensor signal may be sampled at the time triggered by the phase locked loop during the duration of the strong short pulse. The characteristics of the transmitted light may be used to synchronize between the writing instrument and the sensor. The transmitted light may include low frequency modulation cycles and high frequency modulation bursts, and the sensor signal associated with the high frequency bursts may be used to lock onto a modulation clock.

In general, in another aspect, the invention features a method that includes transmitting light from a moving writing instrument in a time-varying directional pattern using two or more light sources

More sensors located at two or more different locations spatially separated from the writing instrument sense the light, and the position of the writing instrument is determined by detecting a phase difference between the signals measured by the two or more sensors.

According to an aspect of the present invention, there is provided an apparatus for tracking a writing instrument position, comprising: two or more sensors, the sensors comprising a plurality of pixels and being configured to receive sensed light from a writing instrument on the two-dimensional writing surface and to generate a sequence of signals representative of the sensed light for use in determining a position of the writing instrument on the two-dimensional writing surface, at least one of the sensors being characterized by an instability in data output by the plurality of pixels of the sensor; an optical device configured to enhance the light intensity of light received from the writing instrument; and a processor configured to receive the signals and determine a partial center of gravity output by a sub-array of the plurality of pixels, the sub-array centered on a pixel closest to a sub-pixel location of the light from the writing instrument, for increasing a stability of the calculated sub-pixel location of the light from the writing instrument.

Implementations of the invention may include one or more of the following features. The time-varying direction pattern may include a rotation pattern relative to an X-Y plane on which the writing instrument is moved. This signal for light emitted in the positive X-axis direction is in phase quadrature with the signal for light emitted in the Y-axis direction.

In general, in another aspect, the invention features apparatus that includes a sensor configured to receive light from a writing instrument moving across an X-Y writing surface, the light having a change in intensity along a Z-axis perpendicular to the writing surface, and optics configured to enhance the intensity of the light received from the writing instrument.

Implementations of the invention may include one or more of the following features. The optical device may be a spherical lens or an aspherical lens. The optics may comprise a single spherical lens and the lens, together with the corresponding sensor, is set to enhance the light intensity of light received at a larger angle or longer distance or received at an adverse location of the writing instrument. The optical device may include a dedicated lens configured to enhance the light intensity of the light received from a location on the X-Y surface other than a predetermined location. The optical device may include two cylindrical lenses, one proximate to the sensor for projecting light horizontally onto the sensor and the other positioned for collecting light in the Z-axis direction, the other lens having a shape that curves around the first lens. The algorithmic processing may enhance the immunity of the signal to changes in the intensity of the received light caused by tilting or distance from the writing instrument. The process may determine an integer power of the overall signal distribution across the sensor and calculate the location of a sub-pixel based on half the integer power location. The processing may use a polynomial approximation to the signal distribution and calculate a sub-pixel location at the approximated maximum location. The calibration process may generate parameters for use in conjunction with the sensor characteristics.

The calibration parameters can correct manufacturing defects of the optical device and the sensor, and the algorithm

The process may use an orthotriangulation technique to determine a position of the writing instrument. The calibration parameters may correct for manufacturing defects of the optics and sensors, and the algorithmic process may determine the position of the writing instrument using a polynomial series when coefficients in a polynomial are determined during the calibration process.

In general, in another aspect, the invention features a method that includes receiving light from a moving writing instrument with an array of sensing elements of a sensor, sequentially reading the sensing elements to produce a sequence of signals indicative of the light sensed by the array of elements, and resetting each element after it is read and before at least some other elements in the array are read.

Implementations of the invention may include one or more of the following features. The array may comprise a CMOS or CCD position sensor. All elements are read before they are reset.

In general, in another aspect, the invention features a method that includes determining a sequence of three-dimensional positions of a moving writing instrument based on a signal.

In general, in another aspect, the invention features a writing instrument combination including an elongated housing configured to be hand-held, a light source in the housing, and a lens in the housing configured to receive light from the light source and transmit the light via a vacuum path to a light sensor spatially separated from the writing instrument, the lens configured to be semi-reflective.

In general, in another aspect of the invention, the light source includes an array of light sources arranged about an axis of the writing instrument and configured to emit light in a direction perpendicular to the axis.

Implementations of the invention may include one or more of the following features. The lens may be configured to internally reflect and concentrate the light and emit the light by reflection from a reflective outer surface of the lens. The lens may include a cylindrical body having an upper surface receiving the light, and a lower annular curved surface reflecting the light to the light quantity sensor. The reflective outer surface may include a conical curve that is angled at 45 degrees to the longitudinal axis of the writing instrument. The light source in the pen tip may comprise LEDs arranged on a circle.

In general, in another aspect, the invention features an apparatus configured to turn on and off a light source in response to a user applying pressure from a writing instrument to a writing surface, the light source configured such that an amount of pressure required to trigger the apparatus is not so great as to disrupt normal writing movement of the writing instrument on the writing surface.

Implementations of the invention may include one or more of the following features. The writing instrument may comprise a ballpoint pen barrel having a writing tip, and the device may be positioned at an end of the barrel remote from the writing tip. The device may be a switch or a pressure sensor.

In general, in another aspect, the invention features a holder having a receptacle for receiving at least a portion of a writing instrument for storing the writing instrument, the writing instrument and the holder including respective components that enable wireless transmission of a signal associated with movement of the writing instrument and tracking of the writing movement based on the signal.

In the practice of the invention, the holder may be a pen cap and may include a clip configured to attach the holder to a stack of papers or a notebook. The holder may include at least two light sensors and a processor that processes signals from the light sensors to determine a sequence of positions of the writing instrument. The holder may include a socket for holding the writing instrument and enabling the battery in the writing instrument to be recharged.

In general, in another aspect, the invention features an element for enabling wireless transmission of a signal associated with movement of a writing instrument and tracking the writing movement based on the signal, the element being embedded in a cellular phone, a PDA, a WEB personal digital assistant, or a tablet with a paper clip.

In general, in another aspect, the invention features a holder having a mechanism for attaching the holder to a writing board in a direction that enables the component to be used in conjunction with the wireless transmission. The clamp mechanism may include a switch that activates a function of a processor in the holder when the clamp mechanism is operated.

In general, in another aspect, the invention features a holder including a receptacle for the writing instrument and a recharging circuit connected to recharge the battery when the writing instrument is in the receptacle.

In general, in another aspect, the invention features a CMOS sensor adapted to receive light associated with movement of a writing instrument and to provide signals indicative of an angle at which the light is received relative to a known direction, and a lens positioned to direct the received light at the CMOS array.

In the practice of the present invention, the lens may be optimized for collecting light from the area where the movement of the writing instrument occurs. The lens may be a field lens or a fresnel lens. The lens system may be configured to collect light in a direction perpendicular to a plane of motion of the writing instrument and project the light onto the sensor in a direction parallel to the plane of motion.

In general, in another aspect, the invention features a calibration by positioning a writing instrument at successive positions on a writing surface, receiving light from the writing instrument at the successive positions to generate signals at sensors, and determining calibration parameters for calibrating a process for determining the position of the writing instrument as it moves.

In implementations of the invention, the calibration parameters may include coefficients used in a polynomial series used as part of the position determination process.

In the practice of the invention, this location is not on a regular rectangular grid.

In general, in another aspect, the invention features (1) identifying locations on a writing surface corresponding to input elements to be input into an electronic device, the writing surface being non-electronic and independent of the electronic device, (2) using a writing instrument to tap selected ones of the identified locations corresponding to the input elements to be input, and (3) sensing the locations at which the writing instrument is tapped and inputting corresponding data into the electronic device.

In implementations of the invention, the writing surface comprises a sheet of paper and the input elements comprise characters or commands of a language printed on the writing surface.

In general, in another aspect, the invention features moving a writing instrument over a non-electronic writing surface to indicate a path, and remotely sensing the path and generating signals for inputting the path into an electronic device independent of the writing surface.

In general, in another aspect, the invention features modulating light at a predetermined frequency, the light being transmitted from a moving writing instrument to a photosensitive element spatially separated from the writing instrument, and using a phase-locked loop associated with the sensor to lock the phase of the modulated light.

In general, in another aspect, the invention features circuitry for tracking a writing instrument through the use of wireless signal transmission between the writing instrument and a static element including a main processor coupled to perform at least data acquisition related to movement of the writing instrument and a separate preprocessor coupled to perform at least data communication related to the tracking.

In the practice of the invention, the preprocessor may also be connected to perform user interface functions and sub-pixel data storage, while the main processor may also be connected to perform background elimination and sub-pixel computation.

In general, in another aspect, the invention features a reflective element configured to reflect light received from outside the writing instrument to a sensor for tracking movement of the writing instrument.

In the practice of the present invention, the reflective element may reflect light to the sensor when the writing instrument is being used to write, and inhibit the reflective element from reflecting light to the sensor when the writing instrument is not being used to write.

In general, in another aspect, the invention features receiving light from a moving writing instrument with an array of photosensitive elements having a photosensitive pixel element, and determining a location in the array where the light received from the writing instrument has been at a maximum intensity, the location determined with sub-pixel accuracy.

In implementations of the invention, the sub-pixel location is determined by determining an integer pixel location proximate to the sub-pixel location, and finding a partial center of gravity of a sub-array centered at the integer pixel location.

In general, in another aspect, the invention features indicating locations on a non-electronic surface corresponding to inputs to an electronic device and detecting the locations to input them into the electronic device.

In general, in another aspect, the invention features a clip for securing a sheet of paper on which the writing instrument is to be moved to the sensor.

In the practice of the invention, the mechanism may be a tablet or part of a notebook having a paper clamping device and the clamp may include a mechanism to enable a user to clamp or release the paper. The clip may include an activation button and a spring and a lever operated by the button. The lever may be configured to rotate in response to the button. The button may be configured to be pushed or pulled. Other advantages and features will become apparent from the following description and claims.

Drawings

Fig. 1 illustrates pen tracking.

Fig. 2 shows a pen.

Figure 3 shows the lens in a pen.

Figure 4 shows the lens in a pen.

Fig. 5 and 6 show the reflection of light in a pen.

Fig. 7 illustrates a tracking method.

Fig. 8 shows a pen.

Fig. 9 shows a pen.

Fig. 10 shows a holder.

Fig. 11 shows a lens in front of a sensor.

Fig. 12 shows a holder.

Figure 13 shows a holder.

Fig. 14 and 15 show one half of a holder.

Fig. 16 shows the other half of the holder.

Fig. 17 shows a view.

Fig. 18 shows a block circuit diagram.

Fig. 19 shows a state diagram.

Fig. 20 shows a timing chart.

Fig. 21 shows a sequence diagram.

Fig. 22 shows the tracked geometry.

Fig. 23 shows a circuit diagram.

Fig. 24 shows a sequence diagram.

Fig. 25 and 26 illustrate a rotating beam technique.

Fig. 27 shows a channel diagram.

Fig. 28 shows a channel loop.

Fig. 29 shows a paper keyboard.

Fig. 30 shows a spherical lens.

Fig. 31 shows an aspherical lens.

Fig. 32 shows a two-lens approach.

Fig. 33 shows a clip.

Fig. 34 shows a clip.

Fig. 35 shows a sliding belt clip.

Fig. 36 shows two views of a clip.

Fig. 37 shows two views of a clip.

Detailed Description

We describe an electronic wireless pen that, in addition to its conventional function of leaving a visible trace on the writing surface, emits Infrared (IR) which is picked up by an external IR sensor to measure the position of the pen relative to the sensor. The sensor is a CMOS or CCD linear or two-dimensional array, a Position Sensitive Detector (PSD) or other photosensitive detector. The sensor may be clipped to the edge of the writing surface, providing a reproduction of the writing on those pages. The position of the pen is determined by mapping the sensor readings to the actual XY position of the pen tip on the paper.

The electronic input device appears as a conventional pen with a holder.

The user writes with it just as with any ordinary pen on paper, a notebook, or any other flat surface. It is used to capture handwritten text or drawings. By recording the measurements of the sensor in its memory, the pen stores all its movements during its use. The pen then downloads it to a computer, personal digital assistant, portable computer or portable telephone. Then, when the handwriting appears on a page, it is automatically reproduced according to the sensor information.

As shown in fig. 1, a pen or other writing instrument 10 that leaves a visible trace 12 on a sheet of paper or other writing surface 14 that is written or drawn in a conventional manner may also have a source 16, which source 16 emits Infrared (IR) light 18 for automatically tracking the motion of the pen. The light is detected by infrared sensors 20, 22 at a nearby location, for example, near the edge 23 of the paper and held stationary relative to the pen.

The sensor emits a sequence of signals indicative of the position of the pen on the writing surface (e.g., angle 24) at which light from the pen is received at each instant of a continuous measurement time. The circuitry associated with the sensors uses an algorithm for processing the directional information (and the known distance 26 between the sensors) to determine the successive positions of the pen as it moves across the writing surface. The algorithm may use a mathematical model to translate the pixel signals of the sensor to locations on the writing surface. The algorithm may be a quasi-triangulation algorithm using calibration parameters (distance from the lens to the sensor and horizontal offset between the centers of their refractive indices) or may be a polynomial approximation.

The tracked pen motion may be used to recognize handwriting created using the pen or to capture drawings created by the pen or for other various applications. The tracked motion information may be sent to a local personal computer or to a central computer via a personal digital assistant, a portable computer or a portable telephone for central storage and processing.

Tracking light sources with two-dimensional or one-dimensional sensors

The problem of tracking XY from one pen can be formalized as follows.

The pen carries a limited light source close to the pen tip. The source emits a light ray whose intensity at a test point depends on the XYZ location of the test point with the source as the origin.

A multi-channel detector is located at another location. It collects some portion of the light emitted by the source on the pen. The intensity delivered to the different channels varies according to the XYZ location of the channel input relative to the location of the source. The intensity data is sufficient to calculate the XYZ position of the source relative to the detector. Intensity data is also affected by noise including source instability, detector noise, and other classes.

We are only interested in obtaining an XY map from the pen. In fact, all three coordinates will vary due to thickness irregularities on the writing surface and due to changes in the slope of the pen during writing.

With noise this will cause complex variations in the channel readout data.

There are different ways to process such signals: weighted average (center of gravity), median calculation, thresholding, etc. They mostly consider noise cancellation and consider the Z-direction motion of the source as noise.

Our goal is to establish the nature of the detected signal that is invariant to Z-motion of the pen and to some noise sources.

For this purpose we introduce an aperture between a detector and the source. For example, the aperture may comprise a lens. Whereby we obtain a spatially limited signal. This means that there is a set of closed detector channel devices excited simultaneously by signal and noise (which in the case of a linear array detector is part of it). The group is surrounded by channels excited only by noise. In the absence of an aperture, all channels are excited by both signal and noise.

After creating such a signal we establish a certain point (e.g. a maximum) and define a processing window around this point in a way that extends beyond the spatially limited signal. A cumulative distribution function of the data within the processing window is then calculated against the channel numbers. The projection of the half-size points of these functions on the channel number yields the invariant property. When the channel number is an integer, the invariant property may be fractional.

There are basically two types of detectors, two-dimensional or one-dimensional. Each detector may be a dual channel detector such as a PSD, or a "multi-channel" detector such as a CMOS and CCD device. The PSD detector has two output signals, the ratio of which defines the relative position of the incident spot. CMOS and CCD detectors have many pixels. Each pixel defines a particular space on the detector and has an analog output. These analog outputs may be digitized for subsequent processing in firmware or software, or may be processed by analog devices. The algorithm used in software may be alternatively implemented by hardware by the same method.

Calibration process

It is not necessary to obtain a linear response of the detector to the pen movements, as other known methods have been suggested.

If simple triangulation is used to interpret a detector readout as the XY map position of a pen, a linear response will be required

There is a clear dependency between the position of the device map of a pen and the read-out of the left and right detectors (L and R), as follows:

X＝Fx(L，R)；

Y＝Fy(L，R)。；

these functions may be expressed as polynomial series.

The coefficients in these polynomials may be determined during a calibration procedure.

During the calibration process, the pen is placed in a known different XY position on the paper and the readings of both detectors are obtained and stored for future processing. After a sufficient number of points have been accumulated, the coefficients in (1) are calculated using a common linear algebraic method.

We know that the system (1) is substantially non-linear. It is important that the calibration points are positioned in such a way that the necessary resolution can be obtained over the entire writing area. We do not locate the calibration points at the nodes in a conventional rectangular grid. Instead, we use a mathematical model to match a calibration grid to the specific non-linear nature of our detector.

Another calibration procedure

If a pen's XY map position is calculated using simple triangulation from the detector data, we intentionally introduce an error into the geometry of our detector to account for the non-linear nature of the above discussion.

By virtue of our design, we know exactly the refractive index of the lens in our detector and the distance between the lens and the sensor. It has also been demonstrated that by varying these values in the triangulation calculations one can effectively compensate for the non-linear nature of the detector.

To obtain the effective value of the refractive index and the effective value of the distance between the lens and the sensor, we run another calibration procedure. Calibration points for different XY map positions are required for proper resolution on the writing area. The positions of the calibration points for these interrupted triangulation measurements are obtained by a mathematical model.

Pen with writing-in function

As shown in FIG. 2, in one example, the infrared source in the pen may be an LED13 that emits infrared light 15 at the tip 17 when pressure is applied during writing. In this example, the LED source 13 consists of a ring of LEDs 19 (only two LEDs are shown) arranged around the longitudinal axis 21 of the pen.

Light from the LED is projected down the nib and into a pen body/lens 18 of acrylic material used as a light pipe. The acrylic lens scatters and transmits the received light so that the light emitted by the pen is transmitted along light paths in various directions around the pen.

As shown in fig. 3, the tube 18 is polished and reflective, concentrating the light 502 from the inside of the LED19 without allowing the light to escape to the side. The bottom of the tube is polished with a 45 degree conical surface 504 at the bottom of the tube. A reflective cylindrical housing 506 helps to confine and induce mixing of the light emitted by the LED. A cone 508 supports the light pipe. Direct light directed downward within the tube is reflected by the tapered surface 504 and is transmitted into the air at all angles around the pen.

Fig. 5 and 6 illustrate side and top views of internal reflection of light rays in the light pipe. Most of the light from the LED passes through the length of the pen housing and is reflected at a 90 degree angle towards the sensor. Other rays find their way out of the pen at angles other than 90 degrees.

As shown in fig. 2, the light emitted by the pen is confined to a vertical space 11 near the writing surface 13 so that as much light as possible reaches the sensor (not shown), which is also positioned within a small distance of the writing surface.

Other configurations with different light pipe and lens shapes may be used, including the one shown in FIG. 4, which may provide better coupling between the LED and the light pipe, and may more effectively split and direct the light to a reflective surface located at the bottom of the pipe.

The pen in this example (fig. 2) comprises a ballpoint pen barrel 23 terminating in a writing tip 25. A pressure switch 26 emits a signal that can be used to turn on the LED and also to trigger the function of a circuit 28 mounted on the pen when the user applies pressure to the writing tip during writing. The circuit 28 and LED19 are powered by a battery 30. These elements are all held in a single housing 15.

As shown in fig. 7, the position of the pen in an x-y coordinate system 40 parallel to the writing surface is determined by the two alpha and beta angles sensed by the two sensors 20, 22 and the known distance 26 between the sensors.

In another example, as shown in fig. 8, the pen is powered by three miniature AAA type nickel cadmium rechargeable batteries 51 stored behind the pen back (the batteries are moved closer to the pen tip and the circuitry is placed behind for better weight distribution).

The battery supplies power directly to the electronic circuitry 28 without any dc to dc converter. The power is delivered only when the pressure switch 26 is activated. The light activated switch moves only a short distance (e.g., 0.25 mm). The switch is preloaded by a spring mechanism to minimize cartridge travel, which should not exceed 0.008-0.010 inches.

The pressure sensor may be an effective way to match the pressure on the cartridge to the activation of the LED, as many popular switches have an activation pressure in excess of the required 20 to 30 gram level.

The electronic board 28 behind the battery generates modulated pulses of approximately 100Hz and 50% duty cycle for the infrared LEDs and pen on and pen off signals in bursts of 1 to 10kHz for sleep mode.

The light emitted by the pen is visible in all directions so that the pen can be used in any direction in the hand. The closer the emitted light is to the pen tip, the less error due to the change in pen-to-paper angle and the more accurate the tracking of the pen tip. The LED light should be located in the infrared range, far from the visible spectrum, so that ambient light from the sun and lighting fixtures does not unduly interfere with the infrared emission and is also invisible to the human eye.

The orientation of the infrared source of the pen and the sensor in the holder are set to ensure that its infrared beam reaches the sensor when the pen is tilted sideways and rotated during normal writing or drawing.

Figure 9 shows a more detailed isometric view of a partially assembled pen.

Pen holder

As shown in fig. 1 and 10, the sensor may be housed in a typical pen cap 70 in which the pen may be stored when not in use.

When the pen is in use, the pen cap is removed and the cap is placed in a fixed position near the writing surface and in the vicinity of the pen. In some examples, the sensor is a linear CMOS array (e.g., a 1024 pixel array from photo-vision system, llc (pvs) (mailbox 509, kotland, NY13045) (element number LIS1024) may be used). Other linear CMOS sensors of PVS or other companies with the same or different number of pixels may also be used. The analog output of each sensor is a sequence of 1024 analog signals, one from each sensor pixel.

As shown in FIG. 10, the holder may include a clip 62 for attaching the holder to the edge of a stack of papers or a notebook.

When writing begins (e.g., the pen begins to emit light), the processor is awakened from a sleep mode (described below) using a third sensor in the form of a photodiode 56 located in the middle of the holder.

The third sensor signal may also be used to synchronize circuitry in the sensor system with circuitry in the pen. All three sensors are covered by an infrared filter window facing the writing surface.

As shown in FIG. 11, in one example, the front surface 100 of each primary sensor has a vertical height 104 of 125 microns and a distance 106 of four millimeters from the front surface 108 of the lens 110. The FOV112 is10 degrees. The pen tip 114 directs infrared light into the FOV when the pen tip is on the paper 116.

As shown in fig. 17, the two sensors 88.90 are positioned 100mm apart. Each sensor has a field of view (FOV)94, 96 centered about a FOV axis 195, 197. The axes of the FOVs are not parallel but are canted at an angle 199 to increase the amount of overlap of the FOVs. The FOV of each sensor has a width of 150 ° in the horizontal (x y) plane and a height of +/-5 ° in the vertical plane.

The FOV does not cover certain locations 101 on the writing surface 15 near the edge 303 of the paper and is set so that the dead zone 98 does not extend beyond the holder 25 mm.

In another example of a pen holder, as shown in FIG. 12, the sensor 117 and the lens 119 are mounted on a holder frame 121 having an infrared filter 123 on the front. The mount is mounted on a printed circuit board 125 and is contained in a housing 127. The centers of the two primary sensors are separated by 100 mm.

Light from the pen is collected by two sensors to identify the linear position of a modulated light source within a defined range (8.5 x 11 inches). The linear position may be calculated using triangulation, a lookup table, a polynomial approximation, or any combination of these methods.

The sensor is a planar, linear multi-pixel sensor. When the light source is at different positions within the field, different pixels of each sensor are illuminated. As the light source moves across the field, the corresponding motion of the light rays passing through the pixels of the sensor may not be linear, but this lack of linearity can be addressed because the linear position can be calculated by a combination of mathematical and optical knowledge and calibration data from the pair of sensors.

Instead of looking for a linear response from the sensor, we try to optimize the light falling within the writing area on the sensor. A correct reproduction of the writing is obtained by using the parameters stored by the previous calibration procedure. In some implementations (in the polynomial example shown below, the number of parameters may be larger and provided in a single file), the system uses only four parameters, transmitted by each pen to a host or server to process and linearize the data. The specific calibration parameters for a pen are stored in the memory of the pen during production verification and calibration. The parameters may also be stored on a server or a personal computer used by the user without being transmitted from the pen during download.

A lens or set of lenses is attached to each sensor. The goal of the optical system is to optimize the efficiency of light transmission, cover the field of the entire field of view, provide a uniform signal response across the field, and make the optical system as small and inexpensive as possible. The target part is realized by the following steps:

as shown in fig. 30, a spherical lens 753 is used to focus the light on the sensor 755. The focal plane has a semicircular shape. The distance 759 from the lens to the sensor is optimized, while other optical and mechanical properties of the lens include focal length, diameter, thickness and material.

As shown in fig. 31, an aspheric lens 760 may be designed to have a focal point that is located on the sensor when the light source is moved around the periphery of the field, where the total energy supplied to the sensor by the light source is weakest. Thus, the aspheric lens is designed to have a focal plane that coincides with the plane of the sensor only for points on the periphery of the page. A point within the page will not be in focus, but the amount falling on the sensor will be significantly larger (close to the sensor or better angle) and the signal will be stronger.

As shown in fig. 32 (including a top view above and a side view below), one lens may be replaced with two vertical cylindrical lenses 770, 771. The distance of the sensor limits the focal length of the lens in the horizontal direction axis. The lens diameter must be small and the lens must be located close to the sensor. In the vertical axis, the lens may be further away from the sensor, so the diameter may be larger. This larger diameter allows more light to be collected from the light source. A first cylindrical surface (near the lens) focuses the light along a horizontal axis. The lens may be spherical in that the axial spot diameter is not very important in the horizontal direction and does not vary much within our given field. The second column has energy in the vertical axis direction. It is important to focus as much light as possible on the sensor in the vertical direction. To make this true, the light must travel an equal distance from the cylinder to the sensor for any angle. To accomplish this, the second cylindrical surface should be curved into an aspherical shape. Either of the two cylindrical lenses may be a fresnel lens in order to save space.

In a more detailed example of the pen holder shown in fig. 13, 14, 15 and 16, the sensor system is housed in a housing 79 having a bottom 80 and a top 82. The base 80 contains a clip 62 (not shown). When a clip button 86 is depressed, paper may be inserted between the clip 62 and the bottom of the pen holder. When the button is released, the clamp grips the paper. The pen clip contains a stack of 7mm thick paper or a standard notepad 83. The clip places the pen holder on the paper so that the side 87 facing the pen is vertical, inclined no more than +/-1 deg., to ensure that the sensor receives infrared light from the pen when writing on a surface using the pen.

The holder may also be located just above the paper or notebook without the use of a clip.

In the holder shown in fig. 13-16, the two sensors are mounted behind the ir filter windows 89, 91 with the photodiode 93 in between. An "ink reservoir" 95 may receive the pen tip 97 for temporary storage, while a tube 99 provides a place to store the pen. The pen may be fully inserted into the tube and the battery in the pen may be recharged during storage.

Various mechanisms for operating the clip may include the example shown in U.S. patent application No. 09/376,837, filed 8, 18, 1999.

In one version, a clip mechanism shown in fig. 33 is used. The figure shows the steps in operating the mechanism as follows.

Step 1: the mechanism is not activated.

Step 2: when the button 780 presses on a spring 782, the latter releases the mount 733 and allows the hinge spring 784 to open the clip 785.

And step 3: paper 786 is inserted between the clip and the housing 787 of the pen holder.

And 4, step 4: the lever 785 (the clip) is against the paper and the spring 788, which is significantly weaker than the hinge spring, gives way and begins to sag. The clip rotates about a rotation point 789.

And 5: the clip presses the paper against the bottom of the holder. The spring 788 sags and both levers drop according to the amount of paper inserted.

A clip button may be used to convert the horizontal motion to vertical motion via a lever to lower the clip. That is, the button may press a lever that rotates to translate horizontal motion into vertical motion, thereby lowering the clip.

This mechanism is illustrated in fig. 34. The clip has two longitudinal slide strips 790, 791 connected to a horizontal strip 792. A spring 793 is provided between the horizontal bar and the housing of the holder. There are two longitudinal guide devices 794, 795 for sliding the longitudinal strips up and down.

In the side view at the bottom of the figure, when the button is not pressed, the spring is all the way up and the clip is pressed toward the barrel casing (left side of the figure). When the button is pressed (to the right in the figure), the spring is pressed and the clip (now looking at the front view) descends between the slide devices. After a paper is inserted and the button is released, the spring pushes the clip upward and the mechanism catches the paper between the clip and the barrel housing.

The button may move a lever along multiple axes to lower the clip.

The button may activate a lever that rotates and moves linearly to lower the clip.

In another arrangement, as shown in fig. 36, the operator presses on a sliding plate (button) 901. The slide button contacts the center of the two top spring rods 903, 905 moving the fulcrums downward in the same direction of the button/plate. The bottom end of the lever is secured around the swivel pins 907,909 so that the bottom end moves down approximately twice the distance of the button. The top terminal end of the lever is fitted with prongs 911, 913 which are received in guide slots 915, 917 and two small protrusions bent vertically up from the bottom clip. The net result is a vertical downward movement of the clip, which is approximately twice the distance of movement of the button/plate.

Alternatively, as shown in fig. 35, the slide plate 930 may be placed horizontally and with the aid of a rigid elastic band, the desired result, i.e. a horizontal pushing movement, opposite to the vertical movement of the push button, is obtained.

Another method may be obtained by a pulling movement of the button 951, as shown in fig. 37, if the force acting on the two levers 953, 955 in the mechanism moves from the middle of the lever to the bottom terminal and the fulcrum of the lever moves to a point 1/3 located a distance from the top terminal to the bottom terminal. This will provide the necessary mechanical advantage to maintain the 2: 1 ratio of the distance the clip moves relative to the travel of the button.

In both mechanisms, this positioning of the point of application of force on the lever relative to the center of rotation can be used to increase the mechanical displacement of the clip.

Pen holder circuit

A block circuit diagram of the holder is shown in fig. 18. An application specific integrated circuit ASIC205 is powered by a battery 511, or by an ac voltage converter 513, or by connecting a USB to a host 211. The two CMOS sensors 201, 203 have outputs 165, 167 connected to the asic by operational amplifiers 169, 171 via a multiplexer 180 and a 12-bit a-D converter.

The analog output of the CMOS sensor is signal processed in the form of offset cancellation and automatic gain control. The signal-to-noise ratio of the process requires the use of a 5v power supply for the CMOS sensor and all analog signal processing circuitry. Some analog-to-digital converters operate with a 2.5v reference, and the signal from the CMOS sensor can be scaled down by a factor of 2 using a resistor divider.

The application specific integrated circuit ASIC may be model class 2B from Sound Vision, located in flamingham MA) and is based on an ARM7 core. The ASIC firmware implements data collection, data storage, file system, I/O services (LED and switch), RS232 and USB communications, power conditioning for idle and sleep modes, optical calibration and test modes.

The multiplexer enables the a-D converter 182 to be alternately implemented between the two CMOS arrays 201 and 203 to minimize the time difference between the two sensors. The clock frequency of the a-D converter is 1.2 MHz. Each CMOS sensor is a 600khz clock. Data acquisition uses the Direct Memory Access (DMA) device of the ASIC.

A wake-up input of the asic is driven by a phase-locked loop 513, the phase-locked loop 513 receiving the input signal from the photodiode. The photodiode is driven by modulated light from the pen.

The ASIC is clocked by a 48mhz crystal and a clock divider 517. I/O features are provided via a USB port 211 and an RS232/IrDA port 209. The firmware and data are processed in the SDRAM207 and stored in a flash memory 519. An optional LCD172 may be provided for the user display.

USB provides two ways of data transfer from the holder to the personal computer: batch and real-time. Real-time transmission is interrupt driven and used as an alternative application to keyboards and mice.

A dual function transceiver is used to implement both RS232 and IrDA communications. This RS232 communication is used as a dial-up connection to a server through a cellular phone.

The holder may support various external connections including USB, serial, parallel, IrDA, Bluetooth, firewall, or any kind of communication port. When powered down, the holder has an automatic power supply capability if it is connected to any external device. It also has the ability to power itself down when an external device is disconnected. There are two connections to the holder:

one connection is an external memory type connection. Such a connection is made with a computer or other device called a host that is capable of displaying graphics and has a user interface that meets the requirements. The holder corresponds to an external memory device when connected to a host device. A user of the host device may browse the file system through the holder, copy, view and edit files previously collected by the holder. Software resident in the host is capable of converting, displaying, printing and editing files stored on the holder on the host screen or copying from the holder to the host. The pen may also be used as a real-time input device when connected to the host.

Another type of connection is the use of a portable network or modem enabled device, such as a cellular telephone. When such a connection is detected, the pen holder automatically begins transmitting all previously collected data in the form of an email or fax.

The optional LCD display informs the user of the status of the pen, for example, regarding connection and download via internet-ready cellular phones for which communication is not reliable. The display screen may be mounted on the upper side of the holder. If an LCD is used, the LED may not be needed.

A single three-color (green, yellow, red) LED170 (see fig. 18) indicates the normal collection of written data, download to a personal computer via cellular phone, and monitoring of battery and memory status.

The pen, clip and reservoir switches 141, 143 and 145 are used to control the ASIC, and a reset switch 147 is used to reset the ASIC.

Fig. 19 shows a state diagram of the operating states of the invention. The shaded blocks indicate the study status. The bright block indicates an intermediate state. In other cases, the map identifies the manner in which the operating status is indicated using a multi-colored LED.

At power up, the green light flashes for the same amount of time when there are pages in memory. When the memory is empty, the green light is not on at power up, and stops flashing if the user starts writing 30 seconds or shortly thereafter.

During writing, the LED is pale green when data acquisition is properly performed. The green light goes off due to a fault detection triggered by, for example, blocking light, the pen leaving the writing surface, or the battery running out.

When no writing occurs, a flashing yellow light indicates a low battery condition. However, when writing, the yellow and greenish lights blink intermittently if the battery capacity is low.

The memory is indicated as almost full by a double flashing of the yellow light when writing is not occurring, and the memory is indicated as almost full by flashing of the yellow and greenish light intermittently when writing is occurring.

After a successful download, the download status is indicated by a bright green light (whether the pen is in the holder or ink reservoir or outside it can be started independently). A flashing green color indicates that a download is in progress. A red light flashes when there is no valid download service or the download signal is weak. For an internet problem, such as when a server is dropped, the red light doubles in flash. A red light flashing three times indicates that the wrong setting for the communication includes the wrong user identifier or server address. This requires that a code be returned from the server to the pen after an unsuccessful match of the data from the pen with the data recorded in the database.

The battery recharge status is indicated by a continuous green light after successful recharge and by a multiple flashing green light when the holder is inserted into the ac rectifier and charged. A combination of a signal from the battery monitoring circuit and a fast charge signal (high when not charging) from a charger can identify the state, whether charging in the process or trickle charging.

The pen can be used during charging. If the pen is removed from the ink reservoir and used during charging, the yellow light is replaced with all of the normal indicators as described above.

The flash status is written by a continuous yellow light indicator.

All faults are reset by activating either the pen or the switches of the ink reservoir in question. The only exception is when the download is successful and the user begins writing. The constant bright green light will then turn into a light green light.

In the sleep mode, all fault indications, low storage capacity and low battery capacity continue in a normal manner. All download failures continue to remain.

The power-up indication gets priority if the ASIC needs to indicate a low storage capacity or low battery capacity condition during power-up. The fault indication is then displayed after a timeout of 30 seconds. If the ASIC needs to indicate a low storage capacity or low battery capacity condition during a download, the download indication takes precedence. The fault indication is displayed after resetting the download state.

In the four holders that are on, the clip switch 141 indicates that the clip is opening and closing in a manner to inform the circuit that the user is beginning a page change. The pen switch 143 indicates when a pen is in or out of the holder. A reservoir switch 145 indicates when the pen is in the reservoir or out of the reservoir. The reset switch 147 is hidden from use a paper clip is accessible through a hole in the bottom.

The pen switch and the ink reservoir switch indicate when the pen is in the holder or the ink reservoir and remove power to the data acquisition and storage electronics when the pen is in the holder or the ink reservoir. The pen switch also opens a new document (or page) when activated, while the ink reservoir switch does not.

The clip switch indicates when the clip is activated and also a page change and the start of a new document (each page is a document).

If the software freezes, the reset switch resets the ASIC. The switch is normally opened as follows:

the pen opens when the pen is outside the holder.

The ink reservoir is opened when the pen is outside the ink reservoir.

The clip opens when the clip button is released.

Reset is turned on when the switch is depressed.

The holder also includes a small connector for USB and RS232 interfaces and an antenna for using Bluetooth (r) or other wireless technology. The USB and RS232 connectors are also connected to the wake up power circuit so that the pen holder can power itself when the cable is plugged into the miniature connector.

The angle signals generated by the sensor are processed by the ASIC and stored in flash memory for transmission to other devices such as cellular phones, PDAs and PCs (not shown) where they are used for handwriting recognition or to capture drawings. For example, the transmission may be performed using USB, RS232, IrDA, or Bluetooth (Bluetooth) protocols.

File system

The flash memory is structured as a FAT (file allocation table) compatible file system, where each file represents a page of handwritten information. Each file has a unique name of 12 characters, including an extension of 3 characters and a separating "dot".

Data file creation

When a user brings the pen into writing mode by taking the pen out of the holder or by pressing the page-changing clip button when the pen is already in writing mode, a new file is created and the subsequent writing is saved in a new file. If the user does not actually perform more writing after creating a new file, the newly created file is deleted and the same file name is reused when the pen next enters writing mode.

During data collection, uncompressed data is stored in a temporary buffer in the SDRAM and compressed by a data storage job before being stored in a file in flash memory. Each page is stored in a separate file. The previous page is compressed before the acquisition of the new page is started.

Data file format

We use a binary compression format based on variable rate huffman coding with a cubic approximation. Such a format includes encoded data coordinates and a time stamp.

Before being compressed, the file has the following format:

the file is constructed with four byte segments. Each segment corresponds to a pixel or a time stamp. Each pixel has a Most Significant Bit (MSB) of zero and consists of two 15-bit numbers, which are the sub-coordinates of the corresponding CMOS sensor. The timestamp is identified by the most significant bit of one and may store the complete date and time of the next pixel (referred to as the complete timestamp) or may store an incremental counter of pixels since the last complete timestamp.

Each file starts with the full timestamp. A time stamp with 1 is inserted at the end of each writing stroke. Because all pixels are scanned uniformly in time, such a combination of timestamps enables the entire history of the handwriting to be effectively recovered in future processing.

Downloading of data

When the holder is connected to a personal computer using a USB cable, the personal computer automatically recognizes the holder as a PC compatible USB device and the contents of the holder file system become visible to the personal computer through the personal computer file system extension. Through which the user can browse and use a handwriting viewer to view the document.

When an RS232 cable is connected between the holder and, for example, a portable telephone, the holder automatically supplies power to itself and starts the transfer of data files from the memory of the holder to the telephone. Infrared data transmission to the phone may also be performed.

The data is sent to the server in compressed form and stored there until requested for delivery to a recipient. It is then decompressed and converted into one of the following formats: GIF,. pdf,. GIF, ps dedicated to email or fax communication services.

Sensor signal preprocessing

In some examples, a pre-processor (not shown) may be used for background removal and storage into flash memory while the application specific integrated circuit ASIC processor performs all communication and I/O functions. The preprocessor may be implemented in the form of a programmable device such as a PLD, FPGA or digital application specific integrated circuit ASIC or a DSP. In this example, a frequency doubling operation is performed to generate a high frequency pixel clock and a clock for the preprocessor according to the LED modulation frequency of the pen recovered by the Phase Locked Loop (PLL).

The second processor may be a processor of another portable device, such as a portable telephone or PDA.

Data acquisition

The target coordinates are acquired in successive sample spaces 10 milliseconds apart to capture sufficient writing motion at a typical 5 centimeter per second rate for a 0.5 millimeter resolution. The ASIC operates as a master, generating the clock and all necessary signals for the sensor.

The sensor in the holder uses the pixel clock from the ASIC. A frame signal is generated by each sensor and read back into the ASIC. Thus, in some implementations, the LED pulses from the pen and the signal acquisition performed on the holder are not synchronized. In other examples, the data acquisition is synchronized to the modulation frequency of the pen. The synchronization effectively improves the angular resolution.

In each sampling period, the position coordinates of the pen are obtained, and data is captured from two sensors. One approach to this background elimination algorithm (asynchronous to the pen) requires the capture of three adjacent frames at each sensor. Another frame is used by the ASIC architecture for sensor reset.

To minimize the amount of coordinate misalignment of the two sensors, the multi-channel data acquisition alternates between the two sensors for each pixel.

The a-D converter is operated at a maximum sampling rate of 1.2MHz and alternates between the two sensors, allowing pixel sampling frequencies up to 600 kHz. Each CMOS array has 1024+4 pixels, which results in a frame rate of approximately 600 Hz. A slow rate of 300Hz can be used to obtain more pixel exposures and correspondingly better signal-to-noise ratio.

Each sensor operates in a mode in which each pixel is reset after being read into the a/D converter.

The infrared LED duty cycle is 50% of the three frame period. For this duty cycle, the LED frequency cannot exceed 200 Hz.

For the purpose of eliminating background noise and low frequency crosstalk without synchronization, three 1024 pixel data frames are required, as described below.

In addition to the main analog output, each CMOS also asserts the END FRAME signal. With each CMOS, the acquisition cycle for each of the three successive data FRAMEs begins with the END FRAME signal, which coincides with the last pixel of the FRAME. Each a-D transition occurs on the falling edge of the PIXEL CLOCK pulse. The total number of dots is substantially (1024+4) × 3, where 1024 is the length of the CMOS array, 4 is the number of clock pulses between the END FRAME signal and the start of the next FRAME, and 3 is the number of consecutive FRAMEs required to implement background compensation.

From the obtained waveform, the asic extracts three arrays, each corresponding to 1024 pixels. The array must be correctly positioned so that the ith cell in each of them corresponds to the ith pixel of the CMOS.

Let us call these arrays A1, A2 and A3. Background compensation is made by the fact that the LEDs in the pen are modulated at a frequency of 1/3 equal to the frame rate and a duty cycle of 50%. To achieve background compensation, the following calculations are performed on the array in a unit fashion: a12 ═ abs (a1-a 2); a23 ═ abs (a 2-A3); a13 ═ abs (a 1-A3). Arrays A12, A13, and A23 are then added in cell form to form a new array, called A.

The array a is1024 units long and carries beam information with background cancellation.

In order to reliably remove large peaks appearing in the pixel waveform during the END FRAME pulse, a sub-array shorter than 1024 cells, for example, three sub-arrays 1020 cells long, starting from pixels 3, 1032, and 2061 (based on 0) may be extracted.

The readout data of the two sensors is digitized simultaneously (or quasi-simultaneously with multiple channels when only one a-D converter is utilized).

Peak position lookup along CMOS array with sub-pixel resolution

Determining the angle at which each sensor receives light depends on the pixel location along the array of sensors at which the peak light intensity is determined. This algorithm for finding peak positions at sub-pixel resolution uses two parameters: t, intensity threshold in volts, and W, window width in pixels. Typical values for these parameters are T-0.1V and W-15.

As an initial step, the peak and its subscript in the array are looked up, referred to as Amax and M. If either of the two Amax values (corresponding to the two sensors) is less than T, then the point is discarded. In this case, the LED is considered to be turned off as the pen does not contact the paper. If M < W/2 or M > (1024-W/2), the point is discarded because it too close to the edge of the video.

A subarray of W elements long starting at element M-W/2 is extracted from A. Its partial center of gravity is found as follows: an array of running sums of the elements of the extracted sub-array (called S) is created. The value of its last element is obtained. It is divided by 2. The partial index for the value position in S is looked up using a linear interpolation/look-up table. M-W/2 is added to this value. This will become the partial index of the center of gravity of the beam in the original 1024 element array. After reversing its sign, 512 (510 for 1024 units long a or 510 for 510 units long a). The result P is the partial position (in pixels) of the beam relative to the axis of the sensor.

The use of a sub-pixel algorithm allows the pixel resolution to be increased by a factor of 8 to 10.

Calculating the angle of the light source relative to the sensor axis

As a result of the previous calculation, we have the angular coordinates (in pixels) of the beam of each sensor. We refer to them as Pleft and right (the sensor is seen from the point of view of the pen endpoint). We recalculate Ps in radians based on the geometry of the sensor. In one example, the pixel pitch L is 7.77 microns, the lens to CMOS distance D is 4800 microns (typical), and the lens material has an index of refraction N (1.5 for glass, 1.4 for plastic, and 1.8 for SF 6). Calibration data for correcting the written image will be used to adjust the parameters, distance D, refractive index N, and horizontal offset and Off.

The angle (in radians) is then calculated as F ═ arcsin (arctan ((P x L)/D)).

As shown in fig. 22, the following parameters are required to calculate the light source position in a cartesian coordinate system:

sensor argument (inclination) C (radian), typically 30/57

Reference B, distance between sensors (mm); typically 150 deg.f

A left sensor: kleft ═ tan (C-Fleft)

A right sensor: kright tan (C + Fright)

X(mm)＝B*Kright/(Kleft+Kright)；

Y(mm)＝Kleft*X

Criterion for accepting a point as a valid point

The points are stored in the form of coordinate pairs (X, Y). When storing points in memory, the coordinates are stored continuously, except as follows:

if the signal is found to be below the threshold (as described above) a flag (pair of single values) is written to the memory, e.g. (NaN ), which will later indicate that the pen is held up (NaN stands for no number, as defined in the IEEE algorithm standard). Then, no new data points are added to the file until the signal is detected again. This method allows the pen to accurately inform the playback program of the trace break restored there.

If the signal is valid but the pen position does not change significantly compared to the previous position, no new data points are added to the memory, but unlike the no-signal case, no marks are written to the memory. The size of the moving square was calculated as (X1-X0)2+ (Y1-Y0) 2. An effective typical value for this moving square is 0.04 square millimeters.

The time stamp is not included in one file because this information is not needed to restore the pen trace.

The coordinates are stored in a temporary buffer and compressed only before being stored in flash memory. Each page is stored in a separate file. Therefore, a mark of an end of a page is not required.

The full timestamp will be inserted before the first active pixel. Every time the pen lifts off the sheet, all other time stamps on a page (document) will be summed by 1 and inserted. Only one time stamp is inserted regardless of how long the pen leaves the paper.

Sleep mode

When the pen is removed from the holder or the ink reservoir for writing, the ASIC is turned on in sleep mode and waits until a light signal from the pen is detected.

When the holder wakes up and detects that the writing is interrupted for a predetermined period of time, the holder returns to a power-saving sleep mode. The ASIC enters sleep mode by reducing its normal 48MHz clock frequency to 750 kHz. The SDRAM refresh rate is also changed accordingly to keep the data intact.

When the pen is within the holder or the ink reservoir, the holder is almost completely powered off. The RS232 receiver and the USB monitoring circuit consume very little standby current. When the circuit is connected to a portable telephone via a cable or to a personal computer via a USB cable, the pen holder is completely disconnected when the pen is within the holder, upon detecting an activation level of RS232 or USB, the circuit wakes up and powers the remaining electronics.

In the sleep mode, the only function of the holder electronics is to wait for a WAKEUP input from the photodiode and associated phase locked loop circuit to indicate that the pen is activated. In sleep mode, when the pen verifies the photodiode, it consumes little power between the periods.

During writing, the pen transmits IR pulses that have been modulated. The pulse is detected at the holder causing the phase locked loop to wake up the processor and start normal acquisition mode once the ASIC has switched to the 48MHz system clock.

Phase-locked loop (PLL)

When the modulated IR light from the pen is detected, the pen LED's modulation clock (represented by the 1kHz pulse train in the output light) is extracted by PLL circuit 132 to adjust to the infrared light's modulation frequency.

All collected data is initially stored by DMA to SDRAM 134. The refresh rate of the SDRAM remains unchanged from the acquisition mode to the sleep mode. This storage requires 1 Mbyte of data for 50 pages that have been compressed or 10 pages that have not been compressed. The 5:1 compression algorithm must have fast and computationally simple encoding without limitations on decoding.

The data collected during writing is initially stored in SDRAM. When the pen is returned to the ink reservoir or the pen, or page switch 136 is activated, the ASIC writes all data from the SDRAM to flash memory 138. Writing a full page of handwritten text data to flash only requires a short time. The transmission is indicated to the user on the holder by illuminating a yellow LED 140.

An 8 mbit flash memory stores a compressed file representing a maximum of 50 pages of handwritten text. The compression algorithm allows at least 6 to 1 compression without visible text distortion.

Power supply for the holder

The holder provides 3.0V power through two AA NiMH batteries in series. The three nickel cadmium batteries of the pen are charged by trickle when the pen is in the ink reservoir or holder. The battery of the pen has a large capacity and has hardly been fully charged. The trickle charge is sufficient to maintain the battery charge. A special mode is provided when both the pen and the pen holder are in the charger to charge all batteries including the pen battery with the full charge current.

Battery life is the average amount of ten handwritten pages or one week without data compression for storage in memory. A typical user can write 2 characters per second, i.e. 120 characters/minute, i.e. 7200 characters/hour. A typical handwritten page is approximately 700 characters. To produce ten pages, the battery must operate for 5 hours.

When connected to a USB port, the holder may draw power from the USB host. The charge on the battery is maintained at a sufficiently high level to enable the circuit before transitioning to USB power. Power from the USB connector is provided only after the ASIC establishes communication over the USB link and informs the personal computer at the other end of the USB link that the connection is high power. In response, the personal computer provides up to 0.5A of power. The battery charge current is set at 0.4A and monitored to convert the charger to a trickle charge.

The pen holder circuitry is activated when the pen is removed from the holder or ink reservoir. Some support circuits such as RS232 drivers and wake-up power circuits draw power directly from the battery.

Other circuits draw power from a 3.3v power supply generated from a battery voltage of 2 to 3 volts by an on-board switching regulator. When connected to a USB link, the 3.3v is generated from the USB power source.

5v is generated for the analog circuit from the 3.3v power supply.

Synchronization of pen and holder

Synchronization of the pen and the pen receiver may result in better signal resolution and correspondingly better written resolution and angular resolution.

As shown in fig. 20, for synchronization, the pen generates a periodic train 401 of pulses of relatively high frequency, such as 1-10kHz pulses (as a suggestion we show some timing diagrams), which can be easily detected by a phase locked loop. The phase-locked loop PLL detects not only the actual modulation clock but also its phase, which can generate a signal for initiating data acquisition and synchronizing it with the pen LED.

As shown in fig. 21, the control signals LED _ ON and LED _ OFF trigger signal acquisition. In this case, only two frames are required for background removal, one for the IR signal when the pen LED is on and the other for the IR signal when the LED is off.

For a CMOS sensor, a shutter mode is provided to reset all pixels simultaneously.

Having only two frames per sample improves the sampling rate and resolution and may allow the processor to enter an idle mode between samples to save power.

Use of 2-dimensional CMOS arrays

Suppliers to manufacturers of digital sensors produce small, power-saving sensors and sensors with image processing circuitry that can be integrated into paper pen or three-dimensional pen applications.

Three-dimensional positioning of the light spots may use two-dimensional photo arrays. The projection of the light spots on the two planes defines a single point in 3D space. The motion of an IR stylus can control a three-dimensional object on a personal computer screen while a three-dimensional locus of positions is active. When the pen is moved in space, it drags or rotates the object in any direction.

Slave MODE (SLAVE MODE)

In other implementations, the application specific integrated circuit ASIC based ARM7 is used in a slave mode, the DMA can handle data acquisition but the vertical synchronization signal is provided by the light detection circuit (PLL).

Two alternative analog channels

Two separate channels may be used for analog signal processing and a-D conversion. Such an implementation may use more economical components that do not require fast setup times, bandwidths, and slew rates.

Frame change replacement

CMOS sensors have a limited dynamic range. Although a tunable electronic gain may be used by both CMOS after the CMOS output, this approach may not be ideal for two reasons.

First, the signals for the two CMOS may differ in magnitude when the pen is moving over a certain area of the paper, so changing the gain of both may fix one signal while lowering the other signal to an unacceptable level. To obtain the proper signal on the page, it is useful to have a separate gain for each CMOS. Secondly, using an electronic gain does nothing to prevent actual CMOS saturation, which is inevitable for the area that the sensor must cover.

The gain of the CMOS can be varied by independently varying the irradiance of each CMOS. As shown in FIG. 21, the pen transmission rate is maintained at 100Hz, while the CMOS frame rate 601 is shifted among 300Hz, 600Hz, and 1200 Hz. At 300Hz, the background subtraction is straightforward. For 600Hz, the algorithm uses every other frame (frames 1, 3 and 5). For 1200Hz, the algorithm uses every fourth frame (frames 1, 5 and 9). The pixel rates are 300kHz, 600kHz and 1.2 MHz. Changing the frame rate can be done by an application specific integrated circuit ASIC without any other hardware.

Each CMOS may be connected directly to its own analog-to-digital converter ADC, or both may be connected through to an analog-to-digital converter ADC that will be able to process 1.5 mega samples/second and have a 4V reference voltage. The analog-to-digital converter ADC may then be connected to a digital multiplexer for feeding the signal into the application specific integrated circuit ASIC.

Method-based PSD

Instead of a CMOS array, two PSDs can be used to detect the infrared light from the pen. Each PSD determines the angle between the line of sight between the pen and the PSD and the page. The two angles determined by the two PSDs and the distance between the PSDs are sufficient to calculate the position of the pen tip.

Even with the use of infrared filters, ambient light will introduce errors in the PSD position measurement. To reduce this error, the infrared of the pen is modulated to produce pulses at a modulation frequency and with a 50% duty cycle, as described above.

Two analog techniques can be used to discriminate the PSD signal that is converted to angles for triangulation.

In one method, called synchronous demodulation and used in test electronics, the incoming synchronous light pulse is split at the light modulation frequency and opposite gains (+ 1 and-1 respectively) are applied to those signals depending on whether the LED is on or off. This can subtract out background noise. The signal is then integrated using a time constant responsive to the change in the signal on the one hand, and the noise on the other hand, is balanced. In one example, the modulation frequency may be 3kHz and the pulse amplitude may be ILEDpeak Xma.

A method for deciding to use a sample and hold technique. The shape of the optical signal has a 50% duty cycle at a modulation frequency of 3kHz, as previously described, and a significantly intense short pulse applied at the modulation frequency. The modulation frequency is determined by a phase locked loop PLL and used to trigger the sample and hold circuit, while the intense light pulse is actually sampled. The pulse amplitude is ILEDpeak Xma and the pulse duration T Y usec.

The PSDs are extremely accurate in sensing and measuring the position of light on their photosurfaces. They are inexpensive and require only very little power consumption. The implementation of the PSD is also simpler than CMOS.

As shown in fig. 23, current-to-voltage conversion is performed on each of the four channels, two for each PSD. The four analog signals are low frequency filtered 605, synchronously detected 607, integrated 609 and digitally converted 611 by a microcontroller 613(12 bit analog to digital converter a/D). The a-D conversion is performed at a 100Hz sampling rate. The processor is activated when the pen makes a mark on the paper. The processor performs signal acquisition and stores periodically into flash memory. FIG. 24 shows a system sequence diagram. The microcontroller enters an idle mode when there is no manufacturing mark and enters a sleep mode after an arbitrary period of time.

The microcontroller is awakened from an idle mode or a sleep mode by an interrupt or poll (TBD) input from one of: one of the four analog channels when the signal at its modulation frequency exceeds a certain threshold of the comparator; an interrupt from a USB port when the presence of activity from a host is detected; one of its buttons is pressed.

The processor writes data from the RAM to the flash memory when the RAM becomes full or/and reaches a boundary condition for a flash memory page. If the acquisition continues and the page is full, the microcontroller starts writing to the flash. However, when there is no write, most of the flash operations should be performed during idle periods.

Each PSD has two dual channels of analog signal processing. Each channel has a current to voltage converter whose output is ac coupled to a first gain amplifier. This signal is shunted with the modulated frequency of the pulse generating IR LED (on the pen), currently 1 kHz.

The chopper has a gain of +1 when the LED emits light. When there is no lamp light, the gain is-1, so the signal is synchronously demodulated. The final stage is an integrator whose output is close to dc. More precisely, it is a sawtooth waveform that is charged and discharged due to the integrating capacitance in the feedback of the amplifier.

The analog-to-digital converter a/D, either a PC-based DAQ or an analog-to-digital converter a/D of the microcontroller, samples the output synchronously at the modulation frequency at given time intervals to eliminate errors due to sawtooth waveforms.

To use all 12 bits of the a/D resolution of the analog-to-digital converter, a dynamic change of the reference voltage of the converter is used. The microcontroller always reads the analog-to-digital converter channel with the highest range and then halves it until the range is optimal for the signal.

The chopper amplifier uses a replica of the modulation frequency detected with an analog circuit on each channel (four channels total). The signal is acquired after the second gain stage and processed for signal transition detection, and the recovered modulated pulses are then passed through an or gate to drive the chopper amplifier analog switches to change gain between +1 and-1.

Rotating nib using phase shift of photodiode

If a rotating light source 617 is used at the pen tip, as shown in FIG. 25, the phase difference between the signals on the three photodiodes 619, 620, 621 on the holder can be measured to find the pen position.

The rotating light on the pen tip can be implemented using several (e.g., eight) LEDs 623 that are activated at spaced apart times by T/N, where T is the entire period of the LED cycle and N (e.g., 8) is the number of LEDs.

The signal source is located at a position on an X-Y plane. The two signal detectors 619, 620 are located at two other fixed locations on the same plane. If the source has a radiation pattern such that the signal emitting light in the positive X-axis direction is in phase quadrature (rotated in the signal frequency space) with the signal emitting in the Y-axis direction, and the propagation delay is negligible compared to the signal period, then the angle a1 formed by the two intersecting lines 637, 639 trailing from the detector towards the source will be the same as the phase difference between the signals measured at the detector.

If a third fixed position detector 621 is added, a second angle A2 will be formed because the three lines cross at the signal source. Likewise, the angle a2 between the lines at the intersection will be the same as the phase difference of the measured signal between the receivers. By applying some basic triangulation, the location of the signal source in the X-Y plane can be found by a known fixed position of the detector and measuring the phase difference of the signals of the detector. This calculation becomes insignificant if the three detectors are arranged along a straight line with equal distances between them.

Referring to fig. 26, B and a angles are calculated from the angles measured by the sensors, "a" and "B" as follows:

comprises the following steps: a/A ═ d/R (1)

And B/B ═ d/R (2)

And B + a + B + a ═ 180o (3), according to the basic geometric theorem,

we obtained: b/a ═ B/a, (4),

and correspondingly B ═ A × B/a (5)

A＝B×a/b(6)；

Now substituting (3) into (5) and (6) we get:

axb/a + b + a ═ 180o (7) and

B×a/b+B+b+a＝180o(8)；

we use them to solve a and B:

A＝a×(180o-b-a)/(a+b)(9)

B＝b×(180o-b-a)/(a+b)(10)

the rotating light on the pen tip can be implemented using several (e.g., eight) LEDs 623 that are activated at spaced apart times by T/N, where T is the entire period of the LED cycle and N is the number of LEDs. For example, eight Light Emitting Diodes (LEDs) may be arranged in an outwardly directed circle, spaced 45 degrees apart and driven by a single oscillator with a 45 degree phase difference between adjacent LEDs.

As shown in fig. 27 and 28, the three detectors 641, which may be positive-intrinsic-negative (PIN) diode photodetectors, drive a signal processing chain 642 of a transimpedance amplifier 643 and a high gain limiter 645 to remove any amplitude modulation in the detected signal.

Phase detection may be accomplished with two edge-triggered one-bit up-down counter type phase detectors 649, two binary counters, and a clock running a number of decimals at the frequency of the signal. If the counters are connected such that they count up by 1 per clock cycle in which the phase of one sensor is advanced with respect to the other, and count down for each clock cycle in which the phase of one sensor is delayed with respect to the other, and a third counter is provided to count up by 1 consecutively, a microprocessor can periodically read and reset all counters, by reading the consecutively running counters, scaling the readings from the two counters connected to the phase detector (driven by it). This number is the phase difference (in slope) between the three sensors and is the same as the angle between the intersection of the lines from the sensors to the source. It is then a trivial task to calculate the position of the source relative to the sensor.

Pen light activated switch replacement

Different pen light activation methods may be used, including conductive rubber, pressure sensitive materials or strain gauges.

Pressure sensitive materials allow a varying pressure threshold and coordination between the switching point and the ink flow. This will prevent data loss when the ink makes a trace but the pen is not inactive. For example, most ballpoint pen cores release ink at 20 to 30gf +/-30%, while one current switch is activated at 50 to 100gf and +/40gf, making reliable coordination of ink flow and data acquisition impossible. The dedicated cartridge may be designed to prevent ink flow below 50gf so that current chip switches may be used.

Pen optics replacement

Another method may be used for emitting light from the nib. Optical fibers may be used to collect light from an LED and emit the light in a 360 ° fashion around the nib. Individual LED chips may be placed around the nib and emit light through a semi-reflective lens/window such that 50% of the light is emitted and the other 50% is internally reflected to mix with the other light, ultimately producing a uniform 360 ° illumination. A dedicated ring may be used to mix the light from a single LED to redistribute the light for uniformity.

Passive pen refill

If the IR light source is placed next to the sensor, the pen may be completely passive. A reflective surface is provided near or at the tip. The sensor will see the reflected light from the infrared of the pen tip and calculate the angle as described above.

The nib must reflect light only when pressure is applied to the paper and the ink forms a mark. Otherwise there will be false marks in the form of numbers without corresponding marks on the paper.

The activation of the light reflecting mechanism may be mechanical or electrical. In a mechanical implementation, pressure on the nib will open a sheath and expose the reflective surface around the nib. In an electrical implementation, pressure on the nib will activate a liquid crystal or other electro-optical technology that causes the material to reflect light. Reflected light from other objects, such as fingernails and rings, can be addressed by using polarized infrared light.

Passive pen holder

Conversely, the holder may have two mirrors, while the pen both emits light and receives reflected light. The sensing element on the pen may be a 2-dimensional PSD or CMOS array. If a planar two-dimensional sensor is used, the pen will not be omni-directional, but it may form a custom circular two-dimensional sensor, which will have 360 ° coverage.

Keyboard and mouse replacement architecture

As mentioned above, the pen can be used to replace standard PC input devices such as a mouse and keyboard.

When used as a replacement for a keyboard or mouse, paper, plastic or other flat surface may carry a printed keyboard pattern for use as a keyboard and mouse pad for, for example, personal computers, portable computers and cellular phones.

Today, when inputting data into a personal computer, portable computer or cellular telephone, users are mostly limited to keyboard, keypad or finger touch input on the screen. It is efficient and convenient when the keyboard is of a real size, but not for portable devices such as palm top computers and portable phones. The keypad of a cellular telephone is effective for dialing telephone numbers, but requires excessive keystrokes in an attempt to generate ASCII letters and symbols, making any type of data entry a very tedious and time-consuming process. Finger touch input on the screen of a palm device requires the user to use a single writing style such as "graffiti" (rough carving), or requires the user to tap on a virtual keyboard displayed on the screen, in order to minimize the amount of handwriting recognition required by the device. Both finger-touch methods often produce incorrect inputs, limiting the functionality of the device.

The electronic pen can be used for entering text characters and for recording handwritten images and lines by providing a highly reliable mode of the method. All that the user needs to type in with a pen is a piece of paper or any other surface (with or without a printed pattern of the keyboard).

The electronic pen along with the spatial transcription capabilities of the keyboard template are used in place of a mouse or keyboard.

The paper keyboard may be of various sizes depending on the needs of the user. The size can vary from 81/2 x 11 sheets to the size of a cell phone cover. The user first selects the keyboard size he desires and then calibrates by touching the pen to the designated characters on the keyboard. To type a message, the user touches the pen tip on the appropriate key. When the pen touches a square area of the paper corresponding to a certain letter, the position of the pen tip is calculated and the designated letter is determined. The method allows the user to generate text on a computer with an electronic pen without relying on any handwriting recognition software.

This method is an improvement over the current embedded keypad, software keyboard, or on-screen finger-touch input methods on increasingly smaller personal devices. The paper keypad allows the user to type messages on handheld and portable phones more quickly and reliably than alternative methods. The paper may also be used in other modes to record drawn and handwritten notes and images. When completed, the paper keyboard may be discarded or folded for future use.

The keyboard may contain shortcut and function keys in addition to characters, enabling more efficient interaction with small devices. The shortcut keys may minimize the number of keystrokes required to activate the command. The shortcut keys may be customized according to the type of device being entered.

The keyboard may also include a portion that functions as a paper mouse pad. By using the spatial transcription capabilities of the electronic pen, the user can move within a specified square space on the paper so as to subsequently move a point on the screen of a device. Therefore, the paper mouse is used as a substitute for the buttons and the finger touch on the screen as a device for navigating on the screen of a handheld or portable telephone.

The paper keyboard also has the flexibility to enter foreign characters. Keyboards can be created for several languages, such as japanese, korean, spanish, french, and russian. If it is desired to enter a different letter, the user may simply print out a new keypad.

As shown in fig. 29, to type a message, the user touches the tip 701 on the appropriate keys 703 printed on a paper keyboard 705. The touched position of the pen is tracked by the tracker 707, converted to text and sent to a portable device such as a portable phone 711 or a palm-top computer 709.

After use, the paper keyboard may be folded or discarded.

In other implementations, the keyboard is made of paper that is not printed; rather, when the pen is used to write on any surface, the pen's motion is tracked via handwriting recognition to derive text, commands, and drawings.

In one implementation of the method, the mouse and keyboard may continue to be used while the pen is used as a replacement. The pen may be operated in a "pointing device" (mouse) mode or a "character input" (keyboard) mode. This mode may be selected by the pen holder or a dedicated hardware switch or button on the pen, or by sending a command from the personal computer to the holder.

In the mouse mode, pen operation is indistinguishable from operation of a second (USB) mouse. It is a relative positioning pointing device that moves a cursor on a screen. In keyboard mode, pen input may be received by a specially designed application that uses an efficient character recognizer to convert graphical input (strokes) into characters. Other applications are unaware of the presence of the pen and continue to operate using a conventional (legacy) keyboard.

In another approach, the pen is the only input device to the system. In this case, a software driven stacked memory is modified to provide system wide keyboard functionality. Mouse mode operation does not affect and is consistent with the first described method. When operating in keyboard mode, pen input is recognized by active handwriting recognition software embedded within the keyboard filter driver and then passed to the system input queue in a manner similar to conventional keyboard input.

This second approach requires a platform availability model and (likely) modification of some system component, such as the basic input/output system (BIOS).

Both approaches increase artifacts and usability problems. Specifically, there are two basic approaches to handwriting recognition: discrete (each individual character) and continuous (each word, phrase or page). In the former case, the user must continuously rely on computer screen output for feedback. This can be inflexible as the fixation of the computer screen for feedback must often interrupt the handwriting processing. In the latter case, the user only occasionally has to look at the screen, when a written unit (word or phrase) is completed and corrected as necessary.

Switching from a mouse to a pen input mode may be performed by a retractable pen core action. When the new supplement is inside the pen (pen cannot write), it is used as a mouse. The pen acts as a keyboard when writing is activated.

Other embodiments are within the scope of the following claims.

The holder need not include an ink reservoir as described above, but may be any kind of device that can accommodate the sensor. The holder may be a simple cap, as previously shown, or may be any other type of device, whether it is a tight fit or covers the pen, and whether it includes a clip. For example, the holder may incorporate a tablet with a paper clip or a notebook.

The light in the pen may be optical fibres which transmit the light to the pen tip and transmit it in a pattern, such as a disc, in all directions around the pen.

Claims (22)

1. An apparatus, comprising:

a writing instrument; and

a holder having a socket for receiving at least a portion of the writing instrument for storing the writing instrument;

the writing instrument and the holder contain respective elements capable of wirelessly transmitting a signal associated with the movement of the writing instrument and tracking the movement in accordance with said signal.

2. The apparatus of claim 1, wherein the writing instrument includes a light source configured to transmit light via a vacuum path to at least one light sensor spatially separated from the writing instrument.

3. The apparatus of claim 2, wherein the light source comprises a light pipe.

4. The apparatus of claim 3, wherein the light pipe is configured to direct light parallel to the writing surface for any orientation of the writing instrument relative to the writing surface.

5. The apparatus of claim 3, wherein the light pipe is configured to internally reflect and concentrate light and emit light by reflection from a reflective outer surface.

6. The apparatus of claim 5, further comprising a mechanism that enables the reflective element to reflect light when the writing instrument is being used for writing and that disables the reflective element from reflecting light when the writing instrument is not being used for writing.

7. The apparatus of claim 5 wherein the reflective outer surface comprises a conically curved surface angled with respect to a longitudinal axis of the writing instrument.

8. The apparatus of claim 3, wherein the light pipe comprises a cylinder having an upper surface for receiving light and a lower annular surface for reflecting light to the at least one light sensor.

9. The apparatus of claim 2, wherein the light source emits light in a direction toward a writing end of the writing instrument.

10. The apparatus of claim 2, wherein the light source comprises one or more LEDs.

11. The apparatus of claim 2, wherein the light source comprises a ring of light sources.

12. The apparatus of claim 1, wherein the writing instrument comprises: a light source configured to transmit light to a plurality of sensors included in the holder; and a device configured to turn the light source on and off.

13. The apparatus of claim 12, wherein the device operates in response to pressure applied between the writing instrument and a writing surface.

14. The apparatus of claim 13, wherein the device comprises a switch configured such that the amount of pressure required to activate the switch is not so great as to disrupt normal writing movement of the writing instrument on the writing surface.

15. The apparatus of claim 12, wherein the writing instrument comprises a ball point pen barrel having a writing tip, and the device is disposed at an end of the barrel distal from the writing tip.

16. The apparatus of claim 13, wherein the device comprises a switch.

17. The apparatus of claim 13, wherein the device comprises a pressure sensor.

18. An apparatus as set forth in claim 1, wherein the holder includes a mechanism for attaching the holder to a writing board.

19. The apparatus of claim 18, wherein the mechanism is configured to attach the holder to the writing board in a direction that enables the member to receive the optical signal emitted by the writing instrument.

20. The apparatus of claim 18, wherein the attachment mechanism includes a switch for activating a function of a processor in the holder when the attachment mechanism is operated.

21. The apparatus of claim 18, wherein one of the functions is a new page function.

22. The apparatus of claim 1, wherein the writing instrument includes a rechargeable battery, and the holder includes a charging circuit connected to charge the battery when the writing instrument is in the holder.