CN1114147C - "Eyes" Mouse for Computer Systems - Google Patents

Abstract

An optical mouse (1, 15) images the spatial features of generally any micro-textured or micro-fine work surface (5) beneath the mouse in the form of an array of pixels (10). The response of the photosensitive detector (10) is digitized and stored in the form of frames in a memory. The motion produces successive frames of a transferred pattern of pixel information that are compared by autocorrelation to determine the direction and amount of movement.

Description

"Eyes" Mouse for Computer Systems

This application is related to the subject matter described in the following two U.S. patents: 5,578,813, filed March 2, 1995, and published November 26, 1996, entitled "Freehand Image Scanning Apparatus Compensating for Nonlinear Movement"; 5,644,139, Applied on August 14, 1996, published on July 1, 1997, titled "Navigation suitable for detecting movement of navigation sensor relative to target". Both patents have the same inventors: Ross R. Allen, David Beard, Mark T. Smith, and Barclay J. Tullis. Both patents were assigned to Hewlett-packad Corporation. This application also relates to what is described in another U.S. patent application: <currently unknown... S/N08/540,355 permitted but not yet published> filed October 6, 1995, entitled "Method and System for Tracking Attitudes", This patent is also assigned to Hewlett-packard Company. These three patents describe techniques for tracking location movements. These techniques are an integral part of the preferred embodiments described below. Accordingly, US Patents 5,578,813; 5,644,139 and <such as published as S/N 08/540,335> are hereby incorporated by reference.

BACKGROUND OF THE INVENTION Hand operated pointing devices adapted for use with computers and their displays are very common. By far the most popular of the various devices is the ordinary (mechanical) mouse. Conventional mice typically have a bottom surface loaded with three or more downwardly projecting pads of low friction material that raise the bottom surface a short distance above a mating mouse pad working surface. Located in the center of the bottom surface of the mouse is a hole through which a lower portion of a leather-like steel ball (hereinafter referred to as the rubber ball) protrudes; during operation, the ball is pulled down to the upper surface of the mouse pad by gravity. Mouse pads are generally dense mesh foam rubber covered with a suitable fabric. Those low-friction pads slide easily on the fabric, but the rubber ball doesn't slide, it rolls when the mouse is moved. Inside the mouse are scroll wheels, or pulleys, that contact the ball at its great circle (the great circle parallel to the bottom surface of the mouse) and convert its rotation into an electrical signal. The housing of the mouse is constructed so that it appears to have a "front-to-back" axis (along the user's forearm) and an orthogonal "side-to-side" axis when it is covered by the user's hand. The inner pulleys that are in contact with the great circle of the ball are arranged so that one pulley responds only to rolling of the ball, which takes the component of movement of the mouse along the front-to-back axis, and so that the other pulley responds only to rolling resulting from the component of movement along the left-right axis. . The resulting rotation of the pulley or contact roller generates electrical signals representative of these moving components. (For example, F/B stands for front or back, and L/R stands for left or right.) These electrical signals F/B and L/R are connected to a computer, where software responds to these signals by pressing Δx according to mouse movement. and Δy change the position displayed by the pointer (cursor). The user moves the mouse as needed so that the displayed pointer reaches the desired location or position. Once the on-screen pointer points to an object or location of interest, one of the one or more buttons on the mouse is activated with the finger of the hand holding the mouse. The initiation serves as an instruction to take some action, the nature of which is dictated by software in the computer.

Unfortunately, mice of the general kind described above suffer from a number of disadvantages. Among these disadvantages are deterioration of the mouse ball or damage to its surface, deterioration or damage to the surface of the mouse pad, and reduced ease of rotation for the contact roller (for example, (a) due to accumulation of dirt or lint, or (b) Because of wear, or (c)(a) and (b) both). All of these situations can be the cause of erratic or complete failure of the mouse to function when desired. These events can be quite frustrating for the user, who complains perhaps that he cannot move the cursor, say, down, although the cursor on the screen moves in all other directions. Accordingly, the industry has committed to making removable mouse balls for easy replacement and cleaning of the ball recess. Enhanced mouse ball hygiene was also a major motivation for the introduction of mouse pads. However, some users absolutely loathe their particular mouse at the moment when these supplementary balls seem to be of no avail. Mouse and mouse pad replacement is a living business.

The glaring reason for this whole malaise is that the common mouse is essentially mechanical in its construction and operation, and relies to a large extent on a rather careful handling of how mechanical forces are generated and transmitted.

There have been several earlier attempts to replace mechanical methods with optical methods. These include photosensitive detectors that respond to mouse movement on specially marked mouse pads and that respond to movement of specially striped mouse balls. US Patent 4,799,055 describes an optical mouse which does not require any special pre-marked surfaces. (Its disclosure of two linear arrays of one-pixel-wide photosensors orthogonal to each other in the X and Y directions and its state machine motion detection mechanism make it a distant cousin of the introduced patented technology, but our opinion is , shifted and correlated array (pattern of pixels in an area) technique is significantly more complex and laborious.) To date, none of these earlier optical techniques has been widely adopted, although users have been frustrated by mechanical mice for over a decade. Accepted as a satisfactory replacement for common mechanical mice. Thus, if there were a non-mechanical mouse that was viable, relatively cheap, and reliable from a manufacturing standpoint; hopefully. This need may be met by a new type of optical mouse with a familiar "tactile feel" and unsurprising behavior. It would be even better if this new optical mouse was not dependent on mating with a mouse pad (whether dedicated or not), but could navigate on almost any arbitrary surface.

One solution to the problem of replacing the common mechanical mouse with an optical counterpart is by directly imaging various specific spatial features of the working surface beneath the mouse in the form of an array of pixels, much as human vision is Think do that. Broadly speaking, such a work surface can be almost any flat surface; in particular, the work surface need not be a mouse pad, dedicated or not. For this purpose, the working surface below the imaging unit is illuminated from the side, for example, with infrared (IR) light-emitting diodes (LEDs). A fairly wide variety of surfaces produce dense collections of lights and shadows when illuminated at appropriate angles of incidence. This angle is generally very low, say, on the order of 5 to 20 orders of magnitude, and we will call it the "grazing" angle of incidence. Paper, wood, bakelite, and lacquered surfaces all work well; about the only surface that doesn't work is smooth glass (unless it's covered in fingerprints!). The reason these surfaces work is that they have a microtexture that is impossible for the human senses to perceive without any tools.

The IR light reflected from the microtextured surface is focused onto a suitable photosensitive detector array (say, 16x16 or 24x24). The LEDs can be illuminated continuously for a steady or variable amount of illumination that is servoed to maximize some aspect of performance (eg, the dynamic range of the photosensitive detector together with the reflectance of the work surface). Additionally, the charge accumulation mechanism coupled to the photosensitive detector can be "shunted" (with a current shunt switch); LEDs can be pulsed on and off to control exposure by servo averaging. Turning off the LED also saves power; in a battery operating environment, this is an important consideration. The response of each photosensitive detector is digitized to the appropriate resolution (say, 6 or 8 binary bits) and stored as a frame in the corresponding cell within the array memory. Having thus given our mouse an "eye", we go a step further by equipping it to "see" movement by comparing it to successive frames.

The size of the image projected onto the photosensitive detector is preferably a slight magnification, say 2 to 4 times, of the original feature being imaged. However, if the photosensitive detector is small enough, it would be possible and desirable to avoid amplification. The dimensions of the photosensitive detectors and their spacing are such that each image feature is likely to have one or several adjacent photosensitive detectors, rather than the other way around. Thus, the pixel size represented by each photosensitive detector corresponds to a spatial region on the working surface that is generally smaller than the size of a typical spatial feature on that working surface. A spatial feature might be a fiber in the cloth covering a mouse pad, a fiber in a sheet of paper or cardboard, a subtle variation in a paint finish, or a raised microtexture on a plastic laminate . The overall size of the photosensitive detector array is preferably large enough to receive images of several features. It can be seen that the image of such spatial features produces a pattern of pixel information that is translated as the mouse is moved. The number of photosensitive detectors in the array and the frame rate at which their contents are digitized and captured combine to affect how quickly the eye-eye mouse can be moved across the work surface and still be tracked. Tracking is accomplished by comparing a newly captured sampled frame with a previously captured reference frame to determine the direction and amount of motion. A feasible method is to test the 8 directions allowed by a pixel offset in turn (one over, one over and one down, one down, one up, one up and one over, in the other direction Each direction in one of the above, etc.), shift the entire content of one of these frames by a pixel distance (corresponding to the photosensitive detector). This amounts to eight heuristics, but we must not forget that there won't be any more motion, so we also need a ninth heuristic "zero shift". After each tentative shift, those portions of the frame that overlap each other are subtracted on a pixel-by-pixel basis, and the final differences are (preferably squared) summed to form the similarity (correlation ) measure. Of course, larger tentative shifts are possible (e.g., 2 over and one down); however, at some point the attendant complexity destroys convenience, so it is better to have small tentative shifts and high Frame rate. The tentative shift with the smallest difference (largest correlation) can be taken as an indication of motion between two frames, ie it provides a raw F/B and L/R. Raw movement information can be scaled and/or accumulated to provide display pointer movement information ([Delta]x and [Delta]y) of convenient quantification and with appropriate information exchange rates.

The actual algorithm described in the cited patent (and used by the eye-eye mouse) is an improved and refined version of the above algorithm. For example, let's say the photosensitive detector is a 16x16 array. We can say that we initially take a frame of reference by storing the digitized value of the output of the photosensitive detector occurring at some time t ₀ . At a later time t ₁ , we take a frame of samples and store another set of digitized values. We wish to correlate a new batch of nine comparison frames (considered zero, one over, one over and one up, etc.) to a reference frame pattern representing "the last time we were". Our comparison frames are temporally shifted versions of sample frames; note that a comparison frame will no longer exactly reselect the reference frame when moved. One side, or two adjacent sides will not match as well as the original. Pixel positions along a non-matching edge will not contribute to the corresponding correlation (ie, for a particular shift), but all other pixel positions will. These other pixel locations are then a substantial number of pixels that yield a good signal-to-noise ratio. For "nearest neighbor" operations (i.e., limited to zero, over direction, up/down direction, and combinations thereof), the correlation yields nine "correlation values" that can be derived from A pixel location of , which is indeed paired with a pixel location in another frame - a non-matching edge will have no such pairing) is derived as the sum of the squared differences of all pixel locations.

A brief note perhaps in turn about how the shift is done and how the associated value is obtained. Shifting is accomplished by addressing offsets to the memory, which can simultaneously output an entire row or column of an array. Dedicated computational circuitry is coupled to the memory array containing the shifted reference frame and the memory array containing the sampled frame. Correlation value formation for a particular tentative shift (nearest or nearest neighbor set component) is done quickly. The best mechanical analog is to imagine a transparent (reference) membrane of light and dark patterns arranged as if it were a checkboard, except that the arrangement is perhaps random. Now imagine a second (sample) film with the same general pattern superimposed on top of the first, except that it is a negative image (light and dark swapped). Now the pair of membranes are aligned and raised towards the light. As the reference film is moved relative to the sample film, the amount of light entering through the composition will vary according to the degree of image superposition. The configuration with the least amount of light coming in is the best correlation. If the negative image pattern of the reference film is one or two squares displaced from the image of the sample film, then the configuration of minimum light entry will be the configuration commensurate with that displacement. We pay attention to which displacement enters the least light; for eye-eye mice, we notice the configuration with the best correlation and say the mouse moves that much. In fact, this takes place within an integrated circuit (IC) with photosensitive detectors, memory and arithmetic circuitry arranged to perform the image correlation and tracking techniques we are describing.

It would be desirable if a given reference frame could be reused for successive sample frames. At the same time, each new set of nine (or 25) correlation values (at ti, ti+1, etc.) derived from a new image at the photosensitive detector should contain a satisfactory correlation. For a handheld mouse, a contiguous set of several comparison frames can usually be obtained from a (16x16) reference frame taken at _t0 . What allows this to be achieved is maintaining the orientation and displacement data of the most recent movement (this is equivalent to knowing the velocity and the time interval since the previous measurement). This allows "prediction" how to (permanently) shift the set of pixels in the reference frame, so that for the next sample frame a "nearest neighbor" can be expected to make a connection. This shift that provides prediction discards or removes some reference frames, resulting in reduced reference frame size and associated statistical quality. When one edge of the shifted and reduced reference frame begins to approach the center of the original reference frame, it is time to take a new reference frame. This mode of operation is called "prediction" and can also be used for 5x5 comparison frames and an extended "nearest neighbor" (zero, 2 over/one up, one over/two up, One over/one up, 2 over, one over...) algorithm. The benefit of prediction is to speed up the tracking process and reduce the percentage of time spent acquiring reference frames by streamlining the internal correlation procedure (avoiding the comparison of two arbitrary correlated 16x16 arrays of data).

In addition to the usual buttons that mice typically have, our sight-eye mice can have another button that pauses to generate a movement signal to the computer, allowing the mouse to be physically repositioned on the work surface without disturbing the position of the pointer on the screen. This button may be needed if the operator is out of range of physically moving the mouse, but the screen pointer still needs to be moved away. This situation occurs, for example, in UNIX systems that use a display system called "Single Logical Screen" (SLS). In it, perhaps as many as 4 monitors are arranged, each displaying some subsection of the total "screen". If these monitors were arranged to be 1 high by 4 wide, the left-to-right distance required for a single corresponding maximum mouse movement would be much wider than would normally be allowed. The usual motion performed by the operator for, say, an extended rightward excursion is to simply lift the mouse on the right side of the work surface (a mouse pad, or perhaps just a cleared corner of his desk's cluttered table) and lower it on the left it, and keep moving it to the right. What is needed is a method of avoiding motion indication signals being affected by spurious motion during such motion so that the pointer on the screen behaves in a desired and unobtrusive manner. The function of the "hold" button can be accomplished automatically by a proximity sensor on the underside of the mouse, which determines that the mouse is not in contact with the work surface; or by noticing that all or most pixels in the image have "dimmed" (it's actually a bit more complicated than the idea that we'll talk more about in the next section) to autocomplete. Without the hold feature, there would be some slight distortion of the image when the mouse is removed and replaced, due to: (a) a tilted field of view when the mouse is lifted; or (b) some anomalous bug where the Frames of two unrelated and widely separated spatial features imaged at completely different times when removed and replaced are still taken to represent the small distance between two frames of the same feature. A convenient place to actually hold the buttons is on the sides of the mouse near the bottom, where the thumb and opposing ring finger grip the mouse to lift it. The natural increase in grip force used to lift the mouse will also ensure retention functionality. A hold feature may contain an optional short delay in release of the hold button, detection of appropriate proximity, or return of a reasonable digitized value. During this delay, any lighting control servo loops or internal automatic gain controls will have time to stabilize and take a new reference frame before motion detection resumes.

So now comes the "darkening" of pixels in an image. What happens, of course, is that the IR light from the illuminating LED no longer reaches the photosensitive detector in the same amount, if at all, it did; the reflective surface is too far away or simply invisible. But if the mouse is lifted due to the sight eye, it is turned over, or its underside is exposed to a brightly lit environment, the output of the photosensitive detector may be at any level. The point is that they will be uniform, or close to uniform. The main reason they are uniform is that there is no longer an image that is in focus; all image features are blurred and they are each scattered across the entire set of photosensitive detectors. So the photosensitive detector uniformly reaches a certain average level. This is in sharp contrast to the situation when there is a focused image. In the case of being in focus, a correlation between frames (recall one over, one over and one down, etc.) appears to be apparent.

It is assumed that the spatial features being tracked are accurately mapped onto the photosensitive detectors via the lens system, and that the movement of the mouse is not stationary in exactly the amount and direction required by a feature from detector to detector. Now for simplicity, also assume that there is only one feature, and its image is of photosensitive detection size. So all but one of the photosensitive detectors are at exactly the same level due to this feature, and the one detector which is not at this level is at a significantly different level. Under these highly idealized conditions, it is clear that the correlation will be well represented; eight "big" differences and one small difference in a system (on what would otherwise be a fairly flat surface One sinkhole on ) using the nearest neighbor algorithm for nine heuristics (remember also that there may be no moves). (Note: Astute readers will note that the "big" difference in this rather contrived example actually corresponds to or originates from only one pixel, and probably doesn't deserve to be called "big"... Recall that An earlier simulation of a shifted film. For this example the only light passed by the film is the one pixel for the feature. A more normal image with a considerably different set of pixels increases the difference to the point where it is indeed a "Big" difference.)

Thus, such highly idealized conditions are not the usual case. More normally, the images of the spatial features being tracked are slightly larger and smaller than the size of the photosensitive detectors, and the movement of the mouse is continuous, following paths that allow these images to fall on less than one detector simultaneously, some Some detectors will only receive a partial image, that is, some detectors will perform analog addition of light and dark. The result is at least a "widening" of the counterbore (by the number of photosensitive detectors associated with it) and probably a corresponding reduction in the depth of the counterbore. This can be assumed by imagining a heavy ball rolling along a tensioned but stretchable membrane. The film has a discrete integer Cartesian coordinate system associated with it. How much does the film expand at any integer coordinate as the ball rolls? First imagine the ball has a small diameter but is heavy, then imagine the ball has a large diameter and still have the same diameter. The simulation is not exact, but it serves to illustrate the "counterbore" idea above. The general situation is that the roughly flat face with the well-defined counterbore becomes the broad concave face, or bowl.

We will refer to the surface generated or described by the various correlation values as the "correlation surface", and at various times will be most interested in the shape of this surface.

We say that all these form two points. First, the concave shape that shifts on the relevant face as the eye-mouse moves allows for the insertion of finer grains than the mere size/spacing of photosensitive detectors. We point this out as a pro forma that our eye-mouse can do that, and stop there. Full details of the insertion are described in the incorporated patent. There is no need to discuss insertion further. Second, and this is the real reason for our discussion in the previous sections, is the observation that what happens when the eye-mouse is lifted is that the concave surface in the correlation plane leaves, replaced by roughly equal correlation values (i.e., "flat" correlation surface) instead. It is when this happens that we can safely say that the eye-mouse is floating in the air, and then automatically invoke the hold feature until such a moment later that an appropriate concave surface (bowl) reappears.

Another way to invoke or initialize the hold feature is to just notice that the mouse is moving faster than some threshold velocity (so in an attempt to move the screen pointer farther than the available physical space in which the mouse is operating, presumably is undergoing a sharp return movement.). Once the velocity threshold is exceeded, the movement of signals that would otherwise be associated with this movement is suppressed until such time that the velocity falls below an appropriate level.

Figure 1 is a simplified schematic cutaway side view of a prior art imaging and navigation device;

Figure 2 is a bottom view of a mouse constructed in accordance with the present invention;

Figure 3 is a side perspective view of a mouse constructed in accordance with an aspect of the present invention; and

Figure 4 is a simplified side sectional view of a proximity sensor on the base of the mouse of Figures 2 and 3 for automatically activating the hold feature;

Fig. 5 is a simplified flowchart describing the operation of one aspect of the intra-eye mouse, which when used with a feature called prediction, is related to the operation of maintaining features.

Figure 6 is an improved simplified portion of the flow chart of Figure 5, illustrating the speed detection method for calling hold features;

Figure 7 is a perspective view of the associated face drawn with a suitable concavity.

Description of the preferred embodiment

Referring now to FIG. 1, there is shown a simplified cross-sectional side view representation of a prior art imaging and navigation device 1, generally of the type described by the incorporated patent. Perhaps an IR LED and LED2 emits light, projected by lens 3 (which is not separate and may be an integral part of the LED package), through aperture 13 on bottom surface 6 onto area 4 which is part of working surface 5 . The average incident angle is preferably in the range of 5 to 20 degrees. Although one window has been omitted for clarity, the aperture 13 also contains this window, which is transparent to the light from LED 2, for preventing dust, dirt or other contamination from entering the internal structure of the eye-mouse. The work surface 5 may be attached to a special object, such as a mouse pad, or more generally, it is not the surface of a mouse pad, but could be almost any surface except smooth glass. Examples of suitable materials include, but are not limited to, paper, cloth, laminated plastic tops, lacquered surfaces, frosted glass (glossy side down, thank you), table mats, solid wood, fake wood, etc. In general, any microtextured surface with features ranging in size from 5 to 100 microns will do.

Illumination of microtextured surfaces is most effective when illuminated from the side, as this enhances light and shadow images due to surface height inhomogeneities. Suitable incident angles of illumination include a range of about 5 to 20 degrees. Very smooth or flat surfaces (eg ground and polished surfaces) with simple changes in reflectivity due to (minor) compositional changes are also effective. In this case (assuming it can be guaranteed), the incidence angle of the illumination can be close to 90 degrees, because the urgent need to produce shadows is gone. However, such a smooth and fine surface is not what we usually think of when we say "arbitrary surface". Also, eye-mouses for use on "arbitrary surfaces" that are more likely to be micro-textured will work best if equipped to provide a grazing angle of incident illumination.

The image of the illuminated area 4 is projected onto the photosensitive detector array 10 via the optical window 9 in the packaged portion 8a of the integrated circuit. This is done by means of the lens 7 . The package part 8a can also omit the separate window 9 and lens 7, combining them into one and the same component. The photosensitive detectors may consist of a square array of, say, 12 to 24 detectors on one side, each detector being a phototransistor with a photosensitive area of 45 x 45 microns and a center-to-center spacing of 60 microns. A phototransistor charges a capacitor whose voltage is continuously digitized and stored in a memory. The array 10 is mounted on a portion of an integrated circuit die 12 which is adhered with an adhesive 11 to the encapsulation portion 8b. Things not shown are any details of how the integrated circuit is held in place (possibly with a printed circuit board), the shape or composition of the lenses, or how the lenses are arranged; it is clear that these things can be done by conventional means of. It is also clear that the overall illumination level of zone 4 can be controlled by noting the output level of the photosensitive detector and adjusting the intensity of light emitted from LED2. This may be continuous control or pulse width modulation, or some combination of the two.

Once again, the reader is reminded that details concerning the operation of motion sensing are fully described in the incorporated patents (and also briefly in this summary); therefore, they need not be repeated here.

Referring now to FIG. 2, there is a bottom view of a mouse 14 constructed in accordance with the present invention. In short, this bottom view of this particular eye-eye mouse 14 looks very similar to the bottom view of a particular conventional mouse, C1413A, from the Hewlet-Packard Company. The main difference is that where there should be a ball there is a protective lens or window 16 transparent to IR light. This is the transparent window omitted in the diaphragm 13 mentioned in the description of FIG. 1 . Also missing is a generally rotatable annular sleeve which acts as a removable holder allowing access to the ball for cleaning or replacement. What is shown in the figure is the underside 15 of the mouse 14 (corresponding to 6 in FIG. 1 ), the low-friction slider 19 and the connecting cable 17 with its strain relief 18 . Certainly, our visual eye mouse 14 also may be a cordless mouse, has light or radio communication link with computer.

Referring now to FIG. 3, there is shown a side perspective view of a mouse 14 constructed in accordance with an aspect of the present invention. This aspect of the invention is a retaining feature. The hold feature is an aspect of eye mouse operation that suspends the generation of movement information or signals to the computer when the mouse is determined to be inappropriately close to a work surface whose spatial features are being tracked. This allows the eye-mouse to be lifted, moved, and put back down, as we call such an operation "swiped" across the work surface.

Specifically, eye-eye mouse 14 in FIG. 3 includes at least one hold button 24 positioned on the side edge near bottom surface 15 so as to be under the right thumb or left ring finger, depending on which hand is in use. There may also be another button (not shown) located symmetrically on the other side, which contacts either the left thumb or the right ring finger.

Mouse 14 generally includes a surface 21 that rests in the palm of the hand and first and second "front view" mouse buttons 22 and 23 that are actuated by the index and middle fingers. These work in their normal way.

The button or 2 buttons 24 are activated by the natural increase in grip force required to lift the mouse 14 upon swiping. When one or both of these buttons are pressed, the hold feature is activated. During hold, the transmission of motion signals to the computer is suspended. When the hold ends (the button is released), a new reference frame is acquired before any new motion signals are sent to the computer. This allows punching, and has the advantage that the user has the ability to deliberately force the hold feature to start.

The hold feature may also be automatically activated by the action of a separate proximity sensor on the bottom of the mouse. This is what is shown in Figure 4. In it, a stepped aperture 26 in the base 6 accommodates a stepped plunger 25 which is latched with the lever arm of a switch 28 above. The switch 28 is activated by movement of the plunger 25 such that when the plunger moves appreciably in the direction of the arrow 27 the hold feature is activated. The exact nature of the individual proximity sensors is a matter of choice, although it could be as simple as a microswitch operated by the weight of the mouse via the plunger 25, other non-mechanical methods are possible.

Yet another method of automatically activating and deactivating the hold feature is to check the nature of the digitized data of the photosensitive detector array 10 . When the output of the photosensitive detectors becomes sufficiently uniform, it can be concluded that there is no longer a variable image projected onto the photosensitive detector array 10. This uniformity will occur by making the relative surfaces flat or nearly flat. Rather than probing the degree of uniformity alone (this would use hardware that only exists now), we would rather examine the shape of the associated surface (we need this surface for other reasons anyway). The most likely cause of a flat related surface is that the mouse has been lifted. This mode of operation probably dictates that there should be a fairly limited depth of field to avoid undue delay there when hold is activated. Such delays can produce artifacts in screen pointer movement. These artifacts may include slight unwanted screen pointer movement due to the tilt of the mouse when it is either lifted or repositioned. As long as activating the hold feature (however done, whether manually or automatically) forces acquisition of a new reference frame before restarting motion signal generation, there should be no artifacts due to combinations of old data with some new data The danger of a false indication that occasionally appears as a characteristic small movement in some inappropriate direction. However, it may be difficult to ensure that there are no optical effects (reflection of bright sources) that would confuse the algorithm when moving through the air, such as in the case of only uniformity detection (say, of sampled frames). It will be appreciated that the shape of the relevant faces is a more reliable indicator. All that said, one must still remember that the so-called manipulation of the screen pointer is an incrementally driven servo-like operation performed by a human; if the screen pointer is not there yet, just keep moving the mouse as needed! A small disturbance in swiping is not fatal and may not even be noticeable, depending on the specific application being performed.

Referring now to FIG. 5, there is shown a flow chart 29 describing one aspect of the operation of the sight-eye mouse including hold and predict features. We may assume a certain starting state or position 30, from here to step 31: "acquire a frame of reference". This means illuminating LED2 and storing the set of digitized photodetector values into a memory array (not shown). The next step 32 is "acquire a frame of samples". This refers to the same action, except the data is stored in a different memory array and may reflect mouse movement relative to where it was when step 31 was performed. In step 33, "Compute Correlation Values", 9 (or possibly 25) correlation values are computed rapidly by some heavily loaded special purpose computing hardware assisted by automatic address transfer and a wide path out of the memory array . In step 34, "Is the relevant surface properly concave?", the nature of the relevant surface described by the set of correlation values computed in step 33 is checked. We wondered if it was shaped like a bowl, and if so, how much water would it hold?

If the shape of the relevant face is a good bowl, then path 36 takes us to optional step 37: "Is the hold button pressed?"; more on this in the next section. Otherwise, we have a flat relative surface, or a "bad bowl", and continue along path 35 to optional step 42, "Delay". There are several possible reasons for this derivation from hedge 34: eg, extreme speed, a suddenly featureless work surface, and a mouse floating in the air. In the absence of an obvious "hold" button, we will rely on the pinout path 35 to provide proper sight-eye mouse behavior by suppressing the motion signal to the computer when swiping the air segment of the operation.

If the eye-mouse does have a "hold" button, an optional hedge 37 exists, and here the state of the "hold" button 24 (pressed or not pressed) is determined. The pressed case is handled the same as the bad bowl case at hedge 34. That is to take the path 38 which also passes through the optional step 42 .

Optional step 42 provides a delay which can be useful in several ways. First, if there is a swipe in the process, it takes some time, and some battery power can be saved by not imaging during this time. It is also assumed that the nature of this delay is slightly more complicated than pausing in the motion of moving a finger on the flowchart. Assume that the "get reference frame" step 31 is affected by a delay in step 42 . During this delay, a light level control operation is initiated. This can provide time for readjustment of lighting levels etc. Regardless of whether there is a "delay" at optional step 42, path 43 leads back to step 31 where another motion detection cycle begins again.

To proceed, path 39 passes through step 40: "Predict displacement in reference frame". As noted above, it is generally not necessary to acquire and maintain actual velocities in X and Y, and time interval information, in order to obtain the displacement required for prediction. One can imagine measurement environments that might be required, but the one shown here is not one of them. Instead, the predicted shift can be taken by the amount of movement corresponding to the correlation at step 34 above.

The next step 44 is "Output Δx and Δy". It is here that we note how much mouse movement has occurred since the last measurement cycle. The amount of shift needed to obtain the correlation is the required amount. These values can be found by noting which compared frames actually correlate (assuming no interpolation). These "raw" Δx and Δy motion values may be accumulated into motion values that are sent to the computer at a lower rate than the rate at which the raw values of step 44 were generated.

At limit word 45, we ask if we "need a new frame of reference?". If the answer is YES, path 46 leads to step 48: "store current sampled frame in reference frame". (Some considerations will justify this re-use of sampled frames in conjunction with not having to maintain the actual speed and time interval of the prediction process. If we take a single new reference frame, this complicates things a lot and may force the use of predicted D =RT...that is, the distance formula).

When a reference frame has been sufficiently shifted, we need a new reference frame, since comparing frames for reliable relative overlap is insufficient. Somewhere in the range of 3 to 5 shifts (they don't fold back on themselves) is about the limit of a 16x16 reference frame.

If the answer to qualifier 45 is NO, and we do not need to replace the reference frame, path 47 takes us to the same step 49 as the path derived from step 48 leads to. Step 49, "Shift Reference Frame", performs the actual permanent shift of the values in the memory array representing the reference frame. The shift is measured in advance, and the removed data is lost. After the reference frame is shifted, path 50 returns to step 32, "Get a sample frame", where the next measurement cycle begins.

Referring now to FIG. 6, there is shown a simplified flowchart segment 50 showing how step 44 of flowchart 29 in FIG. 5 is replaced with steps 51-55. The effect of doing like this is similar to the various holding operation modes described above, and can be combined or used together with this. The general idea of the improvement represented by FIG. 6 is to hide this by skipping step 55A and not transmitting any updated information or (optionally, with step 55B) transmitting zeros for Δx and Δy, even when this is not true. over the computer. This is done whenever step 52 determines that the rate of mouse motion exceeds, say, 3 to 6 inches per second. Such a limit can easily be expressed as a displacement of a certain number of pixels in some number of measurement cycles for a given sight-eye mouse, assuming that the measurement cycle rate is fast compared to normal mouse motion. The idea is that normal irregular mouse motion may not require a new nearest neighbor reference frame per measurement cycle during some large (say, 10 to 25) number of consecutive measurement cycles (let alone for 5× One of the largest shifts for 5 nearest neighbor operations). Because if this is true, the eye-eye mouse will work on the hold mode thin edge through the NO answer to the hedge 34 and the path 35. (According to this assumption, any higher speed will lead to associated losses!) That is, it is usually very infrequent to wish to fetch a new reference frame. Of course, anyway, the speed of the mouse is really high, so it may happen that path 35 is used. This is as it should be. However, it may not be appropriate to use the technique of FIG. 6 if the measurement cycle rate is not sufficiently high relative to normal expected mouse motion.

Step 51 represents anything in the old step 44 except the actual transfer of the values Δx and Δy to the computer. A subtle example of this difference might be an internal accumulation of motion that has not yet been communicated to the computer due to the eye-mouse's internal motion measurement cycle rate being higher than the computer's message exchange rate. Now, it may indeed be true that in some systems this accumulated information is used for internal mouse purposes and not strictly used to keep the computer communicated. If so, it needs to be saved because all the qualifier 52, path 53 (and bypass step 55A) needs to accomplish is "no" telling the computer that there has been motion; we want to fool the computer without making the mouse lose its mind.

It will be noted that if such accumulation is allowed to continue during the fast retrace, to simulate lifting the mouse, the computer still wins at the end when the speed drops to a normal amount and the accumulation is transmitted last; depends on how the whole system works , the screen cursor can be moved to the correct position anyway. In such a case, a single set of accumulations should be maintained, in those cases where the computer maintained in bypass step 55A.

Of course, it may be true that there is no accumulated Δx and Δy used internally by the mouse, other than to transfer it to the computer. In such a case, there is nothing to do but leave this accumulation in bypass step 55A. It is also possible that there is simply no accumulation in the mouse that raises such concerns; any such accumulation is done by software in the computer, for example.

Finally, see Fig. 7, which is a plot 56 of the adjacent (5 x 5) correlation face 57 with a suitable concave face. The two horizontal axes 58 and 59 represent the X and Y axes of mouse movement; the units indicated along these axes are pixels. Drawn onto the plane of axes 58 and 59 are smoothly interpolated contour lines 60 which are intended to further represent the shape of the associated surface directly above. Vertical axis 61 is a measure of correlation expressed in essentially arbitrary units.

Claims (3)

1. what be suitable for computer system etc. holds indicator device (1,14,15), and this indicator device comprises:

But shell that the flat bottom surface (6) that the workplace (5) with respect to imaging characteristic moves is arranged;

The end face (21) that the also promising acceptor's hand of this shell forms;

This shell also has the shell (20) that the flat bottom surface periphery is coupled together with end face;

This shell has first, substantially with from the palm root by end face in depend on the end face place direction stretch, with second axle vertical with first, two axles all are parallel to the bottom surface;

Diaphragm (13,16) on the bottom surface;

A light source (2,3) is arranged at enclosure, but near diaphragm and shine imaging features on the workplace;

Be installed in the lens in the shell, these lens are near diaphragm, but interception is by the light of illuminated imaging features reflection;

The motion detection circuit (12) of an optics, be installed in the inside of shell and near lens, this motion detection circuit produces motion indicator signal, but this signal indication along first and second axle and with respect to moving through the direction of the visible irradiated imaging features of lens; And

The motion detection circuit of optics comprises a plurality of photosensitive detectors (10) therein, each photosensitive detector has an output, a storer, described storer comprise the reference frame (31) of a digitizing photosensitive detector output valve and the sampling frame (32) of a digitizing photosensitive detector output valve obtaining after this reference frame; A plurality of therein in addition relatively frames are relevant so that produce indication along the motor message that moves on the direction of first and second axle with the sampling frame, and each described relatively frame is the shifted version of reference frame.

2. as the device in claim 1, an existing reference frame is by being shifted with the relevant corresponding quantity (40) with a comparison frame of front therein.

3. as the device in right claim 1, an existing sampling frame is got periodically and is made a new reference frame therein.

Images