CN102047203B - Pose-based control using 3D information extracted within an extended depth of field - Google Patents

Abstract

Systems and methods for gesture-based control using three-dimensional information extracted over an extended depth of field are described. The system includes a plurality of optical detectors coupled to at least one processor. The plurality of optical detectors images the body. At least two of the plurality of optical detectors comprise wavefront coded cameras. The processor automatically detects a posture of the body, wherein the posture includes an instantaneous state of the body. The detecting includes aggregating pose data for the pose for only a moment and excluding use of background data in the detecting of the pose. The pose data includes focus-resolved position data of the body relative to a neutral position as an absolute position and orientation in space, wherein the position data is three-dimensional information. The processor translates the gesture into a gesture signal and uses the gesture signal to control a component coupled to the processor.

Description

Pose-based control using 3D information extracted within an extended depth of field

related application

This application is a continuation-in-part of US Patent Application Serial No. 11/350,697, filed February 8,2006.

This application claims priority to US Patent Application No. 61/041,892, filed April 2, 2008.

This application is a continuation-in-part of US Patent Application Serial No. 12/109,263 filed April 24,2008.

This application claims priority to US Patent Application No. 61/105,243, filed October 14, 2008.

This application claims priority to US Patent Application No. 61/105,253, filed October 14, 2008.

technical field

The present invention relates generally to the field of computer systems, and more particularly to systems and methods for gesture-based control by extracting three-dimensional information within an extended depth of field.

Background technique

When extracting three-dimensional information over an extended depth of field in an imaging system, the distance to a point in the scene can be estimated from its position in two or more images that are captured simultaneously. When the three-dimensional (3D) relationship between images is known, the 3D position of the point can be calculated from the underlying geometric relationship. The challenge in computing spatial location from multiple images (often referred to as stereo correlation or stereo depth computation) is to automatically and accurately relate the mapping of a point in one image to its mapping in another image. This is most commonly done by correlating image features from one image to one or more other images. However, a fundamental assumption in all stereo matching methods is that there must be some identifiable local contrast or feature in an image in order to match a point with its location in another image. Therefore, when there is no local contrast or features in the image due to defocus, a problem arises - stereo matching does not produce accurate results in areas of the image that are out of focus.

A conventional means for extending the depth of focus of an image is to reduce the diameter of the camera lens pupil ("zoom out"). However, two side effects limit the utility of this technique. First, the sensitivity of the imaging system is reduced by a factor equal to the square of the ratio of the inner and outer diameters of the pupil. Second, the maximum spatial frequency response is reduced by a factor equal to the ratio of the inner and outer pupil diameters, which limits resolution and contrast in the image. Therefore, there is a trade-off between depth of field, exposure time, and overall contrast in conventional imaging systems. In the case of a multi-camera ranging system, the net effect will be a tradeoff between stereo depth accuracy and working range.

incorporation by reference

Each patent, patent application and/or publication mentioned in this specification is hereby incorporated by reference in its entirety as if each patent, patent application and/or publication were specifically and individually indicated to be incorporated by reference Same.

Description of drawings

Figure 1 is a diagram of one embodiment of the system of the present invention;

Figure 2 is a diagram of one embodiment of the marking label of the present invention;

Figure 3 is a diagram of gestures in a gesture vocabulary in one embodiment of the invention;

Figure 4 is a diagram of orientations in a gesture vocabulary in one embodiment of the invention;

Figure 5 is a diagram of the combination of both hands in the gesture vocabulary in one embodiment of the present invention;

Figure 6 is a graph of orientation blending in a gesture vocabulary in one embodiment of the invention;

Figure 7 is a flowchart illustrating operation in one embodiment of the system of the present invention;

Figure 8 is an example of a command in one embodiment of the system;

9 is a block diagram of a gesture-based control system for extracting three-dimensional information within an extended depth of field, under one embodiment;

Figure 10 is a block diagram of a wavefront encoded imaging system for use in an attitude-based control system, under one embodiment;

11 is a block diagram of an attitude-based control system, under one embodiment, that employs a wavefront-encoded imaging system including two wavefront-encoded cameras to extract three-dimensional information over an extended depth of field;

Figure 12 is a flowchart of gesture-based control using 3D information extracted within an extended depth of field, under one embodiment;

Figure 13 is a block diagram of a wavefront code design process for use in a gesture-based control system, under one embodiment.

Contents of the invention

Systems and methods for gesture-based control using three-dimensional information extracted within an extended depth of field are described below. A system includes a plurality of optical detectors coupled to at least one processor. The plurality of optical detectors images the body. At least two optical detectors of the plurality of optical detectors include wavefront encoded cameras. The processor automatically detects a pose of the body, wherein the pose includes a momentary state of the body. The detection involves gathering gesture data for only a moment of the gesture and excluding the use of background data in the detection of the gesture. The pose data includes focus-resolved position data of the body relative to a neutral position which is an absolute position and orientation in space, wherein the position data is three-dimensional information. The processor translates the gesture into a gesture signal and uses the gesture signal to control components coupled to the processor.

One method includes imaging a body with an imaging system, where the imaging includes generating a wavefront encoded image of the body. The method automatically detects the pose of the body, where the pose includes the instantaneous state of the body. The detection involves gathering gesture data for only a moment of the gesture and excluding the use of background data in the detection of the gesture. The pose data includes focus-resolved position data of the body relative to a neutral position which is an absolute position and orientation in space, wherein the position data is three-dimensional information. The method includes translating the gesture into a gesture signal, and controlling components coupled to the computer in response to the gesture signal.

In the following description, various features are described in detail in order to provide a more thorough understanding of the embodiments described herein. It is evident that the present invention may be practiced without these specific details.

detailed description

system

A block diagram of one embodiment of the invention is shown in FIG. 1 . A user places hands 101 and 102 within the field of view of camera arrays 104A-104D. The camera detects the position, orientation and movement of the fingers and hands 101 and 102 and generates output signals to the pre-processor 105 . The pre-processor 105 translates the camera output into attitude signals, which are provided to the computer processing unit 107 of the system. Computer 107 uses the input information to generate commands to control one or more on-screen cursors and provides video output to display 103 .

Although the system is shown with a single user's hand as input, the invention can also be implemented with multiple users. Additionally, instead of or in addition to hands, the system may track any one or more parts of the user's body, including head, feet, legs, arms, elbows, knees, and the like.

In the illustrated embodiment, four cameras are employed to detect the position, orientation and movement of the user's hands 101 and 102 . It should be understood that the invention is equally applicable to more or fewer cameras without departing from the scope or spirit of the invention. Also, although the cameras are arranged symmetrically in the example embodiment, such symmetry is not required in the present invention. In the present invention, any number or positioning of cameras that allows for the positioning, orientation and movement of the user's hand may be used.

In one embodiment of the invention, the camera used is a motion capture camera capable of capturing grayscale images. In one embodiment, the camera used is a camera manufactured by Vicon, such as the Vicon MX40 camera. This camera includes on-camera processing and is capable of image capture at 1000 frames per second. Motion capture cameras are able to detect and locate markers.

In the described embodiment, a camera is used for light detection. In other embodiments, cameras or other detectors may be used for electromagnetic, magnetostatic, RFID, or any other suitable type of detection.

The preprocessor 105 is used to generate 3D space point reconstruction and bone point label setting. Gesture translator 106 is used to translate the 3D spatial information and marker motion information into a command language that can be interpreted by a computer processor to update the position, shape and motion of the cursor on the display. In an alternative embodiment of the invention, pre-processor 105 and pose translator 106 may be combined into a single device.

Computer 107 may be any general purpose computer, such as manufactured by Apple, Dell, or any suitable manufacturer. Computer 107 runs applications and provides display output. Cursor information that would have come from a mouse or other prior art input device now comes from the gesture system.

mark label

The present invention contemplates the use of marker tags on one or more of the user's fingers so that the system can locate the user's hand, identify whether it is looking at the left or right hand and which fingers are visible. This allows the system to detect the position, orientation and movement of the user's hand. This information allows multiple gestures to be recognized by the system and used as commands by the user.

The marking label in one embodiment is a physical label comprising a substrate (in this embodiment adapted to be attached to various locations on a human hand) and discrete indicia arranged on the surface of the substrate in a uniquely identifying pattern.

The marker and associated external sensing system can operate in any domain (optical, electromagnetic, magnetostatic, etc.) that allows accurate, precise, rapid and continuous acquisition of its three-dimensional position. The markers themselves can work actively (eg by emitting structured electromagnetic pulses) or passively (eg by optical retroreflective markers as in this embodiment).

At each acquisition frame, the detection system receives an aggregated "cloud" of recovered 3D spatial positions including the positions on the labels that are currently within the instrumented workspace volume (within view of the camera or other detector). All tags. The markers on each label are of sufficient diversity and are arranged in a unique pattern that the detection system can perform the following tasks: (1) Segmentation, where each recovered marker position is assigned to the number of points that make up a single label. one and only one subset; (2) label setting, wherein each segmented subset of points is identified as a specific label; (3) localization, wherein the three-dimensional position of the identified label is recovered; and (4 ) orientation, where the three-dimensional spatial orientation of the identified tag is restored. Tasks (1) and (2) are made possible by specific properties of the marking pattern, as described below and shown in one embodiment in FIG. 2 .

Indicia on the label in one embodiment are affixed to a subset of regular grid locations. This basic grid may be of the traditional Cartesian type as in this embodiment; or instead may be some other regular planar tessellation (eg a triangular/hexagonal tiling). The scale and spacing of the grid is established in view of the known spatial resolution of the marker sensing system such that adjacent grid positions are less likely to be confused. The marking patterns of all labels shall be selected to meet the following constraints: the pattern of the label shall not be consistent with the pattern of any other label through any combination of rotation, translation or mirroring, and the diversity and arrangement of markings may be further Loss (or occlusion) of a specified number of component labels. After any arbitrary transformation, it should still be unlikely to confuse the damaged module with any other module.

Referring now to FIG. 2, a plurality of tags 201A-201E (left hand) and 202A-202E (right hand) are shown. Each label is rectangular and in this embodiment consists of a 5x7 grid array. Choose a rectangular shape to help orient the label and reduce the possibility of mirror duplication. In the illustrated embodiment, there are labels for each finger on each hand. In some embodiments, it may be sufficient to use one, two, three or four tags per hand. Each label has a border of a different grayscale or shade of color. Inside this boundary is a 3x5 grid array. Markers (represented by black dots in Figure 2) are placed at certain points of this grid array to provide information.

By segmenting each pattern into "common" and "unique" sub-patterns, identification information can be encoded with the tag's marking pattern. For example, this embodiment specifies two possible "boundary patterns" (distribution of markers around a rectangular boundary). A "family" of labels is thus established - labels intended for the left hand may thus all use the same border pattern as shown in labels 201A-201E, while labels attached to the fingers of the right hand may be assigned as in labels 202A-202E Different patterns shown. This sub-pattern is chosen such that the left and right patterns are distinguishable in all orientations of the label. In the example shown, the left hand pattern includes markings at each corner and at the second grid position from the corner. The right-hand pattern has marks at only two corners, and two marks at non-corner grid locations. From this pattern it can be seen that as long as any three of the four markings are visible, the left-hand pattern can be clearly distinguished from the right-hand pattern. In one embodiment, the color or shading of the border can also be used as an indication of chirality.

Each label must of course still adopt a unique internal pattern, with the marks distributed within the common boundaries of its family. In the illustrated embodiment, it has been found that two markers in the internal grid array are sufficient to uniquely identify each of the ten fingers without duplication due to rotation or orientation of the fingers. Even if one of the marks is obscured, the combination of the tag's hand shape and pattern produces a unique identifier.

In this embodiment, the grid locations are visually present on the rigid substrate to aid in the manual task of affixing each retroreflective marker in its intended location. These grids and intended marking positions are precisely printed onto a substrate, here a sheet of initially flexible "shrink film", by means of a color inkjet printer. Each module is cut from the sheet and then oven baked, during which heat treatment each module undergoes precise and repeatable shrinkage. In a short interval after this process, the cooling label can deform slightly - to mimic, for example, the longitudinal bending of a finger; thereafter, the substrate is suitably rigid and markings can be adhered to the indicated grid points.

In one embodiment, the markings themselves are three-dimensional, such as small reflective spheres attached to the substrate by means of adhesive or other suitable means. The three-dimensional nature of markers can aid in the detection and localization of two-dimensional markers. However, either may be used without departing from the spirit and scope of the present invention.

Currently, tags are attached to gloves worn by the operator by means of Velcro or other suitable means, or alternatively directly to the operator's fingers using double-sided tape. In a third embodiment, the rigid substrate can be omitted entirely, and the individual indicia can be affixed (or "painted") to the operator's fingers and hands.

gesture vocabulary

The present invention considers a gesture vocabulary consisting of hand poses, orientations, hand combinations, and orientation blends. A notational language is also implemented to design and communicate gestures and poses in the gesture vocabulary of the present invention. A pose vocabulary is a system for representing the instantaneous 'pose state' of a kinematic linkage in compact text form. The linkage in question may be biological (such as a human hand; or a whole human body; or a grasshopper leg; or the articulated spine of a lemur) or alternatively may be non-biological (such as a robotic arm). In any case, the linkage can be simple (spine) or branched (hands). The pose vocabulary system of the present invention builds a string of constant length for any particular linkage; then, the particular set of ASCII characters occupying the 'character positions' of that string are unique descriptions of the linkage's instantaneous state or 'pose'.

hand pose

Figure 3 illustrates hand gestures in one embodiment using the gesture vocabulary of the present invention. The present invention assumes that each of the five fingers on the hand is used. These fingers are codes such as p-little, r-ring, m-middle, i-index, and t-thumb. A number of finger and thumb gestures are defined and shown in FIG. 3 . The pose vocabulary string establishes a single character position for each expressible degree of freedom in the linkage (in this case, the fingers). Furthermore, each such degree of freedom is understood to be discretized (or 'quantized') such that its full range of motion can be expressed by assigning one of a limited number of standard ASCII characters to the position of the string. These degrees of freedom are expressed relative to an origin and coordinate system specific to the body (the back of the hand, the center of the grasshopper's body; the base of a robotic arm, etc.). Therefore, a small number of additional gesture vocabulary character positions are used to express the position and orientation of the linkage 'as a whole' in a more global coordinate system.

Still referring to FIG. 3 , multiple gestures are defined and identified using ASCII characters. Some of these poses are divided between thumb and non-thumb. The invention uses encodings in this embodiment so that the ASCII characters themselves imply gestures. However, any character may be used to represent a gesture, implied or not. In addition, it is not necessary to use ASCII characters for token strings in the present invention. Any suitable symbols, numbers or other representations may be used without departing from the scope and spirit of the invention. For example, the indicia could take two or some other number of digits per finger, if desired.

A bent finger is indicated by the character "^", while a bent thumb is indicated by ">". A straight finger or thumb pointing upward is indicated by "1", and a straight finger or thumb pointing at an angle is indicated by "\" or "/". "-" indicates a thumb pointing straight to the side, and "x" indicates a thumb pointing in-plane.

Using these individual finger and thumb descriptions, a considerable number of hand gestures can be defined and written using the scheme of the present invention. Each pose is represented by five characters in the order p-r-m-i-t as described above. Figure 3 illustrates a number of gestures, and a few gestures are described herein by way of illustration and example. A hand kept flat and parallel to the ground is indicated by "11111". A fist is represented by "^^^^>". The "OK" symbol is represented by "111^>".

Strings provide an opportunity for shallow "legibility" when using suggestive characters. With a view to fast recognition and straightforward simulation, the set of possible characters that describe each degree of freedom can often be chosen. For example, a vertical bar ('|') is intended to indicate that a linkage element is 'straight', an L-shape ('L') may indicate a ninety-degree bend, and a circumflex ('^') may indicate an acute-angle bend. As above, any character or encoding can be used as desired.

Any system that employs pose vocabulary strings as described here benefits from the high computational efficiency of string comparisons—identifying or searching for any specified pose effectively becomes a 'string comparison' between the desired pose string and the instantaneous actual string (e.g. UNIX's 'strcmp()' function). Furthermore, the use of 'wildcards' provides programmers or system designers with additional common capacities and functions: degrees of freedom whose instantaneous state is not relevant for matching can be specified as question marks ('?'); additional wildcard meanings can be given .

orientation

In addition to finger and thumb gestures, hand orientation can also represent information. Obviously, the characters describing the orientation in the global space can also be chosen: the characters '<', '>', '^' and 'v' can be used to represent the concepts of left, right, up and down when they appear in orientation character positions. Figure 4 illustrates an example of encoding combining pose and orientation and a hand orientation descriptor. In one embodiment of the invention, the two character positions first dictate the orientation of the palm, and then the orientation of the fingers (if the fingers are straight, regardless of the actual curvature of the fingers). The possible characters for these two positions express the 'centre of body' notation of orientation: '-,'+','x','*','^' and 'v' describe medial, lateral, anterior (forward , away from the body), posterior (backward, away from the body), cranial (upward) and caudal (downward).

In the notation scheme of one embodiment of the invention, the character representing the five-finger gesture is followed by a colon and two orientation characters to define the complete command gesture. In one embodiment, the starting position is referred to as the "xyz" pose, where the thumb points straight up, the index finger points forward, the middle finger is perpendicular to the index finger, and points left when the pose is done with the right hand. This is represented by the string "^^x1-:-X".

"XYZ-Hand" is a technique that exploits the geometry of the human hand to allow navigation of all six degrees of freedom of visually presented three-dimensional structures. Although this technique relies only on the overall translation and rotation of the operator's hand - so that his fingers can in principle be held in any desired pose - a static configuration is preferred in this embodiment, with the index finger pointing away from the body; Thumb points to the ceiling; middle finger points left-right. These three fingers thus describe (roughly but clearly intended to describe) the three mutually orthogonal axes of the three-dimensional spatial coordinate system: thus 'XYZ-hand'.

Then, XYZ-hand navigation is performed, wherein the hand, fingers assume the posture as described above and remain in front of the operator's body, in a predetermined 'neutral position'. The understanding of the three translational and three rotational degrees of freedom of an object (or camera) in 3D space is achieved in the following natural way: Left-right movement of the hand (relative to the body's natural coordinate system) results in movement along the x-axis of the computing environment ; up and down movement of the hand results in movement along the y-axis of the controlled environment; forward and backward movement of the hand (towards or away from the operator's body) results in z-axis movement within the environment. Similarly, rotation of the operator's hand about the index finger results in 'roll' changes in the orientation of the computing environment; similarly, 'pitch' and 'roll' changes are achieved by rotating the operator's hand about the middle finger and thumb, respectively.

Note that although 'computing environment' is used here to refer to entities controlled by an XYZ-hand approach - and seems to imply synthetic 3D objects or cameras, it should be understood that this technique can equally be used to control various degrees of freedom of real world objects : such as a camera equipped with an appropriate rotary actuator or a pan/tilt/roll control of the camera. Furthermore, the physical degrees of freedom provided by the XYZ-hand pose may be somewhat imprecisely mapped in the virtual domain: in this example, the XYZ-hand is also used to provide navigational access to a large panoramic Left and right and up and down movements of the hand result in a desired left and right or up and down 'pan' around the image, while back and forth movement of the operator's hand is mapped to a 'zoom' control.

In all cases, the coupling between the motion of the hand and the resulting calculated translation/rotation can be direct (i.e., the position or rotation offset of the operator's hand is mapped one by one to The position or rotational offset of an object or camera within the computing environment) or indirect (i.e., the position or rotational offset of the operator's hand is mapped one-to-one to the position/orientation within the computing environment by some linear or non-linear function the first or higher derivatives of ; the ongoing integration then achieves a non-stationary change in the actual zero-order position/orientation of the computing environment). This latter means of control is analogous to the use of a 'pneumatic pedal' in a car, where a constant deflection of this pedal results in a more or less constant vehicle speed.

The 'neutral position', which serves as the origin of the real-world XYZ-hand's local 6DOF coordinates, can be (1) established as an absolute position and orientation in space (relative to, say, an enclosure); (2) established relative to the manipulation the operator's own fixed position and orientation (e.g., eight inches in front of the body, ten inches below the chin, and in line with the plane of the shoulders on the side), regardless of the operator's general position and 'orientation'; or (3) by Deliberate secondary actions of the operator (e.g. employing gesture commands by the operator's 'other' hand indicating that the current position and orientation of the XYZ-hand should henceforth be used as the translation and rotation origin) to is established interactively.

Furthermore, it is convenient to provide a 'blockade' zone (or 'dead zone') around the neutral position of the XYZ-hand, so that movements within this volume do not map to movements within the controlled environment.

Other poses can be included:

[|||||:vx] is a flat hand with the palm facing down and the fingers facing forward (thumb parallel to the fingers).

[|||||:x^] is a flat hand with the palm facing forward and the fingers facing the ceiling.

[|||||:-x] is a flat hand with the palm facing the center of the body (right in left-handed case, left in right-handed case) with fingers facing forward.

[^^^^-:-x] is a one-handed thumbs up (thumb pointing to the ceiling)

[^^^|-:-x] is an imitation gun pointing forward

combination of hands

The present invention contemplates one-handed commands and gestures, as well as two-handed commands and gestures. Figure 5 shows an example of a combination of hands and associated notation in one embodiment of the invention. Looking at the notation of the first example, "full stop" indicates that it involves two closed fists. The "snapshot" example has the thumb and forefinger of each hand extended, with the thumbs pointing toward each other, defining the frame of the goalpost shape. The "rudder and throttle home position" is with fingers and thumb pointing up and palm facing the screen.

mixed orientation

Figure 6 illustrates an example of orientation mixing in one embodiment of the invention. In the example shown, this blending is indicated by enclosing pairs of orientation tokens in parentheses after the string of finger gestures. For example, the first command shows all pointing finger positions. The first pair of orientation commands will cause the palm to face flat against the display, and the second pair will rotate the hand to a 45 degree incline towards the display. Although paired mixes are shown in this example, any number of mixes are contemplated in the present invention.

example command

Figure 8 shows a number of possible commands suitable for use with the present invention. Although some of the discussion herein is about controlling a cursor on a display, the invention is not limited to this work. In fact, the invention has great application in manipulating any and all data and portions of data on the screen, as well as the state of the display. For example, these commands can be used to override video controls during playback of video media. These commands can be used to pause, fast forward, rewind, etc. Additionally, commands may be executed to zoom out or zoom in on the image, change the orientation of the image, pan in any direction, etc. The present invention can also be used in place of menu commands such as open, close, save, and the like. In other words, any conceivable command or job can be performed with gestures.

operate

Figure 7 is a flowchart illustrating the operation of the invention in one embodiment. In step 701, a detection system detects markers and labels. At decision block 702, it is determined whether tags and markers are detected. If not detected, the system returns to step 701. If tags and markers are detected at step 702, the system proceeds to step 703. At step 703, the system identifies hands, fingers, and gestures from the detected tags and markers. At step 704, the system identifies the orientation of the gesture. At step 705, the system identifies the detected three-dimensional spatial location of one or more hands. (Note that any or all of steps 703, 704 and 705 may be combined into a single step).

At step 706, the information is translated into gesture tokens as described above. At decision block 707, it is determined whether the gesture is valid. This can be achieved by a simple string comparison using the generated token string. If the gesture is invalid, the system returns to step 701. If the gesture is valid, the system sends the marker and location information to the computer at step 708 . The computer determines the appropriate action to take in response to the gesture at step 709 and updates the display at step 710 accordingly.

In one embodiment of the present invention, steps 701-705 are implemented by an on-camera processor. In other embodiments, this process can be implemented by the system computer, if desired.

analysis and translation

The system is capable of "analyzing" and "translating" the stream of low-level gestures recovered by the underlying system, and turning those analyzed and translated gestures into commands or event data that can be used to control a wide range of computer applications and systems flow. These techniques and algorithms may be embodied in a system of computer code that provides both an engine for implementing the techniques and a platform for building computer applications that exploit the capabilities of the engine.

One embodiment is dedicated to enabling rich gestural use of the human hand in computer interfaces, but is also capable of recognizing objects made of other body parts (including, but not limited to, arms, torso, legs, and head) as well as a wide variety of non-hand physical tools (static articulated) gestures made by non-hand physical tools including, but not limited to, calipers, two-squares, flexible flexures, and pointing devices of various shapes. Markings and labels can be applied as desired to items and implements that can be carried and used by the operator.

The system described here incorporates several innovations that make it possible to build gesture systems that are rich in the range of gestures that can be recognized and acted upon, while providing simple integration into applications.

The gesture analysis and translation system in one embodiment consists of the following:

1) A compact and efficient way of specifying (encoded for use in a computer program) the following poses with different levels of aggregation:

a. "Pose" of a single hand (configuration and orientation of parts of the hand relative to each other) Orientation and position of a single hand in three-dimensional space.

b. Combinations of hands, for either hand, consider posture, position, or both.

c. Multiplayer combinations; the system can track more than two hands, so more than one person can cooperatively (or competitively, in the case of gaming applications) control the target system.

d. Sequential poses, where poses are combined into a series; we call them "active" poses.

e. "Semantic graph" poses, in which the operator traces shapes in space.

2) Programming techniques for registering specific gestures in each of the above categories relevant to a given application environment.

3) Algorithms for analyzing the gesture stream so that registered gestures can be identified and events encapsulating these gestures can be delivered to the relevant application environment.

The prescribed system (1) having constituent elements (1a) through (1f) provides the basis for exploiting the gesture analysis and translation capabilities of the system described herein.

A one-handed "pose" is denoted as

i) the string formed by the relative orientation between the fingers and the back of the hand,

ii) is quantized into a small number of discrete states.

The use of relative articulation orientations allows the systems described herein to avoid problems associated with different hand sizes and geometries. The system does not require "operator calibration". Additionally, specifying gestures as strings or sets of relative orientations allows for the easy creation of more complex gesture specifications by combining gesture representations with additional filters and specifications.

The use of a small number of discrete states for pose specification enables concise pose specification and the use of a variety of basic tracking techniques (e.g., passive optical tracking using cameras, active tracking using luminous points and cameras, electromagnetic field tracking, etc.) Gesture recognition becomes possible.

The poses in each category of (1a) to (1f) may be specified partially (or minimally) such that non-critical data is ignored. For example, a gesture in which the position of two fingers is unambiguous and the position of the other fingers is unimportant can be represented by a single specification in which the manipulative positions of the two relevant fingers are given, and within the same string, a "wildcard character" is listed for the other fingers ” or the general “ignore these” directive.

All of the innovations described here for gesture recognition (including but not limited to multilayered specification techniques, use of relative orientations, quantization of data, and allowing for partial or minimal specification at each level) go beyond the specification of human poses Rather, it extends to the provision of gestures made with other body parts and "artificial" tools and objects.

The programming technique for "registering gestures" (2) consists of a defined set of application programming interface calls that allow the programmer to define which gestures the engine should make available to other parts of the running system.

These API routines can be used at application build time, thereby creating static interface definitions that are used throughout the lifetime of the running application. They can also be used on the fly, allowing interface properties to change on the fly. This live change of interface makes it possible to:

i) Construct complex environments and conditional control states,

ii) dynamically add hysteresis to the control environment, and

iii) Create applications that enable users to change or extend the interface vocabulary of the running system itself.

The algorithm used to analyze the pose stream (3) compares the poses specified in (1) and registered in (2) to the incoming low-level pose data. When a match to a registered gesture is identified, event data representing the matched gesture is stacked to the running application.

Efficient real-time matching is desired in the design of this system, processing a given pose as a tree of possibilities that is processed as quickly as possible.

In addition, simple comparison operators used internally to recognize specified gestures are also exposed for use by application programmers, so that further comparisons (e.g., flexible state checks in complex or compound gestures) can even occur from within the application environment .

Recognizing "lock" semantics is an innovation of the system described here. These semantics are implied by the registry API (2) (and, to a lesser extent, embedded within the specified vocabulary (1)). Register API calls include,

i) "Entering" state notifier and "Ongoing" state notifier, and

ii) Gesture priority specifier.

If a gesture has been recognized, its "persistent" state takes precedence over all "entry" states of gestures of the same or lower priority. This distinction between entry and persistence states significantly increases perceived system availability.

The system described here includes algorithms for robust operation in the face of real world data errors and uncertainties. Data from low-level tracking systems may be incomplete (due to a number of reasons, including marker occlusion in optical tracking, network drops or processing lag, etc.).

Missing data is flagged by the analysis system and inserted into a "last known" or "probable" status, depending on the amount and context of the missing data.

If data about a particular pose component (e.g., the orientation of a particular joint) is missing, but the "last known" state of that particular component can be analyzed as physically possible, the system uses this last known state in its real-time matching. known state.

Conversely, if the last known state is analyzed as physically impossible, the system falls back to a "best guess range" for that component and uses this synthetic data when it matches in real time.

The specification and analysis system described here was carefully designed to support "handedness agnosticism" such that for multi-handed gestures, either hand is allowed to satisfy the gesture requirements.

Consistent virtual/display and physical spaces

The system may provide an environment in which a virtual space depicted on one or more display devices ("screens") is processed to coincide with the physical space in which one or more operators of the system reside. One embodiment of such an environment is described here. This current embodiment consists of three projector-driven screens at fixed locations, driven by a single desktop computer, and controlled using the gesture vocabulary and interface system described here. Note, however: the technology described supports any number of screens; these screens can be mobile (rather than fixed); these screens can be driven simultaneously by many independent computers; and the entire system can be controlled by any input device or technology .

The interface system described in this disclosure shall have a method of determining the dimensions, orientation and position of the screen in physical space. Given this information, the system is able to dynamically map the physical space in which these screens reside (and the operator of the system resides) as a projection into the virtual space of computer applications running on the system. As part of this automatic mapping, the system also translates the scale, angle, depth, scale, and other spatial characteristics of the two spaces in various ways, depending on the needs of the application hosted by the system.

This continuous translation between physical and virtual spaces enables the consistent and ubiquitous use of multiple interface technologies that are either difficult to implement on existing application platforms or must be specific to each application running on an existing platform implemented. These techniques include (but are not limited to):

1) Use "precision pointing" - using the hand in a gestural interface environment, or using a physical pointing tool or device - as a pervasive and natural interface technique.

2) Automatic compensation for movement or repositioning of the screen.

3) Graphics rendering that varies with operator position, such as simulating parallax displacement to enhance depth perception.

4) Include physical objects in the screen display - taking into account real world position, orientation, state, etc. For example, an operator standing in front of a large opaque screen can see both the application graphics and a representation of the true position of the scale model behind the screen (and possibly moving or changing orientation).

It is important to note that precise pointing differs from the abstract pointing used in mouse-based windowing interfaces and most other modern systems. In those systems, the operator must learn to manage the translation between the virtual pointing device and the physical pointing device, and must map cognitively between the two.

By comparison, in the system described in this disclosure, there is no difference between virtual and physical spaces, either from an application or user perspective (except that virtual spaces are more amenable to mathematical transformations), so no cognitive translation is required by the operator .

The closest analog to the precise pointing provided by the embodiments described herein is a touch screen (eg, found on many ATM machines). A touch screen provides a one-to-one mapping between the two-dimensional display space on the screen and the two-dimensional input space on the screen surface. In a similar fashion, the system described herein provides a flexible mapping (possibly but not necessarily a one-to-one mapping) between the virtual space displayed on one or more screens and the physical space in which the operator is located. Regardless of the practicality of the simulation, it is worth understanding that the extension of this "mapping method" to three dimensions, arbitrarily large architectural environments, and multiple screens is not trivial.

In addition to the components described here, the system implements a continuous system-level mapping (possibly modified by rotation, translation, scaling, or other geometric transformations) between the physical space of the environment and the display space on each screen. algorithm.

The rendering stack, which takes compute objects and maps, and outputs a graphical representation of the virtual space.

An input event processing stack that takes event data from the control system (in this example, gesture and pointing data from the system and mouse input) and maps the spatial data from the input events to coordinates in virtual space. The translated events are then delivered to the running application.

A "glue layer" that allows the system to host applications running between several computers on a local area network. Pose-based control using 3D information extracted within an extended depth of field

FIG. 9 is a block diagram of a gesture-based control system 900 including an imaging system that extracts three-dimensional information within an extended depth of field, under one embodiment. The user places hands 101 and 102 within the field of view of camera arrays 904A-904D. At least two of the cameras in arrays 904A-904D are wavefront-encoded cameras, each of which includes a wavefront-encoded imaging system that includes a wavefront-encoded mask (also referred to herein as an "aspherical optic" or "optical element") elements, as described in detail below. The user's hands and/or fingers may or may not include the aforementioned marking labels.

Cameras 904A-904D detect or capture images of fingers and hands 101 and 102 including their position, orientation and movement and generate output signals to pre-processor 905 . Preprocessor 905 may include or be coupled to wavefront encoded digital signal processing 908, as described below. Alternatively, wavefront encoded digital signal processing may be included in, coupled to, or distributed among one or more other components of system 900 . Wavefront encoded digital signal processing 908 is configured to greatly extend the depth of field of the imaging system.

The pre-processor 905 translates the camera output into attitude signals, which are provided to the computer processing unit 907 of the system. In doing so, the pre-processor 905 produces 3D space point reconstructions and bone point labeling sets. Gesture translator 906 translates the 3D spatial information and marker motion information into a command language that can be interpreted by a computer processor to update the position, shape and motion of the cursor on the display. Computer 907 uses the input information to generate commands to control one or more on-screen cursors and provides video output to display 903 .

One or more of the pre-processor 905, gesture translator 906 and computer 907 of an alternative embodiment may be combined into a single device. Regardless of system configuration, the function and/or functionality of each of pre-processor 905, pose translator 906, and computer 907 is as described above with reference to Figures 1-8 and elsewhere.

Furthermore, although the present example shows four cameras for detecting the position, orientation and movement of the user's hands 101 and 102, the present embodiment is not limited thereto. System configurations may include two or more cameras as required by the system or workstation configuration. Also, although the cameras are arranged symmetrically in the example embodiment, such symmetry is not required in the present invention. Hence, at least two cameras allowing any positioning of the position, orientation and movement of the user's hand may be used in the following.

Although the system is shown with a single user's hand as input, the system can track the hands of any number of multiple users. Additionally, instead of or in addition to hands, the system may track any one or more parts of the user's body, including head, feet, legs, arms, elbows, knees, and the like. Furthermore, the system can track any number of animate or inanimate objects and is not limited to tracking body parts.

In particular, for a gesture analysis system that positions an optical sensor in deliberate or potential proximity to an operator's hand (or equivalently tracked implement), the elements thus perceived will generally encompass the entire natural order of the operator's Several or many orders of magnitude of motion, relative distance. Sustained focus-resolved recording of events across this range of distances is beyond the capabilities of conventional optical imaging systems. However, these near mid-range geometries are often desirable in the context of tracking objects or operators for macroscopic device and product design purposes. Therefore, it is worthwhile to provide techniques that ensure local contrast or prominent feature stability over the expected range of operator motion (for this purpose, conventional optics are insufficient).

When extracting three-dimensional information within an extended depth of field in the system described here, the distance to a point in the scene can be estimated from its position in two or more images captured simultaneously. When the three-dimensional (3D) relationship between images is known, the 3D position of the point can be calculated from the underlying geometric relationship. The challenge in computing spatial location from multiple images (often referred to as stereo correlation or stereo depth computation) is to automatically and accurately relate the mapping of a point in one image to its mapping in another image. This is most commonly done by correlating image features from one image to one or more other images. However, a fundamental assumption in all stereo matching methods is that there must be some identifiable local contrast or feature in an image in order to match a point with its location in another image. Therefore, when there is no local contrast or features in the image due to defocus, a problem arises - stereo matching does not produce accurate results in areas of the image that are out of focus.

A conventional method for extending the depth of focus of an image is to reduce the diameter of the camera lens pupil ("zoom out"). However, two side effects limit the utility of this technique. First, the sensitivity of the imaging system is reduced by a factor equal to the square of the ratio of the inner and outer diameters of the pupil. Second, the maximum spatial frequency response is reduced by a factor equal to the ratio of the inner and outer pupil diameters, which limits resolution and contrast in the image. Therefore, there is a trade-off between depth of field, exposure time, and overall contrast in conventional imaging systems. In the case of a multi-camera ranging system, the net effect will be a tradeoff between stereo depth accuracy and working range.

An alternative to increasing the depth of field without shrinking the lens is to introduce a phase mask as required by regulation in the pupil of the camera lens. With a properly chosen phase function, the extended depth of field can be recovered by subsequent electronic processing of the image captured on the sensor. This technique, known as wavefront encoding, typically provides a trade-off between depth of field, camera dynamic range, and signal-to-noise ratio. Wavefront coding makes it possible to optimize camera parameters for specific applications. Applications that do not require a high dynamic range and where lighting is under user control, such as gesture recognition as described here, can greatly benefit from wavefront encoding to achieve high accuracy within a specified volume of space.

As noted above, the system of one embodiment includes techniques for using the processed output of multiple wavefront-encoded cameras to determine the extent and location of selected objects within a scene. The extended depth of field produced by wavefront encoding can be used in a variety of applications, including pose recognition and a host of other task-based imaging tasks, to significantly improve their performance. Although a minimum of two cameras is required, there is no upper limit to the number of cameras that can be used in this embodiment. Scene extraction may include any of a variety of processing techniques, such as correlation, for range extraction with two or more cameras. The embodiments described here include all wavefront encoded phase functions and their corresponding decoding kernels that produce extended depth of field after processing.

Wavefront encoding as used in wavefront encoding imaging systems is a general technique that uses generalized aspheric optics (device) and digital signal processing to greatly increase the performance and/or reduce the cost of an imaging system. The type of aspheric optics employed yields optical imaging properties that are very insensitive to defocus-related deviations. Sharp and clear images are not produced directly from the optics (device), however, digital signal processing applied to the sampled images produces a final image that is sharp and clear, also insensitive to defocus related biases.

Wavefront coding is used to greatly increase the performance of imaging systems while also reducing the size, weight, and cost of imaging systems. Wavefront encoding combines non-rotationally symmetric aspheric optics with digital signal processing in a fundamental way to greatly extend the depth of field of imaging systems. Using wavefront encoding, for example, for a given aperture size or F#, the depth of field or focus of an imaging system can be increased by a factor of ten or more relative to conventional imaging systems. The wavefront encoding optical element of one embodiment is a phase plane and thus does not absorb light or increase exposure or illumination requirements. It is impossible for traditional imaging technology to achieve this kind of extended depth-of-field performance without incurring a huge optical power loss (for example, when reducing the aperture). Increased depth of field/focus also enables imaging systems to be physically cheaper, smaller, or lighter by controlling defocus-related aberrations that have traditionally been controlled by adding lens elements or increasing lens complexity of. Defocus-related aberrations that can be controlled using wavefront encoding include chromatic aberration, Petzval curvature, astigmatism, spherical aberration, and temperature-dependent defocus.

Wavefront encoding as a hybrid imaging approach combines optics and electronics to increase depth of field and reduce the number of optical components, manufacturing tolerances, and overall system cost. Figure 10 is a block diagram of a wavefront encoded imaging system 1000 for use in a gesture-based control system, under one embodiment. The optical portion 1001 of the wavefront encoded imaging system 1000 is a conventional optical system or camera, but modified to place a wavefront encoded optical element 1002 near the aperture stop. Adding this encoded optical element results in an image with a specialized sharp blur or point spread function that is insensitive to defocus. Digital processing 1003 applied to the sampled image produces a sharp and clear image 1004 that is very insensitive to defocus effects.

FIG. 11 is a block diagram of a gesture-based control system 1100 employing a wavefront-encoded imaging system including two wavefront-encoded cameras to extract three-dimensional information over an extended depth of field, under one embodiment. As described above with reference to FIG. 10 , system 1100 includes at least two wavefront encoded cameras 1101 and 1102 . A processor is coupled to receive the outputs of the wavefront encoded cameras 1101 and 1102 and to perform data processing on the camera outputs. Data processing includes, for example, deconvolution 1120 and range extraction 1130 and produces an expanded focus range map 1140 .

In wavefront encoding system 1100 , the optical portion of the system (eg, wavefront encoding cameras 1101 and 1102 ) “encodes” the resulting image to produce intermediate image 1110 . The intermediate image 1110 appears defocused because a wavefront encoding element (such as element 1002 in Figure 10) purposely blurs all points in any image. In this intermediate image 1110, nearly all objects in the field of view are blurred, but they are equally blurred. In contrast, conventional optics typically form images with variable blur functions that depend on the distance from each object in the scene.

To produce a sharp and clear image from the intermediate wavefront encoded image 1110, the blurred intermediate image is processed or "decoded" 1120 using electronics (e.g., wavefront encoded digital signal processing) by removing system-dependent image blur and 1130. Digital filtering can be performed in real-time by software or using specialized hardware solutions.

As described above with reference to FIG. 10, the system optics of one embodiment include conventional components with at least one additional optical element performing the wavefront encoding function. This element is placed in the light path, usually near the system's aperture stop to minimize vignetting. The signal processing performed on the detected image relies on the first-order properties of the optics (device), wavefront encoding elements, and digital detectors.

Typical wavefront encoding elements are rotationally asymmetric and smooth, although diffractive surfaces can also be used. This element can be a separate component, or it can be integrated onto a conventional lens element by adding a generalized aspheric surface. All encoding elements redirect the light so that no rays hit the focal point of conventional geometry, except on-axis rays. In fact, no two rays hit the same point along the optical axis. The system does not produce sharp images at any image plane.

The main role of the optical part of a wavefront encoded imaging system is to make the resulting image insensitive to focus related deviations such as defocus, spherical aberration, astigmatism or field curvature. The intermediate blurred image is insensitive or invariant to changes in the object or imaging system, including defocus bias. From a system analysis point of view, the modulation transfer function (MTF) and point spread function (PSF) of a wavefront encoded system do not vary with respect to defocus.

Although the MTF of the intermediate image from the wavefront encoded system exhibits little variation with defocus, the MTF does have reduced power relative to an in-focus conventional system. Since no apodization is used, the total optical power is maintained. Digital filtering or image reconstruction processing is used to form a clear image. These final MTFs are very insensitive to defocus—thus, wavefront-encoded imaging systems have a large depth of field. Similarly, the intermediate PSFs from wavefront encoded systems differ from conventional system PSFs, but they vary very little with defocus.

Referring again to FIG. 10, specialized aspheric optical elements are placed at or near the aperture stop of a conventional imaging system to form a wavefront encoded imaging system. This optical element modifies the imaging system in such a way that the resulting PSF and optical transfer function (OTF) are insensitive to a range of defocus or defocus-related deviations. However, the PSF and OTF are not the same as those obtained with a high quality in-focus imaging system. The process of making the imaging system insensitive to defocus deviations produces images with specialized sharp blurring; this blurring is removed using wavefront encoded digital signal processing.

For example, the PSF from a conventional imaging system varies sharply with defocus, while the PSF from a wavefront encoded imaging system exhibits little noticeable variation with defocus. The digital processing applied to defocus by conventional imaging systems to remove defocus blur is processed depending on the amount of defocus present in different regions of the image. In many cases, the amount of defocus is unknown and difficult to calculate. In addition, the MTF of a defocused conventional imaging system may often contain zero or null values, further complicating the digital processing. In contrast, the constant nature of PSF with defocus from wavefront-encoded systems is exactly what is needed to remove the dependence of digital processing on defocus. The digital processing applied to the charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) detected image is independent of defocus and the actual scene being imaged. Additionally, the MTFs of both in-focus and out-of-focus wavefront encoded imaging systems contain zero or null values, allowing for high quality final images.

Wavefront encoding for extended depth of field can add value to imaging applications that would normally not be amenable to traditional methods (ie, reduced aperture). Constraints on illumination levels, exposure time, or spatial resolution often limit the application of existing optical methods. By using wavefront encoding, applications can benefit from a reduction in defocus-related issues without sacrificing exposure time or requiring large amounts of illumination.

As mentioned above, wavefront encoded imaging systems include digital signal processing of the resulting images and unconventional optical designs. The signal processing used depends on the particular optical system. The wavefront encoding optics (device) depend on the type and amount of signal processing to be used. Since optics (device) and signal processing are tightly coupled, systems in which optical and digital components are jointly optimized during the design process are naturally expected to have the best performance. The optical components are configured to minimize optical (device) variation with or sensitivity to defocus effects and to enable efficient signal processing. The digital components are designed to minimize algorithmic complexity, processing time, and the impact of digital processing on image noise.

Figure 12 is a flowchart of gesture-based control using three-dimensional information extracted within an extended depth of field, under one embodiment. Gesture-based control of an embodiment includes imaging 1202 the body with an imaging system. Imaging 1202 includes generating a wavefront encoded image of the body. The gesture-based control of one embodiment includes automatically detecting 1204 the gesture of the body, including the instantaneous state of the body. Detecting 1204 includes aggregating gesture data for a moment in time for the gesture. The pose data includes focus resolved data of the body within the depth of field of the imaging system. The gesture-based control of one embodiment includes translating 1206 the gesture into a gesture signal. Gesture-based control of one embodiment includes controlling 1208 components coupled to the computer in response to the gesture signal.

The basic routine for wavefront encoding for one embodiment may include a ray tracing program that traces rays through typical spherical and aspheric surfaces as well as general wavefront encoding surface forms. A ray tracing program is used to calculate the exit pupil and optimize a given set of optical and numerical merit functions or operands. Figure 13 is a block diagram of a wavefront code design process 1300 for use in an attitude-based control system, under one embodiment. The output of this design includes, but is not limited to: conventional optical surfaces, materials, thicknesses, and spacings; parameters of the wavefront-encoded surfaces; and digital filter coefficients.

A general optical/digital design loop is now described with reference to FIG. 13 . The ray tracing procedure 1302 traces rays through an optical surface to calculate an exit pupil path difference (OPD) 1304 and optimize a given set of optical and numerical quality functions or operands. Inputs to the ray tracing program 1302 include, for example, optical surfaces, thicknesses, and operating conditions (wavelength, field of view, temperature range, sample object images, etc.). OTFs are calculated or generated at 1306 and pixel OTFs related to the detector geometry are added at 1308 . At 1310 the sampled OTF and PSF are calculated. For a processing algorithm selected based on the sampled PSF, 1312 digital filter coefficients are generated. The process then forms the figure of merit for the filter (e.g., wavefront-encoded operands) based on minimizing both: sampled PSF and MTF due to aliasing, due to temperature variation, due to color, due to Variations in field angle, pass focus, etc.; digital processing parameters such as processing volume, processing form, processing related image noise, digital filter noise gain, etc. Wavefront encoded operands are combined with conventional optical operands (Seidel wavefront deviation, RMS wavefront error, etc.) by optimization routines to modify optical surfaces. Operation returns to generating 1302 the exit pupil path difference (OPD) by conventional ray tracing.

Use the theoretically calculated wavefront-encoded surface form as a starting point for optical optimization. A general family of rectangularly separable surface forms is given in normalized coordinates:

S(x)＝|β|sign(x)|x| ^α

Wherein for x>0, sign(x)=+1; for x≤0, sign(x)=-1.

The exponential parameter α controls the height of the MTF in the defocus range, and the parameter β controls the sensitivity to defocus. In general, increasing the parameter β reduces the sensitivity to defocus while reducing the height of the MTF and increasing the length of the resulting PSF.

The filtering process used to reconstruct the intermediate image and produce the final image can impose a computational burden. Depending on the depth-of-field enhancement introduced by the encoding process and the optics, the size of the filter kernel required for image reconstruction can be as much as 70×70 coefficients. Generally, the larger the depth of field extension, the larger the filter kernel and the worse the noise characteristics or the noise gain. Furthermore, since each pixel in the image is blurred by wavefront encoding, each pixel needs to be filtered; therefore, larger images may require more computation than smaller images. For image sizes approaching 10 million pixels, computationally efficient schemes are used for practical and economical systems. Computational implementations such as rectangularly separable filter approximations can help reduce kernel size. For example, the wavefront encoding element used may have the form of a rectangularly separable cubic phase as described by:

S(x,y)=a(x ³ +y ³ )

Filtering a blurred image to remove blur essentially applies magnification and phase shifting according to spatial frequency. This amplification increases both the signal and the noise in the final image. For very large (eg, more than 10 times) depth enhancements, the noise gain in the wavefront encoding system may be quadrupled or quintupled. For moderate depth-of-field enhancements of two to four times, the noise gain is usually two times or less.

For uncorrelated Gaussian noise (a good assumption for most images), the noise gain is the RMS value of the filter coefficients. For systems where the depth of field extension is too large to produce suitably small noise gain values, reducing the resolution or spatial bandwidth of the digital filter can reduce the noise gain. Reducing the contrast of the final image can also reduce the overall effect of increased noise. Specialized nonlinear filtering is the best solution to remove noise in wavefront encoded images.

Since the wavefront encoding optics used to form the MTF and PSF are rectangularly separable in one embodiment, the signal processing used may also be rectangularly separable. Rectangle separable processing can reduce the required computation by one or more orders of magnitude. Since digital filtering is performed using spatial convolutions, the computation method of one embodiment includes: a series of multiplications that scale the data by the filter coefficients; and a summation of all scaled data values throughout the kernel and. The basic unit of this calculation is the multiply-accumulate operation. A typical 2D wavefront encoding filter kernel for large depth-of-field augmentation may be 30x30 coefficients. A rectangular separable version of this filter contains a row filter with a length of 30 coefficients and a column filter with a height of 30 coefficients, or 60 total coefficients. Although the wavefront encoding elements may be rectangularly separable by design, they are not limited thereto, and non-separable filtering may be used for highly skewed systems.

By combining optical imaging techniques with electronic filtering, wavefront encoding improves the performance of a wide variety of imaging systems. Performance gains in high-performance imaging systems can include achieving a large depth of field without sacrificing light collection or spatial resolution. Increased performance of lower cost imaging systems can include achieving good image quality with fewer physical parts than traditionally required.

Embodiments described herein include a system comprising: a plurality of optical detectors, wherein at least two of the plurality of optical detectors comprise wavefront encoded cameras, wherein the plurality of optical detectors body imaging; a processor coupled to the plurality of optical detectors, the processor automatically detecting a pose of the body, wherein the pose includes an instantaneous state of the body, wherein the detecting includes gathering pose data for a moment in time of the pose, the pose data Including focus-resolved data of the body within a depth of field of the imaging system, the processor translates the pose into a pose signal and uses the pose signal to control components coupled to the processor.

The wavefront encoded camera of one embodiment includes wavefront encoded optics.

Imaging of one embodiment includes generating a wavefront encoded image of the body.

The wavefront encoded camera of one embodiment includes a phase mask that increases the depth of focus of imaging.

The pose data of one embodiment includes focus resolution range data of the body within the depth of field.

The body's focus resolution range data within depth of field of one embodiment comes from the output of a wavefront encoded camera.

The pose data of one embodiment includes focus-resolved position data of the body within the depth of field.

The focus resolved position data of the body within the depth of field of one embodiment comes from the output of the wavefront encoded camera.

The system of one embodiment includes a modulation transfer function and a point spread function that do not vary with distance between the body and the imaging system.

The system of one embodiment includes a modulation transfer function and a point spread function that do not vary with respect to defocus.

The processor of one embodiment generates the intermediate image by encoding the image collected by the wavefront encoded camera.

The intermediate image of one embodiment is blurred.

The intermediate image of one embodiment is insensitive to multiple optical detectors or body variations including defocus bias.

The gesture data of one embodiment is three-dimensional spatial position data representing gestures.

The detecting of an embodiment includes at least one of detecting a position of the body, detecting an orientation of the body, and detecting includes detecting motion of the body.

The detecting of an embodiment includes identifying a pose, where identifying includes identifying a pose and orientation of a part of the body.

The detecting of an embodiment includes detecting at least one of the first set of appendages and the second set of appendages of the body.

The detecting of an embodiment comprises dynamically detecting the position of at least one tag.

The detecting of an embodiment includes detecting the position of a set of tags coupled to the body part.

Each label of the set of labels of an embodiment includes a pattern, wherein each pattern of each label of the set of labels is different from any pattern of any remaining labels of the plurality of labels.

The detection of one embodiment includes dynamically detecting and locating markers on the body.

The detecting of one embodiment includes detecting the position of a set of markers coupled to the body part.

The set of markings of one embodiment form a plurality of patterns on the body.

The detecting of an embodiment includes detecting the position of the plurality of appendages of the body using a set of markers coupled to each appendage.

The translation of one embodiment includes translating information of gestures into gesture tokens.

The gesture token of one embodiment represents a gesture vocabulary, and the gesture signal includes a conveyance of the gesture vocabulary.

The pose vocabulary of one embodiment represents in textual form the instantaneous pose state of the body's kinematic linkages.

The gesture vocabulary of one embodiment represents the orientation of the body's kinematic linkages in textual form.

The gesture vocabulary of one embodiment represents, in textual form, combinations of orientations of the body's kinematic linkages.

The gesture vocabulary of one embodiment includes character strings representing states of the body's kinematic linkages.

The kinematic linkage of an embodiment is at least one first appendage of the body.

The system of one embodiment includes assigning each position in the string to a second appendage, the second appendage being connected to the first appendage.

The system of one embodiment includes assigning a character of the plurality of characters to each of the plurality of positions of the second appendage.

The plurality of positions of one embodiment are established relative to a coordinate origin.

The system of one embodiment includes: establishing an origin of coordinates using an absolute position and orientation in space; establishing an origin of coordinates using a fixed position and orientation relative to the body regardless of the overall position and orientation of the body; or responding to motion of the body Instead, establish the coordinate origin interactively.

The system of one embodiment includes assigning a character of the plurality of characters to each of the plurality of orientations of the first appendage.

The detecting of one embodiment includes detecting when the inferred position of the body intersects a virtual space, wherein the virtual space includes a space depicted on a display device coupled to the computer.

Controlling the component of one embodiment includes controlling the virtual object when the inferred position intersects the virtual object in the virtual space.

Controlling the component of one embodiment includes controlling a position of the virtual object in the virtual space in response to the inferred position in the virtual space.

Controlling the component of an embodiment includes controlling a gesture of the virtual object in the virtual space in response to the gesture.

The system of one embodiment includes proportional control of detection and control to create a correspondence between a virtual space including a space depicted on a display device coupled to a processor, and a physical space including a space in which a body is located. space.

The system of one embodiment includes controlling at least one virtual object in a virtual space in response to movement of at least one physical object in the physical space.

The control of an embodiment includes at least one of: controlling the execution of an application hosted on the processor; and controlling a component displayed on the processor.

Embodiments described herein include a method comprising: imaging a body with an imaging system, the imaging comprising generating a wavefront encoded image of the body; automatically detecting a pose of the body, wherein the pose includes a momentary state of the body, wherein the Detecting includes aggregating pose data for a moment in time of the pose, the pose data including focus resolved data of the body within a depth of field of the imaging system; translating the pose into a pose signal; or controlling a component coupled to the computer in response to the pose signal.

The imaging system of one embodiment includes a plurality of optical detectors, wherein at least two of the optical detectors are wavefront-encoded cameras that include wavefront-encoded optics.

Imaging of one embodiment includes generating a wavefront encoded image of the body.

The imaging system of one embodiment includes a plurality of optical detectors, wherein at least two of the optical detectors are wavefront encoded cameras including a phase mask that increases the depth of focus of imaging.

The pose data of one embodiment includes focus resolution range data of the body within the depth of field.

The focus resolution range data of the body within the depth of field of one embodiment comes from the output of the imaging system.

The pose data of one embodiment includes focus-resolved position data of the body within the depth of field.

The focus resolved position data of the body within the depth of field of one embodiment comes from the output of the imaging system.

The method of one embodiment includes generating a modulation transfer function and a point spread function that are invariant to the distance between the body and the imaging system.

The method of one embodiment includes generating a modulation transfer function and a point spread function that do not vary with respect to defocus.

The method of one embodiment includes generating an intermediate image by encoding images collected by a wavefront encoded camera.

The intermediate image of one embodiment is blurred.

The intermediate image of one embodiment is insensitive to multiple optical detectors of the imaging system or to changes in the body including defocus bias.

The gesture data of one embodiment is three-dimensional spatial position data representing gestures.

The detecting of one embodiment includes detecting the position of the body.

The detection of one embodiment includes detecting the orientation of the body.

The detection of one embodiment includes detecting motion of the body.

The detecting of an embodiment includes identifying a pose, where identifying includes identifying a pose and orientation of a part of the body.

The detecting of an embodiment includes detecting at least one of the first set of appendages and the second set of appendages of the body.

The detecting of an embodiment comprises dynamically detecting the position of at least one tag.

The detecting of an embodiment includes detecting the position of a set of tags coupled to the body part.

Each label of the set of labels of an embodiment includes a pattern, wherein each pattern of each label of the set of labels is different from any pattern of any remaining labels of the plurality of labels.

The detection of one embodiment includes dynamically detecting and locating markers on the body.

The detecting of one embodiment includes detecting the position of a set of markers coupled to the body part.

The set of markings of one embodiment form a plurality of patterns on the body.

The detecting of an embodiment includes detecting the position of the plurality of appendages of the body using a set of markers coupled to each appendage.

The translation of one embodiment includes translating information of gestures into gesture tokens.

The gesture token of one embodiment represents a gesture vocabulary, and the gesture signal includes a conveyance of the gesture vocabulary.

The pose vocabulary of one embodiment represents in textual form the instantaneous pose state of the body's kinematic linkages.

The gesture vocabulary of one embodiment represents the orientation of the body's kinematic linkages in textual form.

The gesture vocabulary of one embodiment represents, in textual form, combinations of orientations of the body's kinematic linkages.

The gesture vocabulary of one embodiment includes character strings representing states of the body's kinematic linkages.

The kinematic linkage of an embodiment is at least one first appendage of the body.

The method of one embodiment includes assigning each position in the string to a second appendage, the second appendage being connected to the first appendage.

The method of one embodiment includes assigning a character of the plurality of characters to each of the plurality of positions of the second appendage.

The plurality of positions of one embodiment are established relative to a coordinate origin.

The method of one embodiment includes: establishing the coordinate origin using an absolute position and orientation in space; establishing the coordinate origin using a fixed position and orientation relative to the body, regardless of the body's overall position and orientation; or responding to body motion Instead, establish the coordinate origin interactively.

The method of one embodiment includes assigning a character of the plurality of characters to each of the plurality of orientations of the first appendage.

The detecting of one embodiment includes detecting when the inferred position of the body intersects a virtual space, wherein the virtual space includes a space depicted on a display device coupled to the computer.

Controlling the component of one embodiment includes controlling the virtual object when the inferred position intersects the virtual object in the virtual space.

Controlling the component of one embodiment includes controlling a position of the virtual object in the virtual space in response to the inferred position in the virtual space.

Controlling the component of an embodiment includes controlling a gesture of the virtual object in the virtual space in response to the gesture.

The method of one embodiment includes scaling the sensing and controlling to create a coincidence between a virtual space including a space depicted on a display device coupled to a processor, and a physical space including a physical space where a body is located. Space.

The method of one embodiment includes translating scale, angle, depth and scale between virtual space and physical space as required by at least one application coupled to the processor.

The method of an embodiment includes controlling at least one virtual object in a virtual space in response to movement of at least one physical object in the physical space.

The controlling of one embodiment includes controlling the execution of applications hosted on the processor.

Control of one embodiment includes controlling components displayed on the processor.

The systems and methods described herein include and/or operate under and/or in association with a processing system. As is known in the art, a processing system includes components of a processing system or device, or any collection of computing or processor-based devices working together. For example, a processing system may include one or more of a portable computer, a portable communication device operating in a communication network, and/or a network server. A portable computer may be any number and/or combination of devices selected from, but not limited to, a personal computer, a mobile phone, a personal digital assistant, a portable computing device, and a portable communication device. The processing system may include components within a larger computer system.

The processing system of one embodiment includes at least one processor and at least one memory device or subsystem. The processing system may also include or be coupled to at least one database. The term "processor" is used broadly herein to refer to any logical processing unit, such as one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), and the like. The processor and memory may be monolithically integrated on a single chip, distributed among multiple chips or components in the host system, and/or provided by some algorithmic combination. The methods described here can be implemented by any combination of one or more of software algorithms, programs, firmware, hardware, components, and circuits.

System components embodying the systems and methods described herein may be located together or in separate locations. Thus, system components embodying the systems and methods described herein may be components of a single system, multiple systems, and/or geographically separated systems. These components may also be subcomponents or subsystems of a single system, multiple systems, and/or geographically separated systems. These components may be coupled to one or more other components of the host system or a system coupled to the host system.

Communication paths couple system components and include the media used to communicate or transfer files between the components. Communication paths include wireless connections, wired connections, and hybrid wireless/wired connections. Communication paths also include couplings or connections to networks including Local Area Networks (LANs), Metropolitan Area Networks (MANs), World Wide Webs (WANs), private networks, office or backend networks, and the Internet. Additionally, communication paths include removable fixed media such as floppy disks, hard drives, and CD-ROM disks, as well as flash RAM, Universal Serial Bus (USB) connections, RS-232 connections, telephone lines, bus, and email messages.

Unless the context clearly requires otherwise, throughout this description the words "comprise", "comprising", etc. are to be construed in an inclusive sense as opposed to an exclusive or exhaustive meaning; that is, with The meaning of "including but not limited to" shall be interpreted. Likewise, words using the singular or the plural include the plural or the singular respectively. Additionally, the words "herein," "hereafter," "above," "below," and words of similar import refer to this application as a whole and not to any particular portions of this application. When the word "or" is used with reference to a list of two or more items, that word covers all of the following constructions of that word: any item in that list, all items in that list, and any combination of items in that list .

The above description of embodiments of a processing environment is not intended to be exhaustive, and the systems and methods described are not limited to the precise forms disclosed. While specific embodiments of, and examples for, processing environments are described herein for illustrative purposes, various equivalent modifications are possible within the scope of other systems and methods, those skilled in the art will recognize. The teachings of the processing environment provided here can be applied to other processing systems and methods, not just the ones described above.

The elements and operations of the various embodiments described above can be combined to provide further embodiments. These and other changes to the processing environment can be made in light of the detailed description above.

Claims (90)

1. use the three-dimensional information extracted in the depth of field of extension to carry out a system for control based on attitude, including:

Multiple fluorescence detectors, at least two fluorescence detector in wherein said multiple fluorescence detectors includes wavefront coded phase Machine, wherein said multiple fluorescence detectors are to body imaging；And

The processor coupled with the plurality of fluorescence detector, described processor detects the attitude of health, wherein said appearance automatically State include the instantaneous state of described health, wherein said detection include only assembling flashy described attitude attitude data and Getting rid of the use to background data in the detection to described attitude, described attitude data includes that described health is relative to as sky Absolute position and the focus of the neutral position of orientation between differentiate position data, and wherein said position data is three-dimensional information, Described attitude is translated into attitude signal and uses described attitude signal to control to couple with described processor by described processor Parts.

System the most according to claim 1, wherein said wavefront coded camera includes wavefront coded optical element.

System the most according to claim 1, wherein said imaging includes the wavefront coded image producing described health.

System the most according to claim 1, wherein said wavefront coded camera includes the phase increasing the depth of focus of described imaging Bit mask.

System the most according to claim 1, wherein said attitude data includes that the focus of the described health in the depth of field is differentiated Range data.

System the most according to claim 5, the described focus resolving range data of the described health in the wherein said depth of field Come from the output of described wavefront coded camera.

System the most according to claim 1, wherein said attitude data includes that the focus of the described health in the depth of field is differentiated Position data.

System the most according to claim 7, the described focus of the described health in the wherein said depth of field differentiates position data Come from the output of described wavefront coded camera.

System the most according to claim 1, including with the distance between described health and the plurality of fluorescence detector And the modulation transfer function (MTF) changed and point spread function.

System the most according to claim 1, the modulation transfer function (MTF) changed including being not in relation to defocus and point spread function Number.

11. systems according to claim 1, wherein said processor is by being collected by described wavefront coded camera Image carries out coding to produce intermediate image.

12. systems according to claim 11, wherein said intermediate image is fuzzy.

13. systems according to claim 11, wherein said intermediate image defocuses the plurality of optics of deviation to comprising The change of detector or described health is insensitive.

14. systems according to claim 1, wherein said attitude data is the three-dimensional space position number representing described attitude According to.

15. systems according to claim 1, wherein said detection includes detecting the position of described health, detecting described body At least one in the orientation of body, and detection include detecting the motion of described health.

16. systems according to claim 1, wherein said detection includes identifying described attitude, and wherein said mark includes Identify posture and the orientation of a part for described health.

17. systems according to claim 1, wherein said detection includes the first group of appurtenance and detecting described health At least one in two groups of appurtenances.

18. systems according to claim 1, wherein said detection includes the position dynamically detecting at least one label.

19. systems according to claim 18, wherein said detection includes detecting what the part with described health coupled The position of one group of label.

20. systems according to claim 19, wherein each label in this group label includes pattern, wherein this group label In any pattern of any residue label of being different from this group label of each pattern of each label.

21. systems according to claim 1, wherein said detection includes dynamically detecting and position the mark on described health Note.

22. systems according to claim 21, wherein said detection includes detecting what the part with described health coupled The position of one group echo.

23. systems according to claim 22, wherein this group echo forms the multiple patterns on described health.

24. systems according to claim 21, wherein said detection includes using in the multiple appurtenances with described health Each appurtenance coupling a group echo detect the plurality of appendicular position.

25. systems according to claim 1, wherein said translation includes the information of described attitude is translated into attitude note Number.

26. system according to claim 25, wherein said attitude mark represents attitude vocabulary, and described attitude signal bag Include the reception and registration of described attitude vocabulary.

27. systems according to claim 26, wherein said attitude vocabulary represents the motion of described health in the form of text Learn the momentary gesture state of link gear.

28. systems according to claim 26, wherein said attitude vocabulary represents the motion of described health in the form of text Learn the orientation of link gear.

29. systems according to claim 26, wherein said attitude vocabulary represents the motion of described health in the form of text Learn the combination of the orientation of link gear.

30. systems according to claim 26, wherein said attitude vocabulary includes the kinesiology linkage representing described health The character string of the state of mechanism.

31. systems according to claim 30, wherein said kinesiology link gear be described health at least one One appurtenance.

32. systems according to claim 31, distribute to the second appurtenance including by each position in described character string, Described second appurtenance is connected to described first appurtenance.

33. systems according to claim 32, distribute to institute including by the character in the multiple characters in described character string That states in second appendicular multiple positions is each.

34. systems according to claim 33, wherein said multiple positions are established relative to zero.

35. systems according to claim 34, including: use the absolute position in space and orientation to establish described coordinate Initial point；Use and establish described zero, regardless of whether described health is total relative to the fixed position of described health and orientation Body position and towards how；Or interactively establish described zero in response to the action of described health.

36. systems according to claim 33, distribute including by the character in the plurality of character in described character string Each in described first appendicular multiple orientations.

37. systems according to claim 31, wherein said detection include detecting the inferred position of described health when with Virtual Space is intersected, and wherein said Virtual Space includes the space being depicted in the display device coupled with described processor.

38. according to the system described in claim 37, wherein controls described parts and includes: when described inferred position is virtual with described Dummy object in space controls described dummy object when intersecting.

39., according to the system described in claim 38, wherein control described parts and include: in response to the institute in described Virtual Space State inferred position and control the position of the described dummy object in described Virtual Space.

40. according to the system described in claim 38, wherein controls described parts and includes: control described in response to described attitude The attitude of the described dummy object in Virtual Space.

41. systems according to claim 1, control including described detection and control carry out ratio to produce Virtual Space Consistent with between physical space, wherein said Virtual Space includes being depicted in the display device coupled with described processor Space, wherein said physical space includes the space residing for described health.

42. systems according to claim 41, including in response at least one physical objects in described physical space Move and control at least one dummy object in described Virtual Space.

43. systems according to claim 1, wherein said control includes at least one in following control: control described The operation of the application administered on processor；With the parts of display on the described processor of control.

44. 1 kinds use the method that the three-dimensional information extracted in the depth of field of extension carries out control based on attitude, including:

Utilizing imaging system to body imaging, described imaging includes the wavefront coded image producing described health；

Automatically detecting the attitude of health, wherein said attitude includes that the instantaneous state of described health, wherein said detection only include Assemble the attitude data of flashy described attitude and in the detection to described attitude, get rid of the use to background data, institute State attitude data and include that described health differentiates position relative to the focus of the neutral position as the absolute position in space and orientation Putting data, wherein said position data is three-dimensional information；

Described attitude is translated into attitude signal；And

The parts coupled with computer are controlled in response to described attitude signal.

45. methods according to claim 44, wherein said imaging system includes multiple fluorescence detector, wherein said light At least two learned in detector is to include the wavefront coded camera of wavefront coded optical element.

46. methods according to claim 44, wherein said imaging includes the wavefront coded image producing described health.

47. methods according to claim 44, wherein said imaging system includes multiple fluorescence detector, wherein said light Learn the wavefront coded camera that at least two in detector is the phase mask including increasing the depth of focus of described imaging.

48. methods according to claim 44, wherein said attitude data includes that the focus of the described health in the depth of field is divided Distinguish range data.

49. methods according to claim 48, the described focus resolving range number of the described health in the wherein said depth of field According to the output coming from described imaging system.

50. methods according to claim 44, wherein said attitude data includes that the focus of the described health in the depth of field is divided Distinguish position data.

51. methods according to claim 50, the described focus of the described health in the wherein said depth of field differentiates positional number According to the output coming from described imaging system.

52. methods according to claim 44, including producing not with the distance between described health and described imaging system And the modulation transfer function (MTF) changed and point spread function.

53. methods according to claim 44, including producing the modulation transfer function (MTF) being not in relation to defocus and change and some expansion Dissipate function.

54. methods according to claim 44, including by carrying out the image collected by described wavefront coded camera Coding produces intermediate image.

55. methods according to claim 54, wherein said intermediate image is fuzzy.

56. methods according to claim 54, wherein said intermediate image defocuses the described imaging system of deviation to comprising Multiple fluorescence detectors or the change of described health insensitive.

57. methods according to claim 44, wherein said attitude data is the three-dimensional space position representing described attitude Data.

58. methods according to claim 44, wherein said detection includes the position detecting described health.

59. methods according to claim 44, wherein said detection includes the orientation detecting described health.

60. methods according to claim 44, wherein said detection includes the motion detecting described health.

61. methods according to claim 44, wherein said detection includes identifying described attitude, and wherein said mark includes Identify posture and the orientation of a part for described health.

62. methods according to claim 44, wherein said detection include detecting described health first group of appurtenance and At least one in second group of appurtenance.

63. methods according to claim 44, wherein said detection includes the position dynamically detecting at least one label.

64. methods according to claim 63, wherein said detection includes detecting what the part with described health coupled The position of one group of label.

65. methods according to claim 64, wherein each label in this group label includes pattern, wherein this group label In any pattern of any residue label of being different from this group label of each pattern of each label.

66. methods according to claim 44, wherein said detection includes dynamically detecting and position the mark on described health Note.

67. methods according to claim 66, wherein said detection includes detecting what the part with described health coupled The position of one group echo.

68. methods according to claim 67, wherein this group echo forms the multiple patterns on described health.

69. methods according to claim 66, wherein said detection includes using in the multiple appurtenances with described health Each appurtenance coupling a group echo detect the plurality of appendicular position.

70. methods according to claim 44, wherein said translation includes the information of described attitude is translated into attitude note Number.

71. methods according to claim 70, wherein said attitude mark represents attitude vocabulary, and described attitude signal bag Include the reception and registration of described attitude vocabulary.

72. represent the motion of described health in the form of text according to the method described in claim 71, wherein said attitude vocabulary Learn the momentary gesture state of link gear.

73. represent the motion of described health in the form of text according to the method described in claim 71, wherein said attitude vocabulary Learn the orientation of link gear.

74. represent the motion of described health in the form of text according to the method described in claim 71, wherein said attitude vocabulary Learn the combination of the orientation of link gear.

75. according to the method described in claim 71, and wherein said attitude vocabulary includes the kinesiology linkage representing described health The character string of the state of mechanism.

76. according to the method described in claim 75, wherein said kinesiology link gear be described health at least one One appurtenance.

77., according to the method described in claim 76, distribute to the second appurtenance including by each position in described character string, Described second appurtenance is connected to described first appurtenance.

78., according to the method described in claim 77, distribute to institute including by the character in the multiple characters in described character string That states in second appendicular multiple positions is each.

79. according to the method described in claim 78, and wherein said multiple positions are established relative to zero.

80. according to the method described in claim 79, including: use the absolute position in space and orientation to establish described coordinate Initial point；Use and establish described zero, regardless of whether described health is total relative to the fixed position of described health and orientation Body position and towards how；Or interactively establish described zero in response to the action of described health.

81., according to the method described in claim 78, distribute including by the character in the plurality of character in described character string Each in described first appendicular multiple orientations.

82. according to the method described in claim 76, wherein said detection include detecting the inferred position of described health when with Virtual Space is intersected, and wherein said Virtual Space includes the space being depicted in the display device coupled with described computer.

83. methods described in 2 according to Claim 8, wherein control described parts and include: when described inferred position is virtual with described Dummy object in space controls described dummy object when intersecting.

84. methods described in 3 according to Claim 8, wherein control described parts and include: in response to the institute in described Virtual Space State inferred position and control the position of the described dummy object in described Virtual Space.

85. methods described in 3 according to Claim 8, wherein control described parts and include: control in response to described attitude described The attitude of the described dummy object in Virtual Space.

86. methods according to claim 44, control including described detection and control carry out ratio to produce virtual sky Between consistent with between physical space, wherein said Virtual Space includes being depicted in the display device coupled with described computer Space, wherein said physical space includes the space residing for described health.

87. methods described in 6 according to Claim 8, including the needs according at least one application coupled with described computer, Ratio, angle, the degree of depth and yardstick is translated between described Virtual Space and described physical space.

88. methods described in 6 according to Claim 8, including in response at least one physical objects in described physical space Move and control at least one dummy object in described Virtual Space.

89. methods according to claim 44, the application that wherein said control includes controlling administering on described computer Run.

90. methods according to claim 44, wherein said control includes controlling the parts of display on described computer.