HK1183105B - See-through display brightness control - Google Patents

Description

See-through display brightness control

Technical Field

The present invention relates to see-through displays, and more particularly to see-through display brightness control.

Background

Augmented or mixed reality is a technique that allows virtual images to be mixed with the user's actual view of the real world. The see-through, near-eye display may be worn by a user to view a blended image of virtual and real objects. The display displays a virtual image in a field of view of a user.

However, there is a need to protect the eyes of the wearer from too much light from the display. The brightness of a display is a combination of how much external light passes through the display and how much light is used to display an image. Thus, if too much external light is illuminated through the display, the user may experience eye damage or discomfort. Also, if too much light is used to display the image, the user may suffer eye discomfort or injury. Furthermore, there is a need to protect the eyes of the user while still allowing the user to properly see virtual and real objects.

Disclosure of Invention

The technology provides various embodiments for controlling the brightness of a see-through, near-eye hybrid display device based on the light intensity of the object at which the user is looking. The opacity of the display can be changed so that if the wearer is seeing a bright object, the external light is reduced. The wearer's pupil size may be determined and used to adjust the brightness for displaying the image ("image brightness") and the opacity of the display. A suitable balance between opacity and image brightness can be achieved to allow real and virtual objects to be seen clearly without causing damage or discomfort to the wearer's eyes.

One embodiment includes estimating an area at which a wearer of a see-through display is gazing using an eye-tracking camera, determining a light intensity of the area at which the user is gazing, and adjusting a brightness of the see-through display based on the light intensity of the area.

One embodiment includes a display system including a see-through, near-eye display device including a respective display optical system for each eye positioned to be seen through by the respective eye. The system further comprises an image generation unit for each eye attached to the see-through display device for generating at least one virtual image for display in the display optical system. Furthermore, the system has a corresponding arrangement of gaze detection elements positioned on the display device. The system also includes logic coupled to the gaze detection element. The logic determines a gaze estimate for a wearer of the see-through display device. The logic accesses brightness data for a field of view of the system. The logic determines the light intensity of the area being gazed at. The logic adjusts the brightness of the see-through display device based on the light intensity of the region.

One embodiment includes a method comprising the following. A field of view of a wearer of the see-through display is tracked using a first camera. A gaze vector of at least one eye of the wearer is determined using a second camera. Determining a brightness of a region in the field of view at which the wearer is gazing based on the tracking and the gaze vector. The pupil size of the wearer is determined. The brightness of the see-through display is adjusted based on the light intensity of the region and the pupil size.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Drawings

FIG. 1A is a block diagram depicting example components of one embodiment of a see-through, mixed reality display device with brightness control.

FIG. 1B is a block diagram depicting example components of another embodiment of a see-through, mixed reality display device with brightness control.

Fig. 1C illustrates an exemplary arrangement of locations of respective sets of gaze detection elements in a gaze detection system for each eye positioned to face the respective eye on a mixed reality display device provided in a pair of glasses.

Fig. 1D illustrates another exemplary arrangement of locations of respective sets of gaze detection elements in a gaze detection system for each eye positioned to face the respective eye on a mixed reality display device provided in a pair of glasses.

Fig. 1E illustrates yet another exemplary arrangement of the locations of respective sets of gaze detection elements in a gaze detection system for each eye positioned to face the respective eye by the pair of glasses.

Fig. 1F is a block diagram of one embodiment of a system for adjusting brightness based on gaze estimation and pupil size.

Fig. 1G is a flow diagram of one embodiment of a process of adjusting brightness based on gaze estimation and optionally pupil size.

FIG. 2 is a side view of a temple arm in an embodiment of a mixed reality display device that provides support for hardware and software components.

Fig. 3A is a top view of an embodiment of a display optical system of a see-through, near-eye, mixed reality device including an arrangement of gaze-detection elements in a gaze-detection system.

Fig. 3B is a top view of another embodiment of a display optical system of a see-through, near-eye, mixed reality device including an arrangement of gaze-detection elements in a gaze-detection system.

Fig. 3C is a top view of a third embodiment of a display optical system of a see-through, near-eye, mixed reality device including an arrangement of gaze-detection elements in a gaze-detection system.

Fig. 3D is a top view of a fourth embodiment of a display optical system of a see-through, near-eye, mixed reality device including an arrangement of gaze-detection elements in a gaze-detection system.

FIG. 4A is a block diagram of one embodiment of hardware and software components of a see-through, near-eye, mixed reality display unit that may be used with the embodiment of FIG. 2.

FIG. 4B is a block diagram of one embodiment of hardware and software components of a processing unit associated with a see-through, near-eye, mixed reality display unit.

FIG. 5A is a flow diagram of one embodiment of determining the brightness of an area at which a wearer of a see-through display is gazing.

Fig. 5B is a top view illustrating an example of gaze vectors that intersect at a gaze point at which a user's eyes are focused.

Figure 5C is a flow diagram of one embodiment of determining pupil size.

Figure 5D illustrates one embodiment of a model for corneal and pupil images.

FIG. 5E shows one embodiment of a model for the cornea, with a center E_iThe associated pupil image of (a).

FIG. 5F is a flow diagram of one embodiment of determining user-specific parameters for eye tracking.

Figure 5G is a flow diagram of one embodiment of a process for determining baseline user-specific parameters.

FIG. 5H is a flow diagram of one embodiment for gradually changing the display brightness based on a user baseline value.

FIG. 5I is a flow diagram of one embodiment of varying the brightness of a display based on the distance of the user's eyes from the see-through display.

FIG. 5J is a flow diagram of one embodiment of adjusting display brightness based on image context.

FIG. 6A is a flow diagram of an embodiment of a method for determining a three-dimensional user field of view.

FIG. 6B is a flow diagram of an embodiment of a method for identifying one or more real objects in a user field of view.

FIG. 6C is a flow diagram of an embodiment of a method for generating a three-dimensional model of a user space.

FIG. 6D is a flow diagram of an embodiment of a method for identifying one or more real objects in a user field of view.

FIG. 6E is a flow diagram of an embodiment of a method for identifying one or more real objects in a user field of view.

FIG. 6F is a block diagram of an embodiment of a system for determining a position of an object within a user's field of view of a see-through, near-eye display device.

Fig. 7 is a flow diagram of an embodiment of a method for determining gaze in a see-through, near-eye, mixed reality display system.

FIG. 8 is a flow diagram of an embodiment of a method for identifying flashes in image data.

Fig. 9 is a flow diagram of an embodiment of a method that may be used to determine boundaries of a gaze-detection coordinate system.

Fig. 10 is a flow diagram illustrating an embodiment of a method for determining the location of the corneal center in a coordinate system using an optical gaze detection element of a see-through, near-eye, mixed reality display.

Fig. 11 provides an illustrative example of defining a plane using the geometry provided by the arrangement of optical elements to form a gaze-detection coordinate system that may be used by the embodiment of fig. 10 to find the center of the cornea.

Figure 12 is a flow chart illustrating an embodiment of a method for determining pupil center from sensor generated image data.

Fig. 13 is a flow diagram illustrating an embodiment of a method for determining a gaze vector based on a determined pupil center, a corneal center, and a center of rotation of an eyeball.

Fig. 14 is a flow diagram illustrating an embodiment of a method for determining gaze based on glint data.

Fig. 15A is a flow diagram illustrating an embodiment of a method for generating a set of training data sets for comparison-based determination of gaze.

Fig. 15B is a flow diagram illustrating an embodiment of a method for determining gaze based on a training data set.

FIG. 15C is a flow diagram of an embodiment of an interpolation method that may be used with the comparison step of FIG. 14.

Fig. 16 is a flow diagram illustrating an embodiment of a method for checking whether a recalibration of the training gaze data set is to be completed.

FIG. 17 is a block diagram of an exemplary mobile device that may operate in embodiments of the present technology.

FIG. 18 is a block diagram of one embodiment of a computing system that may be used to implement a hub computing system.

Detailed Description

The techniques described herein provide various embodiments for adjusting the brightness of a see-through, near-eye display. One possible adjustment is to change the brightness for displaying the image ("image brightness"). Another possible adjustment is to change the opacity of the display so that more or less external light will pass into the wearer's eye. These adjustments may be balanced against each other so that real and virtual objects may be properly viewed.

In one embodiment, the wearer's eyes are tracked to determine where the user is looking. The light intensity of the real world object the user is looking at is then determined. The light intensity may be used to adjust opacity and/or image brightness. In one embodiment, the pupil size of the wearer is tracked. The pupil size may be used to determine how to adjust the opacity and/or image brightness. Note that both the light intensity from the object and the pupil size may be factored in the determination.

In some embodiments, the gaze estimation is used to determine how to adjust the display brightness. Gaze is sometimes referred to as the line of sight or visual axis of the eye. The visual axis extends from the fovea (sometimes referred to as the foveal centre) of the retina through the centre of the pupil. Extending the visual axis from the fovea through the pupil and the see-through lens of each eye, the point of regard can be determined in the user's field of view, which can include an image of a virtual object as well as an actual direct view of the real world. In one embodiment, the light intensity of the area at which the user is gazing is determined.

One technique for determining gaze may be based on glint data or a combination of glints and image data. In certain embodiments, determining the coordinate system based on gaze of a predetermined location of at least one light sensor and the illuminators on the display device provides a three-dimensional (3D) spatial relationship between the display device and each respective eye. A gaze vector for each eye may be determined based on the 3D spatial relationship. Based on the gaze vectors of both eyes or on a map of the gaze vectors from at least one eye and an estimate of the environment, a gaze point may be determined that indicates one or more objects at which the user is gazing, or more generally expressed as real or virtual at which the user is looking.

Other embodiments use both image data of the eye and data representing glints in the context of the geometry of the illuminators and the at least one image sensor to determine the boundaries of a three-dimensional (3D) spatial relationship between the positions of the portions of the eye and the corresponding system of gaze-detection elements. Examples of these portions of the eye are a center of the cornea determined based on the glint data, a center of the pupil determined from the image data of the eye, and a center of rotation of the eye, the position of the center of rotation of the eye being estimated based on the position of the center of the cornea. For accuracy considerations in gaze determination purposes, in one embodiment, the center of rotation of the eyeball may be considered fixed. However, in one embodiment, the center of rotation of the eyeball is not considered to be fixed. Note that in one embodiment, the center of rotation of the eyeball is not considered fixed if the HMD is repositioned on the face. The gaze vector of the respective eye is determined based on a corneal center, a pupil center, and a center of rotation forming an optical axis of the respective eye. An angular offset may be applied to the optical axis to obtain the visual axis of the eye that may be selected as the gaze vector.

Different gaze detection techniques may be used within the same system. For example, due to obstructions of the eye or update processing time, less computationally intensive techniques (like some version of the method based on correlating glint intensity values to pupil positions) may be used more frequently in conjunction with more computationally intensive techniques (like some version of the 3D spatial relationship between the cornea center, pupil center, center of rotation, and optical element based gaze detection system to determine the gaze vector) with longer time intervals in between. For example, in embodiments using a training gaze data set, changes in the spatial relationship, including changes in depth between the eyes and gaze detection elements, may also be determined as an indication to trigger recalibration of the system.

In some embodiments, the see-through display device is in a pair of glasses, but other Head Mounted Display (HMD) formats and near-eye display stands suitable for daily use by consumers may also be used.

Fig. 1A is a block diagram depicting example components of one embodiment of a mixed reality display system that controls display brightness based on gaze determination and/or pupil size. For example, opacity and/or image brightness may be adjusted. System 10 includes a see-through display device that is a near-eye, head-mounted display device 2 in communication with a processing unit 4 via line 6.

In other embodiments, head mounted display device 2 communicates with processing unit 4 through wireless communication. The processing unit 4 may take various embodiments. In some embodiments, processing unit 4 is a separate unit that may be worn on the user's body (e.g., the wrist in the illustrated example) or placed in a pocket, and includes most of the computing power for operating near-eye display device 2. Processing unit 4 may communicate wirelessly (e.g., WiFi, bluetooth, infrared, or other wireless communication means) with one or more hub computing systems 12. In other embodiments, the functionality of the processing unit 4 may be integrated in the software and hardware components of the display device 2.

The head mounted display device 2 (which in one embodiment is in the shape of eyeglasses with frames 115) is worn on the head of a user so that the user can see through the display (which in this example is implemented as display optics 14 for each eye) and thus have an actual direct view of the space in front of the user. The frame 115 provides a support for holding the elements of the system in place and a conduit for electrical connections. In this embodiment, the frame 115 provides a convenient frame for the glasses as a support for the elements of the system discussed further below. In other embodiments, other support structures may be used. Examples of such structures are a visor or goggles. The frame 115 includes a temple or side arm for resting on each ear of the user. Temple 102 is representative of an embodiment of a right temple. The nose bridge 104 of the frame comprises a microphone 110 for recording sound and transmitting audio data to the processing unit 4.

In some embodiments, the opacity of the see-through display is adjustable to control the amount of external light allowed to pass to the user's eyes. Controlling opacity may help to help the image on the display look more realistic. For example, it may help make virtual objects appear more realistically integrated with real objects. Controlling opacity may also protect the user's eyes from injury or discomfort.

Hub computing system 12 may be a computer, a gaming system or console, or the like. According to an example embodiment, hub computing system 12 may include hardware components and/or software components such that hub computing system 12 may be used to execute applications such as gaming applications, non-gaming applications, and the like. In one embodiment, hub computing system 12 may include a processor, such as a standardized processor, a specialized processor, a microprocessor, or the like that may execute instructions stored on a processor readable storage device for performing the processes described herein.

Hub computing system 12 also includes one or more capture devices, such as capture devices 20A and 20B. In other embodiments, more or less than two capture devices may be used to capture a room or other physical environment of the user. The capture devices 20A and 20B may be, for example, cameras that visually monitor one or more users and the surrounding space so that gestures and/or movements performed by the one or more users and the structure of the surrounding space may be captured, analyzed, and tracked to perform one or more controls or actions in an application and/or animate an avatar or on-screen character. The application may execute on hub computing system 12, on display device 2, on mobile device 5 as described below, or on a combination of these devices.

The hub computing environment 12 may be connected to an audiovisual device 16 such as a television, a monitor, a high-definition television (HDTV), or the like that may provide game or application visuals. In some cases, the audiovisual device 16 may be a three-dimensional display device. For example, hub computing system 12 may include a video adapter such as a graphics card and/or an audio adapter such as a sound card that may provide audiovisual signals associated with the game application, non-game application, or the like. The audiovisual device 16 may receive the audiovisual signals from the hub computing system 12 and may then output game or application visuals and/or audio associated with the audiovisual signals. According to one embodiment, the audiovisual device 16 may be connected to the hub computing system 12 via, for example, an S-video cable, a coaxial cable, an HDMI cable, a DVI cable, a VGA cable, a component video cable, an RCA cable, or the like. In one example, the audiovisual device 16 includes a built-in speaker. In other embodiments, the audiovisual device 16, a separate stereo system, or the hub computing system 12 is connected to the external speakers 22.

Fig. 1B is a block diagram depicting example components of another embodiment of a mixed reality display system that adjusts display brightness based on gaze determination and/or pupil size. In this embodiment, near-eye display device 2 communicates with a mobile computing device 5, which is an example embodiment of processing unit 4. In the example shown, the mobile device 5 communicates via a line 6, but in other examples the communication may also be wireless.

Further, as in hub computing system 12, gaming and non-gaming applications may execute on the processor of mobile device 5, where user motion control or user motion animates an avatar that may be displayed on display 7 of device 5. The mobile device 5 also provides a network interface for communicating with other computing devices like hub computing system 12 over the internet or another communication network via a wired or wireless communication medium using a wired or wireless communication protocol. A remote network accessible computing system like hub computing system 12 may be leveraged by processing unit 4 like mobile device 5 to provide processing power and remote data access. Examples of hardware and software components of the mobile device 5 (such as may be implemented in a smartphone or tablet computing device) are described in fig. 17, and these components may include hardware and software components of the processing unit 4, such as those discussed in the embodiment of fig. 4A. Some other examples of mobile devices 5 are laptop or notebook computers and netbook computers.

As described above, in some embodiments, gaze detection for each of the user's eyes is based on a three-dimensional coordinate system of gaze detection elements on a near-eye, mixed reality display device like glasses 2, in relation to one or more human eye elements such as corneal center, eyeball center of rotation, and pupil center. Examples of gaze detection elements that may be part of the coordinate system include an illuminator that generates glints and at least one sensor for capturing data representative of the generated glints. As discussed in the embodiment of fig. 7, the corneal center may be determined using planar geometry based on two glints. The corneal center links the pupil center and the center of rotation of the eyeball, which can be considered as a fixed position for determining the optical axis of the user's eye at a particular gaze or viewing angle.

Fig. 1C illustrates an exemplary arrangement of locations of respective sets of gaze-detection elements in a see-through, near-eye, mixed reality display system implemented in a pair of glasses 2. The system also has one or more cameras that can determine pupil size. Furthermore, the system is able to track each eye in 3D.

Represented as lenses for each eye are display optics 14, e.g., 14r and 14l, for each eye. The display optical system includes see-through lenses, such as 118 and 116 in fig. 3A-3D, as a pair of ordinary eyeglasses, but also contains optical elements (e.g., mirrors, filters) for seamlessly blending virtual content with the actual and direct real world views seen through the lenses 118, 116. The display optics 14 have an optical axis generally centered on the see-through optics 118, 116, where the light is generally collimated to provide an undistorted view. For example, when an eye care professional fits a pair of ordinary eyeglasses to the user's face, the goal is for the eyeglasses to fall on the user's nose at a location where each pupil is aligned with the center or optical axis of the corresponding lens, typically so that the collimated light reaches the user's eyes for a clear or undistorted view.

In the example of fig. 1C, the detection regions 139r, 139l of at least one sensor are aligned with the optical axes of their respective display optics 14r, 14l such that the centers of the detection regions 139r, 139l capture light along the optical axes. If the display optical system 14 is aligned with the user's pupil, each detection region 139 of the respective sensor 134 is aligned with the user's pupil. The reflected light of the detection region 139 is transmitted via one or more optical elements to the actual image sensor 134 of the camera, the sensor 134 being shown by a dashed line inside the frame 115 in this example.

In one example, a visible light camera, also commonly referred to as an RGB camera, may be the sensor, and an example of an optical element or light directing element is a partially transmissive and partially reflective visible light reflector. The visible light camera provides image data of the pupil of the user's eye, while the IR photodetector 152 captures glints, which are reflections in the IR portion of the spectrum. If a visible light camera is used, reflections of the virtual image may appear in the eye data captured by the camera. Image filtering techniques can be used to remove virtual image reflections if desired. The IR camera is insensitive to virtual image reflections on the eye.

In one embodiment, the at least one sensor 134 is an IR camera or a Position Sensitive Detector (PSD) to which IR radiation may be directed. For example, the heat reflective surface may transmit visible light but reflect IR radiation. The IR radiation reflected from the eye may be from incident radiation of the illuminator 153, other IR illuminators (not shown) or from ambient IR radiation reflected from the eye. In some examples, the sensor 134 may be a combination of RGB and IR cameras, and the optical directing elements may include visible light reflecting or turning elements and IR radiation reflecting or turning elements. In some examples, the camera may be small, e.g., 2 millimeters (mm) by 2mm or 3mm by 3 mm. An example of such a camera sensor is omnivision ov 7727. In other examples, the camera may be small enough (e.g., omnivision ov 7727), for example, so that the image sensor or camera 134 can be centered on the optical axis or other location of the display optical system 14. For example, the camera 134 may be embedded in a lens of the system 14. Additionally, image filtering techniques may be applied to blend the cameras into the user's field of view to mitigate any interference to the user.

In the example of fig. 1C, there are four sets of illuminators 153, the illuminators 153 being paired with photodetectors 152 and separated by a barrier 154 to avoid interference between incident light generated by the illuminators 153 and reflected light received at the photodetectors 152. To avoid unnecessary clutter in the drawings, reference numbers are shown for a representative pair. Each illuminator may be an Infrared (IR) illuminator that generates a narrow beam of light of approximately a predetermined wavelength. Each of the photodetectors may be selected to capture light at about the predetermined wavelength. Infrared may also include near infrared. Because the illuminator or photodetector may have a wavelength drift or a small range around the wavelength is acceptable, the illuminator and photodetector may have a tolerance range related to the wavelength used for generation or detection. In embodiments where the sensor is an IR camera or an IR Position Sensitive Detector (PSD), the photodetector may be an additional data capture device and may also be used to monitor the operation of the luminaire, such as wavelength drift, beam width change, etc. The photodetector may also provide flash data with a visible light camera as the sensor 134.

As noted above, in some embodiments where the corneal center is calculated as part of determining the gaze vector, two glints (and thus two illuminators) will be sufficient. However, other embodiments may use additional glints when determining the pupil location and thus the gaze vector. Because the eye data representing glints is captured repeatedly, for example at a frame rate of 30 frames per second or greater, the data for one glint may be obscured by the eyelids or even by the eyelashes, but the data may be collected by the glints generated by another illuminator.

Fig. 1D illustrates a respective set of gaze-detection elements and a position of one or more cameras for detecting pupil size implemented in a mixed reality display device in a pair of glasses. In this embodiment, two sets of illuminator 153 and photodetector 152 pairs are located near the top of each frame portion 115 around the display optics 14, and another two sets of illuminator and photodetector pairs are located near the bottom of each frame portion 115 to illustrate another example of the geometric relationship between the illuminators and thus the flashes they generate. This arrangement of the flashes may provide more information about the pupil position in the vertical direction.

Fig. 1E illustrates a further exemplary arrangement of respective sets of gaze detection elements and positions of a camera system for detecting pupil size. In this example, the sensors 134r, 134l are in line or aligned with the optical axes of their respective display optics 14r, 14l, but are located below the system 14 on the frame 115. Additionally, in some embodiments, the camera 134 may be a depth camera or include a depth sensor. A depth camera may be used to track the eye in 3D. In this example, there are two sets of illuminators 153 and photodetectors 152.

Fig. 1F is a block diagram of one embodiment of a system for adjusting brightness based on gaze estimation and pupil size. The system includes one or more camera systems 252. Any of the camera systems, image sensors, photodetectors, etc., described herein may be used in the camera system 252, such as (but not limited to) the capture devices 20A and 20B, the image sensor 134, the detection area 139, the illuminator 153, the photodetector 152. Camera system 252 provides data for brightness control logic 258, gaze estimation logic 254, and pupil size determination logic 256.

Brightness control logic 258 controls the brightness of the see-through display. Brightness control logic 258 has opacity control logic 260 for controlling the opacity of the see-through display and image brightness control logic 262 for controlling the brightness of the image presented on the see-through display. In one embodiment, brightness control logic 258 outputs control signals for controlling opacity and image brightness.

In one embodiment, the gaze estimation logic 254 determines a gaze vector. Pupil size determination logic 256 determines the pupil size. The gaze vector and pupil size are provided to brightness control 258.

In one embodiment, brightness control 258 inputs a user baseline value, which is a parameter related to the eyes of a particular user. These user baseline values may be used to determine how to control display brightness. Further details are described below.

The system may be implemented by any combination of hardware and/or software. In one embodiment, brightness control 258, gaze estimation logic 254, and pupil size determination logic 256 are implemented by a processor. As an example, the devices 4, 5 may have a processor. Note that the system may have other elements, but they are not shown in order not to obscure the illustration.

Fig. 1G is a flow diagram of one embodiment of a process to adjust the see-through display brightness based on gaze estimation and optionally pupil size. This process may be used to adjust the opacity and/or image brightness of the see-through display. Controlling opacity controls how much external light passes through the see-through display. Controlling the image brightness controls how brightly the see-through display presents the image. Note that controlling the image brightness can help save power and extend the life of the see-through display.

Controlling opacity and image brightness has many advantages. It may allow, among other things, virtual objects to more realistically integrate with real objects, protect the user's eyes from damage and discomfort, save power, and extend the useful life of the display.

At step 402, an estimate is made of the region that the wearer of the see-through, near-eye display is looking at. In one embodiment, the estimation is made using an eye tracking camera system. In one embodiment, a gaze vector is determined for each eye. These gaze vectors may be established in a 3D coordinate system around the user. 1C-1G illustrate several embodiments that may be used to track eye gaze. Further details and embodiments of eye tracking to estimate gaze are discussed below.

In optional step 404, the pupil size of the wearer is determined. The example see-through displays of FIGS. 1C-1G may be used. Further details and embodiments of determining pupil size are discussed below.

At step 406, the light intensity of the area (outside the HMD) at which the wearer is gazing is determined. In one embodiment, a camera system is used to determine the light intensity of the wearer's environment. The camera system may be located on the see-through display or elsewhere. The camera image may be correlated with the eye gaze location to determine the light intensity of the area (e.g., real world object) at which the wearer is gazing. In one embodiment, a 3D map of the environment is made in step 406. The gaze vector may be used to determine a point or region in the 3D map. Thus, the light intensity outside the HMD at which the wearer is looking can be determined.

Note that the 3D environment may have objects with a large light intensity range. For example, the wearer may be looking at a black table or a bright wall in the same room. Note that the average light intensity of a room does not convey these large differences in light intensity.

In one embodiment, the brightness of the see-through display is adjusted based on the light intensity of the area being gazed at (step 408). In one embodiment, the brightness of the see-through display is adjusted based on the light intensity of the region and the pupil size (step 410).

As described, adjusting the brightness may include adjusting the opacity and the image brightness. For example, if a user looks at a bright wall to a black table, the opacity may be adjusted to allow the virtual image to be realistically displayed. If the user's pupil size changes when the user's gaze shifts, the image brightness of the display may be adjusted accordingly. However, when changing the image brightness, appropriate changes can be made to the opacity to maintain a realistic rendering of the virtual image. Thus, there may be an interplay between image brightness and opacity.

Fig. 2 is a side view of temple 102 of frame 115 in an embodiment of a see-through, mixed reality display device implemented as eyeglasses that provide support for hardware and software components. In front of the frame 115 is a video camera 113 facing the physical environment that can capture video and still images. The video camera 113 may be used to determine the light intensity of the field of view of a see-through, mixed reality display device. In some embodiments where display device 2 does not operate in conjunction with depth cameras such as capture devices 20a and 20b of hub system 12, camera 113 facing the physical environment may be a depth camera and a camera sensitive to visible light. For example, the depth camera may include an IR illuminator emitter and a heat reflective surface, such as a hot mirror in front of a visible image sensor, that transmits visible light and directs reflected IR radiation within a wavelength range emitted by the illuminator or around a predetermined wavelength to a CCD or other type of depth sensor. Data from the sensors may be sent to the processor 210 of the control circuit 13, or the processing units 4, 5, or both, which may process the data, but in some embodiments the units 4, 5 may also send the data to a hub computing system 12 like FIG. 1A or over a network to one or more computer systems (e.g., like the hub computing system 12) for processing. The process identifies and maps the user's real world field of view. In addition, the camera 113 facing the physical environment may further include an exposure meter for measuring ambient light. In certain embodiments, discussed further below, a certain amount of change may trigger a message regarding the recalibration of the training gaze data set. However, note that embodiments that control the brightness of the display based on the light intensity at which the user is looking may use other brightness data than that collected by the exposure meter to determine the light intensity.

Control circuitry 136 provides various electronics that support other components of head mounted display device 2. More details of the control circuit 136 are provided below with reference to fig. 4A. The earpiece 130, inertial sensor 132, GPS transceiver 144, and temperature sensor 138 are internal to the temple 102 or mounted on the temple 102. In one embodiment, inertial sensors 132 include a three axis magnetometer 132A, three axis gyroscope 132B, and three axis accelerometer 132C (see FIG. 4A). Inertial sensors are used to sense the position, orientation, and sudden acceleration of head mounted display device 2. From these movements, the head position can also be determined.

The display device 2 provides an image generation unit that can create one or more images including one or more virtual objects. In some embodiments, a microdisplay may be used as the image generation unit. The microdisplay assembly 173 includes light processing elements and a variable focus adjuster 135. An example of a light processing element is a micro display unit 120. Other examples include one or more optical elements, such as one or more lenses of lens system 122 and one or more reflective elements, such as 124a and 124B in fig. 3A and 3B or 124 in fig. 3C and 3D. Lens system 122 may include a single lens or multiple lenses.

A microdisplay unit 120 is mounted on or within the temple 102, which includes an image source and generates an image of a virtual object. The microdisplay unit 120 is optically aligned with the lens system 122 and the reflecting surface 124 or reflecting surfaces 124a and 124b shown in the following figures. The optical alignment may be along an optical axis 133 or an optical path 133 that includes one or more optical axes. The microdisplay unit 120 projects an image of the virtual object through the lens system 122, which can direct image light onto the reflective element 124, the reflective element 124 directs the light into the light guide optical element 112 in fig. 3C and 3D or onto a reflective surface 124a (e.g., a mirror or other surface), the reflective surface 124a directs the light of the virtual image to a partially reflective element 124b, and the partially reflective element 124b combines a virtual image view along path 133 with a natural or actual direct view along optical axis 142 in fig. 3A-3D. The combination of views is directed to the user's eyes.

Variable focus adjuster 135 changes the displacement between one or more light processing elements in the optical path of the microdisplay assembly or the optical power (optical power) of elements in the microdisplay assembly. The optical power of a lens is defined as the inverse of its focal length, e.g. 1/focal length, so that a change in one affects the other. The change in focal length causes a change in a region of the field of view, such as a region at a particular distance, that is in focus for the image generated by the microdisplay assembly 173.

In one example of a displacement change made by microdisplay assembly 173, the displacement change is guided within an armature 137, which armature 137 supports at least one light processing element such as lens system 122 and microdisplay 120 in this example. The armature 137 helps stabilize the alignment along the optical path 133 during physical movement of the components to achieve a selected displacement or optical power. In some examples, the adjuster 135 may move one or more optical elements, such as a lens in the lens system 122 within the armature 137. In other examples, the armature may have a slot or space in the region around the light processing element so that it slides over the element (e.g., microdisplay 120) without moving the light processing element. Another element in the armature, such as lens system 122, is attached so that system 122 or the lens therein slides or moves with moving armature 137. The displacement range is typically on the order of a few millimeters (mm). In other examples, the armature 137 may provide support for a focus adjustment technique involving adjustment of other physical parameters besides displacement to the lens system 122. An example of such a parameter is polarization.

In one example, the adjuster 135 may be an actuator such as a piezoelectric motor. Other techniques for actuators may also be used, and some examples of such techniques are voice coils formed from coils and permanent magnets, magnetostrictive elements, and electrostrictive elements.

There are different image generation techniques that can be used to implement microdisplay 120. For example, microdisplay 120 can be implemented using a transmissive projection technology where the light source is modulated by an optically active material, backlit with white light. These techniques are typically implemented using LCD-type displays with powerful backlights and high optical power densities. Microdisplay 120 can also be implemented using a reflective technology where external light is reflected and modulated by an optically active material. Depending on the technology, the illumination is forward lit by a white light source or an RGB source. Digital Light Processing (DLP), Liquid Crystal On Silicon (LCOS), and from Qualcomm, IncThe display techniques are all examples of efficient reflection techniques, as most of the energy is reflected from the modulated structure and can be used in the systems described herein. Additionally, microdisplay 120 can be implemented using an emissive technology, where light is generated by the display. For example, PicoPTM engines from Microvision limited use micro mirror steers to emit laser signals onto a small screen acting as a transmissive element or to emit a beam of light (e.g., laser light) directly to the eye.

As described above, the configuration of the light processing elements of the microdisplay assembly 173 creates a focal distance or focal region where virtual objects appear in the image. Changing the configuration changes the focal region of the virtual object image. The focal region determined by the light processing element may be determined and changed based on equation 1/S1+1/S2= 1/f.

Symbol f denotes the focal length of a lens, such as the lens system 122 in the microdisplay assembly 173. Lens system 122 has a front nodal point and a rear nodal point. If a ray is directed forward to either of the two nodes at a given angle to the optical axis, the ray will exit the other node at an equal angle to the optical axis. In one example, the rear node of lens system 122 would be between itself and microdisplay 120. The distance from the rear node to microdisplay 120 can be represented as S2. The front nodal point is typically within a few millimeters of the lens system 122. The target position is the position in three-dimensional physical space of the virtual image to be generated by microdisplay 120. The distance from the front node to the target position of the virtual image may be represented as S1. The sign convention shows that S1 has a negative value because the image is a virtual image appearing on the same side of the lens as microdisplay 120.

If the focal length of the lens is fixed, S1 and S2 are changed to focus the virtual object at a different depth. For example, the initial position may have S1 set to infinity and S2 equal to the focal length of lens system 122. Assuming that the lens system 122 has a focal length of 10mm, consider an example in which the virtual object is to be placed approximately 1 foot (i.e., 300 mm) in the user's field of view. S1 is now approximately-300 mm, f is 10mm and S2 is currently set at the initial position of the focal length of 10mm, meaning that the rear node of the lens system 122 is 10mm from the microdisplay 122. A new distance or new displacement between lens 122 and microdisplay 120 is determined based on 1/(-300) +1/S2=1/10 (all items are in millimeters). The result was that S2 was approximately 9.67 mm.

In one example, one or more processors (e.g., in the control circuitry, processing units 4, 5, or both) may calculate displacement values of S1 and S2, while causing the focal length f to be fixed and causing the control circuitry 136 to cause the variable adjuster driver 237 (see fig. 4A) to send drive signals to cause the variable virtual focus adjuster 135 to move the lens system 122, for example, along the optical path 133. In other embodiments, the microdisplay unit 120 may be moved instead of or in addition to moving the lens system 122. In other embodiments, the focal length of at least one lens in lens system 122 may also be changed, either in addition to or instead of the change in displacement along optical path 133.

Fig. 3A is a top view of an embodiment of display optical system 14 of a see-through, near-eye, mixed reality device that includes an arrangement of gaze-detection elements in a gaze-detection system. As shown, a portion of the frame 115 of the near-eye display device 2 will be around the display optics including support for one or more lenses. To illustrate the various components of display system 14 (in this case right eye system 14 r), the top portion of frame 115 around the display optics is not depicted.

In this embodiment, display optical system 14 has an optical axis 142 and includes see-through optics 118 that allow a user's actual direct view of the real world. In this example, see-through lens 118 is a standard lens used in eyeglasses and may be made according to any prescription (including not according to the prescription). In another embodiment, the see-through lens 118 is replaced with a variable prescription lens. In some embodiments, see-through, near-eye display device 2 will include additional lenses.

Display optical system 14 also includes reflective surfaces 124a and 124 b. In this embodiment, light from microdisplay 120 is directed along optical path 133 via reflective element 124a to partially reflective element 124b embedded in mirror 118, and partially reflective element 124b combines the virtual object image view traveling along optical path 133 with a natural or actual direct view along optical axis 142, so that the combined view is directed to the user's eye (the right eye in this example) at a location on the optical axis that has the most collimated light for the clearest view.

The detection area 139r of the light sensor is also part of the display optical system 14 r. The optical element 125 implements the detection region 139r by capturing reflected light from the user's eye received along the optical axis 142 and directs the captured light to the sensor 134r, which in this example is located in the bridge of the nose 104. As shown, this arrangement allows the detection region 139 of the sensor 134r to have its center aligned with the center of the display optical system 14. For example, if the sensor 134r is an image sensor, the sensor 134r captures a detection region 139 such that an image captured at the image sensor is centered on the optical axis because the center of the detection region 139 is the optical axis. In one example, the sensor 134r is a visible light camera or a combination of RGB/IR cameras, and the optical element 125 includes an optical element that reflects visible light reflected from the user's eye, such as a partially reflective mirror surface.

In other embodiments, the sensor 134r is an IR sensitive device such as an IR camera, and the element 125 includes a heat reflective surface that passes visible light through it and reflects IR radiation to the sensor 134 r. The IR camera may capture not only glints, but also infrared or near-infrared images of the user's eyes, including the pupils.

In other embodiments, the IR sensor device 134r is a Position Sensitive Device (PSD), sometimes referred to as an optical position sensor. The location of the detected light on the surface of the sensor is identified. The PSD may be selected to be sensitive to the wavelength range of the flashing IR illuminator or about a predetermined wavelength. When light in the wavelength range of, or about, its predetermined wavelength of the position sensitive device is detected on a sensor or photosensitive component of the device, an electronic signal is generated that identifies the position on the surface of the detector. In some embodiments, the surface of the PSD is divided into discrete sensors (like pixels) from which the position of the light can be determined. In other examples, a PSD isotropic sensor may be used, where changes in local impedance on the surface may be used to identify the location of the light spot on the PSD. Other embodiments of PSDs may also be used. By operating the illuminator 153 in a predetermined sequence, the glint reflection locations on the PSD may be identified and thus correlated with locations on the corneal surface.

The light detecting elements (in this case, reflective elements) 125, 124a, and 124b depicted in fig. 3A-3D are representative of their function. These elements may take any number of forms and may be implemented with one or more optical components in one or more arrangements for directing light to its intended destination (e.g., a camera sensor or a user's eye).

In this embodiment, the display optical system 14 includes other gaze detection elements. In this embodiment, at least two (2) but possibly more Infrared (IR) illumination devices 153 are attached to the frame 115 and to the side of the lens 118 that direct a narrow beam of infrared light within a particular wavelength range or about a predetermined wavelength to the user's eye to each generate a respective glint on the surface of a respective cornea. In other embodiments, the illuminator and any photodiodes may be on the lens, e.g., on the corners or on the sides. In this embodiment, there is an Infrared (IR) photodetector 152 in addition to at least 2 IR illumination devices 153. Each photodetector 152 is sensitive to IR radiation within a particular wavelength range of its corresponding IR illuminator 153 through the lens 118 and is positioned to detect a respective flash of light. As shown in fig. 1C-1E, the illuminator and the photodetector are separated by a barrier 154 so that incident IR light from the illuminator 153 does not interfere with reflected IR light received at the photodetector 152. In the case where the sensor 134 is an IR sensor, the photodetector 152 may not be needed or may be an additional flash data capture source. Using a visible light camera, the photodetector 152 captures light from the flash and generates a flash intensity value.

Fig. 3B is a top view of another embodiment of display optical system 14 of a see-through, near-eye, mixed reality device that includes an arrangement of gaze-detection elements in a gaze-detection system. In this embodiment, there is an Infrared (IR) photodetector 152 in addition to at least 2 IR illumination devices 153. In this embodiment, the heat reflective surface 125 has been removed to illustrate operation without a position sensitive detector.

In the fig. 3B embodiment, photodetector 134r may be implemented as a visible light camera (sometimes referred to as an RGB camera), or it may be implemented as an IR camera or a camera capable of processing light in the visible and IR ranges, such as a depth camera. In this example, the image sensor 134r is the detection region 139r, and the image sensor 134 of the camera is vertically located on the optical axis 142 of the display optical system. In some examples, the camera may be located above or below the see-through lens 118 on the frame 115, or embedded in the lens 118. In some embodiments, the illuminator 153 provides light to the camera, while in other embodiments the camera captures images using ambient light or light from its own light source.

In one embodiment, glint reflections may estimate gaze based on several intensity value data points detected for glints, rather than processing much larger eye image datasets. The position of the illuminator 153 on the frame 115 or other support structure of the near-eye display device may be fixed such that the location of the flash light detected by the one or more sensors is fixed in the sensor detection area. The cornea, and thus the iris and pupil, rotate with the eyeball about a center (which may be treated as stationary, but this is not required). As the user's gaze changes, the iris, pupil, and sclera (sometimes referred to as the white portion of the eye) move under the glints. Thus, glints detected at the same sensor location may result in different intensity values due to different reflectivities associated with different eye regions. Since the pupil is an aperture with tissue that absorbs most of the incident light, its intensity value will be very low or near zero, while the intensity value of the iris will be a higher intensity value due to its higher reflectivity. Since the sclera has the highest reflectivity, the intensity value of the sclera may be the highest. In some examples, the illuminators may be located on either side of the display optical system 14 as in fig. 3A-3D, and thus on either side of the pupil of the user's eye. In other embodiments, additional illuminators may be located on the frame 115 or lenses 118, for example, four illuminators may be placed to generate a surrounding geometric shape (e.g., a box) of glints on the eye that would be approximately centered on the pupil when the user looks straight ahead. The microdisplay assembly 173 can display a virtual image or send a message, such as a visual virtual image or audio instructions, to the user so that the user looks straight ahead to initiate a flash on or near the pupil. In other embodiments, glint-based gaze detection is based on intensity values generated from luminaires where glints are located independently of being centered on the pupil.

Fig. 3C is a top view of a third embodiment of display optical system 14 of a see-through, near-eye, mixed reality device that includes an arrangement of gaze-detection elements in a gaze-detection system. The display includes a light guide optical element 112 between an additional see-through lens 116 and a see-through lens 118. The light guide optical element 112 guides the artificial light to the eye.

Light guide optical element 112 transmits light from microdisplay 120 to the eye of a user wearing head mounted display device 2. The light guide optical element 112 also allows light from the front of the head mounted display device 2 to be transmitted through the light guide optical element 112 to the user's eye, allowing the user to have an actual direct view of the space in front of the head mounted display device 2 in addition to receiving a virtual image from the microdisplay 120. Thus, the walls of the light guide optical element 112 are see-through. The light guide optical element 112 includes a first reflective surface 124 (e.g., a mirror or other surface). Light from microdisplay 120 passes through lens 122 and is incident on reflecting surface 124. Reflective surface 124 reflects incident light from microdisplay 120 such that light is trapped by internal reflection within the planar substrate comprising light guide optical element 112.

After a number of reflections at the surface of the substrate, the captured light waves reach the array of selectively reflective surfaces 126. Note that only one of the five surfaces is labeled 126 to prevent the drawings from being too crowded. The reflective surfaces 126 couple light waves exiting the substrate and incident on these reflective surfaces to the user's eye. More details of light-guiding optical elements can be found in U.S. patent application publication No. 2008/0285140, serial No. 12/214,366 "Substrate-guided optical devices," published 11/20 of 2008, which is incorporated by reference in its entirety.

In this embodiment, as in fig. 1E and one of the examples of fig. 3B, the display optical system 14 is similarly arranged with the IR illuminator 153 and photodetector 152, and the visible or IR camera 134r located on the frame 115 or lens 118 below or above the optical axis 142 (typically centered on the lenses 116 and 118 that support the light guide optical element 112).

Fig. 3D is a top view of a fourth embodiment of display optical system 14 of a see-through, near-eye, mixed reality device that includes an arrangement of gaze-detection elements in a gaze-detection system. This embodiment is similar to the embodiment of fig. 3C, including a light guide optical element 112. However, the light detector has only an IR photodetector 152, so this embodiment relies only on glint detection for gaze detection, as discussed in the examples below.

In the embodiment of fig. 3A-3D, the positions of gaze detection elements, such as the detection region 139 and the illuminator 153 and photodetector 152, are fixed relative to each other. In these examples, they are also fixed relative to the optical axis of the display optical system 14.

In the above embodiments, the specific number of lenses shown is merely an example. Other numbers and configurations of lenses operating according to the same principles may be used. Additionally, in the above example, only the right side of the see-through, near-eye display 2 is shown. As an example, an all-near-eye, mixed reality display device would include another set of lenses 116 and/or 118, another light guide optical element 112 for the embodiments of fig. 3C and 3D, another microdisplay 120, another lens system 122, possibly another environment-facing camera 113, another eye-tracking camera 134 for the embodiments of fig. 3A-3C, earphones 130, and temperature sensor 138.

Fig. 4A is a block diagram of one embodiment of hardware and software components of a see-through, near-eye, mixed reality display unit that may be used with the various embodiments described in this disclosure. Fig. 4B is a block diagram describing the components of the processing unit 4. In this embodiment, near-eye display device 2 receives instructions regarding the virtual image from processing unit 4 and provides sensor information to processing unit 4. The processing unit 4, whose components are depicted in fig. 4B, will receive sensor information from the display device 2 and may also receive sensor information from the hub computing device 12 (see fig. 1A). Based on this information, the processing unit 4 will determine where and when to provide the virtual image to the user and send instructions to the control circuitry 136 of the display device 2 accordingly.

Note that some of the components of fig. 4A (e.g., camera 113 facing the physical environment, eye camera 134, variable virtual focus adjuster 135, photodetector interface 139, microdisplay 120, lighting device 153 (i.e., illuminator), headphones 130, temperature sensor 138) are shown shaded to indicate that each of these devices is at least two — at least one on the left and at least one on the right of head mounted display device 2. Fig. 4A shows the control circuit 200 in communication with the power management circuit 202. The control circuit 200 includes a processor 210, a memory controller 212 in communication with a memory 214 (e.g., D-RAM), a camera interface 216, a camera buffer 218, a display driver 220, a display formatter 222, a timing generator 226, a display output interface 228, and a display input interface 230. In one embodiment, all components of control circuit 220 communicate with each other via dedicated lines of one or more buses. In another embodiment, each component of the control circuit 200 is in communication with the processor 210.

The camera interface 216 provides an interface to both the physical environment facing cameras 113 and each eye camera 134 and stores the respective images received from the cameras 113, 134 in a camera buffer 218. Display driver 220 will drive microdisplay 120. Display formatter 222 may provide information about the virtual image displayed on microdisplay 120 to one or more processors of one or more computer systems (e.g., 4, 12, 210) performing the processing of the augmented reality system. A timing generator 226 is used to provide timing data to the system. The display output 228 is a buffer for providing images from the physical environment facing camera 113 and the eye camera 134 to the processing unit 4. Display input 230 is a buffer for receiving images, such as virtual images to be displayed on microdisplay 120. The display output 228 and the display input 230 communicate with a band interface 232 that is an interface to the processing unit 4.

Power management circuit 202 includes voltage regulator 234, eye tracking illumination driver 236, variable adjuster driver 237, photodetector interface 239, audio DAC and amplifier 238, microphone preamplifier and audio ADC240, temperature sensor interface 242, and clock generator 244. Voltage regulator 234 receives power from processing unit 4 through band interface 232 and provides the power to the other components of head mounted display device 2. The illumination driver 236 controls the illumination device 153 to operate at about a predetermined wavelength or within a certain wavelength range, for example, via a driving current or voltage. Audio DAC and amplifier 238 receives audio information from headphones 130. Microphone preamplifier and audio ADC240 provide an interface for microphone 110. Temperature sensor interface 242 is an interface for temperature sensor 138. Power management unit 202 also provides power to and receives data back from three axis magnetometer 132A, three axis gyroscope 132B, and three axis accelerometer 132C. The power management unit 202 also provides power to, receives data from, and transmits data to the GPS transceiver 144.

Variable adjuster driver 237 provides control signals, such as drive current or drive voltage, to adjuster 135 to move one or more elements of microdisplay assembly 173 to achieve a displacement of the focal region calculated by software executing in control circuit 13 or processing units 4, 5 or processor 210 of the hub computer 12 or both. In embodiments that sweep through a range of displacements, and thus a range of focal regions, the variable adjuster driver 237 receives timing signals from the timing generator 226, or alternatively from the clock generator 244, to operate at a programmed rate or frequency.

The photodetector interface 239 performs any analog-to-digital conversion required for the voltage or current reading from each photodetector, stores the reading in memory in a processor readable format via the memory controller 212, and monitors the operating parameters of the photodetectors 152, such as temperature and wavelength accuracy.

Figure 4B is a block diagram of one embodiment of the hardware and software components of the processing units 4, 5 associated with the see-through, near-eye display unit. The mobile device 5 may include this embodiment of hardware and software components as well as similar components that perform similar functions. Fig. 4B shows control circuitry 304 in communication with power management circuitry 306. Control circuitry 304 includes a Central Processing Unit (CPU) 320, a Graphics Processing Unit (GPU) 322, a cache 324, a RAM326, a memory controller 328 in communication with a memory 330 (e.g., D-RAM), a flash controller 332 in communication with a flash memory 334 (or other type of non-volatile storage), a display output buffer 336 in communication with see-through, near-eye display device 2 via band interface 302 and band interface 232, a display input buffer 338 in communication with near-eye display device 2 via band interface 302 and band interface 232, a microphone interface 340 in communication with an external microphone connector 342 for connecting to a microphone, a pci express interface for connecting to a wireless communication device 346, and a USB port 348.

In one embodiment, the wireless communication component 346 may include a Wi-Fi enabled communication device, a Bluetooth communication device, an infrared communication device, and the like. The USB port may be used to interface the processing units 4, 5 to the hub computing device 12 to load data or software onto the processing units 4, 5 and to charge the processing units 4, 5. In one embodiment, CPU320 and GPU322 are the main load devices used to determine where, when, and how to insert virtual images into a user's field of view.

Power management circuitry 306 includes a clock generator 360, an analog-to-digital converter 362, a battery charger 364, a voltage regulator 366, a see-through, near-eye display power supply 376, and a temperature sensor interface 372 (located on the wrist band of processing unit 4) that communicates with a temperature sensor 374. The AC-to-DC converter 362 is connected to a charging receptacle 370 to receive AC power and generate DC power for the system. The voltage regulator 366 communicates with a battery 368 for providing power to the system. The battery charger 364 is used to charge the battery 368 (via the voltage regulator 366) upon receiving power from the charging receptacle 370. The device power interface 376 provides power to the display device 2.

FIG. 5A is a flow diagram of one embodiment of determining the brightness of an area at which a wearer of a see-through display is gazing. The process is one embodiment of steps 402 and 406 from FIG. 1F. At step 420, the wearer's field of view is tracked. In one embodiment, step 420 includes generating a 3D map of the user's environment. In one embodiment, a 3D position of an object is determined. One embodiment of this is depicted in FIG. 6A. Note, however, that identifying objects is not required. In some embodiments, tracking the field of view determines only the luminance map. The luminance map for one embodiment is a 2D map of luminance values.

At step 422, a gaze vector is determined for at least one eye of the wearer. The gaze vector may be determined with a camera system such as that described in fig. 1C-1E. In one embodiment, two gaze vectors are determined. Points in 3D space may be determined based on the intersection of the gaze vectors. In some cases, the gaze vectors may be disjoint (or may intersect at infinity). Further details of determining the gaze vector are described below.

In step 424, the brightness of the area in the field of view where the wearer is looking is determined based on the tracking information from step 420 and the one or more gaze vectors. In one embodiment, a determination is made as to which object the wearer is looking at. The brightness of the object can then be used as a result. However, some objects may not have uniform brightness (or near uniform brightness). Thus, step 424 may determine a certain area or point on the object.

Note, however, that it is not required to determine what object the wearer is looking at. In one embodiment, one or more gaze vectors are used to determine points or regions of a luminance map (e.g., a 2D luminance map). Thus, the brightness of what the wearer is looking at can be determined.

Fig. 5B is a top view illustrating an example of gaze vectors that intersect at a gaze point at which a user's eyes are focused. The eye models 1601, 160r for each eye are shown based on the gullsland schematic eye model. For each eye, eyeball 160 is modeled as a sphere having a center of rotation 166, and includes a cornea 168 that is also modeled as a sphere and has a center 164. The cornea rotates with the eyeball. In one embodiment, the center of rotation 166 of the eyeball may be considered a fixed point. However, the center of rotation 166 of the eyeball need not be considered a fixed point. The cornea overlies the iris 170 with the pupil 162 centered on the iris 170. In this example, on the surface 172 of the respective cornea are glints 174 and 176.

An axis 178 formed from the center of rotation 166 through the corneal center 164 to the pupil 162 is the optical axis of the eye. Gaze vector 180 is sometimes referred to as the line of sight or visual axis extending from the fovea through pupil center 162. The fovea is a small area located at about 1.2 degrees in the retina. The angular offset between the optical axis and the visual axis calculated in the embodiment of fig. 9 has horizontal and vertical components. The horizontal component is up to 5 degrees from the optical axis and the vertical component is between 2 and 3 degrees. In many embodiments, an optical axis is determined and a small correction determined by user calibration is applied to obtain the visual axis selected as the gaze vector. For each user, a small virtual object may be displayed by the display device at each of a plurality of predetermined locations at different horizontal and vertical positions. During the display of the object at each position, the optical axis of each eye can be calculated and the ray modeled as extending from that position into the user's eye. The offset angle with horizontal and vertical components can be determined based on how the optical axis must be moved to align with the modeled ray. From different positions, the average offset angle with horizontal and vertical components can be selected as a small correction to be applied to each calculated optical axis. In some embodiments, only the horizontal component is used as the offset angle correction.

In the illustrated embodiment of FIG. 5B, the sensor detection area 139 is aligned with the optical axis of each display optical system 14 within the frame 115. In this example, the respective image sensors are cameras capable of capturing image data representing flashes 174l and 176l generated by illuminators 153a and 153b, respectively, on the left side of the frame 115 and data representing flashes 174r and 176r generated by illuminators 153c and 153d, respectively.

Through the display optics (14 l and 14r in frame 115), the user's field of view includes real objects 190, 192, and 194 and virtual objects 182, 184, and 186. In this example, the cornea 168l of the left eye is rotated to the right or toward the user's nose, and the cornea 168r of the right eye is rotated to the left or toward the user's nose. Both pupils are gazing at virtual object 186. The gaze vectors 180l and 180r from each eye center enter the Panum's fusional area 195 where the virtual object 186 is located. The Panum's fusion area is an area of single vision in a binocular viewing system like human vision. The intersection of gaze vectors 180l and 180r indicates that the user is looking at virtual object 186.

For a see-through, mixed reality display device, a gaze vector is determined to identify a gaze point in a three-dimensional user field of view that includes real objects that are not typically under computer control, as well as virtual objects generated by some application. The gaze vectors may intersect at an object 10 feet away, or at virtually infinite distance, or at any other distance. The following figures briefly discuss embodiments for determining a 3D user field of view.

Reference to forward facing image data refers to image data from one or more forward facing cameras, such as camera 113 in fig. 1A and 1B. In these embodiments, the field of view of the forward facing camera 113 approximates the field of view of the user when the camera is located at a relatively small offset from the optical axis 142 of each display optical system 14. The offset amount may be considered in the image data.

Figure 5C is a flow diagram of one embodiment of determining pupil size. The process is one embodiment of step 404 from FIG. 1F. At step 430, at least one eye of the wearer is tracked. In one embodiment, step 430 includes tracking the eye in 3D. The camera system of the example device of fig. 1C-1E may be used to track the eye in 3D.

At step 432, a 3D model of the eye is constructed. In one embodiment, a model such as that depicted in FIG. 5B is constructed. Fig. 5D and 5E show other models. However, still other models may be used.

At step 434, a pupil image is determined. Figure 5D illustrates one embodiment of a model for the cornea 172 and pupil image 462. Pupil image 462 is formed in imaging plane 163 of the camera system. The cornea is modeled in this example as a sphere having a center C and a radius Rc. Pupil 162 may be modeled as a circle having a center E and a radius r (not shown in fig. 5D), but may appear elliptical from an angle. The distance between the cornea center and the pupil center is h. In one embodiment, the corneal radius, pupil radius, and h are subject specific and determined for the wearer of the see-through display.

As the pupil passes through the corneal surface, the pupil may be refracted. This is depicted in fig. 5D by refractive pupil 467. Pupil image 462 is an image in the camera and may be a projection of refracted pupil 467. Thus, pupil image 462 may be an ellipse in camera imaging plane 463. Note that in 3D, the center of pupil image 462 need not be the center of the projection of pupil center E. Compensation can be made to adjust for this if desired.

In the embodiment of fig. 5D, an assumption is made that the center of pupil image 462 is the refraction of the fiber starting at pupil center E. Note, however, that refraction on non-planar surfaces (such as the cornea) is non-linear. Thus, in one embodiment, the estimated pupil center E' is determined to compensate for the non-linear refraction. FIG. 5E shows another embodiment of a model for cornea 172, with center E_iAssociated pupil image 462. The pupil 162 has an estimated center E'. The estimated center E' may be based on the known 3D location of the corneal center C, the corneal radii Rc and h through the center E from the pupil image_iBack tracking to determine. Note that the estimated center E' of the pupil does not have to be the same as the true pupil center E, because of the non-linear refraction.

At step 436, a pupil size is determined based on pupil image 462. Note that by tracking the eye in 3D, as well as the 3D pupil 162 within the cornea 172, embodiments are able to account for perspective imaging effects as the eye moves relative to the perspective display. Thus, the pupil profile can be accurately determined. In one embodiment, the pupil measurements performed in 3D are invariant to eye movement.

As mentioned above, certain parameters of the eye model may be user-specific. FIG. 5F is a flow diagram of one embodiment of determining user-specific parameters for eye tracking. These parameters may be used in determining pupil size. They may also be used in determining the eye gaze vector. Example parameters that may be determined include, but are not limited to, corneal radius r_cThe distance from the center of the cornea to the center of the pupil, the angle between the visual axis and the optical axis, and the pupil radius r_pFrom which statistical values, such as minimum, maximum and mean values, can be derived.

At step 452, the wearer is instructed to look at a series of points projected in front of them, one at a time. The system knows the 3D positions of these points. These may be determined based on generating a 3D model of the user's environment. One embodiment of which is depicted in fig. 6C.

The system then determines a gaze vector corresponding to each point using the current values of a set of one or more user-specific parameters in step 454. The initial value may be a default value based on an expected average value of the user.

In step 456, the system compares the gaze vector to the expected gaze vector. If the desired accuracy has been reached (step 457), the process ends. If not, the process continues to step 458.

In step 458, the system adjusts the user-specific parameters. Example parameters that may be determined include, but are not limited to, corneal radius r_cThe distance from the center of the cornea to the center of the pupil ("h" in fig. 5D and 5E), and the angle between the visual axis and the optical axis. The process then determines the gaze vector again in step 454. The process continues until the desired accuracy is achieved.

In one embodiment, a baseline pupillary enlargement response is determined for the wearer. This baseline may be used in determining how to adjust the see-through display. For example, some users may have pupils that change size quickly in response to changes in light intensity, while others are slower. Thus, the rate at which the brightness of the screen changes may be user specific.

Figure 5G is a flow diagram of one embodiment of a process for determining baseline user-specific parameters. At step 462, the image brightness of the see-through display is changed. In one embodiment, the image brightness is maintained at one intensity for a certain period of time, then changed to another level for a certain period of time, and so on. A light source other than a see-through display may be used in step 462. In one embodiment, step 462 determines or estimates the total amount of light at the user's eye. The surroundings may be taken into account.

At step 464, the pupil size is determined as a function of the light intensity. Step 464 may include determining a final pupil size for each light intensity from step 462.

At step 466, the rate of change of pupil size is determined. At step 466, the size of the user's pupil may be tracked over time, starting from the point in time when the light intensity is changed. This may be performed for one or more changes in light intensity. A value may be determined that represents how quickly the user's pupil reacts to changes in light intensity.

In one embodiment, the change in opacity and/or image brightness of the see-through display is performed gradually to allow the user's eyes to adjust to a new level. This may be based on user-specific baseline values, such as those determined in the process of fig. 5G. FIG. 5H is a flow diagram of one embodiment for gradually changing the display brightness based on a user baseline value. At step 472, the user specific baseline value is accessed. At step 474, a determination is made whether the display brightness should be changed. At step 476, a rate of change of the display brightness is determined based on the user-specific baseline. In one embodiment, step 476 is based on general parameters of the user.

In one embodiment, the distance between the wearer's eye and the see-through display is used to determine how to make adjustments to the opacity and/or image brightness of the see-through display. FIG. 5I is a flow diagram of one embodiment of varying the brightness of a display based on the distance of the user's eyes from the see-through display. At step 482, a distance between the user's eye and the see-through display is determined. In one embodiment, this is based on 3D image data for tracking the eye and determining the pupil size.

At step 484, the effect that the distance will have on the light intensity at the eye is determined. In other words, it is expected that the light intensity will decline as a function of distance from the display. Note that there may be two components of light intensity. One component is the image brightness. The other is how much external light passes through the see-through display. These two components may not be affected to the same extent due to the distance between the user's eyes and the see-through display.

At step 486, display brightness is adjusted based on the distance of the user's eyes from the see-through display.

In one embodiment, the display brightness is based on the context of what is being displayed. In one embodiment, certain items should be displayed brightly, such as warnings, priorities, etc. Items such as pictures or videos may be displayed at a different intensity than items such as text. FIG. 5J is a flow diagram of one embodiment of adjusting display brightness based on image context. At step 492, the context is determined. In one embodiment, the context is associated with the type or format of content to be presented on the see-through display. In one embodiment, the context is determined by analyzing the format of the image data to determine whether it is a picture, video, or text. In one embodiment, metadata associated with the image is analyzed to determine whether the image is associated with an alert or whether the image has a priority associated therewith. For example, the image may be associated with some electronic communication such as an email, text message, and the like.

In one embodiment, the context is associated with the user's environment. For example, if potentially dangerous or important objects are nearby, the user needs to be able to see. For example, if the user is walking, the user needs to see where they are going. If the car is in the vicinity, the user needs to know about this. In one embodiment, the analysis of the 3D image (and/or other data) is used to determine context in step 492.

At step 494, an appropriate display brightness is determined based on the context. Note that this may include adjusting image brightness and/or opacity. In step 496, the display brightness is adjusted based on step 494.

In one embodiment, a 3D position of an object is determined. One embodiment of this is depicted in FIG. 6A. Note, however, that identifying objects is not required. In some embodiments, tracking the field of view determines only the luminance map. The luminance map for one embodiment is a 2D map of luminance values.

FIG. 6A is a flow diagram of one embodiment for determining a three-dimensional user field of view. This process may be used in one embodiment to track the user's field of view (step 420 of FIG. 5A). At step 510, the one or more processors of control circuitry 136, processing units 4, 5, hub computing system 12, or a combination of these receive image data from one or more front facing cameras and at step 512 identify one or more real objects in the front facing image data. Data from orientation sensors 132 (e.g., three axis accelerometer 132C and three axis magnetometer 132A) may also be used with the front facing camera 113 image data to map the user's surroundings, the user's face and head positions to determine which objects (real or virtual) he or she may be focusing on at the time. Based on the executing application, at step 514, the one or more processors identify a location of the virtual object in a user field of view, where the user field of view may be determined to be a field of view captured in the forward facing image data. At step 516, the three-dimensional position of each object in the user field of view is determined. In other words, where each object is located with respect to the display apparatus 2, for example, with respect to the optical axis 142 of each display optical system 14.

FIG. 6B is a flow diagram of an embodiment of a method for identifying one or more real objects in a user field of view. This embodiment may be used to implement step 512. The implementation examples in fig. 6B, 6D, and 6E may be used separately or in conjunction with one another to identify the location of an object in the user's field of view. At step 520, the location of the user wearing display device 2 is identified. For example, the location of the user may be identified via GPS data of GPS unit 965 in mobile device 5 or GPS transceiver 144 on display device 2. At step 522, the one or more processors retrieve one or more images of the location from a database (e.g., 470) and use pattern recognition to select one or more images that match image data from the one or more front facing cameras at step 524. In certain embodiments, steps 522 and 524 may be performed remotely by a more powerful computer (e.g., the hub 12 having access to an image database). Based on the GPS data, the one or more processors determine the relative position of the one or more objects in the forward facing image data relative to the one or more GPS-tracked objects 528 in that position at step 526, and determine the position of the user from the one or more real objects based on the one or more relative positions at step 529.

In some embodiments, such as fig. 1A, a user wearing a see-through, near-eye display may be at a location where a computer system or one or more computers provide a three-dimensional mapping of objects within a space (e.g., a store). FIG. 6C is a flow diagram of an embodiment of a method for generating a three-dimensional model of a user space. At step 530, a computer system like hub system 12 with accessible depth cameras of capture devices 20A and 20B creates a three-dimensional model of the space based on the depth images. The depth images may be from multiple perspectives and may be combined based on a common coordinate space (e.g., a store space) and create a volume or three-dimensional description of that space. In step 532, an object is detected in the space. For example, edge detection may be performed on the depth images to distinguish objects (including people) from each other. At step 534, the computer system 12 identifies one or more detected objects, including their location in the space. Objects may also be identified based on a comparison of shapes and pattern recognition techniques, including facial recognition techniques using reference images of objects and people from an image database.

FIG. 6D is a flow diagram of an embodiment of a method for identifying one or more objects in a user field of view based on depth data transmitted to the see-through, mixed reality display device 2. At step 540, the processing units 4, 5 send the front-facing image data to a three-dimensional modeling system, such as may be implemented by a depth image processing application executing on a computer system, such as hub computing system 12 communicatively coupled to depth cameras 20A and 20B. Data from orientation sensor 132 may also be sent for identifying face or head position. For example, when a user enters a store, a computer system at the store provides a 3D mapping of the store and what and who are in the store. At step 542, display device 2 receives data identifying one or more objects in the user's field of view, as well as their locations in the 3D model of the space. The image data from the one or more front facing cameras 113 approximates the user field of view such that the hub system 12 identifies the object in the front facing image data, for example, by image recognition or pattern recognition software. Orientation data may also be used with the front-facing image data to refine the user field of view and to identify objects tracked by computer system 12 that fall within the user field of view. (hub system 12 also adjusts (align) the front facing image data as it is received from the two or more cameras 113 to identify the user field of view.) in step 544, the processing unit 4, 5 receives the position of the user in the 3D model of the space, and in step 546, the processing unit 4, 5 or the processor 210 of the control circuitry 136, or both, determines the position of the one or more objects in the user field of view based on the position of the user and the one or more objects in the 3D model of the space. In another example, the processing units 4, 5 receive the location of the user and one or more objects as determined by the computer system 12.

FIG. 6E is a flow diagram of an embodiment of a method for identifying one or more objects in a user's field of view when front facing camera 113 is a depth camera providing depth image data or has a depth sensor for providing depth data that may be combined with image data to provide depth image data. At step 550, one or more processors of display device 2 (e.g., processor 210 of the control circuitry, or processing units 4, 5, or both) identify one or more real objects in the user's field of view, including their three-dimensional positions, based on depth image data from one or more front-facing cameras. In addition to the image data, the one or more processors may also map the user field of view based on the orientation data from the orientation sensor 132. The one or more processors perform step 514 of identifying a virtual object location in the user field of view based on the application being executed, and step 516 of determining a three-dimensional location of each object in the user field of view. Additionally, remote computer system 12 may also provide additional processing power to other processors for performing the steps of FIG. 6E.

Each of the method embodiments of fig. 6A-6E is typically performed repeatedly as the user and objects within the user environment move around.

FIG. 6F is a block diagram of an embodiment of a system for determining a position of an object within a user's field of view of a see-through, near-eye display device. This embodiment illustrates how devices leverage a networked computer to map a three-dimensional model of a user's field of view and real and virtual objects within the model. An application 456 executing in a processing unit 4, 5 communicatively coupled to the display device 2 may communicate with the computing system 12 for processing image data to determine and track a three-dimensional user field of view over one or more communication networks 50. Computing system 12 may remotely execute applications 452 for processing units 4, 5 to provide images of one or more virtual objects. Either or both of the applications 456 and 452 working together may map a 3D model of the space around the user. The depth image processing application 450 detects objects, identifies objects, and their locations in the model. The application 450 may perform its processing based on depth image data from depth cameras like 20A and 20B, two-dimensional or depth image data from one or more front facing cameras 113, and GPS metadata associated with objects in the image data obtained from the GPS image tracking application 454.

The GPS image tracking application 454 identifies images of the user's location in one or more image databases 470 based on GPS data received from the processing unit 4, 5 or other GPS units identified as being in the vicinity of the user, or both. Additionally, the image database may provide accessible location images with metadata like GPS data and data identifying uploads by users who wish to share their images. The GPS image tracking application provides the distance between objects in the image to the depth image processing application 450 based on the GPS data. Additionally, the application 456 may perform the processing for mapping and locating objects in 3D user space locally and may interact with a GPS image tracking application to receive distances between objects. By leveraging network connectivity, many combinations of sharing processing between applications are possible.

Fig. 7 is a flow diagram of an embodiment of a method for determining gaze in a see-through, near-eye, mixed reality display system. The process is one embodiment of step 422 of FIG. 5A. At step 602, one or more processors, such as processing unit 4, mobile device 5, control circuitry 136, or hub computing system 12, alone or in combination, determine the boundaries of the gaze-detection coordinate system. At step 604, a gaze vector for each eye is determined based on the reflected eye data including glints, and at step 606, a point of gaze, e.g., what the user is viewing, in a three-dimensional (3D) user field of view for both eyes is determined. Because the location and identity of objects in the user field of view are tracked, for example, by embodiments like those in fig. 6A-6F, any object at the point of gaze in the 3D user field of view is identified at step 608. In many embodiments, the three-dimensional user field of view includes actual direct views of displayed virtual and real objects. The term object includes a person.

The method embodiment in fig. 7 and other method embodiments discussed below that use glint data for other ways of detecting gaze may identify such glints from image data of the eye. When an IR illuminator is used, an IR image sensor is also generally used. The following method also works with discrete surface Position Sensitive Detectors (PSDs), such as PSDs with pixels. FIG. 8 is a flow diagram of an embodiment of a method for identifying flashes in image data. As mentioned above, glints are very small and often very bright reflections of light from a light source on a specular reflective surface such as the cornea of an eye. In the following method embodiments, each of the steps is performed on a set of data samples. In some examples, this may include data from one image or image frame, and in other examples, the set of data samples may be multiple images or image frames. At step 605, the processor identifies each connected set of pixels that have their own intensity value within a predetermined intensity range, e.g., the intensity value range may begin at 220 and end at the brightest pixel value of 255. In step 607, candidate glints are pruned by identifying each connected set of pixels that satisfy the glint geometry criteria as candidate glints. Examples of flash geometry criteria are size and shape. Some may be too large, too small, or have an excessively irregular shape. In addition, the illuminators are positioned so that the resulting flashes have a spatial or geometric relationship to each other. For example, the illuminator 153 is arranged so that the flash forms a rectangle. In embodiments where the pupil center discussed in fig. 9 is also determined from the image data, the spatial relationship to the pupil may also be a criterion, e.g., too far away from the pupil may indicate that the connected set is not a candidate glint.

In step 609, the one or more processors determine whether the candidate flashes are less than a predetermined number. For example, for four luminaires, four flashes are expected, but the predetermined number may be two. In the example of a rectangle as the geometric relationship, two flashes forming a horizontal line or a diagonal line of a predetermined length may be selected as candidates. For other glints, there may be eyelid or eyelash obscuration. If there are less than the predetermined number of flashes, the set of data samples is discarded for further processing and processing returns to step 605 at step 611 to process the next set of data samples. If the candidate is not less than the predetermined number, step 613 determines if the candidate flash is more than the predetermined number. If there are more candidates, the one or more processors select a predetermined number of candidates as flashes that most closely fit a predetermined geometric relationship between the flashes at step 615. For example, for rectangles, those candidates that most closely form a rectangle of the predetermined size and shape. If there are no more candidates than this number, the number of candidates matches the predetermined number of flashes, and these candidates are selected as flashes at step 617.

Due to the geometry of the placement of the illuminator used to generate the glints discussed above, the glints appear in the same location unless the frame 115 is moved relative to the eye. In addition, since the positioning of the illuminators relative to each other on the support structure of the frame 115 or lens 118 is fixed, the spatial relationship of the flashes to each other in the image is also fixed. As for size, since the flash is very small, the number of pixels making up the flash area on the sensor and in the sensed image will be correspondingly small. For example, if the image sensor of the camera has 1000 pixels, each flash may occupy less than ten pixels. Flashes in each image frame taken at a rate of, for example, 30 or 60 frames per second may be monitored, and an area may be identified as a flash from a certain number of frame samples. Flash data may not be present in every frame. The sampling accommodates or smoothes out obstructions to the glint and pupil data in the different image frames, such as due to factors like the eyelids or eyelashes that cover the glints and/or pupils. Image frames are an example of an image format.

Fig. 9 is a flow diagram of an embodiment of a method that may be used to implement step 602 of determining boundaries of a gaze-detection coordinate system. At step 612, the one or more processors determine the location of the corneal center 164 of each eye relative to the illuminator 153 and the at least one light sensor (e.g., 134 or 152) based on the glints. Based on the image data provided by the at least one sensor, the one or more processors determine the pupil center of each eye at step 614. At step 616, the location of the center of rotation of the eyeball is determined relative to the cornea and pupil center. For example, based on the pupil center, the light rays may extend back to the eyeball center of rotation 166 through the determined corneal center 164. In addition, the distance or length approximation is used to approximate the length on the optical axis between the pupil and the cornea (e.g., about 3 mm) and the length on the optical axis between the center of curvature of the cornea and the center of rotation of the eyeball (about 6 mm). These values have been determined based on demographic studies on human ocular parameters, as gathered by gullsland. (see Hennessey, page 88).

Optionally, at step 618, the one or more processors determine a location of an eyeball center of rotation relative to the illuminator and the at least one sensor for the respective eye. In one embodiment, this position determined at step 618 provides a depth distance between a fixed point (or a point that may be approximated as fixed to accurately account for gaze detection) and the display optical system. However, the eyeball center of rotation 166 need not be fixed. In practice, a depth axis is defined for the gaze detection coordinate system. The detected changes along the depth axis may be used to indicate that the near-eye display system has moved and again trigger the determination of coordinate system boundaries or recalibration of the training gaze data set, as discussed below.

Figure 10 illustrates an embodiment of a method for determining the location of the corneal center in a coordinate system using an optical element of a see-through, near-eye, mixed reality display. At step 622, the one or more processors generate a first plane including points including a location of a first illuminator used to generate the first flash, a location of a pupil center of the at least one image sensor, such as a camera entrance pupil center, and a location of the first flash. As in the embodiment of fig. 3A, the pupil center of the camera may be located relative to the detection region 139, the detection region 139 acting as an image plane and directing light it receives into an image sensor in another location. In other examples, as in fig. 3B and 3C, the detection region 139 itself may be an image sensor, which is an image plane. The first plane will also include the location of the center of the cornea. Similarly, at step 624, the one or more processors generate a second plane including points including a location of a second illuminator used to generate a second flash, a location of a same pupil center of the at least one sensor, and a location of the second flash. The two planes share the same camera pupil center as the origin, and the distance vector to each illuminator is fixed relative to the camera pupil center because the image sensor and illuminator are positioned at predetermined locations on the near-eye display device. These predetermined positions allow points in the plane to be related to each other in a third coordinate system that includes the two illuminators, the position of the camera pupil center, and the corneal center of curvature. At step 626, the processor determines the location of the center of curvature of the cornea based on the intersection of the first and second planes.

Fig. 11 provides an illustrative example of the geometry of a gaze-detection coordinate system 500 that may be used by the embodiment of fig. 10 to find the center of the cornea. In this embodiment, the at least one sensor is a camera modeled as a pinhole camera. The geometry depicted is a slightly modified version of fig. 3 on page 89 of Hennessey et al, "single camera eye gaze tracking system with free head motion" (ETRA 2006, san diego, california, ACM, page 88, pages 87-94 (hereinafter Hennessey), which is incorporated herein by reference). The following provides a list of variables:

is a luminaire_iThe light of the illuminator producing a flash of light(e.g. 174) of the first,

is a luminaire_i(153) On the surface of the corneaThe generated flash of light is emitted to the outside,

is the camera pupil center of the pinhole camera model,

is a flash of lightAn image on an image plane, which is the detection area 139 of the camera sensor,

is a slave pointToThe scalar distance or length of (a) or (b),

is from the center of the pupil of the cameraTo flash lightImage on image sensorThe vector of (a) is determined,

is from the center of the pupil of the cameraTo the illuminator_iPosition ofThe vector of (a) is determined,

in the case of the present example,shaft edgeThe definition of the method is that,

and in a coordinate systemThe axis causes a flash on the connected image plane 139 (detection area)Image of (2)Is/are as followsIs located at the position ofAndin the plane formed by the shaft.Is thatIn-plane representation of slave illuminator (153) positionGlints onto the corneal surface(174) The angle formed between the lines 502 of incident light rays.Is thatIn-plane representation slave flashTo the camera pupil centerThe camera pupil center is also the origin of the coordinate system.Is the central position of cornea, which is also positionedIn a plane.

Because the cornea is modeled as a sphere, r is the radius of the corneal sphere, and each glintIs a point on the first or outer surface of the sphere so that each glint is separated from the corneal center by a radius r. In the above example, the flashModeled as a point on the outer or first surface of the cornea. In such a model, the illuminator's light reflects off the cornea in the same medium (air) of the same refractive index as the reflected light of the glint directed back to the camera sensor.

As shown in fig. 11, with the glints on the surface of the corneaA perpendicular line or ray 506 may extend from the glint in the direction of the cornea, and also to the coordinate systemOf planeThe axes intersect. As also shown in FIG. 11, incident ray 502 and reflected ray 504 are associated with the illuminatorAnd the camera pupil centerLine betweenForming a right triangle. Thus, angle A and angle D each consist ofIs shown in whichAnd is

According to Hennessey, center of corneaMay be based on unknown parameters in the coordinate system 500To obtain the number of unknowns for 4: () The following three equations:

including the center of the corneaAnother flashCamera pupil center of cameraAnd location of another luminaireIs also formed. Camera pupil center of cameraAnd the cornea center is the same in each plane, but the camera pupil centerThe location is known. This would yield 6 equations with 8 unknowns. In Hennessey, the gaze detection coordinate system is taken as an auxiliary coordinate system, the rotation matrixThe points may be translated between an auxiliary coordinate system for each plane and a single world coordinate system such as a third coordinate system that correlates the position of the detection region 139 with the illuminator 153. There is a constraint where the corneal center defined for each glint is in the world coordinate systemAre the same as in (1), e.g.And 3 equations for different axis components, e.g.Andthus providing 9 equations with 8 unknowns. Hennessey (page 90) states that the solution is solved using a gradient descent algorithmThe numerical value of (c). Thus, the location center 164 of the cornea 168 is defined relative to the location of the illuminator and the image plane or detection region 139.

Figure 12 illustrates an embodiment of a method for determining the pupil center from sensor generated image data. At step 642, the one or more processors identify black pupillary regions in a plurality of image data samples of the respective eye, and at step 644, average the black pupillary regions in the plurality of image data samples to adjust for panning head. An assumption can be made that the pupil is circular and elliptical when viewed from an angle. One axis (major axis) of the ellipse remains unchanged because it represents an unchanged pupil diameter, assuming the illumination does not change, because the pupil size changes as the illumination changes.

When the pupil is looking straight ahead through the display, the pupil appears circular in image format, such as an image frame of a camera whose detection area is centered on the optical axis of the display. As the pupil changes its gaze and moves away from the center of the image frame, the pupil appears elliptical because from one angle the circle appears elliptical. The width of the minor axis of the ellipse changes as the gaze changes. The narrow ellipse to the left of the center of the image frame indicates that the user is looking far to the right. A wider ellipse at a smaller distance to the right of the center of the image frame indicates that the user is looking to the left but not far to the left.

The center of the pupil is the center of the ellipse. An ellipse is fitted from the detected edge points in the image. Because such edge points are noisy and not all of them are on an ellipse, the ellipse fitting process is repeated multiple times on a randomly selected subset of all edge points. The subset that is most consistent with all edge points is used to obtain the final ellipse. At step 646, the processor performs an ellipse fitting algorithm on the average black pupil region to determine an ellipse representative of the pupil, and at step 648, determines the pupil center by determining the center of the ellipse representative of the pupil.

Where the center of rotation, the center of the cornea, and the center of the pupil are identified, the optical axis of the eye can be obtained by extending rays from the center of rotation through the cornea and the center of the pupil. However, as mentioned above, the human gaze vector is the visual axis or line of sight from the fovea through the center of the pupil. Photoreceptors in the foveal region of the human retina are more densely packed than those in the rest of the retina. This region provides the highest visual acuity or visual clarity and also provides stereo vision of neighboring objects. After determining the optical axis, a default offset angle may be applied such that the optical axis approximates the visual axis and is selected as the gaze vector.

Fig. 13 illustrates an embodiment of a method for determining a gaze vector based on the determined pupil center, corneal center, and eyeball center of rotation, and which may be used to implement step 604. At step 652, the one or more processors model the optical axis 178 of the eye as rays extending from the center of rotation of the eyeball through the determined corneal center and pupil center, and at step 654, apply a correction to the modeled optical axis to estimate the visual axis. At step 656, the one or more processors extend the estimated visual axis from the pupil through display optics of the see-through, near-eye display into the user's field of view.

In one embodiment, using the fixed positioning of the illuminator as a basis, the effect of different regions of the eye on the reflectivity (and thus on the amount or intensity of reflected light) is used as a basis for gaze detection. Intensity data from IR or visible light sensors can be used to determine gaze, so reflectivity data can be based on IR-based reflectivity or visible light reflectivity. To illustrate, the sclera is more reflective than other regions of the eye (e.g., the pupil and iris). The illuminator 153, located on the frame 115, at a distance from the user's right side, causes a flash reflection on the right sclera of the user's right eye if the user looks farther to the user's left side. The PSD134r or as shown in fig. 3B, the photodetector 152 on the inner right frame near the nasal bridge 104 receives more of the reflected light represented in a data reading, while light from the reflection at other photodetectors 152 or other locations on the PSD receives less of the amount of reflected light in the range associated with the black pupil when the illuminator 153 closest to the nasal bridge is turned on. The reflectivity of the iris may also be captured by the camera 134 and stored for the user by the processor 210, the processing unit 4, or the mobile device 5 including the processing unit 4.

The accuracy may not be as high as that of an image based on the entire eye, but is sufficient for many applications. Additionally, such gaze detection may be used to assist or backup gaze detection techniques. For example, such flash-based techniques mitigate some processor overhead during computationally intensive time periods when complex virtual images are generated. Furthermore, such flash-based techniques may be performed more times over a period of time than image-based techniques, which process more data or are computationally intensive but more accurate techniques that can be run at a lower rate to periodically recalibrate the accuracy of gaze detection. An example of a gaze detection technique that is both image-based and more computationally intensive is a technique for determining a gaze vector relative to an interior portion of the eye based on glint data and pupil image data, such as the embodiments described in fig. 7-13, which can be run at a lower rate to periodically recalibrate the accuracy of gaze detection. For example, embodiments of more computationally intensive techniques based in part on image data may operate at a rate of ten (10) times per second, while glint-based gaze detection techniques may operate at a faster rate of one hundred (100) times per second, or even five hundred (500) times per second in some cases.

Fig. 14 is a flow diagram illustrating an embodiment of a method for determining gaze based on glint data. At step 673, data representing each flash intensity value is captured. Based on the specular reflectivity of the different eye regions and the location of the illuminator, an eyeball portion of each glint location that is geometrically related to the glint is identified based on the detected intensity values in step 674. At step 675, a gaze angle is estimated based on the eye portion associated with each of the flash locations. As described in the previous examples, the eyeball portion may be the iris, pupil, or sclera of the eyeball. The position of the illuminator forms the geometry of the flash, e.g. a frame, circle, rectangle, etc. that frames or encloses the pupil at least on both sides. A gaze vector is determined based on the gaze angle at step 676, and a gaze point in the 3D user field of view is determined based on the intersection of the determined gaze vectors for the two eyes at step 677.

As described above, different methods with different accuracies can be used at different cycle rates in exchange for speed in accuracy. Method embodiments based on flash intensity values, such as the method embodiment described in fig. 14, are examples of techniques with low computational intensity that may be used. In another example, the training gaze data set may be used for comparison with current pupil location data to determine a gaze vector.

Using the training data set for gaze determination relies on the assumption that the near-eye display device 2 has not moved relative to the eyes. If movement is detected, the training gaze data set is to be recalibrated. The illumination change may also be the basis for recalibration.

For each of a set of predetermined gaze directions, a training gaze data set is obtained. For example, the training data sets may be acquired for different portions of display optics 14, with the user's pupil gazed at a gaze angle or pupil angle through the display optics. In one example, there are nine (9) sections, one for each of the four (4) corners of the display optical system, a middle left frame or region, a middle right frame or region, a middle top frame, a middle bottom frame, and a center region. In the case of a flash, a comparison of the intensity values of the current data at the four flash locations with the training data set may be used.

Fig. 15A is a flow diagram illustrating an embodiment of a method for generating a set of training data sets for comparison-based determination of gaze. The method may be used to determine a training set of gaze angles based on glint intensity value data representing pupil locations. The method embodiment is presented in an exemplary loop structure beginning at step 702 and ending at step 714. At step 704, for each of the plurality of training gaze data sets, the control circuitry 136, the processing unit 4, the mobile device 5, the one or more processors of the networked hub computing environment 12, individually or in combination, generate virtual images at predetermined different locations in the user field of view for a predetermined period of time for the respective training data set. As discussed previously, microdisplay 120 generates virtual images at different locations in the user's field of view.

At step 706, data for each eye is captured during a predetermined time period based on the glints. From the captured data, the one or more processors determine data representative of the pupil location, such as a set of intensity values from a plurality of glints, at step 708. At step 710, a gaze vector is determined based on the pupil location data and the predetermined different location of the virtual image determined in the user field of view. In the case where pupil and glint data are captured, as a test, the gaze vector may be determined based on the corneal center, pupil center, and center of eyeball rotation as discussed above with reference to the embodiments of fig. 7-13, as well as the location of the virtual image. In the case of glint data only, the intensity values of the glints may be correlated with stored values reflecting different reflection areas on the eyes and associated with a gaze vector extending to the position of the virtual image in the user's field of view. The flash value may be checked against a set of values for a desired pupil angle at which the virtual image at the predetermined location is viewed. At step 712, the one or more processors store the pupil location data and the gaze vector for the corresponding training gaze data set, and continue at steps 714 and 702 to begin processing the next training gaze data set until a predetermined number of sets are reached.

Fig. 15B is a flow diagram illustrating an embodiment of a method for determining gaze based on a training data set. At step 722, at least one sensor captures data for each eye based on the reflected light, and at step 724 one or more processors determine data representative of the current pupil position from the captured data. At step 726, the one or more processors determine a gaze vector based on a comparison of the current pupil location data to the one or more training data sets, and at step 728, determine a gaze point based on the gaze vectors for the two eyes, e.g., where the two vectors intersect in the 3D user field of view.

FIG. 15C is a flowchart of an embodiment of an interpolation method that may be used with the comparison step 726 of FIG. 15B. For example, this embodiment may be used when comparing sensor data (e.g., PSD data) of the spatial relationship between flashes. At step 732, the one or more processors determine one or more distance vectors between the current pupil location data and pupil location data of at least a subset of the training gaze data sets according to the mapping criteria. In the detection area of a sensor (e.g., a camera sensor or a discrete position sensitive detector), the mapping may be a distance in mm to pixel mapping. For an isotropic PSD, the mapping may be of the area on the detector area to the distance in mm.

A flash of a box or other geometric shape provides another example. The distance vector for each current glint from the training gaze data set of glint intensity values indicates the direction of the intensity change because the glints are fixed unless the coordinate system moves.

At step 734, the one or more processors select a training gaze data set whose sum of the one or more distance vectors is minimal, and at step 736, interpolate a position change vector between a pupil position of the selected training gaze data set and a pupil position of the current data. At step 738, the one or more processors estimate the gaze vector of the selected training gaze data set moved by the change of position vector as the current gaze vector.

Especially when using training data for comparison, the movement of the gaze detection coordinate system is the reason for the recalibration of the training data set. The locations of the corneal center and the center of rotation may be periodically re-determined to determine whether their spatial relationship with the illuminator and the at least one sensor has changed.

Other tests for movement may be performed based on facial features having fixed characteristics in the image data. In one embodiment, the eye camera may capture an area of about 5 to 10mm around the visible eyeball portion of the corneal prominence, the white of the eye, the iris, and the pupil in order to capture portions of the eyelids and eyelashes. Facial features that are fixed in position, such as moles or spots on the skin, such as eyelids, or on the bottom edge of the skin encompassing the lower eye, may also be presented in the image data of the eye. In the image sample, the location of the nevus or plaque can be monitored for changes in location. If a facial feature moves up, down, right, or left, a vertical or horizontal shift may be detected. If the facial features appear larger or smaller, a depth change in the spatial relationship between the eyes and the display device 2 may be determined. Due to things like camera resolution, there may be a range of criteria for triggering a change in position of the recalibration of the training gaze data set.

In another example, although illumination is a factor that changes the pupil size and the ratio of the pupil area to the iris periphery or visible iris area within the periphery, the size of the iris periphery or periphery does not change with gaze changes or illumination changes, and thus, the periphery or periphery is a fixed characteristic of the iris as a facial feature. By fitting an ellipse to the iris, the one or more processors may determine whether the iris has become larger or smaller in the image data based on criteria. If larger, the display device 2 with its illuminator 153 and at least one sensor 134 has moved in depth closer to the user's eyes; if it gets smaller, the display device 2 has moved further away. A change in the fixed characteristic may trigger a recalibration of the training data set.

Fig. 16 is a flow diagram illustrating an embodiment of a method for checking recalibration of a gaze determination system. At step 742, the one or more processors of the display device 2 or in communication therewith check whether a change has occurred that caused the recalibration of the training data set. One check is to determine whether movement according to some criteria has occurred at step 744. The examination may be periodically determining the gaze vector in three dimensions, as discussed in accordance with fig. 7-13, and noting that the position of the eye rotation has changed relative to one or more gaze examination elements of the see-through, near-eye display device. The criterion may be a distance of movement in any of the three dimensions. Based on the determination in step 744 that movement has occurred indicating no movement, the one or more processors determine whether a change in illumination according to some criteria has occurred in step 746. In response to a negative determination in step 746, further processing prior to the next planned movement check is performed in step 747. If movement is indicated, it may have been detected in image-based techniques based on facial features. Accordingly, an optional step 748 of determining boundaries of the gaze detection coordinate system may be performed as discussed with respect to the embodiments of fig. 7-13. In step 750, a new set of training gaze data sets is generated in response to the movement. Further, if it is determined at step 746 that there is an illumination change that exceeds a threshold or other criteria, the new set of training gaze data sets may also be triggered at step 750.

FIG. 17 is a block diagram of an exemplary mobile device that may operate in embodiments of the present technology. Exemplary electronic circuitry of a typical mobile telephone is depicted. The phone 900 includes one or more microprocessors 912, and memory 1010 (e.g., non-volatile memory such as ROM and volatile memory such as RAM) that stores processor readable code that is executed by one or more processors of the control processor 912 to implement the functions described herein.

The mobile device 900 may include, for example, a processor 912, memory 1010 including applications and non-volatile storage. The processor 912 may implement communications as well as any number of applications, including the interactive applications described herein. The memory 1010 can be any variety of memory storage media types including non-volatile and volatile memory. The device operating system handles the different operations of the mobile device 900 and may contain user interfaces for operations such as making and receiving phone calls, text messaging, checking voicemail, and the like. The application 1030 may be any kind of program, such as a camera application for photos and/or videos, an address book, a calendar application, a media player, an internet browser, games, other multimedia applications, an alarm application, other third party applications, the interaction applications discussed herein, and so forth. The non-volatile storage component 1040 in memory 1010 contains data such as web caches, music, photos, contact data, scheduling data, and other files.

The processor 912 also communicates with the RF transmit/receive circuitry 906, which circuitry 906 in turn is coupled to the antenna 902, which also communicates with the infrared transmitter/receiver 908, with any additional communication channel 1060 like Wi-Fi or bluetooth, and with the movement/orientation sensor 914 like an accelerometer. An accelerometer is included in the mobile device to enable applications such as intelligent user interfaces that let the user input commands through gestures, indoor GPS functionality that calculates the movement and direction of the device after disconnecting from GPS satellites, and to detect the orientation of the device and automatically change the display from portrait to landscape when the phone is rotated. The accelerometer may be provided, for example, by a micro-electromechanical system (MEMS), which is a tiny mechanical device (micron-scale) built on a semiconductor chip. Acceleration direction, as well as orientation, vibration and shock can be sensed. The processor 912 also communicates with a ringer/vibrator 916, a user interface keypad/screen, a biometric sensor system 918, a speaker 1020, a microphone 922, a camera 924, a light sensor 926, and a temperature sensor 928.

The processor 912 controls the transmission and reception of wireless signals. During a transmit mode, processor 912 provides a voice signal or other data signal from microphone 922 to RF transmit/receive circuitry 906. Transmit/receive circuitry 906 transmits the signal to a remote station (e.g., a fixed station, carrier, other cellular telephone, etc.) for communication via antenna 902. The ringer/vibrator 916 is used to signal an incoming call, text message, calendar reminder, alarm clock reminder, or other notification to the user. During a receive mode, the transmit/receive circuitry 906 receives voice or other data signals from a remote station via the antenna 902. The received voice signals are provided to the speaker 1020, while other received data signals are also processed appropriately.

In addition, a physical connector 988 may be used to connect the mobile device 900 to an external power source, such as an AC adapter or powered docking station. The physical connector 988 may also be used as a data connection to a computing device. The data connection allows operations such as synchronizing mobile device data with computing data on another device.

A GPS transceiver 965 that relays the location of the user application using satellite-based radio navigation is enabled for such services.

The example computer systems illustrated in the figures include examples of computer-readable storage media. Computer readable storage media are also processor readable storage media. Such media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, cache, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, memory sticks or cards, magnetic cassettes, magnetic tape, a media drive, a hard disk, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer.

FIG. 18 is a block diagram depicting one embodiment of a computing system that may be used to implement the hub computing system of FIGS. 1A and 1B. In this embodiment, the computing system is a multimedia console 800, such as a gaming console. As shown in fig. 18, the multimedia console 800 has a Central Processing Unit (CPU) 801 and a memory controller 802 that facilitates processor access to various types of memory, including a flash Read Only Memory (ROM) 803, a Random Access Memory (RAM) 806, a hard disk drive 808, and a portable media drive 806. In one implementation, CPU801 includes a level 1 cache 810 and a level 2 cache 812 to temporarily store data and thus reduce the number of memory access cycles made to hard disk drive 808, thereby improving processing speed and throughput.

The CPU801, the memory controller 802, and various memory devices are interconnected via one or more buses (not shown). The details of the bus used in this implementation are not particularly relevant to understanding the subject matter of interest discussed herein. It should be understood, however, that such a bus may include one or more of serial and parallel buses, a memory bus, a peripheral bus, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures can include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an enhanced ISA (eisa) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus also known as a mezzanine bus.

In one implementation, the CPU801, memory controller 802, ROM803, and RAM806 are integrated onto a common module 814. In this implementation, ROM803 is configured as a flash ROM that is connected to memory controller 802 via a PCI bus and a ROM bus (neither of which are shown). RAM806 is configured as multiple Double Data Rate Synchronous Dynamic RAM (DDRSDRAM) modules that are independently controlled by memory controller 802 via separate buses (not shown). Hard disk drive 808 and portable media drive 805 are shown connected to memory controller 802 by a PCI bus and an AT attachment (ATA) bus 816. However, in other implementations, different types of dedicated data bus structures may alternatively be applied.

Graphics processing unit 820 and video encoder 822 form a video processing pipeline for high speed and high resolution (e.g., high definition) graphics processing. Data is transmitted from a Graphics Processing Unit (GPU) 820 to a video encoder 822 over a digital video bus (not shown). Lightweight messages (e.g., popups) generated by system applications are displayed using the GPU820 interrupt to schedule code to render popup into an overlay. The amount of memory used for the overlay depends on the overlay area size, and the overlay preferably scales with the screen resolution. Where the concurrent system application uses a full user interface, it is preferable to use a resolution that is independent of the application resolution. A scaler (scaler) may be used to set this resolution, thereby eliminating the need to change the frequency and cause a TV resynch.

An audio processing unit 824 and an audio codec (coder/decoder) 826 form a corresponding audio processing pipeline for multi-channel audio processing of various digital audio formats. Audio data is communicated between audio processing unit 824 and audio codec 826 via a communication link (not shown). The video and audio processing pipelines output data to an a/V (audio/video) port 828 for transmission to a television or other display. In the illustrated implementation, the video and audio processing component 820 and 828 are installed on the module 214.

FIG. 18 shows a module 814 including a USB host controller 830 and a network interface 832. USB host controller 830 is shown in communication with CPU801 and memory controller 802 via a bus (e.g., a PCI bus) and hosts peripheral controllers 804(1) - (804) - (4). The network interface 832 provides access to a network (e.g., the internet, home network, etc.) and may be any of a wide variety of various wired or wireless interface components including an ethernet card, a modem, a wireless access card, a bluetooth module, a cable modem, and the like.

In the implementation depicted in fig. 18, the console 800 includes a controller support subassembly 840 for supporting four controllers 804(1) -804 (4). The controller support subassembly 840 includes any hardware and software components necessary to support wired and wireless operation with external control devices such as, for example, media and game controllers. The front panel I/O subassembly 842 supports the multiple functions of the power button 812, the eject button 813, as well as any LEDs (light emitting diodes) or other indicators exposed on the outer surface of the console 802. Subassemblies 840 and 842 are in communication with module 814 via one or more cable assemblies 844. In other implementations, the console 800 may include additional controller subcomponents. The illustrated implementation also shows an optical I/O interface 835 configured to send and receive signals that may be passed to module 814.

MUs 840(1) and 840(2) are shown as being connectable to MU ports "a" 830(1) and "B" 830(2), respectively. Additional MUs (e.g., MUs 840(3) -840 (6)) are shown as connectable to controllers 804(1) and 804(3), i.e., two MUs per controller. Controllers 804(2) and 804(4) may also be configured to receive MUs (not shown). Each MU840 provides additional storage on which games, game parameters, and other data may be stored. In some implementations, the other data can include any of a digital game component, an executable gaming application, an instruction set for expanding a gaming application, and a media file. When inserted into console 800 or a controller, MU840 may be accessed by memory controller 802. The system power supply module 850 supplies power to the components of the gaming system 800. A fan 852 cools the circuitry within console 800. A microcontroller unit 854 is also provided.

An application 860 comprising machine instructions is stored on hard disk drive 808. When console 800 is powered on, various portions of application 860 are loaded into RAM806, and/or caches 810 and 812, for execution on CPU801, with application 860 being one such example. Various applications may be stored on hard disk drive 808 for execution on CPU 801.

Gaming and media system 800 may be used as a standalone system by simply connecting the system to monitor 16 (FIG. 1A), a television, a video projector, or other display device. In this standalone mode, gaming and media system 800 allows one or more players to play games or enjoy digital media, such as watching movies or listening to music. However, with the integration of broadband connectivity made possible through network interface 832, gaming and media system 800 may also be operated as a participant in a larger network gaming community.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims (9)

1. A method for controlling the brightness of a see-through display, comprising:

estimating, using an eye-tracking camera, an area of a see-through display that a wearer is looking at;

determining a light intensity of the area at which a user is gazing;

determining a distance between the wearer's eye and the see-through display based on 3D imaging; and

adjusting the brightness of the see-through display based on the light intensity of the region and further based on the distance.

2. The method of claim 1, wherein adjusting the brightness of the see-through display based on the light intensity of the region comprises:

adjusting an opacity of the see-through display.

3. The method of claim 1 or 2, wherein adjusting the brightness of the see-through display based on the light intensity of the region comprises:

adjusting an intensity of light projected by the see-through display.

4. The method of any of claims 1-2, further comprising:

determining a pupil size of the wearer, the adjusting the brightness of the see-through display based on the light intensity of the region further based on the pupil size of the wearer.

5. The method of any of claims 1-2, wherein adjusting the brightness of the see-through display is further based on one or more of: the wearer's environment, the type of content being presented on the see-through display, or the format of the content being presented on the see-through display.

6. The method of any of claims 1-2, further comprising:

determining a baseline magnification response of the wearer, the adjusting the brightness of the see-through display further based on the baseline magnification response of the wearer.

7. A display system, comprising:

a see-through, near-eye display device including a respective display optical system for each eye positioned to be seen through by the respective eye;

an image generation unit for each eye attached to the see-through display device for generating at least one virtual image for display in the display optical system;

a respective arrangement of gaze detection elements positioned on the display device;

logic coupled to the gaze detection element, the logic to determine a gaze estimate of a wearer of the see-through display device, the logic to access brightness data of a field of view of the system, the logic to determine a light intensity of an area in the field of view being gazed, the logic to adjust a brightness of the see-through display device based on the light intensity of the area and further based on a distance between an eye of the wearer and the see-through display device.

8. The display system of claim 7, wherein the logic adjusts an opacity of the see-through display and/or a brightness of light projected by the see-through display based on a light intensity of the region.

9. The display system of claim 7 or 8, wherein the logic determines a pupil size of the wearer, the logic adjusting a brightness of the see-through display and/or an opacity of the see-through display based on the pupil size of the wearer.