HK1241033B - Virtual and augmented reality systems and methods - Google Patents

Description

Virtual and augmented reality systems and methods

This application is a divisional application of application No. 201480074051.7, filed on November 27, 2014, and entitled “Virtual and Augmented Reality Systems and Methods”.

Technical Field

The present disclosure relates to virtual reality and augmented reality imaging and visualization systems.

Background Art

Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears realistic or can be perceived as realistic. Virtual reality or "VR" scenarios generally involve presenting digital or virtual image information without being able to see other actual real-world visual input; augmented reality or "AR" scenarios generally involve presenting digital or virtual image information as a visual augmentation of the actual world around the user. For example, referring to FIG1 , an augmented reality scene (4) is depicted in which a user of AR technology sees a real-world setting (6) resembling a park, which features people, trees, buildings in the background, and a concrete platform (1120). In addition to these items, the user of AR technology also perceives that he "sees" a robotic figure (1110) standing on the real-world platform (1120) and a cartoon-like virtual figure (2) flying around that looks like an incarnation of a bumblebee, even though these elements (2, 1110) do not exist in the real world. In reality, the human visual perception system is extremely complex, and it is very challenging to produce VR or AR technology that promotes the comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements.

Referring to FIG2A , a stereoscopic wearable glasses (8) type configuration has been developed that is characterized by typically including two displays (10, 12) configured to present displayed images through slightly different elements, thereby causing the human visual system to perceive a three-dimensional perspective. It has been found that such a configuration causes many users to experience discomfort due to a mismatch between vergence and accommodation, which must be overcome in order to perceive a three-dimensional image; in fact, some users cannot tolerate the stereoscopic configuration. FIG2B shows another stereoscopic wearable glasses (14) type configuration that is characterized by including two front-mounted cameras (16, 18) configured to capture images for presenting augmented reality to the user through a stereoscopic display. When the glasses (14) are worn on the user's head, the position of the cameras (16, 18) and the displays typically obstructs the user's natural field of view.

Referring to FIG2C , an augmented reality configuration (20) is shown that features a visualization module (26) coupled to an eyeglass frame (24) that also holds a conventional eyeglass lens (22). With this system, a user is able to see at least partially unobstructed real-world scenes and is provided with a small display (28) through which a digital image can be presented to one eye in an AR configuration—for monocular AR presentation. FIG2D features a configuration in which a visualization module (32) can be coupled to a hat or helmet (30) and configured to present a monocular augmented digital image to the user via the small display (34). FIG2E shows another similar configuration in which a frame (36) can be coupled to the user's head in a manner similar to that of eyeglasses so that a visualization module (38) can be used to capture images and present the monocular augmented digital image to the user via the small display (40). For example, such a configuration is marketed by Google, Inc. of Mountain View, California, under the trademark Google Glass (RTM). None of these configurations is optimal for presenting rich, binocular, 3D AR experiences to users in a comfortable and maximally useful manner, in part because existing systems fail to address certain fundamental aspects of the human perceptual system, including the retinal photoreceptors used to generate the user's visual perception and their collaboration with the brain.

Referring to FIG3 , a simplified cross-sectional view of the human eye is shown, characterized by the cornea (42), iris (44), lens (46), sclera (18), choroid (50), macula (52), retina (54), and the optic nerve pathway (56) to the brain. The macula is the center of the retina and is used to see moderate detail; in the center of the macula is a part of the retina called the "fovea" that is used to see the finest detail and contains more photoreceptors than any other part of the retina (approximately 120 cones per degree of vision). The human visual system is not a passive sensor type system, it is configured to actively scan the environment. In a manner somewhat similar to the use of a desktop scanner to capture an image, or the use of fingers to read Braille from paper, the eye's photoreceptors are triggered in response to changes in the stimulus, rather than continuously responding to a constant state of the stimulus. Therefore, an action is required (the action of the linear scanner array across the paper in a desktop scanner, or the action of the finger across the Braille printed on the paper) to present the photoreceptor information to the brain. In fact, experiments using substances such as cobra venom (used to numb eye muscles) have shown that if a subject stares at a static scene with both eyes open, using the venom-numbed eye, they become blind. In other words, if the stimulus remains unchanged, the photoreceptors will not provide input to the brain, and blindness may occur. This is believed to be at least one reason why normal people are observed to move or jitter back and forth in side-to-side movements called "microsaccades."

As mentioned above, the fovea of the retina contains the greatest density of photoreceptors, and while humans are generally thought to have high-resolution vision across their entire field of view, in reality, they typically have only a small, high-resolution center, and frequently mechanically scan around this center while maintaining a persistent memory of the high-resolution information captured by the fovea. In a somewhat similar manner, the eye's focus control mechanism (the ciliary muscle, which is coupled to the lens in a manner whereby relaxation of the ciliary body causes the taut ciliary connective tissue fibers to flatten the lens for farther focus; contraction of the ciliary body causes the ciliary connective tissue fibers to loosen, which allows the lens to adopt a more rounded geometry for closer focus) oscillates back and forth by approximately 1/4 to 1/2 a diopter to cyclically introduce a small amount of so-called "refractive blur" at both the near and far ends of the target focal distance; this is used as cyclical negative feedback by the brain's visual accommodation control loop, which helps to continuously correct the orientation and sharpen the retinal image of a fixed object.

The brain's visualization centers also derive valuable perceptual information from the movement of the eyes and their components relative to each other. The vergence movement of the eyes relative to each other (i.e., the rolling of the pupils toward or away from each other to fix the eyes' line of sight on an object) is closely related to the focusing (or "accommodation") of the eye's lens. Typically, changing the focus of the eye's lens, or adjusting the eye to focus on an object at a different distance, automatically matches the change in vergence to the same distance, according to a relationship known as the "accommodation and convergence reflex." Likewise, a change in vergence will typically trigger a matching change in accommodation. Working according to this reflex (as most traditional stereoscopic AR or VR configurations do) can cause users to experience eyestrain, headaches, or other forms of discomfort.

Movement of the head, where the eyes are located, also has a significant impact on the visualization of objects. Humans move their heads to observe the world around them, and they are generally in a very constant state of repositioning and reorienting their heads relative to objects of interest. Furthermore, most people are willing to move their heads if their eye gaze needs to be more than about 20 degrees off-center to focus on a particular object (that is, people generally don't like to look at things out of the corner of their eyes). People also scan or move their heads relative to sounds—to improve sound signal capture and to take advantage of the geometric relationship of the ears relative to the head. The human visual system derives important depth cues from so-called "head motion parallax," which involves the relative movement of objects at different distances based on head movement and eye convergence distance (i.e., if a person moves their head from side to side and maintains their gaze on an object, the items farthest from the object will move in the same direction as the head, and items in front of the object will move in the opposite direction of the head movement. When objects are spatially located relative to the person's environment, these are very significant cues—perhaps as important as stereo parallax). Of course, head movement is also used to observe the surroundings of an object.

Further, head and eye movements are coordinated with something called the "vestibulo-ocular reflex," which stabilizes image information relative to the retina during head rotation, so that the object image information is nearly centered on the retina. In response to head rotation, the eyes rotate proportionally in the opposite direction to maintain a stable gaze on the object. As a result of this compensatory relationship, many people can read a book while shaking their head back and forth (interestingly, the same result cannot be achieved if the book is shaken back and forth at the same speed while the head is barely moving—people are unlikely to read a moving book, and the vestibulo-ocular reflex is one of the head and eye movement coordinations that is not generally performed for hand movements). This paradigm may be very important for augmented reality systems because the user's head movements can be relatively directly related to eye movements, and such systems are more likely to work by exploiting this relationship.

In practice, given the various relationships described above, when placing digital content (e.g., 3D content such as a virtual chandelier object rendered to augment the real-world scene of a room) or 2D content such as a flat/smooth virtual painting object rendered to augment the real-world scene of a room), design choices can be made to control the behavior of the object. For example, a 2D painting object can be head-centered, in which case the object rotates with the user's head (e.g., as in the Google Glass approach); or the object can be world-centered, in which case the object is rendered as if it were part of a real-world coordinate system, so the user can move their head or eyes without moving the object's position relative to the real world.

Thus, when placing virtual content in an augmented real world presented by an augmented reality system, objects should be presented as world-centered (i.e., virtual objects are located at real-world locations such that a user can move their body, head, eyes without changing their position relative to surrounding real-world objects (such as real-world walls); body or torso-centered, in which case the virtual elements can be fixed relative to the user's torso so that the user can move their head or eyes without moving the objects, but such movement follows torso movement; head-centered, in which case the displayed objects (and/or the display itself) can move with head movement, as described above with reference to Google Glass; or eye-centered, as in a "foveated display" configuration (described below), where the content rotates with the position of the eyes.

For world-centric configurations, it may be necessary to have inputs such as accurate head pose measurements, accurate representation and/or measurements of real-world objects and geometry around the user, low-latency dynamic rendering based on head pose in the augmented reality display, and generally low-latency displays.

The systems and techniques described herein are configured to work with normal human visual configuration to address these challenges.

Summary of the Invention

Embodiments of the present invention relate to apparatus, systems, and methods for facilitating virtual reality and/or augmented reality interaction for one or more users. In one aspect, a system for displaying virtual content is disclosed.

In one or more embodiments, the system includes a light source that multiplexes one or more light patterns associated with one or more frames of image data in a time-sequential manner, and a reflector array that receives the one or more light patterns and variably directs light toward an exit pupil.

In one or more embodiments, the system includes: an image generation source that provides one or more frames of image data in a time-sequential manner; a light modulator that is configured to emit light associated with the one or more frames of the image data; a substrate that directs image information toward the user's eyes, wherein the substrate accommodates a plurality of reflectors; a first reflector of the plurality of reflectors that reflects the emitted light associated with the first frame of the image data toward the user's eyes at a first angle; and a second reflector that reflects the emitted light associated with the second frame of the image data toward the user's eyes at a second angle.

In one or more embodiments, the reflection angles of the multiple reflectors can be variable. In one or more embodiments, the reflectors can be switchable. In one or more embodiments, the multiple reflectors can be electro-optically active. In one or more embodiments, the refractive index of the multiple reflectors can be changed to match the refractive index of the substrate. In an optional embodiment, the system can also include a high-frequency gating layer, which can be configured to be located between the substrate and the user's eyes, the high-frequency gating layer having a controllably movable hole. In one or more embodiments, the hole of the high-frequency gating layer can be moved in a manner that selectively transmits image data only by light reflected through the hole. One or more reflectors of the transmissive beam splitter substrate can be blocked by the high-frequency gating layer. In one or more embodiments, the hole can be an LCD hole. In one or more embodiments, the hole can be a MEMS array. In one or more embodiments, the first angle can be the same as the second angle. In one or more embodiments, the first angle can be different from the second angle.

In one or more embodiments, the system may further include a first lens that directs a set of light rays through the node and to the user's eyes. In one or more embodiments, the first lens may be configured to be located on the substrate and in front of the first reflector so that the set of light rays emitted from the reflector pass through the first lens before reaching the user's eyes.

In one or more embodiments, the system may further include a second lens that compensates for the first lens, and the second lens may be configured to be located on the substrate and on a side opposite to the side where the first lens is located, thereby resulting in zero magnification.

In one or more embodiments, the first reflector among the plurality of reflectors may be a curved reflective surface that collects a group of light rays associated with the image data into a single output point before the group of light rays is transmitted to the user's eyes. In one or more embodiments, the curved reflector may be a parabolic reflector. In one or more embodiments, the curved reflector may be an elliptical reflector.

In another embodiment, a method for displaying virtual content includes providing one or more light patterns associated with one or more frames of image data in a time-sequential manner; and reflecting the one or more light patterns associated with the one or more frames of image data to an exit pupil via a transmissive beam splitter having multiple reflectors to variably direct light to the exit pupil.

In one or more embodiments, the reflection angles of the multiple reflectors are variable. In one or more embodiments, the reflectors are switchable. In one or more embodiments, the multiple reflectors may be electro-optically active. In one or more embodiments, the refractive index of the multiple reflectors may be changed to match the refractive index of the substrate. In an optional embodiment, the system may further include a high-frequency gating layer that may be configured to be located between the substrate and the user's eyes, the high-frequency gating layer having a controllably movable hole. In one or more embodiments, the hole of the high-frequency gating layer may be moved in a manner that selectively transmits image data only by light reflected through the hole. One or more reflectors of the transmissive beam splitter substrate may be blocked by the high-frequency gating layer. In one or more embodiments, the hole may be an LCD hole. In one or more embodiments, the hole may be a MEMS array. In one or more embodiments, the first angle may be the same as the second angle. In one or more embodiments, the first angle may be different from the second angle.

In one or more embodiments, the system may further include a first lens that directs a set of light rays through the node and to the user's eyes. In one or more embodiments, the first lens may be configured to be located on the substrate and in front of the first reflector so that the set of light rays emitted from the reflector pass through the first lens before reaching the user's eyes.

In one or more embodiments, the system may further include a second lens that compensates for the first lens, and the second lens may be configured to be located on the substrate and on a side opposite to the side where the first lens is located, thereby resulting in zero magnification.

In one or more embodiments, the first reflector among the plurality of reflectors may be a curved reflective surface that collects a group of light rays associated with the image data into a single output point before the group of light rays is transmitted to the user's eyes. In one or more embodiments, the curved reflector may be a parabolic reflector. In one or more embodiments, the curved reflector may be an elliptical reflector.

In one or more embodiments, the wavefront can be a collimated wavefront. In one or more embodiments, the wavefront can be a curved wavefront. In some embodiments, the collimated wavefront can be perceived as a plane at infinite depth. In some embodiments, the curved wavefront can be perceived as a plane at a depth closer than optical infinity.

In another embodiment, a system for displaying virtual content to a user includes: a light source that multiplexes one or more light patterns associated with one or more frames of image data in a time-sequential manner; and a reflector array that receives the one or more light patterns, the reflector array being oriented at a particular angle; and a plurality of optical elements coupled to the reflector array to variably direct the light patterns toward an exit pupil.

In one or more embodiments, the reflector array may be separated from the optical element. In one or more embodiments, the reflector array includes a plane mirror. In one or more embodiments, the optical element may be a microlens coupled to the reflector array. In one or more embodiments, one or more reflectors in the reflector array may be arc reflectors. In one or more embodiments, the optical element is integrated into the reflector array. In one or more embodiments, the multiple optical elements may expand the exit pupil.

In one or more embodiments, the system may further include a first lens that directs a set of light rays through the node and to the user's eye, wherein the first lens may be configured to be located on the substrate and between the first reflector and the eye so that the set of light rays emitted from the reflector pass through the first lens before reaching the user's eye.

In one or more embodiments, the system may further include a second lens that compensates for the first lens, and the second lens may be configured to be located on the substrate and on a side opposite to the side where the first lens is placed, thereby resulting in zero magnification. In one or more embodiments, the plurality of reflectors may include wavelength selective reflectors. In one or more embodiments, the plurality of reflectors may include half-silvered mirrors. The plurality of optical elements may include refractive lenses. In one or more embodiments, the plurality of optical elements may include diffractive lenses. In one or more embodiments, the curved reflector may include a wavelength selective notch filter.

In another embodiment, a method for displaying virtual content to a user includes: providing one or more light patterns associated with one or more frames of image data in a time-sequential manner; reflecting the one or more light patterns associated with the one or more frames of image data to an exit pupil via a transmissive beam splitter having multiple reflectors to variably direct light toward the exit pupil; and expanding the exit pupil via multiple optical elements coupled to the multiple reflectors of the transmissive beam splitter.

In one or more embodiments, the reflector array may be separate from the optical element. In one or more embodiments, the reflector array comprises a plane mirror. In one or more embodiments, the optical element may be a microlens coupled to the reflector array.

In another embodiment, a system for displaying virtual content to a user includes a light source that multiplexes one or more light patterns associated with one or more frames of image data in a time-sequential manner, a waveguide that receives the one or more light patterns at a first focus level, and a variable focus element (VFE) coupled to the waveguide to place at least a portion of the light patterns at a second focus level.

In one or more embodiments, the VFE is telecentric. In one or more embodiments, the VFE is non-telecentric. In one or more embodiments, the system further includes a compensating lens so that the user's view of the external world is not distorted. In one or more embodiments, multiple frames are presented to the user at a high frequency so that the user perceives the frames as part of a single coherent scene, wherein the VFE changes the focus from a first frame to a second frame. In one or more embodiments, the light source is a scanning light display, and wherein the VFE changes the focus on a line-by-line basis. In one or more embodiments, the light source is a scanning light display, and wherein the VFE changes the focus on a pixel-by-pixel basis.

In one or more embodiments, the VFE is a diffractive lens. In one or more embodiments, the VFE is a refractive lens. In one or more embodiments, the VFE is a reflector. In one or more embodiments, the reflector is opaque. In one or more embodiments, the reflector is partially reflective. In one or more embodiments, the system further includes a visual accommodation module that tracks the visual accommodation of the user's eye, and wherein the VFE changes the focus of the light pattern based at least in part on the visual accommodation of the user's eye.

In yet another embodiment, a system for displaying virtual content to a user includes a light source that multiplexes one or more light patterns associated with one or more frames of image data in a time-sequential manner; a waveguide that receives the one or more light patterns and directs the light patterns to a first focal point; and a variable focus element (VFE) coupled to the waveguide to direct at least a portion of the light patterns to a second focal point, wherein the VFE is integrated within the waveguide.

In another embodiment, a system for displaying virtual content to a user includes: a light source that multiplexes one or more light patterns associated with one or more frames of image data in a time-sequential manner; a waveguide that receives the one or more light patterns and directs the light patterns to a first focal point; and a variable focus element (VFE) coupled to the waveguide to direct at least a portion of the light patterns to a second focal point, wherein the VFE is separate from the waveguide.

On the other hand, a method for displaying virtual content to a user includes: providing one or more light patterns associated with one or more frames of image data; converging the one or more light patterns associated with the one or more frames of image data to a first focus through a waveguide; and changing the first focus of the light through a variable focus element (VFE) to produce a wavefront at a second focus.

In one or more embodiments, the VFE is separate from the waveguide. In one or more embodiments, the VFE is integrated within the waveguide. In one or more embodiments, the one or more frames of image data are provided in a time sequence. In one or more embodiments, the VFE changes the focus of the one or more frames of image data on a frame-by-frame basis. In one or more embodiments, the VFE changes the focus of the one or more frames of image data on a pixel-by-pixel basis. In one or more embodiments, the VFE changes the first focus to produce a wavefront at a third focus, wherein the second focus is different from the third focus. In one or more embodiments, the wavefront at the second focus is perceived by the user as coming from a particular depth plane.

In some embodiments, multiple frames are presented to the user at a high frequency such that the user perceives the frames as part of a single coherent scene, wherein the VFE changes the focus from a first frame to a second frame. In one or more embodiments, the light source is a scanning light display, and wherein the VFE changes the focus line by line.

In another embodiment, a system for displaying virtual content to a user includes: a plurality of waveguides that receive light associated with image data and emit the light toward the user's eyes, wherein the plurality of waveguides are stacked in a direction facing the user's eyes; and a first lens coupled to a first waveguide of the plurality of waveguides to modify the light emitted from the first waveguide so as to transmit light having a first wavefront curvature; and a second lens coupled to a second waveguide of the plurality of waveguides to modify the light emitted from the second waveguide so as to transmit light having a second wavefront curvature, wherein the first lens coupled to the first waveguide and the second lens coupled to the second waveguide are horizontally stacked in a direction facing the user's eyes.

In one or more embodiments, the first wavefront curvature is different from the second wavefront curvature. In one or more embodiments, the system further includes a third waveguide of the plurality of waveguides that transmits collimated light to the user's eye so that the user perceives the image data as coming from an optically infinite plane. In one or more embodiments, the waveguide is configured to emit collimated light to the lens.

In one or more embodiments, the system further includes a compensating lens layer that compensates for the aggregated power of the lenses stacked in a direction facing the user's eyes, wherein the compensating lens layer is stacked farthest from the user's eyes. In one or more embodiments, the waveguide includes a plurality of reflectors that can be configured to reflect light injected into the waveguide toward the user's eyes.

In one or more embodiments, the waveguide is electrically active. In one or more embodiments, the waveguide is switchable. In one or more embodiments, the light having the first wavefront curvature and the light having the second wavefront curvature are transmitted simultaneously. In one or more embodiments, the light having the first wavefront curvature and the light having the second wavefront curvature are transmitted sequentially. In one or more embodiments, the second wavefront curvature corresponds to a margin of the first wavefront curvature, thereby providing a zoom range that enables the user to adapt. In one or more embodiments, the system further includes a visual accommodation module that tracks the visual accommodation of the user's eyes, and wherein the VFE changes the focus of the light pattern based at least in part on the visual accommodation of the user's eyes.

In yet another embodiment, a system for displaying virtual content to a user includes: a light source that multiplexes one or more light patterns associated with one or more frames of image data in a time-sequential manner; a plurality of waveguides that receive the one or more light patterns and focus the light into an exit pupil, wherein the plurality of waveguides are stacked along a Z-axis and away from the user's line of sight; and at least one optical element coupled to the stacked waveguides to change the focus of the light emitted by the plurality of waveguides.

In one or more embodiments, a waveguide among the plurality of waveguides may include a waveguide for distributing projected light across the length of the waveguide, and a lens for altering the light in a manner that produces a wavefront curvature, wherein the produced wavefront curvature corresponds to a focal plane when viewed by the user.

In one or more embodiments, a waveguide in the plurality of waveguides comprises a diffractive optical element (DOE). In one or more embodiments, the DOE can be switched between an on and an off state. In one or more embodiments, the waveguide in the plurality of waveguides comprises a refractive lens. In one or more embodiments, the waveguide in the plurality of waveguides comprises a Fresnel zone plate. In one or more embodiments, the waveguide in the plurality of waveguides comprises a substrate guided optics (SGO) element. In one or more embodiments, the waveguide can be switched between an on and an off state. In one or more embodiments, the waveguide is a static waveguide. In one or more embodiments, the first frame of image data and the second frame of image data are transmitted to the user's eye simultaneously. In one or more embodiments, the first frame of image data and the second frame of image data are transmitted to the user's eye sequentially.

In one or more embodiments, the system further comprises a plurality of corner reflectors that transmit light to the user's eyes, wherein the first waveguide assembly and the second waveguide assembly direct light toward the one or more corner reflectors. In one or more embodiments, the system further comprises a beam distributing waveguide optical element coupled to the waveguide assembly, wherein the beam distributing waveguide optical element can be configured to propagate the projected light across the waveguide assembly such that light injected into the beam distributing waveguide optical element is replicated and injected into the waveguide assembly of the waveguide assembly.

In another embodiment, a system for displaying virtual content to a user includes: an image generation source that provides one or more frames of image data in a time-sequential manner; a light modulator that projects light associated with the one or more frames of image data; and a waveguide assembly that receives the projected light and transmits the light toward the user's eyes, wherein the waveguide assembly includes at least a first waveguide component and a second waveguide component, the first waveguide component being configurable to modify light associated with a first frame of the image data so that the light is perceived as coming from a first focal plane, the second waveguide component being configurable to modify light associated with a second frame of the image data so that the light is perceived as coming from a second focal plane, and wherein the first waveguide component and the second waveguide component are stacked along a Z-axis in front of the user's eyes.

In certain embodiments, the waveguide component of the waveguide assembly includes a waveguide for distributing the projected light across the length of the waveguide, and a lens for modifying the light in a manner that produces a wavefront curvature, wherein the produced wavefront curvature corresponds to a focal plane when viewed by the user. In one or more embodiments, the waveguide component of the waveguide assembly includes a diffractive optical element (DOE).

In one or more embodiments, the DOE can be switched between an on and an off state. In one or more embodiments, the waveguide component of the waveguide assembly includes a refractive lens. In one or more embodiments, the waveguide component of the waveguide assembly includes a Fresnel zone plate. In one or more embodiments, the first frame of image data and the second frame of image data are transmitted to the user's eye simultaneously. In one or more embodiments, the first frame of image data and the second frame of image data are transmitted to the user's eye sequentially.

In one or more embodiments, the system further comprises a plurality of corner reflectors that transmit light to the user's eyes, wherein the first waveguide assembly and the second waveguide assembly direct light toward the one or more corner reflectors. In one or more embodiments, the system further comprises a beam distributing waveguide optical element coupled to the waveguide assembly, wherein the beam distributing waveguide optical element can be configured to propagate the projected light across the waveguide assembly such that light injected into the beam distributing waveguide optical element is replicated and injected into the waveguide assembly of the waveguide assembly.

The waveguide assembly of the waveguide assembly includes a reflector that can be configured to reflect the projection light at a desired angle toward the user's eye. In one or more embodiments, the first waveguide assembly includes a first reflector configured to reflect the projection light at a first angle, and the second waveguide assembly includes a second reflector that reflects the projection light at a second angle. In one or more embodiments, the first reflector is staggered relative to the second reflector to expand the field of view of the user viewing the image.

In one or more embodiments, the reflector of the waveguide assembly is positioned to form a continuous curved reflective surface across the waveguide assembly. In one or more embodiments, the continuous curved reflective surface comprises a parabola. In one or more embodiments, the continuous curved reflective surface comprises an elliptical curve.

In yet another embodiment, a method for displaying virtual content to a user includes transmitting light associated with a first frame of image data to the user through a first waveguide, the light having a first wavefront curvature; and transmitting light associated with a second frame of image data to the user through a second waveguide, the light having a second wavefront curvature, wherein the first waveguide and the second waveguide are stacked along a Z-axis facing an eye of the user.

In one or more embodiments, the first wavefront curvature and the second wavefront curvature are transmitted simultaneously. In one or more embodiments, the first wavefront curvature and the second wavefront curvature are transmitted sequentially. In one or more embodiments, the first and second wavefront curvatures are perceived by the user as first and second depth planes. In one or more embodiments, the first and second waveguides are coupled to one or more optical elements. In one or more embodiments, the method may further include compensating for the effects of the one or more optical elements via a compensating lens.

In one or more embodiments, the method may further include determining an accommodation of an eye of the user; and transmitting light through at least one of the first and second waveguides based at least in part on the determined accommodation.

In another embodiment, a method for displaying virtual content to a user includes: determining visual accommodation of an eye of the user; transmitting light having a first wavefront curvature through a first waveguide in a waveguide stack based at least in part on the determined visual accommodation, wherein the first wavefront curvature corresponds to a focal length of the determined visual accommodation; and transmitting light having a second wavefront curvature through a second waveguide in the waveguide stack, wherein the second wavefront curvature is associated with a predetermined margin of the focal length of the determined visual accommodation.

In one or more embodiments, the margin is a positive margin. In one or more embodiments, the margin is a negative margin. In one or more embodiments, the second waveguide increases the zoom range that the user can adapt to. In one or more embodiments, the first waveguide is coupled to a variable focus element (VFE), wherein the VFE changes the focal point at which the waveguide focuses the light. In one or more embodiments, the focus is changed based at least in part on a determined visual accommodation of the user's eye. In one or more embodiments, the first wavefront curvature and the second wavefront curvature are transmitted simultaneously.

In one or more embodiments, the first and second wavefront curvatures are perceived by the user as first and second depth planes. In one or more embodiments, the waveguide is a diffractive optical element (DOE). In one or more embodiments, the waveguide is a substrate guided optical element (SGO). In one or more embodiments, the first and second waveguides are switchable. In one or more embodiments, the waveguide includes one or more switchable elements.

In yet another embodiment, a system for displaying virtual content to a user includes an image generation source that provides one or more frames of image data in a time-sequential manner; a display assembly that projects light associated with the one or more frames of image data, the display assembly including a first display element and a second display element, the first display element corresponding to a first frame rate and a first bit depth, the second display element corresponding to a second frame rate and a second bit depth; and a variable focus element (VFE) that can be configured to change the focus of the projected light and emit the light to the user's eyes.

In one or more embodiments, the first frame rate is higher than the second frame rate, and the first bit depth is lower than the second bit depth. In one or more embodiments, the first display element is a DLP projection system. In one or more embodiments, the second display element is a liquid crystal display (LCD). In one or more embodiments, the first display element projects light onto a portion of the second display element such that the periphery of the LCD has constant illumination. In one or more embodiments, only light emitted from the first display element is focused by the VFE.

In one or more embodiments, the VFE is optically conjugate with the exit pupil, thereby changing the focus of the projected light without affecting the magnification of the image data. In one or more embodiments, the first display element is a DLP and the second display element is an LCD, and wherein the DLP has a low resolution and the LCD has a high resolution. In one or more embodiments, the intensity of the backlight varies over time to equalize the brightness of the sub-images projected by the first display element, thereby increasing the frame rate of the first display element.

In one or more embodiments, the VFE may be configured to change the focus of the projected light on a frame-by-frame basis. In one or more embodiments, the system further comprises software for compensating for optical magnification associated with operation of the VFE. In one or more embodiments, the image generation source produces multiple segments of a particular image that, when projected together or sequentially, produce a three-dimensional volume of an object. In one or more embodiments, the DLP operates in binary mode. In one or more embodiments, the DLP operates in grayscale mode.

In one or more embodiments, the VFE modifies the projected light such that a first frame is perceived as originating from a first focal plane and a second frame is perceived as originating from a second focal plane, wherein the first focal plane is different from the second focal plane. In one or more embodiments, the focal length associated with the focal plane is fixed. In one or more embodiments, the focal length associated with the focal plane is variable.

In another embodiment, a method for displaying virtual content to a user includes: providing one or more image segments, wherein a first and a second image segment of the one or more image segments represent a three-dimensional volume; projecting light associated with the first image segment through a spatial light modulator; focusing the first image segment to a first focus through a variable focus element (VFE); transmitting the first image segment with the first focus to the user; providing light associated with the second image segment; focusing the second image segment to a second focus through the VFE, wherein the first focus is different from the second focus; and transmitting the second image segment with the second focus to the user.

In one or more embodiments, the method may further include determining an accommodation of an eye of the user, wherein the VFE focuses the projected light based at least in part on the determined accommodation. In one or more embodiments, the image segments are provided in a frame sequence. In one or more embodiments, the first image segment and the second image segment are transmitted simultaneously. In one or more embodiments, the first image segment and the second image segment are transmitted sequentially.

In another embodiment, a method for displaying virtual content to a user includes: combining a first display element and a second display element, the first display element corresponding to a high frame rate and a low bit depth, and the second display element corresponding to a low frame rate and a high bit depth, so that the combined display element corresponds to a high frame rate and a high bit depth; and projecting light associated with one or more frames of image data through the combined display element; and switching the focus of the projected light frame by frame through a variable focus element (VFE) so as to project a first image segment at a first focus and a second image segment at a second focus.

In another embodiment, a system for displaying virtual content to a user includes: a plurality of light guides that receive coherent light associated with one or more frames of image data and produce an aggregate wavefront; a phase modulator that is coupled to one or more of the plurality of light guides to induce a phase delay in light projected by the one or more light guides; and a processor that controls the phase modulator in such a manner that the aggregate wavefront produced by the plurality of light guides changes.

In one or more embodiments, the wavefronts generated by the light guides of the plurality of light guides are spherical wavefronts. In one or more embodiments, the spherical wavefronts generated by at least two light guides constructively interfere with each other. In one or more embodiments, the spherical wavefronts generated by at least two light guides destructively interfere with each other. In one or more embodiments, the aggregate wavefront is substantially a planar wavefront.

The plane wavefront corresponds to a depth plane at optical infinity. In one or more embodiments, the aggregate wavefront is a spherical wavefront. In one or more embodiments, the spherical wavefront corresponds to a depth plane closer than optical infinity. In one or more embodiments, an inverse Fourier transform of the desired beam is injected into a multi-core optical fiber to produce the desired aggregate wavefront.

On the other hand, a system for displaying virtual content to a user includes: an image generation source that provides one or more frames of image data; a multi-core assembly that includes multiple multi-core optical fibers for projecting light associated with the one or more frames of the image data, one of the multiple multi-core optical fibers emitting light through a wavefront so that the multi-core assembly produces an aggregate wavefront of the projected light; and a phase modulator that induces phase delays between the multi-core optical fibers in a manner that changes the aggregate wavefront emitted by the multi-core assembly, thereby changing the focal length of the one or more frames of the image data perceived by the user.

On the other hand, a method for displaying virtual content to a user includes: emitting light through a multi-core optical fiber, the multi-core optical fiber including multiple single-core optical fibers, wherein the single-core optical fibers emit spherical wavefronts; the light emitted through the multiple single-core optical fibers provides a polymeric wavefront; and inducing a phase delay between the single-core optical fibers in the multi-core optical fiber so as to change the polymeric wavefront generated by the multi-core optical fiber at least in part based on the induced phase delay.

In one or more embodiments, the aggregate wavefront is a plane wavefront. In one or more embodiments, the plane wavefront corresponds to optical infinity. In one or more embodiments, the aggregate wavefront is a spherical wavefront. In one or more embodiments, the spherical wavefront corresponds to a depth plane closer than optical infinity. In one or more embodiments, the method further comprises injecting an inverse Fourier transform of a desired wavefront into the multi-core optical fiber such that the aggregate wavefront corresponds to the desired wavefront.

In another embodiment, a system for displaying virtual content to a user includes: an image generation source that provides one or more frames of image data; a multi-core assembly that includes multiple multi-core optical fibers for projecting light associated with the one or more frames of the image data; and an image injector that inputs an image into the multi-core assembly, wherein the input injector can be further configured to input an inverse Fourier transform of a desired wavefront into the multi-core assembly so that the multi-core assembly outputs the Fourier transform by generating light associated with the image data in the desired wavefront, thereby allowing the user to perceive the image data at a desired focal length.

In one or more embodiments, the desired wavefront is associated with a hologram. In one or more embodiments, the inverse Fourier transform is input to modulate the focus of the one or more light beams. In one or more embodiments, the multi-core fiber of the plurality of multi-core fibers is a multimode fiber. In one or more embodiments, the multi-core fiber of the plurality of multi-core fibers is configured to propagate light on multiple paths along the fiber. In one or more embodiments, the multi-core fiber is a single-core fiber. In one or more embodiments, the multi-core fiber is a coaxial core fiber.

In one or more embodiments, the image injector is configured to input a wavelet pattern into the multi-core assembly. In one or more embodiments, the image injector is configured to input Zernike coefficients into the multi-core assembly. In one or more embodiments, the system further comprises an accommodation tracking module that determines accommodation of the user's eye, wherein the image injector is configured to input an inverse Fourier transform of a wavefront corresponding to the determined accommodation of the user's eye.

In yet another embodiment, a method of displaying virtual content to a user includes determining an accommodation of an eye of the user, wherein the determined accommodation is associated with a focal length corresponding to a current focus state of the user; projecting light associated with one or more frames of image data through a waveguide; changing a focus of the projected light based at least in part on the determined accommodation; and transmitting the projected light to the eye of the user so that the light is perceived by the user as being from the focal length corresponding to the current focus state of the user.

In one or more embodiments, the visual accommodation is measured directly. In one or more embodiments, the visual accommodation is measured indirectly. The visual accommodation is measured by an infrared autorefractor. In one or more embodiments, the visual accommodation is measured by decentered photo-optography. In one or more embodiments, the method further includes measuring a convergence level of the user's eyes to estimate the visual accommodation. In one or more embodiments, the method further includes blurring one or more portions of one or more frames of the image data based at least in part on the determined visual accommodation. In one or more embodiments, the focus varies between fixed depth planes. In one or more embodiments, the method further includes compensating a lens to compensate for optical effects of the waveguide.

In one or more embodiments, a method of displaying virtual content to a user includes: determining visual accommodation of an eye of the user, wherein the determined visual accommodation is associated with a focal length corresponding to a current focus state of the user; projecting light associated with one or more frames of image data through a diffractive optical element (DOE); changing a focus of the projected light based at least in part on the determined visual accommodation; and transmitting the projected light to the eye of the user so that the light is perceived by the user as coming from the focal length corresponding to the current focus state of the user.

In another embodiment, a method of displaying virtual content to a user includes: determining visual accommodation of an eye of the user, wherein the determined visual accommodation is associated with a focal length corresponding to a current focus state of the user; projecting light associated with one or more frames of image data through a freeform optical element; changing a focus of the projected light based at least in part on the determined visual accommodation; and transmitting the projected light to the eye of the user so that the light is perceived by the user as being from the focal length corresponding to the current focus state of the user.

On the other hand, a method of displaying virtual content to a user includes: determining visual accommodation of an eye of the user, wherein the determined visual accommodation is associated with a focal length corresponding to a current focus state of the user; projecting light associated with one or more frames of image data; changing a focus of the projected light based at least in part on the determined visual accommodation; and transmitting the projected light to the eye of the user so that the light is perceived by the user as being from the focal length corresponding to the current focus state of the user.

In one or more embodiments, the light is transmitted to the user through a substrate guided optical assembly. In one or more embodiments, the light is transmitted to the user through a free-form optical element. In one or more embodiments, the light is transmitted to the user through a diffractive optical element (DOE). In one or more embodiments, the light is projected through a waveguide stack, a first waveguide in the waveguide stack being configured to output light on a specific wavefront, a second waveguide being configured to output a positive margin wavefront relative to the specific wavefront, and a third waveguide being configured to output a negative margin wavefront relative to the specific wavefront. In one or more embodiments, the method further includes blurring a portion of one or more frames of the image data in such a manner that the portion is out of focus when the projected light is transmitted to the user's eye.

In yet another embodiment, a system for displaying virtual content to a user includes an image generation source that provides one or more frames of image data in a time-sequential manner; a light emitter that provides light associated with the one or more frames of image data; an accommodation tracking module that tracks accommodation of an eye of the user; and a waveguide assembly that changes the focus of the light associated with the one or more frames of image data, wherein different frames of image data are focused differently based at least in part on the tracked accommodation.

On the other hand, a system for displaying virtual content to a user includes: a visual accommodation tracking module that determines the visual accommodation of the user's eyes; an image generation source that provides one or more frames of image data in a time-sequential manner; a light emitter that projects light associated with the one or more frames of image data; a plurality of waveguides that receive light associated with the image data and emit the light to the user's eyes, wherein the plurality of waveguides are stacked in a direction facing the user's eyes; and a variable focus element (VFE) that changes the focus of the emitted light based at least in part on the determined visual accommodation of the user's eyes.

In one or more embodiments, one of the plurality of waveguides is a waveguide element, wherein a first frame of image data emitted from a first waveguide of the plurality of waveguides has a different focus than a second frame of image data emitted from a second waveguide of the plurality of waveguides. In one or more embodiments, the first frame is a first layer of a 3D scene, and the second frame is a second layer of the 3D scene. In one or more embodiments, the system further includes a blur module that blurs a portion of one or more frames of image data in such a manner that the portion is out of focus when viewed by the user.

In one or more embodiments, the VFE is common to the plurality of waveguides. In one or more embodiments, the VFE is associated with one of the plurality of waveguides. In one or more embodiments, the VFE is coupled to one of the plurality of waveguides such that the VFE is interleaved between two of the plurality of waveguides. In one or more embodiments, the VFE is embedded within one of the plurality of waveguides. In one or more embodiments, the VFE is a diffractive optical element. In one or more embodiments, the VFE is a refractive element.

In one or more embodiments, the waveguide is electrically active. In one or more embodiments, one or more of the plurality of waveguides are switched off. In one or more embodiments, one of the plurality of waveguides corresponds to a fixed focal plane. In one or more embodiments, the system further comprises an exit pupil, wherein the exit pupil has a diameter no greater than 0.5 mm. The light emitter is a scanning fiber display. In one or more embodiments, the system further comprises an exit pupil array.

In one or more embodiments, the system further comprises a plurality of light emitters, one light emitter coupled to an exit pupil. In one or more embodiments, the system further comprises an exit pupil expander. In one or more embodiments, the exit pupil can be switched based at least in part on the determined visual accommodation of the user's eye.

In another aspect, a system includes an accommodation tracking module that determines accommodation of an eye of a user; a fiber scanning display that scans a plurality of light beams associated with one or more frames of image data, wherein one of the plurality of light beams is movable; and blur software that renders simulated refractive blur in the one or more frames of the image data based at least in part on the determined accommodation of the eye of the user.

In one or more embodiments, the diameter of the beam is no greater than 2 mm. In one or more embodiments, the diameter of the beam is no greater than 0.5 mm. In one or more embodiments, the scanning beam is replicated to create multiple exit pupils. In one or more embodiments, the scanning beam is replicated to create a larger eye box. In one or more embodiments, the exit pupil is switchable.

In another embodiment, a method for displaying virtual content includes: determining the visual accommodation of a user's eye; scanning a plurality of light beams associated with one or more frames of image data through a fiber optic scanning display, wherein the diameter of the light beams is no greater than 0.5 mm so that the frames of image data appear in focus when viewed by the user; and blurring one or more portions of the frames based at least in part on the determined visual accommodation of the user's eye using blurring software.

In one or more embodiments, multiple exit pupils are created. In one or more embodiments, the light beam is generated by a single core optical fiber. In one or more embodiments, the light beam is replicated to create multiple exit pupils. In one or more embodiments, the exit pupils are switchable.

In another embodiment, a method for displaying virtual content to a user includes determining a position of a pupil of the user relative to a beam of light projectors corresponding to a sub-image in an image to be presented to the user, and driving light corresponding to the sub-image into a portion of the user's pupil based on the determined position of the user's pupil.

In one or more embodiments, the method further includes driving light corresponding to another sub-image of the image to be presented to another portion of the user's pupil via another light projector. In one or more embodiments, the method further includes mapping one or more light projectors of the fiber-optic scanning display to one or more portions of the user's pupil. In one or more embodiments, the mapping is a 1:1 mapping.

In one or more embodiments, the diameter of the light is no greater than 0.5 mm. In one or more embodiments, the beam of light projectors produces a convergent wavefront. In one or more embodiments, the beamlets produced by the beam projectors form a discretized convergent wavefront. In one or more embodiments, the beamlets approach the user's eyes in parallel, and the eyes deflect the beamlets to converge on the same point on the retina. In one or more embodiments, the user's eyes receive a superset of beamlets corresponding to the multiple angles at which they intersect the pupil.

In another embodiment, a system for displaying virtual content to a user includes: a light source that provides light associated with one or more frames of image data; and a light display assembly that receives light associated with the one or more frames of the image data, wherein the light display assembly corresponds to a plurality of proximate exit pupils, and wherein the plurality of exit pupils emit light into the pupils of the user.

In one or more embodiments, the multiple exit pupils are arranged in a hexagonal lattice. In one or more embodiments, the multiple exit pupils are arranged in a quadrilateral lattice. In one or more embodiments, the multiple exit pupils are arranged in a two-dimensional array. In one or more embodiments, the multiple exit pupils are arranged in a three-dimensional array. In one or more embodiments, the multiple exit pupils are arranged in a time-varying array.

In one or more embodiments, a method for displaying virtual content to a user includes: grouping a plurality of light projectors to form an exit pupil; driving a first light pattern into a first portion of the user's pupil through the first exit pupil; and driving a second light pattern into a second portion of the user's pupil through a second exit pupil, wherein the first light pattern and the second light pattern correspond to sub-images of an image to be presented to the user, and wherein the first light pattern is different from the second light pattern. In one or more embodiments, the method further includes creating a discretized aggregate wavefront.

In yet another embodiment, a method for displaying virtual content to a user includes determining a position of a pupil of the user relative to a light display assembly and calculating a focus for directing light toward the pupil based at least in part on a finite eyebox around the determined position of the pupil.

In one or more embodiments, the diameter of the light is no greater than 0.5 mm. In one or more embodiments, the method further comprises creating a discretized aggregate wavefront. In one or more embodiments, the method further comprises aggregating a plurality of discrete adjacent collimated light beams based at least in part on a center of a radius of curvature of a desired aggregate wavefront. In one or more embodiments, the method further comprises determining an accommodation of an eye of the user, wherein the focus is calculated based at least in part on the determined accommodation.

In one or more embodiments, the method further comprises selecting angular trajectories of light in a plurality of beamlets to create an out-of-focus beam. In one or more embodiments, the plurality of beamlets represent pixels of image data to be presented to the user. In one or more embodiments, the beamlets illuminate the eye at a plurality of angles of incidence.

In yet another embodiment, a system for displaying virtual content to a user includes an image generation source that provides one or more portions of an image to be presented to the user; and a plurality of micro-projectors that project light associated with the one or more portions of the image, the micro-projectors being positioned facing a pupil of the user, and wherein one of the plurality of micro-projectors is configured to project a set of light rays representing a portion of the sub-image, the set of light rays being projected onto a portion of the pupil of the user.

In one or more embodiments, a first portion of the user's pupil receives light from a plurality of micro-light projectors. In one or more embodiments, the system further comprises a reflective surface that reflects the light from the plurality of micro-light projectors to one or more portions of the user's pupil. In one or more embodiments, the reflective surface is positioned such that the user can view the real world through the reflective surface. In one or more embodiments, the diameter of the light is no greater than 0.5 mm. In one or more embodiments, the system further comprises creating a discretized aggregate wavefront.

In another embodiment, a system includes: a processor that determines a position of a user's pupil; and a spatial light modulator (SLM) array that projects light associated with one or more frames of image data, wherein the SLM array is positioned based at least in part on the determined position of the user's pupil, and wherein the SLM array produces a light field when viewed by the user.

In another aspect, a system for displaying virtual content to a user includes an image generation source that provides one or more frames of image data; a first spatial light modulator (SLM) that is configured to selectively emit light associated with the one or more frames of image data; a second SLM positioned relative to the first SLM, the second SLM also being configured to selectively emit light associated with the one or more frames of image data; and a processor that controls the first and second SLMs in a manner that produces a light field when the user views the emitted light.

In one or more embodiments, the system further includes a visual accommodation tracking module that determines the visual accommodation of the user's eyes. In one or more embodiments, the SLM is an LCD. In one or more embodiments, the LCD is attenuated. In one or more embodiments, the LCD rotates the polarization of the emitted light. In one or more embodiments, wherein the SLM is a DMD. In one or more embodiments, the DMD is coupled to one or more lenses. In one or more embodiments, the SLM is a MEMS array. In one or more embodiments, the MEMs array includes a sliding MEMS shutter array. In one or more embodiments, the MEMS array is a MEMS array.

In another embodiment, a system for displaying virtual content to a user includes: a plurality of optical fibers that project light associated with one or more frames of image data to be presented to the user, wherein one of the plurality of optical fiber cores is coupled to a lens configured to change a diameter of a light beam projected by the scanning optical fiber, wherein the lens includes a gradient refractive index.

In one or more embodiments, the lens is a GRIN lens. In one or more embodiments, the lens aims the light beam. In one or more embodiments, the system further comprises an actuator coupled to the optical fiber core of the plurality of optical fiber cores to scan the optical fiber. In one or more embodiments, the actuator is a piezoelectric actuator. In one or more embodiments, one end of the optical fiber core is polished at an angle to produce a lens effect. In one or more embodiments, one end of the optical fiber core is melted to produce a lens effect.

In one or more embodiments, a method for displaying virtual content to a user includes: projecting light associated with one or more frames of image data, wherein the light is projected through a plurality of optical fiber cores; altering the light projected through the plurality of optical fiber cores with the aid of a lens, wherein the lens is coupled to the tips of the plurality of optical fiber cores; and transmitting the altered light to the user.

In one or more embodiments, a system for displaying virtual content comprises: a multi-core assembly comprising a plurality of optical fibers to multiplex light associated with one or more frames of image data; and a waveguide that receives the light pattern and emits the light pattern such that a first viewing area receives only light associated with a first portion of the image and a second viewing area receives only light associated with a second portion of the image, wherein the first and second viewing areas are no larger than 0.5 mm. In one or more embodiments, the system further comprises blur software that blurs one or more portions of the frame of the image. In one or more embodiments, the system further comprises a visual accommodation module that determines the visual accommodation of the user's eyes. In one or more embodiments, the waveguide projects light directly to the user's eyes without intermediate viewing optical elements.

In one or more embodiments, a system includes: a multi-core assembly comprising a plurality of optical fibers to multiplex light associated with one or more frames of image data; a waveguide that receives a light pattern and emits the light pattern such that a first viewing area receives only light associated with a first portion of an image and a second viewing area receives only light associated with a second portion of the image, wherein the first and second viewing areas are no larger than 0.5 mm; and an optical assembly coupled to the waveguide to modify the light beams emitted to the first and second viewing areas.

The plurality of optical fibers projects light into a single waveguide array. In one or more embodiments, the multi-core assembly is scanned. In one or more embodiments, a time-varying light field is generated. The optical assembly is a DOE element. In one or more embodiments, the optical assembly is an LC layer.

In one or more embodiments, a method includes: projecting light associated with one or more frames of image data through a multi-core assembly, the multi-core assembly comprising multiple optical fiber cores; and transmitting the projected light through a waveguide so that a first portion of the user's pupil receives light associated with the first portion of the image and a second portion of the user's pupil receives light associated with the second portion of the image.

In one or more embodiments, the diameter of the first and second portions is no greater than 0.5 mm. In one or more embodiments, the plurality of optical fibers projects light into a single waveguide array. In one or more embodiments, the multi-core assembly is scanned. In one or more embodiments, the waveguide comprises a plurality of reflectors. In one or more embodiments, the angle of the reflectors is variable. In one or more embodiments, a set of optical elements alters the light transmitted to the first and second viewing areas. The set of optical elements is a DOE element. The set of optical elements is a free-form optical element. In one or more embodiments, the set of optical elements is an LC layer.

In one aspect, a system comprises: an array of micro-projectors that projects light associated with one or more frames of image data to be presented to a user, wherein the array of micro-projectors is positioned relative to a position of a pupil of the user, and wherein the light is projected into the pupil of the user. In one or more embodiments, the fiber-optic scanning display of claim 407, wherein the first and second light beams overlap. In one or more embodiments, the fiber-optic scanning display of claim 407, wherein the first and second light beams are deflected based at least in part on a critical angle of a polished fiber bundle. The fiber-optic scanning display of claim 407, wherein the polished fiber bundle is used to increase the resolution of the display. In one or more embodiments, wherein the polished fiber bundle is used to generate a light field.

In another embodiment, a system includes: a micro-projector array that projects light associated with one or more frames of image data to be presented to a user, wherein the micro-projector array is positioned relative to the position of the user's pupil and wherein the light is projected into the user's pupil; and an optical element coupled to the micro-projector array to modify the light projected into the user's pupil.

In yet another embodiment, a system includes: a plurality of multi-core optical fibers that emit light beams, the plurality of light beams being coupled together; and a coupling element that bundles the plurality of multi-core optical fibers together, wherein the multi-core optical fiber bundle is polished at a critical angle relative to a longitudinal axis of the optical fibers such that a first light beam emitted from a first optical fiber in the bundle has a first path length and a second light beam emitted from a second optical fiber in the bundle has a second path length, and wherein the first path length is different from the second path length such that the first light beam is out of phase with respect to the second light beam.

In one or more embodiments, the first and second light beams overlap. In one or more embodiments, the first and second light beams are deflected based at least in part on a critical angle of the polished fiber bundle. In one or more embodiments, the polished fiber bundle is used to increase the resolution of the display. In one or more embodiments, the polished fiber bundle is used to generate a light field.

In another embodiment, a system for displaying virtual content to a user includes: an image generation source that provides one or more frames of image data; a plurality of optical fiber cores that emit light beams associated with the one or more frames of image data; and an optical element coupled to the plurality of optical fiber cores to receive collimated light from the optical fiber cores and transmit the light beams to the user's eyes, wherein the light beams are transmitted to the user's eyes at multiple angles such that a first light beam is transmitted to a portion of the user's eye at a first angle and a second light beam is transmitted to the same portion of the user's eye at a second angle, wherein the first angle is different from the second angle. In one or more embodiments, the optical element is a waveguide. In one or more embodiments, the system further includes a phase modulator that modulates the light transmission through the optical fiber.

In yet another embodiment, a method includes providing one or more frames of image data; emitting light beams associated with the one or more frames of image data through a plurality of optical fiber cores; and transmitting the light beams to the user's eyes at a plurality of angles.

In one or more embodiments, the method further comprises modulating the phase delay of the plurality of optical fibers. In one or more embodiments, the method further comprises coupling an optical element to the plurality of optical fibers. In one or more embodiments, the optical element is a light guide. The optical element is a free-form optical element. In one or more embodiments, the optical element is a DOE. In one or more embodiments, the optical element is a waveguide.

In one or more embodiments, a virtual reality display system includes: a plurality of optical fiber cores that generate light beams associated with one or more images to be presented to a user; and a plurality of phase modulators coupled to the plurality of optical fiber cores to modulate the light beams, wherein the plurality of phase modulators modulate the light in a manner that affects a wavefront generated as a result of the plurality of light beams.

In one or more embodiments, one or more optical fiber cores are skewed at one or more angles. In one or more embodiments, one of the plurality of optical fibers is coupled to a GRIN lens. In one or more embodiments, the plurality of optical fibers are physically actuated to scan the optical fibers.

On the other hand, a method includes providing one or more frames of image data to be presented to a user; projecting light associated with the one or more frames of image data through a plurality of optical fiber cores; and modulating the light projected by the plurality of optical fiber cores through a plurality of phase modulators in a manner that affects an aggregate wavefront produced by the plurality of optical fiber cores.

In one or more embodiments, light projected by the one or more optical fibers is deflected at one or more angles. In one or more embodiments, the one or more optical fibers are coupled to a GRIN lens. In one or more embodiments, the method further comprises scanning the light beam, wherein the plurality of optical fibers are physically actuated to scan the optical fibers.

In another aspect, a system for displaying virtual content includes an optical fiber array that emits light beams associated with an image to be presented to a user; and a lens coupled to the optical fiber array to deflect multiple light beams output by the optical fiber array through a single node, wherein the lens is physically attached to the optical fiber such that movement of the optical fiber causes the lens to move and wherein the single node is scanned.

In one or more embodiments, the light beams output by the fiber array represent pixels of an image to be presented to the user. In one or more embodiments, the lens is a GRIN lens. In one or more embodiments, the fiber array is used to display a light field. In one or more embodiments, another set of light beams output by another fiber array represents another pixel of an image to be presented to the user. In one or more embodiments, a plurality of fiber arrays are combined to represent pixels of an image to be presented to the user. In one or more embodiments, the fiber array is configured to transmit the light beams to a predetermined portion of the user's pupil. In one or more embodiments, the output light beams diverge. In one or more embodiments, the output light beams converge.

In one or more embodiments, the numerical aperture of the output beam is increased relative to the beams emitted by the individual optical fibers. In one or more embodiments, the increased numerical aperture enables higher resolution. In one or more embodiments, the optical fiber array is beveled in a manner that allows multiple focal lengths of the optical beams to be delivered to the user's eye by causing a first optical beam to have a different path length than a second optical beam to have a different path length than a second optical beam to have a different path length.

In another aspect, a system for displaying virtual content to a user includes an array of optical fiber cores that projects light associated with one or more frames of image data, wherein one or more optical fiber cores in the array of optical fiber cores are polished at an angle to deflect the projected light, and wherein the polishing angle results in a path length difference between first and second optical fiber cores in the array of optical fiber cores relative to an optical element; and an optical scanner that receives the deflected light beam and scans the deflected light beam along at least one axis.

On the other hand, a system for providing at least one of a virtual or augmented reality experience to a user includes: a frame; an array of micro-projectors, which is carried by the frame and can be placed in front of at least one eye of the user when the frame is worn by the user; and a local controller, which is communicatively coupled to the micro-projector array to provide image information to the micro-projectors, the local controller including at least one processor, and at least one non-transitory processor-readable medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable medium storing at least one of processor-executable instructions or data, which, when executed by the at least one processor, causes the at least one processor to perform at least one of data processing, caching and storage, and provide the image information to the micro-projectors to produce at least one of a virtual or augmented reality visual experience for the user.

In one or more embodiments, the system further comprises at least one reflector supported by the frame and positioned and oriented to direct light from the micro-light projector toward at least one eye of the user when the frame is worn by the user. In one or more embodiments, the micro-light projector comprises a corresponding display of a plurality of scanning fiber displays. In one or more embodiments, each of the scanning fiber displays has a corresponding collimating lens at a distal end thereof. In one or more embodiments, the corresponding collimating lens is a gradient index (GRIN) lens.

In one or more embodiments, the respective collimating lenses are curved lenses. In one or more embodiments, the respective collimating lenses are fused to the ends of the respective scanning fiber displays. In one or more embodiments, the scanning fiber displays have respective diffractive lenses located at their ends. In one or more embodiments, each of the scanning fiber displays has respective diffusers located at their ends.

In one or more embodiments, the diffuser is etched into the respective end. In one or more embodiments, each of the scanning fiber displays has a respective lens located at its end, the lens extending a sufficient distance from the end to freely vibrate in response to an excitation. In one or more embodiments, each of the scanning fiber displays has a respective reflector located at its end, the reflector extending a sufficient distance from the end to freely vibrate in response to an excitation. In one or more embodiments, the scanning fiber displays each include a respective single-mode optical fiber.

In one or more embodiments, the scanning fiber displays each include a corresponding mechanical transducer coupled to move at least one end of the single-mode optical fiber. In one or more embodiments, the corresponding mechanical transducer is a piezoelectric actuator. In one or more embodiments, each single-mode optical fiber has an end, and the end has a hemispherical lens shape. In one or more embodiments, each single-mode optical fiber has an end, and a refractive lens is attached to the end.

In one or more embodiments, the system further comprises a transparent support substrate that holds the plurality of single-mode optical fibers together. In one or more embodiments, the transparent support substrate has a refractive index that at least approximately matches the refractive index of the single-mode optical fiber cladding. In one or more embodiments, the transparent support substrate holds the plurality of single-mode optical fiber cores, each of which is angled relative to a common point.

In one or more embodiments, the system further comprises at least one mechanical transducer coupled to move the plurality of single-mode optical fiber cores in unison. In one or more embodiments, the at least one mechanical transducer vibrates the plurality of single-mode optical fiber cores at a mechanical resonant frequency of a single-mode optical fiber core partially cantilevered from the transparent support substrate. In one or more embodiments, the micro-light projector comprises a respective waveguide of a plurality of planar waveguides, a portion of each of the planar waveguides extending cantilevered from the support substrate. In one or more embodiments, the system further comprises at least one mechanical transducer coupled to move the plurality of planar waveguides in unison.

In one or more embodiments, the at least one mechanical transducer vibrates the support substrate at a mechanical resonant frequency of the planar waveguide. In one or more embodiments, the micro-projector comprises a corresponding piezoelectric actuator of a plurality of piezoelectric actuators coupled to move a corresponding one of the planar waveguides relative to the support substrate. In one or more embodiments, each of the planar waveguides defines a total internal reflection path along a corresponding length of the planar waveguide, and the planar waveguide comprises a corresponding one of a plurality of electrically switchable diffractive optical elements (DOEs) operable to propagate light outside of the corresponding total internal reflection path. In one or more embodiments, the micro-projector array comprises an array of optical fibers, each optical fiber having an end and at least one bevel. In one or more embodiments, the at least one bevel is located at the end, and the end is a polished end.

In one or more embodiments, each of the optical fibers has a reflective surface located at its respective end. In one or more embodiments, the ends have an output edge located at the ends, the output edge forming a defined critical angle with the longitudinal axis of the respective optical fiber. In one or more embodiments, the defined critical angle is approximately 45 degrees with the longitudinal axis of the respective optical fiber. In one or more embodiments, the system further includes a focusing lens positioned in the optical path of light emitted from the ends of the optical fiber cores to receive multiple beams of the light that are out of phase with each other. In one or more embodiments, the system further includes at least one transducer coupled to move at least one of the optical fiber cores in an X-Y Cartesian coordinate system and to move the light emitted by the at least one optical fiber in an X-Z Cartesian coordinate system. In one or more embodiments, the at least one transducer is a first piezoelectric actuator that resonates the cantilevered portion of the optical fiber in a direction perpendicular to the direction in which the cantilevered portion extends.

In one or more embodiments, the optical fiber comprises a filament of an optical fiber. In one or more embodiments, the at least one transducer is a second piezoelectric actuator that moves at least the cantilever portion of the optical fiber core in a direction parallel to the direction in which the cantilever portion extends. In one or more embodiments, the micro-projector comprises at least one uniaxial mirror operable to provide slow scanning along the longitudinal axis of at least one of the optical fiber cores. In one or more embodiments, the optical fiber array comprises a multi-core optical fiber. In one or more embodiments, the multi-core optical fiber comprises a plurality of sparsely positioned clusters within a single conduit, the clusters being approximately seven, each cluster comprising three optical fibers, each optical fiber carrying a corresponding one of three different colors of light.

In one or more embodiments, the multi-core optical fiber includes a plurality of sparsely positioned clusters within a single conduit, the clusters being approximately nineteen, each cluster including three optical fiber cores, each optical fiber carrying a respective one of three different colors of light to produce three overlapping points of three different colors. In one or more embodiments, the multi-core optical fiber includes at least one cluster within a single conduit, the cluster including at least three optical fiber cores, each of the optical fiber cores carrying at least two different colors of light.

In one or more embodiments, the multi-core optical fiber comprises at least one cluster within a single conduit, the at least one cluster comprising four optical fibers, each optical fiber carrying a corresponding one of four different colors of light, wherein one of the four colors is infrared or near-infrared. In one or more embodiments, the multi-core optical fiber comprises a plurality of cores bundled together, and further comprises at least one transducer coupled to move the cores in a sparse spiral pattern. In one or more embodiments, the at least one bevel is spaced inwardly from the end. In one or more embodiments, the at least one bevel is polished.

In one or more embodiments, the system further includes at least one transducer coupled to move at least one of the optical fiber cores in an X-Y Cartesian coordinate system and to move light emitted by the at least one optical fiber in an X-Z Cartesian coordinate system.

In one or more embodiments, the system further comprises a focusing lens positioned in the optical path of light emitted from the hypotenuse of the optical fiber core to receive multiple beams of the light that are out of phase with each other. In one or more embodiments, the system further comprises a laser; and at least one phase modulator optically coupling the output of the laser to the multiple cores of the multi-core optical fiber to achieve mutual coherence.

In one or more embodiments, the system further includes a microlens array optically coupled upstream of the input end of a corresponding core among the multiple cores of the multi-core optical fiber; and a prism array optically coupled between a plurality of collimating lenses and the input end of the core of the multi-core optical fiber to deflect light from the microlens array to the core of the multi-core optical fiber.

In one or more embodiments, the system further includes a microlens array optically coupled upstream of the input end of a corresponding core among the multiple cores of the multi-core optical fiber; and a shared focusing prism optically coupled between the microlens and the input end of the core of the multi-core optical fiber to deflect light from the microlens array to the core of the multi-core optical fiber.

In one or more embodiments, the micro-projector array further comprises at least one reflector operable to generate a scan pattern and optically coupled to the optical fiber array. In one or more embodiments, the at least one reflector is operable to generate at least one of a raster scan pattern, a Lissajous scan pattern, or a spiral scan pattern of a multi-focus light beam. In one or more embodiments, each core in the multi-core optical fiber addresses a corresponding portion of an image plane that does not overlap. In one or more embodiments, each core in the multi-core optical fiber addresses a corresponding portion of an image plane that significantly overlaps.

In another embodiment, a system for displaying virtual content includes an image source that provides one or more frames of image data to be presented to a user; a fiber-scanning display comprising a plurality of optical fibers to project light associated with the one or more frames of image data, wherein the plurality of optical fibers are scanned using an actuator; and a processor that controls the fiber-scanning display in such a manner that a light field is presented to the user.

In one or more embodiments, the actuator is shared among all optical fibers of the fiber-optic scanning display. In one or more embodiments, each optical fiber has its own individual actuator. In one or more embodiments, the plurality of optical fibers are mechanically coupled via a lattice so that the plurality of optical fibers move together. In one or more embodiments, the lattice is a graphene plane. In one or more embodiments, the lattice is a lightweight strut.

In another embodiment, a system for providing at least one of a virtual or augmented reality experience to a user includes: a frame; a display system carried by the frame and capable of being positioned in front of at least one eye of the user when the frame is worn by the user; and a local controller communicatively coupled to the display system to provide image information to the display system, the local controller including at least one processor and at least one non-transitory processor-readable medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable medium storing at least one of processor-executable instructions or data that, when executed by the at least one processor, cause the at least one processor to perform at least one of data processing, caching, and storage, and provide the image information to the display to produce at least one of a virtual or augmented reality visual experience for the user.

In one or more embodiments, the display includes at least one wedge-shaped waveguide having at least two planar surfaces opposing each other across the thickness of the first wedge-shaped waveguide and a length along which light entering the wedge-shaped waveguide at a defined angle through an entrance portion of the wedge-shaped waveguide propagates via total internal reflection, the thickness of the wedge-shaped waveguide varying linearly along the length of the wedge-shaped waveguide. In one or more embodiments, the wedge-shaped waveguide provides dual-mode total internal reflection.

In one or more embodiments, the system further comprises at least two light projectors optically coupled to the wedge-shaped waveguide at respective different locations along the entrance portion of the wedge-shaped waveguide. In one or more embodiments, the system further comprises a first linear array of a plurality of light projectors optically coupled to the wedge-shaped waveguide at respective different locations along the entrance portion of the wedge-shaped waveguide.

In one or more embodiments, the light projectors in the first linear array of the plurality of light projectors are scanning fiber displays. In one or more embodiments, the system further comprises a stack of a plurality of spatial light modulators optically coupled to the wedge-shaped waveguide along an entrance portion of the wedge-shaped waveguide. In one or more embodiments, the system further comprises a multi-core optical fiber optically coupled to the wedge-shaped waveguide at one or more locations along the entrance portion of the wedge-shaped waveguide.

In one or more embodiments, the light projectors in the first linear array of light projectors are optically coupled to the wedge-shaped waveguide to inject light into the wedge-shaped waveguide at a first angle, further comprising a second linear array of multiple light projectors optically coupled to the wedge-shaped waveguide at corresponding different positions along the entrance portion of the wedge-shaped waveguide, wherein the light projectors in the second linear array of light projectors are optically coupled to the wedge-shaped waveguide to inject light into the wedge-shaped waveguide at a second angle, which is different from the first angle.

In one or more embodiments, the entrance portion is a longitudinal end of the wedge-shaped waveguide. In one or more embodiments, the entrance portion is a lateral side of the wedge-shaped waveguide. In one or more embodiments, the entrance portion is one of the planar surfaces of the wedge-shaped waveguide. In one or more embodiments, the system further includes at least one optical component optically coupled to the light projector and altering the angle of light received from the light projector so as to optically couple the light into the wedge-shaped waveguide at an angle that achieves total internal reflection of the light within the wedge-shaped waveguide.

On the other hand, a system for displaying virtual content to a user includes: an array of micro-projectors that projects light beams associated with one or more frames of image data to be presented to the user, wherein the micro-projectors can be configured to be movable relative to one or more micro-projectors in the array of micro-projectors; a frame that houses the array of micro-projectors; and a processor that is operatively coupled to one or more micro-projectors in the array of micro-projectors to control the one or more light beams emitted from the one or more micro-projectors in a manner that modulates the position of the one or more micro-projectors relative to the array of micro-projectors, thereby enabling a light field image to be transmitted to the user.

In one or more embodiments, the micro-light projectors in the array of micro-light projectors are coupled to a lens. In one or more embodiments, the array of micro-light projectors is arranged in a manner based on a desired resolution of an image to be presented to the user. In one or more embodiments, the array of micro-light projectors is arranged in a manner based on a desired field of view. In one or more embodiments, the light beams in multiple micro-light projectors overlap. In one or more embodiments, the system further includes an actuator, wherein the actuator is coupled to one or more micro-light projectors, and wherein the actuator can be configured to move the one or more micro-light projectors.

In one or more embodiments, the actuator is coupled to a plurality of micro-light projectors. In one or more embodiments, the actuator is coupled to a single micro-light projector. In one or more embodiments, a micro-light projector in the array of micro-light projectors is mechanically coupled to the dot matrix.

In yet another embodiment, a contact lens that contacts the cornea of an eye of a user of a virtual or augmented reality display includes a partially hemispherical substrate and a selective filter. In one or more embodiments, the selective filter is configured to selectively allow a light beam to pass through the user's eye. In one or more embodiments, the selective filter is a notch filter. In one or more embodiments, the notch filter substantially blocks wavelengths of approximately 450 nm (blue peak) and substantially passes other wavelengths in the visible portion of the electromagnetic spectrum. In one or more embodiments, the notch filter substantially blocks wavelengths of approximately 530 nm (green) and substantially passes other wavelengths in the visible portion of the electromagnetic spectrum. In one or more embodiments, the notch filter substantially blocks wavelengths of approximately 650 nm and substantially passes other wavelengths in the visible portion of the electromagnetic spectrum.

In one or more embodiments, the notch filter comprises a multilayer dielectric material carried by the substrate. In one or more embodiments, the filter has a pinhole opening having a diameter less than 1.5 mm. In one or more embodiments, the pinhole opening allows light beams of multiple wavelengths to pass through. In one or more embodiments, the size of the pinhole varies based at least in part on a desired depth of focus of the display. In one or more embodiments, the contact lens further comprises a plurality of operating modes. In one or more embodiments, the contact lens further comprises a multi-depth of focus display configuration for the virtual content.

In one or more embodiments, the contact lens further comprises an accommodation tracking module that determines accommodation of the user's eye. In one or more embodiments, a depth of focus of a particular displayed object changes based at least in part on the determined accommodation. In one or more embodiments, an image is relayed via a waveguide, the relayed image being associated with a particular depth of focus.

In another embodiment, a method for displaying virtual content to a user includes providing one or more frames of image data to be presented to the user; projecting light associated with the one or more frames of image data; and receiving the projected light through a local hemispherical substrate coupled to the user's pupil and selectively filtering out the light beam to the user's pupil.

In another embodiment, a system for displaying virtual content to a user includes: a light projection system that projects light associated with one or more frames of image data into the user's eyes, the light projection system being configured to project light corresponding to a plurality of pixels associated with the image data; and a processor that modulates the depth of focus of the plurality of pixels displayed to the user.

In one or more embodiments, the depth of focus is spatially modulated. In one or more embodiments, the depth of focus is temporally modulated. In one or more embodiments, the system further comprises an image generation source that provides one or more frames of the image data in a time-sequential manner. In one or more embodiments, the depth of focus is modulated on a frame-by-frame basis. In one or more embodiments, the light projection system comprises a plurality of optical fibers, and wherein the depth of focus is modulated across the plurality of optical fibers such that a portion of the optical fibers is associated with a first depth of focus and another portion of the optical fibers is associated with a second depth of focus, wherein the first depth of focus is different from the second depth of focus.

In one or more embodiments, a first display object of a particular frame is displayed at a first focal depth and a second display object of the particular frame is displayed at a second focal depth, wherein the first focal depth is different from the second focal depth. In one or more embodiments, a first pixel of a particular frame is associated with a first focal depth and a second pixel of the particular frame is associated with a second focal depth, wherein the first focal depth is different from the second focal depth. In one or more embodiments, the system further includes a visual accommodation tracking module that determines visual accommodation of the user's eye, wherein the focal depth is modulated at least in part based on the determined visual accommodation.

In one or more embodiments, a lighting pattern associated with the lighting system dynamically changes in response to the determined visual accommodation. In one or more embodiments, the pattern is a scanning pattern of a plurality of optical fibers. In one or more embodiments, the system further comprises a blur module that blurs one or more portions of the image data, wherein the blur is generated to smooth a transition between a first scanning pattern and a second scanning pattern, or to smooth a transition from a first resolution scanning pitch to a second resolution scanning pitch.

In another embodiment, a system for displaying virtual content to a user includes: a light projection system that projects light associated with one or more frames of image data into the user's eyes, the light projection system being configured to project light corresponding to a plurality of pixels associated with the image data; and a processor that modulates a size of the plurality of pixels displayed to the user.

In one or more embodiments, the light projection system is a fiber optic scanning display. In one or more embodiments, the projected light is displayed via a scanning pattern. In one or more embodiments, the processor modulates the size of a particular pixel based at least in part on the type of scanning pattern. In one or more embodiments, the size of one or more pixels can be modulated based in part on the distance between scan lines of the scanning pattern. In one or more embodiments, the size of a first pixel is different from the size of a second pixel in the same frame.

On the other hand, a method for displaying virtual content to a user includes: projecting light associated with one or more frames of image data, wherein one or more beams of the projected light correspond to one or more pixels, wherein the projected light is scanned by an optical fiber display; and modulating the size of the one or more pixels displayed to the user.

In one or more embodiments, the size of the particular pixel varies based at least in part on a scan pattern of the fiber-optic scanning display. In one or more embodiments, the size of the one or more pixels is modulated based at least in part on a distance between scan lines of the scan pattern. In one or more embodiments, the size of the one or more pixels is variable.

In yet another embodiment, a system for displaying virtual content to a user includes: a display system that transmits light associated with one or more frames of image data, wherein the display system includes a plurality of pixels, wherein the display system scans the light with a variable line spacing; a blur module that variably blurs one or more of the plurality of pixels to change the size of the one or more pixels; and a processor that controls the blur module in a manner that changes the pixel size based at least in part on the line spacing of the display system. In one or more embodiments, the display system is a fiber scanning system. In one or more embodiments, the pixel size is enlarged. In one or more embodiments, the pixel size is reduced. In one or more embodiments, the line spacing is sparse. In one or more embodiments, the line spacing is dense.

On the other hand, a method for displaying virtual content to a user includes projecting light associated with one or more frames of image data to be presented to the user; and selectively attenuating at least a portion of the projected light based at least in part on characteristics of the image data; and transmitting the attenuated light beam to the user's eyes.

In one or more embodiments, the light beam is selectively attenuated based at least in part on an angle of incidence of the light beam. In one or more embodiments, different portions of the frame are attenuated to different amounts. In one or more embodiments, the depth of focus of the attenuated light beam is variable.

In one or more embodiments, a system for displaying virtual content to a user includes: an image generation source that provides one or more frames of image data; a stack of two or more spatial light modulators (SLMs) that are positioned so that the stack transmits light associated with the one or more frames of image data to the user, wherein the SLMs spatially attenuate light from an external environment; and a processor that controls the SLM stack in a manner that modulates the angle of a light beam as it passes through one or more cells of the SLMs.

In one or more embodiments, the system further comprises a set of optical display elements, wherein the set of optical display elements is positioned between the user's eyes and the external environment. The SLMs in the SLM stack are cholesteric LCDs. In one or more embodiments, at least one of the SLMs is a cholesteric LCD. In one or more embodiments, the SLM stack is positioned such that the user can view the external world through the SLM stack, wherein the SLMs are at least translucent.

In one or more embodiments, the spatial light modulator array comprises at least one of a plurality of liquid crystal arrays, a plurality of digital mirror device elements of a digital light processing system, a plurality of microelectromechanical systems (MEMS) arrays, or a plurality of MEMS shutters. In one or more embodiments, the system further comprises a light shield comprising at least one optical component, and wherein the processor controls the at least one optical component of the light shield to produce a dark field representation of a black virtual object.

On the other hand, a system for displaying virtual content includes: a spatial light modulator array, which is configured to generate a light pattern, and wherein the spatial light modulator array includes at least two modulators; and a processor that controls the spatial modulator array in a manner such that the at least two spatial modulators form moiré fringes, wherein the moiré fringes are periodic spatial patterns that attenuate light during a time period different from a time period during which the light pattern is formed on the at least two spatial light modulators.

In one or more embodiments, the spatial light modulator array includes at least two spatial light modulator arrays, the at least two spatial light modulators being optically coupled to each other and controlling the passage of light via a moiré effect. In one or more embodiments, the at least two spatial light modulator arrays each carry a corresponding attenuation pattern. In one or more embodiments, the at least two spatial light modulator arrays each carry a corresponding fine-pitch sinusoidal wave pattern printed, etched, or otherwise inscribed thereon or therein. In one or more embodiments, the at least two spatial light modulator arrays are registered with each other. In one or more embodiments, the at least two spatial light modulator arrays each carry a corresponding attenuation pattern.

In yet another embodiment, a system for displaying virtual content to a user includes a light source that provides light associated with one or more frames of image data, wherein the light source is a spatial light modulator; and a pinhole array positioned relative to the spatial light modulator such that a pinhole in the pinhole array receives light from multiple elements of the spatial light modulator, and wherein a first light beam passing through the pinhole corresponds to a different angle than a second light beam passing through the pinhole, and wherein the elements of the spatial light modulator selectively attenuate light.

In one or more embodiments, an external environment is viewed through the pinhole array and the SLM, and wherein a light beam is selectively attenuated based at least in part on an angle of incidence of the light beam. In one or more embodiments, light from different portions of a field of view is selectively attenuated. In one or more embodiments, the system further comprises a selective attenuation layer selectively operable to attenuate transmission of light therethrough, the selective attenuation layer being optically coupled in series with the pinhole layer.

In one or more embodiments, the selective attenuation layer comprises a liquid crystal array, a digital light projection system, or a spatial light modulator array carrying a corresponding attenuation pattern. In one or more embodiments, the pinhole array is positioned approximately 30 mm from the cornea of the user's eye, and the selective attenuation plate is located on the side of the eye opposite to the pinhole array. In one or more embodiments, the pinhole array comprises a plurality of pinholes, and wherein the processing controls the SLM in such a manner that light is attenuated according to the angle at which the light beam passes through the plurality of pinholes, thereby generating a converged light field. In one or more embodiments, the converged light field results in occlusion at a desired focal length.

In another embodiment, a system includes: a light source that provides light associated with one or more frames of image data, wherein the light source is a spatial light modulator; and a lens array that is positioned relative to the spatial light modulator such that a lens in the lens array receives light from multiple elements of the spatial light modulator, and wherein a first light beam received by the lens corresponds to a different angle than a second light beam received by the lens, and wherein the elements of the spatial light modulator selectively attenuate light.

In one or more embodiments, an external environment is viewed through the lens array and the SLM, and wherein a light beam is selectively attenuated based at least in part on an angle of incidence of the light beam. In one or more embodiments, light from different portions of the field of view is selectively attenuated. In one or more embodiments, the lens array includes a plurality of lenses, and wherein the processing controls the SLM in a manner such that light is attenuated based on the angles at which the plurality of lenses receive the light beams, thereby producing a converged light field. In one or more embodiments, the converged light field results in obscuration at a desired focal length.

In another embodiment, a system for displaying virtual content to a user includes: a light projector that projects light associated with one or more frames of image data; at least one polarization-sensitive layer that receives the light and rotates the polarization of the light; and a polarization modulator array that modulates the polarization of the polarization-sensitive layer, wherein the state of cells in the array determines how much light passes through the polarization-sensitive layer. In one or more embodiments, the system is positioned in a near-eye configuration. In one or more embodiments, the polarization modulator is a liquid crystal array.

In one or more embodiments, the system further comprises a parallax barrier that offsets the polarizer so that different exit pupils have different paths through the polarizer. In one or more embodiments, the polarizer is an X-pole polarizer. In one or more embodiments, the polarizer is a multipole polarizer. In one or more embodiments, the polarizer is a patterned polarizer. In one or more embodiments, the light interacts with one or more MEMS arrays.

In one or more embodiments, the system further includes an SLM that projects light, wherein the SLM is positioned between one or more optical elements, wherein the optical elements correspond to a zero-magnification telescope. In one or more embodiments, the user views the external environment through the zero-magnification telescope. In one or more embodiments, at least one SLM is positioned at an image plane within the zero-magnification telescope. In one or more embodiments, the system further includes a DMD, wherein the DMD corresponds to the transparent substrate.

In one or more embodiments, the system further comprises a light shield comprising at least one optical component, and wherein the processor controls the at least one optical component of the light shield to produce a dark field representation of a black virtual object. In one or more embodiments, the system further comprises one or more LCDs, wherein the one or more LCDs selectively attenuate a light beam. In one or more embodiments, the system further comprises one or more LCDs, wherein the one or more LCDs function as polarization rotators. In one or more embodiments, the light shield is a shutter-type MEMS device.

In one or more embodiments, the shutter MEMS device is opaque, and wherein the shutter MEMS device changes the incident angle pixel by pixel. In one or more embodiments, the light shielding plate is a sliding plate MEMS device, wherein the sliding plate MEMS device slides back and forth to change the shielding area.

In another embodiment, a method for displaying virtual content includes: projecting light associated with one or more frames of image data; rotating the polarization of the light through a polarization-sensitive layer at a substrate that receives the projected light; and modulating the polarization of the light to selectively attenuate light passing through the polarization layer.

In one or more embodiments, the polarization modulator is a liquid crystal array. In one or more embodiments, the method further comprises creating a parallax barrier that offsets the polarizer so that different exit pupils have different paths through the polarizer. In one or more embodiments, the polarizer is an X-pole polarizer. In one or more embodiments, the polarizer is a multi-pole polarizer. In one or more embodiments, the polarizer is a patterned polarizer.

In another embodiment, a system for displaying virtual content includes: a light source that provides light associated with one or more frames of image data, wherein the light source is a spatial light modulator; an array of microelectromechanical (MEMS) shutters, wherein the MEMS shutters are housed within a substantially transparent substrate, and wherein the MEMS shutters are configurable to change the angle at which light is transmitted to pixels, and wherein the angle at which light is transmitted to a first pixel of the user is different from the angle at which light is transmitted to a second pixel of the user.

In one or more embodiments, the at least one optical component comprises a first micro-electromechanical system (MEMS) shutter array. In one or more embodiments, the MEMS shutter array comprises a plurality of substantially opaque shutters carried by an optically transparent substrate. In one or more embodiments, the micro-electromechanical system (MEMS) shutter array has a shutter pitch that is sufficiently fine to selectively block light on a pixel basis. In one or more embodiments, the system further comprises at least one optical component of the light shield, the at least one optical component comprising a second MEMS shutter array, the second MEMS shutter array being in a stacked configuration with the first MEMS shutter array.

In one or more embodiments, the MEMS shutter array comprises a plurality of polarization shutters carried by an optically transparent substrate, wherein the respective polarization state of each of the shutters is selectively controllable. In one or more embodiments, the shutters in the first and second MEMS plate arrays are polarizers. In one or more embodiments, at least one optical component of the light shield comprises a first micro-electromechanical system (MEMS) plate array mounted for movement in a frame.

In one or more embodiments, the plates in the first MEMS plate array are slidably mounted to move in a frame. In one or more embodiments, the plates in the first MEMS plate array are rotatably mounted to move in a frame. In one or more embodiments, the plates in the first MEMS plate array are simultaneously translatably and rotatably mounted to move in a frame. In one or more embodiments, the plates are movable to generate moiré fringes. In one or more embodiments, the at least one optical component of the light shield further includes a second MEMS plate array mounted to move in a frame, the second array being in a stacked configuration with the first array. The plates in the first and second MEMS plate arrays are polarizers. In one or more embodiments, the at least one optical component of the light shield includes an array of reflectors.

In another embodiment, a system includes at least one waveguide that receives light from an external environment and directs the light to one or more spatial light modulators, wherein the one or more spatial light modulators selectively attenuate the light received in different portions of the user's field of view. In one or more embodiments, the at least one waveguide includes first and second waveguides, and wherein the second waveguide is configured to transmit light emitted from the SLM to the user's eye.

In another embodiment, a method includes receiving light from an external environment; directing the light to a selective attenuator; and selectively attenuating, by the selective attenuator, the received light in different portions of a field of view of the user.

In one or more embodiments, the at least one waveguide comprises a first and a second waveguide, and wherein the second waveguide is configured to transmit light emitted from the SLM to the user's eye. In one or more embodiments, the selective attenuator is a spatial light modulator. In one or more embodiments, the spatial light modulator is a DMD array. In one or more embodiments, the light is directed to the one or more spatial light modulators via one or more waveguides. In one or more embodiments, the method further comprises recoupling the light back into the waveguide, thereby causing the light to be emitted partially toward the user's eye. In one or more embodiments, the waveguide is oriented substantially perpendicular to the selective attenuator.

In another embodiment, a system for displaying virtual content to a user includes: a light source that provides light associated with one or more frames of image data, wherein the light source includes a plurality of micro-light projectors; and a waveguide configured to receive light from the plurality of micro-light projectors and emit the light to the user's eyes.

In one or more embodiments, the micro-light projectors are placed in a linear array. In one or more embodiments, the micro-light projectors are placed in one edge of the waveguide. In one or more embodiments, the micro-light projectors are placed in multiple edges of the waveguide. In one or more embodiments, the micro-light projectors are placed in a two-dimensional array. In one or more embodiments, the micro-light projectors are placed in a three-dimensional array. In one or more embodiments, the micro-light projectors are placed at multiple edges of the substrate. In one or more embodiments, the micro-light projectors are placed at multiple angles.

In another embodiment, a system for displaying virtual content includes: an image generation source that provides one or more frames of image data, wherein the image data includes one or more virtual objects to be presented to a user; and a rendering engine that renders the one or more virtual objects in a manner such that the user perceives a halo around the one or more virtual objects.

In one or more embodiments, the system further comprises a light attenuator, wherein the light attenuator balances the light intensity of the light ring across the user's field of view.

In another embodiment, a method for displaying virtual content includes: providing one or more frames of image data, wherein the image data includes one or more virtual objects to be presented to a user; and rendering the one or more virtual objects in a manner such that the user perceives a halo around the one or more virtual objects, thereby making it easier for the user to view the virtual objects, wherein the virtual objects are black virtual objects.

In one or more embodiments, the method further includes selectively attenuating light received from the external environment via a light attenuator, wherein the light attenuator balances the light intensity of the light ring across the field of view of the user.

In another embodiment, a system for displaying virtual content includes: a camera system that captures a view of a real environment; a see-through optical system that displays one or more virtual objects superimposed on the view of the real environment, wherein the captured view is used to render one or more virtual objects to be presented to the user; and a light intensity module that modulates the light intensity of the view of the real environment based at least in part on an association between one or more real objects and the one or more virtual objects so that black virtual objects are visible compared to the one or more real objects.

In one or more embodiments, the captured view is used to generate a halo around one or more virtual objects, wherein the halo fades across space. In one or more embodiments, the system further includes a light attenuator, wherein the light attenuator balances the light intensity of the halo across the user's field of view.

In yet another embodiment, a method of driving an augmented reality display system includes rendering a first virtual object at a location in a user's field of view; and rendering a visual focus in the user's field of view that is at least spatially close to the rendered first virtual object substantially simultaneously with the rendering of the first virtual object.

In one or more embodiments, rendering the visual emphasis includes rendering the visual emphasis with an intensity gradient. In one or more embodiments, rendering the visual emphasis includes rendering the visual emphasis with a blur proximate a perimeter of the visual emphasis.

In one or more embodiments, rendering the visual emphasis at least spatially proximate to the rendered first virtual object includes rendering a halo visual effect spatially proximate to the rendered first virtual object. In one or more embodiments, rendering the halo visual effect spatially proximate to the rendered first virtual object includes rendering the halo visual effect so as to be brighter than the rendered first virtual object.

In one or more embodiments, rendering the halo visual effect so as to be brighter than the rendered first virtual object is in response to determining that the rendered first virtual object is darker than a dark threshold. In one or more embodiments, rendering the halo visual effect includes rendering the halo visual effect in a focal plane that is separated from the rendered first virtual object in the perceived three-dimensional space. In one or more embodiments, rendering the halo visual effect includes rendering the halo visual effect with an intensity gradient. In one or more embodiments, rendering the halo visual effect includes rendering the halo visual effect with an intensity gradient that matches a dark halo caused by a rendered occlusion applied to the first virtual object to compensate for a dark field effect of the occlusion.

In one or more embodiments, rendering the halo visual effect includes rendering the halo visual effect with a blur proximate a perimeter of the halo visual effect. In one or more embodiments, the rendered first visual object has a non-circular perimeter, and the rendered halo visual effect is consistent with the non-circular perimeter. In one or more embodiments, rendering the visual emphasis at least spatially proximate to the rendered first virtual object includes rendering the visual effect in a focal plane that is separated from the rendered first virtual object in the perceived three-dimensional space. In one or more embodiments, rendering the visual effect in a focal plane that is separated from the rendered first virtual object in the perceived three-dimensional space includes rendering the visual effect in a focal plane that is relatively far away from the user with respect to the focal plane in which the rendered first virtual object is rendered.

In another embodiment, a system for displaying virtual content includes: an image generation source that provides one or more frames of image data to be presented to a user, wherein the one or more frames of image data include at least one black virtual object; and a rendering engine that renders the one or more frames of image data, and wherein the rendering engine renders the black virtual object as a blue virtual object so that the black virtual object is visible to the user.

In one or more embodiments, rendering a first virtual object at a location in a user's field of view includes first changing any black tones of the first virtual object to a dark blue.

In yet another embodiment, a system for emitting a light beam to display virtual content includes: at least one waveguide having a first end and a second end spaced apart from the first end across a length of the at least one waveguide, along which light entering the corresponding waveguide at a defined angle propagates via total internal reflection; at least one edge reflector positioned at least proximate the first end of the at least one waveguide for coupling light back to the first end of the at least one waveguide by optical reflection; and at least one edge reflector positioned at least proximate the second end of the at least one waveguide for coupling light back to the second end of the at least one waveguide by optical reflection.

In one or more embodiments, the at least one waveguide has a plurality of lateral reflective and/or diffractive surfaces within the waveguide that redirect at least a portion of the light laterally to the exterior of the waveguide. In one or more embodiments, the lateral reflective and/or diffractive surfaces are low diffraction efficiency diffractive optical elements (DOEs). In one or more embodiments, the at least one edge reflector positioned at least proximate the first end of the at least one waveguide comprises a plurality of reflectors positioned at least proximate the first end of the at least one waveguide.

In one or more embodiments, the at least one edge reflector positioned at least proximate the second end of the at least one waveguide comprises a plurality of reflectors positioned at least proximate the second end of the at least one waveguide.In one or more embodiments, the at least one waveguide is a single waveguide.

In yet another embodiment, a system for emitting a light beam to display virtual content includes: a waveguide assembly comprising: a plurality of planar waveguides, each of the planar waveguides having at least two flat, parallel major faces opposing each other across the thickness of the planar waveguide; a first end; a second end opposing the first end across the length of the waveguide, along which light entering the corresponding waveguide at a defined angle propagates via total internal reflection; and two flat major sides opposing each other across the width of the waveguide, the plurality of planar waveguides having a stacked configuration along a first axis parallel to the thickness direction of the planar waveguide and along a second axis parallel to the width of the planar waveguide, thereby forming a three-dimensional planar waveguide array.

In one or more embodiments, there are at least three planar waveguides stacked along the direction of the first axis. In one or more embodiments, there are at least three planar waveguides stacked along the direction of the second axis. In one or more embodiments, there are at least three planar waveguides stacked along the direction of the second axis. In one or more embodiments, successive planar waveguides stacked along the first axis are immediately adjacent to each other, and successive planar waveguides stacked along the second axis are immediately adjacent to each other. In one or more embodiments, the waveguide assembly further comprises a plurality of reflective layers carried by at least one plane of at least one of the planar waveguides.

The reflective layer comprises a fully reflective metal coating. In one or more embodiments, the reflective layer comprises a wavelength-specific reflector. In one or more embodiments, the reflective layer separates the planar waveguides in each successive pair of planar waveguides along at least one of the first or second axis. In one or more embodiments, the reflective layer separates the planar waveguides in each successive pair of planar waveguides along both the first and second axes.

In one or more embodiments, each of the plurality of planar waveguides includes a plurality of lateral reflective and/or diffractive facets that redirect at least a portion of light received by the corresponding planar waveguide laterally toward the exterior of the planar waveguide. In one or more embodiments, the lateral reflective and/or diffractive facets include a diffractive optical element sandwiched within the corresponding planar waveguide between the principal faces of the corresponding planar waveguide. In one or more embodiments, the diffractive optical element is selectively operable to vary focal length.

In one or more embodiments, the first axis is an arcuate axis, and at least one of the major sides of each of the planar waveguides in at least one group of the waveguide assemblies is oriented to focus on a single line that is parallel to the length of the planar waveguide.

In one or more embodiments, a system for displaying virtual content to a user includes: a light projector that projects light associated with one or more frames of image data, wherein the light projector is a fiber optic scanning display; and a waveguide assembly that variably deflects light to the user's eyes, wherein the waveguide is concavely curved toward the eyes.

In one or more embodiments, the curved waveguide extends the field of view. In one or more embodiments, the curved waveguide effectively directs light toward the user's eyes. In one or more embodiments, the curved waveguide includes a time-varying grating, thereby creating an axis along which the light is scanned for the fiber-optic scanning display.

In another embodiment, a system for displaying virtual content to a user includes: a transmissive beam splitter substrate having an inlet for receiving light and a plurality of internal reflective or diffractive surfaces at an angle relative to the inlet, the internal reflective or diffractive surfaces redirecting at least a portion of the light received at the inlet laterally to the exterior of the transmissive beam splitter substrate and toward the user's eyes, wherein the plurality of internal reflective or diffractive surfaces include a plurality of lateral reflective and/or diffractive surfaces spaced apart along a longitudinal axis of the transmissive beam splitter substrate, each of the lateral reflective and/or diffractive surfaces being at an angle or capable of being at an angle relative to the inlet to redirect at least a portion of the light received at the inlet laterally to the exterior of the transmissive beam splitter substrate and toward the user's eyes. a light emitting system that emits light to the transmissive beam splitter; and a local controller that is communicatively coupled to a display system to provide image information to the display system, the local controller comprising at least one processor and at least one non-transitory processor-readable medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable medium storing at least one of processor-executable instructions or data that, when executed by the at least one processor, cause the at least one processor to perform at least one of data processing, caching, and storage, and provide the image information to the display to produce at least one of a virtual or augmented reality visual experience for the user.

In one or more embodiments, the lateral reflective and/or diffractive surface comprises at least one diffractive optical element (DOE), wherein a collimated light beam entering the beam splitter at a plurality of defined angles is totally internally reflected along its length and intersects the DOE at one or more locations. In one or more embodiments, the at least one diffractive optical element (DOE) comprises a first grating. In one or more embodiments, the first grating is a first Bragg grating.

In one or more embodiments, the DOE includes a second grating, the first grating being located on a first plane and the second grating being located on a second plane, the second plane being spaced apart from the first plane such that the first and second gratings interact to produce a moiré beat waveform. In one or more embodiments, the first grating has a first pitch and the second grating has a second pitch, the first pitch being the same as the second pitch. In one or more embodiments, the first grating has a first pitch and the second grating has a second pitch, the first pitch being different from the second pitch. In one or more embodiments, the first grating pitch is controllable to change the first grating pitch over time. In one or more embodiments, the first grating comprises an elastic material and undergoes mechanical deformation.

In one or more embodiments, the first grating is supported by a mechanically deformable elastic material. In one or more embodiments, the first grating pitch can be controlled to change the first grating pitch over time. In one or more embodiments, the second grating pitch can be controlled to change the second grating pitch over time. In one or more embodiments, the first grating is an electro-active grating having at least an on state and an off state. In one or more embodiments, the first grating comprises a polymer dispersed liquid crystal, wherein a plurality of liquid crystal droplets of the polymer dispersed liquid crystal are controllably activated to change the refractive index of the first grating.

In one or more embodiments, the first grating is a time-varying grating, wherein the first grating is a time-varying grating and the local controller controls at least the first grating to expand the field of view of the display. In one or more embodiments, the first grating is a time-varying grating and the local controller employs time-varying control of at least the first grating to correct chromatic aberration. In one or more embodiments, the local controller drives at least the first grating to change the placement of a red sub-pixel in a pixel of the image relative to at least one of a blue or green sub-pixel in a corresponding pixel of the image. In one or more embodiments, the local controller drives at least the first grating to shift the exit pattern laterally to fill gaps in the exit image.

In one or more embodiments, at least one DOE element has a first circularly symmetric term. In one or more embodiments, at least one DOE element has a first linear term, the first linear term being added to the first circularly symmetric term. In one or more embodiments, the circularly symmetric term is controllable. In one or more embodiments, at least one DOE element has another first circularly symmetric term. In one or more embodiments, the at least one diffractive optical element (DOE) comprises a first DOE. In one or more embodiments, the first DOE is a circular DOE.

In one or more embodiments, the circular DOE is a time-varying DOE. In one or more embodiments, the circular DOE is layered relative to the waveguide to provide focus modulation. In one or more embodiments, the diffraction pattern of the circular DOE is a static pattern. In one or more embodiments, the diffraction pattern of the circular DOE is a dynamic pattern. In one or more embodiments, the system includes additional circular DOEs, wherein the additional circular DOEs are positioned relative to the circular DOE to achieve multiple focus levels with a small number of switchable DOEs.

In one or more embodiments, the system further comprises a matrix of switchable DOE elements. In one or more embodiments, the matrix is used to expand the field of view. In one or more embodiments, the matrix is used to expand the size of the exit pupil.

In one or more embodiments, a system for displaying virtual content to a user includes: a light projection system that projects a light beam associated with one or more frames of image data; and a diffractive optical element (DOE) that receives the projected light beam and transmits the light beam at a desired focus, wherein the DOE is a circular DOE.

In one or more embodiments, the DOE is stretchable along a single axis to adjust the angle of the linear DOE term. In one or more embodiments, the DOE comprises a membrane and at least one transducer operable to selectively vibrate the membrane in the Z-axis through an oscillating action to provide Z-axis control and focus change over time. In one or more embodiments, the DOE is embedded in a stretchable medium so that the spacing of the DOE can be adjusted by physically stretching the medium. In one or more embodiments, the DOE is bi-directionally stretched, and wherein the stretching of the DOE affects the focal length of the DOE. In one or more embodiments, the system of claim 762 further comprises a plurality of circular DOEs, wherein the DOEs are stacked along the Z-axis. The circular DOEs are layered in front of the waveguide for focus modulation. In one or more embodiments, the system of claim 768, wherein the DOE is a static DOE.

In one or more embodiments, a system for displaying virtual content to a user includes: a light projection system that projects a light beam associated with one or more frames of image data; a first waveguide that does not have any diffractive optical element (DOE), the first waveguide propagating light received by the first waveguide at multiple defined angles via total internal reflection along at least a portion of the length of the first waveguide, and providing the light from the first waveguide to the outside as collimated light; a second waveguide that has at least a first circularly symmetric diffractive optical element (DOE), the second waveguide being optically coupled to receive the collimated light from the first waveguide; and a processor that controls the grating of the DOE.

In one or more embodiments, the first DOE is selectively controllable. In one or more embodiments, the display includes a plurality of additional DOEs in addition to the first DOE, the DOEs being arranged in a stacked configuration. In one or more embodiments, each of the plurality of additional DOEs is selectively controllable. In one or more embodiments, a local controller controls the first DOE and the plurality of additional DOEs to dynamically modulate the focus of light passing through the display. In one or more embodiments, the processor selectively switches the first DOE and the plurality of additional DOEs, respectively, to achieve a plurality of focus levels, the number of achievable focus levels being greater than the total number of DOEs in the stack.

In one or more embodiments, each of the DOEs in the stack has a corresponding optical power, and the optical powers of the DOEs are summed with each other in a statically controllable manner. In one or more embodiments, the corresponding optical power of at least one of the DOEs in the stack is twice the corresponding optical power of at least another of the DOEs in the stack. In one or more embodiments, the processor selectively switches the first DOE and the plurality of additional DOEs to modulate the respective linear terms and radial terms of the DOEs over time. In one or more embodiments, the processor selectively switches the first DOE and the plurality of additional DOEs based on a frame sequence.

The DOE stack includes a polymer-dispersed liquid crystal element stack. In one or more embodiments, in the absence of an applied voltage, the refractive index of a host medium matches the refractive index of a set of dispersed molecules of the polymer-dispersed liquid crystal element. In one or more embodiments, the polymer-dispersed liquid crystal element includes lithium niobate molecules and a plurality of transparent indium tin oxide layer electrodes located on either side of the host medium, wherein the dispersed lithium niobate molecules controllably alter the refractive index and functionally form a diffraction pattern within the host medium.

In another embodiment, a method for displaying virtual content includes: projecting light associated with one or more frames of image data to a user; receiving the light at a first waveguide, the first waveguide being devoid of any diffractive optical elements and propagating the light by internal reflection; receiving collimated light at a second waveguide having at least a first circularly symmetric diffractive optical element (DOE), the second waveguide being optically coupled to receive the collimated light from the first waveguide, wherein a grating of the circularly symmetric DOE is variable, and wherein the first waveguide and the second waveguide are assembled in a DOE stack.

In one or more embodiments, an optical element for displaying virtual content to a user includes: at least one diffractive optical element (DOE) positioned to receive light, the at least one DOE including a first array of multiple individually addressable segments, each of the individually addressable sub-segments having at least one electrode, each of the individually addressable sub-segments responding to at least one corresponding signal received via the corresponding at least one electrode to selectively switch between at least a first state and a second state, the second state being different from the first state.

In one or more embodiments, the field of view is extended by multiplexing adjacent addressable sub-segments. In one or more embodiments, the first state is an on state and the second state is an off state. In one or more embodiments, each of the individually addressable sub-segments has a corresponding set of at least two indium tin oxide electrodes. In one or more embodiments, the first array of the plurality of individually addressable segments of the at least one DOE is a one-dimensional array. In one or more embodiments, the first array of the plurality of individually addressable segments of the at least one DOE is a two-dimensional array. In one or more embodiments, the first array of the plurality of individually addressable segments is segments of a first DOE located on a first planar layer.

In one or more embodiments, the at least one DOE includes at least a second DOE, the second DOE including a second array of multiple individually addressable segments, each of the individually addressable sub-segments having at least one electrode, each of the individually addressable sub-segments responding to at least one corresponding signal received via the corresponding at least one electrode to selectively switch between at least a first state and a second state, the second state being different from the first state, the second DOE array being located on a second planar layer, and the second planar layer being in a stacked configuration with the first planar layer.

In one or more embodiments, the at least one DOE includes at least a third DOE, the third DOE including a third array of multiple individually addressable segments, each of the individually addressable sub-segments having at least one electrode, each of the individually addressable sub-segments responding to at least one corresponding signal received via the corresponding at least one electrode to selectively switch between at least a first state and a second state, the second state being different from the first state, the third DOE array being located on a third planar layer, the third planar layer being in a stacked configuration with the first and second planar layers.

In one or more embodiments, the first array of individually addressable sub-segments is embedded within a single planar waveguide. In one or more embodiments, the local controller controls the individually addressable sub-segments to selectively emit collimated light from the planar waveguide at a first time and to emit divergent light from the planar waveguide at a second time, the second time being different from the first time. In one or more embodiments, the local controller controls the individually addressable sub-segments to selectively emit light from the planar waveguide in a first direction at a first time and to emit light from the planar waveguide in a second direction at the first time, the second direction being different from the first direction.

In one or more embodiments, the local controller controls the individually addressable sub-segments to selectively scan light across a direction over time. In one or more embodiments, the local controller controls the individually addressable sub-segments to selectively focus light over time. In one or more embodiments, the local controller controls the individually addressable sub-segments to selectively change the field of view of the exit pupil over time.

In one or more embodiments, a system includes: a first free-form reflective and lens optical component that increases the field of view size of a defined set of optical parameters, the first free-form reflective and lens optical component including: a first curved surface, a second curved surface, and a third curved surface, the first curved surface being at least partially capable of optical transmission and refraction and imparting a focusing change to light received by the first free-form reflective and lens optical component via the first curved surface, the second curved surface at least partially reflecting light received by the second curved surface from the first curved surface to the third curved surface, and transmitting light received by the second curved surface from the third curved surface, the third curved surface at least partially reflecting light out of the first free-form reflective and lens optical component via the second curved surface.

In one or more embodiments, the first curved surface of the first free-form reflective and lens optical assembly is a corresponding free-form curved surface. In one or more embodiments, the first curved surface of the first free-form reflective and lens optical assembly adds anastigmatism to the light. In one or more embodiments, the third curved surface of the first free-form reflective and lens optical assembly adds an opposite anastigmatism to offset the anastigmatism added by the first curved surface of the first free-form reflective and lens optical assembly. In one or more embodiments, the second curved surface of the first free-form reflective and lens optical assembly is a corresponding free-form curved surface. In one or more embodiments, the second curved surface of the first free-form reflective and lens optical assembly reflects light of a defined angle reflected by total internal reflection to the third curved surface.

In one or more embodiments, a system includes: a fiber-optic scanning display that projects light associated with one or more frames of image data, wherein the fiber-optic scanning display is configured to transmit the light to a first free-form optical element; and a first free-form reflective and lens optical component that increases the field of view size of a defined set of optical parameters, the first free-form reflective and lens optical component including: a first curved surface, a second curved surface, and a third curved surface, the first curved surface being at least partially capable of optical transmission and refraction and imparting a focus change to light received by the first free-form reflective and lens optical component via the first curved surface, the second curved surface at least partially reflecting light received by the second curved surface from the first curved surface to the third curved surface, and transmitting light received by the second curved surface from the third curved surface, the third curved surface at least partially reflecting light out of the first free-form reflective and lens optical component via the second curved surface.

In one or more embodiments, the freeform optical element is a TIR freeform optical element. In one or more embodiments, the freeform optical element has a non-uniform thickness. In one or more embodiments, the freeform optical element is a wedge-shaped optical element. In one or more embodiments, the freeform optical element is a tapered element. In one or more embodiments, the freeform optical element corresponds to an arbitrary curve.

In one or more embodiments, a system includes: an image generation source that provides one or more frames of image data to be presented to a user; a display system that provides light associated with the one or more frames of image data; and a freeform optical element that modifies the provided light and transmits the light to the user, wherein the freeform optical element includes a reflective coating, wherein the display system is configured to illuminate the freeform optical element with light so that a wavelength of the light matches a corresponding wavelength of the reflective coating.

In one or more embodiments, the one or more free-form optical elements are tiled relative to each other. In one or more embodiments, the one or more free-form optical elements are tiled along the Z-axis.

In one or more embodiments, a system includes: an image generation source that provides one or more frames of image data to be presented to a user; a display system that provides light associated with the one or more frames of image data, wherein the display system includes a plurality of microdisplays; and a freeform optical element that modifies the provided light and transmits the light to the user.

One or more freeform optical elements are tiled relative to each other. In one or more embodiments, the light projected by the plurality of microdisplays increases the field of view. In one or more embodiments, the freeform optical elements are configured such that a particular freeform optical element transmits only one color. In one or more embodiments, the tiled freeform shapes are star-shaped. In one or more embodiments, the tiled freeform optical elements increase the size of the exit pupil. In one or more embodiments, the system further includes another freeform optical element, wherein the freeform optical elements are stacked together in a manner to produce a uniform material thickness. In one or more embodiments, the system further includes another freeform optical element, wherein the another optical element is configured to capture light corresponding to an external environment.

In one or more embodiments, the system further comprises a DMD, wherein the DMD is configured to shade one or more pixels. In one or more embodiments, the system further comprises one or more LCDs. In one or more embodiments, the system further comprises a contact lens substrate, wherein the freeform optical element is coupled to the contact lens substrate. In one or more embodiments, the plurality of microdisplays provide an aggregated array of small exit pupils that is functionally equivalent to a large exit pupil.

In one or more embodiments, at least one image source includes: at least a first monochromatic image source providing light of a first color; at least a second monochromatic image source providing light of a second color, the second color being different from the first color; and at least a third monochromatic image source providing light of a third color, the third color being different from the first and second colors. In one or more embodiments, the at least first monochromatic image source includes a first scanning fiber subset, the at least second monochromatic image source includes a second scanning fiber subset, and the at least third monochromatic image source includes a third scanning fiber subset.

The system further includes a light shield positioned in an optical path between the first free-form reflective and lens optical assembly and the at least one reflector, the light shield operable to selectively shield light on a pixel-by-pixel basis. The first free-form reflective and lens optical assembly forms at least a portion of a contact lens. In one or more embodiments, the system further includes a compensating lens optically coupled to a portion of the first free-form reflective and lens optical assembly.

In one or more embodiments, a system includes: a first free-form reflective and lens optical component that increases the field of view size of a defined set of optical parameters, the first free-form reflective and lens optical component including: a first surface, a second surface, and a third surface, the first surface at least partially capable of optically transmitting light received by the first free-form reflective and lens optical component via the first surface, the second surface being a curved surface that at least partially reflects light received by the second surface from the first surface to the third surface and transmits light received by the second surface from the curved surface, the third surface being a curved surface that at least partially reflects light out of the first free-form reflective and lens optical component via the second surface; and a second free-form reflective and lens optical component, the second free-form reflective and lens optical component including: a first surface, a second surface, and a third surface, the first surface at least partially capable of optically transmitting light received by the first free-form reflective and lens optical component via the first surface, the second surface being a curved surface The first surface of the second free-form reflective and lens optical component is at least partially capable of optically transmitting light received by the second free-form reflective and lens optical component via the first surface, the second surface of the second free-form reflective and lens optical component is a curved surface, which at least partially reflects the light received by the second surface from the first surface of the second free-form reflective and lens optical component to the third surface of the second free-form reflective and lens optical component, and transmits the light received by the second surface from the third surface of the second free-form reflective and lens optical component, the third surface of the second free-form reflective and lens optical component is a curved surface, which at least partially reflects the light out of the second free-form reflective and lens optical component via the second surface, wherein the first and second free-form reflective and lens optical components are in a stacked configuration oriented oppositely along the Z axis.

In one or more embodiments, the second surface of the second free-form reflective and lens optical component is adjacent to the third surface of the first free-form reflective and lens optical component. In one or more embodiments, the second surface of the second free-form reflective and lens optical component is concave, the third surface of the first free-form reflective and lens optical component is convex, and the third surface of the first free-form reflective and lens optical component closely receives the second surface of the second free-form reflective and lens optical component. In one or more embodiments, the first surface of the first free-form reflective and lens optical component is planar, and the first surface of the second free-form reflective and lens optical component is planar, and further includes at least a first light projector optically coupled to the first free-form reflective and lens optical component via the first surface of the first free-form reflective and lens optical component; and at least a second light projector optically coupled to the second free-form reflective and lens optical component via the first surface of the second free-form reflective and lens optical component.

In one or more embodiments, the system further comprises at least one wavelength selective material carried by at least one of the first or second free-form reflective and lens optical components. In one or more embodiments, the system further comprises at least a first wavelength selective material carried by the first free-form reflective and lens optical component; and at least a second wavelength selective material carried by the second free-form reflective and lens optical component, the first wavelength selective material selecting a first set of wavelengths and the second wavelength selective material selecting a second set of wavelengths, the second set of wavelengths being different from the first set of wavelengths.

In one or more embodiments, the system further includes at least a first polarizer carried by the first free-form reflective and lens optical component; at least a second polarizer carried by the second free-form reflective and lens optical component, the first polarizer having a polarization orientation different from that of the second polarizer.

In one or more embodiments, the optical fiber cores are located in the same optical fiber cladding. In one or more embodiments, the optical fiber cores are located in different optical fiber claddings. In one or more embodiments, the visual accommodation module indirectly tracks visual accommodation by tracking the vergence or gaze of the user's eyes. In one or more embodiments, the partially reflective mirrors have a relatively high reflectance for the polarization of light provided by the light source and a relatively low reflectance for other polarization states of light provided by the external world. In one or more embodiments, a plurality of partially reflective mirrors include a dielectric coating. In one or more embodiments, the plurality of reflective mirrors have a relatively high reflectance for waveguides of the wavelength of light provided by the light source and a relatively low reflectance for other waveguides of light provided by the external world. In one or more embodiments, the VFE is a deformable mirror whose surface shape is capable of changing over time. In one or more embodiments, the VFE is an electrostatically actuated membrane mirror, wherein the waveguide or additional transparent layer includes one or more substantially transparent electrodes, and wherein a voltage applied to the one or more electrodes deforms the membrane mirror electrostatically. In one or more embodiments, the light source is a scanning light display, and wherein the VFE varies focus by line segment. In one or more embodiments, the waveguide includes an exit pupil expansion function, wherein an input light ray is split and decoupled into multiple light rays emitted from the waveguide at multiple locations. In one or more embodiments, before the waveguide receives the one or more light patterns, the processor scales the image data according to and compensates for changes in optical image magnification so that the image magnification appears to remain substantially constant while the focus level is adjusted.

In another embodiment, a system for displaying virtual content includes: an image generation source that provides one or more frames of image data in a time-sequential manner; a display assembly that projects light associated with the one or more frames of the image data; the display assembly includes a first display element corresponding to a first frame rate and a first bit depth and a second display element corresponding to a second frame rate and a second bit depth; and a variable focus element (VFE) that can be configured to change the focus of the projected light and emit the light to the user's eyes.

In yet another embodiment, a system for displaying virtual content includes an optical fiber array that emits light beams associated with an image to be presented to a user, and a lens coupled to the optical fiber array to deflect multiple light beams output by the optical fiber array through a single node, wherein the lens is physically attached to the optical fiber such that movement of the optical fiber causes movement of the lens, and wherein the single node is scanned.

In another embodiment, a virtual reality display system includes: a plurality of optical fiber cores that generate light beams associated with one or more images to be presented to a user; and a plurality of phase modulators coupled to the plurality of optical fiber cores to modulate the light beams, wherein the plurality of phase modulators modulate the light in a manner that affects a wavefront generated as a result of the plurality of light beams.

In one embodiment, a system for displaying virtual content includes: a light projection system that projects light associated with one or more frames of image data into a user's eyes, the light projection system being configured to project light corresponding to a plurality of pixels associated with the image data; and a processor that modulates the depth of focus of the plurality of pixels displayed to the user.

In one embodiment, a system for displaying virtual content to a user includes: an image generation source that provides one or more frames of image data; a multi-core assembly that includes multiple multi-core optical fibers for projecting light associated with the one or more frames of the image data, one of the multiple multi-core optical fibers emitting light through a wavefront so that the multi-core assembly produces an aggregate wavefront of the projected light; and a phase modulator that induces a phase delay between the multi-core optical fibers in a manner that changes the aggregate wavefront emitted by the multi-core assembly, thereby changing the focal length of the one or more frames of the image data perceived by the user.

In another embodiment, a system for displaying virtual content to a user includes: an array of micro-projectors that projects light beams associated with one or more frames of image data to be presented to the user, wherein the micro-projectors can be configured to be movable relative to one or more micro-projectors in the array of micro-projectors; a frame that houses the array of micro-projectors; and a processor that is operatively coupled to one or more micro-projectors in the array of micro-projectors to control the one or more light beams emitted from the one or more micro-projectors in a manner that modulates the position of the one or more micro-projectors relative to the array of micro-projectors, thereby enabling a light field image to be transmitted to the user.

Additional and other objects, features, and advantages of the present invention are described in the detailed description, accompanying drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 illustrates augmented reality (AR) viewed by a user through a wearable AR user device in an exemplary embodiment;

2A-2E illustrate various embodiments of wearable AR devices;

FIG3 shows a cross-sectional view of a human eye in an exemplary embodiment;

4A-4D illustrate one or more embodiments of various internal processing components of a wearable AR device;

5A-5H illustrate an embodiment of transmitting focused light to a user through a transmissive beam splitter substrate;

6A and 6B illustrate an embodiment of coupling a lens element to the transmissive beam splitter substrate of FIGs. 5A-5H;

7A and 7B illustrate an embodiment using one or more waveguides to transmit light to a user;

8A-8Q illustrate embodiments of a diffractive optical element (DOE);

9A and 9B illustrate wavefronts generated from a light projector according to an exemplary embodiment;

FIG10 illustrates one embodiment of a stacked configuration of multiple transmissive beam splitter substrates coupled to optical elements according to an exemplary embodiment;

11A-11C illustrate a set of light beams projected into a user's pupil according to an exemplary embodiment;

12A and 12B illustrate configurations of micro-light projector arrays according to exemplary embodiments;

13A-13M illustrate embodiments of coupling a micro-light projector to an optical element according to exemplary embodiments;

14A-14F illustrate embodiments of a spatial light modulator coupled to an optical element according to example embodiments;

15A-15C illustrate the use of a wedge-shaped waveguide with multiple light sources according to an exemplary embodiment;

16A-16O illustrate embodiments of coupling an optical element to an optical fiber according to exemplary embodiments;

FIG17 illustrates a notch filter according to an exemplary embodiment;

FIG18 illustrates a spiral pattern of a fiber scanning display according to an exemplary embodiment;

19A-19N illustrate the masking effect when presenting a dark field to a user according to an exemplary embodiment;

20A-20O illustrate various waveguide assembly embodiments according to exemplary embodiments;

21A-21N illustrate various configurations of DOEs coupled to other optical elements according to example embodiments;

22A-22Y illustrate various configurations of freeform optical elements according to example embodiments.

DETAILED DESCRIPTION

Referring to Figures 4A-4D , which illustrate some common component options, the detailed description following Figures 4A-4D describes various systems, subsystems, and components that achieve the goal of providing a high-quality and comfortable display system for VR and/or AR.

As shown in FIG4A , an AR system user (60) is shown wearing a frame (64) structure coupled to a display system (62) located in front of the user's eyes. In the illustrated configuration, a speaker (66) is coupled to the frame (64) and located near the user's ear canal (in one embodiment, another speaker (not shown) is located near the user's other ear canal to provide stereo/shapeable sound control). The display (62) is operatively coupled (68) (e.g., via a wired or wireless connection) to a local processing and data module (70), which can be mounted in a variety of configurations, such as, for example, fixedly attached to the frame (64) as shown in the embodiment of FIG4B , fixedly attached to a helmet or hat (80); embedded in headphones as shown in the embodiment of FIG4C , removably attached to the torso (82) of the user (60) via a backpack-type configuration; or removably attached to the hips (84) of the user (60) via a belt-type connection as shown in the embodiment of FIG4D .

The local processing and data module (70) may include an energy-efficient processor or controller, and digital memory, such as flash memory, which may be used to facilitate processing, caching, and storing the following data: a) data captured from sensors that may be operatively coupled to the frame (64), such as image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, and/or gyroscopes; and/or b) data acquired using a remote processing module (72) and/or a remote data repository (74), which data may be transmitted to the display (62) after such processing or retrieval. The local processing and data module (70) may be operatively coupled to the remote processing module (72) and the remote data repository (74) (e.g., via a wired or wireless communication link) such that the remote modules (72, 74) are operatively coupled to each other and available as resources to the local processing and data module (70). In one embodiment, the remote processing module (72) may include one or more relatively powerful processors or controllers configured to analyze and process data and/or image information. In one embodiment, the remote data repository (74) may comprise a relatively large digital data storage facility available via the Internet or other network configuration in a "cloud" resource configuration. In one embodiment, all data is stored and all computations are performed in the local processing and data module, allowing for fully autonomous use from any remote module.

Referring to Figures 5A to 22Y, various display configurations are provided that are designed to provide the human eye with photon-based radiation patterns that can be comfortably perceived as an augmentation of physical reality with advanced image quality and three-dimensional perception, while also being able to present two-dimensional content.

Referring to FIG5A , in a simplified example, a transmissive beam splitter substrate (104) having a 45 degree reflective surface (102) directs incident radiation (106) (possibly output from a lens (not shown)) through the pupil (45) of the eye (58) and onto the retina (54). The field of view of such a system is limited by the geometry of the beam splitter (104). To achieve comfortable viewing with minimal hardware, in one embodiment, a larger field of view can be generated by aggregating the output/reflection of various different reflective and/or diffractive surfaces and using, for example, a frame sequential configuration (where a series of frames are provided to the eye (58) at a high frequency, thereby providing the perception of a single coherent scene). Instead of or in addition to providing different image data in a time-sequential manner via different reflectors, these reflectors can separate content by other means such as polarization selectivity or wavelength selectivity. In addition to being able to relay two-dimensional images, these reflectors can also relay three-dimensional wavefronts associated with a true three-dimensional view of actual physical objects.

Referring to FIG5B , a substrate (108) is shown including a plurality of reflectors positioned at a plurality of angles (110), each reflector being actively reflective in the illustrated configuration as an example. The reflectors may be switchable elements to facilitate temporal selectivity. In one embodiment, the reflective surfaces are activated sequentially using a frame sequence of input information (106), wherein each reflective surface provides a narrow field of view sub-image that is tiled with other narrow field of view sub-images provided by other reflective surfaces to form a composite wide field of view image. For example, referring to FIG5C , 5D and 5E , a surface (110) approximately in the middle of the substrate (108) is switched to an “on” state (reflective state) to reflect incoming image information (106), thereby providing a relatively narrow field of view sub-image in the middle of a larger field of view, while the other possible reflective surfaces are in a transmissive state.

Referring to FIG5C , incoming image information (106) from the right side of the narrow field of view sub-image (as indicated by the angle of the incoming beam 106 relative to the input interface 112 of the substrate 108 and the final angle at which the incoming beam 106 exits the substrate (108)) is reflected from the reflective surface (110) to the eye (58). FIG5D shows the same reflector (110) in an active state, where image information is from the middle of the narrow field of view sub-image, as indicated by the angle of the input information (106) at the input interface (112) and the final angle at which the image information exits the substrate (108). FIG5E shows the same reflector (110) in an active state, where image information is from the left side of the narrow field of view sub-image, as indicated by the angle of the input information (106) at the input interface (112) and the final angle at which the image information exits the substrate (108). FIG5F shows a configuration in which the bottom reflector (110) is in an active state, where image information (106) is from the far right side of the entire field of view. For example, Figures 5C, 5D, and 5E can show one frame representing the center of a frame-by-frame tile image, while Figure 5F can show the second frame representing the rightmost side of the tile image.

In one embodiment, light carrying image information (106) can be directed to the reflective surface (110) after entering the substrate (108) from the input interface (112) without first being reflected from a surface of the substrate (108). In one embodiment, light carrying image information (106) can be reflected from one or more surfaces of the substrate (108) after entering from the input interface (112) and before being reflected from the reflective surface (110); for example, the substrate (108) can be used as a planar waveguide to propagate the light carrying image information (106) by total internal reflection. Light can also be reflected from one or more surfaces of the substrate (108) by a partially reflective coating, a wavelength selective coating, an angle selective coating, and/or a polarization selective coating.

In one embodiment, the corner reflectors can be constructed using electroactive materials such that when a voltage and/or current is applied to a particular reflector, the refractive index of the material comprising such reflector changes from a refractive index that substantially matches that of the rest of the substrate (108) (in which case the reflector assumes a transmissive configuration) to a reflective configuration in which the refractive index of the reflector does not match that of the substrate (108), thereby producing a reflective effect. Example electroactive materials include lithium niobate and electroactive polymers. Suitable substantially transparent electrodes for controlling a plurality of such reflectors can include materials such as indium tin oxide, which are used in liquid crystal displays.

In one embodiment, the electro-active reflector (110) may comprise liquid crystal embedded in a substrate (108) host medium such as glass or plastic. In certain variations, the liquid crystal may be selected to change its refractive index in response to an applied electrical signal, thereby enabling a more analog, as opposed to binary, change (from a transmissive state to a reflective state). In embodiments where six sub-images are presented to the eye in a frame sequence to form a large tiled image with a total refresh rate of 60 frames per second, it would be desirable to have an input display capable of refreshing at a rate of approximately 360 Hz with an array of electro-active reflectors capable of keeping up with this frequency. In one embodiment, lithium niobate may be used as an electro-active reflective material as opposed to liquid crystal; lithium niobate is used in the optoelectronics industry for high-speed switches and fiber optic networks and is capable of switching its refractive index in response to a voltage applied at an extremely high frequency; this high frequency may be used to control frame-sequential or pixel-sequential sub-image information, particularly when the input display is a scanned light display (e.g., a fiber-scanning display or a scanning mirror-based display).

In another embodiment, a switchable variable tilt mirror configuration may include one or more high-speed mechanically repositionable reflective surfaces, such as a MEMS (micro-electromechanical system) device. The MEMS device may include a so-called "digital mirror device" or "DMD" (typically part of a "digital light processing" or "DLP" system, such as those manufactured by Texas Instruments, Inc.). In another electro-mechanical embodiment, multiple air-gapped (or vacuum) reflective surfaces are mechanically positioned and shifted at high frequencies. In another electro-mechanical embodiment, a single reflective surface can be moved up and down and repositioned at extremely high frequencies.

With reference to FIG5G , it is noted that the switchable variable tilt reflector configuration described herein is capable of delivering not only collimated or flat wavefront information to the retina (54) of the eye (58), but can also deliver curved wavefront (122) image information, as illustrated in the diagram of FIG5G . This is not true for other waveguide-based configurations, where total internal reflection of curved wavefront information results in non-ideal conditions, and thus the input generally must be calibrated. The ability to deliver curved wavefront information enables configurations such as those shown in FIG5B-5H to provide input to the retina (54) that is perceived as focused at various distances from the eye (58), not just at optical infinity (which would interpret collimated light in the absence of other cues).

5H , in another embodiment, an array of static partially reflective surfaces (116) (i.e., always in reflective mode; in another embodiment, they can be electrically active, as described above) can be embedded in a substrate (114) with a high-frequency gating layer (118) that controls the output to the eye (58) by allowing only transmission through controllably movable apertures (120). In other words, everything can be selectively blocked except transmission through apertures (120). The gating layer (118) can include a liquid crystal array, a lithium niobate array, an array of MEMS shutter elements, an array of DLP DMD elements, or an array of other MEMS devices that are configured to transmit or transmit with relatively high frequency switching and high transmittance when switched to transmissive mode.

6A-6B , other embodiments are shown in which arrays of optical elements can be combined with an exit pupil expansion configuration to help users achieve a comfortable virtual or augmented reality experience. With a larger "exit pupil" in the optical configuration, movement of the user's eyes relative to the display (which, as in FIG4A-4D , can be mounted on the user's head in the form of a glasses configuration) does not disrupt the user experience—because the system has a larger exit pupil, there is a larger acceptable region within which the user's anatomical pupil can be positioned to continue receiving information from the display system as desired. In other words, with a larger exit pupil, the system is less likely to notice that the display is slightly misaligned with the user's anatomical pupil, and can make the user more comfortable by reducing the geometric constraints on the user's relationship to the display/glasses.

As shown in FIG6A , the display (140) on the left feeds a set of parallel rays into the substrate (124). In one embodiment, the display can be a scanning fiber display, as shown, which scans a thin beam back and forth at an angle to project an image through a lens or other optical element (142), which can be used to collect the angled scanned light and convert it into a parallel beam of rays. The rays can be reflected from a series of reflective surfaces (126, 128, 130, 132, 134, 136), which can be configured to partially reflect and partially transmit the incoming light, thereby sharing the light substantially equally across the set of reflective surfaces (126, 128, 130, 132, 134, 136). Through a small lens (138) placed at each exit point of the waveguide (124), the exiting light can be guided through a node and scanned toward the eye (58) to provide an exit pupil array, or the equivalent of a large exit pupil, which the user can use when he or she looks at the display system.

For a virtual reality configuration that ideally also allows the real world (144) to be seen through the waveguide, a similar lens group (139) can be provided on the opposite side of the waveguide (124) to compensate for the lens group below, thereby producing a function equivalent to a zero-magnification telescope. As shown, the reflecting surfaces (126, 128, 130, 132, 134, 136) can all be aligned at approximately 45 degrees, or can be configured to have different arrangements (e.g., configurations similar to those of Figures 5B-5H). The reflecting surfaces (126, 128, 130, 132, 134, 136) can include wavelength-selective reflectors, bandpass reflectors, half-silvered mirrors, or other reflective configurations. The lenses (138, 139) shown are refractive lenses, but diffractive lens elements can also be used.

Referring to FIG6B , a somewhat similar configuration is shown in which a plurality of curved reflective surfaces (148, 150, 152, 154, 156, 158) can be used to effectively combine the functionality of the lens (element 138 of FIG6A ) and reflector (elements 126, 128, 130, 132, 134, 136 of FIG6A ) of the embodiment of FIG6A , thereby avoiding the need for two sets of lenses (element 138 of FIG6A ). The curved reflective surfaces (148, 150, 152, 154, 156, 158) can be selected to simultaneously reflect and impart angular variation in various curved configurations, such as parabolic or elliptical surfaces. With a parabolic shape, a set of parallel incoming rays can be collected into a single output point; with an elliptical configuration, a set of rays diverging from a single origin are collected into a single output point. For the configuration of FIG6A , the curved reflective surfaces (148, 150, 152, 154, 156, 158) are preferably configured to partially reflect and partially transmit, thereby sharing the incoming light across the length of the waveguide (146). The curved reflective surfaces (148, 150, 152, 154, 156, 158) may include wavelength-selective notch reflectors, half-silvered mirrors, or other reflective configurations. In another embodiment, the curved reflective surfaces (148, 150, 152, 154, 156, 158) may be replaced by diffractive reflectors to simultaneously reflect and deflect.

Referring to FIG7A , a waveguide can be used in conjunction with a variable focus optical configuration to facilitate the perception of Z-axis difference (i.e., linear distance from the eye along the optical axis). As shown in FIG7A , image information from a display (160) can be collimated and injected into a waveguide (164), and this image information can be distributed in a wide exit pupil manner using, for example, the configuration described with reference to FIG6A and 6B , or other substrate-guided optical methods known to those skilled in the art—the variable focus optical capability can then be used to alter the wavefront focus of the light exiting the waveguide and cause the eye to perceive the light from the waveguide (164) as originating from a specific focal distance. In other words, because the incoming light has been collimated to avoid the challenges of total internal reflection waveguide configurations, the light exits in a collimated manner, requiring the viewer's eye to accommodate to a far point to bring the light into focus on the retina, and is naturally interpreted as originating from optical infinity—unless some other intervention causes the light to be refocused and perceived as originating from a different viewing distance; one suitable such intervention is a variable focus lens.

In the embodiment of FIG7A , collimated image information is injected into a lens (162) or other material at an angle so as to be totally internally reflected and transferred into an adjacent waveguide (164). The waveguide (164) can be configured in a manner similar to the waveguides of FIG6A or 6B (124, 126, respectively) to distribute collimated light from the display so as to be emitted with a somewhat uniform distribution across the reflector or diffractive features along the length of the waveguide. When emitted toward the eye (58), in the configuration shown, the emitted light passes through a variable focus lens element (166), where the light emitted from the variable focus lens element (166) and into the eye (58) will have various degrees of focus depending on the controlled focus of the variable focus lens element (166) (representing a collimated flat wavefront at optical infinity, representing more beam divergence/wavefront curvature at closer viewing distances relative to the eye 58).

To compensate for the variable focus lens element (166) between the eye (58) and the waveguide (164), another similar variable focus lens element (166) is placed on the opposite side of the waveguide (164) to cancel the optical effect of the lens (166) with respect to light from the world (144), thereby achieving augmented reality (i.e., as described above, one lens compensates for the other lens, thereby producing the equivalent of a zero-magnification telescope).

The variable focus lens element (166) can be a refractive element, such as a liquid crystal lens, an electro-active lens, a conventional refractive lens with moving elements, a lens based on mechanical deformation (such as a liquid-filled film lens or a lens similar to the human lens (in which an elastic element contracts and expands under the action of an actuator)), an electrowetting lens, or a plurality of fluids with different refractive indices. The variable focus lens element (166) can also include a switchable diffractive optical element (such as an element characterized by polymer dispersed liquid crystal technology, in which a host medium such as a polymer material has liquid crystal droplets dispersed within the material; when a voltage is applied, the molecules reorient so that their refractive index no longer matches that of the host medium, thereby producing a high frequency switchable diffraction pattern).

One embodiment includes a host medium in which droplets of an electroactive material based on the Kerr effect, such as lithium niobate, are dispersed to enable pixel-by-pixel or line-by-line refocusing of image information when coupled to a scanning light display, such as a fiber-scanning display or a scanning mirror-based display. In a variable focus lens element (166) configuration in which a pattern is presented using liquid crystal, lithium niobate, or other technology, the pattern spacing can be modulated to change the optical power not only of the variable focus lens element (166), but also of the entire optical system (for a zoom lens type function).

In one embodiment, the lens (166) can be a telecentric element so that the focus of the displayed image can be changed while maintaining the same magnification - similarly, a photographic zoom lens can be configured to separate the focus from the zoom position. In another embodiment, the lens (166) can be a non-telecentric element so that the focus changes also follow the zoom changes. With this configuration, such magnification changes can be compensated in software with dynamic zooming of the graphics system output to keep pace with the focus changes.

Returning to the issue of the projector or other video display unit (160) and how images are fed into the optical display system, in a "frame sequential" configuration, a stack of sequenced two-dimensional images can be fed sequentially into the display to produce a three-dimensional perception over time; this is similar to the way computed tomography systems use stacked image segments to represent three-dimensional structures. A series of two-dimensional image segments can be presented to the eye, each segment at a different focal distance from the eye, and the eye/brain assembles these stacks into a coherent three-dimensional volume perception. Depending on the type of display, line-by-line or even pixel-by-pixel sequencing can be performed to produce a three-dimensional visual perception. For example, for a scanned light display (such as a scanning fiber display or a scanning mirror display), the display sequentially provides a line or pixel at a time to the waveguide (164).

If the variable focus lens element (166) can keep up with the high frequency of pixel-by-pixel or line-by-line presentation, each line or each pixel can be presented and dynamically focused by the variable focus lens element (166) so as to be perceived from different focal distances from the eye (58). Pixel-by-pixel focus modulation generally requires an extremely fast/high frequency variable focus lens element (166). For example, a 1080P resolution display with a total frame rate of 60 frames per second generally presents approximately 125 million pixels per second. Such a configuration can also be constructed using solid-state switchable lenses, such as lenses using electroactive materials (e.g., lithium niobate or electroactive polymers). In addition to being compatible with the system shown in FIG7A , the frame sequential multifocal display drive technology can also be used in conjunction with the various display systems and optical element embodiments described in the present disclosure.

7B , for an electroactive layer (172) (such as one comprising liquid crystal or lithium niobate) surrounded by functional electrodes (170, 174) (which may be made of indium tin oxide), a waveguide (168) with a conventional transmissive substrate (176) (such as a substrate made of glass or plastic with known total internal reflection characteristics and a refractive index matched to the on or off state of the electroactive layer 172) can be controlled so that the path of the incoming light beam can be dynamically changed to essentially produce a time-varying light field.

8A , a stacked waveguide assembly (178) can be used to provide three-dimensional perception to the eye/brain by having a plurality of waveguides (182, 184, 186, 188, 190) and a plurality of weak lenses (198, 196, 194, 192) configured together to send image information to the eye at multiple wavefront curvature levels for each waveguide level (indicating the perceived focal length for that waveguide level). Multiple displays (200, 202, 204, 206, 208) or a single multiplexed display (in another embodiment) can be used to inject calibrated image information into the waveguides (182, 184, 186, 188, 190), each of which can be configured as described above to distribute incoming light substantially equally across the length of each waveguide, thereby directing the light downward toward the eye.

The waveguide (182) closest to the eye is configured to transmit collimated light injected into the waveguide (182) to the eye, which can represent a focal plane at optical infinity. The next waveguide (184) is configured to emit collimated light, which passes through a first weak lens (192; for example, a weak diverging lens) before reaching the eye (58); the first weak lens (192) can be configured to produce a slightly convex wavefront curvature so that the eye/brain perceives the light from the next waveguide (184) as coming from a first focal plane that moves inward from optical infinity toward the person. Similarly, the third waveguide (186) then causes the output light to pass through both the first lens (192) and the second lens (194) before reaching the eye (58); the combined optical power of the first lens (192) and the second lens (194) can be configured to produce another wavefront divergence increment so that the eye/brain perceives the light from the third waveguide (186) as coming from a second focal plane that is inward from optical infinity and closer to the person than the light from the next waveguide (184).

The other waveguide layers (188, 190) and weak lenses (196, 198) are similarly configured, with the highest waveguide (190) in the stack having its output passed through all weak lenses between it and the eye to achieve a collective optical power representing the focal plane closest to the person. To compensate for the lens stack (198, 196, 194, 192) when viewing/perceiving light from the world (144) on the other side of the stacked waveguide assembly (178), a compensating lens layer (180) is placed on top of the stack to compensate for the collective power of the lens stack (198, 196, 194, 192) below. Such a configuration provides a number of perceived focal planes equal to the number of available waveguide/lens pairs, again with a larger exit pupil configuration as described above. Both the reflective aspects of the waveguides and the focusing aspects of the lenses can be static (i.e., not dynamic or electrically active). In an alternative embodiment, they may be dynamic aspects using the electro-active features described above, enabling a small number of waveguides to be multiplexed in a time-sequential manner to produce a large number of effective focal planes.

Referring to Figures 8B-8N, various aspects of a diffraction configuration for focusing and/or redirecting a collimated light beam are shown. Other aspects of a diffraction system for this purpose are disclosed in U.S. patent application Ser. No. 61/845,907 (U.S. patent application Ser. No. 14/331,281), which is incorporated herein by reference in its entirety. Referring to Figure 8B, passing a collimated light beam through a linear diffraction pattern (210) (such as a Bragg grating) will deflect or "steer" the beam. Passing the collimated light beam through a radially symmetric diffraction pattern (212) or "Fresnel zone plate" will change the focus of the beam. Figure 8C illustrates the deflecting effect of passing the collimated light through a linear diffraction pattern (210); Figure 8D illustrates the focusing effect of passing the collimated light beam through a radially symmetric diffraction pattern (212).

Referring to Figures 8E and 8F, a combined diffraction pattern having both linear and radial elements (214) simultaneously produces a deflection and focusing of a collimated input beam. These deflection and focusing effects can be produced in reflection mode as well as in transmission mode. These principles can be applied through waveguide configurations to achieve additional optical system control, as shown in Figures 8G-8N. As shown in Figures 8G-8N, a diffraction pattern (220) or "diffractive optical element" (or "DOE") has been embedded within a planar waveguide (216) so that the collimated beam is totally internally reflected along the planar waveguide (216), which intersects the diffraction pattern (220) at multiple locations.

Preferably, the DOE (220) has a relatively low diffraction efficiency so that at each intersection with the DOE (220), only a portion of the light beam is deflected away from the eye (58), while the remainder continues to pass through the planar waveguide (216) via total internal reflection; the light carrying the image information is thus split into a plurality of related beams that are emitted from the waveguide at a plurality of locations, resulting in a very uniform pattern of emitted light beams toward the eye (58), as shown in FIG8H , for this particular collimated light beam that is reflected throughout within the planar waveguide (216). The emitted light beams toward the eye (58) are shown in FIG8H as being substantially parallel because, in this case, the DOE (220) has only a linear diffraction pattern. As shown in the comparison between FIG8L , 8M and 8N , variations in the spacing of the linear diffraction patterns can be used to controllably deflect the emitted parallel light beams, thereby producing a scanning or tiling function.

Referring back to FIG8I , as the radially symmetric diffraction pattern component of the embedded DOE (220) changes, the outgoing beam pattern becomes more divergent, requiring the eye to adapt to a closer distance to focus on the retina, and the brain perceives the light as coming from a viewing distance closer to the eye than optical infinity. Referring to FIG8J , after adding another waveguide (218) into which a beam can be injected (e.g., by a projector or display), the DOE (221) (such as a linear diffraction pattern) embedded in the other waveguide (218) can be used to spread light across the entire large planar waveguide (216), depending on the specific DOE configuration at work, which can provide a very large inbound area (the inbound area of inbound light emitted from the larger planar waveguide (216) (i.e., the large eye box)) for the eye (58).

The DOEs (220, 221) are shown as bisecting the associated waveguides (216, 218), but need not do so; they can be placed adjacent to, or on either side of, either waveguide (216, 218) to serve the same function. Thus, as shown in FIG8K , with the injection of a single collimated beam, the entire replicated collimated beam field can be directed toward the eye (58). Furthermore, through combined linear diffraction pattern/radially symmetric diffraction pattern scenarios such as those shown in FIG8F (214) and 8I (220), a beam distributing waveguide optical element with Z-axis focusing functionality is provided (for functions such as pupil expansion; with configurations such as FIG8K , the exit pupil can be as large as the optical element itself, which is a significant advantage for user comfort and ergonomics), where the divergence angles of the replicated beams and the wavefront curvature of each beam represent light from a point closer than optical infinity.

In one embodiment, one or more DOEs can be switched between an on state (in which they actively diffract) and an off state (in which they do not significantly diffract). For example, a switchable DOE can include a layer of polymer dispersed liquid crystal, wherein the droplets include a diffraction pattern located in a host medium, and the refractive index of the droplets can be switched to substantially match that of the host material (in which case the pattern does not significantly diffract incident light), or the droplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light). Further, with dynamic variations in diffraction terms (such as the linear diffraction pitch terms shown in Figures 8L-8N), beam scanning or tiling functionality can be achieved. As discussed above, it is desirable to have a relatively low diffraction grating efficiency in each DOE (220, 221) because this facilitates distribution of the light, and also because light that is ideally transmitted through the waveguide (e.g., light emitted from the world 114 and toward the eye 58 in an augmented reality configuration) is less affected when the diffraction efficiency of the DOE (220) through which it passes is low - thereby enabling a better real-world view with such a configuration.

Configurations such as that shown in FIG8K are preferably driven by image information injection using a time-sequential approach, with frame-sequential drive being the easiest to implement. For example, an image of the sky at optical infinity can be injected at time 1, utilizing a diffraction grating that preserves optical calibration. An image of a nearby tree branch can then be injected at time t2, with the DOE controllably imparting a focus change of one diopter unit or one meter, causing the eye/brain to perceive the tree branch light information as originating from a closer focal range. This paradigm can be repeated in a rapid time-series fashion so that the eye/brain perceives the input as all parts of the same image. This is just one example of a dual-focal-plane setup; preferably, the system would be configured with more focal planes to provide smoother transitions between objects and their focal distances. This configuration generally assumes that the DOE switches at a relatively slow rate (i.e., synchronized with the displayed frame rate of the injected image—in the range of thousands of cycles per second).

The opposite extreme could be a configuration where the DOE element is capable of changing focus at tens to hundreds of MHz or more, which facilitates switching the focus state of the DOE element pixel by pixel as the pixels are scanned into the eye (58) using scanned light display technology. This is desirable because it means that the overall display frame rate can be kept quite low; low enough to ensure that "flicker" is not an issue (in the range of about 60-120 frames/second).

Between these ranges, if the DOE can be switched at a KHz rate, the focus on each scan line can be adjusted line by line, thus giving the user a visual advantage with respect to temporal artifacts, for example, during eye movement relative to the display. For example, in this way, different focal planes in a scene are interleaved to minimize visual artifacts in response to head motion (as discussed in more detail below in this disclosure). The line-by-line focus modulator can be operatively coupled to a line scan display (such as a grating light valve display), in which a linear array of pixels is scanned to form an image; the focus modulator can also be operatively coupled to a scanned light display (such as a fiber scanned display and a mirror scanned light display).

A stacked configuration similar to that of FIG8A can use dynamic DOEs (rather than the static waveguides and lenses of the embodiment of FIG8A ) to provide multi-plane focus simultaneously. For example, with three simultaneous focal planes, a primary focal plane (e.g., based on measured eye accommodation) can be presented to the user, and + and - margins (i.e., one focal plane closer, one farther away) can be used to provide a large focus range within which the user can accommodate before needing to update the plane. This increased focus range can provide a time advantage when the user switches to a closer or farther focus (i.e., as determined by accommodation measurements); the new focal plane can then be made an intermediate depth of focus, with + and - margins again ready to quickly switch to either while the system continues to perform.

Referring to FIG8O , there is shown a stack (222) of planar waveguides (244, 246, 248, 250, 252), each having a reflector (254, 256, 258, 260, 262) at the end thereof and configured such that collimated image information injected into one end by a display (224, 226, 228, 230, 232) is reflected downwardly by total internal reflection to the reflector, at which point some or all of the light is reflected outward toward the eye or other target. Each reflector can have a slightly different angle to reflect the outgoing light toward a common target, such as the pupil. This configuration is somewhat similar to the configuration of FIG5B , except that each of the different tilted reflectors in the embodiment of FIG8O has its own waveguide, which reduces interference in the projected light as it travels to the target reflector. Lenses (234, 236, 238, 240, 242) can be inserted between the display and the waveguide to achieve beam steering and/or focusing.

FIG8P shows a geometrically staggered version in which the reflectors (276, 278, 280, 282, 284) are positioned in the waveguides (266, 268, 270, 272, 274) at staggered lengths so that the exiting beam can be relatively easily aligned with an object such as an anatomical pupil. Knowing how far the stack (264) will be from the eye (such as 28 mm between the cornea of the eye and the lens of the eye, a typical comfortable geometry), the geometry of the reflectors (276, 278, 280, 282, 284) and the waveguides (266, 268, 270, 272, 274) can be established to fill the pupil (typically about 8 mm across or less) with the exiting light. By directing the light to an eyebox that is larger than the diameter of the pupil, the viewer can move the eye while still being able to see the displayed image. Referring back to the discussion related to Figures 5A and 5B regarding field of view extension and reflector size, an extended field of view is also presented by the configuration of Figure 8P, but that figure is less complex than the switchable reflective elements in the embodiment of Figure 5B.

Figure 8Q shows a version in which a number of reflectors (298) form a relatively continuous curved reflective surface in aggregate or discrete planes oriented to align with the overall curved surface. The curved surface can be parabolic or elliptical and is shown cut across multiple waveguides (288, 290, 292, 294, 296) to minimize any crosstalk issues, although it can also be used in a monolithic waveguide configuration.

In one embodiment, a high frame rate and low persistence display can be combined with a low frame rate and high persistence display and a variable focus element to form a relatively high-frequency frame sequential stereoscopic display. In one embodiment, the high frame rate display has a lower bit depth, and the low frame rate display has a higher bit depth, and the two are combined to form an efficient high frame rate and high bit depth display that is well suited for presenting image segments in a frame sequential manner. In this way, the ideally represented three-dimensional volume is functionally divided into a series of two-dimensional segments. Each of these two-dimensional segments is sequentially projected into the eye frame, and the focus of the variable focus element is also changed in synchronization with the presentation.

In one embodiment, to achieve sufficient frame rates to support such a configuration, two displays can be integrated: a full-color, high-resolution liquid crystal display ("LCD"); in another embodiment, a backlit ferroelectric flat panel display can also be used; in yet another embodiment, a scanning fiber display can be used; and aspects of a higher-frequency DLP system. In addition to illuminating the back of the LCD panel in a conventional manner (i.e., with a full-scale fluorescent lamp or LED array), the conventional illumination configuration can be removed in favor of using a DLP projector to project a masking pattern onto the back of the LCD (in one embodiment, the masking pattern can be a binary pattern, in that the DLP projects both illumination and non-illumination; in another embodiment, described below, the DLP can be used to project a grayscale masking pattern).

DLP projection systems can operate at extremely high frame rates; in one embodiment, operating at 60 frames per second for six depth planes, the DLP projection system operates at 360 frames per second for the back of an LCD display. The DLP projection system is used to selectively illuminate portions of the LCD panel in synchronization with a high-frequency variable focus element (such as a deformable membrane mirror) positioned between the viewing side of the LCD panel and the user's eye. The variable focus element is used to change the global display focus on a frame-by-frame basis at 360 frames per second. In one embodiment, the variable focus element is positioned optically conjugate to the exit pupil, allowing focus adjustment without simultaneously affecting image magnification or "zoom." In another embodiment, the variable focus element is not conjugate to the exit pupil, allowing image magnification to vary with focus adjustment, and software is used to compensate for these optical magnification changes and any distortion caused by pre-scaling or warping the image to be presented.

Operationally, it is necessary to consider again the following example: a three-dimensional scene is to be presented to the user in which the sky within the background is to be located at a viewing distance of optical infinity, and in which a branch connected to a tree located at a particular position closer to the user than optical infinity extends from the trunk in a direction toward the user so that the top of the branch is closer to the user than the nearest part of the branch connected to the trunk.

In one embodiment, for a given global frame, the system can be configured to present a full-color, fully focused image of a tree branch in front of the sky on the LCD. Then, at subframe 1 within the global frame, a DLP projector in a binary masking configuration (i.e., illuminated or not illuminated) can be used to illuminate only the portion of the LCD representing a cloudy sky, while functionally masking (i.e., not illuminating) portions of the LCD representing tree branches or other elements that are not perceived as being at the same focal distance as the sky. Furthermore, a variable focus element (such as a deformable film mirror) can be used to position the focal plane at optical infinity, so that the sub-image in subframe 1 appears to the eye as clouds at infinite distance.

Then, at subframe 2, the variable focus element can be switched to focus on a point approximately 1 meter from the user's eye (or any desired distance; the 1 meter for the tree branch location is used for illustrative purposes only). The DLP's illumination pattern can be switched so that the system illuminates only the portion of the LCD representing the tree branch, while functionally blacking out (i.e., not illuminating) the portion of the LCD representing the sky or other elements not perceived as being at the same focal distance as the tree branch. In this way, the eye quickly glances past the clouds at optical infinity, then quickly glances past the tree at 1 meter, and the eye/brain integrates these sequences to form a three-dimensional perception. The tree branch can be located diagonally to the viewer so that it extends across a range of viewing distances. For example, it may connect to the trunk at a viewing distance of approximately 2 meters, with the top of the branch located closer at 1 meter.

In this case, the display system can break the 3-D volume of the tree branches into multiple segments, rather than a single segment at 1 meter. For example, the sky can be represented using one focal segment (using DLP to mask all areas of the tree when rendering that segment), while the branches can be segmented across five focal segments (using DLP to mask all but the sky and a portion of the tree, which is the branch being rendered). Preferably, the depth segments are placed at a spacing equal to or less than the eye's focal depth, so that the viewer is less likely to notice the transitions between segments and instead perceives a smooth, continuous flow of branches across the range of focus.

In another embodiment, rather than using a DLP in binary (light-only or darkfield) mode, the DLP can be used to project a grayscale (e.g., 256 levels of gray) mask onto the back of the LCD panel to enhance 3D perception. Grayscale gradients can be used to trick the eye/brain into perceiving something as residing between adjacent depth or focal planes. Returning to the tree branch or cloud scene, if the leading edge of the branch closest to the user is at focal plane 1, then at subframe 1, the portion of the branch located on the LCD can be illuminated with full intensity white by the DLP system (with a variable focus element located at focal plane 1).

Then, in subframe 2, no illumination is generated for the variable focus element located at focal plane 2 (just behind the illuminated portion). These steps are similar to the binary DLP masking configuration described above. However, if there is a portion of a tree branch that is to be perceived between focal planes 1 and 2 (e.g., halfway), grayscale masking can be utilized. DLP can project an illumination mask onto this portion during both subframes 1 and 2, but with half illumination (e.g., 128 out of 256 grayscale levels) for each subframe. This allows for the perception of a blend of multiple depths of focus, with the perceived focal length proportional to the illumination ratio between subframes 1 and 2. For example, for a branch portion that should be located three-quarters of the way between focal planes 1 and 2, the LCD portion located in subframe 1 can be illuminated with a grayscale mask of approximately 25% intensity, and the same LCD portion located in subframe 2 can be illuminated with a grayscale mask of approximately 75%.

In one embodiment, the bit depths of the low frame rate display and the high frame rate display can be combined to achieve image modulation and create a high dynamic range display. High dynamic range driving can be performed in tandem with the focal plane addressing function described above to form a high dynamic range multi-focal 3-D display.

In another embodiment that is more efficient with respect to computing resources, only a specific portion of the display (i.e., LCD) output may be masked and illuminated by the DMD and zoomed on its way to the user's eye. For example, the middle portion of the display may be masked and illuminated, and the peripheral portion of the display may not provide varying accommodation cues to the user (i.e., the peripheral portion may be uniformly illuminated by the DLP DMD, while the middle portion is actively masked and zoomed on its way to the eye).

In the embodiment described above, a refresh rate of approximately 360 Hz allows for refreshing of six depth planes at approximately 60 frames per second per plane. In another embodiment, by increasing the operating frequency of the DLP, even higher refresh rates can be achieved. Standard DLP configurations use MEMS devices and arrays of micromirrors. These micromirrors switch between a mode in which they reflect light towards the display or user and a mode in which they reflect light away from the display or user (such as into a light trap), making them binary in nature. DLPs typically produce grayscale images using a pulse width modulation mechanism in which the mirrors are "on" for a variable amount of time with a variable duty cycle in order to produce brighter pixels or temporarily brighter pixels. Therefore, in order to produce grayscale images at a moderate frame rate, they operate at a much higher binary rate.

In the configuration described above, this setup is well suited for producing grayscale masking. However, if the DLP drive mechanism is modified to flash the sub-images in the binary pattern, the frame rate can be significantly increased (thousands of frames per second). This allows hundreds to thousands of depth planes to be refreshed at 60 frames per second, which can be exploited to avoid the grayscale interpolation between depth planes described above. The typical pulse-width modulation mechanism used in Texas Instruments DLP systems has an 8-bit command signal (the first bit is the first long pulse of the mirror; the second bit is a pulse half the first long pulse; the third bit is a pulse half the second pulse, and so on), allowing the configuration to produce second through eighth illumination levels of varying power. In one embodiment, the intensity of the backlight from the DLP can be varied in sync with the different pulses of the DMD to equalize the brightness of the generated sub-images, a practical solution for enabling existing DMD drive electronics to produce significantly higher frame rates.

In another embodiment, direct control changes of the DMD drive electronics and software can be used to always have an equal on-time configuration for the mirrors, rather than the traditional variable on-time configuration, which facilitates higher frame rates. In another embodiment, the DMD drive electronics can be configured to provide low bit depth images at a higher frame rate than high bit depth images but lower than the binary frame rate, thereby allowing some grayscale blending between focal planes while modestly increasing the number of focal planes.

In another embodiment, when limited to a limited number of depth planes (e.g., 6 depth planes in the example above), it is ideal to functionally move these 6 depth planes so that they are most effectively used for the scene being presented to the user. For example, if a user is standing in a room and has a virtual monster placed in their augmented reality view that is approximately 2 feet deep from the user's eyes in the Z axis, it makes sense to cluster all 6 depth planes around the monster's current position (and dynamically move the depth planes so that it appears as if the monster is moving relative to the user) so that the user can be provided with richer visual accommodation cues and all six depth planes are located in the monster's immediate area (e.g., 3 in front of the center of the monster and 3 behind the center of the monster). This depth plane allocation is context-sensitive.

For example, in the scenario described above, the same monster is placed in the same room, while also presenting a virtual window frame element to the user, and then presenting a virtual view of optical infinity outside the virtual window frame. A useful approach is to use at least one depth plane for optical infinity, at least one depth plane for the wall on which the virtual window frame is mounted, and then perhaps the remaining four depth planes for the monster within the room. If the content causes the virtual window to disappear, two depth planes can be dynamically reallocated to the area around the monster, and so on. Dynamic allocation of focus plane resources based on content provides the richest experience for the user, given the computing and rendering resources.

In another embodiment, phase delays in a multi-core fiber or a single-core fiber array can be used to generate variable-focus light wavefronts. Referring to FIG9A , a multi-core fiber (300) can include an aggregation of multiple individual optical fibers (302); FIG9B shows a close-up view of the multi-core assembly, which emits light from each core in the form of a spherical wavefront (304). If the cores send coherent light, such as from a shared laser source, these small spherical wavefronts eventually interfere with each other constructively or destructively, and if they emerge from the multi-core fiber in phase, they collectively form a nearly planar waveguide (306), as shown. However, if phase delays are induced between the cores (using a conventional phase modulator (such as one using lithium niobate) to slow the path of some cores relative to others), it is possible to collectively generate curved or spherical wavefronts to represent objects from points closer than optical infinity at the eye/brain, representing another option that can be used instead of the variable-focus elements described above. In other words, such a phased multi-core configuration or phased array can be used to produce multiple degrees of optical focus from a light source.

In another embodiment related to the use of optical fibers, the well-known Fourier transform aspect of multimode optical fibers or light guides or pipes can be used to control the wavefront output from such optical fibers. Optical fibers generally come in two forms: single-mode and multimode. Multimode fibers generally have larger core diameters and allow light to propagate along multiple angular paths, rather than just along a single-mode fiber. It is known that if an image is injected into one end of a multimode fiber, the angular differences encoded into the image are preserved to some extent as it propagates through the multimode fiber. For certain configurations, the output from the optical fiber can resemble the Fourier transform of the input image.

Thus, in one embodiment, an inverse Fourier transform of an input wavefront (such as a diverging spherical wavefront representing a focal plane closer to the user than optical infinity) can be taken, and after passing through an optical fiber that optically transmits the Fourier transform, the output becomes a wavefront with a desired shape or focus as desired. For example, such an output can be fully scanned to function as a scanning fiber display, or can be used as a light source for a scanning mirror to form an image. Thus, such a configuration can be used as yet another focus modulation subsystem. Other types of light patterns and wavefronts can be injected into a multimode fiber to emit a specific spatial pattern at the output. This can be used to achieve the equivalent functionality of wavelet patterns (in optics, optical systems can be analyzed in terms of what are known as Zernike coefficients; images can be characterized in a similar manner and decomposed into smaller principal components, or weighted combinations of simpler image components). Thus, if the principal components are used on the input side to scan light into the eye, a higher resolution image can be recovered at the output of the multimode fiber.

In another embodiment, the Fourier transform of the hologram can be injected into the input of a multimode fiber to output a wavefront that can be used for three-dimensional focus modulation and/or resolution enhancement. Certain single-core fibers, multi-core fibers, or coaxial core fiber + cladding configurations can also be used in the above-mentioned inverse Fourier transform configuration.

In another embodiment, rather than physically manipulating a wavefront approaching a user's eye at a high frame rate without regard to the user's specific accommodation state or eye gaze, the system can be configured to monitor the user's accommodation, and instead of presenting a diverse set of light wavefronts, provide a single wavefront at a time corresponding to the accommodation state of the eye. Accommodation can be measured directly (such as by infrared autorefractor or eccentric photo-optormetry) or indirectly (such as by measuring the level of convergence of the user's eyes; as discussed above, vergence and accommodation are very closely neurally linked, so accommodation estimates can be made based on vergence geometry). Thus, with accommodation determined at, for example, 1 meter from the user, the presentation of the wavefront at the eye can be configured to be at a 1 meter focal length using any of the variable focus configurations described above. If an accommodation change to a 2 meter focus is detected, the presentation of the wavefront at the eye can be reconfigured to be at a 2 meter focal length, and so on.

Therefore, in one embodiment incorporating accommodation tracking, a variable focus element can be placed in the optical path between the output combiner (e.g., a waveguide or beam splitter) and the user's eye so that the focus can be changed as the eye's accommodation changes (i.e., preferably at the same frame rate). Software effects can be used to create a variable amount of blur (e.g., Gaussian blur) for objects that should not be in focus to simulate refractive blur (expected at the retina if the object is at viewing distance) and enhance the eye/brain's 3D perception.

A simple embodiment is a single plane whose focus follows the viewer's level of accommodation, but if only a few planes are used, the performance requirements of the accommodation tracking system can be relaxed. Referring to FIG10 , in another embodiment, a stack (328) of approximately three waveguides (318, 320, 322) can be used to simultaneously create three focal planes equivalent to wavefronts. In one embodiment, the weak lens (324, 326) can have a static focal length, and the variable focus lens (316) can follow the eye's accommodation tracking so that the output of one of the three waveguides (such as the middle waveguide 320) is considered to be the sharp wavefront, while the other two waveguides (322, 318) output a +margin wavefront and a -margin wavefront (i.e., slightly farther than the detected focal length and slightly closer than the detected focal length), which can improve three-dimensional perception and also provide enough differentiation for the brain/eye accommodation control system to perceive some blur as negative feedback, which enhances reality perception and allows a range of accommodation before the focus level must be physically adjusted.

Also shown is a variable focus compensating lens (314) to ensure that light from the real world (144) in an augmented reality configuration is not refocused or magnified by the combination of the stack (328) and the output lens (316). As described above, the zoom in these lenses (316, 314) can be achieved through refraction, diffraction, or reflection techniques.

In another embodiment, each waveguide in the stack can contain its own zoom functionality (e.g., by having an embedded electronically switchable DOE), thus eliminating the need to center the zoom element as in the stack (328) of the configuration of Figure 10.

In another embodiment, variable focus elements can be interleaved between waveguides in the stack (i.e., rather than fixed focus weak lenses as in the embodiment of Figure 10) to eliminate the need to combine fixed focus weak lenses with the entire stack of refocus variable focus elements.

Such a stacked configuration can be used for the accommodation tracking changes described herein, and can also be used for frame sequential multi-focus display techniques.

In configurations where light enters the pupil through a small exit pupil (such as 1/2 mm in diameter or smaller), one configuration has the functional equivalent of a pinhole lens configuration, in which the light beam is always perceived by the eye/brain as being in focus—for example, a scanned light display uses a 0.5 mm diameter beam to scan an image to the eye. This type of configuration is referred to as a Maxwell view configuration, and in one embodiment, accommodation tracking input can be used to induce blurring using software for image information that is perceived as being behind or in front of the focal plane determined by accommodation tracking. In other words, if one starts with a display that presents a Maxwell view, then everything can theoretically be sharp and provide a rich, natural 3D perception, and simulated refractive blur can be induced by software and can follow the accommodation tracking state.

In one embodiment, a scanning fiber display is well suited for such a configuration because it can be configured to output only a small diameter beam in the form of a Maxwellian beam. In another embodiment, an array of small exit pupils can be created to increase the functional eyebox of the system (while also reducing the impact of light-blocking particles that may reside in the vitreous humor or cornea of the eye), such as by one or more scanning fiber displays, or by a DOE configuration such as that described with reference to FIG8K , wherein the spacing in the array of provided exit pupils ensures that at any given time, only one exit pupil is incident on the user's anatomical pupil (e.g., if the average anatomical pupil diameter is 4 mm, a configuration can include 1/2 mm exit pupils spaced approximately 4 mm apart). These exit pupils can also be switched in response to eye position so that only one and only one small exit pupil is always active at a time; thereby allowing for a denser array of exit pupils. Such users will have a large depth of focus, to which software-based blurring techniques can be added to enhance perceived depth perception.

As mentioned above, objects at optical infinity produce essentially planar wavefronts; closer objects (such as 1 meter from the eye) produce curved wavefronts (with a convex radius of curvature of approximately 1 meter). The eye's optical system needs to have sufficient optical power to bend incoming light rays so that they are ultimately focused on the retina (the convex wavefront becomes a concave wavefront and then continues to a focal point on the retina). These are fundamental functions of the eye.

In many of the embodiments described above, the light directed to the eye has been considered as being part of a continuous wavefront, with certain subsets impinging on the pupil of a particular eye. In another approach, the light directed to the eye can be effectively discretized or split into a plurality of beamlets or individual rays, each having a diameter less than approximately 0.5 mm and having a unique propagation path as part of a larger aggregate wavefront that can be functionally created by aggregating the beamlets or rays. For example, an arcuate wavefront can be approximated by aggregating multiple discrete adjacent collimated beams, each approaching the eye from an appropriate angle to represent an origin that matches the center of the radius of curvature of the desired aggregate wavefront.

When the beamlets have a diameter of about 0.5 mm or less, the beamlets can be considered to be from a pinhole lens configuration, which means that each individual beamlet is always relatively focused on the retina, regardless of the eye's state of accommodation, but the trajectory of each beamlet is affected by the state of accommodation. For example, if the beamlets approach the eye in parallel, representing a discrete, collimated light convergent wavefront, then an eye that is properly accommodated to infinity will deflect the beamlets so that they focus on the same shared point on the retina, and will appear sharp. If the eye is accommodated to, say, 1 m, the beamlets will converge to a point in front of the retina, span multiple paths, and land on multiple adjacent or partially overlapping points on the retina (i.e., appear blurry).

If the beamlets approach the eye in a diverging configuration, with their shared origin 1 meter from the viewer, 1 meter of accommodation will cause the beams to steer toward a single point on the retina, resulting in a sharp display. If the viewer accommodates to infinity, the beamlets converge to a point behind the retina, producing multiple adjacent or partially overlapping points on the retina, resulting in a blurred image. More generally, the eye's accommodation determines the degree of overlap of points on the retina, and a given pixel is "sharp" when all points are directed toward the same point on the retina and "blurred" when the points are offset from each other. This means that all beamlets with a diameter of 0.5 mm or less are always sharp, can converge, and are perceived by the eye/brain as being essentially the same as a coherent wavefront, and can be used in configurations that produce comfortable 3D virtual or augmented reality perception.

In other words, a set of multiple beamlets can be used to simulate what would happen with a larger diameter variable focus beam, where if the beamlet diameter is kept to a maximum of about 0.5mm, they maintain a relatively static degree of focus, and when desired, the perception of defocus can be produced, and the beamlet angular trajectories can be selected to produce an effect very much like a larger defocused beam (this defocusing process may be different from the Gaussian blur process used for the larger beam, but will produce a multimode point spread function that can be interpreted in a manner similar to a Gaussian blur).

In a preferred embodiment, the beamlets are not mechanically deflected to create this convergent focusing effect, but rather the eye receives a superset of a large number of beamlets that simultaneously includes multiple angles of incidence and multiple locations of intersection of the beamlets with the pupil; to represent a given pixel from a particular viewing distance, a subset of the beamlets from the superset that include the appropriate angles of incidence and intersections with the pupil (as if they were emitted from the same shared origin in space) become of matching color and intensity to represent the convergent wavefront, while the beamlets in the superset that are not aligned with the shared origin do not become of the aforementioned color and intensity (but some of them may become of some other color and intensity level to represent, for example, different pixels).

Referring to FIG11A , in a discretized wavefront display configuration, each of a plurality of incoming beamlets (332) passes through a small exit pupil (330) relative to the eye (58). Referring to FIG11B , a subset (334) of a set of beamlets (332) can be driven with matching colors and intensity levels to be perceived as if they were part of the same larger ray (the subsets 334 combined together can be considered "convergent beams"). In this case, the subsets of beamlets are parallel to each other, representing collimated, convergent beams from optical infinity (such as light from a distant mountain). The eye is visually accommodated to infinity, and therefore, the subsets of beamlets are offset by the eye's cornea and lens so that they all fall on substantially the same location on the retina and are perceived as comprising a single, sharp pixel.

FIG11C shows another subset of beamlets, which represents a converged collimated beam (336) from the right side of the field of view of the user's eye (58) if the eye (58) is viewed from above in a coronal plane view. Again, the eye is shown as being accommodated to infinity, so that the beamlets fall on the same point on the retina and the pixel is perceived as sharp. In contrast, if a different subset of beamlets is selected (arriving at the eye as a diverging fan of rays), these beamlets will not fall on the same location on the retina (and be perceived as sharp) until the eye changes its accommodation to a near point that matches the geometric origin of the fan of rays.

The pattern of intersections of the beamlets with the anatomical pupil of the eye (i.e., the pattern of the exit pupil) can be organized using a configuration such as a cross-sectioned hexagonal lattice (e.g., as shown in FIG12A ) or a tetragonal lattice or other two-dimensional array. Furthermore, a three-dimensional exit pupil array can be generated, as well as a time-varying exit pupil array.

A variety of configurations can be used to create discrete aggregate wavefronts, such as a microdisplay or microprojector array placed optically conjugate with the exit pupil of an optical viewing element, the microdisplay or microprojector array coupled to a direct field of view substrate (such as a spectacle lens) to project light directly to the eye without the need for additional intermediate optical viewing elements, continuous spatial light modulation array technology, or waveguide technology (such as that described with reference to FIG8K).

Referring to FIG12A , in one embodiment, a light field can be generated by bundling together a group of small projectors or display units (such as scanning light displays). FIG12A shows a hexagonal dot-matrix projector beam (338) that can, for example, produce a 7 mm diameter hexagonal array, where each fiber display outputs a sub-image (340). If an optical system (such as a lens) is placed in front of this array so that the array is optically conjugate with the entrance pupil of the eye, an image of the array is produced at the pupil of the eye, as shown in FIG12B , which essentially provides the same optical setup as the embodiment of FIG11A .

Each small exit pupil of this configuration is produced by a dedicated small display (such as a scanning fiber display) in the projector beam (338). Optically, it is as if the entire hexagonal array (338) is placed right inside the anatomical pupil (45). These embodiments are a means for driving different sub-images into different small exit pupils within the larger anatomical entrance pupil (45) of the eye, including a superset of beamlets with multiple angles of incidence and multiple points of intersection with the pupil. Each of the separate projectors or displays can be driven with a slightly different image to produce sub-images that separate different sets of rays to be driven with different light intensities and colors.

In one variation, strict image conjugation can be produced, as in the embodiment of FIG12B , where there is a direct one-to-one mapping between the array ( 338 ) and the pupil ( 45 ). In another variation, the spacing between the displays in the array and the optical system ( lens 342 in FIG12B ) can be changed so that rather than obtaining a conjugate mapping of the array to the pupil, the pupil can capture rays from an array located at some other distance. With such a configuration, one can still obtain an angular diversity of beams from which a discretized aggregate wavefront representation can be produced, but the mathematics of how to drive the rays and with what power and intensity may become more complex (although, on the other hand, such a configuration may be considered simpler from the perspective of viewing the optics). The mathematics involved in light field image capture can be utilized for these calculations.

Referring to FIG13A , another light field generation embodiment is shown in which an array of microdisplays or microlight projectors (346) can be coupled to a frame (344, such as a spectacle frame) for placement in front of the eye (58). The configuration shown is a non-conjugate arrangement in which no large optical elements are interposed between the display (e.g., a scanning light display) of the array (346) and the eye (58). One can imagine a pair of glasses with multiple displays (such as scanning fiber engines) coupled to the glasses, positioned orthogonally to the surface of the glasses and angled inwardly to point toward the user's pupil. Each display can be configured to generate a set of rays representing a different element of a superset of beamlets.

With such a configuration, the user's anatomical pupil (45) will receive results similar to those received in the embodiment described with reference to FIG11A, where each point of the user's pupil receives rays from different displays having multiple angles of incidence and multiple intersection points. FIG13B illustrates a non-conjugate configuration similar to FIG13A, except that the embodiment of FIG13B has a reflective surface (348) that facilitates moving the display array (346) away from the field of view of the eye (58), while also allowing a view of the real world (144) to be displayed through the reflective surface (348).

Another configuration is provided for producing the angular divergence required for a discretized aggregate wavefront display. To optimize this configuration, the size of the display can be minimized. A scanning fiber display that can be used as a display can have a baseline diameter in the range of 1 mm, but reduction in the housing and projection lens may reduce the diameter of such a display to about 0.5 mm or less, which has little impact on the user. In the case of a fiber scanning display array, another streamlined geometric optimization can be achieved by directly coupling a collimating lens (which can, for example, include a gradient reflectivity, or "GRIN", lens, a conventional curved lens, or a diffractive lens) to the tip of the scanning fiber itself. For example, referring to Figure 13D, a GRIN lens (354) is shown as being fused to the end of a single mode optical fiber. An actuator (350: such as a piezoelectric actuator) is coupled to the optical fiber (352) and can be used to scan the optical fiber tip.

In another embodiment, the ends of the optical fibers can be shaped into a hemispherical shape using a curved polishing process to produce a lens effect. In another embodiment, a standard refractive lens can be coupled to the end of each optical fiber using an adhesive. In another embodiment, a lens can be constructed using a small amount of transmissive polymeric material or glass (such as epoxy). In another embodiment, the tips of the optical fibers can be melted to produce a curved surface for achieving a lens effect.

FIG13C-2 shows an embodiment in which a display configuration such as that shown in FIG13D (i.e., a scanned light display with GRIN lenses; shown in close-up view in FIG13C-1) can be connected together by a single transparent substrate (356) that preferably has a refractive index closely matched to the cladding of the optical fiber (352) so that the optical fiber itself is not very visible to the outside world view across the shown assembly (if the cladding refractive index matching is achieved accurately, the larger cladding/shell becomes transparent and only the thin core (preferably about 3 microns in diameter) will obstruct the view). In one embodiment, the display matrix (358) can all be tilted inward so as to face the user's anatomical pupil (in another embodiment, they are parallel to each other, but such a configuration is less effective).

Referring to FIG13E , another embodiment is shown in which, rather than using a circular optical fiber to move cyclically, a series of thin planar waveguides (358) are configured to be suspended relative to a larger substrate structure (356). In one variation, the substrate (356) can be moved to produce cyclic motion of the planar waveguide relative to the substrate structure (i.e., motion at the resonant frequency of the suspension member 358). In another variation, a piezoelectric actuator or other actuator can be used to actuate the suspended waveguide portion (358) relative to the substrate. Image illumination information can be injected from the right side (360) of the substrate structure to couple to the suspended waveguide portion (358), for example. In one embodiment, the substrate (356) can include a waveguide configured (such as by the integrated DOE configuration described above) to totally internally reflect incoming light (360) along its length and redirect the light to the suspended waveguide portion (358). When a user looks at the suspended waveguide portion (358) and sees the real world (144) behind it, the planar waveguide is configured to minimize any dispersion and/or focus changes through its planar form factor.

In the context of introducing a discretized aggregate wavefront display, it is very valuable to generate some angular diversity for each point in the exit pupil of the eye. In other words, it is ideal to have multiple incoming beams represent each pixel in the displayed image. Referring to Figures 13F-1 and 13F-2, one way to obtain further angular and spatial diversity is to use a multi-core fiber and place a lens such as a GRIN lens at the exit pupil so that the outgoing beam is deflected through a single node (366) that can then be scanned back and forth in a scanning fiber type setup (such as by a piezoelectric actuator 368). If the retinal conjugate is placed in a plane defined by the end of the GRIN lens, a display can be created that is functionally equivalent to the general case discretized aggregate wavefront configuration described above.

Referring to FIG13G , a similar effect is achieved not by using a lens, but by scanning a facet of a multicore system located at the correct conjugate of the optical system (372), with the goal of producing a higher angular and spatial diversity of the light beam. In other words, rather than having a single individually scanned light display as in the bundled example of FIG12A above, a portion of the necessary angular and spatial diversity can be produced by using multiple cores, thereby creating a plane that can be relayed by a waveguide. Referring to FIG13H , a multicore optical fiber (362) can be scanned (such as by a piezoelectric actuator 368) to produce a set of beamlets with multiple angles of incidence and multiple intersection points, which can be relayed to the eye (58) via a waveguide (370). Thus, in one embodiment, a collimated light field image can be injected into the waveguide, and without any additional refocusing elements, the light field display can be directly translated to the human eye.

Figures 13I-13L show some commercially available multi-core optical fiber (362) configurations (from suppliers such as Mitsubishi Cable Industries, Ltd. of Japan), including a variant with a rectangular cross-section (363), as well as variants with a flat exit face (372) and an inclined exit face (374).

Referring to FIG. 13M , some additional angular diversity can be generated by feeding a linear display array ( 378 ), such as a scanning fiber display, to the waveguide ( 376 ).

Reference is made to Figures 14A-14F , which illustrate another set of configurations for generating a fixed-viewpoint lightfield display. Referring back to Figure 11A , if a two-dimensional plane is generated that intersects all beamlets from the left, each beamlet will have a specific intersection point with that plane. If another plane is generated at a different distance from the left, all beamlets will intersect that plane at different locations. Returning to Figure 14A , if individual locations on each of two or more planes can be allowed to selectively transmit or block optical radiation passing therethrough, such a multi-plane configuration can be used to selectively generate a lightfield by independently modulating individual beamlets.

The basic embodiment of FIG14A shows two spatial light modulators, such as liquid crystal display panels (380, 382; in other embodiments, they can be MEMS shutter displays or DLP DMD arrays), which can be independently controlled to block or transmit different rays based on high resolution. For example, referring to FIG14A, if the second panel (382) blocks or attenuates ray transmission at point "a" (384), then all of the rays shown are blocked; but if only the first panel (380) blocks or attenuates ray transmission at point "b" (386), then only the lower incoming ray (388) is blocked/attenuated, while the other rays are transmitted to the pupil (45). Each of the controllable panels or planes can be considered a "spatial light modulator" or "fatte". The intensity of each transmitted beam passing through a series of SLMs depends on the combination of the transparency of the individual pixels in the array of SLMs. Therefore, without any kind of lens elements, a set of thin beams with multiple angles and intersections (or "light fields") can be produced using multiple stacked SLMs. A greater number of SLMs than two provides more opportunities to control which light beams are selectively attenuated.

As briefly described above, in addition to using stacked liquid crystal displays as SLMs, DMD device panels in DLP systems can also be stacked to be used as SLMs, and may be superior to liquid crystal systems as SLMs because they have more efficient light transmission (with a mirror element in a first state, it can be very efficiently reflected to the next element on the way to the eye; with a mirror element in a second state, the mirror angle can be moved, for example, by 12 degrees to direct the light out of the path to the eye). Referring to Figure 14B, in one DMD embodiment, two DMDs (390, 390) can be used in series with a pair of lenses (394, 396) in a periscope-type configuration to maintain a large amount of light transmission from the real world (144) to the user's eye (58). The embodiment of Figure 14C provides six different DMD (402, 404, 406, 408, 410, 412) planes to intervene in the SLM function when the light beam is routed to the eye (58); and provides two lenses (398, 400) to achieve beam steering.

FIG14D shows a more complex periscope-type arrangement with up to four DMDs (422, 424, 426, 428) for SLM functionality and four lenses (414, 420, 416, 418); this configuration is designed to ensure that the image does not flip upside down when transmitted to the eye (58). FIG14E shows an embodiment in which light can be reflected between two different DMD devices (430, 432) without an intermediate lens (the lenses in the above designs can be used to integrate image information from the real world in such a configuration), and also shows a "hall of mirrors" arrangement in which the display can be viewed through a "hall of mirrors" and operates in a manner substantially similar to that shown in FIG14A. FIG14F shows an embodiment in which the non-display portions of two opposing DMD chips (434, 436) can be covered by a reflective layer to transmit light to and from the active display area (438, 440) of the DMD chip. In other embodiments, instead of a DMD for SLM functionality, an array of sliding MEMS shutters (e.g., those available from suppliers such as Pixtronics, a subsidiary of Qualcomm, Inc.) can be used to transmit or block light. In another embodiment, an array of small shutters that are moved out of position to provide light-transmitting apertures can be similarly aggregated to implement SLM functionality.

A light field of numerous thin beams (i.e., less than about 0.5 mm in diameter) can be injected and propagated through a waveguide or other optical system. For example, a conventional "birdbath" type optical system is suitable for transmitting light for a light field input, or a free-form optical element design described below, or any number of waveguide configurations. Figures 15A-15C illustrate the use of a wedge-shaped waveguide (442) and multiple light sources as another configuration for creating a light field. Referring to Figure 15A, light can be injected into the wedge-shaped waveguide (442) from two different locations/displays (444, 446) and emerge according to the total internal reflection properties of the wedge-shaped waveguide at different angles (448) based on the point of injection into the waveguide.

Referring to FIG15B , if an array of linear displays (such as scanning fiber displays) is created (450) projected onto the end of a waveguide as shown, a large number of beams (452) with diverse angles will emerge from the waveguide along one dimension, as shown in FIG15C . In fact, if another array of linear displays is envisioned that is injected into the end of the waveguide, but at slightly different angles, a plurality of beams with diverse angles can be generated that emerge in a fan-shaped pattern similar to that shown in FIG15C , but on a vertical axis; these beams can be used together to generate a two-dimensional fan of rays emerging from each position in the waveguide. Thus, another configuration is provided for generating angular diversity to form a light field display using one or more arrays of scanning fiber displays (or alternatively, using other displays that meet the spatial requirements, such as a micro DLP projection configuration).

Alternatively, a stack of SLM devices can be utilized as the input to the wedge-shaped waveguides shown here, in which case the light field output from the SLM configuration can be used as the output of a configuration such as that shown in FIG15C , rather than using a direct view of the SLM output as described above. An important concept of this paper is that while conventional waveguides are best suited for successfully relaying collimated beams, for light fields of small diameter collimated beams, conventional waveguide technology can be used to further manipulate the output of such light field systems that are injected into one side of the waveguide (into the wedge waveguide) for beam size/collimation reasons.

In another related embodiment, rather than using multiple separate displays for projection, a light field can be separated and injected into a waveguide using a multi-core optical fiber. Further, a time-varying light field can be used as input so that instead of producing a static distribution of beamlets from the light field, there can be some dynamic element that can systematically change the path of a group of beams. This can be achieved using a component such as a waveguide with an embedded DOE (e.g., the DOE described above with reference to Figures 8B-8N, or the liquid crystal layer described with reference to Figure 7B), where two optical paths are generated (a smaller total internal reflection path in which the liquid crystal layer is placed in a first voltage state to have a refractive index that does not match the other substrate materials, which causes total internal reflection to occur only along its waveguide; and a larger total internal reflection optical path in which the liquid crystal layer is placed in a second voltage state to have a refractive index that matches the other substrate materials, causing light to be totally internally reflected through a composite waveguide that includes both the liquid crystal portion and the other substrate portion). Similarly, a wedge-shaped waveguide can be configured to have a dual-mode total internal reflection paradigm (e.g., in one variation, the wedge-shaped element can be configured so that when the liquid crystal portion is activated, not only does the pitch change, but the angle at which the beam is reflected also changes).

One embodiment of a scanned light display can be simply characterized as a scanned fiber display with a lens at the end of a scanned fiber. Many lens types are suitable, such as a GRIN lens, which can be used to collimate or focus light to a point that is smaller than the mode field diameter of the fiber, thereby providing the advantages of increasing the numerical aperture (or "NA") and enclosing an optical invariant that is negatively correlated with the spot size. From a display perspective, a smaller spot size generally facilitates higher resolution opportunities, which is generally preferred. In one embodiment, the GRIN lens can be sufficiently long relative to the fiber so that it can include a vibrating element (i.e., rather than the fiber end vibrating as in conventional scanned fiber displays), which is referred to as a "scanned GRIN lens display" configuration.

In another embodiment, a diffractive lens (i.e., patterned onto the fiber) can be utilized at the exit end of a scanning fiber display. In another embodiment, a curved mirror can be placed at the end of a fiber operating in a reflective configuration. Essentially, any configuration known to be capable of collimating and focusing a light beam can be used at the end of a scanning fiber to produce a suitable scanned light display.

Two important effects of coupling a lens to or including the end of a scanning fiber (i.e., compared to a configuration in which an uncoupled lens can be used to direct light exiting the fiber) are: a) the exiting light can be collimated to avoid the use of other external optical elements to do this; and b) the NA, or angle of the cone of light exiting the end of the single-mode fiber core, can be increased, thereby reducing the fiber's associated spot size and increasing the usable resolution of the display.

As described above, a lens such as a GRIN lens can be fused or otherwise coupled to the end of an optical fiber, or formed as part of the end of an optical fiber using techniques such as polishing. In one embodiment, a typical optical fiber having an NA of approximately 0.13 or 0.14 can have a spot size of approximately 3 microns (also known as the "mode field diameter" of the optical fiber at a given NA). Given industry-standard display resolution paradigms (e.g., typical microdisplay technologies such as LCDs or organic light-emitting diodes, or "OLEDs" have a spot size of approximately 5 microns), this provides the potential for relatively high-resolution displays. As such, the above-described scanning light display can have a minimum pixel pitch of 3/5 that of conventional displays; further, by using a lens at the end of the optical fiber, the above-described configuration can produce a spot size in the range of 1-2 microns.

In another embodiment, instead of using a scanning cylindrical fiber, a suspended portion of a waveguide (such as a waveguide created using micromachining processes such as masking and etching rather than drawn microfiber technology) can be placed in a scanning oscillating motion and can be coordinated with a lens at the output end.

In another embodiment, the increased numerical aperture of the optical fiber to be scanned can be created using a diffuser (i.e., a diffuser configured to scatter light and create a large NA) covering the optical fiber's exit end. In one variation, the diffuser can be created by etching the end of the optical fiber to create a small band that scatters light; in another variation, the scattering band can be created using welding or sandblasting techniques, or direct grinding/gluing techniques. In yet another variation, an engineered diffuser similar to a diffractive element can be created to maintain a clean spot size with a desired NA (which is incorporated into the intention of using the diffractive lens described above).

Referring to FIG16A , an optical fiber array (454) is shown coupled to a coupler (456) configured to hold the optical fibers together in parallel so that the ends of the optical fibers are ground and polished to have output edges that form a critical angle (458; for example, 45 degrees for most glasses) with the longitudinal axis of the input optical fibers so that light exiting the bevel exits as if it had passed through a prism and is bent and becomes almost parallel to the surface of the polished face. The light beams exiting the optical fibers (454) in the bundle will overlap but will not be out of phase in the longitudinal direction due to different path lengths (for example, for different cores, the path lengths from the exit bevel to the focusing lens are different for different cores as shown in FIG16B ).

The fact that the X-axis separation in the fiber bundle before exiting the inclined surface will become the Z-axis separation facilitates the generation of a multi-focus light source with this configuration. In another embodiment, instead of using multiple single-mode fibers bundled/coupled, a multi-core optical fiber such as that manufactured by Mitsubishi Cable Industries, Ltd. of Japan can be angle polished.

In one embodiment, if an optical fiber is polished at a 45-degree angle and then covered with a reflective element (such as a mirror coating), the exit light can be reflected from the polished surface and emerge from the side of the optical fiber (in one embodiment, at a location where a flat-polished exit window has been created on the side of the optical fiber). In this way, when the optical fiber is scanned in what is generally referred to as an X-Y Cartesian coordinate system, the optical fiber functionally performs an equivalent X-Z scan, where the distance varies during the scan. This type of configuration can also be used to advantage to change the focus of a display.

Multi-core fibers can be configured to play a role in display resolution enhancement (i.e., higher resolution). For example, in one embodiment, if individual pixel data is sent to 19 cores tightly bundled together in a multi-core fiber, and the cluster is scanned across in a sparse spiral pattern with a spiral pitch approximately equal to the diameter of the multi-core, then the back-and-forth scanning effectively produces a display resolution that is approximately 19 times the resolution of a similarly scanned single-core fiber. In practice, it is more efficient to position the fibers more sparsely relative to each other, as in the configuration of FIG16C , which has 7 clusters (464; 7 is for illustrative purposes only, as it is an effective tiled/hexagonal lattice pattern; other patterns or numbers may be used; for example, 19 clusters, and the configuration may be larger or smaller), each cluster having 3 fibers housed in conduits (462).

With the sparse configuration shown in FIG16C , multi-core scanning will scan the local area of each core separately, as opposed to a configuration where the cores are all tightly packed together for scanning (where the cores overlap during scanning, and if the cores are too close together, their NA will not be large enough, and the tightly packed cores will be somewhat blurred and will not produce a resolvable spot for display). Therefore, to increase resolution, it is preferable to have a sparse tiling rather than a very dense tiling, although both are effective.

The idea that tightly bundled scanned cores can produce blur at the display can be used to advantage in one embodiment by densely bundling multiple cores (i.e., three cores carrying red, green, and blue light) so that each three cores form three overlapping light spots characterized by red, green, and blue light. This configuration allows for an RGB display that does not require combining red, green, and blue into a single-mode core, an advantage since conventional approaches for combining multiple (e.g., three) small lightwaves into a single core lose significant optical energy. Referring to FIG16C , in one embodiment, each tightly bundled cluster of three fiber cores contains one core relaying red light, one core relaying green light, and one core relaying blue light, where the three cores are close enough together that their positional differences are not discernible by subsequent relay optics, thereby forming effectively overlapping RGB pixels. Thus, the sparse tiling of the seven clusters yields increased resolution, while the tight bundling of the three cores within the cluster facilitates seamless color mixing without the need for smooth RGB fiber combiners (e.g., those using wavelength division multiplexing or evanescent wave coupling).

Referring back to FIG16D , in another simpler variation, there may be only one cluster (464) housed in a conduit (468), for example, red/green/blue (in another embodiment, another core for infrared may be added for applications such as eye tracking). In another embodiment, additional cores may be placed in a tight cluster carrying additional wavelengths of light to include a multi-master display to increase color saturation. Referring to FIG16E , in another embodiment, a sparse array of single cores (470) may be used; in a variation, a combined red, green, and blue color may be used, all within the conduit (466); such a configuration may work, although it does not provide much in the way of resolution gain, but is not optimal for the red/green/blue combination.

Multi-core optical fibers can also be used to generate lightfield displays. In practice, rather than spacing the cores so far apart that they are not scanned over every other local area of the display panel, as described above in the context of using lightfield displays to generate scanned light displays, it is desirable to scan multiple fibers densely packed back and forth, since each beam generated represents a specific portion of the lightfield. If the optical fibers have a small NA, the light emitted from the bundled fiber tips can be relatively narrow; a lightfield configuration can exploit this property and have an arrangement in which multiple slightly different beams are received from the array at the anatomical pupil. Thus, there are optical configurations that scan multiple cores that are functionally equivalent to an array of single scanning fiber modules, thereby generating a lightfield by scanning multiple cores rather than a set of single-mode optical fibers.

In one embodiment, multi-core phased array technology can be used to create a large exit pupil variable wavefront configuration to facilitate three-dimensional perception. A single laser configuration with a phase modulator is described above. In a multi-core embodiment, phase delays can be introduced into different channels of a multi-core fiber so that light from a single laser is injected into all cores of the multi-core configuration, thereby generating mutual coherence.

In one embodiment, a multi-core optical fiber can be combined with a lens such as a GRIN lens. Such a lens can be, for example, a refractive lens, a diffractive lens, or a polished surface used as a lens. The lens can be a single optical surface, or can include multiple stacked optical surfaces. In fact, in addition to a single lens having an extended multi-core diameter, it is ideal to also provide a small lens array at the light exit point of each core in the multi-core, for example. Figure 16F shows an embodiment in which a multi-core optical fiber (470) injects multiple light beams into a lens (472) such as a GRIN lens. The lens collects the light beams to a focal point (474) located in the space in front of the lens. In many conventional configurations, the light beams will be emitted from the multi-core optical fiber when diverging. For example, a GRIN or other lens is configured to direct the light beam to a single point and calibrate the light beam, thereby scanning the calibration results back and forth for the light field display.

Referring to FIG16G , a small lens (478) can be placed in front of each core in a multi-core (476) configuration, and these lenses can be used for calibration; the shared lens (480) can then be configured to focus the collimated light onto a diffraction-limited spot (482) aligned for all three spots. The net result of such a configuration is that, as shown, by combining three collimated narrow beams with narrow NA, all three beams are effectively combined into a much larger emission angle, which translates to a smaller spot size, for example, in a head-mounted optical display system that is next in the light delivery chain to the user.

Referring to FIG16H , one embodiment features a multi-core optical fiber (476) having an array of lenslets (478) that feeds light into an array of prisms (484) that deflects the light beams produced by the individual cores to a common point. Alternatively, the lenslet array can be offset relative to the cores to deflect and focus the light to a single point. Such a configuration can be used to increase numerical aperture.

Referring to FIG16I , a dual-step configuration is shown with an array of lenslets (478) capturing light from a multi-core fiber (476), followed by a shared lens (486) that focuses the beam to a single point (488). This type of configuration can be used to increase the numerical aperture. As mentioned above, a larger NA corresponds to a smaller pixel size and a higher possible display resolution.

Referring to FIG16J , a reflective device (494; such as a DMD module of a DLP system) can be used to scan an array of angled optical fibers that can be held together with a coupler (456) such as described above. By coupling to multiple single-core optical fibers (454) within the array, or conversely through multi-core optical fibers, the superimposed light can be directed through one or more focusing lenses (490, 492) to produce multifocal beams; by superimposing and angling the array, different sources are at different distances from the focusing lens, so that when the beams are emitted from the lens (492) and directed to the retina (54) of the user's eye (58), different degrees of focus are produced in these beams. For example, the farthest optical path/beam can be set as a collimated beam representing an optically infinite focus position. The closer paths/beams can be associated with diverging spherical wavefronts at closer focus positions.

The multifocal beam can be delivered to a scanning mirror that can be configured to produce a raster scan (or, for example, a Lissajous or spiral scan pattern) of the multifocal beam, which can then pass through a series of focusing lenses before reaching the cornea and lens of the eye. The multiple beams emerging from the lenses produce distinct pixels or voxels with superimposed, variable focal lengths.

In one embodiment, different data can be written to each of the optical modulation channels located at the front end, thereby generating an image that is projected into the eye through one or more focusing elements. By changing the focal length of the lens (i.e., through visual accommodation), the user can bring different incoming pixels into and out of focus, as shown in Figures 16K and 16L, in which the lens is at different Z-axis positions. In another embodiment, the fiber array can be actuated/moved by a piezoelectric actuator. In another embodiment, when the piezoelectric actuator is activated, a relatively thin ribbon array can be resonated in a suspended form along an axis perpendicular to the array fiber arrangement (i.e., in the thin direction of the ribbon). In a variation, a single piezoelectric actuator can be used to vibrate and scan in a perpendicular long axis. In another embodiment, a slow scan can be performed along the long axis using single mirror axis scanning while the fiber ribbon resonates.

Referring to FIG16M, an array (496) of scanning fiber displays (498) can be advantageously bundled/tiled to achieve an effective resolution increase, with the goal that with such a configuration, each scanning fiber in the bundle is configured to write to a different portion of the image plane (550), such as shown in FIG16N, where each portion of the image plane is addressed by emission from at least one fiber bundle. In other embodiments, these optical configurations can be used to allow the beams to be slightly magnified as they exit the fibers, thereby slightly overlapping in a hexagonal or other dot pattern, and incident upon the display plane, thereby having a good fill factor, while also maintaining a sufficiently small spot size in the image plane and understanding that there is a slight magnification in the image plane.

Rather than having individual lenses located at the end of each scanning fiber housing, in one embodiment, a monolithic lenslet array can be utilized so that the lenslets are packed as tightly together as possible. This allows for a smaller spot size in the image plane because a lower magnification can be used in the optical system. Thus, fiber scanning display arrays can be used to increase the resolution of the display, or in other words, they can be used to increase the field of view of the display because each engine is used to scan a different portion of the field of view.

For light field configurations, the emissions can ideally overlap in the image plane. In one embodiment, a light field display can be generated using multiple small-diameter optical fibers that are scanned in space. For example, rather than having all fibers address different portions of the image plane as described above, there can be more overlap, more fibers can be tilted inward, or the lens power can be modified so that the small spot size is not conjugate to the tiled image plane configuration. Such a configuration can be used to generate a light field display by scanning a large number of small-diameter rays back and forth, intercepted in the same physical space.

Referring back to FIG. 12B , one way to create a light field display is to collimate the output of the elements on the left side through a narrow beam and then conjugate the projection array to the pupil on the right side.

Referring to Figure 16O, for a common substrate block (502), a single actuator can be used to uniformly actuate multiple optical fibers (506) together. Similar configurations are described above with reference to Figures 13C-1 and 13C-2. In practice, it may be difficult to keep all optical fibers at the same resonant frequency, vibrate in an ideal phase relationship with each other, or have the same hanging size from the substrate block. To meet this challenge, the tips of the optical fibers can be mechanically coupled to a matrix or sheet (504) (such as a very thin, very light but rigid graphene sheet). Through this type of coupling, the entire array can vibrate similarly and have the same phase relationship. In another embodiment, a carbon nanotube matrix can be used to couple the optical fibers, or extremely thin flat glass (such as the type used in the production of liquid crystal display panels) can be coupled to the ends of the optical fibers. Further, a laser or other precision cutting equipment can be used to cut all associated optical fibers to the same hanging length.

Referring to FIG17 , in one embodiment, it is desirable to have a contact lens that interfaces directly with the cornea and is configured to facilitate focusing of the eye on a display that is very close (e.g., the typical distance between the cornea and a spectacle lens). Rather than arranging the optical lens as a contact lens, in one variation, the lens can include a selective filter. FIG7 shows a drawing (508) of what can be considered a “notch filter” in that it is designed to block only specific wavelength bands, such as 450 nm (blue peak), 530 nm (green), and 650 nm, and generally pass or transmit other wavelength bands. In one embodiment, multiple dielectric coatings can be polymerized to provide the notch filtering function.

Such a filtering configuration can be coupled with a scanning fiber display that produces very narrow band illumination for red, green, and blue, and a contact lens with notch filtering will block all light from the display (e.g., a microdisplay such as an OLED display, mounted in the space typically occupied by a spectacle lens) except for the transmitted wavelength. A narrow pinhole can be created in the middle of the contact lens filter layer/film so that the small hole (i.e., less than about 1.5 mm in diameter) allows the passage of wavelengths that would otherwise be blocked. Thus, the resulting pinhole lens configuration operates in a pinhole manner for red, green, and blue, capturing only the image from the microdisplay while light from the real world (typically broadband illumination) passes relatively unimpeded through the contact lens. Thus, a large depth of focus virtual reality configuration can be assembled and operated. In another embodiment, due to the large depth of focus pinhole configuration, a collimated image emitted from the waveguide will be visible at the retina.

Creating a display that can change its depth of focus over time is very useful. For example, in one embodiment, a display can be configured to have different display modes that can be selected by an operator (preferably quickly switched between the two modes upon the operator's command), such as a first mode that combines a very large depth of focus with a small exit pupil diameter (i.e., so that everything is always sharp), and a second mode characterized by a larger exit pupil and a narrower depth of focus. In operation, if a user is playing a three-dimensional video game that includes objects perceived at a large depth of field, the operator may select the first mode; alternatively, if the user is typing long prose (i.e., over an extended period of time) using a two-dimensional word processing display configuration, it may be more desirable to switch to the second mode to take advantage of the larger exit pupil and sharper image.

In another embodiment, it may be desirable to have a multi-depth of focus display configuration in which certain sub-images are presented with a large depth of focus, while other sub-images are presented with a small depth of focus. For example, one configuration may have red and blue wavelength channels, both of which are configured with very small exit pupils so that they are always in focus. The green channel then has only a large exit pupil configuration with multiple depth planes (i.e., because the human visual accommodation system preferentially targets the green wavelength to optimize focus). Therefore, in order to reduce the cost associated with having too many elements represented by all red, green, and blue depth planes, the green wavelength can be prioritized and represented at various different wavefront levels. The red and blue can be relegated to being represented by a more Maxwellian approach (e.g., software can be used to introduce Gaussian-level blur, as described above with reference to the Maxwell display). This display will present multiple depths of focus simultaneously.

As mentioned above, there are multiple parts of the retina with a higher density of light sensors. The fovea, for example, is typically populated with approximately 120 cones per degree of visibility. In the past, display systems have been created that use eye or gaze tracking as input. These systems save computing resources by only generating a true high-resolution rendering for the location where the user is looking at the moment, while rendering a low-resolution rendering for the rest of the retina. In such a configuration, the positions of the high-resolution and low-resolution portions can dynamically change with the tracked gaze position, which can be called a "foveated display."

Improvements to this type of configuration can include a scanning fiber display with a pattern spacing that can dynamically follow the tracked eye gaze. For example, when a typical scanning fiber display is operated in a spiral mode, as shown in FIG18 (for comparison, the leftmost image portion 510 in FIG18 shows the spiral motion pattern of a scanned multi-core optical fiber 514, and the rightmost image portion 512 in FIG18 shows the spiral motion pattern of a scanned single optical fiber 516), a fixed pattern spacing provides uniform display resolution.

In a foveated display configuration, a non-uniform scan spacing can be utilized, where smaller/tighter spacing (and therefore higher resolution) dynamically follows the detected gaze location. For example, if the user's gaze is detected moving toward the edge of the display screen, the spiral can be more densely packed at that location, creating a circular scan pattern for the high-resolution portion, with the remainder of the display in low-resolution mode. In configurations where gaps can be created in the portion of the display in low-resolution mode, blur is intentionally dynamically generated to smooth the transition between scans, as well as the transition from high-resolution to low-resolution scan spacing.

The term "lightfield" can be used to describe a 3D representation of light transmitted from an object to the viewer's eye. However, a transmissive optical display only reflects light to the eye, not the absence of light. Ambient light from the real world is added to any light representing the virtual object. In other words, if the virtual object presented to the eye contains black or extremely dark portions, ambient light from the real world can pass through this dark portion and obscure the portion that would otherwise be dark.

In summary, there is a need to be able to present dark virtual objects against a light, real-world background, with the dark virtual objects appearing to occupy a volume at a desired viewing distance; that is, it is useful to create a "darkfield" representation of the dark virtual object, in which the absence of light is perceived as being located at a specific point in space. Certain aspects of the spatial light modulator or "SLM" configuration described above are relevant even in well-lit, real-world environments, with respect to masking elements and presenting information to the user's eye so that he or she perceives the virtual object's darkfield. As described above, for a light-sensitive system such as the eye, one way to achieve selective perception of a darkfield is to selectively attenuate light from those portions of the display, since the primary display system involves the manipulation and presentation of light; in other words, a darkfield cannot be exclusively projected—an absence of illumination can be perceived as a darkfield, and therefore, configurations for selective illumination attenuation have been developed.

Referring back to the introduction to the SLM configuration, one way to selectively attenuate dark field perception is to block all light from one angle while allowing light from other angles to pass. This can be accomplished using multiple SLM planes, which include elements such as liquid crystals (which are not optimal due to their relatively low transparency in the transmissive state), DMD elements of DLP systems (which have relatively high transmission/reflectivity in this mode), and MEMS arrays or shutters configured to controllably block or allow light radiation to pass, as described above.

For appropriate liquid crystal display ("LCD") configurations, cholesteric LCD arrays can be utilized as controllable shielding/blocking arrays. In contrast to the conventional LCD paradigm, where the polarization state changes with voltage, for cholesteric LCD configurations, pigments are bonded to the liquid crystal molecules, which then physically tile in response to an applied voltage. Such configurations can be designed to achieve greater transparency than conventional LCDs when in transmissive mode, without the need for polarizing film stacks as with conventional LCDs.

In another embodiment, multiple layers of controllably interrupted patterns can be utilized to controllably block selected light presentations via the moiré effect. For example, in one configuration, two attenuation pattern arrays can be presented to a user's eye at a sufficiently close distance. Each array can, for example, comprise precisely spaced sine waves printed or painted on a transparent planar material (such as a glass substrate). Thus, when a viewer views either pattern individually, the view is substantially transparent. However, when the viewer views both patterns simultaneously, even if the attenuation patterns are sequentially positioned relatively close to the eye, the viewer sees only a spatial beat-frequency moiré attenuation pattern.

The beat frequency depends on the pattern spacing on the two attenuation planes, so in one embodiment, the attenuation pattern for selectively blocking specific light transmission for dark field perception can be generated using two sequential patterns, each of which is otherwise transparent to the user, but together produce a spatial beat frequency moiré attenuation pattern selected as the attenuation for dark field perception desired in the augmented reality system.

In another embodiment, a multi-view display-style masking plate can be used to create a controllable masking paradigm for achieving a dark field effect. For example, one configuration can include a pinhole layer that completely blocks light except for the apertures and pinholes, along with a selective attenuation layer in series. This attenuation layer can include an LCD, DLP system, or other selective attenuation layer configurations such as those described above. In one case, the pinhole layer is placed at a typical eyeglass lens distance of approximately 30 mm from the cornea, with the selective attenuation plate positioned on the side of the eye opposite the pinhole array. This can produce a sharp mechanical edge perceived in space. Essentially, if the configuration allows light at certain angles to pass while blocking or shielding other light, the perception of an extremely sharp pattern (such as a sharp edge projection) can be produced. In another related embodiment, the pinhole array layer can be replaced by a second dynamic attenuation layer to provide a somewhat similar configuration, but with more control than a static pinhole array layer (which can be simulated, but need not be).

In another related embodiment, the pinholes can be replaced by cylindrical lenses. The same pattern in the pinhole array layer configuration can be achieved, but with cylindrical lenses, and the array is not limited to extremely small pinhole geometries. To avoid the distortion caused by the lens seen by the eye when viewing the real world, a second lens array can be added on the side of the pinhole or lens array opposite the side closest to the eye to perform compensation and provide viewing illumination with a substantially zero-power telescope configuration.

In another embodiment, rather than physically blocking the light used to shade and create a dark field perception, the light can be bent or reflected, or, if a liquid crystal layer is utilized, the polarization of the light can be altered. For example, in one variation, each liquid crystal layer can act as a polarization rotator so that, if a patterned polarizing material is integrated on one side of the panel, the polarization of individual rays from the real world can be selectively manipulated, thereby capturing a portion of the patterned polarizer. There are polarizers known in the art that have a checkerboard pattern where half of the "checkerboard" has vertical polarization and the other half has horizontal polarization. Furthermore, if a material such as liquid crystal is used in which polarization can be selectively manipulated, light can be selectively attenuated by this material.

As mentioned above, a selective reflector can provide greater transmission efficiency than an LCD. In one embodiment, if the lens system is positioned so that it absorbs light from the real world and focuses a plane from the real world onto an image plane, and if a DMD (i.e., DLP technology) is placed at this image plane to reflect light to another set of lenses that transmit light to the eye when in the "on" state, and these lenses also have DMDs at their focal lengths, the DMD can produce an attenuation pattern that is not clearly visible to the eye. In other words, in a zero-magnification telescope configuration, the DMD can be used with the selective reflector plane, as shown in FIG19A, to controllably mask and promote the perception of a dark field.

As shown in FIG19A , a lens (518) absorbs light from the real world (144) and focuses it on an image plane (520); if a DMD (or other spatial attenuation device) (522) is placed at the focal length of the lens (i.e., at the image plane 520), the lens (518) will absorb all light from optical infinity and focus it on the image plane (520). The spatial attenuator (522) can then be used to selectively block the light to be attenuated. FIG19A shows an attenuator DMD in transmission mode, in which these attenuation devices transmit the light beam shown across the device. The image is then placed at the focal length of a second lens (524). Preferably, the two lenses (518, 524) have the same optical power so that they are zero-power telescopes or "repeaters" and do not magnify the view of the real world (144). Such a configuration can be used to present an unmagnified view of the world while also allowing for the selective blocking/attenuation of specific pixels.

In another embodiment, as shown in Figures 19B and 19C, additional DMDs can be added to reflect light from each of the four DMDs (526, 528, 530, 530) before being transmitted to the eye. Figure 19B shows an embodiment with two lenses, which preferably have the same optical power (focal length "F") and are positioned with a 2F relationship to each other (the focal length of the first lens is conjugate with the focal length of the second lens) to have a zero-power telescope effect; Figure 19C shows an embodiment without lenses. For ease of illustration, the orientation angle of the four reflective plates (526, 528, 530, 532) in the illustrated embodiments of Figures 19B and 19C is shown as approximately 45 degrees, but a specific relative orientation is required (e.g., typical DMDs reflect at an angle of approximately 12 degrees).

In another embodiment, the reflective plate can be a ferroelectric reflective plate, or any other type of reflective or selective attenuator plate or array. In an embodiment similar to that shown in Figures 19B and 19C, one of the three reflector arrays can be a simple mirror, allowing the other three to be selective attenuators, thereby still providing three independent plates to controllably block various portions of the incoming illumination, thereby facilitating darkfield perception. By connecting multiple dynamic reflective attenuators in series, it is possible to generate blockage at different focal lengths relative to the real world.

Alternatively, referring back to Figure 19C, a configuration can be created in which one or more DMDs are placed in a reflective periscope configuration without any lenses. Such a configuration can be driven by a light field algorithm to selectively attenuate specific rays while letting other rays pass.

In another embodiment, a DMD or similar matrix of controllable mobile devices can be created on a substrate opposite a generally opaque substrate for use in transmissive configurations such as virtual reality.

In another embodiment, two LCD panels can be used as light field shields. In one variation, they can be considered attenuators due to their aforementioned attenuation function; alternatively, they can be considered polarization rotators with a shared polarizer stack. Suitable LCDs can include components such as blue-phase liquid crystal, cholesteric liquid crystal, ferroelectric liquid crystal, and/or twisted nematic liquid crystal.

One embodiment may include an array of directionally selective shading elements, such as a MEMS device, that features a set of shutters that can be rotated to allow most light from a particular angle to pass through, but provide a wider surface for light from another angle (somewhat similar to how farm shutters are used for typical human-friendly windows). The MEMS/shutter configuration can be placed on an optically transparent substrate, where the shutters are essentially opaque. Ideally, the shutters of such a configuration are spaced finely enough to selectively block light on a pixel-by-pixel basis. In another embodiment, two or more shutter layers or shutter stacks can be combined to provide even further control. In another embodiment, the shutters can be polarizers configured to controllably change the polarization state of light, rather than selectively blocking light.

As described above, another embodiment for selectively blocking light can include an array of sliders in a MEMS device, such that the sliders can be controllably opened (i.e., by sliding in a planar manner from a first position to a second position; or by rotating from a first orientation to a second orientation; or, for example, by a combination of reorientation and placement) to transmit light through a small frame or aperture, and controllably closed to block the frame or aperture and prevent transmission. The array can be configured to open or block various frames or apertures so that they maximally attenuate the radiation to be attenuated and only minimally attenuate the radiation to be transmitted.

In embodiments where a fixed number of slides can occupy a first position that blocks a first aperture and opens a second aperture, or a second position that blocks a second aperture and opens a first aperture, the same overall amount of light is always transmitted (because for such a configuration, 50% of the apertures are blocked and 50% are open), but local positional variations of the shutters or doors can create directional moiré or other effects to achieve darkfield perception through dynamic positioning of the various slides. In one embodiment, the slides can include sliding polarizers that can be used to selectively attenuate light if stacked with other dynamic or dynamic polarization elements.

Referring to FIG19D , another configuration is shown that provides an opportunity for selective reflection (such as via a DMD-style reflector array (534)), enabling the use of a stack of two waveguides (536, 538) and a pair of focusing elements (540, 542) and a reflector (534; such as a DMD) to capture a portion of the light entering through an entrance reflector (544). The reflected light can be totally internally reflected along the length of the first waveguide (536) to the focusing element (540) to focus the light on the reflector (534) (such as a DMD array), after which the DMD can selectively attenuate a portion of the light and reflect it back to the second waveguide (538) through a focusing lens (542; the lens is configured to facilitate injection of the light back into the second waveguide) for subsequent total internal reflection to an exit reflector (546), which is configured to direct the light out of the waveguide and toward the eye (58).

Such a configuration can have a relatively thin form factor and be designed to allow light from the real world (144) to be selectively attenuated. Since waveguides are best suited for collimated light, such a configuration is well suited for virtual reality configurations where the focal length is in the optical infinity range. For closer focal lengths, a light field display can be used as a layer on top of the silhouette created by the selective attenuation/dark field configuration described above, providing the user's eye with additional cues that the light is coming from another focal distance. The mask may be out of focus, although this is not desirable, so in one embodiment, a light field on top of the masking layer can be used to hide the fact that the dark field may be at the wrong focal distance.

Referring to FIG. 19E , an embodiment is shown featuring two waveguides ( 552 , 554 ), each having two reflectors ( 558 , 544 ; 556 , 546 ) tilted at approximately 45 degree angles for ease of illustration; in actual configurations, the angles may be different depending on the reflective surface, the reflective/refractive properties of the waveguides, etc., the two reflectors directing a portion of the light from the real world along each side of the first waveguide (or along two separate waveguides if the top layer is not a monolithic structure) so that the light strikes a reflector ( 548 , 550 ) at each end, such as a DMD, which can be used for selective attenuation, after which the reflected light can be injected back into the second waveguide (or back into two separate waveguides if the bottom layer is not a monolithic structure) and back into the two tilted reflectors (again, they do not need to be at 45 degrees as shown) to be emitted toward the eye ( 58 ).

A focusing transparency may also be placed between the reflector at each end and the waveguide. In another embodiment, the reflector (548, 550) at each end may comprise a standard mirror (such as an aluminized mirror). Further, the reflector may be a wavelength selective reflector, such as a dichroic mirror or a thin film interference filter. Further, the reflector may be a diffractive element configured to reflect incoming light.

FIG19F shows a configuration in which four reflective surfaces in a pyramidal configuration are used to guide light through two waveguides (560, 562), where incoming light from the real world can be split and reflected onto four different axes. The pyramidal reflector (564) can have more than four faces and can be located within the substrate prism, just like the reflector in the configuration of FIG19E. The configuration of FIG19F is an extension of the configuration of FIG19E.

19G , a single waveguide (566) can be used to capture light from the world (144) via one or more reflective surfaces (574, 576, 578, 580, 582) and relay it (570) to a selective attenuator (568; such as a DMD array), and then couple it back into the same waveguide again so that it continues to propagate (572) and encounters one or more other reflective surfaces (584, 586, 588, 590, 592) that cause the light to at least partially exit the waveguide (594) and toward the user's eye (58). Preferably, the waveguide includes a selective reflector such that one set of reflective surfaces (574, 576, 578, 580, 582) can be switched on to capture incoming light and direct it toward the selective attenuator, while a separate set of reflective surfaces (584, 586, 588, 590, 592) can be switched on to direct light returning from the selective attenuator toward the eye (58).

For simplicity, the selective attenuator is shown as being oriented substantially perpendicular to the waveguide; in other embodiments, various optical components such as refractive or reflective optical elements may be used to position the selective attenuator in a different and more compact orientation relative to the waveguide.

Referring to FIG19H , a variation of the configuration described with reference to FIG19D is shown. This configuration is somewhat similar to the configuration described above with reference to FIG5B , in which a switchable reflector array can be embedded in each of a pair of waveguides (602, 604). Referring to FIG19H , a controller can be configured to sequentially turn reflectors (2598, 600) on and off to operate multiple emitters in a frame sequence; then the DMD or other selective attenuator (594) can be driven sequentially in synchronization with the other reflectors being turned on and off.

Referring to FIG19I , a pair of wedge-shaped waveguides similar to those described above (e.g., as described with reference to FIG15A-15C ) are shown in side view or cross-section to illustrate that the two long surfaces of each wedge-shaped waveguide (610, 612) are not coplanar. A "turning film" (606, 608; such as that manufactured by 3M under the trade name "TRAF," which essentially comprises an array of microprisms) can be used on one or more surfaces of the wedge-shaped waveguides to either turn incoming rays to an angle for capture by total internal reflection or to turn outgoing rays as they exit the waveguides toward the eye or other target. The incoming rays are directed along the first wedge-shaped waveguide to a selective attenuator (614), such as a DMD, an LCD (such as a ferroelectric LCD), or an LCD stack that acts as a mask.

After the selective attenuator (614), the reflected light is coupled back into the second wedge-shaped waveguide, which then relays the light along the wedge by total internal reflection. The nature of the wedge-shaped waveguide is such that each reflection of the light results in a change in angle; this point becomes the exit point of the wedge-shaped waveguide: at this point, the angle has changed enough to become the critical angle for escape from total internal reflection. Typically, the light will exit at an oblique angle, so another turning film layer can be used to "turn" the exiting light toward a directional target such as the eye (58).

Referring to FIG19J , several arched lenslet arrays (616, 620, 622) are positioned relative to the eye and configured so that the spatial attenuator array (618) is located at the focal plane/image plane so as to be aligned with the focus of the eye (58). The first (618) and second (620) arrays are configured so that, in convergence, light passing from the real world to the eye passes essentially through a zero-power telescope. The embodiment of FIG19J shows a third lenslet array (622) that can be used to improve optical compensation, but such a third layer is generally not required. As described above, a telescope with a diameter of the viewing optics can produce an excessively large form factor (somewhat similar to placing a stack of small binoculars in front of the eyes).

One way to optimize the overall geometry is to reduce the lens diameter by dividing the lens into smaller lenslets, as shown in FIG19J (i.e., an array of lenses, rather than a single large lens). The lenslet array (616, 620, 622) is shown wrapped in a radial or arcuate manner around the eye (58) to ensure that the light beams entering the pupil are aligned through the appropriate lenslets (otherwise, the system may suffer from optical problems such as dispersion, overlap, and/or poor focus). Thus, all lenses are oriented to be "front-in" and pointed toward the pupil of the eye (58), and the system facilitates avoiding situations where rays are propagated to the pupil through unintended lens groups.

19K-19N , various software techniques can be used to assist in rendering dark scenes in virtual or augmented reality replacement cases. Referring to FIG19K , a typical challenging scenario (632) for augmented reality is shown, which has a patterned carpet (624) and uneven background architectural features (626), both of which are lightly colored. The black box (628) shown indicates the area where one or more augmented reality features will be presented to the user for three-dimensional perception, and within this black box, a robotic creature (630) is presented, which may be part of an augmented reality game in which the user participates. In the example shown, the robotic creature (630) is heavily colored, which helps achieve a challenging rendering in three-dimensional perception, especially for the background selected for this example scenario.

As briefly mentioned above, the main challenge in rendering dark-field augmented reality objects is that systems generally cannot add "darkness" or paint in "darkness"; displays are typically configured to add light. Thus, referring to FIG19L , without any specialized software processing to enhance dark-field perception, the rendering of a robotic figure in an augmented reality scene results in a scene in which parts of the robotic figure that are essentially matte black in the rendering are invisible, and parts of the robotic figure that have some light (such as the light tinting of the outer cover of the gun slung over the robotic figure's shoulder) are only slightly visible (634), appearing as slight grayscale interruptions to the normal background image.

Referring to FIG19M , the use of a software-based global attenuation process (similar to digitally putting on a pair of sunglasses) provides enhanced visibility of the robotic figure, as the brightness of the nearly black robotic figure is effectively increased relative to the rest of the space, now appearing darker (640). FIG19M also shows a digitally added halo (636) that can be added to enhance the now more visible shape of the robotic figure (638) and distinguish it from the background. Through the halo process, even parts of the robotic figure that would otherwise appear matte black are made visible by contrast with the white halo, or by a "halo" around the robotic figure.

Preferably, the halo can be presented to the user at a perceived focal length that is behind the focal length of the robotic figure in three-dimensional space. In configurations where a dark field is presented using a single-panel shading technique, such as that described above, the bright halo can be presented with an intensity gradient to match the dark halo accompanying the shading, thereby minimizing the visibility of any dark field effect. Furthermore, the halo can be presented with a background blurred behind the presented halo to achieve a further distinguishing effect. More subtle halo or ring effects can be produced by at least partially matching the color and/or brightness of a lighter-colored background.

19N , some or all of the black tones of the robot figure may be changed to a darker cool blue to provide further distinction from the background, as well as provide relatively good robot visualization ( 642 ).

Wedge-shaped waveguides have been described above, for example, with reference to Figures 15A-15C and 19I. With a wedge-shaped waveguide, each time a ray bounces off one of the non-coplanar surfaces, it experiences a change in angle, ultimately leading to total internal reflection when the ray's angle of approach to one of the surfaces exceeds the critical angle. Turning films can be used to redirect the outgoing light so that the outgoing beam exits along a trajectory that is more or less perpendicular to the exit surface, depending on the geometry and ergonomic considerations at hand.

By injecting image information into a series or array of displays of wedge-shaped waveguides (as shown in FIG15C ), for example, the wedge-shaped waveguides can be configured to create a finely spaced array of angularly deflected rays exiting the wedges. Somewhat similarly, as described above, light field displays or variable wavefronts generating waveguides can both generate multiple beamlets or beams to represent a single pixel in space, such that no matter where the eye is located, multiple different beamlets or beams specific to that particular eye position in front of the display panel will be emitted into the eye.

As further described above in the context of light field displays, multiple viewing zones can be created within a given pupil, each of which can be used for a different focal length, where the aggregate produces a perception that is similar to the perception of a variable wavefront generated by a waveguide, or to the actual optical and physical characteristics of the reality of the viewed object being real. Thus, a light field can be generated using a wedge-shaped waveguide with multiple displays. In one embodiment similar to that of FIG15C (with a linear array of displays injected with image information), an exit ray fan is generated for each pixel. This concept can be extended to embodiments in which multiple linear arrays are stacked to all inject image information into a wedge-shaped waveguide (in a variation, one array can inject at one angle relative to the wedge waveguide face, while a second array can inject at a second angle relative to the wedge waveguide face), in which case the exit beams fan out from the wedge waveguide along two different axes.

Thus, such a configuration can be used to generate multiple light beams emitted at multiple different angles, and each light beam can be driven individually, as in effect, each light beam is driven using a separate display in this configuration. In another embodiment, one or more arrays or displays can be configured to inject image information into the wedge waveguide through a side or face other than the side or face of the wedge waveguide shown in FIG15C , such as by using a diffractive optical element to bend the injected image information into a total internal reflection configuration relative to the wedge waveguide.

Consistent with such wedge-shaped waveguide embodiments, various reflectors or reflective surfaces can also be used to decouple light from the wedge-shaped waveguide and manage such light. In one embodiment, the entrance aperture of the wedge-shaped waveguide, or the injection of image information through surfaces other than those shown in FIG. 15C , can be used to facilitate the interleaving (geometric and/or temporal) of different displays and arrays, thereby creating a Z-axis difference as a means of injecting three-dimensional information into the wedge-shaped waveguide. For configurations larger than a three-dimensional array, various displays can be configured to enter the wedge-shaped waveguide at different edges, with these edges stacked in an interleaved manner to achieve higher dimensional configurations.

Referring now to FIG. 20A , there is shown a configuration similar to that shown in FIG. 8H , in which a waveguide (646) has a diffractive optical element (648; or "DOE" as described above) sandwiched therebetween (alternatively, as described above, the diffractive optical element can be located on the front or back of the waveguide shown). Rays can enter the waveguide (646) from a light projector or display (644). Once in the waveguide (646), each time a ray intersects the DOE (648), a portion of the ray exits the waveguide (646). As described above, the DOEs can be designed so that the exit illumination is somewhat uniform across the length of the waveguide (646) (e.g., a first such DOE intersection can be configured to exit approximately 10% of the light; then a second DOE intersection can be configured to exit approximately 10% of the remaining light, so that 81% continues to be transmitted, and so on; in another embodiment, the DOEs can be designed to have variable diffraction efficiency along their length (such as a linearly decreasing diffraction efficiency) so as to exit more uniform exit illumination across the length of the waveguide).

To further distribute the remaining light that reaches the ends (in one embodiment, allowing the selection of a relatively low diffraction efficiency DOE, which is advantageous from a view-world transparency perspective), a reflective element (650) may be included at one or both ends. Further, referring to the embodiment of FIG20B , additional distribution and retention is achieved by including an elongated reflector (652) (e.g., comprising a thin-film dichroic coating with wavelength selectivity) across the length of the waveguide as shown; preferably, this reflector will block light that is accidentally reflected upward (back toward the real world 44 to be emitted in a manner that is unavailable to the viewer). In certain embodiments, such an elongated reflector facilitates the user's perception of a phantom effect.

In one embodiment, this ghosting effect can be eliminated by providing a dual waveguide (646, 654) recirculating reflective configuration, such as that shown in FIG20C, which is designed to keep the light moving around until it is emitted toward the eye (58) in a preferably substantially uniform distribution across the length of the waveguide assembly. Referring to FIG20C, light can be injected through a light projector or display (644) and travel across the DOE (656) of the first waveguide (654), which projects a preferably substantially uniform light pattern toward the eye (58); the light remaining in the first waveguide is reflected by the first reflector assembly (660) into the second waveguide (646). In one embodiment, the second waveguide (646) can be configured without a DOE, thereby simply transmitting or recycling the remaining light back to the first waveguide using the second emitter assembly.

In another embodiment (as shown in FIG20C ), the second waveguide (646) may also have a DOE (648) configured to uniformly emit a portion of the traveling light to provide a second focal plane for achieving three-dimensional perception. Unlike the configurations of FIG20A and 20B , the configuration of FIG20C is designed so that light travels through the waveguide in one direction, which avoids the ghosting problem described above (associated with having light travel back through a waveguide with a DOE). Referring to FIG20D , a smaller retroreflector array (662) or retroreflective material may be used instead of having a mirror-like or box-like reflector assembly (660) at the end of the waveguide to recycle light.

Referring to FIG20E , there is shown an embodiment that utilizes certain light recycling configurations of the embodiment of FIG20C such that light, after being injected through a display or light projector (644), meanders through a waveguide (646) having a DOE (648) sandwiched therein, such that it traverses the waveguide (646) multiple times before reaching the bottom, where it can be recycled back to the top layer for further recycling. Such a configuration not only recycles light and facilitates the use of less diffraction-efficient DOE elements to direct light toward the eye (58), but also distributes the light to provide a large exit pupil configuration similar to the configuration described with reference to FIG8K .

Referring to FIG20F , there is shown an exemplary configuration similar to that of FIG5A , in which the incoming light is injected into the reflector (102) along a conventional prism or beam splitter substrate (104) and no total internal reflection is performed (i.e., the prism is not treated as a waveguide) because the input projection (106), scanning, or other operations remain within the confines of the prism—which means that the geometry of such a prism becomes an important constraint. In another embodiment, a waveguide can be utilized instead of the simple prism of FIG20F , which facilitates the use of total internal reflection to provide more geometric flexibility.

Other configurations described above are also configured to benefit from the inclusion of waveguides to achieve similar light manipulation. For example, referring back to FIG7A , the general concept illustrated therein is that collimated light injected into the waveguide can be refocused before exiting toward the eye, a configuration also designed to facilitate viewing of light from the real world. Instead of the refractive lens shown in FIG7A , a diffractive optical element can be used as the variable focus element.

Referring back to FIG7B , another waveguide configuration is shown in the context of multiple layers stacked on top of each other, with controllable access switching between a small path (through total internal reflection in the waveguide) and a large path (through total internal reflection in a hybrid waveguide comprising the original waveguide and a liquid crystal isolation region, where the liquid crystal is switched to a mode where the refractive index between the primary and auxiliary waveguides is substantially matched). This allows a controller to tune the path taken on a frame-by-frame basis. High-speed switching electroactive materials such as lithium niobate facilitate path changes at gigahertz rates through such a configuration, allowing the optical path to be altered pixel by pixel.

Referring back to FIG8A , which shows a waveguide stack paired with a weak lens to demonstrate a multifocal configuration in which the lens and waveguide elements can be static elements, each waveguide and lens pair can be functionally replaced by a waveguide with an embedded DOE element (which can be a static element (much like the configuration of FIG8A ) or a dynamic element), such as described with reference to FIG8I .

Referring to FIG20G , if a transparent prism or block (104; i.e., not a waveguide) is used in a periscope-type configuration to support a mirror or reflector (102) to receive light from other components (such as a lens (662) and a light projector or display (644)), the field of view is limited by the size of the reflector (102; the larger the reflector, the wider the field of view). Therefore, to have a larger field of view in such a configuration, a thicker substrate is required to support a larger reflector; otherwise, the functional field of view can be increased by using the functionality of multiple reflectors aggregated, as described with reference to FIG8O , 8P , and 8Q . Referring to FIG20H , a stack (664) of planar waveguides (666) can be used to aggregate toward the functionality of a larger single reflector, each fed with light through a display or light projector (644; or, in another embodiment, multiplexed through a single display) and having an exit reflector (668). In some cases, the exit reflectors may be located at the same angle, and in other cases they may be located at different angles, depending on the positioning of the eye (58) relative to the assembly.

FIG201 shows a related configuration in which the reflectors (680, 682, 684, 686, 688) within each of the planar waveguides (670, 672, 674, 676, 678) have been offset from one another, and in which each waveguide absorbs light from a projector or display (644), which is sent through a lens (690) and ultimately emitted to the pupil (45) of the eye (58) by means of the reflectors (680, 682, 684, 686, 688) within each of the planar waveguides (670, 672, 674, 676, 678). A useful field of view has been achieved if the full range of all angles desired to be seen in a scene can be produced (i.e., preferably with no blind spots in the critical field of view). As described above, the eye (58) operates at least in part based on the angle at which light enters the eye, and this action can be simulated. The rays do not need to pass through the same point in pupil space - rather, the rays only need to pass through the pupil and be perceived by the retina. Figure 20K shows a variation in which the shaded portion of the optical assembly can be used as a compensating lens to functionally allow light from the real world (144) to pass through the assembly as if through a zero-power telescope.

20J, each of the above rays is also a relatively wide beam, which is reflected by the associated waveguide (670, 672) by total internal reflection. The size of the reflector (680, 682) determines the width of the outgoing beam.

Referring to FIG20L , a further discretization of the reflector is shown, wherein a plurality of small right-angle reflectors can be aggregated to form a roughly parabolic reflective surface (694) via a waveguide or waveguide stack (696). Light from a display (644; or, for example, a single multiplexed display) (such as through a lens (690)) is all directed to the same shared focal point on the pupil (45) of the eye (58).

Referring back to Figure 13M, a linear array of displays (378) injects light into a shared waveguide (376). In another embodiment, a single display can be multiplexed to a series of entrance lenses to provide similar functionality to the embodiment of Figure 13M, where the entrance lenses generate parallel ray paths through the waveguide.

In conventional waveguide technology, where total internal reflection is relied upon for light propagation, the field of view is limited because only rays within a certain angular range propagate through the waveguide (others may escape). In one embodiment, if red/green/blue (or "RGB") laser line reflectors are placed on one or both ends of the plane (similar to thin-film interference filters that are highly reflective only for certain wavelengths but less reflective for other wavelengths), the angular range of light propagation can be functionally increased. Uncoated windows can be provided to allow light to exit at predetermined locations. Furthermore, the coating can be chosen to be directionally selective (somewhat similar to a reflective element that is highly reflective only for certain angles of incidence). Such coatings are most relevant for larger waveguide planes/sides.

Referring back to FIG13E , a variation of a scanning fiber display is described which is viewed as a scanning thin waveguide configuration such that a plurality of very thin planar waveguides (358) are collimated or vibrated such that if a plurality of injected beams are emitted by total internal reflection, the configuration will functionally provide a linear array of beams exiting the edge of the vibrating element (358). The configuration shown has approximately five externally projected planar waveguide sections (358) located within a transparent host medium or substrate (356), but the host medium or substrate preferably has a different refractive index so that the light is in total internal reflection within each of the small waveguides affixed to the substrate, which ultimately feed the externally projected planar waveguide sections (358) (in the embodiment shown, there is a 90 degree turn in each path at which point a planar reflector, curved reflector, or other reflector may be used to reflect the light outward).

The externally projected planar waveguide portions (358) can be vibrated individually or as a group in response to the oscillatory motion of the substrate (356). Such scanning motion can provide horizontal scanning, and for vertical scanning, aspects of the input (360) of the assembly can be used (i.e., such as one or more scanning fiber displays that scan in the vertical axis). Thus, a variation of a scanning fiber display is provided.

Referring back to FIG13H , a light field can be created using a waveguide (370). From a perceptual perspective, waveguides are best suited for collimated beams that may be associated with optical infinity, as all beams in focus may cause perceptual discomfort (i.e., the eye does not distinguish differences in refractive blur based on accommodation; in other words, narrow diameter (such as 0.5 mm or less) collimated beamlets may open-loop control the eye's accommodation/convergence system, causing discomfort).

In one embodiment, a single beam can be fed through multiple outgoing cone-shaped beamlets, but if the incoming beam's introduction vector is varied (i.e., the beam injection location of the projector/display is moved laterally relative to the waveguide), the position of the beam exiting the waveguide as it is directed toward the eye can be controlled. Thus, a waveguide can be used to create a light field by producing a narrow diameter collimated beam, and such a configuration does not rely on actual variations in the light wavefront associated with the eye's desired perception.

If a set of angularly and laterally diverse beamlets is injected into the waveguide (e.g., by using a multi-core fiber and driving each core individually; another configuration could use multiple fiber scanners from different angles; another configuration could use a high-resolution panel display with a lenslet array on top), multiple exit beamlets can be generated at different exit angles and exit positions. Because the waveguide can scramble the light field, the decoding is preferably predetermined.

Referring to Figures 20M and 20N, the waveguide (646) assembly (696) is shown as including waveguide components stacked on the vertical axis or the horizontal axis. The purpose of these embodiments is not to have a monolithic structure planar waveguide, but to stack multiple small waveguides (646) closely adjacent to each other so that light introduced into a waveguide and continues to propagate along the waveguide by total internal reflection (i.e., propagating along the Z axis by total internal reflection in +X, -X) is also totally internally reflected in the vertical axis (+Y, -Y) so that the light does not overflow into other areas. In other words, if the total internal reflection is from left to right and returns during the Z axis propagation, a configuration is established in which any light that is injected into the top or bottom side is totally internally reflected at the same time; each layer can be driven separately without interference from other layers. As described above, for a predetermined focal length configuration (as shown in Figure 20M, in the range of 0.5 meters to optical infinity), each waveguide can have a DOE (648) that is an embedded element and is configured to cause light to exit with a predetermined distribution along the length of the waveguide.

In another variation, a very dense stack of waveguides with embedded DOEs can be made so that the stack spans the size of the eye's anatomical pupil (i.e., requiring multiple layers 698 of composite waveguides to span the exit pupil, as shown in FIG20N ). With such a configuration, a collimated image can be fed for one wavelength, and then the next millimeter produces a diverging wavefront that represents an object from, say, 15 meters away at a focal distance, etc., to illustrate that the exit pupil comes from multiple different waveguides as a result of the DOEs and total internal reflection through the waveguides and across the DOEs. Thus, rather than creating a unified exit pupil, such a configuration creates multiple strips that, through convergence, contribute to the eye/brain perceiving different depths of focus.

This concept can be extended to configurations including waveguides with switchable/controllable embedded DOEs (i.e., switchable to different focal lengths), such as the configurations described with reference to Figures 8B-8N, which allow for more efficient capture of light in the axis across each waveguide. Multiple displays can be connected to each layer, and each waveguide with a DOE can emit rays along its length. In another embodiment, rather than relying on total internal reflection, laser line reflectors can be used to increase the angular range. Between the layers of the composite waveguide, a fully reflective metallized coating (such as aluminized layers) can be used to ensure total reflection, or alternatively, a dichroic style reflector or narrow band reflector can be used.

Referring to FIG20O , the entire composite waveguide assembly (696) can be curved and concave toward the eye (58) so that each individual waveguide is directed toward the pupil. In other words, this configuration can be designed to more efficiently direct light toward where the pupil is likely to be located. Such a configuration can also be used to increase the field of view.

As described above with reference to Figures 8L, 8M, and 8N, a variable diffraction configuration allows scanning in one axis, somewhat similar to a scanning light display. Figure 21A shows a waveguide (698) with an embedded (i.e., sandwiched therein) DOE (700) having a linear grating term that can be altered to change the exit angle of the exit light (702) from the waveguide, as shown. High frequency switching DOE materials such as lithium niobate can be used. In one embodiment, such a scanning configuration can be used as the sole mechanism for scanning the beam in one axis; in another embodiment, the scanning configuration can be combined with other scanning axes and can be used to create a larger field of view (i.e., if the normal field of view is 40 degrees, then by changing the linear diffraction pitch, it can be turned another 40 degrees, and the effective usable field of view of the system becomes 80 degrees).

Referring to FIG21B , in a conventional configuration, a waveguide (708) can be positioned perpendicular to a display panel (704) (such as an LCD or OLED panel) so that a light beam can be injected from the waveguide (708) through a lens (706) and then enter the panel (704) in a scanning configuration to provide a visual display for a television or other device. Thus, in such a configuration, the waveguide can be used as a scanning image source, in contrast to the configuration described with reference to FIG21A , in which a single light beam can be steered by a scanning fiber or other element to scan through different angular positions, and in addition, a high-frequency diffractive optical element can be used to scan in another direction.

In another embodiment, a single axis scanning fiber display can be used (such as scanning fast scan because the scanning fiber has a higher frequency) to inject a fast scan into the waveguide, and then relatively slow DOE switching (i.e., in the 100 Hz range) can be used to scan the lines in the other axis to form an image.

In another embodiment, a DOE with a fixed-pitch grating can be combined with an adjacent layer of an electroactive material (such as liquid crystal) with a dynamic refractive index to redirect light to the grating at different angles. This is an application of the basic multipath configuration described above with reference to FIG7B , in which an electroactive layer including an electroactive material such as liquid crystal or lithium niobate can change its refractive index to change the angle at which rays exit the waveguide. A linear diffraction grating can be added to the configuration of FIG7B (in one embodiment, sandwiched within the glass or other material comprising the larger lower waveguide) so that the diffraction grating can be maintained at a fixed pitch, but the light is deflected before entering the grating.

FIG21C shows another embodiment featuring two wedge-shaped waveguide elements (710, 712), one or more of which can be electro-active, thereby modifying the relative refractive indices. These elements can be configured so that when the wedges have matching refractive indices, light undergoes total internal reflection through the waveguide element pair (the two waveguide elements collectively perform similarly to a planar waveguide with two matching wedges), with the wedge interface inactive. However, if one of the refractive indices is modified to create a mismatch, the light beam at the wedge interface (714) is deflected, resulting in total internal reflection from that surface back into the associated wedge. A controllable DOE (716) with a linear grating can then be coupled along one of the long sides of the wedge to allow light to exit at the desired exit angle and reach the eye.

In another embodiment, a DOE such as a Bragg grating can be configured to change the pitch versus time, for example, by mechanical stretching of the grating (e.g., if the grating is on or includes an elastic material), a moiré beat pattern between two gratings in two different planes (the gratings can have the same or different pitches), Z-axis motion of the grating (i.e., toward the eye, or away from the eye) or an electro-active grating (functionally similar to grating stretching) that can be switched on or off (e.g., electro-active gratings created using a polymer dispersed liquid crystal approach, in which liquid crystal droplets are controllably activated to change the refractive index, thereby becoming an active grating, and turning off the voltage and allowing switching back to index matching that of the host medium).

In another embodiment, a time-varying grating can be used to extend the field of view using a tiled display configuration. Furthermore, a time-varying grating can be used to correct for chromatic aberration (the inability to focus all colors/wavelengths at the same focal point). One property of a diffraction grating is its ability to deflect a light beam depending on the angle of incidence and wavelength (i.e., a DOE will deflect different wavelengths by different angles, somewhat similar to how a simple prism splits a light beam into its multiple wavelength components).

In addition to field of view expansion, time-varying grating control can be used to compensate for chromatic aberration. Thus, for example, in the configuration described above with a waveguide embedded with a DOE, the DOE can be configured to drive the red wavelength to a slightly different position than the green and blue wavelengths to account for undesirable chromatic aberration. The DOE can be time-varying (i.e., resulting in red, green, and blue colors diffracting outward in a similar manner) by having a stack of elements that switch on and off.

In another embodiment, a time-varying grating can be used for exit pupil expansion. For example, referring to FIG21D , a waveguide (718) with an embedded DOE (720) can be positioned relative to the target pupil so that none of the beams emitted in baseline mode actually enter the target pupil (45), causing the user to miss the relevant pixels. A time-varying configuration can be used to fill in the gaps in the exit pattern by laterally shifting the exit pattern (shown in dashed/dotted lines) to effectively scan each of the five exit beams, ensuring that one of them enters the pupil of the eye. In other words, expanding the functional exit pupil of the display system.

In another embodiment, a time-varying grating can be used with a waveguide to perform one, two, or three-axis optical scanning. In a manner similar to that described with reference to FIG21A, a grating can be used to scan the beam in the vertical axis and an element in the grating can be used to perform scanning in the horizontal axis. Further, if radial elements of the grating are incorporated, as described with reference to FIG8B-8N, the beam can be scanned in the Z axis (i.e., toward/away from the eye), all of which can be time-sequential scans.

Although the description here deals specifically with the treatment and use of SOEs in combination with waveguides, many of these uses of DOEs are applicable regardless of whether the DOE is embedded in a waveguide. For example, the output of a waveguide can be manipulated independently using a DOE; alternatively, a beam can be manipulated by a DOE before being injected into a waveguide; further, one or more DOEs, such as a time-varying DOE, can be used as the input to a freeform optical element configuration, as described below.

As described above with reference to Figures 8B-8N, a DOE element can have a circularly symmetric term that can be added to a linear term to create a controllable output pattern (i.e., the same DOE that decouples light can also focus it, as described above). In another embodiment, the circular term of the DOE diffraction grating can be varied to modulate the focus of the beam representing those pixels of interest. Additionally, a configuration can have a second/separate circular DOE, eliminating the need to include a linear term in the DOE.

21E , there can be a waveguide (722) that outputs collimated light without an embedded DOE element, and a second waveguide with a circularly symmetric DOE that can be switched between multiple configurations—in one embodiment, by having a stack (724) of such DOE elements that can be switched on/off ( FIG. 21F shows another configuration in which, as described above, a functional stack 728 of DOE elements can include a stack of polymer dispersed liquid crystal elements 726 in which, in the absence of an applied voltage, the host medium refractive index matches that of the dispersed liquid crystal molecules; in another embodiment, molecules of lithium niobate can be dispersed to achieve faster response times; in the presence of an applied voltage, such as through transparent indium tin oxide on either side of the host medium, the dispersed molecules change the refractive index and functionally form a diffraction pattern within the host medium).

In another embodiment, a circular DOE can be layered in front of the waveguide for focusing. Referring to FIG21G , the waveguide (722) outputs collimated light that will be perceived as associated with a depth of focus at optical infinity unless otherwise modified. The collimated light from the waveguide can be input to a diffractive optical element (730) that is used for dynamic focusing (i.e., different circular DOE patterns can be turned on or off to impart a variety of focal points to the outgoing light). In a related embodiment, a static DOE can be used to focus the collimated light exiting the waveguide to a single focal depth useful for a particular user application.

In another embodiment, multiple stacked circular DOEs can be used to increase power and a wide range of focus levels with a smaller number of switchable DOE layers. In other words, three different DOE layers can be switched on relative to each other in various combinations, and the optical power of the switched-on DOEs can be increased. In an embodiment where a range of up to four diopters is desired, for example, a first DOE can be configured to provide half of the desired total diopter range (in this example, a 2-diopter focus change); a second DOE can be configured to cause a 1-diopter focus change; and a third DOE can be configured to cause a 1/2-diopter focus change. These three DOEs can be mixed and matched to provide focus changes of 1/2, 1, 1.5, 2, 2.5, 3, and 3.5 diopters. Thus, an extremely large number of DOEs is not required to achieve a relatively wide control range.

In one embodiment, a matrix of switchable DOE elements can be used for scanning, field of view expansion, and/or pupil expansion. Generally in the above DOE discussions, it has been assumed that a typical DOE is either all on or all off. In one variation, the DOE (732) can be subdivided into multiple functional subsections (e.g., a subsection labeled as element 734 in Figure 21H), each of which is preferably uniquely controllable to be on or off (e.g., referring to Figure 21H, each subsection can be operated by its own set of indium tin oxide, or other control wire material, voltage application wires 736 returning to a central controller). Given this level of control over the DOE paradigm, other configurations are facilitated.

Referring to FIG21I , a waveguide (738) with an embedded DOE (740) is viewed from above through the eyes of a user positioned in front of the waveguide. A given pixel can be represented as a beam of light that enters the waveguide and is totally internally reflected together until it can exit the waveguide as a group of light beams through a diffraction pattern. Depending on the diffraction configuration, the light beams can exit in parallel (as shown in FIG21I for convenience) or in a diverging fan configuration (if representing a focal length closer than optical infinity).

The set of parallel outgoing beams shown may, for example, represent the leftmost pixel of what a user would see in the real world when viewed through the waveguide, and the light reaching the rightmost end would be a different set of parallel outgoing beams. In practice, using modular control of DOE subsections as described above, it may take more computational resources or time to generate and manipulate the subset of small beams that may actively address the user's pupil (i.e., because the other beams never reach the user's eye and are effectively wasted). Thus, referring to FIG21J , a waveguide (738) configuration is shown in which only two subsections (740, 742) of the DOE (744) are activated that are considered to potentially address the user's pupil. Preferably, one subsection may be configured to direct light in one direction while the other subsection directs light in a different direction.

FIG21K shows an orthogonal view of two individually controllable subsections (734, 746) of a DOE (732). Referring to the top view of FIG21L, such individual control can be used to scan or focus light. In the configuration shown in FIG21K, an assembly (748) of three individually controllable DOE/waveguide subsections (750, 752, 754) can be used to scan, increase the field of view, and/or increase the exit pupil area. Such functionality can be generated from a single waveguide with such individually controllable DOE subsections, or from a vertical stack of these subsections with the added complexity of the stack.

In one embodiment, if a circular DOE can be controllably stretched radially and symmetrically, the diffraction pitch can be modulated, and the DOE can be used as a tunable lens with analog-type control. In another embodiment, uniaxial stretching (e.g., to adjust the angle of a linear DOE term) can be used for DOE control. Further, in another embodiment, a membrane (similar to an eardrum) can be vibrated in the Z axis (i.e., toward/away from the eye) through an oscillatory motion, thereby providing Z-axis control and time-dependent focus changes.

Referring to FIG21M , a stack of several DOEs (756) is shown that receive collimated light from a waveguide (722) and refocus the collimated light based on the additional power of the activated DOEs. For example, the linear and/or radial terms of the DOEs can be modulated over time according to a frame sequence to produce various treatments (such as a tiled display configuration or an extended field of view) for light exiting the waveguide and preferably toward the user's eyes. In a configuration where one or more DOEs are embedded within the waveguide, low diffraction efficiency is desired to maximize the transparency of light passing from the real world; in a configuration where one or more DOEs are not embedded, high diffraction efficiency may be desired, as described above. In one embodiment, both linear and radial DOE terms can be combined outside the waveguide, in which case high diffraction efficiency will be desired.

Referring to FIG21N , a segmented or parabolic reflector is shown, such as those discussed above in FIG8Q . Rather than implementing a segmented reflector by combining multiple smaller reflectors, in one embodiment, the same functionality can be generated by a single waveguide with a DOE having a different phase profile for each of its segments so that it can be controlled by sub-segments. In other words, while the entire segmented reflector functionality can be turned on or off simultaneously, the DOEs can generally be configured to direct light toward the same area in space (i.e., the user's pupil).

Referring to Figures 22A-22Y, an optical configuration known as a "free-form optical element" can be used to address the above challenges. The term "free-form" is generally used to refer to free-form curved surfaces that can be used in situations where spherical, parabolic, or cylindrical lenses do not meet design complexities such as geometric constraints. For example, referring to Figure 22A, a common challenge with display (762) configurations when a user views through a mirror (and sometimes also lens 760) is that the field of view is limited by the area facing the final lens (760) of the system.

Referring to Figure 22B, in simpler terms, if you have a display (762) that may include some lens elements, there is a simple geometric relationship such that the field of view cannot be larger than the angle subtended by the display (762). Referring to Figure 22C, this challenge is exacerbated if the user is attempting to have an augmented reality experience (where light from the real world also passes through the optical system), because in this case, there is typically a reflector (764) directed toward the lens (760); by inserting the reflector, the overall path length from the eye to the lens is increased, which narrows the angle and reduces the field of view.

Given this, if you want to increase the field of view, you must increase the size of the lens, but from an ergonomic perspective, this may mean pushing the physical lens towards the user's forehead. Furthermore, the reflector may not capture all the light from the larger lens. Therefore, there are practical limitations imposed by the structure of the human head, and achieving a field of view greater than 40 degrees using conventional transparent displays and lenses is generally challenging.

With a freeform lens, rather than having a standard flat reflector as described above, a combined reflector and lens (i.e., curved reflector 766) with power is obtained, which means that the curved lens geometry determines the field of view. Referring to FIG22D , without the circuitous path length of the conventional example as described above with reference to FIG22C , a freeform arrangement can achieve a significantly larger field of view for a given set of optical requirements.

Referring to FIG22E , a typical free-form optical element has three active surfaces. Referring to FIG22E , in a typical free-form optical element (770) configuration, light can be directed from an image plane (e.g., a flat panel display (768)) toward the free-form optical element into a first active surface (772), which is typically a primary transmissive free-form surface that refracts the transmitted light and imparts a focus change (e.g., added astigmatism, since the final reflection from the third surface will add matching/opposite astigmatism, and this astigmatism needs to be canceled). From the first surface, incident light can be directed to a second surface (774), where it can penetrate at a sufficiently shallow angle to cause light to be reflected toward the third surface (776) under total internal reflection.

The third surface can comprise a semi-silvered free-form curved surface configured to reflect light out toward the eye through the second surface, as shown in FIG22E . Thus, in the exemplary free-form configuration shown, light enters through the first surface, reflects from the second surface, reflects from the third surface, and is directed out of the second surface. Since the second surface is optimized to have the necessary reflective properties for a first pass and refractive properties for a second pass when exiting toward the eye, various curved surfaces having higher-order shapes, rather than simple spheres or parabolas, form free-form optical elements.

22F, in an augmented reality configuration, a compensating lens (780) may be added to the freeform optical element (770) so that the overall thickness of the optical assembly is a substantially uniform thickness and preferably does not magnify light incident from the real world (144).

Referring to FIG22G , a freeform optical element (770) can be combined with a waveguide (778) configured to promote total internal reflection of the captured light within specific constraints. For example, as shown in FIG22G , light can be directed into the freeform/waveguide assembly from an image plane (e.g., a flat panel display) and totally internally reflected within the waveguide until it strikes the freeform curved surface and escapes toward the user's eye. The light is thus reflected several times by total internal reflection until it reaches the freeform wedge.

A primary goal with this type of assembly is to try to make the optical assembly longer while maintaining as uniform a thickness as possible (to facilitate transmission via total internal reflection, and also viewing the world through the assembly without further compensation) in order to achieve a larger field of view. Figure 22H shows a configuration similar to Figure 22G, except that the configuration of Figure 22H also depicts the characteristics of the compensating lens portion to further extend the thickness uniformity and facilitate viewing the world through the assembly without further compensation.

Referring to FIG22I , in another embodiment, a freeform optical element (782) is shown having a facet or fourth face (784) in the lower left corner, the fourth face (784) being configured to facilitate the injection of image information at different locations than those typically used for freeform optical elements. The input device (786) may include, for example, a scanning fiber display, which may be designed to have a very small output geometry. The fourth face itself may include various geometries and have its own refractive power (e.g., by using a planar or freeform surface geometry).

Referring to FIG22J , in practice, such a configuration can also describe the properties of a reflective coating (788) along a first surface so that it directs light back to a second surface, which then reflects the light to a third surface, which directs the light across the second surface and toward the eye (58). The addition of a fourth small surface to inject image information facilitates a more compact configuration. In one embodiment in which a conventional free-form input configuration and a scanning fiber display (790) are used, certain lenses (792, 794) may be required in order to properly form the image plane (796) using the output from the scanning fiber display; these hardware components add additional bulk that may be undesirable.

Referring to FIGURE 22K, an embodiment is shown where light from a scanning fiber display (790) is passed through the input optical assembly (792, 794) to the image plane (796) and is then directed across a first surface of the freeform optical element (770) to exit a total internal reflection at a second surface, whereupon another total internal reflection from a third surface causes the light to exit across the second surface and be directed toward the eye (58).

A total internal reflection freeform waveguide can be created such that there is no reflective coating (i.e., such that total internal reflection is relied upon to propagate light until a critical angle of incidence with the surface is met, at which point the light exits in a manner similar to the wedge-shaped optical element described above). In other words, rather than having two planar surfaces, one can have a surface comprised of one or more subsurfaces drawn from a set of conic sections (e.g., parabolas, spheres, ellipses, etc.).

Such a configuration can still produce angles shallow enough to allow for total internal reflection in the optic; thus providing an approach that is somewhat of a hybrid between conventional freeform optics and wedge-shaped waveguides. One motivation for having such a configuration is to avoid the use of reflective coatings, which do help produce reflections, but are also known to block the transmission of a relatively large portion (e.g., 50%) of the light transmitted from the real world (144). Further, such coatings would also block an equal amount of light from the input device from entering the freeform optic. Thus, there is a reason to develop a design without a reflective coating.

As described above, one surface of a conventional free-form optical element may include a semi-silvered reflective surface. Typically, such a reflective surface will have a "neutral density," meaning that it will generally reflect all wavelengths similarly. In another embodiment, such as in an embodiment where a scanning fiber display is used as an input, a wavelength-sensitive narrowband reflector (e.g., a thin film laser line reflector) may be used in place of the conventional reflector paradigm. Thus, in one embodiment, the configuration may reflect specific red/green/blue wavelength ranges and remain inactive for other wavelengths, which will generally increase the transparency of the optical element and is therefore preferred for augmented reality configurations where it is important to transmit image information from the real world (144) across the optical element.

Referring to FIG22L , an embodiment is shown in which multiple freeform optical elements (770) can be stacked in the Z axis (i.e., along an axis substantially aligned with the visual axis of the eye). In one variation, each of the three freeform optical elements shown can have a wavelength selective coating (one highly selective for blue, one highly selective for green, and one highly selective for red) so that an image can be injected into each freeform optical element so that blue is reflected from one surface, green is reflected from another surface, and red is reflected from a third surface. Such a configuration can be used, for example, to address chromatic aberration issues, create a light field, or increase the functional exit pupil size.

Referring to FIG. 22M , an embodiment is shown in which a single freeform optical element ( 798 ) has multiple reflective surfaces ( 800 , 802 , 804 ), each of which may be wavelength or polarization selective so that its reflective properties may be individually controlled.

Referring to FIG22N , in one embodiment, multiple microdisplays (e.g., scanned light displays) (786) can be injected into a single freeform optical element to tile the image (thus providing an increased field of view), increase the functional exit pupil size, or address challenges such as chromatic aberration (i.e., by reflecting a wavelength from each display). Each display shown injects light that takes a different path through the freeform optical element, which will provide a larger functional exit pupil output due to the different positioning of the displays relative to the freeform optical element.

In one embodiment, a package or bundle of scanning fiber displays can be used as input to overcome one of the challenges in operationally coupling scanning fiber displays to freeform optical elements. One such challenge with scanning fiber display configurations is that the outputs of individual optical fibers are emitted with a specific numerical aperture, or "NA," which is analogous to the projection angle of light from the fiber; ultimately, this angle determines the diameter of the beam passing through the various optical elements and ultimately determines the functional exit pupil size; therefore, to maximize the exit pupil size of a freeform optical element configuration, the NA of the fiber can be increased using an optimized refractive relationship (e.g., between the core and cladding), or a lens (i.e., a refractive lens, such as a gradient index lens, or "GRIN" lens) can be placed at the end of the fiber or built into the end of the fiber as described above, or an array of fibers can be created that feeds the freeform optical element, in which case all of the NAs in the bundle remain small and create an array of small exit pupils at the exit pupil, the aggregation of which forms the functional equivalent of a large exit pupil.

Alternatively, in another embodiment, a more sparse array of scanning fiber displays or other displays (i.e., not tightly bundled into a package) can be used to functionally increase the field of view of the virtual image through the freeform optical element. Referring to FIG22O, in another embodiment, multiple displays (786) can be injected through the top of the freeform optical element (770), and another multiple displays (786) can be injected through the lower corners; the display array can be a two-dimensional or three-dimensional array. Referring to FIG22P, in another related embodiment, image information can also be injected from both sides (806) of the freeform optical element (770).

In embodiments where multiple smaller exit pupils are aggregated into a functionally larger exit pupil, each scanning fiber can be chosen to have a single color, so that in a given bundle or bundles of projectors or displays, there can be a subset of separate red fibers, a subset of separate blue fibers, and a subset of separate green fibers. Such a configuration facilitates more efficient outcoupling of light into the fibers; for example, in such embodiments, there is no need to superimpose red, green, and blue into the same band.

22Q-22V, various freeform optical element tiling configurations are shown. Referring to FIG22Q, an embodiment is shown in which two freeform optical elements are tiled side by side, and a microdisplay (e.g., a scanning light display) (786) is configured on each side to inject image information from each side so that one freeform optical element wedge represents half the field of view.

Referring to Figure 22R, a compensating lens (808) may be included to facilitate viewing the real world through the optical assembly. Figure 22S shows a configuration where freeform optical element wedges are tiled side by side to increase the functional field of view while keeping the thickness of such optical assembly relatively uniform.

22T , in the following configuration, a star assembly includes multiple free-form optical element wedges (also shown with multiple displays for inputting image information): This configuration can provide a greater field of view expansion while also maintaining a relatively thin overall optical assembly thickness.

Using tiled freeform optical assemblies, optical elements can be aggregated to produce a larger field of view; this concept has been addressed in the tiled configuration described above. For example, in a configuration where two freeform waveguides are aimed at the eye (such as the configuration shown in FIG22R ), there are several ways to increase the field of view. One option is to "toe in" the freeform waveguides so that their outputs are shared, or to overlay the freeform waveguides in the space of the pupil (e.g., a user can view the left half of the field of view through the left freeform waveguide, and the right half of the field of view through the right freeform waveguide).

Using such a configuration, the field of view has been increased by tiling the freeform waveguides, but the size of the exit pupil has not been increased. Alternatively, the freeform waveguides can be positioned so that they do not toe in as much—thus creating exit pupils that are side-by-side at the anatomical pupil of the eye. In one example, the anatomical pupil can be 8 mm wide, and each side-by-side exit pupil can be 8 mm, so that the functional exit pupil is approximately twice as wide. Thus, such a configuration provides an increased exit pupil, but if the eye moves around the "orbit" defined by the exit pupil, the eye may lose some portion of the field of view (i.e., lose a portion of the incident light on the left or right side due to the side-by-side nature of such a configuration).

In one embodiment using such an approach to tile freeform optical elements (particularly along the Z-axis relative to the user's eye), the red wavelength can be driven by one freeform optical element, the green wavelength by another freeform optical element, and the blue wavelength by another freeform optical element, which can address red/green/blue chromatic aberration. Such a configuration can also be provided with a stack of multiple freeform optical elements, each configured to address a specific wavelength.

Referring to FIG22U , two oppositely oriented freeform optical elements are shown stacked along the Z axis (i.e., they are inverted relative to each other). For such a configuration, a compensating lens may not be required to facilitate accurate viewing of the world through the assembly; in other words, rather than having a compensating lens (such as in the embodiments of FIG22F or FIG22R ), additional freeform optical elements can be used, which can further assist in routing light to the eye. FIG22V shows another similar configuration, in which an assembly of two freeform optical elements is provided as a vertical stack.

To ensure that one surface does not interfere with another in a freeform optical element, wavelength- or polarization-selective reflector surfaces can be used. For example, referring to FIG22V , red, green, and blue wavelengths of 650 nm, 530 nm, and 450 nm can be injected, and red, green, and blue wavelengths of 620 nm, 550 nm, and 470 nm can be injected; different selective reflectors can be used in each freeform optical element so that they do not interfere with each other. In configurations where polarization filtering is used for similar purposes, the reflectance/transmission selectivity of light polarized along a particular axis can be varied (i.e., the image can be pre-polarized before being sent to each freeform waveguide to cooperate with the reflector selectivity).

Referring to Figures 22W and 22X, a configuration is shown in which multiple free-form waveguides can be used simultaneously in series. Referring to Figure 22W, light can enter from the real world and be directed sequentially through a first free-form optical element (770), an optional lens (812), which can be configured to pass the light to a reflector (810) (e.g., a DMD in a DLP system), which can be configured to reflect the light, which has been filtered pixel by pixel (i.e., a mask can be used to block specific elements of the real world, such as for dark field perception as described above; a suitable spatial light modulator can be used, including a DMD, LCD, ferroelectric LCOS, MEMS shutter array, etc. as described above) to another free-form optical element (770), which relays the light to the user's eye (28). Such a configuration can be more compact than a configuration using conventional lenses for spatial light modulation.

22X, in scenarios where keeping the overall thickness to a minimum is important, a configuration can be used that has one highly reflective surface so that it can reflect light directly to another compactly positioned freeform optical element. In one embodiment, a selective attenuator (814) can be inserted between two freeform optical elements (770).

Referring to FIG22Y , an embodiment is shown in which the freeform optical element (770) can comprise an aspect of a contact lens. A miniaturized freeform optical element is shown coupled to a miniaturized compensating lens portion (780) through the cornea of a user's eye (58), similar to that described with reference to FIG22F . Signals can be injected into the miniaturized freeform assembly using a connected scanning fiber optic display, which can be coupled, for example, between the freeform optical element and the user's tear gland area, or between the freeform optical element and another head-mounted display configuration.

Various example embodiments of the present invention are described herein. Reference to these examples is without any limitation. These embodiments are provided to illustrate the broader application aspects of the present invention. Without departing from the true spirit and scope of the present invention, various changes can be made to the described invention, and equivalents can be used for replacement. In addition, many modifications can be made to adapt specific circumstances, materials, components, processes, processing actions (multiple) or steps (multiple) to the target (multiple), spirit or scope of the present invention. Further, it will be understood by those skilled in the art that each modification described and illustrated herein has separate components and features, and without departing from the scope or spirit of the present invention, these separate components and features can be easily separated or combined with the features of any embodiment in other multiple embodiments. All these modifications are intended to be within the scope of the claims associated with this disclosure.

The present invention includes methods that can be performed using subject devices. These methods may include the act of providing such appropriate devices. Such provisioning may be performed by the end user. In other words, the act of "providing" simply requires the end user to obtain, access, approach, locate, set up, activate, power on, or otherwise perform other actions to provide the necessary devices in the subject method. The methods described herein may be performed in any logically feasible order of the described events, and may be performed in the described order of events.

Various exemplary aspects of the present invention, as well as details regarding material selection and manufacturing, have been described above. Further details of the present invention can be understood in conjunction with the patents and publications cited above and the common knowledge of those skilled in the art. For the method-based aspects of the present invention, further actions that are typically or logically employed are also understood in the manner described above.

In addition, although the present invention has been described with reference to a plurality of examples optionally incorporating a plurality of features, the present invention is not limited to the description or indications contemplated with respect to each variation of the present invention. Various changes may be made to the described invention and equivalents (whether described herein or not included for simplicity) without departing from the true spirit and scope of the present invention. In addition, if a range of values is provided, it will be understood that each intermediate value or intermediate value in the range between the upper and lower limits of the range and any other described range is included within the present invention.

In addition, it is contemplated that any optional feature of the described inventive variations may be listed and claimed independently or in combination with any one or more of the features described herein. Reference to a single item includes the possibility that a plurality of the same item may be present. More specifically, as used herein and in the associated claims, the singular forms "a," "an," "said," and "the" include plural referents unless expressly indicated otherwise. In other words, the use of the articles allows for "at least one" subject item in the above description and claims associated with this disclosure. It is further noted that the above claims may be drafted to exclude any optional elements. Therefore, this statement is intended to serve as a preamble to the use of exclusive terms such as "solely," "only," etc., or the use of "negative" limitations in conjunction with the recitation of claim elements.

In the absence of such exclusive terms, the term "comprising" in the claims associated with the present disclosure shall allow for the inclusion of any additional elements (regardless of whether a given number of elements are listed in these claims), or the addition of features may be considered to change the nature of the elements listed in these claims. While maintaining claim validity, all technical and scientific terms used herein, except as expressly defined herein, are given the broadest possible well-understood meaning.

The scope of the present invention is not limited to the examples and/or subject descriptions provided, but is limited only by the scope of the claim language associated with this disclosure.

Claims (19)

1. A system for displaying virtual content to a user, comprising: A visual accommodation tracking module that determines the visual accommodation of the user's eyes; An image generation source that provides one or more frames of image data in a temporal manner; A light emitter that projects light associated with one or more frames of the image data; At least one waveguide that receives light rays associated with image data and emits the light rays to the user's eye; A first variable focus element, located near one side of the at least one waveguide, alters the focus of the emitted light at least in part based on the determined visual accommodation of the user's eye; and A second zoom element is positioned near the other side of the at least one waveguide to compensate for and counteract the optical effects of the first zoom element against light traveling from the real world through the at least one waveguide to the user's eye. 2. The system of claim 1, wherein the at least one waveguide comprises a plurality of waveguides, and one of the plurality of waveguides is a waveguide element, wherein the focus of a first frame of image data transmitted from a first waveguide of the plurality of waveguides is different from the focus of a second frame of image data transmitted from a second waveguide of the plurality of waveguides. 3. The system according to claim 2, wherein the first frame is a first layer of the 3D scene, and the second frame is a second layer of the 3D scene. 4. The system of claim 1, further comprising a blurring module that blurs a portion of one or more frames of the image data in such a way that the portion is out of focus when viewed by the user. 5. The system according to claim 1, wherein the waveguide is electrically active. 6. The system according to claim 1, wherein one of the at least one waveguides corresponds to a fixed focal plane. 7. The system according to claim 1, further comprising an exit pupil, wherein the diameter of the exit pupil is not greater than 0.5 mm. 8. The system of claim 1, wherein the light emitter is a scanning fiber optic display. 9. The system of claim 7, further comprising an exit pupil array. 10. The system of claim 7, further comprising a plurality of emitters, one emitter being coupled to the exit pupil. 11. The system of claim 1, further comprising an exit pupil dilator. 12. The system of claim 7, wherein the exit pupil is capable of switching at least in part based on the determined visual accommodation of the user's eye. 13. The system of claim 1, wherein the at least one waveguide comprises a plurality of waveguides, and the system further comprises at least one lens disposed accordingly between the waveguides. 14. The system of claim 13, wherein each of the at least one lens has a static focal length. 15. The system of claim 13, wherein the at least one lens comprises a plurality of lenses. 16. The system of claim 14, wherein the first variable focus element changes according to the visual accommodation of the determined user's eye, such that one of the plurality of waveguides outputs a clear wavefront. 17. The system of claim 16, wherein the at least one lens causes the plurality of waveguides to simultaneously generate a plurality of different wavefront focal planes. 18. The system of claim 17, wherein the plurality of waveguides comprises three waveguides, and the waveguide that outputs a clear wavefront is the middle waveguide of the three waveguides. 19. The system of claim 18, wherein the first waveguide of the remaining waveguides of the three waveguides outputs a +margin wavefront, and the second waveguide of the remaining waveguides of the three waveguides outputs a -margin wavefront.