TW202524161A - Display apparatus - Google Patents

Abstract

An apparatus that displays synthetic scene images for augmented reality, virtual reality and other display types. One or more pairs of input beams (70, 76) are injected into a display panel incorporating a layer (100) of phase matched nonlinear optical (NLO) material. The input beams of each pair are brought to cross each other in the NLO material in a noncollinear geometry to generate a product beam (84) by sum frequency generation that is directed to the eye (160) of a viewer. The synthetic scene image information is impressed on each input beam pair by amplitude and phase modulation to set the amplitude (i.e. brightness), wavevector (i.e. direction) and wavefront radius of curvature (i.e. depth) of the synthetic scene image. For colour display, three pairs of input beams with respective sum frequencies in three primary colours are used.

Description

Display device

The present invention relates to a display device.

Augmented reality (AR) and virtual reality (VR) displays have been integrated into various wearable formats including headsets. For AR headsets, the glasses (i.e., spectacles) format is common. For VR headsets, the ski goggles format is common. A recent review of AR and VR display technologies is the article “Augmented Reality and Virtual Reality Displays: Perspectives and Challenges” by Zhan, Yin, Xiong, He, and Wu, https://doi.org/10.1016/j.isci.2020.101397. Perspective, Vol. 23, No. 8, 101397 (2020).

In AR, an image of a synthetic scene is superimposed on the natural scene that the viewer is simultaneously viewing. The AR glasses are located at a vertex distance of approximately 12 mm to 15 mm, which is much closer than the near point of the eye, which increases with age from approximately 7 cm (for young people) to over 50 cm (for older people). (The near point of the eye is the closest distance at which the eye can focus). In AR, the synthetic scene is therefore represented by a virtual image that is configured farther from the eye than the AR glasses. VR imaging is simpler than AR because there is no natural scene and only a synthetic scene is generated.

For AR headsets in the form of glasses, the most common technical approach currently is to use a microprojector system. The virtual image that reproduces the synthetic scene is routed to one or both of the lenses of the glasses. (For the sake of convenience in language, we use the term "lens", but it should be understood that the lenses can be invisible, that is, they have no focusing effect in terms of normal vision correction for hyperopia or myopia, etc.). The lenses of the AR glasses are used as a medium to route the light representing the synthetic scene into the path of the natural scene light via a suitable beam combiner. AR beam combiners can be subdivided into reflective types (such as mirrors or prisms) or diffractive types (such as some kind of grating). The virtual image of the synthesized scene should give the impression that one or more objects are depicted in the correct part of the natural scene, meaning not only at the correct position (roughly angle) in the field of vision, but preferably also at the correct distance from the eye.

There are several problems with one or more of the existing AR and VR display technologies.

One problem is how to avoid unwanted light scattering on the lens caused by the projected beam, which detracts from the appearance of the scene and causes the AR glasses to not give the impression of being completely transparent. For example, since the lens is placed approximately 12 mm to 15 mm from the eyeball and since this distance is similar to the focal length of the eye, any unwanted light scattered from the lens is collimated by the eye and spreads widely on the retina. In bright ambient light conditions, this scattering is seen as glare. In dim ambient light conditions, this scattering can be much brighter than the light from the natural scene and therefore severely impairs vision. Light scattering artifacts such as the rainbow effect are particularly problematic when using a grating-based beam combiner.

Another issue related to both AR and VR is how to avoid azimuthal accommodation conflict, which is a property of binocular vision where the normal relationship between accommodation and azimuthal that exists when viewing natural scenes is disrupted by the rendering of synthetic scenes, leading to headaches and nausea. Aazimuthal accommodation conflict occurs when an object in a synthetic scene is perceived to be located at one distance (and the azimuthal is locked to that distance), while the light representing the object originates from a different distance (and the accommodation is locked to that distance). Even in monocular vision, the brain has accommodative expectations based on the brain's understanding of the juxtaposition of objects in familiar scenes, so that headaches and nausea can also be caused by monocular vision of synthetic scenes that cause the eyes to adjust when viewing objects at a specific distance, where the distance is different from the brain's expectation of the object's distance.

The present invention proposes a display device based on a display panel that hosts phase-matched nonlinear mixing of input beam pairs that cross in a non-collinear geometry in the display panel. The crossing occurs in a nonlinear optical (NLO) material region within the display panel. Each crossed input beam pair generates a product beam by sum frequency generation (SFG) or other nonlinear mixing processes in the NLO material.

For both AR and VR, a display device according to the design may be provided for near-eye imaging. A display device according to the design may also be provided for large displays such as large screens used for video conferencing or television.

According to one aspect of the present invention, a display device for displaying a synthetic scene image in response to input of image information is provided, the display device comprising: A first beam source for providing a first input beam of a first frequency and bandwidth; A second beam source for providing a second input beam of a second frequency and bandwidth; A display panel containing an NLO material, the NLO material being phase-matched with the first input beam and the second input beam and a product beam having a sum frequency equal to the sum of the first frequency and the second frequency; An input beam routing component configured to introduce a first input beam and a second input beam into a display panel so that they pass through the display panel in a first path and a second path, the first path and the second path intersecting each other in a non-collinear geometry in the NLO material to define an intersection volume in which a product beam is generated by sum frequency generation; and a controller operable in response to input of image information to form a synthetic scene image by amplitude modulating at least one of the first input beam and the second input beam and by phase modulating at least one of the first input beam and the second input beam to set each of an amplitude, a wave vector, and a wavefront curvature radius of the product beam.

In some embodiments, a first input beam encodes image intensity information defining the intensity of a product beam generated at each intersecting volume of a given synthetic scene image, and a second input beam encodes image depth information defining the wavefront curvature radius of the product beam generated at all intersecting volumes of a given synthetic scene image. This is a convenient arrangement for allowing each synthetic scene image to be associated with a different perceived depth by setting the wavefront curvature radius of each synthetic scene image formed. In particular, when the second input light beam has a substantially planar wavefront, meaning that the wavefront curvature radius is greater than 5 meters, the synthetic scene image is formed at a distance perceived as infinite, whereas when the second input light beam has a wavefront curvature radius less than 5 meters, the synthetic scene image is formed at a distance perceived as finite. It is thus possible to form several synthetic scene images in rapid succession to create different scene layers in an image frame, such as an infinitely distant background and one or more closer layers on which specific image elements are represented.

In some embodiments, the controller is operable to group the synthetic scene images into image frames, different synthetic scene images of a given image frame having different wavefront curvature radii to represent image elements located at different respective perceptual distances.

In other embodiments, a display device is used to display a synthetic scene image in color and includes a plurality of beam sources, the plurality of beam sources including a first input beam source and a second input beam source, wherein the first input beam and the second input beam form one of three input beam pairs generated by the plurality of beam sources, the first input beam pair to the third input beam pair are combined to generate first to third product beams having first to third sum frequencies providing first to third primary colors. In such embodiments, the plurality of beam sources used to form the three input beam pairs may include an emitter array and first to third lasers, wherein the emitter array provides one of the input beams for all three input beam pairs, the other input beam in each pair is provided by one of the first to third lasers, and the first to third primary colors are provided by time division, thereby generating the first to third product beams in sequence. In particular, the array of emitters can be driven to amplitude modulate the input beams it generates. For example, the NLO material can include first to third spatial modulation zones to provide quasi-phase matching for the first to third input beam pairs, respectively. Here, one option is when the first to third spatial modulation zones are formed in the NLO material at the first to third depth portions in the display panel. Here, another option is when the first to third spatial modulation zones are formed as an array of light spot clusters in the NLO material, each light spot cluster includes adjacent light spots of each of the first to third spatial modulation zones.

For a color display in some embodiments, the controller is operable to group the synthesized scene images into image frames, a different synthesized scene image of a given image frame being in each of the first to third primary colors.

For a color display in some embodiments, the controller may be operable to group synthetic scene images into image frames, different synthetic scene images of a given image frame being in each of first to third primary colors and having different wavefront curvature radii, thereby representing color image elements located at different respective perceptual distances.

In some embodiments, the input beam routing component is adjustable to provide a change in at least one of the first path and the second path, thereby changing the angle at which the first input beam and the second input beam intersect in the NLO material, thereby changing the wave vector direction of the product beam.

In some embodiments, the input beam routing component is adjustable to provide a change in at least one of the first path and the second path, thereby changing the location where the first path and the second path intersect in the NLO material.

In some embodiments, the display device further includes an amplitude modulator configured to perform amplitude modulation on the first input light beam and a phase modulator configured to perform phase modulation on the second input light beam. In other embodiments, the display device further includes a combined amplitude and phase modulator for performing amplitude and phase modulation on one of the first input light beam and the second input light beam.

In some embodiments, the NLO material is spatially modulated with respect to its second-order nonlinearity to provide phase matching via quasi-phase matching. In other embodiments, the NLO material is uniform with respect to its second-order nonlinearity to provide phase matching via birefringent phase matching.

Regarding the first frequency and the second frequency and their sum frequency: The sum frequency is preferably in the visible range, i.e., about 790 THz to 405 THz / 380 nm to 740 nm. The first frequency and the second frequency are in the infrared range, i.e., about 400 THz to 150 THz / 750 nm to 2 μm. The ratio of the first frequency to the second frequency is between 0.5 and 2.0.

For AR, display panels containing NLO materials are made transparent in the visible light range (i.e., approximately 790 THz to 405 THz / 380 nm to 740 nm) so that natural scene light can pass through to the viewer's eyes.

In some embodiments, the display panel has a peripheral region containing a material that is absorptive to light at a first frequency and a second frequency, such that when a first input light beam and a second input light beam reach the peripheral region, the first input light beam and the second input light beam are absorbed after the first path and the second path have crossed each other in the NLO material.

In some embodiments, the display device further includes one or more light sensors disposed outside the display panel to detect abnormal leakage of the input light beam from the display panel, the abnormal leakage indicating structural damage of the display panel.

The NLO material may be contained in a display panel in a layer of NLO material. In one embodiment, the display panel includes a front filter layer disposed in front of the NLO material layer, the front filter layer being opaque to the first frequency and the second frequency and transparent to the sum frequency to prevent light from the first input beam and the second input beam from emitting out of the display panel in an outward direction. In another embodiment, the display panel includes a rear filter layer disposed behind the NLO material layer, the rear filter layer being opaque to the first frequency and the second frequency to prevent light from the first input beam and the second input beam from emitting out of the display panel in an inward direction. The rear filter layer may be transparent to visible frequencies. One option is to distribute the NLO material continuously over the NLO material layer so that phase matching and therefore product beam generation can occur anywhere in the NLO material layer. Another option is to distribute the NLO material as an array of spots over the NLO material layer so that phase matching and therefore product beam generation is confined to the locations of those spots.

In some embodiments, the display panel includes a light-shielding layer that includes an array of pixels that are individually addressable by electrical control lines to switch the pixels between a first state that is opaque to visible frequencies and a second state that is transmissive to visible frequencies so that natural scene light can be mixed out from selected areas of the light-shielding layer.

In some embodiments, the first path and the second path pass through the display panel by continuous reflection. The continuous reflection may come from the front surface and the rear surface of the display panel by total internal reflection. Alternatively, the continuous reflection may come from a front mirror layer and a rear mirror layer configured in the display panel, and the front mirror layer and the rear mirror layer are respectively disposed in front of and behind the NLO material. Specifically, the front mirror layer and the rear mirror layer may be reflective at the frequency of the input light beam and transmissive within the visible frequency.

In some embodiments, the NLO material is transparent in the visible range.

In some embodiments, the first frequency and the second frequency are different. In other embodiments, the first frequency and the second frequency are the same, so that the sum frequency generation is a second harmonic generation. When the first frequency and the second frequency are the same, the first input beam source and the second input beam source can be the same beam source, and the first input beam and the second input beam both originate from the beam source.

The display device specified above may be included in a wearable head-mounted headset such that the display panel is configured in front of the wearer's eye at a vertex distance of less than 30 mm for near-eye imaging.

Assuming that SFG is a nonlinear process being utilized, and also assuming that the product beam will be formed at visible wavelengths/frequencies (approximately 380 nm to 740 nm / 790 THz to 405 THz) to create synthetic scene images visible to the human eye, the input beams in each pair will be in the infrared range (e.g., 750 nm to 2 μm / 400 THz to 150 THz), at least with the constraint that they are not too different in wavelength/frequency (e.g., having a ratio of less than 2:1).

As an alternative nonlinear mixing process to SFG, three-photon accumulation mixing can be provided by a combination of collinear mixing second harmonic generation (SHG) in one of the input beams and non-collinear mixing of the second harmonic with the other input beam in the pair.

Phase matching in NLO materials can occur without structuring the material, for example by using sufficiently thin layers of NLO material with high birefringence. However, in many embodiments it will be convenient to use quasi-phase matching (QPM). To achieve QPM, the NLO material is structured to spatially modulate its nonlinear properties. The most common periodic polarization is used to create a grating structure characterized by a single periodicity and alternating sign of the second-order nonlinearity X(2).

The two input beams in a pair are directed through the NLO material at an angle that ensures that the wave vector of the product beam has a direction that propagates to the viewer's eye (ie, through the pupil).

Non-collinear mixing of the first and second input beams in the NLO material allows for the generation of a product beam whose properties are controlled with respect to amplitude (i.e., image brightness), wave vector (i.e., directing the generated light toward the pupil), wavefront curvature (i.e., controlling the perceived distance of the image), and image element location in the field of view (i.e., position on the NLO material).

To create a realistic impression of binocular vision, it is possible to exploit the ability to control the direction of the wave vectors and the positions of the image elements so as to present slightly different images of the same synthetic object to the left and right eye of the viewer, for example, on each lens of a pair of AR glasses, thereby providing natural radiance and associated depth perception (stereoscopic vision) of the synthetic object. For synthetic objects that are close enough for monocular depth perception (i.e., less than about 4-5 meters), radiance simulation can be combined with setting the wavefront curvature of the synthetic object so that the focus (accommodation) of the eye matches exactly or at least approximately the same distance as that associated with the radiance, thereby avoiding radiance-accommodation conflicts. For synthetic objects at greater distances, a planar wavefront can be used.

The nonlinear process produces an oscillatory polarization at the frequency of the product beam (i.e., in the visible range). The process is essentially a wave mixing phenomenon, meaning that the product beam has a lateral (i.e., spatial) coherence with a defined phase relationship between different parts of the emitted waves. Therefore, the product beam produces a wavefront with a specific curvature. This is a fundamental property that is not possessed by light produced by pixel emitters in conventional microdisplays as used in current VR headsets or point emitters as used in current AR glasses. In contrast, the product beam produced by non-collinear mixing of input beam pairs in NLO materials has properties that are more related to conventional diffraction optics methods, which also emit wavefronts via diffraction. A method of generating lateral coherence of light for synthetic scene images is a method that allows setting the radius of curvature of the wavefront, thereby allowing the synthetic scene image to be placed at a specific distance, which specific distance can be changed by changing the radius of curvature of one of the wavefronts of the input light beam. Therefore, an impression of natural depth perception can be left on the synthetic scene image. Different synthetic objects can be given different depth appearances, so they appear at appropriate depth relative to each other, and in the case of AR, at appropriate depth relative to objects in the natural scene. This not only contributes to the perceived realism of the synthetic scene image, but also allows visual azimuthal accommodation conflicts to be avoided.

A display device can be provided that is capable of directing light for a synthetic scene image to an eye from a wide range of angles (i.e., from a wide range of positions across the field of view). This property is beneficial because it accounts for the wide range of angular rotation of the eye. This property is also beneficial because it allows the creation of a synthetic scene image that appears to emanate from an appropriate location within the field of view when the eye is looking straight ahead in a certain direction. In addition, as mentioned above, the product beam light that forms the synthetic scene image can be emitted from the display panel with a defined radius of curvature so that different portions of the synthetic scene image can be placed in the synthetic scene at appropriate perceived distances.

For AR headsets, the display panel may be implemented in a lens format. In a traditional eyeglass format where a pair of lenses are set in a frame for the left and right eye, there will be a physically separate display panel for each lens. In a ski goggle format with a single lens, a single display panel may be used. For AR, the NLO material in the display panel is preferably transparent (within the visible range) so that it can be included in the display panel without reducing the transmission of light propagating from the natural scene through the display panel into the eye. Thus, an AR display panel can be provided that appears truly transparent to a viewer, not only because the NLO material itself is transparent, but also because the wavelength of the input light beam is outside the visible light range (or at least can be easily selected to be outside the visible light range). Not only is the light from the input light beam invisible to the human eye, but more importantly, by configuring a suitable filter layer (e.g., as a bandpass or edge filter) between the NLO material layer and the viewer's eye, it is easy to prevent the input beam light at invisible frequencies/wavelengths from entering the viewer's eyes. The filter is selected to be opaque to the input beam wavelength, but transparent in the visible wavelength range. For AR headsets, it is also possible to adjust the brightness of objects in the synthetic scene image. This can be achieved simply by adjusting the intensity of one or both of the input beams in the beam pair. For VR headsets, the NLO material does not have to be transparent (in the visible range), that is, it can be opaque (in the visible range).

The display panel according to the invention can be considered as a transparent microdisplay, which, like a microdisplay, generates visible light in a localized area (the volume of intersection of the crossed input beams) capable of generating a wavefront on the display. Due to the characteristics of the invention, only visible light is generated for the image, so that the problems associated with stray light and scattering in known microprojection systems for AR (or VR) do not arise. Instead of having visible light beams that pass through the lens to form an image in the field of vision of the eye, we have infrared light beams, i.e. input light beams, which are at a wavelength outside the visible light range and which can in any case be prevented from entering the eye by arranging suitable filter layers between the eye and the NLO material layer. Therefore, there is no glare and bright scattered artifacts that interfere with night vision. Therefore, a display device can be realized that is similar to a microdisplay VR display device in terms of its light generation, but unlike a microdisplay, the display device is transparent (within the visible range), so the display device can be used for AR. Color display

The above discussion of the invention has been restricted to considering the generation of a single product beam from a single input beam pair. This can be considered in terms of describing a monochrome display or, alternatively, describing one of the three color components of a color display. For a color display, three product beams are required, one for each of the three primary colors, so there will be three input beam pairs.

According to another aspect of the present invention, for a color display, a display device for displaying a synthetic scene image in response to input of image information is provided, the display device comprising: A plurality of beam sources having respective frequencies and bandwidths to provide a first input beam pair to a third input beam pair of two input beams, each input beam pair having a pair of frequencies, the pair of frequencies being the sum of first primary color frequencies to third primary color frequencies; A display panel containing an NLO material, the NLO material being phase-matched with the first input beam pair to the third input beam pair and the first product beam to the third product beam at a first primary color frequency to a third primary color frequency equal to the sum of the frequencies of the first input beam pair to the third input beam pair; An input beam routing component configured to introduce two input beams in each input beam pair into the display panel so that the input beams in each pair pass through the display panel in a first path and a second path, the first path and the second path intersecting each other in a non-collinear geometry in the NLO material to define an intersection volume in which a product beam of the input beam pair is generated by sum frequency generation; an amplitude modulator operable to amplitude modulate at least one beam in each input beam pair; a phase modulator operable to phase modulate at least one beam in each input beam pair; and A controller operable to form color image frames, each image frame comprising first to third synthetic scene images generated by first to third input beam pairs, respectively, wherein the controller is operable to form each of the first to third synthetic scene images by controlling an amplitude modulator and a phase modulator in response to input of image information to set a product beam amplitude, a product beam wave vector, and a product beam wavefront curvature radius.

For a color display, the plurality of beam sources may include an emitter array and a first laser to a third laser, wherein the emitter array provides one of the input beams for all three input beam pairs, the other input beam in each pair is provided by one of the first laser to the third laser, and the first primary color to the third primary color are provided by time division, thereby generating a first product beam to a third product beam in sequence.

Although the need to have three input beam pairs implies that there will be six input beams in total, this is not equivalent to the need to have six independently generated input beams. First, the same wavelength can be shared between different input beam pairs, so the total number of wavelengths required to form three pairs does not need to be as high as six, but can be reduced to five, four, or three, with three being the minimum possible number. Second, time division (i.e., multiplexing) can be used so that a common first input beam is used in conjunction with a dedicated second input beam to generate all three colors, one input beam for each color, thereby reducing the total number of independently generated input beams required to four.

For color displays, if QPM is used, the NLO material can be structured with a single grating period for all three colors. The lack of precise phase matching that would exist for at least two of the three colors can be tolerated and compensated for by changing the overall intensity of one or both of the input beams in each beam pair, if desired or necessary. The angles at which the input beams pass through the QPM NLO material can also be chosen to be different for different colors, thereby providing precise phase matching (or at least closer to precise phase matching) for all three beam pairs (i.e., all three colors). Alternatively, to provide quasi-phase matching, the NLO material may be periodically polarized (or equivalently orientation patterned in the case of non-birefringent materials) with three different periods, so that there are three regions of QPM NLO material with different grating periods, each optimized for one of the three colors. Another alternative is to create a single synthetic QPM structure with three phase matching peaks optimized for the desired three colors of the display. The three color-specific QPM regions may be distributed in the image generation plane, for example on a display panel, in a regular or irregular 2D array of spot clusters, where each spot cluster contains three laterally adjacent color-specific QPM regions, i.e., spots. The spots in each cluster are so close together that the fact that they are not at the same location in the field of view will not be noticeable. Alternatively, three color-specific QPM regions may be distributed along an axis perpendicular to the image generation plane, for example, over three different depth portions of the display panel. Here, the color-specific QPM regions may be arranged in a 2D array as in the above-described clusters or may be continuous regions on the display panel as needed to cover the desired field of view for synthetic scene imaging.

In the following, we describe the invention primarily in terms of generating a single product beam (i.e., in terms of generating a monochrome image or generating one color component of a color image). However, it should be understood that a color image can be generated based on three such product beams from three input beam pairs. Image Information for Near-Eye Synthetic Scene Imaging

In order to create a convincing virtual image of a synthetic scene in an image generation plane close to the eye (especially required for AR glasses), it is necessary to accurately reproduce the image position information that would be encoded in the light from the equivalent natural scene. The image position information is a subset of the entire image information and is what defines the perceived 3D placement of objects in the scene. As discussed in more detail below, for monocular vision, the image position information in a natural scene is essentially a combination of the positions at which light from different parts of the natural scene appears in the field of view (i.e., the input angle of the light entering the eye) and the radius of curvature that the light wave from each point in the natural scene has when it reaches the eye. For nearby objects, the wavefront curvature is obvious, while for distant objects, there is no obvious curvature, i.e., the wavefront is effectively a plane wave. For synthetic scene imaging in a display panel close to the eye (e.g., at a vertex distance of about 12 mm to 15 mm, where eyeglass lenses are typically located), the input angle and radius of curvature information from a natural scene can be mimicked by light being generated at a specific location on the image generation plane (so it appears in the correct point in the field of view) and having a specific light propagation direction from that location (so the light generated at that location in the image generation plane enters the eye). In addition to carrying the correct image location information, the light generated on the image generation plane must also have the correct intensity and, for color images, the correct color.

In the context of the present invention, correctly encoding the image location information of the synthetic virtual image involves generating a product beam of light in the NLO material at the correct location (viewing angle) with the correct wave vector direction and with the correct wavefront curvature radius. In order to generate the image, of course, the image intensity information must also be reproduced, which can be done in a straightforward manner by appropriate amplitude modulation of one or both of the first and second input beams. For color images, the image color information also needs to be reproduced, which is done by generating three product beams of three primary colors (e.g. RGB) in the QPM NLO material at each location in the field of view (i.e., providing an appropriate color gamut in the same way as any color projection system).

In order to give a convincing binocular visual perception of the synthesized scene, the respective images formed for each of the left and right eyes will correspond, but will not be identical. Specifically, light from the same point in the synthesized scene is formed with different wave vector directions at different locations in the monocular field of view of each eye for the left and right eyes, thereby mimicking natural radiation and the associated depth perception (stereoscopic vision). It should be understood that the images formed for each of the left and right eyes can be made identical for any portion of the synthesized scene attributable to infinite distance, since in an equivalent natural scene, light from a distant object will arrive at both the left and right eyes as plane waves and at the same location in the respective monocular fields of view of the left and right eyes. Although it should be recognized that in creating a realistic visual scene, any intervening objects can block or occlude the left-eye scene differently from the right-eye scene, so that although both are plane waves, they are emitted to the two eyes from different directions.

We disclose two basic methods for forming a synthetic virtual image in an image generation plane close to the eye. The first basic method is a scanning method that builds each image frame pixel by pixel, such as by rasterization. The second basic method forms the entire image frame simultaneously.

Imaging in angular space (input beam scanning/retina scanning): The first input beam and the second input beam move in both angle and position with respect to their passage through the NLO region to produce a product beam that is scanned over the monocular field of view of the viewer to form an image encoded in angular space using a retina scanning method. Thus, an image frame is created by rapidly scanning visible light over the retina. In more detail, a first input beam and a second input beam are provided, the first input beam and the second input beam having respective beam cross-sections, the dimensions of which are set so that in the case where the first input beam and the second input beam cross in the NLO zone, their intersection volume is small to provide a plane wave that is not diffraction-limited and is mapped onto a point on the retina (focused by the eye), so that the intersection volume at any location in the image generation plane corresponds to a single shot (as a wavefront) of the angular field for creating a synthetic virtual image on the eye. For each intersection volume, the first angle and the second angle at which the first input beam and the second input beam pass through the NLO zone define the wave vector direction of the product beam. The wave vector direction is set to ensure that the product beam is directed to the pupil of the eye. To create a complete image (for one eye), one or both of the first and second input beams are angularly varied such that the angle of the nonlinearly generated light is varied to different angular positions in the (monocular) field of view of the viewer's eye while the input beam intersection volume (i.e., the position where the product beam is generated) remains substantially constant. Similarly, the intensity of the product beam is varied to reproduce the image intensity information, which can be accomplished by amplitude modulating the first and/or second input beams, preferably also taking into account any changes in the conversion efficiency of the nonlinear mixing due to changes in the lateral angle of the input beams as the intersection angle on the image generation plane changes, thereby creating a complete image frame over time. As the input angle into the intersection volume changes, the wave vector direction also changes to ensure that the product beam is always directed to the pupil of the eye. This can be accomplished by appropriately adjusting the angles at which the first and second input beams pass through the NLO material. Scanning the intersection volume within the monocular field of view to create each image frame can, for example, follow a progressive grating pattern. The grating can involve a back-and-forth movement that snakes from left to right, then right to left, and so on.

Overall Imaging (Up-Conversion Analogy): The imaging method can be partially understood by analogy with known up-conversion imaging systems. A first input beam (e.g., up to 1064 nm) contains the image intensity information, and a second input beam (e.g., up to 1550 nm) intersects the first input beam in the NLO material so that an up-converted version of the first input beam is reproduced (as a product beam). This analogy is helpful, but not complete, since it does not take into account the fact that non-collinear phase matching according to the present invention also allows the control of the wavefront curvature of the product beam, thereby including image depth information in the synthetic virtual image. Thus, image depth information is "added" to the "up-converted version" of the intensity modulated first input beam (i.e., the product beam) by phase modulating the second input beam. The wavefront engineering feature of the present invention has no analog in conventional up-conversion imaging systems.

In one embodiment of the invention, there is an implementation that is closest to the up-conversion imaging analogy. The first input beam is amplitude modulated in its beam cross-section to carry image intensity information, which defines a brightness specific to the intersection volume in the product beam. The second input beam is phase modulated in its beam cross-section to carry image depth information, which defines a wavefront curvature specific to the intersection volume in the product beam. In other implementations that are far from the up-conversion imaging analogy, both amplitude and phase are modulated in one or both of the first input beam and the second input beam in their beam cross-sections. For example, the first beam can be amplitude modulated and the second beam can be amplitude and phase modulated.

In a first specific example, both the first input beam and the second input beam are highly coherent laterally (i.e., in the cross-section of the beams). This is the case if the beam sources of both the first input beam and the second input beam are lasers and the beam modulation is performed with a spatial light modulator (SLM) in transmission or with a liquid crystal on silicon (LCOS) device in reflection. Here, we note that both SLM and LCOS devices can be operated to modulate (only) amplitude, (only) phase, or both amplitude and phase.

In a second embodiment, the first input beam is incoherent and is generated by a 2D array of incoherent emitters (e.g. an LED or OLED array) which provides amplitude modulation via the ability to drive each emitter independently. The second input beam is coherent, as required for phase modulation, based on a laser source whose laser beam is phase modulated (e.g. by an SLM or LCOS device) and optionally also amplitude modulated, as already mentioned for the first embodiment.

In the context of an RGB display where three input beam pairs are required for producing three colors, the same incoherent emitter array may be used to produce the first input beams for all three colors, while each color has its own dedicated second input beam, the three second input beams being produced by three different laser sources having three different wavelengths that are added to the wavelength of the incoherent emitter array to produce red, green, and blue light, respectively. Time division multiplexing (i.e., truncation) may then be used to sequence the red, green, and blue composite scene image production via appropriate drive electronics that synchronize the outputs of the IR emitter array and the three laser sources. This sequential driving (or on/off modulation) of the three laser sources provides a temporal RGB imaging similar to the color wheel approach in tabletop projectors. If the refresh rate of the three colors is high enough, the viewer perceives a full-palette RGB image.

Since it is possible to impose both phase modulation and amplitude modulation on one beam to produce a single beam that is combined phase and amplitude modulated (e.g. with an SLM or LCOS device), other embodiments are possible in which the other input beam is not spatially modulated at all. Other Features

Some other features of certain embodiments are now briefly outlined.

To deliver the input beams into the display panel, a convenient method is to introduce the first input beam at or near one end of the display panel (e.g., the left end), and introduce the second input beam at or near the other end of the display panel (e.g., the right end). However, it would also be possible to introduce both the first input beam and the second input beam into the same end of the display panel. This can be achieved by a more sophisticated phase matching scheme that uses appropriate reciprocal lattice vectors for non-collinear phase matching. This can also be achieved by reflecting one of the first input beam and the second input beam from the other end of the display panel so that they cross in the same non-collinear geometry as the first input beam and the second input beam were introduced into the opposite ends.

There are a variety of options for the distribution of the NLO material within the NLO material layer. In some configurations of the device, the NLO material is distributed continuously over the NLO material layer, while in other configurations, it may be attractive to have spots of NLO material arranged in a grid over the NLO material layer. In the context of the spot grid approach, it should be noted that the NLO material we refer to means NLO material that has been phase matched to promote nonlinear mixing, so that if QPM is used, the spot can be a localized region that has been spatially structured for QPM within the same unstructured NLO material layer. It is still possible to produce convincing images using a spot grid approach with confined regions of the product beam of NLO material produced, since the eye is constantly moving. As long as the localized regions are close enough so that light from at least one such region can be directed to the pupil, a convincing image can be formed.

In certain embodiments of the present invention, a first input beam and a second input beam are each introduced into or near one end of a display panel at an angle relative to the front and back of the display panel. Each input beam then propagates across the display panel to a location where non-collinear mixing occurs by one or more reflections. In some embodiments, total internal reflection (TIR) from the display panel-air interface is used to cause each input beam to pass through the display panel, traveling from the location where each input beam is introduced to the location where each input beam intersects with the other input beam in the pair to produce a product beam. In other embodiments, a reflective mirror layer is provided to the front and back of the NLO material layer to reflect at the wavelength of the input beam, so that the input beam passes through the display panel by continuous mirror reflection. One or both of the front and rear mirror layers may be located on the front or back of the display panel, respectively, or may be buried layers within the display panel. Mirror reflection allows reflection at larger angles (i.e., closer to normal) than TIR, which is beneficial if quasi-phase matching is used, since the polarization period can be larger, which is easier to manufacture.

Other functional layers may be added to the display panel. For example, it may be attractive to add a front filter layer to the front of the NLO material layer that absorbs at the wavelength of the input beam (and transmits at the wavelength of the product beam), thereby preventing any scattered input beam light from reaching the eye. In the case where either of the two input beams is a laser beam, this will also help ensure that laser safety requirements are met. A similar rear filter layer may be added to the back of the NLO material layer to prevent the input beam light from escaping into the environment. If the display panel is used for AR, both the front and rear filters should also be transparent in the visible frequencies so that natural scene light is transmitted through the display panel to the eye without attenuation.

Another safety measure is to provide one or more light sensors such as photodiodes outside the display panel to detect abnormal leakage of the input light beam from the display panel, which abnormal leakage indicates structural damage to the display panel. The light sensors can be arranged around the edge or outer edge of the display panel. Then, for example, when the display panel is damaged by a deep scratch, the operation of the display device can be shut down (if necessary). Another possibility for AR is to include a photochromic or electrochromic layer so that the transmission of natural scene light can be reduced, for example in bright ambient light conditions, so that the synthetic scene light is properly mixed with the natural scene light. The dimming operation can be automatic (as in photochromic lenses) or electric. Furthermore, the darkening may be uniform darkening over the entire lens area or may be localized, for example to attenuate light from bright spots in the natural scene. A corrective lens layer may also be provided in front of or behind the NLO material layer (or both in front of and behind the NLO material layer if two corrective lenses are required). If a corrective lens layer is provided, there is preferably an input beam lens layer interposed between the NLO material layer and the corrective lens layer so that the input beam light does not enter the corrective lens layer and thus the input beam path is not affected. In other words, a TIR solution is not preferred when a corrective lens is included.

Optical properties of the human eye and human vision

When designing a display device, it is important to consider the optical properties of the eye and how the eye is controlled individually and in conjunction with the brain, and also to consider how the brain processes image information from the eye.

To enter the eye, light passes through the pupil, cornea, lens and thus reaches the back of the eye where the retina is located. The effective radius of the pupil is controlled by the iris (pupillary muscle). The pupil dilates and contracts depending on the ambient brightness to increase and decrease the amount of light entering the eye. The pupil diameter varies over a range of several millimeters, with a range of 2 mm to 4 mm being the most common. The diameter of the eyeball is about 25 mm.

If you wear glasses, light from distant objects entering the eye passes through an area of the lens that is roughly the same size as the diameter of the iris. The distance from the lens to the front of the eyeball is about 12 mm to 15 mm and is called the apex distance or vertex distance.

The field of view of one eye is called monocular vision and is usually defined in terms of the angular range of what the eye can see when the eye looks straight in a specific direction to fix its vision to a certain point. The monocular field of view extends about 60 degrees laterally inward (toward the nose) and about 107 degrees laterally outward relative to the vertical meridian of the eye. It extends vertically about 70 degrees upward and about 80 degrees downward relative to the horizontal meridian of the eye. Binocular vision is the superposition of the monocular fields of the left and right eyes and is therefore significantly larger laterally. In this document, references to 'field of view' refer to monocular field of view. If reference is made to binocular field of view, this will be explicitly stated.

Normal human vision involves a series of movements of the head from the spine and rotation of the eyes in their sockets. Head movement can be thought of as corresponding to roll, pitch, and yaw angles relative to straight ahead, with yaw resulting from normal left-to-right head rotation, pitch resulting from top-to-bottom nodding motion of the head, and roll resulting from side-to-side motion of the head. The range of eye movement in the human eye is approximately +25 degrees upward, approximately -30 degrees downward, and approximately ±45 degrees left and right. Eye movement is used to place the retinal image of the part of the natural scene that is of greatest concern on the fovea, which is the subregion of the retina with the highest resolution, providing an angular range of approximately 1 to 2 degrees in the field of vision (compared to the approximately 18 degrees of the field of vision of the entire retina (i.e., the macula)). Furthermore, the eyes naturally move around (scanning movements), which keeps the eyes moving to avoid the effects of saturation and unresponsiveness. Thus, the eyes are constantly looking around the scene while the brain performs image processing, which counteracts the blurring effects of eye movement.

For monocular vision, if one considers light emanating from a given point in the natural scene, the light will actually be a spherical wave with its origin at that point. The spherical wave strikes the front of the eye, passes through the cornea, pupil and lens of the eye and reaches the retina. The refractive power of the lens of the eye is modulated by the ciliary muscles to form a focused image of the scene light on the retina. The lens of the eye receives the wavefront, which has a specific central angle and radius of curvature. In the case of light from a distant point in the natural scene (effectively at infinite distance), the wave can be considered to enter the eye as a plane wave, that is, the wavefront is a plane wavefront. For monocular vision perception of most individuals, infinite distance is any distance above about 4 or 5 meters. The angle that the incident light from this distant point makes with the eye axis determines where the light falls on the retina. In the case of light coming from a closer point in the natural scene, the wave enters the eye with a finite radius of curvature, which is perceived in monocular vision by the accommodation of the eye. The eye will accommodate, that is, adjust its focus, to bring the light from the closer point to a sharp focus on the retina. The brain associates the amount of accommodation with the distance of the object, thus creating the perception of distance. (Distance perception also takes into account the radiation effects of binocular vision).

Thus, for monocular vision, we can view a natural scene as a collection of point light sources at different locations in the monocular field of view, where the 3D position information for each point is encoded by the input angle of the light (i.e., the 2D position in the monocular field of view) and the radius of curvature of the light in the eye (i.e., depth). In addition to the 3D position information, this is of course the case: the light from each point also carries intensity information and color information.

In near-eye display systems such as AR and VR, the term "oculomotor frame" (or eye movement frame) is used to refer to the volume of the acceptable view that the eye receives an image, whether that image is real or virtual. A simple definition of the oculomotor frame is the 3D volume visible across the field of view (FOV) for a standard pupil size, taking into account the fact that in human vision, the eye is constantly moving to focus on different areas within the FOV. The specified size of the oculomotor frame in AR or VR goggles is usually larger than the theoretical movement range of the pupil to account for alignment tolerances and variations in pupil distance between people. NONLINEAR OPTICAL (NLO) Materials

One type of nonlinear mixing in NLO materials exploits the second-order nonlinearity X(2). For nonlinear mixing, energy conservation exists, which means that the energy of the photons produced by nonlinear mixing depends on the addition of the energies of the photons of the input beams, which means that the product light frequency is given by the sum or difference of the frequencies of the input beams given the Planck relation.

Phase matching occurs due to the wavelength dependence of the refractive index of the NLO material. The type of phase matching used in embodiments of the present invention is called non-collinear phase matching. Non-collinear phase matching occurs when two (or more) beams with different propagation directions cross to form a beam intersection region located in a phase-matched NLO material. Phase matching ensures that the wave vector of the product beam resulting from nonlinear mixing is the vector sum of the wave vectors of the input beams (and the compensation grating k-vector). The term non-collinear phase matching is used to contrast with the term collinear phase matching, which refers to the situation where the two input beams overlap in space and propagate together so that the wave vectors of the input beams and the product beam are all in the same direction. The non-collinear mixing of two input beams can be characterized by two subtended angles between the product beam and each of the two input beams, which add up to the crossing angle between the two input beams, i.e., the beam crossing angle. In the case of two input beams with wave vectors of equal magnitude (i.e., equal wavelength), the two angles are equal, and it is possible to refer to the angle as the beam crossing half-angle.

In practice, phase matching is often achieved using a method known as quasi-phase matching (QPM), which overcomes the fact that the phase mismatch of the dispersion of the refractive index in the birefringent material does not allow phase matching unless the birefringent material is extremely thin. The NLO material is structured to spatially modulate its nonlinear properties, typically using periodic polarization to create a linear grating structure with alternating sign of the second-order nonlinearity X(2). For noncollinear mixing, the phase matching requirements of the QPM NLO material need to take into account the angle at which each of the input beams passes through the QPM NLO material. Here, it should be noted that the polarization period required for noncollinear mixing will be smaller than that required for collinear mixing, since only the component of the wave vector of each of the input beams will be in the polarization direction.

Known birefringent NLO materials can be used. If the NLO material is quasi-phase matched, known inorganic NLO materials can be used, in particular ferroelectric oxides, which can be patterned by periodic polarization. Examples are lithium niobate (LN) to produce periodically polarized (PP) lithium niobate (PPLN), periodically polarized lithium niobate doped with magnesium oxide (PPMgOLN), lithium tantalate (LT) to produce PPLT, and potassium titanium oxyphosphate (KTP) to produce PPKTP. It is known that ferroelectric NLO materials can be microstructurized using QPM structures (e.g. linear gratings) by electric field polarization. Some non-ferroelectric materials, such as III-V zirconite semiconductor crystals such as GaAs, can also be processed using a recently developed technique called orientation patterning to spatially modulate their nonlinear properties. Although orientation patterned GaAs is opaque in the visible range, other transparent materials (e.g. orientation patterned GaN) can be used. Furthermore, in addition to using inorganic materials such as the above, the polarized NLO material can also be an organic material (or a suitably non-centrosymmetric layer of an organic material). It is also worth noting that electric field polarization of the layer can be achieved. Description of Figures

FIG. 1 is a diagram of how an image is received by the human eye, which shows an eyeball 160 with a pupil 170. The eye receives an image of an object 180, illustrated as a 'stick man'. Light scattered from the object 180 is received at the pupil 170 and then focused by the lens of the eye to the retina at the back of the eyeball, where it is detected by rod cells and cone cells. The light can be seen as light line 184. Typically, the distance between the eyeball 180 and the object is at least 15 cm (or so), which represents the near point of the eye. The distance can be much greater; for example, in the case of starlight, the distance is millions of kilometers or more. The fact that the distance can be 'large' is indicated by line 186 to represent the discontinuity of the scale in the diagram. Light scattered from an object can be represented by wavefronts 188, an inset of which is shown as 190. Each wavefront is separated by a wavelength λ of light. There is a defined phase relationship between any two points on a given wavefront, here indicated as φ_1 and φ_2. In the figure, the wavefront is shown to bend with a wavefront curvature corresponding to a distance 192 from the object 180. The lens of the human eye changes shape under muscle control (the ciliary muscle), and the shape adopted produces a focus of the incident (usually curved wavefront) so that the desired portion of the visual field is in sharp focus on the retina.

FIG. 2 shows an eye 160, an object 180, and the object 180 is disposed at a distance 192 from the eye 160, which is large. The eyeglass lens 20 is disposed in front of the eye. Light scattered from the object 180 that has passed through the eyeglass lens is associated with a wavefront 188. Two sets of light rays are shown emanating from points A and B. With respect to the main visual axis of the eye 160, point A is on the axis and point B is off the axis. Light incident on the eye enters the eye through the pupil as controlled by the iris 172. Light rays emanating from points A and B are associated with respective wavefronts 188A and 188B and are focused by the eye lens 166 onto the retina 168 at two different points 190A and 190B, respectively. The angle that the light makes with the eye's visual axis determines where the image is formed on the retina, while the curvature of the wavefront determines the location of the focus.

FIG. 3 shows that light emitted from a point source close to the eye will have a wavefront curvature that does not match the wavefront curvature of light from a natural scene. An eyeball 160 and a distant object 180 are shown. The wavefront curvature of the light from the object 180 corresponds to the distance from the object, which is "large". A piece of optical material (lens) 20 is close to the eyeball and is shown as housing a point light source 192. The point source 192 can be fluorescence from an optical material (such as a dye or rare earth element) or a scattering point (such as from dust particles). The point source will produce a wavefront 188 with an associated radius of curvature centered on the point source 192. Because the point source is very close to the eyeball (e.g., about 1 cm), the radius of curvature of the light will be small when it enters the eye. This creates two problems to be solved. First, the wavefront emitted from point source 192 is too dispersed to be focused by the eye lens (i.e., the eye cannot produce a sharp image on the retina). Second, the radius of curvature of the wavefront emitted from point source 188 will be very different from the radius of curvature of natural scene light from, for example, object 180. To solve the first problem, the radius of curvature must have a value higher than the threshold value required for the eye to be able to focus an image on the retina. To address the second problem, if the AR light source creates an object on an optical material, then in order to obtain a convincing visual effect, the radius of curvature of the light representing the virtual object needs to be the same or approximately the same as the radius of curvature if the virtual object is a real object. For example, if the real scene is a golf course green with a hole, and the virtual object is a golf ball being pushed into the hole, the radius of curvature of the wavefront representing the golf ball needs to match the radius of curvature of the real scene near the location where the golf ball is presented. This match makes the distance perception realistic for the viewer. This match is also important for a pleasant viewing experience, since large differences in the curvature of the wavefront presented to the viewer in a single scene are known to be unpleasant due to the disruption of the natural match between the point where the eyes focus (accommodation) and the point where the visual axes of the two eyes cross for binocular vision (radius). This is called radius-accommodation conflict, which can induce feelings of headache and nausea, similar to motion sickness. In this context, it should be noted that in AR (or MR) systems, a combination of outward-facing cameras and distance measurement devices such as LiDAR can be used to survey the scene and obtain relevant information about the appropriate layout of virtual objects in the scene. An AR headset may wish to superimpose images of new objects onto a natural scene. This may be to superimpose objects that are not present in the natural scene (e.g. showing two tennis players playing a game superimposed onto a natural scene of an empty tennis court). An AR headset may also wish to replace images of real objects in the scene with virtual objects (e.g. replacing images of farm animals in a natural scene with images of dinosaurs).

FIG . 4 is a schematic diagram illustrating the operating principles used by an embodiment of the present invention. An eye 160 having a retina 168 is shown. A piece of transparent (in the visible range) NLO material 100 in the form of an eyeglass lens having a front (proximal) face and a back (distal) face is disposed in front of the eye at a vertex distance. Light that leaves the spectacle lens from the front face toward the wearer's eye propagates in a proximal or inward direction. Light that leaves the spectacle lens from the back face into the environment propagates in a distal or outward direction. We refer to the lens as an eyeglass lens by virtue of its format, but this does not imply that the lens has any vision correction function.

The first input beam 70 and the second input beam 76 are input into the NLO material 100 and cross, thereby forming an intersection region in the NLO material 100. This is a non-collinear geometry. The first input beam 70 and the second input beam 76 have respective first and second frequencies. The first and second frequencies may be the same or different from each other. In the intersection region, nonlinear mixing occurs by sum frequency generation (SFG), and a product beam is generated. The product beam has a sum frequency equal to the sum of the first and second frequencies.

The product beam produced by nonlinear mixing in the intersection region is effectively a point source, and thus, produces its own wavefront 84 with a wavevector direction that sends the product beam light to the pupil. Product beams are only produced where the beams overlap (i.e., in the intersection region). The NLO material can be chosen to be transparent to visible light, and so a viewer can "see through" the material, as would be required for AR to allow synthetic scene images to be superimposed on natural scenes, as schematically shown by object 180 having a wavefront 188 that is farther away (indicated by discontinuity 186.

As proposed, the physics of an expanded source produced by nonlinear mixing differs from that produced by projection or emission from a fluorescent material in that nonlinear mixing by crossing beams allows wavefront generation and phase matching, which can be achieved, for example, by quasi-phase matching, so that product beams can be produced in the intersection region with in principle any desired radius of curvature.

A synthetic scene image can be generated in an NLO material in a number of different ways. One option is to move the first input beam and the second input beam, as indicated by reference numerals 71 and 77 in the figure, so that the angle at which the beams intersect is a function of the position on the image generation area to "write" the synthetic scene image into the NLO material. To achieve this, it may be necessary to change the entry point of the input beams into the NLO material. If the first input beam and the second input beam are appropriately modulated in intersection angle and amplitude, the synthetic scene image can be created, for example, using some kind of rasterization. Another option is to provide one or both input beams with a patterned cross-section, such as a 2D amplitude modulation, so that when the two input beams mix in the intersection area, a complete object is created by the static superposition of the two input beam cross-sections and their angular spectra. A combination of these two methods can be used with a mix of scanning and beam profile modulation. Within the general concept of light producing synthetic scene images by a phase-matched nonlinear process, it should be further noted that the wavefront 84 produced by this nonlinear mixing can be given a desired direction and a desired radius of curvature, the desired direction being towards the eye, as illustrated.

In summary, an image is created on the retina 168 that combines the wavefront 188 from the natural scene with the wavefront 84 of the synthetic scene image produced in the lens, where the two wavefronts have the same or similar radius of curvature.

Figure 5 illustrates the key concept of quasi-phase matching (QPM). QPM is a property that can be imposed in certain nonlinear crystals via a process called periodic poling. A ferroelectric nonlinear crystal is treated by periodic poling to create a periodic domain structure that modulates the sign of the nonlinear magnetic susceptibility of the crystal so that it alternates positive and negative as light propagates along a certain direction within the crystal. Periodic poling thus simulates the effect of joining slices of birefringent material together into a stack with alternating crystallographic orientations, which is of course not practical, at least for anything other than the very long wavelengths in the extreme far infrared or microwave region. The ferroelectric material most commonly used for periodic polarization is lithium niobate, which after periodic polarization is called periodically poled lithium niobate (PPLN). Another material is periodically polarized potassium titanium oxyphosphate (PPKTP). Recently, a technique called orientation patterning has been developed to allow periodic polarization of non-ferroelectric materials, such as III-V zinc blend semiconductor crystals such as GaAs. In addition to using inorganic materials such as the above materials, the polarized NLO materials can also be organic materials.

As illustrated in the upper portion of the figure, the QPM NLO material 200 comprises a series of alternative layers having an 'upper' domain 202 and a 'lower' domain 204, respectively. The material is structured in this way because the dispersion (variation of the refractive index with wavelength) of the material prevents efficient nonlinear mixing due to phase mismatch. This is shown in the middle portion of the figure, which shows how the refractive index n depends on the wavelength λ. In general, the refractive index of commonly used NLO materials decreases with increasing wavelength (known as normal dispersion). In SFG, three wavelengths are involved. A first wavelength λ_1 and a second wavelength λ_2 mix to produce a third wavelength λ_3, whose wavelength is such that energy is conserved. The wavelengths of the nonlinear mixing (in this case sum frequency mixing) are set such that 1/λ_3 = 1/λ_1 + 1/λ_2 This corresponds to energy conservation.

Because the refractive index varies with wavelength, there is a phase mismatch between the three optical wave vectors. The wave vectors are given by: K_i = 2 * π n_i / λ_i where n_i is the refractive index of the NLO material at wavelength i, and λ_i is the wavelength.

The wave vector matching diagram is shown in the lower part of the figure, where K_3 (longer than the vector sum of K_1 and K_2. This difference occurs because the refractive index n_3 at λ_3 is larger than the refractive indices n_1 and n_2 at wavelengths λ_1 and λ_2. The k_vector mismatch is K_G. In quasi-phase matching, the period of the QPM structure 200 is set so that Lg = 2 * π/ K_G. where Lg is the QPM period. It should be emphasized that in this discussion, the waves are assumed to propagate together in the same direction.

Figure 6 shows a more complex geometry of a polarized NLO material. Instead of a quasi -phase matching structure, which is a simple linear grating structure, a nonlinear crystal is processed by polarizing the regional regions so that their domains point upward out of the plane of the pattern (the 'up' domains) in a nonlinear crystal that naturally has 'down' domains pointing in opposite directions. The shape of the polarized regions may advantageously follow the crystallographic symmetry of the NLO material and may thus be hexagonal, for example, in the case of LiNbO3. The pattern illustrated is of circular polarized domains in a hexagonal close packing (HCP) pattern, but any other distribution may be chosen, such as a square or rectangular stacking of square or rectangular domains. Furthermore, the patterning may be non-periodic. The figure shows the physical structure as having two vectors a_1 and a_2 representing the period of the two-dimensional polarization. The two associated K_G basis vectors b_1 and b_2 are also shown. It should be noted that these can be used for phase matching interactions in non-collinear wave mixing, so that other choices of angles can be used. All that is required is the presence of suitable non-linear grating vectors that allow the phase matching pattern to produce a product beam that is properly directed (i.e., directed to the pupil of the eye).

7 is a schematic diagram of a portion of a block of NLO material 100 wherein the NLO material is spatially modulated in region 108 with respect to its second order nonlinearity to provide quasi-phase matching with crossed input beams 70 , 76. The QPM region 108 is schematically illustrated as a linear grating. The first input beam 70 and the second input beam 76 are introduced into the lens from opposite ends and propagate through one or more reflections (for example, total internal reflection from the inner and outer surfaces of the NLO material 100) (for each of the first input beam 70 and the second input beam 76, one reflection from the outer surface is illustrated) until the first input beam 70 and the second input beam 76 cross in an intersection region overlapping the QPM zone 108. The quasi-phase matching condition is met in the QPM zone 108, and the product beam 84 is generated by nonlinear wave mixing to produce a wavefront that propagates toward the pupil 170 of the eyeball 160.

FIG8 shows another representation of the input beam intersection region. The first input beam 84 and the second input beam 85 have respective wave vectors k_1 and k_2, resulting in a product beam having a wave vector k_g, which is the vector sum of k_1 and k_2. Ideally, for maximum conversion efficiency, the vector sum of k_1 and k_2 (i.e., k_g) should be exactly equal to the wave vector associated with the QPM grating period K_G. A phase matching vector diagram is shown in the figure to the right. It should be noted that the period of the required QPM grating period can be calculated by simple trigonometry from the input wave vectors k_1 and k_2 and taking into account the dispersion properties of the NLO material.

FIG . 9 is a schematic diagram of a display panel including a layer of NLO material 100. It can be seen that a first input light beam 70 and a second input light beam 76 are guided through the display panel 100 by multiple reflections from mirror layers formed on the front and back sides of the NLO material layer, such that the light beams 70, 76 intersect each other in an intersection region 110 and produce a product beam wavefront 84 by nonlinear mixing. It should also be understood that the light beams 70, 76 are focused at the intersection region 110. A distant object 180 is disposed several meters from the eye, indicated by discontinuity 186. Natural light is scattered from the distant object 180 toward the eye 160. Therefore, the product beam light representing the synthetic scene is superimposed on the light from the natural scene light. The light passes through the pupil 170 and into the eye 160 to form a fused synthetic/natural scene image on the retina.

FIG . 10 shows a display panel having a multi-layer structure including an NLO material layer 100, a front mirror layer 136, and a rear mirror layer 130. The front mirror layer and the rear mirror layer are reflective at the frequency of the input beam and transmissive within the visible frequency. Two different input beam paths 70A, 70B of the first input beam are shown to illustrate how the angle and position (θ, z) at which the input beam is introduced into the display panel can be changed, thereby changing the angle at which the input beam intersects the intersection area. The rear mirror layer 138 covers the center portion of the display panel, but does not cover the left end portion of the front face of the display panel. The first input beam is introduced into the left end portion at an angle and position (θ1, z1) or alternatively (θ2, z2). Next, the first input beam is scattered from the front mirror layer 136 onto the rear mirror layer 138, etc., thereby passing through the display panel to the intersection region 108 containing the QPM grating structure 108. Similarly, the second input beam is introduced at the right end portion of the display panel (not shown) and arrives at the intersection region 108 along the beam path 76A or 76B. Next, the first input beam and the second input beam intersect each other in a non-collinear geometry at the QPM region 108 and produce a product beam by sum frequency generation. The required position and angle for introducing the input beam can be calculated simply by tracing the light back from the point where the intersection region 108 is required and at what angle the input beam should intersect the QPM region 108. Because the path length of each light beam changes due to changing the angle and position at which each light beam is injected into the display panel, a lens or other focusing element (not shown) located outside the display panel can be adjusted to ensure that both light beams 70, 76 remain focused in the intersection region 108.

Providing mirror layers 136, 138 allows reflection over a wider range of angles than is possible when relying on total internal reflection from the panel-air interface. Here, it should be noted that if the nonlinear mixing process is frequency-increased, and assuming the common case that the image is formed in the visible region, and further assuming that the frequencies of the mixed light beams are the same or not very different, then the light beams will be in the infrared (IR) wavelength region, so that the reflective coating will be designed to efficiently reflect light at one (or more) infrared wavelengths of the light beam.

Optionally, both the first input beam and the second input beam may be injected from the same end of the lens, and the lens structure may be designed so that the second beam reflects from the reflective element to the right side of the figure, and thus bounces back along path 76A or 76B, as illustrated. It will therefore be appreciated that by appropriately routing and/or scanning the first input beam and the second input beam, they may be introduced at different angles to propagate in the lens, wherein changes in angle require different introduction points to ensure that the intersection region does not move.

It will be appreciated that the first and second input beams can only be accurately phase matched to the QPM NLO material at one particular combination of their respective angles. Therefore, scanning the input beams to vary their angles will result in the phase matching of the crossed input beams to the QPM NLO material becoming progressively less precise as the divergence of the angles from the specified optimal phase matching angle increases. As a result, the conversion efficiency, and therefore the intensity of the mixed product beam light, will progressively decrease as such angles move away from the specified angles. Since the conversion efficiency is a function of the beam angles and will therefore vary as each beam is scanned, for example by rastering, this can be compensated for by temporal variation of the beam intensity during the scan to ensure that the intensity scale produced by the product beam light on the synthetic scene image remains constant.

It is also generally preferred to have the distance between the front and rear mirror layers large enough to keep the number of reflections (bounces) relatively low while allowing a relatively large bounce angle θ. This reduces overlap between light spots (i.e., beam intersection areas), which in turn reduces undesirable waveguide and mode excitation effects.

FIG . 11 shows an example optical component for routing an input beam from a beam source (not shown) to a display panel (not shown). A beam 220 from the beam source hits a first mirror 222A and passes into a second mirror 222B that can be rotated by an angle φ_1, and then passes into a third mirror 222C that can be rotated by an angle φ_2. Thus, different beam paths 224A and 224B can be generated, which arrive at the display panel at different angles and positions. It should be understood that this is only one example of an optical component capable of routing, and other methods can also achieve the same function.

Figure 12a shows another example optical component 230 that may be included in an optical component for routing an input beam from a beam source, in which figure the component is a simple block of optical glass. The optical component 230 allows the beam to be translated, that is, its position to be changed without changing the angle of propagation of the beam. The optical component 230 is mounted so that it can be rotated by an angle θ and is a block of glass or other optical material 230. The optical component 230 is transparent to the beam 220. The beam 220 enters the optical component 230, is refracted at the input surface and passes through the block tracing the path shown. Upon exiting, the beam path is translated downward to the beam path 224A, which is displaced according to the angle θ, but remains parallel to the beam path 220. Thus, a displacement with an angular variation is achieved. For polarized light, the optical element (glass block) 230 may be positioned to utilize the Brewster angle to minimize unwanted reflections, or alternatively, it may be anti-reflection coated. In miniaturized systems, there are various suitable implementation technologies, such as microelectromechanical system (MEMS) mirror technology, electro-optics, and other known beam scanning technologies. The need for miniaturization and limiting the number of moving parts makes solid-state or MEMS devices attractive.

Figure 12b shows a component that directs light to a desired point within a layer of NLO material while changing the angle at which the light approaches the spot. The figure shows an input beam (shown here as coming from the right side of the figure. The input beam (labeled 76 in this case to match the other figures) enters the first lens 240A. The first lens 240A is placed a focal length (f') away from the rotatable mirror 222. Assuming the input beam 76 is roughly collimated, the lens 240A will produce a small focused spot on the mirror 222. . The light reflected from mirror 222 is directed toward the second lens 240B. The second lens 240B is placed at a distance 2f from the mirror 222 and images the small spot from the mirror 222 onto the desired refocused spot marked as 110, since the beam will be delivered into the NLO material where it intersects with another beam. In this example, the distance from the lens 240B to the refocused spot is 2f. The distance is also 2f. These focal length values meet the known imaging requirements. When the mirror 222 is rotated about either axis (see rotation angle θ), the composite beam path will be deflected, but due to the imaging operation of the lens 240B, the light spot will be sent to the same location 110, but will be close at an angle set by the rotation of the mirror 222. Advantageously, the mirror 222 will be rotated so that its center of rotation coincides with the reflective surface of the mirror. It should be understood that different focal length values can be used as long as the mirror is imaged in the beam intersection interaction area. In addition, other optical elements can be inserted into the system to guide the intersection point to the desired area of the display element (AR eyewear system). In addition, additional optical elements can be used to ensure that no matter where the desired interaction area is placed within the display system, it will still be of the appropriate size.

FIG . 13 shows an AR display device according to an embodiment. The first of the input light beams has a phase modulated beam profile and the other of the input light beams has an amplitude modulated beam profile, so that in the intersection region 110 where the two input light beams cross, both the amplitude modulation and the phase modulation are encoded into a product light beam produced by nonlinear mixing of the first input light beam and the second input light beam in the phase matching NLO material layer. The display panel produces an image of the synthetic object. The necessary information for the synthetic scene image is effectively encoded in the angular spectrum of the product beam light that hits the eye. The focusing requirement of the eye lens 166 is to obtain the bent wavefront 181 from the natural scene object 180 and move it to a single focus on the retina 168. The eyeball 162 is shown as having a pupil 170, an eye lens 166 and a retina 168. The pupil 170 of the eye receives light from a natural scene object 180. The natural scene object 180 scatters the incident light, which produces a wavefront 181 having a spherical wave. The light of the wavefront 181 passes through the display panel, shines on the eye, enters the pupil 170 and is focused by the eye lens 166 to form a sharply focused image on the retina 168. If light is emitted from other points on the natural scene object 180 and the light is tracked, the light will be at a different angle to the visual axis of the eye and focused at a different position on the retina 168.

When the AR headset is worn, the display panel 88 is disposed in front of the eye 160. The display panel 88 is transparent to light scattered from the natural scene (such as light contained in the wavefront 188), and additionally produces an artificial image in response to a crossed pair of image-generating input light beams (one pair is shown). Image information is delivered to the display panel 88 via a suitable delivery arrangement. The display panel 88 is shown as being curved and having an outer surface 92 and an inner surface 90 that form an interface from the display panel 88 to the air.

The display panel 88 shown has a composite layered construction having a layer of NLO material and a front view mirror layer 136 and a rear view mirror layer 138. The mirror layers 136, 138 are reflective at the infrared wavelength of the input light beam.

The NLO material layer of the display panel 88 has a QPM zone 108, which is configured to cover a portion of the display panel 88, which is illustrated as a relatively small portion, but it should be understood that the QPM zone 108 can extend over the entire field of view of the eye when the eye is looking straight ahead and/or extend over the entire field of view of the eye at all possible rotation angles and/or extend over the entire area of the NLO material layer.

On the left side of the reference figure, an amplitude modulated first input beam 70 is provided. The first input beam 70 is generated by the emitter array 16, which encodes image information in the cross-section of the beam by driving the individual emitters of the array in an appropriate manner to modulate the amplitude (i.e., intensity). The first input beam 70 is emitted from the emitter array and then reflected by the mirror 222A and transmitted through the relay lens 240A to the left end surface of the display panel 88. The mirror 222A can be deflectable, for example, by rotating along one or two axes to change the path of the first input beam 70. Similarly, the lens 240A can be mounted translationally and/or rotationally to change the optical path of the first input beam 70. In addition, other mirrors, lenses or other optical components may be included for beam manipulation of the direction, position and divergence of the first input beam 70. Thus, the position and angle at which the first input beam 70 is introduced into the end face of the display panel can be adjusted. The left end face of the display panel is illustrated as being disposed at an angle relative to the normal (i.e., the 90 degree cutoff of the display panel) to facilitate in-coupling of the first input beam 70. Alternatively, the first input beam 70 may be introduced into the front face of the display panel. Another alternative is to use grating in-coupling of the first input beam 70, wherein the grating may cover a regional portion of the front face or end face of the display panel. After being introduced into the display panel, the first input beam 70 passes through the display panel from left to right by successive reflections from the mirror layers 136, 138. On the right side of the reference figure, a phase modulated second input beam 76 is provided. The second input beam 76 is generated by the laser source 18. From the laser source 18, the second input beam 76 passes through the beam expanders 240A and 240B to the first plane mirror 222B, where the second input beam 76 is reflected on the second mirror 222C. The second mirror in the beam path 222C is a phase retardation mirror such as a liquid crystal phase modulator (allowing a spatially varying phase to be imparted based on the input) and is used to apply a spatially structured phase modulation to the laser beam. With respect to the introduction of the second input beam 76 into the display panel 88 and the propagation on the display panel 88, the same comments as made above for the first input beam 70 also apply to the second input beam 76. The second input beam 76 is illustrated as being introduced into the right end face of the display panel. The first input beam 70 and the second input beam 76 then cross in the intersection region 110 where the QPM NLO material is located, resulting in a product beam having a wavefront 84 and a frequency at the sum frequency of the first input beam and the second input beam 76. The wavefront 84 enters the pupil 170 and is then focused on the retina 168. The product beam is superimposed on the natural scene light 181 from the object 180. By setting the radius of curvature of the wavefront of the second input beam 76 (by applying a spatially structured phase delay to the beam), the synthetic scene image can also be given any desired radius of curvature so that the synthetic scene image is a virtual image formed at any desired distance from the eye. The size of the laser beam can be reduced by using Gaussian beam optics to change the radius of curvature of the second input beam 76. Reducing the cross-sectional area of the laser beam will increase the beam divergence of the resulting nonlinear wavefront.

Thus, it is possible to vary the virtual image distance of the synthetic scene image to provide a fused synthetic and natural scene image at the eye, wherein the synthetic scene image adapts to the natural scene in a realistic manner.

In the above embodiments, it is implicitly assumed that the QPM NLO material is a simple linear grating. However, in other embodiments, the QPM NLO material may have a more complex spatial profile including some curvature in order to produce a diverging (or converging) product beam. In still other embodiments, it is possible to apply both amplitude information and phase information to either or both of the first input beam 70 and the second input beam 76.

The emitter array 16 may be an OLED array, such as a micro OLED array. There is a controller device 324 that controls the output of the OLED array to create intensity information in the image. Other options for emitter arrays include inorganic LED arrays, quantum dot emitter arrays, liquid crystal devices with appropriate backlighting, liquid crystal on silicon (LCOS) with appropriate lighting, VCSEL arrays, and digital micromirror device (DMD) type tilted MEMS display devices, such as the DLP device from Texas Instruments. It should be noted that the system does not require coherence of the amplitude structured beam, and this is advantageous in allowing incoherent emitter light sources (such as micro-OLED arrays), but also in reducing speckle in the system.

Thus, a first input beam 70 is formed by applying different phase modulations to a beam produced by a single emitter, such as a laser, and has, for example, a Gaussian intensity profile. A second input beam 76 is produced by an array of emitters and thereby has a beam cross-section that carries a spatially varying amplitude modulation. In other embodiments, the first input beam 70 may have a cross-section that is spatially modulated in both phase and amplitude, while the second input beam 76 may be modulated in either phase or amplitude or both. A suitable source for combined phase and amplitude modulation would be a phase controlled emitter array, such as a VCSEL array with external phase modulation or an independent emitter array with appropriate phase control.

An advantage of embodiments that allow both the first input beam and the second input beam to have spatial modulation of their intensity is that this can be used to compensate for the fact that different portions of each beam's cross-section will be more or less accurately phase matched to the QPM NLO material. The display device can be designed so that light along the principal axis of each beam is accurately phase matched to the QPM NLO material at the location where they cross in the QPM region, while the phase matching of the cross beam component to the QPM NLO material will become progressively less accurate with increasing distance from the optical axis of each beam. As a result, the conversion efficiency will progressively decrease with increasing distance from the optical axis of each beam. Therefore, the intensity profile of the beams can be modified to increase the intensity at the end of each beam so that the intensity scale of the product beam light remains constant over the entire intersection region of the beams.

FIG . 14 is a schematic diagram of a beam intersection region 110 where a QPM NLO material 108 is located. A first input beam 70 has a beam cross-section with a nearly Gaussian profile 81 to provide a simple planar phase front, i.e., there is no spatial phase variation on the beam or only spatial phase variation inherent to the beam being a Gaussian beam. A second input beam 76 has a beam cross-section that is, in general, both amplitude modulated and phase modulated according to respective amplitude and phase modulation functions 75. The first input beam 70 and the second input beam 76 in the infrared range interact via second-order nonlinearity of the QPM NLO material 108 to utilize a phase matching scheme as further described above with reference to FIG. 8 and an appropriate nonlinear grating K vector. The nonlinear interaction induces nonlinear polarization at a certain frequency (in this case, the frequency is the sum frequency of the first input beam and the second input beam, which sum frequency is in the visible range) to produce an output beam 82 having a wavefront 84 that varies in both amplitude and phase in a manner determined by the amplitude and phase of the input wave. Thus, the information encoded in the second input beam by the amplitude and phase modulation function 75 is transferred to the product beam wavefront 84.

FIG . 15 is a schematic diagram of an arrangement applicable to FIG. 14. That is, there is a first input beam 70 having a beam cross-section approximately Gaussian profile 81 to provide a plane wavefront that is combined with a second input beam having an amplitude and/or phase modulated wavefront. The second input beam is generated by a suitable emitter array 16 located a focal length f from a relay lens 242, which in turn is located a focal length f from the intersection region 110. This configuration places the image information contained in the amplitude and phase modulated wavefronts from the emitter array 16 in the Fourier plane of the imaging system. Thus, the resulting visible light beam produced by the nonlinear mixing of the input light beams will contain the same spatial image information as that contained in the second input light beam, which was then transmitted to the eye, encoded essentially in a plane wave. The visible product beam light produced by the nonlinear mixing will have a wavefront with a radius of curvature that follows the radius of curvature of the first input light beam 70, so that the product beam light will have an equivalent image position determined by the radius of curvature of the first input light beam 70. The perspective foreshortening caused by the finite intersection angle of the two input light beams can be compensated using simple trigonometry and taken into account in either or both of the beam profile and the construction of the image.

Figure 16 shows a more complex example layer structure of an AR display panel 88. Air 128 surrounds the display and has a refractive index very close to 1.

The figure shows the distal outer side of the display panel 88 farthest from the eye (also referred to as the 'front side' to match the colloquial description of an eyeglass lens) with the surface between the air and the display panel. The figure also shows the proximal surface closest to the eyeball, located on the inner side of the display panel (also referred to as the 'back side' to match the colloquial description of an eyeglass lens).

Each layer is mentioned in order from distal to proximal: • Layer 146 is the front surface anti-reflection coating (ARC) layer AR_1 to reduce reflections of visible light propagating from the panel-air interface in air 128. The outer anti-reflection coating layer 146/AR_1 should preferably be anti-reflective over the entire visible wavelength range to accept light from natural scene sources. Such ARC coatings are typically deposited on optical elements such as camera lenses. It is also necessary to make this layer "hard" to resist scratching. • Layer 142 is the front vision correction layer (labeled N1). It is made of an optically transparent material with refractive power (within the visible range) in order to correct the wearer's vision defects, such as myopia, hyperopia, astigmatism, etc. Vision correction may involve changing the curvature of the layer, or including a polarizing filter as is common in sunglasses. Alternatively, layer 142 may be optically neutral (i.e., providing no image enhancement effect) or omitted. Another possible function of the rear vision enhancement layer 142 is to provide dimming in response to certain natural light conditions. For example, the vision enhancement layer 142 may include a UV-sensitive photochromic material, such as silver halides, as used in known dimming display panels. Another option for dimming would be to use an electrochromic switchable material, such as is sometimes used for automatic dimming of automotive rearview mirrors. For example, the vision enhancement layer 142 may include tungsten trioxide. For electrochromic materials, a power source will need to be in contact with the vision enhancement layer 142 in order to power the dimming function. • Layer 130 is a front filter layer (labeled A1). The front filter layer absorbs the wavelength of the input beam, that is, it is opaque at the wavelength of the input beam, which is typically in the infrared range. This will prevent any light from the input beam from being emitted from the front of the display panel into the air 128. In the case where one or both of the first input beam and the second input beam are generated by a laser source, a front filter layer 130 may need to be provided to comply with laser safety regulations. Front filter 130 may be made of an organic material that absorbs at the appropriate infrared wavelength. In addition, it is desirable for the front filter to transmit visible light so as not to attenuate natural scene light. Such filters are sometimes referred to as heat absorbing filters or 'short pass' filters and may be based on glass such as Schott KG1. Alternatively, front filter 130 may be based on thin film interference effects, which may be appropriate when the light of the input beam is highly monochromatic and narrow band such as produced by a laser. • Layer 150A is a front spacer layer of optically transparent material (in the visible range) and is used to separate the IR absorbing layer 150 from the next layer 136. • Layer 136 is a front mirror layer for internally reflecting the input beams as they pass transversely through the display panel 88. Providing the front mirror layer 136 avoids the use of total internal reflection at the far panel-air interface 146 and also avoids any possible loss, scattering or other undesirable effects of the input beams passing through the outer layers 146 to 150. • Layer 150B is another spacer layer that is optically transparent in the visible range and infrared range (at the wavelength of the input beams). The spacer layer 150B is used to increase the thickness of the material through which the input beams pass, thereby reducing the number of internal reflections ("bounces") required for each input beam to reach the intersection area. • Layer 100 is a layer of NLO material that is quasi-phase matched to the input beams, at least in the area that will be covered by the eye's field of view when it is looking straight ahead, taking into account the angles at which the input beams will cross each other. • Layer 150C is another spacer layer having similar functions and properties to spacer layer 150B. • Layer 138 is a rear mirror layer for internally reflecting the input beams as they pass laterally through display element 88, having similar functions and properties to front mirror layer 136. • Layer 150D is a rear spacer layer of optically transparent material (in the visible and infrared wavelengths of the input beams) and is used to separate rear mirror layer 138 from the next layer 1814. • Layer 132 is a rear filter configured to absorb any stray light from the input light beam (i.e., at a wavelength of the input light beam that will typically be in the infrared range), having similar properties to the front filter 130. Thus, the rear filter 132 prevents light at the wavelength of the input light beam from passing into the air 128, thereby protecting the wearer's eyes. • Layer 144 is a rear vision correction layer, which may have any function or properties as described above for the front vision correction layer 142. • Layer 148 is a rear anti-reflection coating (ARC) layer AR_2, having similar properties to the front ARC layer AR_1/146. Its function is to reduce the reflection of visible light propagating in the air 128 from the panel-air interface.

One particular advantage of this layer structure is that by limiting the optical path of the input beam to a particular portion of the layer stack between the front lens layer 136 and the rear lens layer 138, vision correction can be applied independently of managing the input beam and producing nonlinear mixing of the product beam light. That is, vision correction of refractive anomalies in the eye lens can be achieved in one or more layers located outside the sub-stacks 136-138 (preferably, outside the sub-stacks in the distal (anterior) direction), such as in layer 142, as is accomplished in the illustrated layer stack. In particular, the optical path of the input beam can be managed without regard to any vision correction layers.

It should be noted that some of the above layers may be omitted. The anti-reflective coating is optional. In addition, near-end reflection of the input beam may be performed by total internal reflection from the front panel-air interface, in which case the rear-view mirror layer would be omitted. Similarly, if total internal reflection is used for far-end reflection of the input beam, the front-view mirror layer would be omitted. Depending on the properties of the light source, such as whether the input beams are coherent and their wavelength and maximum output power, the rear filter layer may also be omitted, or both the rear and front filters may be omitted. The front and rear vision correction layers are also optional.

It should be noted that additional layers may be added, for example to provide anti-reflective coatings between the different optical materials involved, but it is worth noting that the maximum undesirable reflections are likely to occur between the air and the display panel (due to the large refractive index difference to air) and between the non-linear layer 100, which may have a significant refractive index difference depending on the choice of NLO material.

FIG. 17 shows an eyeball 162 connected to the brain via the optic nerve 174. The eyeball can rotate within the eye socket and thus align its primary visual axis in different directions (e.g., A, B, and C as illustrated). The display panel 88 is constructed in such a way that the quasi-phase-matched grating vector Kg varies across the lens so that in all rotational positions of the eyeball (e.g., with its visual axis along A, B, or C), the QPM NLO material is oriented to produce a product beam of light that will be scattered toward the iris. This can be achieved by orienting the area portion of the NLO material in the appropriate position and orientation within the display panel. Alternatively, this can be achieved by using appropriate curvatures of the display panel and its front and back faces. The front and back curvatures will affect both the wave vector direction of the product beam within the display panel and the path of the input beam. It should be noted that depending on how the light is introduced into the panel (relative position within the panel and curvature of the reflective layer), it may be advantageous to place Kg in a direction that achieves the best regional phase matching (including the effects of input angle and optical axis).

Figure 18 shows that the display panel can have different shapes. Display panel 88A is planar, having parallel and flat front and back surfaces. This provides an input beam path that is easy to determine. Display panel 88B has a front and back surface with the same radius of curvature (that is, the front and back surfaces have respective curvatures associated with respective center points that are offset). Display panel 88C has a front and back surface that are curved with different radii of curvature so that the two circles have a common center point, thereby providing a display panel of constant thickness. In all cases, the curvature can be on the surface of a cylinder or on the surface of a sphere. Alternatively, different curvatures can exist in the two planes to provide a more complex curved surface.

FIG . 19 is a schematic perspective view of a display device in the format of a pair of AR glasses 10. The glasses 10 include a known portion, including left and right glasses arms (i.e., glasses legs) 32, 34 and a frame having left and right frames for accommodating left and right lenses. A display panel as described above forms each lens. The frame includes a nose bridge, a nose pad, and left and right arm points that may be hinged or not hinged. In order to support its AR function, the glasses arms and frame may be modified to accommodate necessary additional components by means of an internal housing and/or external accessories compared to known glasses. These parts may include electronic circuit components, batteries, light sources, and optical elements. A wireless transmitter or transceiver 40 may be included for communicating according to a wireless protocol such as a Bluetooth or WiFi protocol. The arm-mounted wireless transceiver 40 allows the AR glasses 10 to communicate wirelessly via a communication path 42 with an external control device 44, which has its own wireless receiver or transceiver 43 together with processing capabilities via a processor 46 and associated memory 48 and external network communications 49, which may include WiFi, 4G/5G, optical LAN, wired Ethernet, etc. The external control device may also provide positioning means, such as position sensing via a global positioning sensor (GPS) and images via telecommunication wireless signals and/or wireless network signals. The external control device may be a dedicated stand-alone unit dedicated to controlling the AR glasses 10, or it may be a mobile phone, tablet, personal computer, etc. running with a suitable application (computer program). In turn, the external control device may communicate data with other devices, and in particular, may have access to remote computing resources in an ad hoc network (e.g., as provided by cloud computing). Thus, processing-intensive tasks for controlling imaging in the AR glasses may be delegated away from the AR glasses to the external control device, and optionally also to more remote computing resources with which the external control device communicates. Therefore, by limiting the processing tasks performed on the processor integrated into the glasses to those of relatively low processing intensity, the processing power and associated power consumption of the AR glasses themselves can be kept low.

The eyeglass frame houses a left outward facing camera 50 and a right outward facing camera 52, which face outward to view the natural scene. Using a pair of cameras arranged side by side at the same height allows the establishment of stereoscopic images, that is, binocular vision similar to the binocular vision of the wearer's eyes, thereby allowing the distance to objects in the natural scene to be determined, at least when the distance is relatively short. The eyeglass frame 10 further houses a rangefinder, such as a LiDAR device or a point cloud imaging system 58, for detecting objects in the natural scene and determining their distance from the wearer. In a LiDAR system, the LiDAR device includes at least one laser and a detector, and measures the flight time of laser light from the LiDAR device to a scattering source and back. The laser beam may be scanned to create a map of objects in the natural scene, or multiple laser beams may be generated to allow information to be obtained from multiple points in the natural scene in parallel. The movement of the laser beam due to the natural rotation of the wearer's head may also be used to scan the natural scene. In a point cloud imaging system, the laser beam is projected into the natural scene imaged by cameras 50, 52 to allow a model of the geometry of the natural scene to be created. Object information collected by the camera and rangefinder, in particular object distance information, has a particular synergistic effect with the AR glasses according to the present invention, because the QPM nonlinear hybrid process allows the curvature radius of the wavefront generated in the wavefront of the synthetic scene image to be set to a desired value in response to the distance (and position) information measured by the rangefinder (and camera). For example, if the augmented portion of the image is to superimpose features on objects in the natural scene (or completely replace objects in the natural scene), the augmented portion of the image can be generated with wavefronts having a radius of curvature equal to the radius of curvature they would have if the augmented portion of the image was at the measured distance of the associated natural scene object. Similarly, the augmented synthetic scene image can be correctly placed in the natural scene based on position information obtained from the camera. Of course, position and distance information can also be combined to assist in the segmentation process of the natural scene in order to identify objects in the natural scene.

One potentially useful remote computing resource is to provide mapping information associated with the wearer's live view of the natural scene, for example as an aid in segmenting the natural scene and also to provide realistic lighting when rendering augmented reality objects. For example, this mapping information can come from a mapping application, live satellite imaging, or live flight tracking.

The amount of potentially useful computational resources is virtually unlimited. For example, computational resources can be used to reconstruct an entire simulated version of a natural scene in real time based on what is observed by the camera and rangefinder combined with additional mapping information obtained from removing the computational resources. Augmented reality objects can then be placed in the rendered simulated version of the natural scene and volumes to provide realistic lighting based on textures, etc.

FIG. 20 shows another view of additional elements that may be assembled into a display device in an AR glasses format. • 100 is a transparent or near-transparent optical element containing NLO material and the aforementioned IR beam that produces visible light directed to the eye. • 26 is a glasses frame, which together with the arms (glasses legs) houses the necessary electronic and optical components. • 50 and 52 are outward facing left and right cameras. • 58 is a point cloud or LiDAR system • 60 is an infrared light detector configured on the frame around the periphery of each lens. At least one light detector is required for each lens. IR photodetectors are used to detect light leakage from the input beam and should be substantially completely contained within the lens (and before being introduced into the lens in the arm and frame portion of the glasses). Any increase or increase in the photodetector signal is an indicator of damage. Damage may be a deep lens scratch or physical damage to the frame or arm that exposes the lens end face. In the event of IR light leakage or IR excess, the glasses will be shut down, i.e., the light source will be turned off, thereby ensuring safety, including laser safety in the case of using one or more laser sources. • 54 and 56 are other cameras facing inwards (i.e., towards the eyes) that track eye position. This can be used to help determine the optical image that needs to be transmitted and to ensure that light is only directed when it needs to be picked up by the iris. This saves power and therefore extends battery life. These cameras may also be able to collect information about the viewer's stereoscopic vision so that radiation and accommodation conflicts can be handled and artificial images are correctly positioned relative to the natural scene. Importantly, the two cameras can be used to compensate for different pupil distances (or equivalently inter-pupillary distances) of different viewers and ensure that light from the display is directed to the pupil of the viewer's eye. This has the effect of automatically enlarging the eye box and overcomes a major difficulty with grating based or other AR implementations. • 62 indicates a magnetic compass element which may be included to help determine the compass direction in which the glasses and, therefore, the wearer's head is facing. Again, this information may be processed in the electronics on the glasses or using remote processing and may be fused with the mapping information. • 64 is a set of accelerometers which may be used to track movement of the glasses and may also determine parameters such as roll, pitch and yaw angles of the wearer's head assuming a reference direction for those angles when the head is level and the cervical spine is not twisted. The accelerometer data may be processed within the onboard electronics or by external processing in order to track movement of the glasses and, therefore, movement of the wearer's head.

Although not shown, the AR glasses may have an ambient light sensor for measuring the overall (non-directional) brightness of the natural environment, such as a light detector that is sensitive within the visible region. The output from the ambient light sensor may be used to adjust the overall brightness of the synthetic scene image so that it remains visible in bright light conditions while being less bright in low light conditions (such as when in a dim room or at night).

Inward-facing cameras can be used to compensate for different interpupillary distances for different users and the accompanying changes in the position of the eye relative to the glasses to ensure correct 3D composite scene creation and ensure that light enters the pupil correctly.

FIG . 21 shows a display panel 88 having an additional layer 140 for providing dimming in response to certain natural light conditions. For example, the dimming layer 140 may include a UV sensitive photochromic material, such as silver halide, as used in known dimming display panels. Another option for dimming would be to use an electrochromic switchable material, such as is sometimes used for automatic dimming in automotive rearview mirrors, trains, and commercial aircraft windows. For example, the electrochromic material may be tungsten trioxide. For electrochromic materials, a power source 139 (either a voltage source or a current source) would need to be in contact with the dimming layer 140 via appropriate electrical connections 141 in order to power the dimming function. Another option for a dimming layer would be a liquid crystal layer with adjustable transmission, which could be pixel-based. The dimming effect could be applied over the entire display panel (as in photochromic lenses). Another interesting approach would be to selectively apply the dimming effect only in certain areas of the image. A simple example would be to form dark rectangular areas to provide a suitable background for a synthetic scene image component showing written information, which would otherwise be difficult to read if superimposed on a portion of a bright natural scene. A more complex example is to provide selective zone dimming based on information obtained from the natural scene, particularly objects that have been identified by segmenting the natural scene using data collected by the glasses' forward-facing camera and range finder and/or other data collection sources, which may be obtained from the vehicle's systems, such as when driving a car or flying an airplane. The zone dimming may be exactly aligned with the extent of the object in the natural scene to attenuate light (in whole or in part) from the natural scene object, wherein the regional zone dimming may be done in conjunction with inserting a synthetic scene image into the same regional area. There are various examples where it is desirable to attenuate overly bright elements in the natural scene, which makes it impossible or difficult to view the rest of the natural scene. These include, for example, attenuating light from oncoming car headlights, sunlight when looking up. Another example would be suppressing light pollution when viewing the night sky. Another example is when the AR glasses are configured as a driver aid for night driving. Here, other information from cameras and radars that are part of the car may additionally be included in the image processing. A night vision augmented reality driving experience can be produced that can reveal or enhance natural scene objects that are normally visible during the day but are not visible at night or are too dark to be clearly visible, thereby improving road safety.

FIG. 22 shows a schematic diagram of a pair of AR glasses 10, wherein the NLO material in the lens 100 is formed in discrete regions (i.e., spots 104) on the image area, which spots are embedded in regions of material that are not NLO material, or that are NLO material but as regions that have not been periodically polarized (or otherwise quasi-phase matched to the input beam). Therefore, visible light generation is limited to the spots of QPM NLO material, wherein the remaining regions between the spots are inactive, i.e., unable to generate any significant flux of visible light by nonlinear mixing of the input beam. As the eye moves in its orbit, only a relatively small amount of synthetic scene image light needs to be ensured to reach the eye, and considering that the iris can be 4 mm to 8 mm wide and the lens surface is only about 12 mm from the eye, the NLO material can be limited to a relatively small area spot without image perception degradation compared to the full area coverage of the QPM NLO material. In other words, the fill factor of the QPM NLO material can be quite small, for example, between 5% and 20%, considering the total area covered by the QPM NLO material. For example, the spot can be circular, with a diameter of 0.3 mm to 0.5 mm and distributed in a hexagonal close packed (HCP) grid with a grid pitch of 1 mm to 2 mm. The lower portion of the figure shows details in a portion 104 of a lens according to one implementation option for RGB image generation. Each spot is actually a composite set of three spots 106R, 106G and 106B with different QPM structures to generate red, green and blue light, respectively, allowing the formation of RGB images. Alternatively, RGB imaging can be achieved when the three QPM structures for red, green and blue are spatially superimposed (i.e., embedded at different depths z at the same xy location within the lens). Another alternative for RGB imaging is to polarize the NLO material according to a single more complex polarization pattern that is phase matched to all three colors (red, green and blue). There is an advantage in this approach that the lens is designed with a grid of spots where mixing can occur, rather than a design with a continuous area of QPM NLO material over the portion of the lens where the synthetic scene image is to be formed. That is, nonlinear mixing (i.e., visible light generation of the synthetic scene image) can only occur at specific locations where the spots exist. This simplifies beam management, such as beam routing and beam modulation, because beams crossing outside the spots do not generate any visible light and can therefore be allowed.

Figure 23 plots the dependence of the QPM grating period P on the angle θ, which is 90° minus the half angle of reflection from the normal to the eye axis. The example is calculated for the nonlinear mixing of two 1064 nm beams to produce green light at 532 nm. It can be seen that if the display panel is traversed by the input beam by shallow angle reflection (right end of the figure), the required period of the QPM grating becomes quite small, which may make manufacturing more difficult and less accurate. For example, at 60 degrees (i.e., 30 degrees), the required periodicity is about 400 nm in the UV, and at 70 degrees it is about 300 nm. At angles close to the critical angle for total internal reflection (i.e., about 40 degrees for a typical glass-air interface), the grating period is about 800 nm. If the lens has an inner and outer infrared reflective material layer, the input beam passes through the lens by specular reflection which enables the use of larger angles of reflection. For example, at a 2 micron grating periodicity, the angle is about 22 degrees. On the other hand, each reflection is responsible for losses, for example by scattering, so the optimal angle range to be chosen is the one where there is a good compromise between ease of manufacture (and precision) and having sufficiently shallow reflections to keep the total number of reflections required to cross the input beams in each beam pair relatively small.

FIG . 24 is a schematic diagram of a structure for introducing a first input beam 70 and a second input beam 76 into an eyeglass display lens. The first input beam 70 and the second input beam 76 are introduced into the respective distal and proximal lens layers in the dual-layer lens structure in the lens arm or temple side 264 of the lens. Each input beam 70, 76 passes through its lens layer. At the nose bridge side 266 of the lens, the first input beam 70 in the distal layer is reflected by the mirror surface 260 (two mirrors at 90 degrees to each other) back into the proximal layer, so that it propagates in the opposite direction with the second input beam 76 to reach the desired area 110 where the beams overlap within the NLO material layer. There is an additional mirror layer 262 within the display element which is reflective at the infrared wavelengths of the first input beam 70 and the second input beam 76. Thus, with this design, the first input beam 70 and the second input beam 76 are optionally crossed in the region 110 and introduced simultaneously into the same end of the lens, which end may conveniently be the side of the eyeglass leg as illustrated. It should be noted that other methods may be used to introduce the first input beam and the second input beam, including routing around the periphery of the lens using a suitable light pipe or waveguide structure. Likewise, it is not necessary to use two separate layers (as in the figure), as it is also possible to use a single layer to retro-reflect the first input beam 70.

Figure 25 shows a temporal truncation sequence of a single instance frame overlaying both RGB and ROC temporal multiplexing to produce a synthetic scene image. To form an image frame, a series of time slices are provided within time t, each time slice being specific to a primary color, such as red, green, blue (RGB); and a radius of curvature (ROC), such as 0.25 meters. Subframes of three time slices produce a synthetic color image at one ROC. In order to produce a color synthetic scene image containing objects at multiple distances in the scene, multiple such subframes are required, one subframe for each ROC. In the full frame, synthetic sub-images are formed at multiple different perceptual depths in the synthetic scene. Thus, by displaying related display fields in sequence, it is possible to create a convincing white light effect and render any color, as in known displays such as RGB-based projectors. The example curvature radii shown are 0.25 m, 0.5 m, 1.0 m, 2.0 m, and 4.0 m. In this example, they are shown in a monotonically increasing sequence, which is then repeated. However, it should be understood that any given frame may use an arbitrary sequence of time slices. In practice, sequences different from the sequence shown may be more efficiently generated by the hardware and/or produce images that are perceived by human vision as having higher quality. The five example ROC values shown have a minimum ROC value at 0.25 meters, which is approximately equal to or just below the near point of normal vision. This value enables objects that are very close and even slightly too close to be displayed in focus. The maximum distance in this example is 4 meters, which is actually the same as the maximum distance of a normal eye in monocular vision (or infinity). It should be understood that a smaller or larger number of distances can be selected, and it is also possible to control the distances and frames adaptively. In addition, the number of ROCs per frame can vary depending on the content of the synthesized scene, for example it can be reduced or increased as certain objects leave or enter the synthesized scene. As previously discussed, the ROC value is set using a nonlinear blending process and changing the intersection angle of the first input beam and the second input beam.

FIG . 26 shows a schematic cross-sectional view of an eyeglass format system including standard components of an eyeglass frame 26 housing left and right eyeglass lenses 22, 24 separated by a nose bridge 28 and having left and right eyeglass arms (eyeglass temples) 32, 34. In this embodiment, left and right inward facing cameras 54, 56 are mounted on or adjacent to the nose bridge 28 for capturing images of the left and right eyes 162, which are shown in cross-section through the eyeballs. The inward facing cameras 52, 54 are illustrated as being located at the nose wing of each lens. However, it should be understood that alternative arrangements may be used, such as on the side of the temples of each lens. The function of the inward facing cameras 54, 56 is to track the gaze direction of the person wearing the AR glasses. Providing inward facing cameras for eye tracking allows for multiple advantageous features and functionalities to be provided. First, the position of the pupil (and therefore the gaze) allows the system to illuminate the appropriate area of the display so that the viewer sees the image in the correct place. With the help of the outward facing camera and information from LiDAR or other predetermined scene information plus position and orientation (from a compass or accelerometer), appropriate artificial scene information can be displayed. Secondly, the eye position can be used to create a high resolution image that will be received by the fovea of the retina, which is the part of the eye with the highest resolution. Thus, a limited number of spatial pixels on a modulator or emitter array can be used to create a high resolution image, and then, as the gaze shifts, the input beam can be directed to a different area of the eyeglass display lens and a high resolution image is produced in that new location. It should be noted that displaying information (of lower resolution, but with the correct color and intensity) in more peripheral portions of the field of view (by directing appropriate beams into other areas of the lens) can also be helpful in creating a compelling synthetic scene. A third feature of eye tracking is that it can be used to darken or break up the synthetic scene as the eye shifts gaze (over larger angles) so that the viewer does not perceive unwanted ghosting. A fourth advantage of eye tracking with the aid of inward facing cameras 54 and 56 is that they ensure that all generated light is directed towards the pupil of the eye, thereby both increasing the power visual efficiency of the display and also avoiding visible light being directed to other parts of the viewer's eye (or other locations on their eyelid or surrounding skin), which could be perceived by another person observing the wearer of the AR glasses display system. A fifth advantage of eye tracking is that the system can be adapted to compensate for different pupillary distances (the separation between the eyes). Since the eye distances of different people may be different, it is advantageous to be able to adapt the system to direct light to an individual's eyes. This can be achieved by changing the input angles of the two light beams entering the NLO material in order to steer the wavefront direction of the emitted light.

FIG . 27 shows a time-truncated sequence of a single instance frame overlaying RGB and ROC time multiplexing to produce a synthetic scene image that additionally includes a scanning motion blanking period 282. When the system is blanked, the illumination (one or both of the input beams) is turned off or switched to low intensity so that no synthetic scene image is formed during the blanking period. The decision as to when to blank is made by the control system 280, which performs signal processing based on eye tracking information obtained from the inward-facing cameras 54, 56. The purpose of blanking is so that no light is sent to the eye when the eye is moving. This avoids unwanted ghosting and striping in the field of view. This is particularly important in pulsed systems, since the pulse causes a transient image to be projected onto the retina. The pulse can occur because one or both beam sources are pulsed (e.g. pulsed lasers) or because of pulse effects caused by time division of the color and radius of curvature RGB, ROC.

FIG . 28 shows, by way of example, an absorption layer arranged in a peripheral region of a display element lens. The lens 20 is surrounded by an absorption material region 290, which is placed there to absorb the input beams after they have crossed in the NLO material and reached the peripheral region, thereby preventing further propagation of the input beams by reflection from the edge of the lens after they have reached their destination. The absorption material may advantageously have a refractive index similar to that of the material of the lens (to avoid Fresnel reflections), while including the absorption function via molecular absorption, dye absorption, or absorption by black absorbing particles. The same design may be used for display panels of other formats, such as large area displays for conferencing or a single panel for both eyes as may be used in a VR headset format. What these designs have in common is that there is a peripheral region containing material that is absorptive to light at the first and second frequencies, so that when the first and second input beams reach the peripheral region, they are absorbed after the first and second paths have crossed each other in the NLO material. There are also photodiodes 60 disposed at the periphery of the lens 20, which, as previously described, can be used to check the integrity of the lens structure and provide a shut-off safety feature in the event of damage. The lens periphery also includes a non-absorptive region 292 through which the input beam 70/76 is injected into the lens 20, i.e., the absorptive material region does not extend completely around the periphery of the lens 20.

FIG. 29 ( upper portion ) shows a schematic diagram of a pair of AR glasses 10, wherein the NLO material in the lens 100 is formed in discrete regions (i.e., spots 104) on the image area, which spots are embedded in regions of material that are not NLO material, or that are NLO material but as regions that have not been periodically polarized (or otherwise quasi-phase matched to the input beam). Therefore, visible light generation is limited to the spots of QPM NLO material, wherein the remaining regions between the spots are inactive, i.e., unable to generate any significant flux of visible light by nonlinear mixing of the input beam. As the eye moves in its orbit, only a relatively small amount of synthetic scene image light needs to be ensured to reach the eye, and considering that the iris can be 4 mm to 8 mm wide and the lens surface is only about 12 mm from the eye, the NLO material can be limited to a relatively small area spot without image perception degradation compared to the full area coverage of the QPM NLO material. In other words, the fill factor of the QPM NLO material can be quite small, for example, between 5% and 20%, considering the total area covered by the QPM NLO material. For example, the spot can be circular, with a diameter of 0.3 mm to 0.5 mm and distributed in a hexagonal close packed (HCP) grid with a grid pitch of 1 mm to 2 mm. There is an advantage in this approach that the lens is designed with a grid of spots where mixing can occur, rather than a design with a continuous area of QPM NLO material over the portion of the lens where the synthetic scene image is to be formed. That is, nonlinear mixing (i.e., visible light generation of the synthetic scene image) can only occur at specific locations where the spots exist. This simplifies beam management, such as beam routing and beam modulation, because beams crossing outside the spots do not generate any visible light and can therefore be allowed.

FIG. 29 ( lower left portion ) shows a first approach for achieving each spot. That is, spot 103A comprises a stack of three different QPM periods, 206R, 206G and 206B, designed to be phase matched with an appropriate nonlinear mixing process for generating three different wavelengths, such as red, green and blue wavelengths, respectively. In the illustration, the layers are shown as being in contact (as may be achieved by electric field polarization), but it should be understood that gaps of inactive material may be interspersed.

FIG. 29 ( lower right portion ) shows a second method for realizing each spot. That is, the spot 103B is realized as a superstructure grating 208, which is designed so that phase matching can be performed for three colors (e.g., red, green, and blue) at the same time. This can be produced by polarizing the NLO material according to a single more complex polarization pattern that phase matches all three colors (red, green, and blue). A superstructure grating is also illustrated, whose layers are tilted slightly away from the plane of the NLO material layer. The grating for each spot can be tilted in a manner that varies from one spot to another within the lens area so as to guide the phase-matched output beam toward the pupil - as further described above with reference to FIG. 17. The same area tilting method can be used for the multi-stacked light spots 103A described above.

FIG. 30 shows two schematic diagrams of a lens 20 for AR glasses. The upper part of the figure shows a front view of the lens 20, and the lower part of the figure shows a top view, that is, a view as a cross section through the lens. In the front view, the first input light beam 70A, 76A and the second input light beam 70B, 76B enter the lens 20 from the left and right sides of the lens 20 through the entrance areas 292A and 292B, respectively. The first input light beam 70A, 76A and the second input light beam 70B, 76B cross at two different crossing positions 110A and 110B. When the eye behind the lens turns, the line of sight will be guided to many such locations, and it is necessary to create synthetic scene information that will be received by the pupil of the eye. In this example, intersection position 110A relates to a gaze directed upward, while intersection position 110B relates to a gaze directed downward. As examples, two different arbitrary first input beam directions 70A, 76A and two different second input beam directions 70B, 76B are shown. The lower portion (top view) is a slice of the lens showing the rebound path. It should be understood that the respective intersection angles in the nonlinear overlap regions 110A and 110B need to be considered in three dimensions, as can be clearly seen when the front view and the top view are considered together, where the input beam injection direction shown in the front view and the lens passing through due to rebound are related. From this figure, it should be understood that the direction of the vector that crosses at a given point on the lens 20 depends on the location on the lens at which the gaze is directed. The resultant wavefront needs to be directed to the viewer's pupil, and therefore the phase matching diagram (which is now a 3-dimensional vector) must be designed so that the light is properly directed. This can be achieved by placing the QPM direction at the appropriate orientation on the lens surface so that phase matching in the desired direction can be achieved.

It should be noted that although the phase matching period and QPM grating vectors will be determined when the lens is manufactured, there are still controllable degrees of freedom to ensure that the light obtained from the NLO process is directed to the eye. First, at a given crossover position 110A or 110B, the bounce angle θ of the first input beam and the second input beam can be independently controlled (see Figure 10). Second, the wavelength of the first input beam and/or the second input beam can be slightly changed to change the phase matching condition, but this will require at least one of the beam sources to be tunable. As already discussed, by using an inward-facing camera, it is possible to ensure that the light from the system is efficiently directed toward the pupil and is suitable for different pupil distances. As illustrated, the glasses can be configured so that the beam enters each lens at the mid-height positions 292A and 292B from opposite sides of each lens. The light beams are then guided to cross at different locations 110A and 110B as needed. As just discussed, in order to ensure efficient guidance of light from the lens to the wearer's pupil, it is necessary to regionally set the QPM grating vectors at the crossing locations 110A and 110B to ensure that a phase-matched wavefront (or a nearly phase-matched wavefront) is propagated to the pupil, that is, so that the wavefront has an appropriate wave vector.

FIG . 31 shows an example lens 300 suitable for VR goggles. The lens 300 has a layer of NLO material 100 embedded therein. The first input beam 70 and the second input beam 76 enter the lens 300 from the distal side, which is possible for VR where there is no natural scene that is transmitted on the eye and intersects in the NLO material layer 100 at the intersection area 110. The fact that the lens 300 can be made much thicker in VR than is possible for AR also helps with the injection of the input beams from the distal side of the lens 300. The thickness of the lens 300 can be, for example, 50 mm. The illustrated form of lens 300 is a roof shape having first and second inclined facets, the surfaces of which form approximately normal angles at the injection point with the first and second input beams 70, 76. The other features of the system remain the same as previously discussed for end injection of the input beams in a bounce-through manner to reach the intersection region where the input beams cross.

FIG. 32 schematically illustrates image generation in an NLO material layer with a nonlinear QPM grating 108 when the source of the first input beam is an incoherent emitter array 321 that emits multiple objects as incoherent infrared light (e.g., from a monochromatic micro-LED array) as the first input beam. A lens 325 is disposed between the emitter array 320 and the QPM grating 108, at a focal length f from each of them. Three emission regions are shown as pixels 322, 323, and 324. Each of these has the same radius of curvature. An input infrared laser wave 317, used as the second input beam and having an intensity profile and a phase profile (corresponding to a radius of curvature), is shown entering the QPM grating 108 (from the left side). The input laser wave 317 can be given three different radii of curvature 318, 319 and 320 at different times. By varying the ROC of the input laser beam from the left side and by providing different intensity patterns from the micro-LED incoherent emitters, it is possible to produce continuous wavefronts that can be rapidly cycled in sequence so that a viewer perceives different objects to exist at different apparent focal locations of the visible nonlinear waves emitted from the QPM grating 108 as a result of nonlinear mixing of the input beams. Overview of options for phase and amplitude modulation between beams

In the following, we outline possible methods for performing amplitude and phase modulation functions, particularly with respect to whether both are applied to one of the input beams or one is applied to each of the input beams and whether the beams are large area beams (or equivalent beam arrays as produced by an array emitter) or rasterized pencil beams. While the following is intended to provide a comprehensive overview of the options, it is not exhaustive and other variations will be readily understood, for example by combining elements from the following specific examples. It should also be noted that the labeling of the two input beams in the form of Beam 1 and Beam 2 is arbitrary. Table A: Example Method for Amplitude Modulation describe Beam 1 Beam 2 Examples of Modulation Methods Note Ai) Grating scanning the entire beam method Amplitude modulation across the entire beam Phase Modulation (See Table B) Laser current drive, or electro-optic modulator, or uniform liquid crystal modulator Modulation can be performed in reflection or transmission. Aii) Laser-like beams with complex amplitude modulation across the beam The desired amplitude pattern is imprinted across the input laser beam Phase Modulation (See Table B) 2D modulators are required (e.g. reflective LCOS, or transmissive liquid crystal, or DLP-type MEMs arrays) Note that the amplitude of the beam must be sufficient to produce a virtual image for the eye. Amplitude modulators can operate in either reflection or transmission. Aiii) Transmitter Array Beam 1 is a 2D array of individual beams generated by the emitter array in a 2D modulated manner Phase Modulation (See Table B) Micro-LED arrays or OLED arrays, where the drive current is used to set the amplitude for individual pixels The emitter array can be incoherent (such as in an OLED or micro-LED display) or can be coherent (such as a VCSEL array) but with amplitude control Aiv) Composite modulator array/emitter mix Beam 1 is a 2D array of individual beams generated by a hybrid modulator and an emitter array in a 2D modulation manner Phase Modulation (See Table B) Similar to the above, but the emitter function and the modulation function are created via a hybrid or compound approach. For example, by placing an LCD modulator directly on the LED array or similar Av) phase and amplitude of the combination The beam 1 is amplitude modulated and phase modulated and sent to the NLO interaction region. Beam 2 is the unmodified laser beam delivered to the NLO interaction region. See Table B for examples of methods used for phase modulation. The device may include a hybrid structure in which phase and amplitude are modulated together in the same modulator, or phase and amplitude modulation may be applied sequentially to a single one of the input beams by a dedicated modulator. Table B: Example method for phase modulation describe Beam 1 Beam 2 Examples of Modulation Methods Note Bi) Spatial phase modulation across laser beams Adjust according to Table A Produces spatially varying phase modulation of the desired radius of curvature Pockels cell device with radially varying electric field or transmissive LC phase modulator that produces a secondary phase on a beam This example is in transmission and can use materials such as KTP, BBO, KDP, KTN, LiNbO3 or organic electro-optical materials. Bii) Spatial phase modulation across laser beams (II) Adjust according to Table A Produces spatially varying phase modulation of the desired radius of curvature The LCOS mirror device comprises a suitable liquid crystal layer with a locally addressable electric field strength Such a device will typically operate in a reflective manner, for example where the liquid crystal is placed on a reflector and addressed per pixel. However, a transmissive device may be used instead. Biii) Spatial phase modulation across laser beams Adjust according to Table A Produces spatially varying phase modulation of the desired radius of curvature Examples would include MEMS-based deformable mirrors or regional thermally deformable mirrors. Typically, the device will operate in a reflective mode. Biv) Amplitude and phase of the combination Modulation is performed using both amplitude modulation and phase modulation across the beam Unmodified laser beam delivered to the NLO interaction region (guided only on site) As discussed in Table A, by creating a hybrid/composite element Same as Aiv, but with the two beams' roles swapped bv) Diffraction control of ROC Adjust according to Table A Change the intensity of beam 2, and then use the diffraction effect to change the ROC Amplitude modulators are used to spatially control the range of a beam in order to increase or decrease diffraction effects. Spatially varying amplitude modulators (such as LCOS or DLP or variable diameter irises) can be used. [1] If a beam is perforated in this way, compensating changes in the total beam intensity will need to be made. Diffraction control can be performed in transmission or reflection. bvi) Gaussian beam optical control of ROC Adjust according to Table A Change focal length/relay optics to change spot size and input ROC Variable focus lens-type elements, or liquid-filled lenses, or square electro-optical or liquid crystal elements with electric field drive

Figure 33a shows a series of panels illustrating a particular method of controlling the amplitude and phase of two input beams as outlined in Table A. The panels are labeled using reference numbers taken from the corresponding columns of Table A. It will be appreciated that hereinafter, for convenience, the two beams are hereby labeled 70 for the beam from the left side of the interaction region 110 and 76 for the beam from the right side, although of course the directions of the beams may be interchanged so long as the beams cross in the appropriate overlap region 110. It may also be convenient to design the system so that it is mirror-symmetric for the right and left lenses, thereby maintaining the symmetry of the two sides of the eyeglasses. For example, in two glasses, one of the input beams (e.g., the first input beam carrying phase modulation) is injected from the side of the nose bridge, and the other input beam (e.g., the second input beam carrying amplitude modulation) is injected from the side of the glasses leg (or arm).

FIG . 33a shows in panel Ai) a scheme suitable for grating scanning and overall beam modulation methods. It shows two beams 70 and 76 interacting with each other in a non-linear region 110. The first beam is emitted from a laser 18, which is driven by a laser controller unit 19. The laser controller unit provides power to the laser and, by varying the current, can modulate the output power as the beam is scanned at a certain angle to create an image. The figure shows discontinuities 186 to indicate that there will be other optical elements placed in the optical path (e.g., at a scanning angle, etc.). In this embodiment, the power of the laser beam 70 is controlled by the laser controller unit 19. The second beam 76 is shown as having a phase profile 80 suitable for controlling the radius of curvature of a composite non-linear beam (not shown). The intensity of the overall beam 70 may also be controlled by placing a modulator in the beam path as discussed in Table A. The modulating element may be transmissive or reflective.

Figure 33a shows in panel Aii) a beam produced by a laser having a complex amplitude modulation imprinted on the beam. This differs from the previous examples in that the intensity of the entire beam was modulated because a spatial light modulator is used to vary the intensity of the beam. The figure shows two beams 70 and 76. The first of these 70 is amplitude modulated by a spatially varying reflective amplitude modulator 120 (e.g., a reflective LC-based spatial light modulator). The figure shows an amplitude modulator controller 121 electrically connected to the modulator 120. The example also shows a laser 18 and a laser controller 19. The laser controller can control the total amount of light in the beam, while the modulator controller 121 drives the modulator 120 to produce a spatially varying intensity profile on the laser beam. As previously described, discontinuities 186 indicate that other optical elements will be present in the beam path of 70. All controller elements are controlled by a central controller (not shown) that controls the imaging. The second beam 76 has a controlled phase profile 80.

Figure 33a shows in panel Aiii ) a system in which amplitude modulated light is produced by a device having multiple individual emitters, such as a micro LED device. In this method, a first light beam 70 is produced by individual pixel emitter sources from an emitter array 321. The emitter array is driven and controlled by a separate controller 324, which sets the intensity of each emitting element. Light emitted from two pixel locations is shown. Note that in general, the array will be a 2-dimensional array. In this figure, the light emitted from the emitter array 321 is shown as diverging (as is the case with a micro LED device), and a lens 240 is placed to collect the light and direct it in the direction of the first light beam 70. It should be understood that the first beam 70 no longer resembles a laser beam, but rather is radiation emitted by an emitter pixel and directed by the optical system to the interaction region 110. An array of emitters may be incoherent between pixels (such as is the case with a micro-LED array), or may have coherence between pixels (such as is the case with a single large laser spot that would have an amplitude modulator placed on its emitting surface). Alternatively, an array of emitting lasers may be used (such as an array of VCSELs), and again may be coherent or incoherent between each emitter. (Note that a VCSEL is a vertical cavity surface emitting laser). The provision of a hybrid or composite emitter is described in Table A as Example Aiv), which describes placing a modulator (such as a liquid crystal modulator) on the emitting surface of a larger emitter (such as a laser or LED).

Figure 33a shows a combined phase and amplitude modulator in panel Av) . This example is a variation of the example shown in panel Aii. A laser 18 and a laser controller 19 are shown. Light from the laser is reflected by a combined amplitude and phase modulator 123a. This is controlled by a controller 123b which imparts both amplitude information and phase information to the first light beam 70. The combined amplitude and phase modulator 123a may be a single device, such as a modified LCOS device with a phase control layer, or it may be implemented using adjacent separate elements. Similarly, the device may be operated in either a transmissive or reflective mode. The second light beam 76 is no longer imparted with any controlled phase because the function of changing the radius of curvature of the emitted light is now achieved by applying phase information to the first light beam 70. Therefore, the phase 80 is now flat across the beam. However, the beam may still have some more complex phase function (for example it may be focused), but is not modulated when producing the image to create the radius of curvature.

FIG. 33b illustrates a method of controlling the phase on a beam according to Table B. It shows a series of panels illustrating different methods. The panels are labeled using reference numbers taken from the corresponding columns of Table B. The panel schematics focus on illustrating the role of the second beam (labeled 76 in the figure), but it should be understood that the labeling of the first beam and the second beam with 70 and 76 is arbitrary. Similar to the description of FIG. 33a, it should be recognized that it is convenient to construct a system with image symmetry in the left and right lenses of a particular display device (AR glasses).

FIG . 33b shows in panel Bi) a pair of light beams 70 and 76 interacting within the nonlinear region 110. The first light beam (from the left) is labeled 70 and is the beam that is amplitude modulated according to FIG. 33a and Table A. The second light beam 76 is the light beam from the right. It is shown passing through a transmissive spatial phase modulator 122 that imparts a desired spatial phase profile on the light beam 80. There is a discontinuity in scale 186, which indicates that other elements may be added to the optical path. The spatial phase modulator 122 is controlled by a phase modulator controller 123, which is used to create a phase pattern corresponding to a diverging light beam so as to change the radius of curvature of the light emitted from the NLO region 110 and thereby create the appearance of different focal points to a viewer of the display device.

FIG . 33b is similar to B1) in panel Bii) , except that the second light beam 76 is reflected from the phase modulator 122, which is similarly controlled by the spatial phase modulator controller 123. For example, the phase modulator 122 can be a deformable MEMs mirror, a liquid crystal (LCOS) type device, or a thermally deformable mirror element. Note that example Biii) in Table B is not shown in the figure due to the similarity of the configuration to Bii).

Figure 33b is not shown in panel Biv) in Table B because it involves combining amplitude and phase modulation in a single mixing element and is the same as, for example , Av.

Figure 33b illustrates in panel Bv ) a diffraction intensity modulation method that changes the radius of curvature of the light beam 76. It appears superficially similar to Bi, except that the modulator is now a spatially controlled intensity modulator 120. In this figure, it is shown in transmission, but it should be understood that it can also be implemented in reflection. The intensity modulator 120 is controlled by an intensity modulator controller 121. In the simplest configuration, the radius of curvature of the light beam 76 (equivalent to the phase profile 80) can be changed simply by transmission through an aperture of controllable width (such as a pinhole). By reducing the size of the aperture, the beam divergence will increase due to diffraction. Similarly, a transmissive pixel controlled aperture (such as a liquid crystal device) can produce a transmissive pattern (such as a pinhole) that will also increase diffraction as needed. A more elegant approach would be to use a Fresnel zone plate pattern, as this would be more light efficient.

Figure 33b shows in panel Bvi) a system based on changing the properties of a laser-like beam 76 so as to control the phase 80 (equivalent to the radius of curvature) on the beam 76, thereby producing an output of a non-linear intersection region 110 that produces the desired image depth for a display device. The figure shows a pair of lenses 240A and 240B used to create an expanding telescope. The two lenses are controllable (by a controller unit 241) which is used to modify the resulting optical power of the pair of lenses. This can be achieved in many different ways, including liquid lenses or by moving elements to change the lens spacing (such as in a zoom lens for a camera). It should be understood that a different number of lenses can be used and the choice will be based on the desired technical performance. Changing the size of the beam 76 (which will propagate over a certain distance (and other optical elements represented by 186) will cause diffraction expansion, as is well known for Gaussian beam optics. Therefore, the radius of curvature (phase) of the beam will be controlled. Example Control System Inputs, Outputs and Functions

FIG34 shows a simplified schematic block diagram of a controller for a display system as described above . The controller 13 is shown as having three main inputs, namely, image data of the desired artificial scene information 11, eye monitoring data from inward facing cameras 54 and 56, and image data of the natural scene from outward facing cameras 50 and 52. The controller 13 uses the external cameras 50 and 52 to obtain information from the natural scene to determine where the artificial scene information should be located 11. The inward facing cameras 54 and 56 are used to determine where the line of sight is pointing, which in turn is used to derive the location on the display element to which the overlapping input beams should be directed to overlap in the NLO material. The figure also shows the main outputs of the controller 13, which are: laser power 19 for the radius of curvature determining beam; OLED controller 324 for producing an intensity structured beam; phase controller 123 (which sets the radius of curvature/phase on the laser beam); and the output of controlling the input beam manipulation optics (e.g., rotatable mirror 222) used to steer and focus the input beam. It should be understood that the system described is simplified and in a more complex configuration, other information would be included, such as accelerometer data, magnetic compass data, LiDAR data, etc.

As an example, a display system as described herein may have any combination or selection of the following inputs, controller functions, and outputs. Inputs:

Any of the following inputs and associated controller input interfaces may be provided. • WiFi or 4G/5G connection to the Internet • Bluetooth or equivalent (wired, wireless or optical) • Accelerometer and magnetometer • Front camera • Rear (eye tracking) camera • Integrity sensor photodiode • Information used to determine the synthetic scene, including information derived locally and received from the external network • Coordinates and geometry of the local space (locally broadcast or from memory) • LiDAR/3D mapping information • Calibration data in the installation protocol • Light scattering from the eye observed by the built-in camera System controller functions:

The controller may include any of the following functions. • Determine the synthetic scene based on external and internal sources • Determine region geometry and feature identity • Memory and signal processing • Current view angle • Eye spacing and other visual properties, monocular dominance, blinking, etc. • Calculate overlap angle and phase control • Intensity modulation calculations • Determine display integrity • External illumination • Calculate the need to block blocks of real scene light Outputs:

Any of the following outputs and associated controller output interfaces that may be viewed as control signals may be provided. • Position, rotation and acceleration data for external users • Camera output for external processing • Controller for first beam and controller for second beam • Amplitude modulation • Phase modulation to set radius of curvature • Color modulation • Steering optics to steer beam to desired spot • Real scene blocking wavelength combination to control photochromic

For color displays, it is necessary to provide red, green, and blue light. Assuming a nonlinear process of three-beam SFG, the frequencies of the two input beams must be the sum of the frequencies of the product beams of the desired color. There are a plethora of available laser and non-laser sources that provide an almost unlimited number of possible combinations of the two frequencies of the sum of red, green, and blue.

An example combination is: 1064 nm + 1550 nm = 630 nm (red) 1064 nm + 1064 nm = 532 nm (green) 1064 nm + 780 nm = 450 nm (blue).

This example also shows a practical method for reducing the number of source wavelengths required. In this example, only three different wavelengths (frequencies) are used, that is, less than two wavelengths (frequencies) for each color.

The second example combination is: 1064 nm + 1560 nm = 632.5 nm (red) 1064 nm + 1064 nm = 532 nm (green) 1064 nm + 780 nm = 450 nm (blue).

This example shows another method for reducing the number of source wavelengths required. In this example, only two laser sources are required since the 780 nm light can be generated by second harmonic generation of a 1560 nm laser.

It should be understood that the choice of the number of lasers used will depend on the availability, efficiency, price, lifetime and physical size of each technology. It should be noted that it is possible to use a single laser (for example at 3144 nm) which will provide 1572 nm via frequency doubling and its third harmonic will provide 1048 nm.

Thus, a third example combination using a single starting laser at 3144 nm is: 1048 nm + 1572 nm = 628.8 nm (red) 1048 nm + 1048 nm = 524 nm (green) 1048 nm + 786 nm = 449.1 nm (blue).

Another specific example would be: 950 nm + 1800 nm = 621 nm (red) 950 nm + 1210 nm = 532 nm (green) 950 nm + 880 nm = 456 nm (blue).

The 950 nm beam can be generated by an OLED array (or other IR LED array), while the other three beams at 1800 nm, 1210 nm, and 880 nm are generated by separate laser sources. In this example, there are four different wavelengths.

The generally desirable features of the present invention become apparent from the above examples. That is, since the colors of the display are essentially in the visible range, the input beam will be in the infrared range and therefore invisible for most, if not all, practical combinations of the two wavelengths. Thus, the scattering problem of known microprojector displays is essentially solved, since any scattered light from the source beam will be in the infrared range and therefore invisible.

The conversion efficiency in nonlinear processes depends on the product of the input beam intensities. For this purpose, although a CW source can be used, the preferred source is operated in pulse mode to produce high peak power, resulting in higher conversion efficiency. The laser source used to generate the input beam can be operated in any of the following modes: CW, long pulse, nanosecond pulse, picosecond pulse, or femtosecond pulse. The choice of which source to use and in which operating mode will depend on various factors, including the nonlinearity of the NLO material, the damage threshold of the NLO material, the required drive power, form factor, weight, cost, lifetime, etc.

Furthermore, it should be understood that any particular beam source used will produce a beam having a particular frequency response, which is typically defined in terms of a single frequency (or wavelength) - the peak frequency - and bandwidth, as a measure of the spread of emission above and below the peak frequency. To the extent relevant to this document, we define the bandwidth of a beam source as the full width at half maximum (FWHM). When a laser is used as the beam source, the bandwidth will be extremely narrow, while when a non-laser source is used as the beam source, the bandwidth can be significant. Considering the visible and near-infrared wavelengths discussed in the examples above, typical bandwidths are as follows. In single-mode edge-emitting LDs, the bandwidth can be as high as a few nanometers, decreasing to sub-pm for distributed feedback (DFB) or distributed Bragg reflector (DBR) edge-emitting LDs. For non-laser sources, the bandwidth will typically be much higher. For example, OLEDs can have bandwidths of tens of nanometers. In this literature, we usually refer to the "frequency" of a beam source - meaning the peak frequency, and not the bandwidth unless relevant. Other points and variations

Although SFG by nonlinear wave mixing is usually associated with the need to use an input beam with relatively high power (e.g., mW to Watt-level laser power), embodiments of the present invention generally do not require such high-power input beams due to the high sensitivity of the human eye. Sub-microwatt-level optical powers are quite bright to the human eye, so the maximum output power of the beam source (such as AR glasses or VR headsets) required for near-eye embodiments of the present invention will remain quite modest.

Many conventional optical design methods known from existing AR and VR display devices may also be used in display devices embodying the present invention. For example, it is well known to use a relay lens between a modulated light source (e.g., an array of micro OLED emitters) and an AR display panel to transfer the light to an emission point on the display panel where it is effectively transformed into an angular spectrum. In terms of Fourier optics, the concept is familiar, for example in a 4f imaging system, where a simple lens is placed one focal length away from the emission screen and arranged so that the position of the reflected plane wave is also one focal length away. In the simplest case, and assuming that the focal length of the lens matches the focal length of the eye (e.g., approximately 16 mm), a symmetric 4f imaging system is provided. The display panel is effectively located in the middle or Fourier plane and will therefore contain angular information. Modifications of the relay lenses with different focal lengths compared to the eye are possible. Furthermore, the relay lenses can also be modified in more complex ways, for example to compensate for undesired optical distortions, such as correcting chromatic aberrations and other aberrations or to provide other effects, such as providing different magnifications. When designing the relay lenses, it should be noted that the non-collinear geometry causes the spatial distribution of the image information in the product beam to be stretched compared to the corresponding image information in the first input beam and optionally the second input beam that needs to be considered. A simple approach is to use the Scheimpflug principle, where the lens is tilted so as to image onto a plane at a certain angle. More complex optical lenses and lens combinations can be deployed to correct distortions and aberrations, as is already common in optical design theory. There is also an effect due to the curvature of the display panel, and this can be corrected by a combination of determining the field angle of the QPM grating direction and by selecting the image information.

If the AR display device is convincing, the person wearing the AR glasses can place any image anywhere within their field of view. For example, if the wearer looks at their wrist to view the watch, they don't actually need a watch face for presentation. Instead, the watch face can be superimposed on the watch face. Similarly, mobile phones no longer need a display, but can have an arbitrary display area, on which the AR display superimposes the content on the display. The same method can also be used for televisions or personal computers, which no longer need screens, but only a wall or designated area on the device for the wearer of the AR display to perceive the display. Keyboards can also be projected.

It should be recognized that any of the methods disclosed herein in the context of AR will also be applicable to VR (simply by blocking all natural scene light).

It should be noted that when we refer to SFG of two input beams, we include frequency doubling, i.e., second harmonic generation (SHG), which is a special case of SFG in which the two input beams have equal wavelengths.

Although certain features and advantages of the proposed design are specific to AR vision systems, the proposed design is also advantageous for VR vision systems, in particular VR goggles can be made to have much less depth due to the ability to engineer the wavefront curvature of the VR image. The ability to place different synthetic objects at different perceived depths is as advantageous in the context of virtual images formed in VR goggles as it is for AR glasses, as it makes the VR image more convincing and also allows for avoiding azimuthal accommodation conflicts.

In addition, the disclosure of this document can also be used to provide imaging solutions for other optical viewing systems where the eye is in close proximity to where imaging occurs, including but not limited to binoculars, monoculars, telescopes, camera viewfinders, movie camera viewfinders, microscope eyepieces, medical imaging devices, endoscopes, magnifying elements, rifle sights, and military aiming systems.

10: AR glasses 11: Artificial scene information 12: Display device 13,123b,400: Controller 14: Beam source 16: Emitter array 18: Laser source 19: Laser controller unit 20,22,24: Lens 26: Frame 28: Nose bridge 30: Nose pads 32,34: Eye arm 36,38: Lens frame 40: Wireless transmitter and transceiver 42: Communication path 43: Wireless receiver and transceiver 44: External control device 46: Processor 48: Memory 49: External network communication 50,52: Outward-facing camera 54,56: Inward-facing camera 58: Point cloud imaging system 60: Photodiode 62: Compass 64: Accelerometer 70: First input beam 70A,70B: Input beam path 71,77: Figure markers 72: First input beam wave vector 74: First input beam wave front 75: Amplitude and phase modulation function 76: Second input beam 76A,76B,224A,224B: Beam path 78: Second input beam wave vector 80: Phase profile 81: Gaussian profile 82: Output beam 84: Product beam wave front 86: Product beam wave vector 88,88A,88B,88C: Display panel 90: Inner surface 92: outer surface 94: left end face 96: right end face 100: NLO material 103: NLO material superstructure light spot 103A, 103B, 104, 106B, 106G, 106R: light spot 106: light spot cluster 108: QPM area 110: intersection area 110A, 110B: intersection position 120: modulator 121: amplitude modulator controller 122: transmission type spatial phase modulator 123: phase modulator controller 123a: combined amplitude and phase modulator 124: input beam routing component 124b: controller of combined amplitude and phase modulator 125: rotatable mirror element 128: air 130,A1: front filter layer 132: rear filter layer 134: light blocking layer 136: front mirror layer 138: rear mirror layer 139: power supply 140: dimming layer 141: electrical connection line 142,N1: front vision correction layer 144: rear vision correction layer 146,AR_1: front ARC layer 148,AR_2: rear ARC layer 150: IR absorption layer 150A,150B,150C,150D: layer 152: front spacer layer 154: rear spacer layer 156: dimming layer, electrical connection line 160: Eye 162: Eyeball 164: Cornea 166: Lens 168: Retina 170: Pupil 172: Iris 174: Optic nerve 180: Object 181: Bent wavefront 182: Light scattered from object 184: Light 186: Discontinuity 188,188A,188B: Wavefront 190: Illustration 190A,190B: Point 192,2f: Distance 200: QPM NLO material 202: 『up』 domain 204: 『down』 domain 206B,206G,206R: QPM period 208: Superstructure grating 220: Light beam 222: Rotatable lens 222A: First lens 222B: Second lens 222C: Third lens 230: Optical components 240,300,325: Lenses 240A: First lens 240B: Second lens 241: Controller unit 242: Relay lens 260: Lens surface 262: Lens layer 264: Side of the glasses arm or glasses leg 266: Side of the nose bridge 280: Control system 282: Blanking cycle 290: Absorption material area 292: Non-absorption area 292A,292B: Entrance area 317: Input infrared laser wave 318,319,320: radius of curvature 321: incoherent emitter array 322,323,324: pixels 334: controller for incoherent emitter array 402: controller input 404: controller output a_1,a_2: vector A,B,C: direction Ai),Aii),Aiii),Av),Bi),Bii),Bv),Bvi): panel b_1,b_2: K_G basis vectors f,f’: focal length k_1,k_2,k_g: wave vector Kg: quasi-phase-matched grating vector K_G,P: QPM grating period n: refractive index t: time z: depth Z1,Z2: position λ: wavelength λ_1: first wavelength λ_2: second wavelength λ_3: third wavelength θ: rebound angle ϕ_1,ϕ_2: angle

The present invention will now be further described, by way of example only, with reference to the accompanying drawings.

Figure 1 is an explanation of how wavefront information is needed to image distant objects.

FIG. 2 is a diagram showing how the image formed through a lens element and entering the eye depends on both the wavefront direction and the wavefront curvature.

Figure 3 explains why point light emitters cannot be used in near-eye displays.

FIG. 4 illustrates a nonlinear optical approach for generating near-eye artificial images that can be superimposed on natural scene light.

Figure 5 illustrates the concept of quasi-phase matching.

FIG. 6 shows two-dimensional phase matching, and different grating vectors can be constructed.

Figure 7 shows an implementation of NLO mixing for generating synthetic scene light with a bounce path within the lens.

Fig. 8 shows non-collinear phase matching.

FIG. 9 shows a more detailed diagram showing the nonlinear wavefront overlap in layered NLO QPM materials to create image information.

FIG. 10 shows a route that allows beams having different intersection angle pairs relative to the QPM structure to overlap in a common location.

Figure 11 shows a scheme for directing a light beam into the device.

Figure 12a shows a scheme for deflecting a light beam without angular variation.

Figure 12b shows components for directing light to a desired point within a layer of NLO material while changing the angle at which the light approaches the spot.

FIG. 13 shows a lens configuration with an input beam and an emitter capable of creating an image with a desired apparent depth of focus.

FIG. 14 shows the imprinting of the desired wavefront via nonlinear mixing of amplitude and phase structured beams.

FIG. 15 shows a system for creating a far field pattern from an image to allow conversion via NLO material.

FIG. 16 shows a multi-layer optical composite used to create a display system.

Figure 17 shows the changing area Kg position on the eyeglass type display system.

Figure 18 shows different lens designs with respect to cross-sections having appropriate advantages.

Figure 19 shows an eyeglass AR system that includes multiple sensors and communicates with an external computing device.

FIG. 20 shows an eyeglass system that includes a series of diode elements around the exterior to detect damage to the system.

Figure 21 shows an electronically controlled dimming system included in the lens.

FIG. 22 shows a display device including sub-regions of NLO material and having separate red, green and blue color generating regions.

FIG. 23 shows the QPM period as a function of the crossing angle in a simple second harmonic process, indicating the advantage of using a larger crossing angle.

Figure 24 shows a method for creating a lens in which light beams enter from one side but remain crossed in the opposite direction.

FIG. 25 shows a timing display and depth-realistic display for providing RGB and different ROC images to produce convincing full color.

Figure 26 shows an inward-facing camera used to track pupil position.

FIG. 27 illustrates how inward-facing camera based cloaking can be used to reduce undesirable visual effects.

FIG. 28 shows a lens having an absorptive layer dispersed around the edges to prevent unwanted reflection of light.

FIG. 29 shows a lens having sub-regions of NLO material, each sub-region containing elements capable of producing red, green, and blue colors.

Figure 30 shows that the beam crossing angle on the lens requires regional control of the QPM grating direction.

FIG. 31 shows a virtual reality (VR) configuration according to the present invention, in which a near-eye image is created without the complexity of using thin lenses.

FIG. 32 is a schematic diagram of image generation in a NLO material layer using an array of incoherent emitters to generate one of the input beams.

Figure 33a shows various options for modulating two light beams interacting in an NLO material.

Fig. 33b shows various options for phase control on the input beam to impart radius of curvature information.

Figure 34 shows a schematic diagram of the control system showing the inputs and outputs.

Domestic storage information (please note in the order of storage institution, date, and number) None Foreign storage information (please note in the order of storage country, institution, date, and number) None

70: First input beam

71,77: Attached picture mark

76: Second input beam

84: Product beam wavefront

100:NLO material

160: Eyes

168: Retina

180:Object

186: Discontinuity

188: Wavefront

Claims (40)

A display device (12) for displaying a synthetic scene image in response to input of image information, the display device comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel (88) containing a nonlinear optical NLO material (100) that is phase-matched with the first input beam and the second input beam and a product beam (82) having a sum frequency equal to the sum of the first frequency and the second frequency; An input beam routing component (222, 240) configured to introduce the first input beam and the second input beam into the display panel so that they pass through the display panel in a first path and a second path, the first path and the second path intersecting each other in a non-collinear geometry in the NLO material to define an intersection volume (110), in which the product beam is generated by sum frequency generation; and a controller (400) operable in response to the input of the image information to form the synthetic scene images by amplitude modulating at least one of the first input beam and the second input beam and by phase modulating at least one of the first input beam and the second input beam to set each of the amplitude, wave vector and wavefront curvature radius of the product beam. A display device as described in claim 1, wherein the first input beam encodes image intensity information, which image intensity information defines the intensity of the product beam generated at each intersection volume of a given synthetic scene image, and the second input beam encodes image depth information, which image depth information defines the wavefront curvature radius of the product beam generated at all intersection volumes of a given synthetic scene image. A display device as described in claim 1 or 2, wherein the controller is operable to group the synthetic scene images into image frames, different synthetic scene images in a given image frame having different wavefront curvature radii to represent image elements located at different respective perceptual distances. A display device as described in claim 1 or 2, wherein the display device is used to display a synthetic scene image in color and includes a plurality of beam sources, the plurality of beam sources including a first input beam source and a second input beam source, wherein the first input beam and the second input beam form one of three input beam pairs generated by the plurality of beam sources, and the first input beam pair to the third input beam pair are combined to generate first product beams to third product beams having first to third sum frequencies providing first to third primary colors. A display device as described in claim 4, wherein the plurality of beam sources used to form the three input beam pairs include an emitter array and first to third lasers, wherein the emitter array provides one of the input beams for all three input beam pairs, the other input beam in each pair is provided by one of the first to third lasers, and the first to third primary colors are provided by time division, thereby generating the first to third product beams in sequence. A display device as described in claim 5, wherein the emitter array is driven to amplitude modulate the input light beam it generates. A display device as described in any one of claims 4 to 6, wherein the NLO material includes first to third spatial modulation regions to provide quasi-phase matching for the first input light beam pair to the third input light beam pair, respectively. A display device as described in claim 7, wherein the first to third spatial modulation regions are formed in the NLO material at a first depth portion to a third depth portion (206) in the display panel. A display device as described in claim 7, wherein the first to third spatial modulation regions are formed in the NLO material as an array of light spot clusters (106), each light spot cluster including adjacent light spots of each of the first to third spatial modulation regions. A display device as described in any of claims 4 or 9, wherein the controller is operable to group the synthetic scene images into image frames, the different synthetic scene images of a given image frame being presented in each of the first to third primary colors. A display device as described in any of claims 4 or 9, wherein the controller is operable to group the synthetic scene images into image frames, the different synthetic scene images of a given image frame presenting each of the first to third primary colors and having different wavefront curvature radii, thereby representing color image elements located at different respective perception distances. A display device as described in any of the preceding claims, wherein the input beam routing components are adjustable to provide a change in at least one of the first path and the second path, thereby changing the angle at which the first input beam and the second input beam intersect in the NLO material, thereby changing the wave vector direction of the product beam. A display device as described in any of the preceding claims, wherein the input beam routing components are adjustable to provide a change in at least one of the first path and the second path, thereby changing the location where the first path and the second path intersect in the NLO material. The display device as described in any one of claims 1 to 13 further includes an amplitude modulator (120) configured to perform amplitude modulation on the first input light beam and a phase modulator (122) configured to perform phase modulation on the second input light beam. The display device as described in any one of claims 1 to 13 further includes a combined amplitude and phase modulator (123a) for performing amplitude and phase modulation on one of the first input light beam and the second input light beam. A display device as described in any of claims 1 to 15, wherein the NLO material is spatially modulated with respect to its second-order nonlinearity to provide phase matching via quasi-phase matching. A display device as described in any of claims 1 to 15, wherein the NLO material is uniform with respect to its second-order nonlinearity to provide phase matching via birefringent phase matching. A display device as claimed in any preceding claim, wherein the sum frequency is in the visible light range and the first frequency and the second frequency are in the infrared range. A display device as described in any preceding claim, wherein a ratio of the first frequency to the second frequency is between 0.5 and 2.0. A display device as described in any of the preceding claims, wherein the display panel comprising the NLO material is transparent in the visible light range. A display device as described in any preceding claim, wherein the NLO material is contained in the display panel in a NLO material layer. A display device as described in claim 21, wherein the display panel includes a front filter layer (130) arranged in front of the NLO material layer, the front filter layer is opaque to the first frequency and the second frequency and transparent to the sum frequency to prevent light from the first input light beam and the second input light beam from being emitted from the display panel in an outward direction. A display device as described in claim 21, wherein the display panel includes a rear filter layer (132) arranged behind the NLO material layer, the rear filter layer being opaque to the first frequency and the second frequency to prevent light from the first input light beam and the second input light beam from being emitted from the display panel in an inward direction. A display device as described in claim 23, wherein the rear filter layer is transparent to visible frequencies. A display device as described in any of claims 21 to 24, wherein the NLO material is continuously distributed on the NLO material layer so that phase matching and therefore product beam generation can occur at any location in the NLO material layer. A display apparatus as claimed in any one of claims 21 to 24, wherein the NLO material is distributed as an array of spots (106) on the NLO material layer so that phase matching and therefore product beam generation is confined to those portions of those spots. A display device as claimed in any preceding claim, wherein the display panel includes a light shielding layer (134) comprising an array of pixels individually addressable by electrical control lines so as to switch the pixels between a first state opaque to visible frequencies and a second state transmissive to visible frequencies so that natural scene light can be mixed out from selected areas of the light shielding layer. A display device as claimed in any preceding claim, wherein the first path and the second path pass through the display panel by successive reflections. A display device as described in claim 28, wherein the continuous reflections come from the front surface and the rear surface of the display panel by total internal reflection. A display device as described in claim 28, wherein the continuous reflection comes from a front-view mirror layer and a rear-view mirror layer (136, 138) configured in the display panel, and the front-view mirror layer and the rear-view mirror layer are respectively arranged in front of and behind the NLO material. A display device as described in claim 30, wherein the front mirror layer and the rear mirror layer are reflective at the frequencies of the input light beams and are transmissive within visible frequencies. A display device as claimed in any preceding claim, wherein the NLO material is transparent in a visible range. A display device as described in any of the preceding claims, wherein the display panel has a peripheral region (290) containing a material that is absorptive to light at the first frequency and the second frequency, so that when the first input light beam and the second input light beam reach the peripheral region, the first input light beam and the second input light beam are absorbed after the first path and the second path have crossed each other in the NLO material. The display device as claimed in any of the preceding claims further comprises one or more light sensors (60) arranged outside the display panel to detect abnormal leakage of input light beam from the display panel, the abnormal leakage indicating structural damage of the display panel. A display device as described in any of claims 1 to 34, wherein the first frequency and the second frequency are different. A display device as described in any one of claims 1 to 34, wherein the first frequency and the second frequency are the same so that the sum frequency generation is a second harmonic generation. A display device as described in claim 36, wherein the first input beam source and the second input beam source are the same beam source, and the first input beam and the second input beam both originate from the beam source. A wearable headset (10) comprising a display device as claimed in any preceding claim, wherein the display panel is arranged in front of a wearer's eye at a vertex distance of less than 30 mm for near-eye imaging. A display device (12) for displaying a synthetic scene image in response to input of image information, the display device comprising: A plurality of beam sources (16, 18) having respective frequencies and bandwidths to provide a first input beam pair to a third input beam pair of two input beams (70, 76), each input beam pair having a pair of frequencies, the pair of frequencies being the sum of a first primary color frequency to a third primary color frequency; A display panel (80) containing a nonlinear optical NLO material (100), the nonlinear optical NLO material being phase-matched with the first input beam pair to the third input beam pair and the first product beam to the third product beam (82) at a first primary color frequency to a third primary color frequency equal to the sum of the frequencies of the first input beam pair to the third input beam pair; An input beam routing component (222, 240) configured to introduce the two input beams in each input beam pair into the display panel so that the input beams in each pair pass through the display panel in a first path and a second path, the first path and the second path intersecting each other in a non-collinear geometry in the NLO material to define an intersection volume (110), in which a product beam of the input beam pair is generated by sum frequency generation; an amplitude modulator (120) operable to amplitude modulate at least one beam in each input beam pair; a phase modulator (122) operable to phase modulate at least one beam in each input beam pair; and A controller (400) is operable to form color image frames, each image frame comprising first to third synthetic scene images generated by the first input beam pair to the third input beam pair, respectively, wherein the controller is operable to form each of the first to third synthetic scene images by controlling the amplitude modulator and the phase modulator in response to the input of the image information to set the product beam amplitude, the product beam wave vector and the product beam wavefront curvature radius. A display device as described in claim 39, wherein the plurality of beam sources include an emitter array (16) and first to third lasers (18), wherein the emitter array provides one of the input beams for all three input beam pairs, the other input beam in each pair is provided by one of the first to third lasers, and the first to third primary colors are provided by time division, thereby generating the first to third product beams in sequence. [1] A more complex method may utilize a Fresnel zone plate, which has the advantage of being able to use an absorptive spatial modulator and utilizing the coherence of the laser-like input beams