HK1238457B - Projector optimization method and system - Google Patents

Description

Projector optimization method and system

Technical Field

The technology described herein generally relates to methods and systems for optimizing one or more projectors.

Background Art

Image projectors are used to project images onto a projection surface, such as a screen. In some applications, video projectors are used to enhance, supplement, or otherwise augment objects on a surface to create a dynamic and enjoyable user experience, such as an amusement park attraction. For example, characters or objects can be projected onto a surface that virtually "interact" with real objects on the surface.

Traditional video projectors have many limitations. In particular, they have a limited color gamut and brightness. Due to these limitations, presentations using only a traditional video projector can appear dim and unremarkable. Additionally, in the presence of ambient lighting, the generated image can appear washed out and unrealistic. In contrast, laser projectors (such as laser scanning projectors) have increased brightness and color gamut compared to traditional video projectors. In particular, laser projectors can project pure, saturated, i.e., monochromatic, red, green, and blue tones, allowing for a significantly wider color gamut than traditional video projectors.

In contrast to video projectors, laser projectors use galvanometer mirrors to direct light. These mirrors move at such high speeds that, due to the limitations of human visual perception, the projected laser spot is perceived as a static graphic by a human observer. However, the speed at which such mechanical mirrors can move has physical limitations that can cause problems such as blurring, flickering, and sluggishness. Therefore, scanning lasers are generally limited to the number of vertices that can be properly displayed, because visual flicker can occur quickly if the path is traced too slowly. On the other hand, the speed must be reduced to ensure accurate spatial rendering of the content. In addition, a typical laser projector will choose a random path for rendering the content. These problems result in flickering and other artifacts that affect the appearance of the projected content through the laser projector.

Summary of the Invention

One example of the present disclosure relates to a method and system for optimizing a projector for content projection. In one embodiment, the method includes receiving, by a processing component, a plurality of test images corresponding to test patterns projected by a projector onto a projection surface, wherein each of the test patterns includes at least two points, comparing, by the processing component, the plurality of test images to evaluate one or more projector characteristics related to a distance between the two points, generating, by the processing component, a projector model representing the one or more projector characteristics, and determining a projection path for the projector to use for content using the model.

Another example of the present disclosure includes a method for generating a projector model for optimizing a projection path for projecting content by a projector. The method includes capturing, with a camera, multiple pattern images of a test pattern projected by the projector, analyzing the pattern images to determine distance characteristics and angle characteristics for reaching the test pattern in each of the pattern images, and creating a model database representing the projector based on the distance characteristics and angle characteristics of each test pattern.

Yet another example of the present disclosure includes a system for optimizing a projection path for content. The system includes a laser projector for projecting content and a computer electronically connected to the laser projector. The computer includes a memory containing a projector model database corresponding to the display behavior of the projector, and the computer is configured to analyze the projector model to determine a minimum length for the projection path for the content and to determine an optimal angle between each point of the content along the projection path.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1A is a perspective view of an optimization system.

1B is a perspective view of the optimization system of FIG. 1A including a calibration arrangement.

Figure 2 is a simplified block diagram of the optimization system.

FIG3 is a simplified block diagram of the projector of the optimization system.

FIG4 is a flow chart illustrating an optimization method.

FIG5 is a flow chart illustrating a calibration method for optimizing the camera and projector of the system.

6A is a perspective view of an optimization system in which a projector projects a test pattern on a projection surface.

FIG. 6B is an illustration of a pattern image captured corresponding to the test pattern of FIG. 6A .

FIG. 6C is an illustration of the pattern image of FIG. 6B cropped about a center point.

FIG. 7 is an example of model parameters derived for a projector using the method of FIG. 4 .

FIG. 8 is a diagram illustrating a selection operation of the method of FIG. 4 .

FIG. 9 is a schematic representation of the optimization operation of FIG. 4 .

FIG. 10A illustrates a frame of content projected by a projector before optimization.

FIG. 10B illustrates a frame of the content in FIG. 10A after optimization.

DETAILED DESCRIPTION

The present disclosure generally relates to optimizing the projection path for a scanning laser projector. With a scanning laser projector, the appearance of flicker is affected by a variety of factors, such as the projector's total scan path length, the order in which the path is traversed, the number of vertices to be displayed, and temporal path consistency. The disclosed systems and methods minimize these factors while also maximizing the number of displayable points.

In one embodiment, the method includes capturing information about the scanning behavior of a laser projector under various path tracking conditions. Using this information, a model is generated that describes the laser's behavior with respect to relevant properties. The model is used to compute a near-optimal scanning path for one or more images that is a spatially accurate projection of an input point in a minimum amount of time and that minimizes perceived flicker artifacts by taking into account the spatiotemporal path distribution.

As used herein, the term "laser projector" or "projector 110" is intended to refer to substantially any type of light projection assembly, such as, but not limited to, a galvanometer laser scanning projector. In addition, the methods and systems presented herein can be used for other types of projectors that are optimized or that do not guarantee output based on the same input (e.g., input with the same set of points in the same position but displayed in a different order), such as scanning projectors or other projectors with moving elements.

FIG1A is a perspective view of the optimization system 10. FIG1B is a perspective view of the calibration system for FIG1A. FIG2 is a simplified block diagram of the optimization system 10. Referring to FIG1A-2, the optimization system 10 includes one or more laser projectors 110 for projecting content 153, a projection surface 162, one or more cameras 120, and one or more computers 130. The one or more laser projectors 110, cameras 120, and computers 130 can all communicate with each other and include the ability to share and transfer information between each device. Original content 149 is input to the computer 130, which optimizes the projector path for the projector 110, which then projects the optimized content 153. Each of the components will be discussed in turn below.

Projection surface 162 is any type of opaque surface or object. For example, projection surface 162 can be flat, non-planar, or variable, and can include one or more textures or surface variations and/or colors. In some examples, projection surface 162 can include multiple surfaces at different depths or positions relative to each other. The type and configuration of projection surface 162 can vary as desired.

The camera 120 is any device capable of capturing still or video images. The camera 120 captures full-color images and/or monochrome images and can use any type of filter, such as one or more color filters. In one embodiment, the camera 120 is configured to capture substantially the entire dynamic range of the laser projector 110 without significant clipping. The camera 120 is registered with its environment or otherwise placed in a known position so that the specific orientation and position of the camera 120 relative to the projection surface 162 is known.

It should be noted that while one camera 120 is shown in Figures 1A and 1B, in other embodiments, additional cameras may be used. For example, in some embodiments, two cameras may be used for three-dimensional (3D) calibration of the projector, while a single camera may be used for optimization of the projector. Alternatively or additionally, one or more cameras capable of depth perception may be used, such as a 3D stereo camera, a KINECT-type depth camera, a 3D camera, a machine vision camera, or any other type of active or passive depth detection camera (such as a time-of-flight-based 3D camera or an infrared 3D camera).

1A and 2 , optimization system 10 also includes a computer 130. Computer 130 analyzes data from both projector 110 and camera 120 and may also optionally control one or more functions of either device. In the example of FIG1A , only one computer 130 is shown; however, in other embodiments, more than one computer or server may be used. Computer 130 may include one or more processing elements 301, a power supply 302, a display 303, one or more memory components 304, and an input/output (I/O) interface 305. Each of the components of computer 130 may communicate via one or more system buses, wirelessly, or the like.

Processing element 301 is any type of electronic device capable of processing, receiving, and/or transmitting instructions. For example, processing element 301 may be a microprocessor or a microcontroller. Furthermore, it should be noted that selected components of computer 130 may be controlled by a first processor, while other components may be controlled by a second processor, wherein the first and second processors may or may not communicate with each other.

The memory 304 stores data used by the computer 130 to store instructions for processing element 301 and to store rendering, optimization, calibration, and/or projector model data used to optimize the system 10. For example, the memory 304 may store data or content corresponding to various applications, such as, but not limited to, audio files, video files, etc. The memory 304 may be, for example, magneto-optical storage, read-only memory, random access memory, erasable programmable memory, flash memory, or a combination of one or more types of memory components.

Power supply 302 provides power to the components of computer 130 and may be a battery, a power cord, or other element configured to deliver power to the components of laser projector 110 .

The display 303 provides visual feedback to the user and, optionally, can be used as an input element to enable the user to control, manipulate, and calibrate various components of the optimization system 10. The display 303 can be any suitable display, such as a liquid crystal display, a plasma display, an organic light emitting diode display, and/or a cathode ray tube display. In embodiments where the display 303 is used as an input, the display can include one or more touch or input sensors, such as a capacitive touch sensor, a resistive grid, and the like.

The I/O interface 305 provides communication to and from the laser projector 110, the camera 120, and the computer 130, as well as other devices (e.g., other computers, auxiliary scene lighting, speakers, etc.). The I/O interface 305 may include one or more input buttons, communication interfaces such as WiFi, Ethernet, etc., and other communication components such as a universal serial bus (USB) cable.

Optionally, the computer 130 may include a sensor 306. The sensor 306 includes essentially any device capable of sensing a change in a characteristic or parameter and generating an electrical signal. The sensor 306 may be used in conjunction with or in place of the camera 120, or may be used to sense other parameters such as the ambient lighting surrounding the projection surface 162. The sensor 306 and display 303 of the computer 130 may be varied as desired to meet the needs of a particular application.

The laser projector 110 of the optimization system 10 will now be discussed in more detail. FIG3 is a simplified block diagram of the laser projector 110. Referring to FIG1 and FIG3, the laser projector 110 is used to project content 153, such as light patterns, images, etc., onto a projection surface 162 of a scene 160. The laser projector 110 can be a scanning laser projector or other lens-based or non-lens projection assembly. It should be noted that although the embodiments discussed herein are discussed with respect to laser projectors, the optimization methods and systems can be used with substantially any type of projector, particularly projectors that include a moving element for outputting content.

In some embodiments, laser projector 110 projects red, green, and/or blue coherent light. In other embodiments, laser projector 110 projects light of substantially any other color or frequency, including visible light or non-visible light (e.g., ultraviolet, infrared, and others), as desired. Laser projector 110 may be a galvanometer-scanned laser projector, etc., and include a mirror assembly 201, a laser source 202, one or more processing elements 203, one or more memory components 204, an I/O interface 205, and a power supply 206. In some embodiments, computer 130 may provide some or all of the processing and storage functions for projector 110, and in these embodiments, one or more features of projector 110 may be omitted.

Mirror assembly 201 directs light emitted by laser source 202 onto projection surface 162. In some embodiments, mirror assembly 201 can include two or more mirrors connected to a galvanometer servo system or other motion-inducing element. In one embodiment, the mirrors of mirror assembly 201 can be oriented orthogonally relative to each other so that one mirror rotates about a first axis and the other mirror rotates about a second axis orthogonal to the first axis. The motion-inducing element moves the mirrors based on input to projector 110 to change the output and position of light emitted from projector 110.

One or more projector processing elements 203 receive input image data from memory component 204 or I/O interface 205. Projector processing element 203 can be substantially similar to processing element 301 and can be any electronic device capable of processing, receiving, and/or transmitting instructions. For example, projector processing element 203 can be a microprocessor or a microcontroller.

Projector memory 204 stores data used by laser projector 110 and/or computer 130 and can be volatile or non-volatile memory. Memory 204 can be substantially similar to memory 304, but in many embodiments may require less memory than other components.

The power supply 206 provides power to the components of the laser projector. The power supply 206 can be a battery, a power cord, or other element configured to deliver power to the components of the laser projector.

The laser source 202 may be one or more solid-state laser sources, such as a laser diode, or may be a gas laser source and/or other types of coherent light. In other embodiments, the projector may include an incoherent light source.

I/O interface 205 provides communication to and from laser projector 110 and computer 130, as well as other devices. I/O interface 205 may include one or more input buttons, communication interfaces such as WiFi, Ethernet, and other communication components such as a Universal Serial Bus (USB) cable.

The optimization system 10 is used to calibrate and optimize the laser projector 110 so that the content projected by the projector 110 is projected in the most efficient manner to reduce flicker and other artifacts. FIG4 illustrates a method 400 for optimizing the laser projector 110. The method 400 begins with operation 402 and performs a geometric calibration. For example, a camera-based calibration process is used to evaluate information about the scanning behavior of the projector 110 and ensure that the displayed content is projected in the desired spatial location. In this example, the pixels of the camera 120 are mapped to the coordinates of the projected content on the projection surface 162.

FIG5 illustrates a diagrammatic process for operation 402. Referring to FIG5, operation 402 includes process 502, and the laser projector 110 projects a calibration pattern 150 onto the projection surface 161. The calibration pattern 150 can be any type of structured light. Examples of calibration patterns can be found in U.S. application Ser. No. 14/534,855, entitled "Method and System for Projector Calibration," which is incorporated herein by reference in its entirety (the '855 application). The calibration pattern 150 can include multiple pattern elements or dots projected individually or in groups in a temporal binary-coded sequence. For example, in one embodiment, each element 151 of the projected content is identified by a series of on/off events, while in other embodiments, the pattern elements 151 can be projected substantially simultaneously.

In some embodiments, a portion of a calibration pattern 150 containing a subset of pattern elements 151 can be projected, followed by projection of the remainder of the overall calibration pattern 150 having other sets of pattern elements 151. In still other embodiments, a calibration pattern 150 having a first set of pattern elements 151 can be projected, followed by another calibration pattern 150 having a second set of pattern elements 151, which can be the same as or different from the first calibration pattern 150. The second calibration pattern 150 having the second set of pattern elements 151 can be selected or chosen to help further refine features of the projection surface 161. Additionally, the second calibration pattern 150 can be selected based on image feedback provided by the camera 120. That is, the calibration of a projector or projection system can be based on the projection of more than one calibration pattern 150 in an adaptive manner.

The selection of the projection sequence of the pattern 150 and the arrangement of the elements 151 within the pattern 150 may be determined based on the desired projection surface 162 or the content to be projected by the projector.

5 , in process 504, camera 120 captures a calibration image of projected calibration pattern 150. Camera 102 may be adjusted prior to use so that the camera shutter can capture pattern elements 151 without oversaturating the image sensor's photosensors. This adjustment may be done automatically or manually by a user (e.g., by adjusting exposure settings).

One or more calibration images are captured using the camera 120. The calibration images are typically photographs, such as digital images, capturing the pattern 150 as projected onto the projection surface 162. The calibration images allow the projection elements 151 to be identified separately.

In some embodiments, multiple calibration patterns may be projected in a predetermined sequence, and one or more calibration images may be captured for each calibration pattern. By using multiple patterns, the processing element may be able to more easily determine the distortion of the laser projector.

Using the calibration image, operation 402 includes process 506 where the calibration image is analyzed to detect pattern elements 151. Processing element 301 detects the positions of pattern elements 151 using an image analysis algorithm such as a blob detection algorithm or other algorithm that can detect the positions of pattern elements 151.

An image analysis algorithm is selected for detecting pattern elements 151 based on the characteristics of the pattern elements 151. For example, if the pattern elements 151 are dots or spots, then the processing element 301 may use a spot detection algorithm. As an example, the spot detection algorithm may include steps of subtracting a background from an image and thresholding the image to mask out suspicious spots or feature locations. The algorithm is executed, and the processing element then analyzes each masked area to calculate the center of gravity of the spot. In an exemplary embodiment of the present disclosure discussed in more detail below, when the processing element 301 is configured to detect pattern elements 151 using a spot detection algorithm, the processing element 301 analyzes and compares the relative position of the center of gravity for each detected spot to the center of gravity for other spots to determine the spot coordinates within the 2D image plane.

In other embodiments, if the pattern elements 151 are selected as lines or line segments, a line center detection algorithm may be used. Other algorithms for detecting the pattern elements 151 may also or alternatively include feature detection, edge detection, sequential binary coded spots, binary coded horizontal and/or vertical lines, Gray code with long exposure, color coded patterns, intensity coded patterns, more complex pattern element features, etc. Depending on the shape of the pattern elements 151, some algorithms may be more accurate than others, while other algorithms may require less processing power.

Returning to FIG. 5 , once the test pattern has been captured by camera system 120 and processed by computer 130 , operation 502 includes process 506 , in which pattern elements 151 are detected. For example, an image analysis algorithm reviews the calibration image to detect areas that differ in one or more properties (such as, but not limited to, brightness, color, or hue) from the surrounding areas. In this example, the captured pattern elements 151 will be brighter and may have a different color than the surrounding areas of pattern 150 , thus allowing the processing unit to detect their location.

In one example, the processing element analyzes the captured image to determine the location of the center of each pattern element 151 in the two-dimensional (2D) image plane of the corresponding camera 120. That is, the 2D coordinates of each pattern element 151 are determined by using detectable features of each element 151. In one example, a blob detection algorithm can be used to provide sub-pixel accuracy for the location of the center of the pattern element 151. However, as discussed above, other algorithms can also be used.

Using the detected elements, operation 402 proceeds to process 508, where the 2D positions of test pattern elements 151 within scene 160 are determined and a dense lookup table is created between the image planes of camera 120 and projector 110. In other words, the correspondence between the positions of projected pixels and the inputs to the projector is determined. It should be noted that operation 402 is described with respect to a 2D mapping between camera 120 and projector 110. For example, once the mapping between sub-pixel positions on the image plane of camera 120 and the coordinates of projector 110 is generated using blob detection (or other algorithms), thin plate spline-based interpolation is used to interpolate missing points to generate a dense lookup table, which enables warping of the camera image onto the virtual image plane of the projector. Examples of lookup table generation can be found in the '855 application. It should be noted that these steps can be performed in various ways, and other interpolation methods can be used to generate the dense map.

In embodiments where 3D mapping may be desired (e.g., for non-planar projector surfaces), two or more cameras may be used as described in the '855 application. However, for the present method, 2D mapping may be used for simplicity and to reduce errors.

Returning to FIG. 4 , after the geometric registration of the camera 120 and the projector 110 is completed, the optimization and model generation of the projector can begin.

Method 400 proceeds to operation 404, and a test pattern is projected onto projection surface 160. FIG6A illustrates an example of a test pattern 600 projected onto projection surface 608. In many embodiments, test pattern 600 includes three points 602, 604, and 606 arranged at different angles and distances relative to one another, using different accelerations and decelerations of the mirror. In these embodiments, pattern 600 includes a center point 602 located between the other two points. While two points may be used to evaluate the behavior of projector 110 in some embodiments, adding a third point provides additional data regarding angular direction changes between the different points.

The parameters of the test pattern 600 vary between test patterns, and multiple test patterns are used to enhance the sensitivity of the optimization. In these examples, a selection sequence is used in which the first point projected is the upper right point 604, then the center point 602 is projected, followed by the upper left point 606, and in each subsequent test pattern 600, the angles between the points and, optionally, the distances between the points are varied. While three points are disclosed, in other embodiments, additional points can be used in addition to the "center point" to form each point between the first and last projected points in the pattern.

The test pattern 600 (including the number of dots, angle variations, and distance variations) is selected to optimally detect the acceleration and deceleration of the laser projector 110 as it starts and stops moving to project each dot 602, 604, 606 in the pattern 600. The test pattern 600 can be configured as a "V" shape, but can vary depending on the type of projector 110. By varying the dots, stopping points, typically located at the same or similar locations as the projected dots, are typically used by laser projectors to assess when to stop or slow down the mirror as it approaches the location of the projected dots. These stopping points, or speed control points, are blank or invisible points (e.g., projected in the same color as the background or not projected at all) that slow the laser and can be used by the projector to decelerate the mirror. These speed control points vary based on the location of the previously projected dots.

Mathematically, a set of test point patterns with different angles and normalized distances are multiply projected with increasing number of stopping points, which is denoted as .where d corresponds to half the height of the virtual image plane and is equal to the diagonal.

4 , as different test patterns 600 are projected by projector 110, method 400 proceeds to operation 406, and camera 120 or a measurement sensor captures a test image 610 of the projected pattern. FIG6B illustrates an example of a test image 610 captured by camera 120 in FIG6A . It should be noted that a single camera 120 is required to capture the test image, but multiple cameras may also be used. In some embodiments, camera 120 is focused or angled on only the center point, as other points may not be needed in model generation for reasons discussed below.

Test images 610 are typically captured using a short shutter time to help avoid saturation. Furthermore, to ensure the laser path is fully captured, multiple images of the same pattern can be acquired with random delays of a few milliseconds and averaged, which also reduces the amount of noise in the image. In this specific example, eight test images 610 are captured for each test pattern 600. Typically, the shutter speed of camera 120 is selected to be slow enough to capture the entire pattern (e.g., all three points 602, 604, 606) at once. However, in other embodiments, the camera shutter speed can be increased to capture only one or two of the points.

Additionally, in embodiments where the pattern is repeatedly drawn, some points may be brighter than others. Averaging multiple images of the same pattern with random delays helps ensure that all points share approximately the same overall brightness at the end, and is generally unaffected by the scan speed and shutter time of camera 120.

Using the test image 610 captured in operation 406, method 400 proceeds to operation 408, and the processing element removes noise and otherwise prepares the test image 610 for analysis. Specifically, the test image 610 is warped to the virtual image plane of projector 110 so that the image can be more accurately compared to the desired projected content. Operation 408 is accomplished by using the lookup table generated in operation 402 to cause processing element 310 to compare the test image 610 with the rasterized representation of the test pattern data 600 and modify the test image to represent an image of the view from the projector's image plane.

In one embodiment, the series of test images 610 is represented by , and the members of are warped onto the virtual image plane of projector 110 using the dense 2D warping lookup table generated in operation 402. This allows the processing element to compare the test images 610 to the rasterized representation of the test pattern 600 data.

In some embodiments, the test image 610 can be cropped to focus on a desired point. For example, in some embodiments, the center point 202 is analyzed and the test image 610 is cropped to a small area around the center point 202. As will be appreciated below, the center point (or group of center points) may be the only one that needs to be analyzed to determine projector behavior. For example, the center point is the location that demonstrates the change in angle between the first and last points. The number of stopping points or speed control points typically changes only at the center point. In other words, while the first and last points are required to project a pattern, only the center point needs to be analyzed to determine the changes in angle, distance, and stopping points (acceleration and deceleration) that occur as the pattern changes. It should be noted that in instances where more than three points are used for the pattern, the center point evaluated can include multiple points, such as all points that fall between the first and last projected points in the pattern.

An example of a cropped image is shown in FIG6C . A cropped image 612 of results in a single white pixel at the center of the image corresponding to center point 602. Center point 602 will be the same for all and can be denoted as . Other points can also be used, with corresponding changes to the model input.

After warping the image to the projector and optionally cropping it, method 400 proceeds to operation 412 and applies noise reduction and binarization. For example, any method capable of robustly separating and detecting points in the image can be used. As a specific example, intensity thresholding can be used to remove outlier pixels and intensities. In one embodiment, intensity thresholding is applied where a brightness threshold is defined. Typically, surface illumination (and therefore the intensity of the captured image 610) varies primarily with increasing numbers of stop points. Therefore, an adaptive thresholding method combined with a fixed low intensity threshold method can be used to automatically segment all patterns. Operation 412 generates binary images for all n stop points of the test pattern 600, all angles, and all distances for each test image 610. It should be noted that in many instances, test images 610 with three or five stop points may generally appear identical because the same center point is illuminated, but in patterns with more stop points, the center point will be brighter. A new binary image can be generated for each combination of stop point, angle, and distance.

4, once the images have been processed in operation 412, the method 400 proceeds to operation 414 and compares the center point 602 in each of the (R) test images 610. Mathematically expressed, the binary images produced for all increasing stopping points are compared to .

In a specific embodiment, each of the binary test images 610 (analyzed for noise and optionally cropped) is compared to R using two error metrics: hit error and distance error. The hit error is represented by equation (1) and describes whether the desired point location of the pattern is illuminated by projector 110. The distance error is represented by equation (2), which describes how far the drawn pixel is within a predetermined area (e.g., a rectangular area within the image) and can be determined by the number of pixels away from the center of the rectangle. In equation (2), dist is the L2 distance transform.

It should be noted that in some embodiments, edge points rather than center points may be used to analyze the pattern. In these embodiments, the model and error calculations should be modified based on the changing spatial position.

Using the calculations performed in operation 414, method 400 proceeds to operation 416. In operation 416, processing element 301 determines the optimal model parameters using equation (3) below. In one embodiment, only those of are used in the calculation because, at that time, the desired spatial accuracy cannot be achieved using the current number of stopping points for this configuration of and. Therefore, the overall error term for each specific combination of, and is defined by equation (3):

which should be minimized to plot the desired point as accurately as possible.

Once the processing element analyzes the pattern 600 for each image using various stopping points for all and , a model database describing the display behavior of the laser projector 110 is defined using the following relationship. The optimal number of stopping points for and is declared as , and the corresponding error is declared as . The model database entry is initialized with zero stopping points and its initial error value accordingly. The number of stopping points is iteratively increased by 1, and its error value is evaluated and compared to . If exists for it, the optimal number of stopping points is increased by 1. In one example, t is set to . The analysis then continues.

Operations 410 through 416 are illustrated in FIG8 , which illustrates the cropping and comparison of a test image 610 and the selection of the optimal number of generated points. Referring to FIG8 , the analysis workflow for hit and distance error calculation for three points ( n = 3, n = 150, and n = 0.4) is performed. As shown in FIG8 , the test image 610 is warped from the camera image plane to the projector's virtual image plane and cropped to a region around the center point. Adaptive thresholding is then applied, and the error sum is calculated.

It should be noted that both distance and hit error can be sensitive to noise in the input data. In these instances, two additional post-processing steps can be applied to reduce error. First, for consecutive angles at a fixed distance, the model is constrained to have fewer stopping points only if the corresponding error also decreases. Otherwise, it is set to . Second, an outlier removal procedure can be applied to remove single outliers between consecutive angles of the same distance. In some embodiments, methods for smoothing the model and reducing model error can optionally be performed. For example, if the entire sequence of consecutive angles has non-monotonic variation, then all corresponding stops are assigned to a maximum number of stops within the sequence. Figure 7 illustrates a visualization of the results of measurements of database entries for one type of laser projector 110. As shown in Figure 7, the derived model parameters depend on the length (normalized distance) from the consecutive points and the angle relative to the consecutive points. In this example, as the angle between the points increases, more stopping points are required to make the content more accurate.

Once the model parameters are determined, method 400 proceeds to operation 418, and projector 110 is optimized for the selected content. To this end, input content (i.e., the content to be projected) is entered into the model. For example, a computer's memory may store a model or a database of models, and the model is evaluated using the process outlined below to find the optimal angle for traversing the path for the content that minimizes the total path length and requires the least number of stops.

As described above, the speed and spatial accuracy of content projected by the laser projector 110 generally depend on factors affected by the physical movement of the galvanometer mirror. Examples of these factors include: sluggishness, resonance, and the acceleration and deceleration required to display each point at the desired location. In instances where the mirror cannot be directly controlled (e.g., with an off-the-shelf laser projector), the optimization controls global parameters (such as overall scanning speed, number of stops, angle between consecutive points, distance between consecutive points, and number of stops) to improve the speed and spatial accuracy of the projector 110.

The optimization generates a point sequence path where the overall scan time is minimized, but the content will still be displayed accurately at the desired location. In addition, the time sequence (multi-frame content) should be projected so that the temporal and spatial point distribution is as uniform as possible to evenly distribute and further suppress perceived flicker.

Furthermore, in many instances, the scan path for global optimization of a sequence of random input points is an NP-complete problem and cannot be easily solved in a short timeframe. Therefore, the following optimization is chosen to find an approximate solution close enough to the global optimum to achieve the desired performance. However, in instances where more time is available, the optimization path can be fully solved.

Projector 110 can be analyzed for both still frame content and two or more moving or sequential content frames. Still frame optimization minimizes error costs for distance and angle, while sequential frames include an additional time cost.

Still frame optimization is used to minimize the total path length of the projector 110 that projects each point while reducing the time it takes to project the point by considering the optimal angulation. For example, to optimize the scan path of a single still frame containing points, the processing element 301 minimizes the error cost using the following equation (4).

Distance cost is defined as:

in,

.

To make the distance cost independent of the resolution of the virtual image plane, the maximum possible distance on the diagonal is normalized to 1.0.

The angle cost is defined as:

.

In equation (6), is the angle at the point formed by the path, and is the lookup of the nearest entry in the model database storing the required number of stopping points. The total cost is given by the weighted sum of two terms in equation (7):

.

As described above, the optimal values for and are estimated. This error is minimized using the k-opt method. In particular, 2-opt is applied to the path with randomized inputs and is started as soon as there is path improvement. In order to further improve the solution, 3-opt can be applied afterwards. Because the k-opt optimization depends on the starting configuration of the path, different starting configurations can be used to start the process several times and select the best solution. Other solvers can also be used, such as particle swarm optimization, Lin-Kernighan-Helsgaun heuristic, minimum spanning tree heuristic, simulated annealing, tabu search or Christofides heuristic.

In instances where the projected content includes two or more frames of projected content, the optimization considers spatial distribution across the frames. For example, if the content is a sequence of animated points in frames, the optimization additionally balances the spatiotemporal distribution of the points in consecutive frames of the animation. This helps ensure that closely located spatial points are drawn at approximately equal time offsets, ensuring even light distribution across the entire frame to reduce the perception of locally varying flicker. For example, any flicker that cannot be controlled (e.g., due to inherent physical properties) will be evenly distributed across each frame and across multiple frames to be less noticeable.

In particular, the still frame optimization is extended to take into account spatiotemporal consistency. The solution for the frame at position is initialized to the previous solution using a nearest neighbor (NN) method. In one embodiment, the NN method uses a kd-tree with the flann method applied, and for each point in , the ordinal position of the NN in is determined, and the position is normalized by the number of points in . It should be noted that the number of points in each frame can be arbitrary. Depending on the speed constraints, other types of nearest neighbor search methods can be used. Using relative ordinal positions, points are mapped to corresponding absolute ordinal positions for . After finding the NN, these are sorted so that the path with those points is closest to the start of the best path in the previous solution. Using this method, the spatiotemporal error cost is defined by the following equation (8).

The overall error term for time optimization is then defined as an extension of Equation (9):

In one embodiment, the temporal weighting factor is set to 10, while the sums are optimally set to 100 and 1. However, other weighting factors can be used and may vary based on the specific projector being optimized. Alternatively, the camera can be used to generate the weighted estimates. Note that in equations 4, 6, and 8, for out-of-bounds indices, they are mapped back to the index space in a circular fashion.

Figure 9 is a schematic representation of optimization. Referring to Figure 9, in this example, (a) illustrates optimizing a random path traversal for the first frame sequence using the distance and angulation derived from the model parameters (b). In the next frame, all points are mapped to the nearest point from the optimized previous frame, generating an initial path. This path serves as the starting point for optimizing the frame using an additional temporal error cost (c). This same process is repeated for the next and each successive frame in the sequence (d) to obtain a spatiotemporal path that minimizes scan time and flicker (e).

FIG10A illustrates a frame of content projected by projector 110 before optimization. As shown in FIG10A , when content is displayed in a random order without stopping points, the spatial locations for the content (e.g., the lines and vertices of a star design) are not precisely hit, and the points are rendered as strokes. However, using optimization method 400, projector 110 is optimized and renders the content as desired by adding multiple stopping points and an optimal drawing order. In short, by applying global optimization to the point cloud of the content and taking into account temporal effects, the method of the present disclosure can significantly enhance the spatial alignment of rendered points, the aesthetics of the rendering (e.g., points versus dashes when desired), and reduce the overall scanning speed of projector 110, thereby reducing flicker and other artifacts.

The above specification, examples, and data provide a complete description of the structure and use of exemplary embodiments of the invention as defined in the claims. Although various embodiments of the claimed invention have been described above with a certain degree of particularity or with reference to one or more individual embodiments, those skilled in the art may make numerous modifications to the disclosed embodiments without departing from the spirit and scope of the claimed invention. Other embodiments are therefore contemplated. It is intended that all matter contained in the foregoing description and shown in the accompanying drawings shall be interpreted as merely illustrative of particular embodiments and not as limiting. Changes may be made in detail or structure without departing from the essential elements of the invention as defined in the following claims.

Claims (19)

1. A method for optimizing a projector for displaying content, comprising: The processing element receives multiple test images corresponding to two or more test patterns projected by a projector on a projection surface, each of which contains at least two points; The processing element compares the multiple test images to evaluate one or more projector characteristics related to the distance between two points; The processing element generates a projector model representing the characteristics of the one or more projectors; and The model is used to determine the projection path of light corresponding to the pixels of the content projected by the projector onto the projection surface. 2. The method of claim 1, wherein each of the test patterns further comprises at least three points. 3. The method of claim 2, wherein comparing the plurality of test images by the processing element includes comparing the center point within the test pattern in each of the test images. 4. The method of claim 3, wherein the center point comprises a plurality of points sequentially projected between the first projection point and the last projection point. 5. The method of claim 3, wherein comparing the plurality of test images via a processing element comprises: Determine whether the center point is illuminated by the projector; and Determine the distance of the center point in the test image relative to the desired position of the center point in the test pattern. 6. The method of claim 1, wherein the processing element further evaluates projector characteristics related to the angle between consecutive points in the test pattern. 7. The method of claim 1, wherein the processing element further evaluates one or more projector characteristics related to a plurality of stop points between consecutive points in the test pattern by the projector. 8. The method of claim 1, further comprising projecting the content by the projector. 9. The method according to claim 1, wherein the projector is a laser scanning projector. 10. A method for generating a projector model to optimize the projection path of content, the method comprising: Multiple pattern images of the test pattern projected by the projector are captured by the camera; Analyze the pattern images to determine the distance and angle characteristics of each test pattern captured in each of the pattern images; A model database representing the projector is created based on the distance and angle characteristics of each test pattern; and The model database is used to optimize the projection path of light corresponding to the content emitted by the projector onto the projection surface. 11. The method of claim 10, further comprising comparing each pattern image input to the test pattern of the projector based on whether at least one point is illuminated at a desired location. 12. The method of claim 11, further comprising analyzing each pattern image against the test pattern input to the projector to determine the distance between points in the pattern image and points in the test pattern. 13. The method of claim 10, further comprising selecting a region around the center point of the test pattern represented in the pattern image. 14. The method of claim 10, wherein each test pattern comprises a first point, a second point, and a third point, wherein the first point is projected by a first projection, the second point is projected by a second projection, and the third point is projected by a third projection. 15. The method of claim 10, further comprising geometrically calibrating the camera and the projector to determine a dense correspondence between the image plane of the camera and the image plane of the projector. 16. The method of claim 10, wherein the projector is a laser scanning projector. 17. A system for optimizing a projection path for projected content, comprising: A laser projector for projecting the content; and A computer electrically connected to the laser projector, the computer including a memory comprising a projector model database corresponding to the display behavior of the projector, wherein the computer is configured to analyze the projector model to: Determine the minimum length of the total projection path of light on the projection surface for the content; and Determine the optimal angle for the traversal projection path that requires the minimum number of stopping points. 18. The system of claim 17, wherein the content comprises a first frame and a second frame, and the computer is further configured to analyze the projector model to equalize the spatiotemporal distribution of multiple points of the content in the first frame and the second frame. 19. The system of claim 18, wherein the content comprises a third frame and a fourth frame, and the computer is further configured to analyze the projector model to equalize the spatiotemporal distribution of multiple points of the content in the first frame, the second frame, the third frame, and the fourth frame.