HK1224467B - Method and system for projector calibration - Google Patents

Description

Method and system for projector calibration

Technical Field

The technology described herein generally relates to methods and systems for calibrating one or more projectors.

Background Art

Image projectors can be used to project images onto a projection surface, such as a screen or other surface. In some applications, video projectors can be used to enhance, compliment, or otherwise augment objects on a surface to create a dynamic and enjoyable user experience, such as an amusement park attraction. For example, characters or objects can be projected onto a surface that virtually "interact" with real objects on the surface.

Conventional video projectors have several limitations. In particular, they have a limited color gamut and brightness. Due to these limitations, presentations using only a conventional video projector can appear dull and monotonous. Furthermore, in situations where there is ambient lighting, the resulting image can appear washed out and unrealistic. In contrast, laser projectors, such as laser scanning projectors, have increased brightness and color gamut compared to conventional video projectors. In particular, laser projectors can project pure, saturated, i.e., monochromatic, red, green, and blue tones, which allows for a significantly wider color gamut than conventional video projectors.

However, known calibration techniques for laser projectors require a lot of user interaction, are time intensive, and are generally not very accurate.As such, there is a need for techniques that can be used to more accurately and automatically calibrate laser projectors.

The information included in this background section of the specification (including any references cited herein and any description or discussion thereof) is included for technical reference purposes only and is not to be considered subject matter by which the scope of the invention as defined in the claims is delimited.

Summary of the Invention

One example of the present disclosure relates to a method for calibrating a projector having non-uniform distortion characteristics. The method includes receiving, by a processing component, a first calibration image of a calibration pattern projected onto a projection surface, generating, by the processing component, a perspective projection matrix by analyzing the first calibration image and the calibration pattern, and determining, by the processing component, a nonlinear mapping function by analyzing the first calibration image.

Another example of the present disclosure includes a system for calibrating a projector, such as a laser projector or other projector with non-uniform distortion characteristics. The system includes a projector that projects a first calibration pattern onto a projection surface, a camera that captures a first calibration image of the first calibration pattern projected onto the projection surface, and a computing device in communication with the camera. The computing device generates a perspective projection matrix based on the first calibration image and determines a distortion map for the projector by analyzing the first calibration image and the first calibration pattern, the distortion map identifying nonlinear distortion present in the projector that is not accurately approximated by the perspective projection matrix.

Another example of the present disclosure includes a projection system having a laser projector and a video projector calibrated using nonlinear mapping techniques. Using this system, the video projector projects a first image onto a projection surface, and the laser projector projects a second image onto the projection surface. The second image overlays the first image on the projection surface and provides augmentation and/or emphasis for at least a portion of the first image.

This Summary is provided to introduce a selection of concepts that are further described below in a simplified form in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. A more comprehensive presentation of the features, details, utilities, and advantages of the invention, as defined in the claims, is provided in the following written description of various embodiments of the invention and is illustrated in the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

This patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

FIG1 is an isometric view of the projection calibration system of the present disclosure.

FIG. 2 is a block diagram of the calibration system of FIG. 1 .

FIG3 is a block diagram of the laser projector 110 of the system in FIG1 .

4A-4I illustrate various examples of calibration patterns for use in calibrating the projection system of FIG. 1 .

FIG5 is a flow chart illustrating a method of calibrating a laser projector.

FIG. 6 is an example calibration image of a calibration pattern as projected onto a projection surface.

7 is a flow chart illustrating a method for creating a distortion map for calibrating a laser projector.

FIG. 8A is an example of a distortion map created using the method of FIG. 7 .

8B is an example of calibration test pattern elements before and after calibration of the projector of FIG. 1 .

9 is an isometric view of an exemplary projection system using a calibrated laser projector.

FIG. 10 is a flow chart illustrating a method of projecting an image using the system of FIG. 9 .

FIG. 11A is an enlarged image of a background or scene image before emphasizing the image projected onto the scene image.

FIG. 11B is a magnified image of the scene image of FIG. 11A with an emphasis image superimposed on top thereof.

manual

Overview

The present disclosure generally relates to calibrating projectors that exhibit non-uniform distortion characteristics, such as laser projectors, projectors with fisheye lenses, ellipsoidal lenses, or other lenses with varying shapes. Methods also relate to using a calibrated projector (in one example, a laser projector) in one or more presentations, either alone or in combination with one or more laser or other types of projectors. The embodiments disclosed herein are generally applicable to any type of projector, including those with traditional lens projection systems. However, as described in detail below, the calibration methods can be particularly useful when used with projectors or projector systems that have optical components with atypical, non-uniform, or unexpected distortion properties. As used herein, the term "laser projector" means essentially any type of coherent light projection component, such as, but not limited to, electroscopic laser scanning projectors. The term "projector with non-uniform distortion characteristics" can refer to a projector with projection optics that exhibit extreme and/or non-uniform distortion, such as a projector with inferior lens optics, a laser projector, a projection system with a mirror in the optical path, or a projector that uses a fisheye or other wide-field-of-view projection lens (e.g., an image maximization (IMAX) projector). It is noted that when the calibration techniques as described herein are used with a conventional lens-based projector, some steps may not be necessary in order to accurately calibrate the projector or projection system. As used herein, the term "video projector" means essentially any type of lens-based two-dimensional raster projection component, such as a digital light processing projector, a liquid crystal display projector, and the like.

The calibration method includes creating a calibration pattern, projecting one or more calibration patterns onto a projection surface, capturing at least one calibration image of the calibration pattern(s) using one or more cameras, and using the captured calibration image to create a distortion map that transforms an input image into an output image. In some embodiments, two or more calibration patterns may be projected and two or more calibration images may be captured.

The calibration or test pattern used in the calibration method can be essentially any type of structured light pattern comprising an array of elements, such as dots, lines, line segments, shapes, tonal variations, or other types of patterns, which can be random or ordered, repeating or non-repeating. In some examples, the calibration pattern can represent a binary coded sequence, where each element (e.g., dot, line, etc.) within the pattern is uniquely identifiable. In some embodiments, multiple calibration patterns can be projected separately in sequence, providing additional data points detected from each pattern. As discussed below, this allows each element of the calibration pattern to be used to determine the physical location of the corresponding element of the calibration pattern on the projection surface.

Once a calibration pattern has been created or selected, the calibration pattern is provided to the projector for projection onto a projection surface. The projection surface can be a screen, an object, and/or substantially any type of surface. As the calibration pattern is projected, two or more cameras or a single three-dimensional camera captures a calibration image of the calibration pattern on the projection surface. In some embodiments, the laser projector projects the calibration pattern at a specific or known two-dimensional (2D) position on a virtual image plane of the laser projector. For example, the known 2D position can be the original input coordinates used to project light from the laser projector. Furthermore, as described above, two or more calibration patterns can be projected in a predetermined sequence and an image of each calibration pattern can be captured as it is projected.

Using a processor or other computing element, the captured calibration image and known 2D positions can be used to determine the 2D coordinates of the test pattern elements for the laser projector. However, the processor analyzes the identifiers of the test pattern elements used in the calibration pattern to determine the three-dimensional (3D) projected position of each element on the projection surface. For example, the processor can triangulate the 3D projected position for each element based on the element position within the projected calibration image and the known camera position and orientation. However, other methods of determining the 3D position of the test pattern elements can be used, such as using a 3D stereo camera or other imaging device capable of depth measurement, in which instances only a single camera may be required, while triangulating the projected position may require two cameras.

Once the actual 3D positions of the pattern elements have been determined, a perspective projection matrix can be created. For example, the processor can create the perspective projection matrix by assuming that the projection characteristics of a laser projector are approximately represented by a pinhole for a pinhole device. That is, the processor can assume that the laser projector has projection characteristics similar to those of a conventional projector with an optical lens system. The perspective projection matrix created in this manner is typically not as accurate as desired because a laser scanning projector does not behave like a pinhole device due to the use of a mirror assembly rather than a lens assembly and therefore does not resemble a perfect pinhole device. Therefore, the perspective projection matrix does not accurately represent the positions of the pattern elements.

To improve accuracy, additional nonlinear 2D mapping can be performed after applying the estimated perspective projection matrix to more accurately match the transformation of the 3D position of each pattern element. In one embodiment, an interpolation method is used to create a dense lookup table for points between each detected pattern element. Thus, the distortion exhibited by the laser projector at points on its virtual image plane can be accounted for and quantified, which allows for calibration of the laser projector. That is, a matrix transformation combined with the additional nonlinear 2D mapping can be used to transform or modify the input image to provide an output image, where the output image is projected so that each pixel is projected at the desired location on the projection surface.

Because laser projectors can be precisely calibrated to allow input images to be projected at the desired location, they can be used for presentations that require highly accurate projection. For example, a laser projector can be used to project an accent image on top of or in the same location as an image from a video projector. In this example, the laser projector can be calibrated so that specific elements of the projected input image will match corresponding or supplementary image content projected by the video projector. As another example, a laser projector can be used to project an image onto a 3D object or surface, with each point of the image displayed at the desired location.

In one embodiment, a system including a calibrated laser projector and one or more video projectors can be used together to provide presentations, outputs, entertainment features, amusement park attractions, and the like. In particular, the enhanced brightness and color saturation of the laser projector can be used to display high-frequency features, emphasize details, or otherwise amplify the video projector presentation onto one or more projection surfaces. As an example, if an object projected by a conventional video projector should have a shimmering or sparkling quality, such as a snowy scene or a glittering diamond, a laser projector can be used to project a much brighter image feature superimposed on the projected video object. In this example, the features of the laser projection can enhance the image data, creating a "shimmer" or other bright output, resulting in a more realistic and interesting presentation for users such as amusement park guests. Due to the calibration, the image projected by the video projector and the image projected by the laser projection can be substantially aligned to create a desired aesthetic effect.

In some embodiments, the system can display the desired image content from the projector, along with any other visual or audio effects that may accompany or be synchronized to the presentation, such as additional scene lighting, special effects sounds, and/or character dialogue.

DETAILED DESCRIPTION

Turning now to the figures, the calibration method and system of the present disclosure will now be discussed in greater detail. FIG. 1 is a perspective view of a calibration system 10. FIG. 2 is a simplified block diagram of the calibration system 10. Referring to FIG. 1 and FIG. 2, the calibration system 10 may include one or more laser projectors 110, a projection surface 161, one or more cameras 120a, 120b, and one or more computers 130. The one or more laser projectors 110, cameras 120a, 120b, and computer 130 may all be in electrical communication with one another and/or include the ability to share and transfer information between each device. Each component will be discussed in order below.

Projection surface 161 can be substantially any type of opaque surface or object. For example, projection surface 161 can be flat, non-planar, or variable and can include one or more textures or surface variations and/or colors. In some examples, projection surface 161 can include multiple surfaces at different depths or positions relative to each other. The type and configuration of projection surface 161 can vary as desired.

Cameras 120a and 120b can be substantially any device capable of capturing still or video images, such as a charge-coupled device camera or a complementary metal oxide semiconductor (CMOS) image sensor camera. The cameras 120a, 120b, collectively referred to as camera system 120, can be capable of capturing full-color images and/or monochrome images and can use any type of filter, such as one or more color filters. In one embodiment, cameras 120a, 120b are configured to capture substantially the entire dynamic range of laser projector 110 without significant clipping. Each camera 120a, 120b is registered or otherwise placed at a known location in its environment, such that each camera 120a, 120b has a specific orientation and position relative to projection surface 161.

In some embodiments, two or more cameras 120a, 120b are used. In these embodiments, cameras 120a and 120b are physically offset from each other by a predetermined separation distance _SD (i.e., a known separation distance). In addition to the separation distance _SD , the orientation of cameras 120a, 120b relative to each other can also be known or predetermined. For example, the orientation of cameras 120a, 120b can be determined manually or using standard methods. It should be noted that while two cameras 120a, 120b are shown, in other embodiments, more than two cameras may be used, or both cameras 120a, 120b may be replaced by a single camera, such as a 3D camera, stereoscopic, or depth-sensing camera system.

Continuing with Figures 1 and 2, calibration system 10 also includes a computer 130. Computer 130 analyzes data from both projector 110 and camera 120 and may also optionally control one or more functions of either device. In the example of Figure 1, only one computer 130 is shown; however, in other embodiments, more than one computer may be used. Computer 130 may include one or more processing elements 301, a power supply 302, a display 303, one or more memory components 304, and an input/output (I/O) interface 305. Each element of computer 130 may communicate via one or more system buses, wirelessly, or the like.

Processing element 301 can be substantially any type of electronic device capable of processing, receiving, and/or sending instructions. For example, processing element 301 can be a microprocessor or a microcontroller. Furthermore, it should be noted that selected components of computer 130 can be controlled by a first processor, and other components can be controlled by a second processor, wherein the first and second processors may or may not be in communication with each other.

The memory 304 stores electronic data used by the computer 130 to store instructions for the processing element 301, as well as to store presentation and/or calibration data used to calibrate the system 10. For example, the memory 304 may store data or content corresponding to various applications, such as, but not limited to, audio files, video files, etc. The memory 304 may be, for example, a magneto-optical storage device, a read-only memory, a random access memory, an erasable programmable memory, a flash memory, or a combination of one or more types of memory components.

The power supply 302 provides power to the components of the computer 130 and may be a battery, a power cord, or other element configured to deliver power to the components of the laser projector 10 .

Display 303 provides visual feedback to the user and can optionally serve as an input element that enables the user to control, manipulate, and calibrate various components of calibration system 10. Display 303 can be any suitable display, such as a liquid crystal display, a plasma display, an organic light emitting diode display, and/or a cathode ray tube display. In embodiments where display 303 is used as an input, the display can include one or more touch or input sensors, such as a capacitive touch sensor, a resistive grid, and the like.

The I/O interface 305 provides communication to and from the laser projector 110, the camera 120, and the computer 130, as well as other devices (e.g., other computers, auxiliary scene lighting, speakers, etc.). The I/O interface 305 may include one or more input buttons, communication interfaces (such as WiFi, Ethernet, etc.), and other communication components (such as a universal serial bus (USB) cable, etc.).

Optionally, the computer 130 may include a sensor 306. The sensor 306 includes essentially any device capable of sensing a change in a characteristic or parameter and generating an electrical signal. The sensor 306 may be used in conjunction with or in place of the cameras 120a, 120b, or may be used to sense other parameters such as the ambient lighting surrounding the projection surface 161. The sensor 306 and display 303 of the computer 130 may be varied as desired to meet the needs of a particular application.

The laser projector 110 of the calibration system 10 will now be discussed in greater detail. FIG3 is a simplified block diagram of the laser projector 110. Referring to FIG1 and 3, the laser projector 110 is used to project light patterns, images, etc., onto a projection surface 161 of a scene 160. The laser projector 110 can be substantially any type of coherent light and/or non-lens-based projection component. It should be noted that although the embodiments discussed herein are discussed with respect to laser projectors, the calibration methods and systems can be used with substantially any type of projector that exhibits non-uniform distortion characteristics (such as, for example, fisheye lens projectors, ellipsoidal projectors, etc.), and as such, the discussion of any particular embodiment is intended merely as an illustration.

In some embodiments, laser projector 110 can project red, green, and/or blue coherent light. In other embodiments, laser projector 110 can project light of substantially any other color or frequency, including visible or invisible light (e.g., ultraviolet, infrared, and other light), as necessary. Laser projector 110 can be an electroscope scanning laser projector, etc. Laser projector 110 can include a mirror assembly 201, a laser source 202, one or more processing elements 203, one or more memory components 204, an I/O interface 205, and a power supply 206. In some embodiments, computer 130 can provide some or all of the processing and storage functions for projector 110, and in these embodiments, one or more features of projector 110 can be omitted.

Mirror assembly 201 directs light emitted by laser source 202 onto projection surface 161. In some embodiments, mirror assembly 201 may include two or more mirrors connected to an electroscope, servo, or other motion-inducing element. In one embodiment, the mirrors of mirror assembly 201 may be oriented orthogonally relative to each other, such that one mirror rotates about a first axis and the other mirror rotates about a second axis orthogonal to the first axis. The motion-inducing element moves the mirrors based on input to projector 110 to change the output and position of light emitted from projector 110.

One or more processing elements 203 receive input image data from memory component 204 or I/O interface 205. Processing element 203 can be substantially similar to processing element 112 and can be any electronic device capable of processing, receiving, and/or sending instructions. For example, processing element 203 can be a microprocessor or a microcontroller.

Memory 204 stores electronic data used by laser projector 110 and/or computer 130 and can be volatile or non-volatile memory. Memory 204 can be substantially similar to memory 114, but in many embodiments may require less storage than other components.

The power supply 206 provides power to the components of the laser projector. The power supply 206 can be a battery, a power cord, or other element configured to deliver power to the components of the laser projector.

The laser source 202 may be one or more solid-state laser sources, such as laser diodes, or may be a gas laser source and/or other types of coherent light.

I/O interface 205 provides communication to and from laser projector 110 and computer 130, as well as other devices. I/O interface 205 may include one or more input buttons, communication interfaces (such as WiFi, Ethernet, etc.), and other communication components (such as a universal serial bus (USB) cable, etc.).

Referring again to FIG. 1 , the calibration system 10 is used to calibrate the laser projector 110. As part of the calibration process, the laser projector 110 projects a calibration pattern 150 onto the scene 160. FIG. 4A-4I illustrate various examples of the calibration pattern 150. Referring to FIG. 4A-4I , the calibration pattern 150 is a predefined, structured light arrangement having a number of pattern elements 151. For example, the calibration pattern 150 may include one or more clumps or areas having properties that are substantially constant or that vary within a specified range of values. In some examples, each pattern element 151 or point in a clump may be similar to each other in at least one characteristic. The calibration pattern 150 may be constructed in such a way that the position of each pattern element 151 may provide identification information. For example, each pattern element 151 may include an identifier associated therewith that may be explicitly displayed and/or implicitly inferred.

In some embodiments, pattern elements 151 may be arrays of laser dots or other shapes. In other embodiments, pattern elements 151 may be in the shape of a circle, a plus sign, a square, or other suitable geometric shapes. Furthermore, pattern elements 151 may include localized or structured changes or variations in color, hue, intensity, polarization, or shading, where these characteristics can provide additional data points to be tracked or can form pattern elements themselves. In one embodiment, calibration pattern 150 may comprise an array of 64x64 laser dots arranged in rows and columns as pattern elements 151. It should be noted that pattern elements 151 can take on essentially any shape or size as desired. As will be discussed in greater detail below, in some embodiments, each pattern element in pattern 150 may be projected in a temporal binary-coded sequence that uniquely identifies each element 151 through a series of binary on/off cycles. That is, calibration pattern 150 may be formed into any structured light pattern shape and layout that allows each element 151 to be detected and extracted, and its position within the pattern to be determined.

Continuing with reference to Figures 4A-4I, the calibration pattern 150 may include a plurality of laser spot test pattern elements 151 arranged in one or more orientations, arrangements, or patterns. In some embodiments, the pattern elements 151 may be evenly distributed in rows and/or columns, and in other embodiments, the pattern elements 151 may be distributed in a non-uniform but structured manner. For example, with reference to Figure 4A, the elements 151 are arranged in rows and columns, with each element 151 equidistant from each adjacent element 151. As another example, with reference to Figure 4B, the elements 151 are separated into two equal segments spaced apart from each other. In this example, the elements 151 within a segment are equally spaced relative to the other elements within the segment. As shown in Figures 4A-4I, other patterns having different characteristics or features for the arrangement of the elements 151 may be used.

The number of pattern elements 151 within pattern 150 can vary based on the desired accuracy of the calibration. In particular, as the number of elements 151 increases, accuracy can also increase. Furthermore, the separation or distance between each element 151 within pattern 150 can also improve the sensitivity of the calibration and, therefore, improve accuracy. For example, in instances where projection surface 161 may have a dynamic geometry, the separation between pattern elements can be reduced and/or additional pattern elements 151 can be added. By reducing the separation between corresponding pattern elements 151, more pattern elements 151 can be projected onto the same surface area of projection surface 161. In other words, by increasing the number of elements 151 projected onto a particular area, more elements 151 can be positioned on a facet of projection surface 161, which can allow for detection of more elements 151.

Calibration pattern 150 may be selected based on various characteristics associated with scene 160, a desired calibration accuracy for projector 110, etc. As such, any discussion of a particular embodiment is meant to be illustrative only.

The operation of the calibration system 10 and the method for calibrating the projector system will now be discussed in greater detail. FIG5 is a flow chart illustrating a method for calibrating the laser projector 110. Referring to FIG5, the method 500 may begin with operation 501. In operation 501, the laser projector 110 projects a calibration pattern 150 onto the projection surface 161. In one embodiment, the pattern elements 151 are projected individually or in groups in a temporal binary-coded sequence. For example, in one embodiment, each element 151 may be identified by a sequence of on/off occurrences; in other embodiments, the pattern elements 151 may be projected substantially simultaneously.

In some embodiments, a portion of a calibration pattern 150 comprising a subset of pattern elements 151 may be projected, followed by projection of the remainder of the overall calibration pattern 150 comprising a set of other pattern elements 151. In still other embodiments, a calibration pattern 150 comprising a first set of pattern elements 151 may be projected, followed by projection of another calibration pattern 150 comprising a second set of pattern elements 151, which may be the same as or different from the first calibration pattern 150. The second calibration pattern 150 comprising the second set of pattern elements 151 may be selected or chosen to help further refine the characteristics of the projection surface 161. Additionally, the second calibration pattern 150 may be selected based on image feedback provided by cameras 120a, 120b. In other words, calibration of a projector or projection system may be based on the projection of more than one calibration pattern 150 in an adaptive manner.

4A-4I , the configuration of calibration pattern 150 can vary as desired and, in some embodiments, can be tailored to the geometry or other characteristics of scene 160, including the surface and topography of projection surface 161. In other words, the selection of the arrangement of elements 151 within pattern 150 and the sequence in which pattern 150 is projected can be determined based on the desired projection surface 161.

After calibration pattern 150 is projected onto the scene in operation 501, the method proceeds to operation 502. In operation 502, cameras 120a, 120b capture a calibration image of the projected calibration pattern 150. As discussed above, the geometric positions of cameras 120a, 120b within scene 160 are known, i.e., the positions and orientations of cameras 120a, 120b relative to each other and other objects are known. Cameras 120a, 120b can be adjusted prior to use so that the camera shutters can capture pattern elements 151 without oversaturating the image sensor photosensors. This adjustment can be performed automatically or manually by the user (e.g., by adjusting exposure settings).

Using cameras 120a, 120b, one or more calibration images are captured. FIG6 illustrates a calibration image. Referring to FIG6 , calibration image 208 can be a photograph, digital image, or the like, that captures pattern 150 as projected onto projection surface 161. In the example shown in FIG6 , calibration pattern 150 of FIG4A is projected onto surface 161 and captured. Furthermore, as shown in FIG6 , pattern elements 151 of calibration pattern 150 are shown in calibration image 208 and, as will be explained in greater detail below, can be separately identified, as shown in FIG6 . In many embodiments, each camera 120a, 120b captures a calibration image 208, and due to the various positions and separation distances _SD of cameras 120a, 120b relative to scene 160, each calibration image 208 captured by each camera 120a, 120b can correspond to calibration pattern 150 projected onto projection surface 161 from a different perspective.

In some embodiments, multiple calibration patterns may be projected in a predetermined sequence and one or more calibration images may be captured for each calibration pattern 208. By using multiple patterns, the processing element may be able to more easily determine the distortion of the laser projector, as will be discussed in more detail below.

5 , once the calibration images 208 have been captured, the method 500 proceeds to operation 503. In operation 503, the calibration images 208 are provided to the computer 130, and the processing element 301 analyzes the calibration images 208. In particular, the processing element 301 analyzes the calibration images 208 to detect pattern elements 151 in each image 208. The processing element 301 detects the locations of the pattern elements 151 using an image analysis algorithm, such as a blob detection algorithm or other algorithm that can detect the locations of the pattern elements 151.

The image analysis algorithm used to detect the pattern elements 151 can be selected based on the characteristics of the pattern elements 151. For example, if the pattern elements 151 are dots or clumps, the processing element 301 can use a clump detection algorithm. As an example, the clump detection algorithm can include the steps of subtracting the background from the image and thresholding the image to mask suspected clumps or feature locations. The algorithm can then analyze each masked area to calculate the center of gravity of the clump. In an exemplary embodiment of the present disclosure discussed in more detail below, when the processing element 301 is configured to detect the pattern elements 151 using a clump detection algorithm, the processing element 301 can analyze and compare the position of the center of gravity for each detected clump relative to the center of gravity for other clumps in order to determine the coordinates of the clump within the 2D image plane.

In other embodiments, if the pattern elements 151 are selected as lines or line segments, a line center detection algorithm may be used. Other algorithms for detecting the pattern elements 151 may also or alternatively include feature detection, edge detection, sequential binary coded blobs, binary coded horizontal and/or vertical lines, Gray codes using long exposures, color coded patterns, intensity coded patterns, more complex pattern element features, etc. Depending on the shape of the pattern elements 151, some algorithms may be more accurate than others, while other algorithms may require less processing power.

Returning to FIG5 , once the test pattern has been captured by the camera system 120 and processed by the computer 130, the method 500 proceeds to operation 503. For example, the image analysis algorithm can review the calibration image 208 to detect areas that differ from surrounding areas in one or more properties such as, but not limited to, brightness, color, or hue. In this example, the captured pattern element 151 will be brighter and may have a different color than the surrounding area of the pattern 150, thereby allowing the processing component 112 to detect its location.

In one example, the processing component 112 can analyze the captured image 208 to determine the location of the center of each pattern element 151 in the 2D image plane of the corresponding camera 120a, 120b. In other words, the 2D coordinates of each pattern element 151 can be determined using detectable features of each element 151. In one example, a blob detection algorithm can be used to provide sub-pixel accuracy for the center location of the pattern elements 151. However, as discussed above, other algorithms can also be used.

1 and 5 , during operation 503, the processing component 112 of the computer 103 determines the pattern elements 151 visible through both cameras 120 a, 120 b. For example, the processing component 112 can analyze the calibration image 208 of the first camera 120 a to match a unique identifier associated with a particular pattern element 151 (e.g., due to its position, brightness, position in a sequence, color, or other characteristics) and determine whether the element 151 with the same identifier is in the calibration image 208 of the second camera 120 b.

It should be noted that, depending on the position of each camera 120a, 120b and the configuration of the projection surface 161, the number of pattern elements 151 visible through both cameras 120a, 120b may be a subset of the total number of pattern elements 151 in the calibration pattern 150. That is, due to obstructions related to camera positioning, projector positioning, or specific surface features of the projection surface 161, some pattern elements 151 may appear in only one calibration image or in neither calibration image 208.

As briefly discussed above, although two cameras 120a, 120b are used to capture the calibration image 208, in other embodiments, additional cameras may be used. Alternatively or in addition, one or more cameras capable of depth perception may be used, such as a 3D stereo camera, a KINECT-type depth camera, a 3D camera, or any other type of active or passive depth detection camera (such as a time-of-flight-based 3D camera or an infrared 3D camera).

Once pattern elements 151 have been detected in calibration image 208, the method may proceed to operation 504. In operation 504, processing component 112 or, alternatively, a user determines whether a sufficient number of pattern elements 151 have been detected. The sufficient number of detected elements 151 may depend on the desired accuracy of the calibration. In particular, whether the number of detected elements 151 is sufficient to accurately represent the virtual image plane of laser projector 110 and provide sufficient accuracy to adequately track the geometry of projection surface 161.

In operation 504 , a sufficient number of detected elements 151 may be determined based on, for example, a minimum threshold of detected pattern elements 151. Alternatively or additionally, the determination in operation 504 may be based on an amount of empty or blocked areas in the calibration image 208 .

If, in operation 504, the number of detected elements 151 is insufficient, the method may return to operation 501 and project a calibration pattern 150. The calibration pattern 150 may be the same as the originally projected or newly selected calibration pattern 150. For example, the projected calibration pattern 150 may be selected to better eliminate empty and/or blocked areas. If, in operation 504, the number of detected elements 151 is sufficient, the method 500 may proceed to operation 505.

In operation 505, processing component 112 determines the 3D position of test pattern elements 151 within scene 160. Using pattern elements 151 visible to at least two cameras 120a, 120b (or at least one depth-sensing camera), the positions of those test pattern elements 151 on projection surface 161 can be determined using triangulation or another position algorithm. For example, in an embodiment where two cameras are used, the separation distance _SD between the two cameras 120a, 120b, along with the change in perceived position of pattern elements 151 in calibration image 208 (e.g., parallax), allows processing component 112 to determine the distance of the projected pattern elements 151 from each camera 120a, 120b. With the known positions of each camera 120a, 120b and the positions of the projected pattern elements 151, the impact locations of the 3D positions of pattern elements 151 on projection surfaces 161 and/or 162 can be determined.

Once the 3D positions of at least a subset of the pattern elements 151 have been determined, the method proceeds to operation 506. In operation 506, the processing element 112 estimates a perspective projection matrix. In one example, the perspective projection matrix can be created by assuming that the projection of the laser projector 110 approximates a pinhole device. When the laser projector 110 is not a pinhole device and does not have a lens system that approximates a pinhole device, the perspective projection matrix estimated in operation 506 may need to be further corrected, but provides an initial calibration for the laser projector 110.

To create a perspective projection matrix, known inputs such as a predefined and/or preselected structure of a calibration pattern 150 and the detected projected position of each element 151 within the pattern 150 are used to map the 3D projection pattern. Specifically, the 2D position of each pattern element 151 on the virtual image plane of the laser projector 110 is mapped to a 3D projected position on the projection surface 161. In other words, a 2D to 3D correspondence is established and used to generate a perspective projection matrix using standard methods such as direct linear transformations. Examples of creating a perspective projection matrix can be found in the book "Multiple View Geometry in Computer Vision," 2nd edition, by R.I. Hartley and A. Zisserman (which is incorporated herein by reference in its entirety).

Once the estimated perspective projection matrix has been determined in operation 506, the method proceeds to operation 507. In operation 507, a distortion map, such as a lookup table, is created for laser projector 110. As discussed above, due to the mirror assembly 201 of laser projector 110, the distortion exhibited by laser projector 110 will generally not accurately conform to a standard pinhole-based lens distortion model (e.g., a radial distortion model) that can be applied to account for the nonlinearities of lens-based imaging systems. In particular, this approximation accounts for the distortion of video projectors with optical projection lenses. Therefore, the perspective projection matrix estimated in operation 506 (based on the standard lens distortion model) does not account for the full distortion properties of laser projector 110 and the estimated perspective projection matrix, and the actual positions of pattern elements projected by laser projector 110 into scene 160 will be misaligned. Depending on the desired accuracy required for laser projector 110, a perspective projection matrix may be sufficient. However, in many instances, a more precise calibration may be required, and therefore a distortion map, which in some embodiments may be a lookup table, may be created.

In operation 507, the perspective projection matrix generated in operation 506 can be used as an initial estimate of the distortion for laser projector 110 to generate a distortion map. FIG7 is a flow chart illustrating operation 507. Referring to FIG7, operation 507 can begin with process 701. In process 701, processing element 301 compares the estimated projection matrix with captured calibration image 208 to determine the deviation between the estimated position of pattern element 151 and the actual position of pattern element 151. As an example, if laser projector 110 projects pattern element 151 at coordinates (1, 1) on the virtual image plane, distortion due to mirror assembly 201 (and other inherent characteristics of the projector) can cause pattern element 151 to impinge on projection surface 161 at the "actual" or "perceived" coordinates of (1.5, 0.5). In operation 701, the deviation between the estimated perspective projection matrix (i.e., the estimated projected position of pattern element 151) and the actual projected position of pattern element 151 can be determined. Process 701 may be performed for each detected pattern element 151 .

Once the perspective projection matrix has been compared to the captured calibration image(s) 208, operation 507 may proceed to process 702. In process 702, the processor element 301 calculates the warping required to map each detected pattern element 151 in the estimated projection matrix to the intended projection location on the virtual image plane of the laser projector 110. In other words, the amount of distortion that needs to be applied to the input image so that the pattern element 151 strikes the projection surface 161 at the desired location. After determining the amount of warping for each pattern element 151, the method 700 proceeds to create a lookup table in operation 703. The lookup table may include the distortion amounts for those pattern elements 151 detected by the cameras 120a, 120b. To more accurately map the distortion of the laser projector 110, an interpolation method may be performed. Specifically, the processing element 301 may use the detected pattern elements 151 to interpolate additional points for the lookup table to estimate distortion characteristics for points between the projected pattern elements.

Various interpolation algorithms may be utilized, and the interpolation performed may be selected based on several variables, including the number of detected pattern elements 151, the structure or characteristics of the test pattern 150 or the geometry of the projection surface 161, processing resources, etc. In some embodiments, a spline-based, position-varying interpolation method is used to create additional entries to the lookup table for points between detected test pattern elements 151. Additionally, the lookup table may be a simple lookup table, a color map, or other type of lookup table.

FIG8A illustrates an example of a lookup table in the form of a distortion map. As shown in FIG8A , distortion map 900 may include entries assigned to or corresponding to pixel locations within the image plane of laser projector 110. The color assigned to an entry in map 900 may indicate the amount of correction or distortion to be applied at that pixel location to correct distortion caused by laser projector 110. That is, the normalized x and y coordinates of the image plane may be assigned a specific shade, hue, or color intensity, which may be interpreted by processing element 301 to determine the amount of correction. In one embodiment, after creating the lookup table in process 703 of operation 507, processing element 301 may store the distorted or corrected x and y locations in the red and green channels. In other embodiments, entries within distortion map 900 may correspond not only to the amount of distortion, but also to the direction of distortion. It should be noted that distortion map 900 is provided as an example of a lookup table created in operation 604, and other types of lookup tables may be used.

After the lookup table has been created in operation 703, process 507 terminates. Referring to FIG5 , upon completion of operation 507, method 500 may proceed to operation 508. In operation 508, the lookup table (e.g., color map 900) may be used to create a calibrated, emphasized image from the desired input image. Specifically, the image data may be converted or distorted to account for distortion that will occur when the image is projected by laser projector 110.

It should be noted that although the lookup table and the projection matrix are discussed as being calculated in separate operations, in some embodiments, the lookup table and the projection matrix can be determined in a single operation. For example, the projection matrix can be constrained to one or more predetermined properties such as a specific field of view, aspect ratio, etc. By constraining the matrix, the lookup table and the projection matrix can be calculated substantially simultaneously in the same operation.

In one example, in operation 508, the lookup table created in process 604 is used to convert the coordinates of the laser projector 110 virtual image plane so that an image projected by the laser projector 110, including image data elements (e.g., pixels or groups of pixels), will strike the projection surface 161 in a desired location.

FIG8B is an illustration of the transformation from operation 508 using mapping 900. Referring to FIG8B , the uncalibrated position 802 based on the perspective projection matrix is overlaid with the transformed, calibrated position 803 of the pattern element 151. As shown in FIG8B , the distortion exhibited by the laser projector 110 can vary widely across the virtual image plane of the laser projector. The change in position between the calibrated pattern element 803, represented by an "O," and the calibrated pattern element 802, represented by an "X," is a result of the nonlinear transformation performed in operation 508, as discussed above. In many embodiments, the calibrated pattern element 802 can be aligned with the desired position of the calibration pattern element prior to being projected by the uncalibrated projector 100.

Referring again to FIG. 5 , after the emphasized input image has been converted to form a calibrated emphasized image in operation 508, method 500 proceeds to operation 509. In operation 509, processing element 112 may transmit the calibrated emphasized image data to laser projector 110. Alternatively or in addition, processing element 112 may store the converted image in memory 114 or transmit the converted image to another computing device. After operation 509, the method proceeds to an end state 510.

A system for using a calibrated laser projector 110 to provide an emphasized image and/or another projected image that is aligned with features on a projection surface 161 will now be discussed in greater detail. FIG9 is a perspective view of a presentation system 12. The presentation system 12 includes a video projector 100 and one or more laser projectors 110. The presentation system 12 projects one or more images onto a scene 160, which may include any number of objects and may be of substantially any shape, such as a 3D shape, a surface, etc. As described above with respect to FIG1 , the projection surface and/or the scene may include projection surfaces having varying configurations, different colors, textures, etc. It should be understood that the presentation system 12 may be used indoors or outdoors.

Video projector 100 can be positioned substantially anywhere relative to scene 160, but is typically positioned so that it is oriented facing projection surface 163. Video projector 100 is configured to project one or more images, such as pictures and/or videos, onto projection surface 163.

Video projector 100 has a field of view 101 that defines a projection area for video projector 100. It should be noted that video projector 100 of presentation system 12 can be a rasterization-based video projector capable of displaying red, green, and blue (RGB) or another color space image signal. However, a monochrome projector can alternatively be used. Video projector 100 will typically include a light source and a lens. The light source can be any type of light-emitting element, such as, but not limited to, one or more light-emitting diodes (LEDs), an incandescent light bulb, a halogen lamp, etc. The lens is in optical communication with the light source and transmits light from the source to a desired destination (in this case, projection surface 163). The lens varies one or more parameters to affect the light, such as focusing the light at a specific distance. Video projector 100 may also include a white balance adjustment or other color balancing features that can be adjusted automatically and/or manually. This allows the projector's white point to be manipulated to avoid color shifts visible to human observers.

5 , the laser projector 110 may be calibrated so that the desired input image may be transformed or otherwise modified so that image elements of the image are projected at desired locations on the projection scene 160. This allows the projector 110 to align its output image with the image output of the video projector 110 to achieve highlights, emphasis, or other augmented presentations.

With reference to FIG10 , a presentation method 400 using presentation system 12 will now be discussed. The method begins at operation 401 and computer 130 receives input video projector image data from a desired source, such as another computing device, a memory storage component, a network, etc. The input image data may include scene image data, such as texture information, still or moving images, animated images (such as video sequences), etc. For example, the scene image data may include characters virtually "interacting" with each other, elements, or features within scene 160 (e.g., traversing portions of objects forming scene 160). In other words, the scene image data may contain any type of environmental presentation information appropriate for the desired scene.

After operation 401, method 400 may proceed to operation 402. In operation 402, processing element 112 of computer 130 receives an emphasis image corresponding to output for laser projector 110. The emphasis image may include image data that augments, supplements, or otherwise emphasizes scene image data for the video projector in a predetermined manner. In general, the emphasis image may include features or characteristics that may not be replicated or produced by video projector 100 but that can be accurately projected as desired due to the unique characteristics and brightness of laser projector 110.

It should be noted that both the scene image data and the emphasis image data may be stored together, for example, on the memory 114 of the computer 130 for use in controlling the presentation system 12. Alternatively, the scene image data may be stored separately from the emphasis image data and accessed and transmitted separately by the computer 130 to the appropriate projector.

Once computer 130 receives the input scene image and the emphasis image, method 400 may proceed to operation 403. In operation 403, processing element 112 converts the emphasis image to account for the distortion of laser projector 110. For example, using the lookup table created in method 500 of FIG. 5 (such as color map 900 of FIG. 8A ), processing element 112 adjusts the emphasis image. Thus, when the emphasis image is output by projector 110, various image elements, such as each pixel or group of pixels, may be projected at desired locations on scene 160. Alternatively, or in addition, the characteristics of laser projector 110 may be adjusted based on the lookup table to compensate for the distortion. In this example, the emphasis image may not need to be modified because laser projector 110 itself may be modified. That is, the coordinate system used by laser projector 110 to determine the projection position may be adjusted or modified to project light at the corrected emphasis image location as determined above.

Once the emphasis image or laser projector 110 has been modified, method 400 may proceed to operation 404. In operation 404, video projector 100 projects scene image data onto scene 160. For example, referring to FIG9 , scene image data 1200 may be provided to video projector 100 by computing device 130 or other device, and video projector 100 outputs light corresponding to the desired scene image.

7 , when projecting a scene image, method 400 may proceed to operation 405. In operation 405, laser projector 110 projects the transformed or modified emphasis image onto scene 160. Due to the calibration of laser projector 110 and/or the modification of emphasis image 1100, features of the emphasis image projected by laser projector 110 may be projected onto locations within scene 160 that correspond to features of the scene image data projected by projector 110.

9 , the dual projection of the video projected image 1200 and the laser projected image 1100 thus allows the emphasis image to augment the image projected by the video projector to create a combined image 1300. As an example, the scene image data may include a character such as a fairy, and the emphasis image may be used to highlight or emphasize the fairy dust trail following the fairy by applying a shimmering or glowing effect.

FIG11A is a magnified image of a scene image, and FIG11B is a magnified image of the scene image of FIG11A with an emphasis image projected therein. Referring to FIG9 , 11A, and 11B, scene image 1200 is projected by a video projector and may not include high-frequency details, such as very bright or flickering spots. However, in FIG11B , once emphasis image 1100 is projected by laser projector 1100, combined image 1300 can include both the overall details of scene image 1200 and the augmented portion of emphasis image 1100. Emphasis image 1100 provides bright tones, different colors, or other characteristics that cannot be achieved with a video projector.

In one example, the emphasis image 1100 is configured to be projected so as to be overlaid on the scene image 1200 projected by (multiple) video projectors 100. Alternatively, the two images can be projected together in other ways. The scene image 1200 can represent scene geometry, characters, background scenes, etc. As can be seen in Figure 11B, once calibration of the projection system according to the present disclosure has been performed, the scene image data projected by the video projector 100 is overlaid and aligned with the emphasis image data projected by the laser projector 100. And the emphasis image is mixed with the scene image to create the desired aesthetic presentation. The alignment between the scene image and the emphasis image is possible because the emphasis image 1100 is modified using the distortion map 900 to account for the distortion. Alternatively or in addition, the laser projector 110, rather than the input image for the emphasis image, can be calibrated to correct for such distortion.

In some embodiments, the fields of view (FOVs) of the laser projector and the video projector substantially overlap. In other embodiments, the FOVs of the laser projector and the video projector only partially overlap, where the non-overlapping portions of the FOVs of the respective projectors may overlap with the FOV of another projector provided in the projection system, such as a second video projector oriented toward a different portion of the scene or oriented at a different perspective angle or height within the scene 160.

The foregoing specification, examples, and data provide a complete description of the structure and use of exemplary embodiments of the invention as defined in the claims. Although various embodiments of the claimed invention have been described above with a certain degree of particularity or with reference to one or more individual embodiments, those skilled in the art may make numerous changes to the disclosed embodiments without departing from the spirit and scope of the claimed invention. Other embodiments are therefore contemplated. It is intended that all matter contained in the foregoing description and shown in the accompanying drawings be construed as illustrative of particular embodiments only and not as limiting. Changes in detail or structure may be made without departing from the essential elements of the invention as defined in the appended claims.

Claims (17)

1. A method for calibrating a projector, comprising: The processing element receives a first calibration image of a first calibration pattern projected onto a projection surface. The three-dimensional positions of multiple pattern elements are determined by processing the components; The processing element generates a perspective projection matrix by analyzing the first calibration image and the first calibration pattern; and The processing element determines the nonlinear mapping function by analyzing the first calibration image. The first calibration pattern includes multiple pattern elements. The nonlinear mapping function is determined by comparing the first calibrated image with the perspective projection matrix using a processing element. 2. The method according to claim 1 further includes detecting multiple pattern elements using an image analysis algorithm via a processing element. 3. The method of claim 1, further comprising generating a calibrated image by the processing element by transforming the first calibrated image using a nonlinear mapping function. 4. The method of claim 1, further comprising determining a transformation amount for each of the plurality of pattern elements by means of a processing element. 5. The method of claim 1, further comprising creating a lookup table using a nonlinear mapping function via a processing element. 6. The method of claim 5, further comprising interpolating within the lookup table by a processing element to increase the number of values. 7. The method according to claim 1, further comprising: A second calibration image of a second calibration pattern projected onto a projection surface is received by a processing element, wherein the second calibration pattern is different from the first calibration pattern. 8. The method of claim 7, wherein generating the perspective projection matrix further comprises analyzing the second calibration image and the second calibration pattern. 9. A system for calibrating a projector with non-uniform distortion characteristics, comprising: A projector that projects a first calibration pattern onto a projection surface; A camera, wherein the camera captures a first calibration image of a first calibration pattern projected onto a projection surface; and A computing device that is in electrical communication with a camera and has at least one processing element, the computing device performing the following operations: Determine the three-dimensional positions of multiple pattern elements; Generate a perspective projection matrix based on the first calibration image; and A distortion map for the projector is determined by analyzing a first calibration image and a first calibration pattern, wherein the distortion map determines the distortion present within the laser projector, and wherein the distortion map is determined by comparing the first calibration image and the perspective projection matrix. The first calibration image includes a structured light pattern with multiple pattern elements. 10. The system of claim 9, wherein the processing element determines the three-dimensional position of the pattern element by a depth detection algorithm. 11. The system according to claim 9, wherein the depth detection algorithm is a triangulation algorithm. 12. The system of claim 9, wherein the processing element uses a cluster detection algorithm to detect a plurality of pattern elements. 13. The system of claim 12, wherein the processing element further interpolates points falling between pattern elements within the first calibration image. 14. The system of claim 9, wherein the projector is an electroscope laser scanning projector. 15. The system of claim 9, wherein the processing element creates a modified output image by transforming the input image through applying a distortion map, wherein the modified output image is for distortion correction of the laser projector. 16. A projection system, comprising: A video projector projects a first image comprising multiple elements onto a projection surface; and The laser projector projects a second image onto the projection surface, wherein, By modeling the laser projector as a pinhole device with a pinhole approximation, the laser projector is calibrated using a perspective projection matrix to provide a first calibration approximation based on detecting multiple elements in a first image through an image analysis algorithm and a nonlinear mapping function to correct inaccuracies in the first calibration approximation generated by the pinhole approximation and to determine a distortion map to adjust for the inherent distortion of the laser projector, wherein the distortion map is determined by comparing the first image and the perspective projection matrix. The second image overlays the first image onto the projection surface. 17. The projection system of claim 16, wherein the second image is aligned with a first portion of the first image to provide emphasis on the first image.