HK40092035A - Three-dimensional scanner having sensors with overlapping fields of view - Google Patents

Description

3D scanner with sensors that have overlapping fields of view

Technical Field

This invention generally relates to three-dimensional scanners, and more particularly to three-dimensional scanners having sensors with overlapping fields of view.

Background Technology

A 3D scanner is a device used to construct a 3D model of the surface of a physical object. Applications of 3D scanners span many fields, including industrial design and manufacturing, computer animation, science, education, medicine, art, and design.

Summary of the Invention

This disclosure relates to 3D scanning technology. One method of 3D scanning is the use of so-called "structured light," in which a projector projects a known light pattern onto the surface of an object. For example, light from the projector can be guided through a slide with a pattern printed on it. The shape of the object's surface is inferred from the distortion in the light pattern captured by a camera. One or more cameras can be used to obtain an image of the pattern reflected on the object. By measuring the position of the pattern in the image (e.g., measuring the distortion of the pattern), a computer system can determine the position on the object's surface using simple geometric calculations (e.g., triangulation algorithms). The structured light method can be compared to other methods (e.g., time-of-flight methods), in which a laser rangefinder finds the distance to the surface by timing the round-trip time of pulses of light that perform a raster scan on the surface.

To determine the position of an element on the surface of an object, a computer system needs to know which element in the image corresponds to which element on the slide. There are two general approaches to solving this problem: one approach utilizes coded elements, while an alternative approach relies on non-coded elements. With coded elements, elements in the pattern possess unique recognizable properties that allow the computer system to understand which imaging element corresponds to which element on the object. With non-coded elements (e.g., lines), other methods are needed to resolve ambiguity between one imaging element and other imaging elements.

In some embodiments, a method is provided for eliminating ambiguity of imaging elements (e.g., lines) in a non-coded structured light method for 3D scanning. The method is performed using a scanner having a projector and at least two cameras. The projector projects multiple lines onto the surface of an object. Reflective imaging is achieved using a first camera and a second camera. Elements are detected in a first image (e.g., distorted lines) from the first camera, and a correspondence with elements in the projected pattern is assumed. Using this assumption, the position of the element is transferred to a second image from the second camera. If the element is also not present in the second image, the assumption is rejected. In some embodiments, the assumption is rejected if it would result in a position on the object outside the depth of focus of the first camera.

In some embodiments, a 3D scanner is provided. In some embodiments, the 3D scanner uses a structured light method. In some embodiments, the 3D scanner uses non-coded elements. The performance of the 3D scanner is improved by the overlapping fields of view of two or more cameras and a projector. In some embodiments, for improved ease of manufacture and other benefits, the two or more cameras and the projector have substantially parallel optical axes. Thus, two or more cameras (including their sensors and optics) can be mounted on a parallel plane. For example, the sensors of two or more cameras can be mounted on a single printed circuit board. To maximize the overlap of the fields of view of the two or more cameras, the sensor of at least one camera is shifted (e.g., offset) relative to its optical axis.

For this purpose, the 3D scanner includes a projector configured to project a plurality of non-coded elements onto an object. The projector has a first optical axis. The 3D scanner further includes a first camera, the first camera including a first lens and a first sensor. The first lens focuses reflections of a first portion of the plurality of non-coded elements onto the first sensor. The first lens defines a second optical axis. The 3D scanner includes a second camera, the second camera including a second lens and a second sensor. The second lens focuses reflections of a second portion of the plurality of non-coded elements onto the second sensor. The second lens defines a third optical axis. The projector, the first camera, and the second camera are offset from each other in a first direction. The first optical axis is substantially parallel to the second optical axis, and the second optical axis is substantially parallel to the third optical axis. The center of the first sensor is shifted away from the second optical axis along the first direction. In some embodiments, the center of the second sensor is shifted away from the third optical axis along the first direction.

Attached Figure Description

For a better understanding of the various described embodiments, reference should be made to the detailed implementation described below in conjunction with the following figures.

Figure 1 is a schematic diagram of a front view of a 3D scanner according to some embodiments.

Figure 2 is a schematic diagram of a side view of a 3D scanner according to some embodiments.

Figure 3 is a schematic diagram of a camera with overlapping field of view according to some embodiments.

Figure 4 is a schematic diagram of a camera having a sensor that is shifted relative to the optical axis of the camera, according to some embodiments.

Figure 5 is a schematic diagram of a side view of a 3D scanner according to some embodiments.

Figure 6 is a schematic diagram of a side view of a 3D scanner according to some embodiments.

Figure 7 is a schematic diagram of a side view of a 3D scanner according to some embodiments.

Figure 8 illustrates a flowchart of a method for generating a 3D model of an object according to some embodiments.

Figure 9 is a block diagram of a 3D scanner according to some embodiments.

Generally, the various figures described above illustrate different embodiments of the 3D scanner provided in this disclosure. However, it will be understood that certain features (e.g., camera, projector, etc.) of the scanner shown and described in one figure may be similar to features described in other scanners shown with reference to other figures. For the sake of brevity, such details will not be repeated throughout this disclosure.

Detailed Implementation

Examples of the described embodiments are now illustrated in the accompanying drawings with reference to the embodiments. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to those skilled in the art that the various described embodiments can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure the inventive aspects of the embodiments.

Note that, as used herein, the term offset is used to refer to the relative positions of the camera and detector within the scanner body (e.g., housing), while the term shift is used to describe the position of the sensor relative to the optical axis of the camera or the slide relative to the optical axis of the projector.

Figure 1 shows a front view of a 3D scanner 100 including a projector 114 that projects one or more non-coded elements onto an object to be measured. In some embodiments, the one or more non-coded elements are portions of a non-coded pattern of light (i.e., a non-coded pattern of light including one or more non-coded elements). The 3D scanner 100 includes three cameras (a first camera 102, a second camera 106, and a third camera 110). In some embodiments, fewer than three cameras are used. For example, in some embodiments, two cameras are used. In other embodiments, only one camera is used. In some embodiments, more than three cameras are used (e.g., four cameras, five cameras). The first camera 102 includes a first sensor 104, the second camera 106 includes a second sensor 108, and the third camera 110 includes a third sensor 112.

In some embodiments, three cameras are used to capture non-coded patterns of light reflected from an object. In some embodiments, the non-coded pattern of light comprises structured light patterns, such as lines or other repeating elements. The term non-coded pattern means that such lines or repeating elements lack the individual, unique characteristics of specific elements that allow the pattern to be identified in the captured image. In some embodiments, two or more cameras are used to identify specific elements (e.g., lines) recorded in the images of each of the two or more cameras. In some embodiments, the non-coded pattern has a set of simple elements (e.g., lines, dots, stripes) that are relatively small in at least a first direction (e.g., the x-direction, the y-direction). The individual features of these simple elements are not prominent enough to identify the elements from images acquired by a single camera that is not sufficiently close to the projector 114.

Each of the first camera 102, the second camera 106, and the third camera 110 records image data of light emitted by the projector 114 and reflected from the object to be measured, and transmits the image data to a computer system (e.g., the computer system of the 3D scanner 900 and/or the remote device 936 of FIG9).

Line 118 on the first sensor 104 indicates the displacement of the center of the first sensor 104 from the geometric center of the first camera 102 along the x and y directions. In some embodiments, the geometric center of the first camera 102 is defined by the optical axis of its optics. Line 120 on the second sensor 108 indicates the displacement of the center of the second sensor 108 from the geometric center of the second camera 106 along the x and y directions. Further details regarding the displacement of the sensor centers are depicted in FIG. 4. The center of the third sensor 112 of the third camera 110 is not displaced. Therefore, no line indicating displacement is shown on the third sensor 112.

Each camera has a field of view, as described in further detail below. In some embodiments, the field of view of one of the cameras differs from that of the other cameras. For example, in some embodiments, the field of view of an object from the third camera 110 is smaller than the field of view of an object from the first camera 102. In some embodiments, the field of view of an object from the third camera 110 is larger than the field of view of an object from the first camera 102. In various embodiments, each of the fields of view of the cameras may be the same or different.

Projector 114 includes slide 116, and line 122 represents the displacement of the center of slide 116 from the geometric center of projector 114 along the x-direction. In some embodiments, the geometric center of projector 114 is defined by the optical axis of projector 114, as described in subsequent figures. Note that in some embodiments, the displacement of the center of slide 116 is optional (that is, in some embodiments, the slide is centered relative to the projector).

In some embodiments, the projector 114, the first camera 102, the second camera 106, and the third camera 110 are all positioned on the same plane (e.g., in the x-y plane, at a specific value z). In some embodiments, the projector 114, the first camera 102, the second camera 106, and the third camera 110 are located at different z values (e.g., one or more cameras and/or the projector extend above or below the plane of FIG. 1).

Figure 2 illustrates a 3D scanner 200, in which a projector 202 and three cameras (first camera 204, second camera 206, and third camera 208) are arranged in a substantially straight line along the y-direction. For example, the positions of the projector, first camera 204, second camera 206, and third camera 208 do not vary substantially along the x-direction. In some embodiments, the three cameras are arranged on one side of the projector 202. For ease of illustration, the displacement of the center of the slide from the optical axis of the projector 202 (if present) and the displacement of the center of one or more of the sensors of the first camera 204, second camera 206, and third camera 208 from their respective optical axes (if present) are not shown in Figure 2.

Figure 3 shows a view of a 3D scanner 300 along the z-y plane. The 3D scanner 300 includes a projector 302, a first camera 304, and a second camera 306. The projector 302 has an optical axis 308, the first camera 304 has an optical axis 310, and the second camera 306 has an optical axis 312.

In some embodiments, the optical axis 308 of the projector 302 is defined by optics (e.g., a lens system) in the projector 302. In some embodiments, the optical axis 310 of the first camera 304 and the optical axis 312 of the second camera 306 are defined by imaging optics (e.g., imaging lenses or lens systems) in the first camera 304 and the second camera 306, respectively. In some embodiments, the optical axis passes through the center of curvature of each surface of the optical element in the imaging optics and coincides with the rotational symmetry axis of the imaging optics.

Imaging optics in the first camera 304 and the second camera 306 form a clear image of the object located in region 314 (on the corresponding sensors of the first camera 304 and the second camera 306). Region 314 is defined by a near plane 316 and a far plane 318.

Defocusing blurs the image of objects located along the z-direction closer to the near plane 316 or farther than the far plane 318 from the projector 302, the first camera 304, and the second camera 306. In other words, in some embodiments, the near plane 316 and the far plane 318 are defined by a threshold resolution. In some embodiments, the threshold resolution is the resolution required to detect and/or distinguish individual elements on the surface of an object. Outside the region 314 defined by the near plane 316 and the far plane 318, defocusing results in objects not being distinguishable at the threshold resolution. The optimal focus plane 324 is located within region 314. In some embodiments, the surface located close to the z-distance coinciding with the optimal focus plane 324 forms the sharpest image on the sensors of the first camera 304 and the second camera 306.

The field of view of a particular camera is a region or area of the object space captured on the camera's sensor. In some embodiments, region 314 also represents a portion of the field of view of each of the first camera 304 and the second camera 306. The projector 302 also projects a non-coded pattern of light into the field of view of at least one of the first camera 304 and the second camera 306 (note that in some embodiments, the fields of view of the first camera 304 and the second camera 306 overlap, and the projector projects a non-coded pattern of light in the case of overlapping fields of view).

The projector 302, the first camera 304, and the second camera 306 are mechanically arranged to overlap their fields of view, enhancing the performance of the 3D scanner 300. This is because when the projected elements of a pattern are within the field of view of each of the cameras (e.g., the first camera 304, the second camera 306), images obtained by different cameras can be used to identify the projected elements (e.g., to identify the correspondence with the projected pattern), as described in method 800 with reference to FIG8. In some embodiments, for ease of manufacture and other benefits, two or more cameras and the projector have substantially parallel optical axes, as shown in FIG3. Thus, two or more cameras (including their sensors and optics) can be mounted on a parallel plane (as shown in FIG7). For example, the sensors of two or more cameras can be mounted on a single printed circuit board. To maximize the overlap of the fields of view of two or more cameras, the sensor of at least one camera is shifted relative to its optical axis.

For this purpose, the optical axis 308 of the projector 302, the optical axis 310 of the first camera 304, and the optical axis 312 of the second camera 306 are substantially parallel to each other. In other words, the angle formed between any pair of optical axes 308, 310, 312 is close to zero (e.g., within design tolerances). To maximize the overlap of the fields of view of the first camera 304, the second camera 306, and the projector 302, the geometric center of the sensor of each of the first camera 304 and the second camera 306 is shifted from the optical axis of the respective camera. In some embodiments, the geometric center of the sensor is the centroid of the sensor. In some embodiments, the geometric center of the sensor is the center pixel of the sensor.

In some embodiments, the geometric center of the slide in projector 302 is also shifted from the optical axis of the projector, such that the field of view of projector 302 overlaps with the field of view of the first camera 304 and the second camera 306. As used herein, the term field of view of the projector (or equivalently, the field of view of the projection) is used to refer to the area where the slide pattern is projected. In some embodiments, projector 302 includes a light source, such as a lamp, LED, or laser, and the optical system includes a condenser lens. The condenser lens renders a diverging beam from the light source (e.g., a point source) into a substantially parallel beam to illuminate an object, such as the slide in the projector. In some embodiments, the slide defines a plurality of geometric elements. In various embodiments, the geometric elements include points and/or horizontal (or vertical) parallel lines or bands. In some embodiments, the optical system of the projector includes additional optics (e.g., lenses) behind the slide.

In other words, each point in the measurement area is imaged in each camera of the 3D scanner 300. Typically, it is desirable to increase the measurement area of the object (e.g., the area of the object for which usable data is obtained in each image for the purpose of producing a 3D reconstruction of the object). Increasing the overlap of the fields of view of the cameras (e.g., cameras 304 and 306) increases the measurement area of the object. When the fields of view are fully overlapped (e.g., at the optimal focus plane 324), each of the cameras in the 3D scanner 300 receives reflected light from the same measurement area. In some embodiments, when the 3D scanner includes three cameras, the measured object simultaneously has corresponding imaging points on all three cameras. In some embodiments, when the 3D scanner includes four cameras, the measured object simultaneously has corresponding imaging points on all four cameras.

Figure 4 illustrates how the center of a sensor, according to some embodiments, is displaced (e.g., offset) relative to the geometric center of a camera (e.g., any of the cameras shown in previous or subsequent figures, such as camera 102 or 106 in Figure 1). Camera 400 includes a sensor 402 (shown as a rectangle with a dashed outline) having a center 404. In some embodiments, the center 404 of sensor 402 is a pixel at the center of the pixel array of sensor 402. In some embodiments, the center 404 is the centroid of sensor 402. Camera 400 also includes an imaging optics 406 (e.g., one or more lenses) that images one or more objects in a field of view 410 onto sensor 402. Imaging optics 406 defines an optical axis 408. In some embodiments, the optical axis 408 of imaging optics 406 is an axis of symmetry (e.g., a rotational symmetry axis) passing through the center of curvature of a lens (or other optical element) in imaging optics 406. For illustration, imaging optics 406 is represented by a single lens. In some embodiments, imaging optics 406 is an imaging system comprising one or more lenses and/or mirrors.

In some embodiments, the imaging optics 406 captures a larger field of view 410 (with a width 418 along the y-direction) than the field of view detectable by the sensor 402. For example, the imaging optics 406 forms an image 412 of the field of view 410. In FIG. 4, two peripheral rays of the field of view 410 (shown as dotted lines) are schematically shown as passing through the center of the imaging optics 406. For example, outside the peripheral rays, aberrations (e.g., vignetting) become too severe to collect suitable data. The image 412 has a lateral dimension 416 (along the y-direction, as shown) greater than the width of the sensor 402.

As shown in Figure 4, by shifting the center 404 of sensor 402, for example, along the y-direction, the detected field of view 414 of camera 400, as measured by sensor 402, changes. The detected field of view 414 is also schematically illustrated by two edge rays (shown in solid lines) passing through the center of imaging optics 406 and reaching the boundary of sensor 402. Due to the shift 432, the detected field of view 414 is asymmetrical relative to the optical axis 408 of camera 400.

The field of view of sensor 402 (e.g., the coverage angle of sensor 402) depends on the displacement 432 of the center 404 of sensor 402 along the y-direction. The field of view also depends on the distance 420 along the z-direction between imaging optics 406 and sensor 402. Distance 420 is a parameter designed for the device and is known. The dimensions of sensor 402 (e.g., along the y-direction and along the x-direction) are also known.

Figure 4 shows the camera 400 and its field of view 414 in the z-y plane. When the imaging optics 406 includes spherical optics (e.g., spherical lens, spherical mirror), the field of view 410 of the imaging optics 406 extends in the x-y plane and also has a width 418 along the x direction.

In some embodiments, in addition to shifting the center 404 of sensor 402 along the y-direction, the center 404 is also shifted along the x-direction (e.g., in an embodiment where the camera 400 is offset from the projector in the x-direction). For a sensor with a center shifted along both the x and y directions, the field of view of the image is asymmetrical about the optical axis 408 along the y-direction (as shown in FIG. 4). In some embodiments, the center 404 of sensor 402 is shifted only along the x-direction and not the y-direction. For a sensor with a center shifted only along the x-direction and not the y-direction, the field of view of the image is symmetrical about the y-direction but asymmetrical along the x-direction.

Figure 5 illustrates a 3D scanner 500 comprising a projector 502, a first camera 504, a second camera 506, and a third camera 508. The projector 502 includes an optical system 510 configured to direct light from a light source 518 onto an object scene 526. In some embodiments, light is projected through a patterned slide such that the light contains non-coded elements. In some embodiments, the non-coded pattern of light comprises dashed lines, curves, or an array of points having predefined or calibrated positions on the projected pattern. In some embodiments, a common property of the non-coded patterns is that the patterns have a set of simple elements (e.g., lines, dots, stripes) that are relatively small in size in at least a first direction. The individual features of these simple elements are not prominent enough to identify the elements from images acquired from a single camera that is not sufficiently close to the projector. Instead, in some embodiments, each non-coded element projected by the projector and reflected by a measuring object is identified by comparing the coordinates of lines perceived by each of the cameras (e.g., identifying its correspondence with the pattern of the projected slide). In some embodiments, a large offset between the cameras and between the cameras and the projector can result in better accuracy in identifying the correspondence between the imaged elements and the projected pattern (e.g., using the method described with respect to FIG8).

Optical system 510 has an associated optical axis 528. A first camera includes optical system 512 and optical sensor 520. Optical system 512 has an associated optical axis 530. A second camera includes optical system 514 and optical sensor 522. Optical system 514 has an associated optical axis 532. A third camera includes optical system 516 and optical sensor 524. Optical system 516 has an associated optical axis 534. Optical axis 528 is substantially parallel to optical axes 530, 532, and 534.

Figure 5 illustrates a projector 502, a first camera 504, a second camera 506, and a third camera 508 arranged in a substantially straight line along the y-direction. The field of view 538 of the first camera 504 is substantially symmetrical about the optical axis 530 because the center of the optical sensor 520 is not shifted relative to the geometric center of the first camera 504. The center of the optical sensor 522 is shifted to the right along the y-direction in Figure 5, resulting in an asymmetrical field of view 540 of the second camera 506 about the optical axis 532 in the y-direction. The center of the optical sensor 524 is shifted further to the right along the y-direction in Figure 5, resulting in an asymmetrical field of view 542 of the third camera 508 about the optical axis 534 in the y-direction. A slide (not shown) in the light source 518 is shifted to the left along the y-direction in Figure 5 to produce an asymmetrical field of view 536 of the projector 502 relative to the optical axis 528. With appropriate displacement sensors and projector 502, the field of view 536 of projector 502, the field of view 538 of first camera 504, the field of view 540 of second camera 506 and the field of view 542 of third camera 508 substantially overlap at object scene 526.

In some embodiments, the 3D scanner 500 includes one or more processors (e.g., processor 902 of FIG. 9) configured to receive data recorded by an optical sensor. In some embodiments, the data generated by the optical sensor is quite large. For example, some optical sensors provide more than 100 frames per second (fps), and for accuracy reasons, it is generally desirable to use the highest possible frame rate (e.g., processing more frames in the same scan time provides a sharper 3D model with better accuracy). As an example, three ultra-high definition (HD) cameras can produce a throughput of 30 gigabits per second. In some embodiments, to address this issue, one or more processors configured to receive data recorded by the optical sensor are field-programmable gate arrays (FPGAs) capable of having enormous data processing parallelism. However, a disadvantage of FPGAs is that they typically contain a small amount of single-board random access memory (RAM).

In some embodiments, to address this issue, the optical sensors are configured to sequentially provide data along a readout direction parallel to the sensor displacement direction (e.g., the y-direction). In some embodiments, the 3D scanner 500 includes an FPGA, on which data is read from each optical sensor (e.g., data from each sensor is read out onto a single common FPGA to avoid the need for inter-chip communication). At any given time during readout, for each optical sensor, the FPGA stores a portion of an image of the object scene, comprising the same subset (less than all) of elements projected onto the surface of the object (e.g., the same subset is simultaneously stored in the RAM of each FPGA). In some embodiments, while storing the image of the corresponding element from each optical sensor in the RAM of the FPGA, method 800 described below is performed for each corresponding element (e.g., by calculating a spatial point on the surface of the object corresponding to the element). In this way, method 800 is performed when readout occurs.

Figure 6 illustrates a 3D scanner 600 comprising a projector 602 and a single camera 604. The projector 602 includes an optical system 606 with an associated optical axis 608. The projector 602 also includes a slide 620. The center 622 of the slide 620 is shifted a distance 628 to the left of Figure 6 along the y-direction relative to the optical axis 608. This shift of the center 622 of the slide 620 produces an asymmetrical field of view 616 of the projector 602 relative to the optical axis 608.

A single camera 604 includes an optical system 610 with an associated optical axis 612. The single camera 604 also includes a sensor 624. The center 626 of the sensor 625 is shifted a distance 630 to the right of FIG. 6 along the y-direction relative to the optical axis 612. This shift of the center 626 of the sensor 624 produces an asymmetrical field of view 618 of the single camera 604 relative to the optical axis 612. The field of view 616 of the projector 602 and the field of view 618 of the single camera 604 overlap at a plane 614 in the object scene 611. Plane 614 is indicated by a line in the z-y plane. Plane 614 extends in the x-y plane. The optical axis 608 of the projector 602 is substantially parallel to the optical axis 612 of the single camera 604. In other words, the angle formed by the optical axis 608 and the optical axis 612 is close to zero.

Figure 7 illustrates a 3D scanner 700 comprising a projector 702, a first camera 704, and a second camera 706. The projector 702 includes an optical system 708 with an associated optical axis 730. The projector 702 also includes a slide 714. The center of the slide 714 is shifted a distance 736 to the left of Figure 7 along the y-direction relative to the optical axis 730. This shift of the center of the slide 714 produces an asymmetrical field of view 738 for the projector 702 relative to the optical axis 730.

The first camera 704 includes an optical system 710 having an associated optical axis 732. The first camera 704 also includes a sensor 716 having a center that is shifted to the right of FIG7 along the y-direction relative to the optical axis 732. The shift of the sensor 716 produces an asymmetrical field of view 740 of the first camera 704 relative to the optical axis 732.

The second camera 706 includes an optical system 712 having an associated optical axis 734. The second camera 706 also includes a sensor 718 having a center that is shifted relative to the optical axis 734 along the y-direction to the right of FIG7. The shift of the sensor 718 produces an asymmetrical field of view 742 of the second camera 706 relative to the optical axis 734.

In some embodiments, a projector 702, a first camera 704, and a second camera 706 are connected to an optical holder 720. In some embodiments, the projector 702, the first camera 704, and the second camera 706 are mounted on the optical holder 720. In some embodiments, the optical holder 720 is configured to mechanically couple the projector 702, the first camera 704, and the second camera 706. In some embodiments, a sensor 716 of the first camera 704 and a sensor 718 of the second camera 706 are supported on a common mechanical support 722. In some embodiments, the sensors 716 of the first camera 704 and the sensors 718 of the second camera 706 are manufactured (e.g., directly) on the mechanical support 722 (i.e., the mechanical support 722 is an integral structure). In some embodiments, the mechanical support 722 includes a printed circuit board (PCB). In some embodiments, the mechanical support 722 is flat. In some embodiments, the sensors 716 and 718 are therefore positioned in a common plane. In some embodiments, a connecting element 724 couples the mechanical support 722 to the optical holder 720. For example, a single mechanical support for the optics of different cameras is mechanically coupled to the projector 702, the first camera 704, and the second camera 706.

The field of view 738 of the projector 702, the field of view 740 of the first camera 704, and the field of view 742 of the second camera 706 overlap at plane 728 in the object scene 726. Plane 728 is indicated by a line in the z-y plane. Plane 718 extends in the x-y plane. The optical axis 730 of the projector 702 is parallel to the optical axis 732 of the first camera 704 and the optical axis 734 of the second camera 706.

Mounting sensors from different cameras on the same mechanical support (and/or manufacturing sensors directly on the mechanical support) simplifies sensor manufacturing and provides faster temperature stabilization between the first camera 704 and the second camera 706. For example, having a single mechanical support quickly stabilizes the temperature between sensors 716 and 718. When sensors are mounted on the same mechanical support, there is a smaller misalignment error associated with the camera's offset along the z-direction. In some embodiments, optical holder 720 secures and fixes optical systems 708, 710, and 712 to the same substrate. Fixing optical system 708 to the same substrate reduces misalignment between different optical systems in the z and x directions and allows for more accurate maintenance of their relative desired offset in the y-direction.

Monochrome sensors are typically used in 3D scanners to capture patterns of light generated by a projector's light source and reflected by the object being measured.

Monochrome sensors do not include color filters. 3D scanners typically have separate color sensors for capturing texture information of an object. In some embodiments, texture information includes color information and, optionally, one or more non-color properties of the object's appearance (e.g., specularity). For example, a color optical sensor includes one or more built-in filters that detect the color and brightness of incident light. In some embodiments, these two functionalities (e.g., texture sensing and mapping the 3D contours of an object) are combined in a single color camera (e.g., a first camera 504 having a color sensor configured to receive reflected light of different wavelengths from the object). A frame from the color camera captures an image of the pattern of projected light reflected from the measured object, and subsequent frames capture either the object's color or texture information. In some embodiments, the color optical sensor has lower sensitivity due to the presence of color filters. In some embodiments, the remaining cameras (e.g., cameras 506 and 508) are monochrome cameras.

Figure 8 illustrates a flowchart of a method 800 for generating a 3D model of an object according to some embodiments. In some embodiments, method 800 is performed at a 3D scanner (e.g., any of the 3D scanners described in the preceding figures). In some embodiments, certain operations of method 800 are performed by a computer system other than the 3D scanner (e.g., a computer system that receives and processes data from the 3D scanner, such as the remote device 936 of Figure 9). Some operations in method 800 are optionally combined, and/or the order of some operations is optionally changed. For ease of explanation, method 800 is described as being performed by a 3D scanner 700 (Figure 7).

Method 800 includes projecting (802) a plurality of elements onto the surface of an object (e.g., using a projector 702). In some embodiments, the elements are non-coded elements. In some embodiments, the elements are lines.

Method 800 includes capturing (804) a first image of a portion of multiple lines reflected from an object using a first camera (e.g., first camera 704 of FIG. 7). The portions of the multiple lines captured in the first image are those lines within the field of view of the first camera.

Method 800 includes capturing (806) a second image of a portion of multiple lines reflected from an object using a second camera (e.g., second camera 706 of FIG. 7). The portions of the multiple lines captured in the second image are those lines within the field of view of the second camera.

In some embodiments, the first image and the second image are captured substantially simultaneously. In some embodiments, the first image and the second image are captured while multiple elements are projected onto the surface of an object.

In some embodiments, one of the first camera and the second camera is a color (e.g., RGB) camera. In some embodiments, the other of the first camera and the second camera is a monochrome camera. In some embodiments, multiple elements are projected (e.g., illuminated) onto the surface of an object in a stroboscopic manner. Images obtained while the multiple elements are projected onto the surface of the object are used for 3D reconstruction of the object's surface (as described below). Additionally, in some embodiments, the color camera also obtains images when the multiple elements are not projected onto the surface of the object. In some embodiments, the color camera obtains images between the stroboscopic projections of the projector. Therefore, in some embodiments, the color image of the object does not contain reflections of the multiple non-coded elements. In some embodiments, images obtained when the multiple elements are not projected onto the surface of the object are used to generate textures for 3D reconstruction of the object's surface.

Method 800 includes calculating (808) a plurality of possible spatial points on the surface of the object using image points on a first image corresponding to corresponding lines of a plurality of lines projected onto the surface of the object. In some embodiments, calculating the plurality of possible spatial points includes calculating the z-components of the plurality of possible spatial points. In some embodiments, each possible spatial point on the surface of the object is calculated by assuming a correspondence between imaging elements in the first image and elements in a projected pattern on a slide. Once the correspondence is known, spatial points on the surface of the object can be determined by utilizing a triangulation algorithm, taking into account the geometry of the 3D scanner. As used herein, triangulation is the determination of the location of points on the surface of the object using the known locations of elements projected onto the surface of the object within an image, together with knowledge of the scanner geometry.

Some possible (e.g., hypothetical) spatial points may be located outside the depth of field (e.g., focus depth) of the 3D scanner 700 (e.g., outside region 314 of FIG. 3). Therefore, method 800 includes rejecting spatial points outside the depth of field of the first camera from a plurality of possible spatial points, wherein the rejection produces a set of remaining spatial points.

Method 800 includes determining (810) whether a corresponding spatial point in the remaining set of spatial points corresponds to an imaged line from the second image (e.g., mapping points in the first image to points in the second image using the geometry of the 3D scanner 700 and hypothetical spatial points on the surface of the object).

Method 800 further includes storing (814) the corresponding spatial point as the corresponding position on the surface of the object, based on the determination that the corresponding spatial point corresponds to an imaged line in the second image. For example, the fact that the hypothetical spatial coordinates of the line in the first image are mapped to the line in the second image confirms that the hypothesis is correct. In some embodiments, additional images may be needed to eliminate ambiguity in the correspondence between elements in the first image and elements in the projected pattern. Therefore, in some embodiments, additional cameras are used to acquire additional images (e.g., Figure 5 shows a scanner with three cameras).

In some embodiments, method 800 includes removing a corresponding spatial point from memory, which is a possible corresponding location on the surface of an object, based on the determination that the corresponding spatial point does not correspond to an imaged non-coded element in a third image. In some embodiments, method 800 includes eliminating hypothetical correspondences between imaged elements and elements of a projected pattern from a set of possibilities.

It should be understood that the specific order of operations described in Figure 8 is merely an example and is not intended to indicate that the described order is the only possible order in which the operations can be performed. Those skilled in the art will recognize various ways to reorder the operations described herein.

Figure 9 is a block diagram of a 3D scanner 900 according to some embodiments. Note that a 3D scanner described with reference to any of the preceding figures may include any of the features described with respect to 3D scanner 900. For example, any of the 3D scanners described with reference to the preceding figures may include a computer system as described with respect to 3D scanner 900. 3D scanner 900 or the computer system of 3D scanner 900 typically includes a memory 904, one or more processors 902, a power supply 906, a user input/output (I/O) subsystem 908, one or more sensors 903 (e.g., cameras), one or more light sources 311 (e.g., projectors), and a communication bus 910 for interconnecting these components. Processor 902 executes modules, programs, and/or instructions stored in memory 904 and thereby performs processing operations.

In some embodiments, processor 902 includes at least one central processing unit. In some embodiments, processor 902 includes at least one graphics processing unit. In some embodiments, processor 902 includes at least one field-programmable gate array (FPGA).

In some embodiments, memory 904 stores one or more programs (e.g., instruction sets) and/or data structures. In some embodiments, memory 904 or a non-transitory computer-readable storage medium of memory 904 stores the following programs, modules, and data structures, or subsets or supersets thereof:

• Operating System 912, which contains programs for handling various basic system services and for performing hardware-related tasks;

• Network communication module 918, which is used to connect the 3D scanner to other computer systems (e.g., remote device 936) via one or more communication networks 950;

• User interface module 920, which receives commands and/or input from the user via user input/output (I/O) subsystem 908, and provides output for presentation and/or display on user input/output (I/O) subsystem 908;

• Data processing module 924, which is used to process or preprocess data from sensor 903, includes optionally performing any or all of the operations described with respect to method 800 (FIG. 8);

• Data acquisition module 926, which controls the readout of the camera, projector, and sensors; and

• Storage device 930, which includes buffers, RAM, ROM and/or other memory for storing data used and generated by 3D scanner 900.

The modules identified above (e.g., data structures and/or programs containing instruction sets) do not need to be implemented as separate software programs, programs, or modules, and therefore can be combined or otherwise rearranged in various embodiments in various subsets of these modules. In some embodiments, memory 904 stores a subset of the modules identified above. Furthermore, memory 904 may store additional modules not described above. In some embodiments, the modules stored in memory 904 or the non-transitory computer-readable storage medium of memory 904 provide instructions for implementing the corresponding operations in the methods described below. In some embodiments, some or all of these modules may be implemented using dedicated hardware circuitry (e.g., an FPGA) containing some or all of the module functionality. One or more of the elements identified above may be executed by one or more processors 902.

In some embodiments, the user input/output (I/O) subsystem 908 communicatively couples the 3D scanner 900 to one or more devices, such as one or more remote devices 936 (e.g., external displays), via a communication network 950 and/or via wired and/or wireless connections. In some embodiments, the communication network 950 is the Internet. In some embodiments, the user input/output (I/O) subsystem 908 may communicatively couple the 3D scanner 900 to one or more integrated or peripheral devices, such as touch-sensitive displays.

In some embodiments, sensor 903 includes a first optical sensor (e.g., a CCD) (e.g., sensor 716 of FIG. 7) for collecting texture (e.g., color data), a second optical sensor (e.g., a CCD) (e.g., sensor 718) for collecting 3D data, and a motion sensor (e.g., a 9-DOF sensor, which may be implemented using microelectromechanical systems (MEMS), a gyroscope, and one or more Hall sensors). In some embodiments, in addition to collecting texture, the first optical sensor also collects 3D data.

In some embodiments, the light source 911 (e.g., a component of a projector described herein) includes one or more lasers. In some embodiments, the one or more lasers include vertical-cavity surface-emitting lasers (VCSELs). In some embodiments, the light source 911 also includes an array of light-emitting diodes (LEDs) that generate visible light.

The communication bus 910 optionally includes a circuit system (sometimes called a chipset) that interconnects and controls communication between system components.

For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise forms disclosed. In view of the foregoing teachings, many modifications and variations are possible. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to best use the invention and the various described embodiments with various modifications suitable for the particular intended use.

It will also be understood that although the terms first, second, etc., are used in some instances herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the various described embodiments, a first sensor may be referred to as a second sensor, and similarly, a second sensor may be referred to as a first sensor. Both the first sensor and the second sensor are sensors, but they are not the same sensors unless the context clearly indicates otherwise.

The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and in the appended claims, the singular forms “a,” “an,” and “described” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and/or,” as used herein, refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “includes,” “including,” “comprises,” and/or “comprising,” when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and/or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, depending on the context, the term "if" is optionally interpreted as meaning "when," "upon," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if [the stated condition or event] is detected" is optionally interpreted as meaning "when determination," "in response to determination," "when [the stated condition or event] is detected," or "in response to the detection of [the stated condition or event]."

Claims (28)

1. A system comprising: A projector configured to project a plurality of non-coded elements onto an object, the projector having a first optical axis; A first camera includes a first lens and a first sensor, wherein the first lens focuses reflections of a first portion of the plurality of non-coded elements onto the first sensor, and wherein the first lens defines a second optical axis; and A second camera includes a second lens and a second sensor, wherein the second lens focuses reflections from a second portion of the plurality of non-coded elements onto the second sensor, and wherein the second lens defines a third optical axis. in: The projector, the first camera, and the second camera are offset from each other in a first direction. The first optical axis is substantially parallel to the second optical axis, and the second optical axis is substantially parallel to the third optical axis. The center of the first sensor shifts away from the second optical axis along the first direction, and The center of the second sensor is shifted away from the third optical axis along the first direction. 2. The system of claim 1, wherein the projector, the first camera, and the second camera are arranged substantially in a straight line in the first direction. 3. The system of claim 2, wherein the projector is positioned at a first end of the substantially straight line, and wherein the first camera and the second camera are positioned on one side of the projector along the first direction on the substantially straight line. 4. The system according to claim 1, wherein the depth of field of the first camera is substantially the same as the depth of field of the second camera. 5. The system of claim 1, wherein the center of the second sensor, shifted along the first direction, provides an asymmetrical field of view of the object from the second camera relative to the third optical axis. 6. The system of claim 5, wherein the field of view of the projector, the field of view of the object from the first camera, and the field of view of the object from the second camera through which the plurality of non-coded elements are projected onto the object are configured to overlap. 7. The system of claim 6, wherein the field of view of the object from the first camera is greater than the field of view of the projector. 8. The system according to claim 2, further comprising: The third camera has a third lens and a third sensor, wherein: The third lens focuses the reflections of the third portion of the plurality of non-coded elements onto the third sensor. The third lens defines the fourth optical axis. The projector is positioned at the first end of the substantially straight line. The first camera, the second camera, and the third camera are positioned on a substantially straight line along the first direction on one side of the projector. The center of the second sensor, shifted along the first direction, provides an asymmetrical field of view of the object from the second camera relative to the third optical axis, and The fields of view of the projector, the field of view of the object from the first camera, and the field of view of the object from the second camera, through which the plurality of non-coded elements are projected onto the object, are configured to overlap. 9. The system of claim 8, wherein the field of view of the object from the third camera is smaller than the field of view of the object from the second camera. 10. The system of claim 1, wherein the projector includes a slide displaced away from the first optical axis along the first direction, and the field of view of the projector through which the plurality of non-coded elements are projected onto the object is asymmetrical relative to the first optical axis. 11. The system of claim 1, further comprising a mechanical support for mechanically coupling the first camera to the second camera. 12. The system of claim 11, wherein the mechanical support comprises a printed circuit board on which the first sensor and the second sensor are mounted. 13. The system of claim 11, wherein the mechanical support is configured to stabilize the temperature between the first sensor and the second sensor. 14. The system of claim 11, further comprising an optical holder configured to mechanically couple the projector, the first camera, and the second camera. 15. The system of claim 1, wherein the first sensor comprises a color sensor configured to receive reflected light of different wavelengths from the object. 16. The system of claim 15, wherein the reflected light from different wavelengths of the object provides texture information of the object. 17. The system of claim 15, further comprising one or more processors configured to receive a first image from the first sensor and a subsequent second image from the first sensor, wherein the first image includes the reflection of the first portion of the plurality of non-coded elements, and the subsequent second image includes a color image of the object that does not include the reflection of the plurality of non-coded elements. 18. The system of claim 1, further comprising one or more processors configured to receive data recorded by the first sensor and data recorded by the second sensor, wherein the first sensor and the second sensor are configured to be sequentially provided to the one or more processors along a readout direction. 19. The system of claim 18, wherein the readout direction of the first camera is parallel to the first direction. 20. The system of claim 19, wherein the one or more processors include a common processor that receives the data from the first sensor and the second sensor. 21. The system of claim 20, wherein the common processor is a field-programmable gate array (FPGA). 22. The system of claim 1, further comprising one or more processors and a memory, the memory storing instructions for the following operations: Multiple possible spatial points on the surface of the object are calculated using image points from a first image from the first camera that correspond to the corresponding non-coded elements among the plurality of non-coded elements projected onto the surface of the object. Reject spatial points outside the depth of field of the first camera from the plurality of possible spatial points, wherein the rejection produces a set of remaining spatial points; For a corresponding spatial point in the set of remaining spatial points, determine whether the corresponding spatial point corresponds to an imaged non-coded element in the second image from the second camera; and Based on the determination that the corresponding spatial point corresponds to an imaged non-coded element in the second image, the corresponding spatial point is stored as the corresponding position on the surface of the object. 23. The system of claim 22, wherein the memory further stores instructions for the following operations: Determine whether the corresponding spatial point corresponds to an imaged non-coded element in a third image from a third camera; and Based on the determination that the corresponding spatial point corresponds to an imaged non-coded element in the third image, the corresponding spatial point is stored as the corresponding position on the surface of the object. 24. The system of claim 23, wherein the instructions further comprise instructions for the following operations: Based on the determination that the corresponding spatial point does not correspond to an imaged non-coded element in the third image, the corresponding spatial point, which is a possible corresponding position on the surface of the object, is removed from the memory. 25. The system of claim 22, wherein determining whether the corresponding spatial point corresponds to an imaged non-coded element in the second image comprises calculating an image point in the second image corresponding to the corresponding spatial point, and identifying whether a portion of the non-coded element is imaged at the image point. 26. The system of claim 1, wherein the non-coded element is a line. 27. A system comprising: A projector configured to project a plurality of non-coded elements onto an object, the projector having a first optical axis; A first camera includes a first lens and a first sensor, wherein the first lens focuses reflections of a first portion of the plurality of non-coded elements onto the first sensor, and wherein the first lens defines a second optical axis; and A second camera includes a second lens and a second sensor, wherein the second lens focuses reflections from a second portion of the plurality of non-coded elements onto the second sensor, and wherein the second lens defines a third optical axis. in: The first sensor and the second sensor are mounted on a common plane; The second optical axis is substantially parallel to the third optical axis. The center of the first sensor is shifted away from the second optical axis along the first direction. 28. A system comprising: A projector configured to project a plurality of non-coded elements onto an object, the projector having a first optical axis; and A first camera includes a first lens and a first sensor, wherein the first lens focuses reflections from a first portion of the plurality of non-coded elements onto the first sensor, and wherein the first lens defines a second optical axis. The center of the first sensor is shifted away from the second optical axis along the first direction.