CN101846798B - Method and device for obtaining scene depth information - Google Patents

Abstract

The invention relates to a method and a device for obtaining depth information of an scenery, which improve the similarity of point spread functions under different field conditions through an optical device, and the point spread functions on an optical axis of the optical device still have difference, so that the depth information of the scenery can be obtained through the change situation of the point spread functions on the optical axis at different object distances.

Description

Method and device for obtaining scene depth information

technical field

The invention relates to an acquisition of scene depth information, in particular to a method and device for obtaining scene depth information. the

Background technique

How to capture the distance or depth information of different objects in the actual scene has always been an important issue in the field of stereoscopic display, ranging system or depth profile display system. Traditional capture methods and devices can be divided into two types: "active" and "passive". In the "passive" system, the concept of "parallax" is used, and its method mainly uses dual-lens, multi-lens, and porous aperture imaging systems of similar concepts. However, such systems require multiple imaging lenses and multiple image sensors to obtain stereoscopic image information. The "active type" uses an additional active emission signal source (usually a light source), and calculates the distance of different objects according to the difference in time of flight (time of flight) after the signal source illuminates the object to be photographed A specific form of distribution fringe is projected onto the object to be photographed, and the degree of deformation of the fringe on the surface of the object is used to evaluate the relative distance between the objects to be photographed. the

In addition, U.S. Patent No. 5,521,695 discloses that through a specially designed optical component, an object point will form four image points on its imaging surface after passing through the optical component, and the relative position changes of the four image points can be used to calculate the The relative distance of the photographed object. the

On the other hand, in the imaging system, the point spread function (Point Spread Function: PSF) of the lens will change with the object distance. Therefore, the object distance information can be obtained according to the characteristics of the point spread function. However, in addition to changing the point spread function of the lens with different object distances, the point spread function of the lens will also change under the same object distance but different field of view. Therefore, in order to obtain the distance difference and depth information between the objects to be photographed, the influence of the point spread function along with the object distance and the field of view must be considered at the same time in the back-end image processing. the

Invention content

The invention provides a method for obtaining depth information of a scene and an optical obtaining device thereof, which improves the similarity of point spread functions of different viewing field conditions through a device with a coding component, and the point spread along the optical axis of the optical device The function is still different, and then the depth information of the scene can be obtained through the variation of the on-axis point spread function at different object distances. the

The present invention provides an optical acquisition device for scene depth information, which includes an optical component and a coding component. Wherein, the encoding component is placed on the path of the scene light passing through the optical component to adjust the point spread function of the optical acquisition device. the

The present invention proposes a method for obtaining depth information of a scene, which includes: obtaining an image through an optical obtaining device; obtaining at least one point spread function information of the optical obtaining device; and scanning different regions of the image and executing according to the point spread function information. Restore the comparison program. the

Description of drawings

Fig. 1 shows the block diagram of the scene depth information acquisition device of an embodiment of the present invention;

Figure 2A shows the speckle diagrams of the point spread functions of different fields of view when an optical component is used to image a scene whose object distance is 1000mm;

Figure 2B shows the speckle diagrams of the point spread functions of different fields of view when the depth information acquisition device is used to image a scene with an object distance of 1000mm;

Figure 3A shows the speckle diagrams of the point spread functions of different fields of view when an optical component is used to image a scene whose object distance is 790mm;

Figure 3B shows the speckle diagrams of the point spread functions of different fields of view when the depth information acquisition device is used to image a scene with an object distance of 790mm;

Figure 4A shows the speckle diagrams of the point spread functions of different fields of view when the optical component is used to image a scene whose object distance is 513 mm;

Figure 4B shows the speckle diagrams of the point spread functions of different fields of view when the depth information acquisition device is used to image a scene with an object distance of 513 mm;

Fig. 5 shows the speckle diagram of the point spread function at different image height positions when using the depth information acquisition device (without using the xy coupling term) to image different object distance scenes;

Fig. 6 shows the speckle diagram of the point spread function at different image height positions when using the depth information acquisition device (using the xy coupling term) to image different object distance scenes;

Figure 7 shows the similarity comparison diagram of the point spread function at the same object distance but in different fields of view;

Figure 8 shows the similarity comparison diagram of the point spread function on the optical axis of the depth information acquisition device under different object distances;

Fig. 9 shows the flow chart of the steps of the depth information acquisition method of the embodiment of the present invention;

Figure 10 shows a schematic diagram of scanning;

Figure 11A shows a restored image with an object distance of 1000 mm;

Figure 11B shows a restored image with an object distance of 980mm;

Figure 11C shows a restored image with an object distance of 900mm;

Figure 11D shows a restored image with an object distance of 790 mm; and

FIG. 11E shows a restored image with an object distance of 513 mm. the

Description of main component symbols

100 A device for acquiring scene depth information

101 Optical components

102 Encoding components

S901-S905 steps

Detailed ways

FIG. 1 is a block diagram of an apparatus for obtaining depth information according to an embodiment of the present invention. The incident scene light is received by the sensor after passing through the depth information obtaining device 100 . The depth information obtaining device 100 includes an optical component 101 and a coding component 102 . The optical component 101 for imaging can be a single lens, a lens group, or a reflective imaging mirror group. The encoding component 102 can be a wavefront phase encoding component, a wavefront amplitude encoding component or a wavefront phase and amplitude hybrid encoding component, wherein, when the encoding component 102 is a wavefront phase encoding component, its encoding method can be a shaft Symmetrical encoding. The wavefront encoding of the encoding component 102 can be represented by a mutually orthogonal coordinate system. In the present embodiment, the wavefront coding of coding assembly 102 is represented by the following equation:

$\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; E.</span>}}{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; nx</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; lmxy</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; ,</span>$

Where x and y are the coordinate positions of the encoding component in the x-axis direction and y-axis direction respectively, n is a positive integer, l+m is an even number, A _nx and _Any are the nth item in the x-axis direction and y-axis direction respectively The size of the coefficient on A _lmxy is the size of the coefficient of the xy coupling term.

The encoding component 102 can be placed at the aperture of the depth information acquisition device 100, near the aperture, the exit pupil surface, near the exit pupil surface, the entrance pupil surface or near the entrance pupil surface. The coding component 102 can also be combined with the optical component 101 , for example, the coding component 102 can be made on the surface of the lens in the optical component 101 . The combined wavefront code can be expressed by the following equation:

W'(x,y)=W(x,y)+W ₀ (x,y)

Where W'(x, y) is the wavefront after the depth information acquisition device 100 adds the encoding component 102, W ₀ (x, y) is the wavefront without the depth information acquisition device 100 adding the encoding component 102. Those skilled in the art understand that the above W′(x, y), W(x, y) and W ₀ (x, y) can also be represented by Zernike polynomials. The encoding component 102 is placed in such a way that the wavefront of the captured scene light passing through the depth information acquisition device 100 is mainly composed of W(x, y) after being added to the encoding component. In addition, the encoding element 102 can be a refractive element or a diffractive element or an element having both optical properties above.

According to an embodiment of the present invention, the effective focal length of the optical component 101 is 10.82 mm, F# is 5, and the full viewing angle is 10.54 degrees. The diagonal length of the sensor 103 is 2mm. The phase encoding component 102 performs encoding using the above wavefront equation, where n=2, A _2x =A _2y =22.8PI, A _1mxy =0.

Fig. 2A shows when the optical component 101 is used to image a scene with an object distance of 1000mm, the point spread functions of the scene light with a wavelength of 656.3nm, a wavelength of 587.6nm and a wavelength of 486.1nm expressed as red, green and blue light bands in different fields of view Spot diagram. FIG. 2B shows the speckle diagrams of PSFs of different fields of view when the object distance is 1000 mm using the optical component 101 combined with the phase encoding component 102 to image the scene with the depth information acquisition device 100 . the

FIG. 3A shows the point spread function of the scene light with a wavelength of 656.3 nm, a wavelength of 587.6 nm and a wavelength of 486.1 nm expressed as red, green and blue light bands in different fields of view when the optical component 101 is used to image a scene with an object distance of 790 mm. dot plot. 3B shows the speckle diagrams of PSFs of different fields of view when the depth information acquisition device 100 using the optical component 101 combined with the phase encoding component 102 is used to image a scene with an object distance of 790 mm. the

FIG. 4A shows the point spread function of the scene light with a wavelength of 656.3nm, a wavelength of 587.6nm and a wavelength of 486.1nm expressed as red, green and blue light bands in different fields of view when the optical component 101 is used to image a scene with an object distance of 513mm. dot plot. Fig. 4B shows that when the depth information acquisition device 100 using the optical component 101 combined with the phase encoding component 102 is used to image a scene with an object distance of 513mm, the wavelengths of 656.3nm, 587.6nm and 486.1nm expressed as red, green and blue light bands Speckle plots of the point spread function of scene light in different fields of view. the

According to another embodiment of the present invention, the effective focal length of the optical assembly 101 is 10.82 mm, F# is 5, and the full viewing angle is 10.54 degrees. The diagonal length of the sensor 103 is 2 mm. The phase encoding component 102 performs encoding using the above-mentioned wavefront equation, where n=3, A _3x =A _3y =12.7PI, A _lmxy =0. According to another embodiment of the present invention, FIG. 5 shows the depth information acquisition device 100 using the optical component 101 combined with the phase encoding component 102 to image scenes with different object distances (513mm, 790mm and 1000mm), expressed as red, green and Speckle diagrams of the point spread function of the scene light with a wavelength of 656.3nm, a wavelength of 587.6nm and a wavelength of 486.1nm in the blue light band at different fields of view (that is, the image height positions are 0mm, 0.7mm and 1mm).

According to yet another embodiment of the present invention, the effective focal length of the optical assembly 101 is 10.82 mm, F# is 5, and the full viewing angle is 10.54 degrees. The diagonal length of the sensor 103 is 2mm. The phase encoding component 102 performs encoding using the above wavefront equation, where n=2, A _2x =A _2y =19.1PI, A _22xy =9.55PI. According to yet another embodiment of the present invention, FIG. 6 shows the depth information acquisition device 100 using the optical component 101 combined with the phase encoding component 102 to image scenes with different object distances (513mm, 790mm and 1000mm), expressed as red, green and The speckle diagrams of the point spread function of the scene light with a wavelength of 656.3nm, a wavelength of 587.6nm and a wavelength of 486.1nm in the blue light band at different fields of view (that is, the image height positions are 0mm, 0.7mm and 1mm).

Compared with the point spread function using only the optical component 101, it can be seen from Fig. 2A to Fig. 6 that the appearance of the point spread function of the depth information acquisition device 100 according to the embodiment of the present invention is under the same object distance but different field of view. The degree of variation in shape is fairly small. In order to further confirm the degree of improvement of the similarity of the point spread function, the Hilbert space angle (Hilbert space angle) is used to calculate the similarity of the point spread function. Figure 7 shows the similarity comparison diagram of the point spread function at the same object distance but in different fields of view (the reference object distance is 790mm). Figure 8 shows the similarity comparison diagram of the point spread function along the optical axis under different object distances. It can be seen from FIG. 7 that the similarity of the point spread function of the depth information acquisition device 100 of the embodiment of the present invention is improved under different fields of view (in the Hilbert space angle, the smaller the calculated value is, the smaller the similarity is). higher). On the other hand, it can be seen from FIG. 8 that the point spread function on the optical axis of the depth information acquisition device 100 still has differences at different object distances. the

In addition, according to the wavelength band of the scene light when the optical component 101 or the depth information acquisition device 100 actually works, the phase encoding component 102 combined with the optical component 101 can be designed according to the aforementioned wavefront encoding equation, and is not limited to wavelengths of 656.3 nm and 587.6 nm. nm and the scene light band with a wavelength of 486.1nm. the

In order to enable those skilled in the art to implement the present invention through the teaching of this embodiment, another method embodiment is proposed below in combination with the above-mentioned device for obtaining scene depth information. the

FIG. 9 is a flowchart of steps of a method for obtaining depth information according to an embodiment of the present invention. In step S901 , the obtaining device 100 is used to obtain point spread function information of different object distances. Such point spread function information can be obtained by measuring the device 100 or according to design parameters of the device 100 under different object distances. In step S902, the point spread function information is stored. On the other hand, in step S903 the obtaining device 100 obtains an image including depth information. Next, in step S904, the stored point spread function information is used to perform scanning and restoration comparison procedures for different regions of the image. Fig. 10 shows a schematic diagram of scanning. A filter kernel (filterkernel) 104 moves in the X direction and the Y direction, so as to perform scanning and restoration comparison procedures on the scenes 101, 102 and 103 in different regions and different object distances in the image. In the restore comparison program, the Wiener filter or direct inverse operation can be used to restore the image. After the images in the area are restored, the mean square error (MSE) of the restored images is calculated respectively and compared with the threshold value preset by the user. In step S905, the object distance of the image can be determined according to the comparison result, and then the depth information thereof can be obtained. In addition, in step S904 , the quality of the reconstructed image may also be evaluated by determining whether the boundaries and edges are clear, and then in step S905 , the object distance of the image is determined and the depth information thereof is obtained. the

According to an embodiment of the present invention, the shooting range of the scene is between 513mm and 1000mm, and at the same time there is a Lena scene at 513, 790, 900, 980 and 1000mm, these scenes are also placed in the different perspectives. FIG. 11A shows a restored image with an object distance of 1000 mm. FIG. 11B shows a restored image with an object distance of 980 mm. FIG. 11C shows a restored image with an object distance of 900 mm. FIG. 11D shows a restored image with an object distance of 790 mm. FIG. 11E shows a restored image with an object distance of 513 mm. Because the point spread function information when the object distance is 1000mm is used to perform the scanning and restoration comparison procedures, compared with images with other object distances, the restoration effect of the object distance is 1000mm is the best, and its root mean square error (MSE) is 2.4×10 ^-7 . Therefore, in step S905 , it can be known that the depth information of the image is 1000 mm.

In the prior art system, the impact of the point spread function along with the object distance and field of view needs to be considered at the same time during the back-end image processing. In the embodiment of the present invention, the obtaining device 100 with the encoding component 102 improves the similarity of the point spread function of different field of view conditions, and maintains the difference of the point spread function on the axis, and then through the point spread function on the axis at different object distances The change situation of the scene gets the depth information of the scene. The technical content and technical features of the present invention have been disclosed above, but those skilled in the art may still make various substitutions and modifications based on the teaching and disclosure of the present invention without departing from the spirit of the present invention. Therefore, the protection scope of the present invention should not be limited to those disclosed in the embodiments, but should include various replacements and modifications that do not depart from the present invention, and are covered by the scope of the patent application. the

Claims (10)

1. A device for acquiring scene depth information, comprising: an optical assembly; and An encoding component, wherein the encoding component is placed on the path of the scene light passing through the optical component to adjust the point spread function of the optical component, the coding component is an axisymmetric coding component, and the coding of the coding component uses mutual Orthogonal coordinate system representation, and represented by the equation W(x, y)=∑A _nx x ²ⁿ +A _ny y ²ⁿ +A _lmxy x ¹ y ^m , where x and y are the x-axis direction and y of the encoding component respectively The coordinate position in the axial direction, n is a positive integer, 1+m is an even number, A _nx and _Any are the coefficient sizes of the nth item in the x-axis direction and the y-axis direction, Al _lmxy is the coefficient size of the xy coupling item , or the equation W(x, y) is represented by a nononic polynomial, where the encoding component is a wavefront phase encoding component; Scanning the area where the image is obtained, and performing a restoration comparison procedure according to the point spread function information; and Obtain a depth information according to the result of the reduction comparison program. 2. The device according to claim 1, wherein the optical component is a single lens, a lens group or a reflective imaging mirror group. 3. The device according to claim 1, wherein the encoding component is placed near the aperture, the exit pupil plane, near the exit pupil plane, the entrance pupil plane or near the entrance pupil plane of the device. 4. The device according to claim 1, wherein the encoding element is a refractive element or a diffractive element or an element having both optical properties. 5. The device of claim 1, wherein the encoding component is integrated with the optical component. 6. A method for obtaining depth information of a scene, comprising: obtaining an image through an optical device; obtaining at least one point of spread function information of the optical device; Scanning the area where the image is obtained, and performing a restoration comparison procedure according to the point spread function information; and obtaining a depth information according to the result of the reduction comparison program; The optical device has an encoding component, which is an axisymmetric encoding component, and the encoding of the encoding component is expressed by a mutually orthogonal coordinate system, and is represented by the equation W(x, y)=∑A _nx x ²ⁿ +A _ny y ²ⁿ +A _lmxy x ¹ y ^m , where x and y are the coordinate positions of the encoding component in the x-axis direction and y-axis direction, n is a positive integer, 1+m is an even number, A _nx and _{Any are} respectively The magnitude of the coefficient of the n term in the x-axis direction and the y-axis direction, A _lmxy is the coefficient magnitude of the xy coupling term, or the equation W(x, y) is represented by a nononic polynomial, wherein the encoding component is a wavefront phase encoding component. 7. The method according to claim 6, wherein the point spread function information is obtained by measuring the optical device or according to design parameters of the optical device. 8. The method of claim 6, wherein the uncomparison procedure uses a Wiener filter or a direct inverse operation. 9. The method according to claim 6, wherein the back-to-back comparison procedure comprises a root mean square difference operation. 10. The method according to claim 9, wherein the restore comparison procedure further comprises comparing the mean square error calculation result with a user-set threshold.

Images