TWI884034B - Panorama-based three-dimensional scene reconstruction system and method therefor - Google Patents

Abstract

A panorama-based three-dimensional scene reconstruction system and a method therefor are provided. Multiple images in a scene are obtained by an image-retrieving device. An image-processing device is used to perform following steps, including stitching the multiple images for forming a panorama of the scene, performing semantic segmentation and depth prediction on the panoramas over paths in the scene, and afterwards obtaining a point cloud for each path of the scene and depths of the path based on depth prediction. Point cloud alignment is performed on all of the point clouds and the point cloud of each path is projected onto a planar image. A contour edge of each path is obtained based on the planar image. A triangulation algorithm is performed on the contour edge for obtaining a planar path mesh. A three-dimensional mesh for the scene is established based on walls and/or ceilings along the planar path mesh.

Description

System and method for reconstructing stereoscopic scenes based on panoramas

The specification discloses a technology for establishing a stereo grid, particularly a system and method for reconstructing a stereo grid from a continuous panorama and allowing a user to move along a shooting path to reconstruct a stereo scene based on a panorama.

3D Mesh Reconstruction technology is used to reconstruct meshes of indoor or outdoor scenes. In most methods, various sensors of a scanner are used to scan a specific scene environment to construct a point cloud describing the scene and the objects therein. The point cloud is then reconstructed into a corresponding 3D mesh through a mesh reconstruction algorithm. The reconstructed 3D mesh provides related applications of virtual reality (VR) or augmented reality (AR), such as guide systems or map navigation.

The main process of the 3D mesh reconstruction technology includes first scanning the environment image with a scanner to obtain images of various azimuth angles in the environment, forming a point cloud that describes multiple data points in a space. The point cloud is a discrete set of data points formed by multiple data points, where each data point includes the coordinate description and image data of each point in the space, and may include depth information. Then, mesh reconstruction is performed based on the point cloud information, in which an algorithm is used to reconstruct the point cloud into a mesh, and a virtual reality (VR) or augmented reality (AR) can be established with the image data.

For example, Matterport is a known technology for creating stereoscopic images, which is suitable for reconstructing indoor or outdoor scenes. In the process of reconstructing stereoscopic images, a scanner (mobile phone, 360-degree camera, or other) is needed to scan the environment and thereby create an environmental image of each shooting point. Because Matterport technology shoots at fixed points, users need to select a "fixed site" in a scene to scan the environment at each fixed site and obtain a panorama of each site. The number of sites that users can move in a stereoscopic scene actually depends on how much data the user has shot. Ultimately, a three-dimensional model of the scene can be formed, which can be used in virtual reality or augmented reality applications.

After scanning multiple sites in a stereoscopic scene, users can browse in the stereoscopic space described by the established stereoscopic model. Starting from the initial site, users can move towards the next target site in the stereoscopic space. Matterport technology uses the panorama corresponding to the two sites to switch between different sites in the stereoscopic model. However, because the panorama corresponding to the two sites is switched immediately, users will experience a discontinuous image switching process when moving between different sites, and various images will also be distorted.

In order to improve the problem that the stereoscopic images created by the known technology using the panorama of multiple discrete stations in a scene will cause discontinuous image switching when the user switches between different stations, the disclosure document proposes a system and method for reconstructing a stereoscopic scene based on a panorama, which can solve the problems of discontinuous image switching and image distortion between different stations, so as to provide users with free movement in the stereoscopic space and improve the user's experience in browsing images in the stereoscopic space.

According to an embodiment of a method for reconstructing a stereoscopic scene based on a panorama, in the method, multiple images in a scene are first obtained, and the multiple images are stitched together to obtain a panorama of the scene. Semantic segmentation and depth prediction can be performed on all panoramas on the paths in the scene. Then, the point clouds of each path obtained based on the path pixels in the scene and the path depth obtained based on the depth prediction are obtained, and point cloud alignment is performed on all point clouds. When finding the path contour, the point cloud of each path is projected onto a plane image, and the contour edge of each path can be obtained based on the plane image. Triangulation can be performed according to the contour edge of each path to obtain a plane path grid, and finally the walls and/or ceiling of the scene are established along the plane path grid to establish a three-dimensional grid of the scene.

Preferably, a photographic system consisting of one or more fisheye lenses can be used to capture multiple images of a scene.

Furthermore, fisheye correction can be performed on multiple images taken by a photographic system, wherein the process includes: projecting each pixel of each image taken by one or more fisheye lenses to a spherical coordinate system, and then projecting each pixel on the sphere to an equidistant rectangular coordinate system, thereby completing the fisheye correction.

Furthermore, the adjacent images can be seam-cut by using a graph cutting algorithm to find the lowest loss seam in the adjacent images, and then the adjacent images can be repeatedly fused into one image by using a multi-band fusion algorithm to form a panoramic image.

Using the point clouds of each path, the spatial coordinates and rotation angles of the camera system that took multiple images are estimated using simultaneous localization and mapping technology. The point clouds are then converted from the coordinate system of the camera system to the world coordinate system to align all the point clouds to obtain a complete path and smoother point cloud.

Furthermore, when semantically segmenting each panorama, a pre-trained model of a hierarchical multi-scale attention mechanism for semantic segmentation can be used to perform semantic segmentation on the panorama, wherein the categories annotated by a dataset with a large number of high-resolution images are used to predict the panorama and obtain the semantic segmentation result, thereby obtaining the contour of the path within the scene.

Furthermore, the step of obtaining the contour edge of each path using the plane image includes establishing a mask image of corresponding size according to the size of the point cloud, projecting the pixels of the point cloud onto the mask image, and analyzing the approximate contour edge of the path in the point cloud using the mask image, and then filling the gaps in the contour edge to obtain the contour edge of the path of the point cloud.

Furthermore, after creating a 3D grid, you can project a panorama onto the 3D grid and use the panorama as a texture for the 3D grid, switching the panorama during movement.

To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only used for reference and description and are not used to limit the present invention.

The following is a specific embodiment to illustrate the implementation of the present invention. The technical personnel in this field can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments. The details in this specification can also be modified and changed in various ways based on different viewpoints and applications without deviating from the concept of the present invention. In addition, the drawings of the present invention are only for simple schematic illustration and are not depicted according to actual size. Please note in advance. The following implementation will further explain the relevant technical content of the present invention in detail, but the disclosed content is not used to limit the scope of protection of the present invention.

It should be understood that, although the terms "first", "second", "third", etc. may be used in this document to describe various components or signals, these components or signals should not be limited by these terms. These terms are mainly used to distinguish one component from another component, or one signal from another signal. In addition, the term "or" used in this document may include any one or more combinations of the related listed items depending on the actual situation.

The disclosure relates to a system and method for reconstructing a stereoscopic scene based on a panorama, one of the purposes of which is to solve the image distortion problem caused by the known discontinuous panorama switching process. In this way, the system for reconstructing a stereoscopic scene based on a panorama can allow a user to move and browse smoothly in a stereoscopic space, thereby improving the user's experience of moving and browsing in a stereoscopic space.

FIG. 1 shows a system architecture embodiment diagram for reconstructing a stereoscopic scene based on a panoramic image.

The system for reconstructing a three-dimensional scene based on a panoramic image is mainly implemented by a computer system, and cooperates with the image capture device 11 in the diagram to take multiple images of an indoor scene, and then uses the image processing device 100 to perform image correction, image alignment, image stitching, grid reconstruction, semantic segmentation, depth prediction, point cloud creation, mapping and other steps to create a three-dimensional model of the space. After completion, it can be output to the storage device 13 to store the file and used for other applications. It is explained here that the system is mainly implemented by the image capture device 11 and the image processing device 100. The image capture device 11 and the image processing device 100 can be two independent devices, or they can be designed as two subsystems in one device, which are connected to each other and implement a method of reconstructing a three-dimensional scene based on a panoramic image through collaborative operation with software.

The image capture device 11 may be a 360-degree camera, and may be implemented as a photography system composed of one or more fisheye lenses. The image capture device 11 may capture multiple images of one or more locations in a scene, and the multiple images may be stitched together to form a panorama, and then a 3D mesh may be reconstructed using an algorithm. According to one embodiment, when the image capture device 11 is used to capture a scene, it may be moved along a path formed by a road in the scene or a route set by the system to capture multiple images in multiple directions along the way, thereby capturing information about the scene. The 360-degree camera composed of one or more fisheye lenses can take multiple images at each location. In addition to having a photography system with computing capabilities to perform subsequent image processing, the illustrated implementation example is that after being input to the image processing device 100 through the input unit 103 of the image processing device 100, the image processing is performed by the processing unit 101 therein. After the three-dimensional grid is completed, in addition to being stored in the storage medium in the image processing device 100, it can also be output to the storage device 13 through the output unit 105.

It is worth mentioning that when multiple images are taken with one or more fisheye lenses, adjacent images may include overlapping areas, which may cause image misalignment. Therefore, by using a single fisheye lens to move and take multiple images, or by using multiple fisheye lenses to take multiple images, a panoramic image covering the scene can be formed by a stitching process and appropriate processing of the overlapping areas.

The functions running in the above-mentioned image processing device can refer to the functional module implementation example diagram of the method for reconstructing a stereoscopic scene based on a panoramic image shown in Figure 2. The functional modules in the figure are mainly implemented by the collaborative operation of hardware and software in the computer system, and the steps can also refer to the method implementation example flow chart shown in Figure 3.

The system includes an image input unit 21, and the image input unit 21 is an image capture device 11 as shown in FIG. 1. A photography system composed of one or more fisheye lenses or various 360-degree cameras is used to shoot multiple images in a scene (step S301). When adjacent images may have overlapping areas, the adjacent images can be further stitched and cut by a panorama image stitching unit 23. One implementation method is to use a graph-cut algorithm to perform stitching and cutting. The method is shown in the flowchart of the embodiment shown in FIG. 4. The main steps are to first obtain the overlapping area of two adjacent images, and then find a minimum loss seam in the overlapping area. After finding the minimum loss seam, a multiband blending algorithm is used to blend the adjacent images into one image. After repeating these steps, a panorama of multiple images is obtained, wherein multiple panoramas may be needed to describe a scene. For an example of seam cutting, please refer to FIG. 7.

The panorama is then output to the mesh reconstruction module 20, where the semantic segmentation unit 201 performs semantic segmentation on each of the multiple panorama images in the scene to obtain the pixels of all paths in the scene, and the depth estimation unit 203 performs depth estimation to estimate the depth of each path (step S303).

The semantic segmentation embodiment is described as follows.

In order to reconstruct the stereo grid, it is necessary to obtain the outline of the path in the scene, and the method used is to perform semantic segmentation on the panorama, first identify all pixel categories of the panorama for classification. According to one embodiment, the hierarchical multi-scale attention mechanism for semantic segmentation proposed by Andre Tao et al. can be used to perform semantic segmentation on all panoramas, wherein the pre-trained model of the hierarchical multi-scale attention mechanism for semantic segmentation is used to perform semantic segmentation on the panorama. According to one embodiment, the data set used for model training can be MapillaryVitas, which is a street view data of a large scene, including a large number of high-resolution color images, and multiple categories are annotated. Therefore, the obtained panorama can be input into the above algorithm for prediction, to obtain the classification of the panorama, and to obtain the semantic segmentation result.

The depth prediction embodiment is described as follows.

After obtaining the pixels of all paths from the panorama, the depth of the panorama is predicted. The depth map is evaluated by referring to the single image depth prediction based on wavelet decomposition proposed by Michael et al. The Resnet-18 pre-trained model provided by the single image depth prediction based on wavelet decomposition is used for depth prediction. The data set used by the model consists of a large number of calibrated stereoscopic videos, based on which the depth of the panorama can be predicted.

Afterwards, the point cloud building unit 207 obtains the pixels of each path in the scene according to the result of semantic segmentation and the path depth obtained by depth prediction to build the point cloud of each path (step S305).

It is worth mentioning that when the point cloud is established with the results predicted by the above semantic segmentation and depth prediction, all pixels belonging to the path (including various roads) in all panoramas in the scene are obtained, and the actual outline of the path in the scene can be used as the grid bottom for reconstruction, and only the pixels belonging to the path are converted to the point cloud in the three-dimensional space through the depth of the pixel position corresponding to the depth map. However, the point cloud established by the depth map will be a rough and non-smooth point cloud, so the position (height) of the camera system that took the image can be estimated first, and the point cloud can be resampled according to the position of the camera system to obtain a smoother point cloud.

According to an embodiment, before aligning the point cloud, it is necessary to know the exact corresponding position of each image obtained by the camera system of the image shooting scene, wherein a simultaneous localization and mapping (SLAM) unit 205 is used to estimate the corresponding position of each image obtained by the camera system using the technology of simultaneous localization and mapping (SLAM), which can be described by spatial coordinates and rotation angles. According to an embodiment, the OpenVSLAM algorithm can be used. Among them, since the scale unit of simultaneous localization and mapping may not be the scale unit of the correct corresponding depth, as shown in the embodiment, the scaling scale of simultaneous localization and mapping is calculated by the simultaneous localization and mapping scale estimation unit (SLAM scale estimation unit) 209 to scale the path of simultaneous localization and mapping in proportion.

Through the above-mentioned scaling, the scale of the simultaneous positioning and mapping construction is consistent with the actual scale, so that the point cloud will be correctly aligned. In the step of aligning all point clouds, the point cloud alignment unit 211 uses the spatial coordinates and rotation angle of the camera system estimated by the simultaneous positioning and mapping construction unit 205 for each point cloud of each path, such as the height and rotation direction of the image capture device when taking the image in the scene. Afterwards, the point cloud can be converted from the coordinate system of the camera system to the world coordinate system to align all point clouds, and finally a complete path and smoother point cloud can be obtained (step S307).

After the point cloud of each path in the scene is completely established, the point cloud on the path is reconstructed into a grid, wherein the point cloud is projected onto a plane image by a contour finding unit 213, and the rough contour edge of the path can be analyzed through the plane image (step S309). For related embodiments, please refer to the process shown in FIG8. Afterwards, the triangulation unit 215 uses the edge of the contour as a constraint. One embodiment is to use the Constrainted Delaunay Triangulation algorithm to perform triangulation calculation to establish a plane path grid (step S311).

When the plane path mesh is obtained, the 3D mesh building unit 217 can be used to build walls in the indoor scene along the edge of the plane path mesh, and/or the ceiling of the indoor scene can be added (step S313) to complete the 3D mesh reconstruction (step S315).

After the stereo grid is established, other elements obtained by shooting or scanning the scene can be projected onto the stereo grid model with the current panorama through the texture projective unit 219 according to the position of the camera system, and the stereo grid is mapped. When playing the stereo scene video reconstructed by the stereo grid, for the final mapping step, the model can be textured with the current panorama according to the current position, and the mapping is obtained by switching the panorama video.

After the reconstruction of the stereo grid is completed, the stereo scene realized by the stereo grid model can be used in various applications25. For example, navigation arrows, random non-player characters (NPCs) and some stereo interactive objects can be placed in the stereo scene, allowing the user to move forward or backward freely along the path in the stereo scene, or perform 360-degree rotation, allowing the user to observe the surrounding scene everywhere and interact with the objects in the scene at any time.

Regarding the above-mentioned image seam cutting step, the method thereof refers to the flow chart of the embodiment shown in FIG. 4 .

According to one embodiment, when a scene is photographed by a photography system composed of one or more fisheye lenses to obtain multiple images, fisheye correction is required first (step S401). The fisheye correction is mainly divided into two steps. The schematic diagram of the embodiment of fisheye correction can be referred to in FIG5. As shown in FIG5(A), a fisheye image 50 is shown, in which each pixel is represented by P(x', y'); as shown in FIG5(B), each pixel P(x', y') is projected onto the coordinate system of a sphere and converted to describe the pixel with three-dimensional coordinates P(x, y, z); then as shown in FIG5(C), the three-dimensional coordinates of each pixel are projected from the sphere to the coordinate system of an equirectangular (Equirectangular), and each pixel is described by coordinates P(x", y"), and the fisheye correction is completed.

After obtaining the corrected image, the image is then aligned (step S403). According to one embodiment, the Natural Image Stitching with the Global Similarity Prior (NISwGSP) algorithm proposed by Yu-Sheng Chen et al. can be used to perform image alignment and grid deformation.

For example, referring to the implementation example diagram of stitching four images shown in FIG6 , four corrected images are obtained, namely the first image 61, the second image 62, the third image 63 and the fourth image 64. Because all images need to be stitched into a 360-degree panoramic image, it means that the first image 61 and the fourth image 64 are aligned during the stitching process. The method shown in this example cuts the first image 61 into left and right sides, forming a first left image 611 and a first right image 612, and takes the left half of the first left image 611 as the fifth image, which is stitched with the fourth image 64. That is, the first left image 611 is used as input, and the above-mentioned NISwGSP algorithm is used for alignment and cropping, and finally outputs five aligned and deformed images.

Next, color correction is performed on the image (step S405). Since images taken by different lenses may have color differences due to different lighting angles or other factors, resulting in discontinuity in the stitched image, color correction is required.

After the image alignment is completed, multiple images are fused to generate a 360-degree panoramic image. However, in many cases, when the images are deformed and aligned, the parallax between different fisheye lenses may cause the image to not be completely and accurately aligned. Therefore, if image blending is performed on the overlapping areas between the images, obvious artifacts often appear. To solve this problem, according to an embodiment, the images can be seam-cutting to find the best seam and copy each image to the corresponding position of the seam to eliminate the artifacts.

When the adjacent images are subjected to seam cutting processing, according to the embodiment, a graph-cut algorithm may be used to perform seam cutting (step S407). The seam cutting process embodiment diagram may be shown in FIG7, wherein two adjacent images are shown as a first image _I1 and a second image _I2 , and an overlapping area 70 is found in the two images, and then a minimum loss seam 701 is obtained in the overlapping area 70, so that the images can be better fused together. When the minimum loss seam 701 is obtained, the left image 71 and the right image 72 shown in FIG7 are obtained, and then a multiband blending algorithm is used to blend the adjacent images into one image 73 (step S409). In this way, the seam cutting process shown in FIG. 7 can be repeated until all images are merged into a panoramic image.

FIG. 3 shows a step S309 of projecting the point cloud onto a plane image and then analyzing the plane image to obtain a rough outline edge of the path. The implementation method may refer to the embodiment process shown in FIG. 8 .

After the point cloud is created, the path contour analysis will be performed on the point cloud. First, a mask image of the corresponding size is created according to the size of the point cloud (step S801). This mask mainly represents the contour shape of the point cloud, and the actual contour of the path in the point cloud can be analyzed. Then the point cloud is projected onto this two-dimensional mask image (step S803), in which the three-dimensional coordinates of the pixels of the point cloud are converted to the plane coordinates of the mask image. The conversion method is to directly take the x, y coordinate values of each point cloud pixel coordinate value (x, y, z) as the coordinates of the point cloud on the mask image.

Therefore, when each point cloud is converted to the coordinate system of the mask image, the mask image can be used to analyze the approximate contour edge of the path (step S805), and the schematic diagram of the mask image 90 shown in Figure 9 (A) can be referred to. However, before finding the contour edge, some broken gaps can be found in some positions on the mask image, such as the gap 901 shown in Figure 9 (A), and some gaps need to be filled up (step S807). The method of filling the gaps can be to perform Gaussian Blur on the mask image 90, and then perform image dilation (dilate) and image erosion (erode) on the image, so that some gaps can be filled up.

In this way, a mask image 90' after gap filling as shown in FIG. 9(B) can be formed, in which the gaps are shown to have been filled, forming filled gaps 901', so that subsequent processing will not be affected by these gaps, thereby minimizing the problem of too complex contour structure.

Then, the mask image can be analyzed, as shown in FIG9(C), to obtain the contour edge image 90'' of the path (step S809), and the obtained contour edge can be used as the floor shape of the reconstructed three-dimensional grid and the plane grid can be reconstructed.

After obtaining the plane mesh of the plane path, the edge points of the outline can be used as constraint points, such as in the above embodiment, and triangulation is performed using constrained Delaunay triangulation. Since the triangulated mesh may have triangles that are too large or too narrow, the triangles in the path outline will then be refined. For the triangle meshes that are too large or too narrow, the triangle meshes can be refined with newly added vertices to obtain a triangle mesh of more uniform size.

It is worth mentioning that since the triangulated mesh is a convex hull shape, there will be many triangles that are not within the outline. Therefore, it is necessary to find the triangles outside the outline and delete these meshes to finally get a plane mesh of a complete road. Then convert the plane mesh to a three-dimensional mesh. Finally, build walls and/or ceilings along the edges of the mesh to complete the reconstruction of the three-dimensional mesh.

Finally, at the application level, the mapping method of the model using the stereo grid is projected according to the current position. In particular, the mapping is a panorama, and the panoramic image will cover the horizontal 360-degree field of view and the vertical 180-degree field of view. Therefore, all the vertices of the stereo grid model can be extrapolated to the corresponding mapping coordinates. In this way, it is only necessary to calculate the polar angle and azimuth between the vertex and the current camera system, and then the corresponding mapping coordinates can be calculated through the polar angle and azimuth.

The method proposed in the disclosure book can solve the problem of serious texture distortion in the known technology caused by discontinuous switching of textures when moving in a stereoscopic scene. When the user interacts in a stereoscopic scene formed by a stereoscopic grid, the textures on the stereoscopic grid are switched by playing videos. During operation, the user can control the movement and panorama switching in the stereoscopic scene through input devices such as a keyboard or mouse, and the movement method is to move forward or backward along the path calculated by the simultaneous positioning and map construction technology, and switch the textures simultaneously during the movement. That is, when the user moves forward, the system will play the panoramic video forward, and when moving backward, it will play the panoramic video backward, thereby solving the problem of distortion caused by the switching process.

In summary, according to the system and method for reconstructing a stereoscopic scene based on a panorama described in the above embodiments, a panorama is used as an input image, and a stereo grid is established based on it, which can solve the problem of image distortion caused by image discontinuity. Furthermore, after the stereo grid is established, when browsing a stereo image and mapping, it can be obtained by switching the panorama video. The proposed system can also add some interactive objects to the stereo scene, and allow the user to move freely along the route in the stereo scene, and can rotate the lens or interact with the stereo object.

The contents disclosed above are only preferred feasible embodiments of the present invention and are not intended to limit the scope of the patent application of the present invention. Therefore, all equivalent technical changes made using the contents of the specification and drawings of the present invention are included in the scope of the patent application of the present invention.

11: Image capture device 100: Image processing device 103: Input unit 101: Processing unit 105: Output unit 13: Storage device 21: Image input unit 23: Image stitching unit 20: Grid reconstruction unit 201: Semantic segmentation unit 203: Depth estimation unit 205: Simultaneous positioning and mapping unit 207: Point cloud building unit 209: Simultaneous positioning and mapping scale estimation unit 211: Point cloud alignment unit 213: Contour finding unit 215: Triangulation positioning unit 217: Stereo grid building unit 219: Texture projection unit 25: Application 50: Fisheye image 61: First image 62: Second image 63: Third image 64: Fourth image 611: First left image 612: First right image I ₁ : First image I ₂ : Second image 70: Overlapping area 701: Minimum loss seam 71: Left image 72: Right image 73: Image 90: Mask image 901: Gap 90': Mask image 901': Filled gap 90'': Contour edge image Steps S301~S315 Process of reconstructing a stereoscopic scene based on a panorama Steps S401~S409 Process of image stitching Steps S801~S809 Process of analyzing path contours

Figure 1 shows the system architecture for reconstructing a stereoscopic scene based on a panorama.

FIG. 2 shows an example diagram of a functional module for executing a method for reconstructing a stereoscopic scene based on a panoramic image;

FIG3 is a flow chart showing an embodiment of a method for reconstructing a stereoscopic scene based on a panoramic image;

FIG4 shows a flowchart of an embodiment of image stitching;

FIG. 5 (A) (B) (C) shows an example diagram of fisheye correction;

FIG6 shows an example diagram of an implementation of stitching four images;

FIG. 7 shows an example diagram of a seam cutting process;

FIG8 is a flowchart showing an embodiment of analyzing path profiles; and

FIG9(A)(B)(C) shows an example of an analysis path profile.

21: Image input unit

23: Image stitching unit

20: Grid reconstruction group

201: Semantic segmentation unit

203: Depth estimation unit

205: Simultaneous positioning and map construction unit

207: Point cloud creation unit

209: Simultaneous positioning and map construction scale estimation unit

211: Point cloud alignment unit

213: Contour search unit

215: Triangulation unit

217: 3D grid creation unit

219: Texture projection unit

25: Application

Claims (16)

A method for reconstructing a stereoscopic scene based on a panorama is run in a system, wherein the system includes an image capture device and an image processing device, and the method includes: Acquire multiple images in a scene through the image capture device; The image processing device performs the following steps, including: Stitching the multiple images to obtain a panorama of the scene; Performing semantic segmentation and depth prediction on the panorama; Obtaining the point cloud of each path obtained from the path depth obtained by the depth prediction and the path pixels in the scene according to the result of semantic segmentation, and performing point cloud alignment on all point clouds; Projecting the point cloud of each path to a plane image, and obtaining the contour edge of each path according to the plane image; Perform a triangulation calculation according to the outline edge of each path to obtain a planar path grid; and Build the walls and/or ceiling of the scene along the planar path grid to build a three-dimensional grid of the scene. A method for reconstructing a stereoscopic scene based on a panorama as described in claim 1, wherein the multiple images in the scene are captured by a photographic system composed of one or more fisheye lenses. A method for reconstructing a stereoscopic scene based on a panoramic image as described in claim 2, wherein the multiple images taken by the photographic system are subjected to fisheye correction, wherein the process executed by the image processing device includes: Projecting each pixel of each image taken by the one or more fisheye lenses to a spherical coordinate system; and Projecting each pixel on the sphere to an equidistant rectangular coordinate system to complete the fisheye correction. A method for reconstructing a stereoscopic scene based on a panoramic image as described in claim 2, wherein the corresponding position of each image obtained by the photographic system is obtained by using the technology of simultaneous positioning and map construction. A method for reconstructing a stereoscopic scene based on a panorama as described in claim 1, wherein adjacent images are seam-cut by a graph cutting algorithm to find a minimum loss seam in the adjacent images, and then the adjacent images are repeatedly fused into one image by a multi-band fusion algorithm to form the panorama. A method for reconstructing a stereoscopic scene based on a panorama as described in claim 1, wherein the point clouds of each path are used to estimate the spatial coordinates and rotation angles of a photographic system that took the multiple images using simultaneous positioning and mapping techniques, and then the point cloud is converted from the coordinate system of the photographic system to a world coordinate system to align all point clouds to obtain a complete path and smoother point cloud. A method for reconstructing a stereoscopic scene based on a panorama as described in claim 1, wherein the triangulation is performed using a constrained Delaunay triangulation algorithm. A method for reconstructing a stereoscopic scene based on a panorama as described in claim 1, wherein in the step of performing semantic segmentation on each panorama, a pre-trained model of a hierarchical multi-scale attention mechanism for semantic segmentation is used to perform semantic segmentation on the panorama, wherein the panorama is predicted using categories annotated by a dataset with a large number of high-resolution images to obtain a semantic segmentation result, thereby obtaining the outline of the path within the scene. The method for reconstructing a stereoscopic scene based on a panoramic image as described in claim 8, wherein the step of the image processing device obtaining the contour edge of each path according to the planar image includes: Creating a mask image of corresponding size according to the size of the point cloud Projecting the pixels of the point cloud onto the mask image; Analyzing the approximate contour edge of the path in the point cloud with the mask image; Filling the gap of the contour edge; and Obtaining the contour edge of the path of the point cloud. A method for reconstructing a stereoscopic scene based on a panorama as described in any one of claims 1 to 9, wherein after the stereoscopic grid is established, the stereoscopic grid is projected with the panorama, and the mapping on the stereoscopic grid is switched by playing a panoramic video. A system for reconstructing a stereoscopic scene based on a panorama, comprising: an image capture device; and an image processing device, connected to the image capture device, for executing a method for reconstructing a stereoscopic scene based on a panorama, comprising: obtaining multiple images of one or more positions in a scene through the image capture device; stitching the multiple images to obtain a panorama of the scene; performing semantic segmentation and depth prediction on the panorama; obtaining point clouds of each path obtained from path pixels in the scene and path depths obtained from depth prediction based on the result of semantic segmentation, and performing point cloud alignment on all point clouds; projecting the point clouds of each path to a planar image, and obtaining the contour edges of each path based on the planar image; Perform a triangulation calculation according to the outline edge of each path to obtain a planar path grid; and Build the walls and/or ceiling of the scene along the planar path grid to build a three-dimensional grid of the scene. A system for reconstructing a stereoscopic scene based on a panoramic image as described in claim 11, wherein the image processing unit is implemented using a photographic system composed of one or more fisheye lenses. A system for reconstructing a stereoscopic scene based on a panorama as described in claim 11, wherein adjacent images are seam-cut by a graph cutting algorithm to find a minimum loss seam in the adjacent images, and then the adjacent images are repeatedly fused into one image by a multi-band fusion algorithm to form the panorama. A system for reconstructing stereoscopic scenes based on panoramas as described in claim 11, wherein in the step of performing semantic segmentation on each panorama, a pre-trained model of a hierarchical multi-scale attention mechanism for semantic segmentation is used to perform semantic segmentation on the panorama, wherein the panorama is predicted using categories annotated by a dataset with a large number of high-resolution images to obtain a semantic segmentation result, thereby obtaining the outline of the path within the scene. A system for reconstructing a stereoscopic scene based on a panoramic image as described in claim 14, wherein the step of obtaining the contour edge of each path based on the planar image includes: Creating a mask image of corresponding size based on the size of the point cloud Projecting the pixels of the point cloud onto the mask image; Analyzing the approximate contour edge of the path in the point cloud using the mask image; Filling the gap of the contour edge; and Obtaining the contour edge of the path of the point cloud. A system for reconstructing a stereoscopic scene based on a panorama as described in any one of claims 11 to 15, wherein after the stereoscopic grid is established, the stereoscopic grid is projected with the panorama, and the mapping on the stereoscopic grid is switched by playing a panoramic video.

Images