TWI734116B - Method for spherical camera image stitching - Google Patents

Abstract

The present invention provides a method for Spherical Camera Image Stitching. By using two fisheye lens to catch two fisheye images and then being developed into three pairs of flat figures based on Segmented Sphere Projection (SSP) method. Thereafter each corresponding pair is stitched based on a similar-edge method, and then three pairs are combined to form a panoramic image. At the end, the combined panoramic image is projected to a 3-D ball sphere space.

Description

Image stitching method of spherical camera

The present invention relates to a spherical camera image splicing method, and in particular refers to the use of a segmented spherical projection method to splice images based on similar edge techniques, and then project the spliced panoramic image into a 3-D spherical space.

Please see Figure 1, which is a schematic diagram of the existing Equirectangular Projection (ERP) method. Starting from the upper left, a fisheye camera 1 takes images with two fisheye lenses 11 and 12 back-to-back (step 1), and obtains two fisheye diagrams A and B (step 2). The two fisheye diagrams A and B respectively represent the images of the two hemispheres, but each takes about 190° images, so there is an extra circle 13 and 14 of about 5° below each (step 3). Flatten the images of the two hemispheres into a rectangle, which becomes the two expanded plan views at the bottom left (step 4). This method of converting the hemispherical image into a rectangular plan view is known as the equidistant cylindrical projection (ERP) method.

These two expanded plan views are divided into two halves, namely A left, A right, B left, and B right. The left strip 131 on the left of A and the right strip 132 on the right of A are formed by separating the ring 13, each occupying about 10° (-95°~-85° and 85°~95°). The left strip 141 on the left of B and the right strip 142 on the right of B are formed by separating the ring 14, each occupying about 10° (-95°~-85° and 85°~95°).

Splicing the strips 132 and 141 of the two unfolded plan views, and splicing the strips 131 and 142 together to form a panoramic image (step 5), and finally project the spliced panoramic image into a 3-D sphere The space forms a spherical ring field image (step 6).

This equidistant column projection (ERP) method turns the sphere into a flat rectangle, so the north/south pole of the sphere will have considerable image distortion relative to the central equatorial region of the sphere.

The purpose of the present invention is to provide a method for splicing images of a spherical camera, which greatly improves the image distortion of the north/south pole of the image sphere relative to the central equatorial region of the sphere. The main content is to use segmented sphere projection (Segmented Sphere Projection, SSP), the image is divided into three parts of the North Pole, the equator and the South Pole and processed. The processing method is to use the similar edge method and the feather fusion method.

The present invention uses a fisheye camera to take images with two fisheye lenses back-to-back to obtain two hemispherical fisheye images, each taking about 190° images; each of the two fisheye images is divided into three parts: the North Pole, the Equator and Antarctica, the three segments are divided at +45°N latitude and -45°South latitude, forming three parts of the North Pole above +45°N latitude, the equator below +45°N to -45°S latitude, and Antarctica below -45°S latitude; The north pole and the south pole of the two hemisphere fisheye images are respectively converted into a semicircle; the two semicircles of the north pole and the south pole of the two hemisphere fisheye images are respectively spliced into a circle by a similar edge method; At the same time, the overlapping area is feathered and merged, and then the circle is turned into a spherical surface; the equatorial area is merged with traditional ERP (based on the similar edge method and the feathering fusion method), and then the three sections of the North Pole, the Equator and the South Pole are joined to form a panoramic image ; Finally, the stitched panoramic image is projected into the 3-D spherical space to form a spherical ring field image.

1‧‧‧Fisheye Camera

11‧‧‧Fisheye lens

12‧‧‧Fisheye lens

A‧‧‧Fisheye

B‧‧‧Fisheye

13‧‧‧Circle

14‧‧‧Circle

131‧‧‧Strip

132‧‧‧Strip

141‧‧‧Strip

142‧‧‧Strip

31‧‧‧Strip

32‧‧‧Strip

33‧‧‧Strip

34‧‧‧Strip

131'‧‧‧ Strip

132'‧‧‧ Strip

141'‧‧‧ Strip

142'‧‧‧ Strip

Figure 1 is a schematic flow diagram of the existing Equirectangular Projection (ERP) method.

Figures 2, 3, and 4 are explanatory diagrams of the Segmented Sphere Projection (SSP) method of the present invention.

Fig. 5 is an explanatory diagram of the similar edge method of the present invention.

Fig. 6 is an explanatory diagram of the feathering and fusion method of the present invention.

Figures 2, 3, and 4 are explanatory diagrams of the Segmented Sphere Projection (SSP) method of the present invention. The steps in FIG. 2 still use steps 1 to 3 in FIG. 1, and steps 4', 51', 52' in FIG. 3 and step 6'in FIG. 4 are the steps of the segmented sphere projection method.

Please see Figure 2. Starting from the left, a fisheye camera 1 takes images with two fisheye lenses 11 and 12 back-to-back (step 1) to obtain two fisheye diagrams A and B (step 2). The two fisheye diagrams A and B respectively represent the images of the two hemispheres, but each takes about 190° images, so there is an extra circle 13 and 14 of about 5° below each (step 3).

Steps 4', 51', 52' in Figure 3 are segmented sphere projection methods (SSP method). The two fisheye diagrams in step 3 of Figure 2 are divided into three parts: the North Pole, the Equator and the South Pole, the boundary of the three segments In the sphere image's north latitude +45° and south latitude -45°, three parts are formed.

In step 4'of FIG. 3, the equatorial parts of the fisheye diagrams A and B are flattened into a rectangle by the ERP method, and become the two expanded plan views in the middle of FIG. 3.

Convert the north pole above +45° north latitude and the south pole below -45° south latitude of the fisheye diagrams A and B into a semicircle, as shown in step 51'.

Perform stitching based on the similar edge method and Feather blending above the north latitude +45° (see the description of [0024], [0025], and [0026]), that is, the strip 31 and the strip 32 are spliced to form a circle ( Step 52').

Perform stitching based on the similar edge method and Feather blending below the south latitude -45° (see the description of [0024], [0025], and [0026]), that is, the strips 33 and 34 are spliced to form a circle (step 52').

Please refer to Figure 4, the equatorial region from below +45° north latitude to above -45° south latitude is subjected to the ERP stitching and feather blending based on similar edges described in the previous technology (see [0024], [0025] , [0026] description), and then join the upper, middle, and lower three sections (North Pole, Equator and South Pole) to form a panoramic image (step 6').

Finally, the stitched panoramic image is projected into a 3-D spherical space to form a spherical ring field image.

The splicing method based on similar edges for the above three sections of North Pole, Equator and South Pole is explained as follows: After edge detection of the two regions to be spliced, the image result is 1 or 0 (1 is an edge, and 0 is a non-edge). A pixel is composed of three values: the brightness of red light, the brightness of green light, and the brightness of blue light, which are referred to as RGB images, which are between 0-255. The three values are superimposed to get various colors. Grayscale images have only one color per pixel, from the darkest black, multi-level gray to the brightest white. The RGB image can be obtained by weighting and averaging R, G, and B respectively to obtain a grayscale image. Then, edge detection is performed. The so-called edge refers to the area in the image with obvious brightness changes (called edge). After the image is gray-scaled, look for areas with large gray-scale changes, set it as edge 1, and the rest as non-edge 0. This can greatly reduce the amount of data and retain important structural attributes in the image.

Please refer to Figure 5, match the two areas to be spliced (for example 31 and 32 or 33 and 34 or 132' and 141' or 131' and 142') to the left and right, respectively taking the center as the standard, and each taking the range L (Main match) and range W (matched). Move L on W, calculate the similarity between L and L'with the same width on W, find the most similar position of L', and join L and L'. The following formula is the similarity value algorithm. The closer the similarity value is to 1, the more similar. The similarity value is equal to 1 to be exactly the same. A and b are the edge values (0 or 1) of the two matched images respectively, and h is the height of the matched image. w is the width of the matched image, and i and j respectively represent the ordinate value and the abscissa value of the image sampling point.

Please refer to Figure 6, the two most similar overlapping areas L and L'are overlapped in weight, and a weight map is established based on the distance between the overlapped area and the boundary of each photo. The closer you are to yourself, the weight is 1, and the farther away you are The weight of the edge is 0, and the middle is 0.5. The sum of the weights of the two image pixels must be 1. The images L and L'are respectively multiplied by weights and superimposed, so that the overlapping part has a gradual effect, which is smoother and reduces the sense of abruptness. This is Feather blending.

The spirit and scope of the present invention are determined by the scope of the following patent applications and are not limited The above embodiment.

A‧‧‧Fisheye

B‧‧‧Fisheye

13‧‧‧Circle

14‧‧‧Circle

131'‧‧‧ Strip

132'‧‧‧ Strip

141'‧‧‧ Strip

142'‧‧‧ Strip

31‧‧‧Strip

32‧‧‧Strip

33‧‧‧Strip

34‧‧‧Strip

Claims (1)

A method for splicing spherical camera images, including the following steps: (1) Two fisheye images of the hemisphere are obtained by two back-to-back fisheye lenses, each with a 190° image; (2) the two fisheye images are divided into each There are three parts: the North Pole, the Equator and the South Pole. The boundary of the three sections is at +45°N and -45°South, forming the north pole above +45°N latitude, the equator below +45°N to -45°South latitude and -45°South latitude °The three parts of the South Pole below °; (3) Convert the north and south poles of the two hemisphere fisheye images into a semicircle; (4) the two hemisphere fisheye images of the two north and south poles each The semicircles are spliced into a circle by the similar edge method; (5) When splicing, the overlapping area is feathered and merged, and then the circle is turned into a spherical surface; (6) The equatorial region is spliced by traditional ERP (based on the similar edge method and the similar edge method). Feather fusion method), and then join the three sections of the North Pole, the Equator and the South Pole to form a panoramic image; (7) Finally, the stitched panoramic image is projected into a 3-D spherical space to form a spherical ring field image; (8) The similarity The edge method is to convert the pixels of the fisheye image from a red, green and blue (RGB) image to a grayscale image, and then set the value of the point with obvious brightness changes in the grayscale image to 1 (called edge), and the remaining points are Set to 0 (referred to as non-edge); match the two regions to be spliced with each center as the standard, and take the range L (main matching) and range W (matched); move L on W, set L'with the same width on L and W calculates their respective similarity values, finds the most similar L'position, and joins L and L'; take the following formula as the similarity algorithm, the closer the similarity value is to 1, the more similar , If the similarity value is equal to 1, it is exactly the same, a and b are the edge values of the two matched images (0 or 1), h is the height of the matched image, w is the width of the matched image, and i, j respectively represent the vertical of the image sampling point Coordinate value and abscissa value,

(9) The feathering fusion method is to overlap the two most similar overlapping areas L and L'with weights, and build a weight map based on the distance from the overlapped area to the boundary of each photo. The closer you are to yourself, the weight is 1. The farther away from you, the weight is 0, and the middle is 0.5. The weights of the two image pixels are added to 1, and the images L and L'are respectively multiplied by the weights and superimposed.

Images