CN116309041A - Infrared panoramic image stitching method, infrared panoramic monitoring device and electronic equipment - Google Patents

Abstract

The invention discloses an infrared panoramic image stitching method, an infrared panoramic monitoring device and electronic equipment, which comprise the following steps: obtaining infrared images shot by each camera, establishing a camera coordinate system corresponding to each camera, establishing a world coordinate system, taking an origin of the world coordinate system as a sphere center, establishing a spherical surface, r is obtained by calibrating the cameras, establishing an output plane in the world coordinate system, obtaining a mapping relation between points on the output plane and points on the spherical surface, obtaining a first coordinate according to the mapping relation, obtaining an RT relation between the world coordinate system and the camera coordinate system, converting the first coordinate into a second coordinate according to the RT relation, converting the RT relation into a coordinate conversion relation between the world coordinate system and the camera coordinate system, converting the second coordinate into a pixel coordinate point, taking a pixel value of an infrared image corresponding to the pixel coordinate point as a target pixel value, and taking the target pixel value as a pixel value of a point of the output plane to obtain the infrared panoramic image.

Description

Infrared panoramic image stitching method, infrared panoramic monitoring device and electronic equipment

technical field

The invention relates to the technical field of image processing, in particular to an infrared panoramic image splicing method, an infrared panoramic monitoring device and electronic equipment.

Background technique

The existing infrared panoramic image generation technology often obtains multiple infrared images, and stitches and fuses all the infrared images to obtain an infrared panoramic image.

However, the existing infrared panoramic image generation technology has the problems of large amount of data calculation and slow image stitching speed.

Contents of the invention

The present invention aims to solve at least one of the technical problems existing in the prior art. Therefore, the present invention proposes an infrared panoramic image mosaic method, an infrared panoramic monitoring device and electronic equipment, which can solve the problems of large data computation and slow image mosaic speed in the existing infrared panoramic image generation technology.

The infrared panoramic image stitching method according to the embodiment of the first aspect of the present invention includes: obtaining infrared images taken by each camera; establishing a camera coordinate system corresponding to each of the cameras, and taking the center of gravity of the origin of each of the camera coordinate systems as the origin , establish a world coordinate system; use the origin of the world coordinate system as the center of the sphere, and use r as the radius to establish a spherical surface, and the r is obtained by calibrating the camera; establish an output plane in the world coordinate system, and obtain all The mapping relationship between the points on the output plane and the points on the spherical surface, according to the mapping relationship, the first coordinates are obtained, and the first coordinates are the coordinates of the spherical surface mapping points in the world coordinate system, and the spherical surface The mapping point is a point on the spherical surface corresponding to the point on the output plane; the RT relationship between the world coordinate system and the camera coordinate system is obtained, and the first coordinate is converted into a second coordinate according to the RT relationship, so The second coordinate is the coordinate point of the spherical mapping point coordinates in the camera coordinate system, and the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system; the second coordinate is converted into a pixel A coordinate point, using the pixel value of the infrared image corresponding to the pixel coordinate point as a target pixel value, and the target pixel value is a pixel value of a point on the output plane to obtain an infrared panoramic image.

The infrared panoramic image stitching method according to the embodiment of the first aspect of the present invention has at least the following beneficial effects:

By obtaining the infrared images taken by each camera, the camera coordinate system corresponding to each camera is established, and the center of gravity of the origin of each camera coordinate system is used as the origin to establish the world coordinate system, and the origin of the world coordinate system is used as the center of the sphere to establish a spherical surface. The output plane is established in the world coordinate system, and the mapping relationship between the points on the output plane and the points on the spherical surface is obtained. According to the mapping relationship, the first coordinate is obtained. The first coordinate is the coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point For the point on the spherical surface corresponding to the point on the output plane, obtain the RT relationship between the world coordinate system and the camera coordinate system. According to the RT relationship, convert the first coordinate to the second coordinate. The second coordinate is the coordinate of the spherical mapping point in the camera coordinate The coordinate point in the system, the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system, convert the second coordinate into a pixel coordinate point, and use the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value, and the target pixel value In order to obtain the infrared panoramic image by outputting the pixel values of the points on the plane, only the pixel values corresponding to each point on the output plane are calculated, and there is no need to process and calculate all the infrared image data, which reduces the amount of data calculation, and the image splicing speed is fast.

According to some embodiments of the present invention, the acquisition of the mapping relationship between the points on the output plane and the points on the spherical surface, and according to the mapping relationship, the first coordinates are obtained, and the first coordinates are the spherical mapping points at Coordinates in the world coordinate system, the spherical mapping point is a point on the spherical surface corresponding to the point of the output plane, including: obtaining the four vertices P1, P2, P3 and P4 of the output plane in coordinates in the world coordinate system; sampling at equal intervals between the P1, the P2, the P3, and the P4 to obtain the coordinates of each point on the output plane in the world coordinate system; Projecting each point on the output plane to the spherical surface, and obtaining the first coordinates according to a projection relationship between each point on the output plane and the spherical surface.

According to some embodiments of the present invention, the acquisition of the mapping relationship between the points on the output plane and the points on the spherical surface, and according to the mapping relationship, the first coordinates are obtained, and the first coordinates are the spherical mapping points at Coordinates in the world coordinate system, the spherical mapping point is a point on the spherical surface corresponding to the point on the output plane, including: taking N vertical lines and M horizontal lines on the output plane, The N vertical lines divide the output plane horizontally, the M horizontal lines divide the output plane vertically, and the N vertical lines and the M horizontal lines form N*M first Intersection; N warps and M latitudes are taken on the spherical surface, N said longitudes divide the spherical surface equally horizontally, M said transverse lines divide the spherical surface vertically equally, N said longitudes and The M parallels form N*M second intersection points; and the N*M first intersection points are sequentially and one-to-one corresponding to the N*M second intersection points to obtain the first coordinates.

According to some embodiments of the present invention, the acquisition of the mapping relationship between the points on the output plane and the points on the spherical surface, and according to the mapping relationship, the first coordinates are obtained, and the first coordinates are the spherical mapping points at Coordinates in the world coordinate system, the spherical mapping point is a point on the spherical surface corresponding to the point of the output plane, including: establishing an inscribed circular plane in the spherical surface, and the radius of the inscribed circular plane The same as the radius of the spherical surface, the circumscribed square plane of the inscribed circle plane is the output plane; obtain the coordinates of each point of the inscribed circle plane in the world coordinate system, that is, the circumscribed square plane and The coordinates of the intersection point of the inscribed circle plane in the world coordinate system are obtained to obtain the first coordinates.

According to some embodiments of the present invention, the pixel value of the infrared image corresponding to the pixel coordinate point is used as a target pixel value, and the target pixel value is a pixel value of a point on the output plane to obtain an infrared panoramic image, including : performing image fusion on overlapping areas of each of the infrared images corresponding to all the pixel coordinate points to obtain a target infrared image, and each of the pixel coordinate points in the overlapping area corresponds to a plurality of pixel values of the infrared images; The pixel value of the target infrared image corresponding to the pixel coordinate point is used as a target pixel value, and the target pixel value is a pixel value of a point on the output plane to obtain an infrared panoramic image.

According to some embodiments of the present invention, performing image fusion on overlapping areas of the infrared images corresponding to all the pixel coordinate points to obtain the target infrared image includes: acquiring a low-light image corresponding to the infrared image; The low-light image and the infrared image are enhanced to obtain a transitional low-light image and a transitional infrared image; the transitional low-light image and the transitional infrared image are fused to obtain an enhanced infrared image; all the pixel coordinate points Image fusion is performed on overlapping areas of corresponding enhanced infrared images to obtain a target infrared image.

According to some embodiments of the present invention, said enhancing the low-light image and the infrared image to obtain the transitional low-light image and the transitional infrared image comprises the following steps: preset a gamma lookup table; according to the gamma lookup table , mapping the pixel value of the low-light image to the corresponding gamma-corrected pixel value to obtain a transitional low-light image; according to the gamma lookup table, mapping the pixel value of the infrared image to the corresponding gamma-corrected pixel value to obtain a transitional infrared image.

According to some embodiments of the present invention, fusing the transitional low-light image and the transitional infrared image to obtain an enhanced infrared image includes: decomposing the transitional low-light image into a base layer image and a detail layer image; The transition infrared image is decomposed into a base layer image and a detail layer image; the base layer image of the transition low-light image and the base layer image of the transition infrared image are fused to obtain a target base layer image; the transition The detail layer image of the low-light image and the detail layer image of the transition infrared image are fused to obtain a target detail layer image; the target base layer image and the target detail layer image are reconstructed to obtain an infrared panoramic image.

The infrared panoramic monitoring device according to the embodiment of the second aspect of the present invention includes: a camera, the camera is used to collect low-light images and infrared images from multiple angles;

An image processing module, the output end of the camera is connected to the input end of the image processing module, and the image processing module executes the above-mentioned infrared panoramic image stitching method;

A communication module, the output end of the image processing module is connected to the input end of the communication module, and the communication module is used to transmit the infrared panoramic image to the host end.

The infrared panoramic monitoring device according to the embodiment of the second aspect of the present invention has at least the following beneficial effects:

By obtaining the infrared images taken by each camera, the camera coordinate system corresponding to each camera is established, and the center of gravity of the origin of each camera coordinate system is used as the origin to establish the world coordinate system, and the origin of the world coordinate system is used as the center of the sphere to establish a spherical surface. The output plane is established in the world coordinate system, and the mapping relationship between the points on the output plane and the points on the spherical surface is obtained. According to the mapping relationship, the first coordinate is obtained. The first coordinate is the coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point For the point on the spherical surface corresponding to the point on the output plane, obtain the RT relationship between the world coordinate system and the camera coordinate system. According to the RT relationship, convert the first coordinate to the second coordinate. The second coordinate is the coordinate of the spherical mapping point in the camera coordinate The coordinate point in the system, the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system, convert the second coordinate into a pixel coordinate point, and use the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value, and the target pixel value In order to obtain the infrared panoramic image by outputting the pixel values of the points on the plane, only the pixel values corresponding to each point on the output plane are calculated, and there is no need to process and calculate all the infrared image data, which reduces the amount of data calculation, and the image splicing speed is fast.

An electronic device according to the embodiment of the third aspect of the present invention includes: at least one processor; at least one memory for storing at least one program; when at least one of the programs is executed by at least one of the processors, the above-mentioned Infrared panoramic image stitching method.

An electronic device according to an embodiment of the third aspect of the present invention has at least the following beneficial effects:

By obtaining the infrared images taken by each camera, the camera coordinate system corresponding to each camera is established, and the center of gravity of the origin of each camera coordinate system is used as the origin to establish the world coordinate system, and the origin of the world coordinate system is used as the center of the sphere to establish a spherical surface. The output plane is established in the world coordinate system, and the mapping relationship between the points on the output plane and the points on the spherical surface is obtained. According to the mapping relationship, the first coordinate is obtained. The first coordinate is the coordinate of the spherical mapping point in the world coordinate system, and the spherical mapping point For the point on the spherical surface corresponding to the point on the output plane, obtain the RT relationship between the world coordinate system and the camera coordinate system. According to the RT relationship, convert the first coordinate to the second coordinate. The second coordinate is the coordinate of the spherical mapping point in the camera coordinate The coordinate point in the system, the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system, convert the second coordinate into a pixel coordinate point, and use the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value, and the target pixel value In order to obtain the infrared panoramic image by outputting the pixel values of the points on the plane, only the pixel values corresponding to each point on the output plane are calculated, and there is no need to process and calculate all the infrared image data, which reduces the amount of data calculation, and the image splicing speed is fast.

Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The present invention will be further described below in conjunction with accompanying drawing and embodiment, wherein:

Fig. 1 is the flowchart of the infrared panoramic image mosaic method of the present invention;

Fig. 2 is the mapping relationship between the points on the acquisition output plane and the points on the spherical surface obtained by the infrared panoramic image stitching method of the present invention, according to the mapping relationship, the flow chart a of the first coordinate is obtained;

Fig. 3 is the mapping relationship between the points on the acquisition output plane and the points on the spherical surface obtained by the infrared panoramic image stitching method of the present invention, according to the mapping relationship, the flow chart b of the first coordinate is obtained;

Fig. 4 is the mapping relationship between the points on the acquisition output plane and the points on the spherical surface obtained by the infrared panoramic image stitching method of the present invention, according to the mapping relationship, the flow chart c of the first coordinate is obtained;

Fig. 5 is the flow chart of obtaining the infrared panoramic image by using the pixel value of the infrared image corresponding to the pixel coordinate point as the target pixel value in the infrared panoramic image splicing method of the present invention, and the target pixel value is the pixel value of the point of the output plane;

Fig. 6 is a flow chart of the infrared panoramic image mosaic method of the present invention that performs image fusion on overlapping areas of the respective infrared images corresponding to all pixel coordinate points to obtain the target infrared image;

FIG. 7 is a positional diagram of the first infrared image and the second infrared image on the output plane of the infrared panoramic image stitching method of the present invention;

Fig. 8 is a functional block diagram of the infrared panoramic monitoring device of the present invention.

Reference signs:

camera 100,

Image processing module 200,

Communication module 300.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

In the description of the present invention, it should be understood that when it comes to orientation descriptions, for example, the orientation or positional relationship indicated by up, down, etc. is based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description , rather than indicating or implying that the device or element referred to must have a particular orientation, be constructed and operate in a particular orientation, and thus should not be construed as limiting the invention.

In the description of the present invention, a plurality refers to two or more. If the description of the first and second is only for the purpose of distinguishing the technical features, it cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features relation.

In the description of the present invention, unless otherwise clearly defined, words such as setting, installation, and connection should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

An infrared panoramic image stitching method, an infrared panoramic monitoring device and electronic equipment according to an embodiment of the present invention will be described below with reference to FIGS. 1 to 8 .

As shown in Figure 1, the infrared panoramic image stitching method according to the embodiment of the first aspect of the present invention includes the following steps:

S100. Obtain infrared images taken by each camera 100;

S200. Establish a camera coordinate system corresponding to each camera 100, and use the center of gravity of the origin of each camera coordinate system as the origin to establish a world coordinate system;

S300. Taking the origin of the world coordinate system as the center of the sphere and taking r as the radius to establish a spherical surface, where r is obtained by calibrating the camera 100;

S400. Establish an output plane in the world coordinate system, obtain the mapping relationship between the points on the output plane and the points on the spherical surface, and obtain the first coordinate according to the mapping relationship. The first coordinate is the coordinate of the spherical surface mapping point in the world coordinate system, Spherical mapping points are points on the sphere corresponding to points on the output plane;

S500. Obtain the RT relationship between the world coordinate system and the camera coordinate system, and convert the first coordinate into a second coordinate according to the RT relationship. The second coordinate is the coordinate point of the spherical mapping point coordinate in the camera coordinate system, and the RT relationship is the world coordinate. The coordinate transformation relationship between the camera coordinate system and the camera coordinate system;

S600. Convert the second coordinates into pixel coordinate points, use the pixel value of the infrared image corresponding to the pixel coordinate point as a target pixel value, and the target pixel value is a pixel value of a point on an output plane, to obtain an infrared panoramic image.

By obtaining the infrared images taken by each camera 100, establish a camera coordinate system corresponding to each camera 100, use the center of gravity of the origin of each camera coordinate system as the origin, establish a world coordinate system, and use the origin of the world coordinate system as the center of the sphere to establish a spherical surface , establish the output plane in the world coordinate system, obtain the mapping relationship between the points on the output plane and the points on the spherical surface, and obtain the first coordinate according to the mapping relationship. The first coordinate is the coordinate of the spherical surface mapping point in the world coordinate system, and the spherical surface The mapping point is a point on the spherical surface corresponding to the point on the output plane, and the RT relationship between the world coordinate system and the camera coordinate system is obtained. According to the RT relationship, the first coordinate is converted into the second coordinate, and the second coordinate is the coordinate of the spherical mapping point. The coordinate point in the camera coordinate system, the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system, the second coordinate is converted into a pixel coordinate point, and the pixel value of the infrared image corresponding to the pixel coordinate point is used as the target pixel value, the target The pixel value is the pixel value of the point on the output plane, and the infrared panoramic image is obtained. Only the pixel value corresponding to each point on the output plane is calculated, and there is no need to process and calculate all the data of the infrared image, which reduces the amount of data calculation, and the image splicing speed is fast. .

In some embodiments of the present invention, the method for determining the origin (x0, y0, z0) of the world coordinate system is: select the camera coordinate system of one camera 100 as the transition world coordinate system, and then according to the RT of the central camera 100 and other cameras 100 The origin of the camera coordinate system of the other cameras 100 is transformed into the transition world coordinate system by the following calculation formula:

[X, Y, Z] · R · T = [X', Y', Z']

Among them, [X, Y, Z] are the origin coordinates of each camera coordinate system, [X′, Y′, Z′] are the coordinates of the origin coordinates of each camera coordinate system in the transition world coordinate system, where R is the rotation Matrix, T is the translation vector;

Use the following calculation formula to find the center of gravity for the coordinates of the origin of all camera coordinate systems in the transition world coordinate system:

Wherein, N is the number of cameras 100, X _OW is the X-axis coordinate of the center of gravity, Y _OW is the Y-axis coordinate of the center of gravity, Z _OW is the Z-axis coordinate of the center of gravity, and the coordinates of the center of gravity (X _OW , Y _Ow , Z _Ow ) as the origin to establish the world coordinate system.

As shown in Figure 2, in some embodiments of the present invention, in step S400, the mapping relationship between points on the output plane and points on the spherical surface is obtained, and the mapping relationship is a linear projection mapping relationship. According to the linear projection mapping relationship, the first Coordinates, the first coordinate is the coordinate of the spherical mapping point in the world coordinate system, the spherical mapping point is the point on the spherical surface corresponding to the point of the output plane, including the following steps:

S410. Obtain the coordinates of the four vertices P1, P2, P3 and P4 of the output plane in the world coordinate system;

S420. Sampling at equal intervals among P1, P2, P3, and P4 to obtain the coordinates of each point on the output plane in the world coordinate system;

S430. Project each point on the output plane to a spherical surface, and obtain a first coordinate according to a projection relationship between each point on the output plane and the spherical surface.

In some embodiments of the present invention, it is assumed that the vertical viewing angle width of the output plane is 2β, and the horizontal viewing angle width is 2α. In the initial state, the coordinates of each point P1, P2, P3, and P4 in the world coordinate system are as follows:

Among them, p _w1 is the coordinate of P1 in the world coordinate system, p _w2 is the coordinate of P2 in the world coordinate system, p _w3 is the coordinate of P3 in the world coordinate system, P _w4 is the coordinate of P4 in the world coordinate system, r is the radius of the spherical surface, and the optimal value of r can be obtained during stereo calibration of the camera 100 (the distance from the camera 100 to the checkerboard), and can also be manually set as a parameter (that is, setting the splicing distance).

The user needs to set the display resolution of the output plane (horizontal resolution/vertical resolution). After setting the display resolution of the output plane, the output plane composed of P _w1 , P _w2 , P _w3 , and P _w4 in the world coordinate system can be For image output, in order to obtain the coordinates of the coordinate points on the entire output plane in the world coordinate system, the four points P _w1 , P _w2 , p _w3 , and p _w4 are sampled at equal intervals according to Δx, Δy, and Δz, and quantized to obtain the output The coordinates of all points on the plane in the world coordinate system.

When the user adjusts the attitude angle, such as rotating the angles ψ, Φ, and θ around the X-axis, Y-axis, and Z-axis in turn, the total rotation matrix R is equal to the product of each rotation matrix:

R＝R(ψ)·R(Φ)·R(θ)

Among them, R(ψ) is the rotation matrix around the X axis, R(Φ) is the rotation matrix around the Y axis, and R(θ) is the rotation matrix around the Z axis.

The coordinates of p _w1 , p _w2 , P _w3 , and P _w4 after attitude angle transformation can be obtained according to the total rotation matrix.

Project all points in the output plane to the spherical surface, and obtain the coordinates of the points on the corresponding spherical surface in the world coordinate system through the following calculation formula, and obtain the first coordinates (X _W , Y _W , Z _W ):

Among them, X _p is the X-axis coordinate of the point on the output plane in the world coordinate system, Y _p is the Y-axis coordinate of the point on the output plane in the world coordinate system, Z _p is the point on the output plane in the world coordinate system The Z-axis coordinates within.

The second coordinates are obtained according to the RT relationship between the world coordinate system and each camera coordinate system.

As shown in Figure 3, in some embodiments of the present invention, in step S400, according to some embodiments of the present invention, the mapping relationship between points on the output plane and points on the spherical surface is obtained, and the mapping relationship is an equidistant cylindrical projection mapping relationship , according to the mapping relationship from the cylindrical projection, the first coordinate is obtained. The first coordinate is the coordinate of the spherical mapping point in the world coordinate system. The spherical mapping point is the point on the spherical surface corresponding to the point of the output plane, including:

S410. Take N vertical lines and M horizontal lines on the output plane, the N vertical lines divide the output plane into equal parts horizontally, the M horizontal lines divide the output plane into equal parts vertically, and the N vertical lines and M horizontal lines form N *M first intersection points;

S420. Take N meridians and M latitude lines on the spherical surface, the N meridians divide the spherical surface into equal parts horizontally, the M horizontal lines divide the spherical surface into equal parts longitudinally, and the N meridians and M latitude lines form N*M second intersection points;

S430. Sequentially make one-to-one correspondence between the N*M first intersection points and the N*M second intersection points, to obtain first coordinates.

As shown in Figure 4, in some embodiments of the present invention, in step S400, the mapping relationship between the points on the output plane and the points on the spherical surface is obtained, the mapping relationship is a 3D linear projection, and according to the 3D linear projection mapping relationship, the first Coordinates, the first coordinate is the coordinate of the spherical mapping point in the world coordinate system, the spherical mapping point is the point on the spherical surface corresponding to the point of the output plane, including the following steps:

S410. Establish an inscribed circular plane in the spherical surface, the radius of the inscribed circular plane is the same as the radius of the spherical surface, and the circumscribed square plane of the inscribed circular plane is an output plane;

S420. Obtain the coordinates of each point of the inscribed circle plane in the world coordinate system, that is, the coordinates of the intersection point of the circumscribed square plane and the inscribed circle plane in the world coordinate system, to obtain the first coordinate.

In some embodiments of the present invention, the resolution of the output plane is set to w*h. According to the following calculation formula, any point P on the output plane is obtained in the first quadrant of the Z-axis and Y-axis planes in the world coordinate system The coordinates of _w (m, n) in the world coordinate system:

当R>r，说明输出平面上该点不在内接圆平面内，可以直接丢弃。In the same way, it can be obtained, output the coordinates of the plane in other quadrants, and obtain the first coordinates, according to the formula

When R>r, it means that the point on the output plane is not in the inscribed circle plane and can be discarded directly.

When rotating the inscribed circle plane, the coordinates of each point in the new inscribed circle plane in the world coordinate system can be obtained according to the rotation matrix.

As shown in Figure 5, in some embodiments of the present invention, in step S600, the pixel value of the infrared image corresponding to the pixel coordinate point is used as the target pixel value, and the target pixel value is the pixel value of the point of the output plane to obtain an infrared panoramic image ,include:

S610. Perform image fusion on the overlapping areas of the infrared images corresponding to all the pixel coordinate points to obtain the target infrared image, and each pixel coordinate point in the overlapping area corresponds to the pixel values of multiple infrared images;

S620. Using the pixel value of the target infrared image corresponding to the pixel coordinate point as the target pixel value, the target pixel value is the pixel value of a point on the output plane, to obtain an infrared panoramic image.

In some embodiments of the present invention, as shown in Figure 7, the output plane corresponds to two infrared images, respectively the first infrared image and the second infrared image, the first infrared image includes the D1 area and the D2 area, and the second infrared image Including the D3 area and the D4 area, wherein, the D1 area and the D4 area correspond to different areas of the output plane, that is, the D1 area and the D4 area are non-overlapping areas, and the D2 area and the D3 area correspond to the same area on the output plane, that is, the D2 area and the D3 area The area is an overlapping area, and the infrared image of the D2 area and the infrared image of the D3 area are fused to obtain the infrared image of the fusion area, and the infrared images of the D1 area, the fusion area and the D4 area are sequentially overlaid on the output plane to obtain the target infrared image, and the pixel coordinates The pixel value of the target infrared image corresponding to the point is used as the target pixel value, and the target pixel value is the pixel value of the point on the output plane to obtain the infrared panoramic image. The D2 area and the D3 area are fused using a linear fusion algorithm, and dynamically adjusted according to the fusion weight. The fusion weight is determined according to the distance from the pixel on the infrared image to the edge of the infrared image. The closer the distance, the smaller the fusion weight, and the closer the distance. The farther the fusion weight is, the larger the fusion weight is to ensure smooth image transition in the overlapping area. If the overlapping area of the first infrared image and the second infrared image is too large, a mask picture (binarized rectangular picture) can be set to reduce the field of view of the infrared image obtained by the corresponding camera 100, reduce the number of fusion points, and improve fusion efficiency. .

As shown in FIG. 6, in some embodiments of the present invention, in step S610, image fusion is performed on overlapping regions of the respective infrared images corresponding to all pixel coordinate points to obtain the target infrared image, including:

S611. Acquire a low-light image corresponding to the infrared image;

S612. Perform enhancement processing on the low-light image and the infrared image to obtain the transitional low-light image and the transitional infrared image;

S613. Fusion the transitional low-light image and the transitional infrared image to obtain an enhanced infrared image;

S614. Perform image fusion on overlapping regions of the enhanced infrared images corresponding to all pixel coordinate points to obtain the target infrared image.

In some embodiments of the present invention, in step S614, the fusion method of the Laplacian pyramid is used to fuse the transition low-light image and the transition infrared image to obtain an enhanced infrared image, including:

Decompose the transition low-light image into a base layer image and a detail layer image;

Decompose the transitional infrared image into a base layer image and a detail layer image;

Fusing the base layer image of the transition low-light image and the base layer image of the transition infrared image to obtain the target base layer image;

The detail layer image of the transition low-light image and the detail layer image of the transition infrared image are fused to obtain the target detail layer image; the target base layer image and the target detail layer image are reconstructed to obtain an infrared panoramic image.

Through the two-scale decomposition of the transition low-light image and the transition infrared image, decompose it into the base layer image and the detail layer image, use the weighting coefficient method to fuse the base layer image, and use the weighting coefficient method to fuse the detail layer image, and finally The fusion images of each layer are reconstructed, and the image information on each scale is quickly and effectively fused to obtain an infrared panoramic image. The infrared panoramic image has rich image information, high definition and recognizability. Using GPU parallel acceleration computing method to stitch and fuse infrared panoramic images to improve the efficiency of stitching and fusion.

In step S612, the gamma image enhancement algorithm is used to enhance the low-light image and the infrared image to obtain the transitional low-light image and the transitional infrared image, including the following steps:

Preset gamma lookup table;

According to the gamma lookup table, the pixel value of the low-light image is mapped to the corresponding gamma-corrected pixel value to obtain the transitional low-light image;

According to the gamma lookup table, the pixel values of the infrared image are mapped to the corresponding gamma-corrected pixel values to obtain the transition infrared image.

The principle of the gamma image enhancement algorithm is as follows:

S100, normalization: convert the pixel value into a real number between 0 and 1, the calculation formula of normalization is:

f=(i+0.5)/256, where f is the normalized pixel value and i is the pixel value

S200. Pre-compensation: perform pre-compensation on the normalized value, the formula for calculating the pre-compensation is:

F=(1/gamma)*f, F is the pre-compensated pixel value

S300. Denormalization: reversely transform the pre-compensated real value into an integer value between 0 and 255, and the calculation formula for denormalization is:

P=F*256-0.5, P is the denormalized pixel value

Any pixel value in the low-light image and infrared image is limited to an integer from 0 to 255, and the gamma value takes a value between 0 and 10 according to the enhancement intensity requirements. After normalization, pre-compensation, and denormalization operations , the corresponding calculation result is unique, and also falls within the range of 0-255, greatly reducing the amount of calculation, fast processing speed, and improving the recognizability of images in dark areas.

As shown in FIG. 8, the infrared panoramic monitoring device according to the embodiment of the second aspect of the present invention includes: a camera 100, an image processing module 200, and a communication module 300. The camera 100 is used to collect low-light images and infrared images from multiple angles, The output end of the camera 100 is connected to the input end of the image processing module 200, and the image processing module 200 executes the infrared panoramic image stitching method as described above, and the output end of the image processing module 200 is connected to the input end of the communication module 300, and the communication module 300 is used for infrared The panoramic image is transmitted to the host.

By obtaining the infrared images taken by each camera 100, establish a camera coordinate system corresponding to each camera 100, use the center of gravity of the origin of each camera coordinate system as the origin, establish a world coordinate system, and use the origin of the world coordinate system as the center of the sphere to establish a spherical surface , establish the output plane in the world coordinate system, obtain the mapping relationship between the points on the output plane and the points on the spherical surface, and obtain the first coordinate according to the mapping relationship. The first coordinate is the coordinate of the spherical surface mapping point in the world coordinate system, and the spherical surface The mapping point is a point on the spherical surface corresponding to the point on the output plane, and the RT relationship between the world coordinate system and the camera coordinate system is obtained. According to the RT relationship, the first coordinate is converted into the second coordinate, and the second coordinate is the coordinate of the spherical mapping point. The coordinate point in the camera coordinate system, the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system, the second coordinate is converted into a pixel coordinate point, and the pixel value of the infrared image corresponding to the pixel coordinate point is used as the target pixel value, the target The pixel value is the pixel value of the point on the output plane, and the infrared panoramic image is obtained. Only the pixel value corresponding to each point on the output plane is calculated, and there is no need to process and calculate all the data of the infrared image, which reduces the amount of data calculation, and the image splicing speed is fast. , The monitoring and display screen on the host side has good real-time performance.

The number of cameras 100 is six, and the six cameras 100 are located on the same spherical surface. The shooting range of the six cameras 100 covers the upper hemisphere of the spherical surface, and can acquire horizontal 360-degree infrared images and low-light images. The camera 100 adopts a fisheye camera 100 with a wider shooting angle of view. Other numbers of cameras 100 can also be set according to application scenarios.

An electronic device according to an embodiment of the third aspect of the present invention includes: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by at least one processor, the above infrared panoramic image stitching is realized method.

By obtaining the infrared images taken by each camera 100, establish a camera coordinate system corresponding to each camera 100, use the center of gravity of the origin of each camera coordinate system as the origin, establish a world coordinate system, and use the origin of the world coordinate system as the center of the sphere to establish a spherical surface , establish the output plane in the world coordinate system, obtain the mapping relationship between the points on the output plane and the points on the spherical surface, and obtain the first coordinate according to the mapping relationship. The first coordinate is the coordinate of the spherical surface mapping point in the world coordinate system, and the spherical surface The mapping point is a point on the spherical surface corresponding to the point on the output plane, and the RT relationship between the world coordinate system and the camera coordinate system is obtained. According to the RT relationship, the first coordinate is converted into the second coordinate, and the second coordinate is the coordinate of the spherical mapping point. The coordinate point in the camera coordinate system, the RT relationship is the coordinate conversion relationship between the world coordinate system and the camera coordinate system, the second coordinate is converted into a pixel coordinate point, and the pixel value of the infrared image corresponding to the pixel coordinate point is used as the target pixel value, the target The pixel value is the pixel value of the point on the output plane, and the infrared panoramic image is obtained. Only the pixel value corresponding to each point on the output plane is calculated, and there is no need to process and calculate all the data of the infrared image, which reduces the amount of data calculation, and the image splicing speed is fast. .

The embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned embodiments, and within the scope of knowledge of those of ordinary skill in the art, various modifications can be made without departing from the spirit of the present invention. Variety.

Claims (10)

1. The infrared panoramic image stitching method is characterized in that, comprising: Obtain the infrared images taken by each camera (100); Establishing a camera coordinate system corresponding to each of the cameras (100), using the center of gravity of the origin of each of the camera coordinate systems as the origin to establish a world coordinate system; Taking the origin of the world coordinate system as the center of the sphere and taking r as the radius to establish a spherical surface, the r is obtained by calibrating the camera (100); Establish an output plane in the world coordinate system, obtain a mapping relationship between points on the output plane and points on the spherical surface, and obtain a first coordinate according to the mapping relationship, and the first coordinate is a spherical mapping point. Coordinates in the world coordinate system, the spherical mapping point is a point on the spherical surface corresponding to a point on the output plane; Obtain the RT relationship between the world coordinate system and the camera coordinate system, and convert the first coordinate into a second coordinate according to the RT relationship, and the second coordinate is the coordinate point of the spherical mapping point coordinate in the camera coordinate system , the RT relationship is a coordinate transformation relationship between the world coordinate system and the camera coordinate system; Converting the second coordinates into pixel coordinate points, using the pixel value of the infrared image corresponding to the pixel coordinate point as a target pixel value, the target pixel value is the pixel value of a point on the output plane, and obtaining an infrared panoramic image. 2. The infrared panoramic image stitching method according to claim 1, wherein the mapping relationship between the point on the output plane and the point on the spherical surface is acquired, and the first coordinate is obtained according to the mapping relationship , the first coordinate is the coordinate of a spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point of the output plane, including: Obtain the coordinates of the four vertices P1, P2, P3 and P4 of the output plane in the world coordinate system; Sampling at equal intervals between the P1, the P2, the P3, and the P4 to obtain the coordinates of each point on the output plane in the world coordinate system; Projecting each point on the output plane to the spherical surface, and obtaining the first coordinates according to a projection relationship between each point on the output plane and the spherical surface. 3. The infrared panoramic image stitching method according to claim 1, wherein the mapping relationship between the point on the output plane and the point on the spherical surface is acquired, and the first coordinate is obtained according to the mapping relationship , the first coordinate is the coordinate of a spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point of the output plane, including: Take N vertical lines and M horizontal lines on the output plane, the N vertical lines divide the output plane horizontally, the M horizontal lines divide the output plane vertically, and the N vertical lines divide the output plane into equal parts. The vertical lines and the M horizontal lines form N*M first intersections; N warps and M latitudes are taken on the spherical surface, and the N warps divide the spherical surface into equal parts horizontally, the M transverse lines divide the spherical surface into equal parts longitudinally, and the N said longitude lines and the M said parallel lines The weft lines form N*M second intersection points; The first coordinates are obtained by making a one-to-one correspondence between the N*M first intersections and the N*M second intersections in sequence. 4. The infrared panoramic image stitching method according to claim 1, wherein the mapping relationship between the point on the output plane and the point on the spherical surface is acquired, and the first coordinate is obtained according to the mapping relationship , the first coordinate is the coordinate of a spherical mapping point in the world coordinate system, and the spherical mapping point is a point on the spherical surface corresponding to the point of the output plane, including: Establishing an inscribed circle plane in the spherical surface, the radius of the inscribed circle plane is the same as the radius of the spherical surface, and the circumscribed square plane of the inscribed circle plane is the output plane; Obtain the coordinates of each point of the inscribed circle plane in the world coordinate system, that is, the coordinates of the intersection point of the circumscribed square plane and the inscribed circle plane in the world coordinate system, and obtain the first coordinate . 5. The infrared panoramic image stitching method according to claim 1, wherein the pixel value of the infrared image corresponding to the pixel coordinate point is used as the target pixel value, and the target pixel value is the value of the output plane. The pixel value of the point to obtain the infrared panoramic image, including: performing image fusion on overlapping areas of the infrared images corresponding to all the pixel coordinate points to obtain a target infrared image, and each of the pixel coordinate points in the overlapping area corresponds to pixel values of multiple infrared images; Taking the pixel value of the target infrared image corresponding to the pixel coordinate point as a target pixel value, and the target pixel value is a pixel value of a point on the output plane to obtain an infrared panoramic image. 6. The infrared panorama image stitching method according to claim 5, wherein the image fusion is performed on overlapping areas of each of the infrared images corresponding to all the pixel coordinate points to obtain the target infrared image, comprising: Acquiring a low-light image corresponding to the infrared image; performing enhancement processing on the low-light image and the infrared image to obtain a transitional low-light image and a transitional infrared image; fusing the transitional low-light image and the transitional infrared image to obtain an enhanced infrared image; Image fusion is performed on overlapping areas of the enhanced infrared images corresponding to all the pixel coordinate points to obtain the target infrared image. 7. The infrared panorama image stitching method according to claim 6, wherein said step of enhancing said low-light image and said infrared image to obtain a transitional low-light image and a transitional infrared image comprises the following steps: Preset gamma lookup table; According to the gamma lookup table, the pixel value of the low-light image is mapped to the corresponding gamma-corrected pixel value to obtain a transitional low-light image; According to the gamma lookup table, the pixel values of the infrared image are mapped to corresponding gamma corrected pixel values to obtain a transition infrared image. 8. The infrared panoramic image stitching method according to claim 6, characterized in that merging the transition low-light image and the transition infrared image to obtain an enhanced infrared image comprises: Decomposing the transition low-light image into a base layer image and a detail layer image; Decomposing the transitional infrared image into a base layer image and a detail layer image; fusing the base layer image of the transition low-light image and the base layer image of the transition infrared image to obtain a target base layer image; merging the detail layer image of the transition low-light image and the detail layer image of the transition infrared image to obtain a target detail layer image; An infrared panoramic image is obtained by reconstructing the target base layer image and the target detail layer image. 9. The infrared panoramic monitoring device is characterized in that it comprises: A camera (100), the camera (100) is used to collect low-light images and infrared images from multiple angles; An image processing module (200), the output end of the camera (100) is connected to the input end of the image processing module (200), and the image processing module (200) performs the method described in any one of claims 1 to 8 Infrared panoramic image stitching method; A communication module (300), the output end of the image processing module (200) is connected to the input end of the communication module (300), and the communication module (300) is used to transmit the infrared panoramic image to a host end. 10. An electronic device, characterized in that it comprises: at least one processor; at least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, the infrared panoramic image stitching method according to any one of claims 1 to 8 is realized.

Images