CN105979203A - Multi-camera cooperative monitoring method and device - Google Patents

Abstract

Embodiments of the present invention provide a multi-camera collaborative monitoring method and device. The multi-camera collaborative monitoring method includes: combining multiple cameras on the monitoring site into multiple camera combinations; obtaining multiple frames of two-dimensional image images collected by each camera combination in each acquisition period, and performing a process on each frame of two-dimensional image images Foreground extraction and projection to obtain a 3D foreground image; the 3D foreground image combined by each camera at the same acquisition time is fused to form a 3D fusion foreground image; according to the 3D fusion foreground image at each acquisition time, and the points in the real world According to the change of the corresponding position in the 3D fusion foreground map, the target object and its spatial distribution characteristics and operating distribution characteristics are determined; and it is judged whether there is a target scene. The invention makes full use of the three-dimensional space information of the monitoring site, and can obtain better monitoring results even in poor illumination, large field of view, and long-distance monitoring.

Description

A multi-camera collaborative monitoring method and device

technical field

Embodiments of the present invention relate to the field of intelligent monitoring, and more specifically, embodiments of the present invention relate to a multi-camera collaborative monitoring method and device.

Background technique

This section is intended to provide a background or context for implementations of the invention that are recited in the claims. The descriptions herein are not admitted to be prior art by inclusion in this section.

In the research field of intelligent monitoring for equipment and personnel safety, the discovery and analysis of abnormal patterns has always been a key research direction, and has broad application prospects. However, there are many challenges in real-time and accurate detection of people and objects in real scenes, and judging their behavior, such as illumination changes under irregular natural conditions, the impact of camera deployment angle of view, and resolution reduction caused by long distances. All are unfavorable factors that affect accurate detection.

Based on the camera to detect scene abnormality and alarm, the mainstream method is mainly based on the abnormal mode detection of the color or grayscale two-dimensional image information obtained by the camera. One of the major categories is the method based on human detection and tracking. This method is realized through the detection and tracking of a single person, which can roughly calculate the trajectory of the person in the two-dimensional image coordinate system, and finally calculate the Trajectory analysis is used to analyze abnormal events. This method is usually only suitable for low-density, medium and short-distance scenes. In real scenes, cluttered backgrounds, occlusions between people, and changes in illumination usually make detection Tracking algorithms fail to give accurate results. Another type is the method based on the underlying features. This method generally first establishes the background model of the scene, and then uses the background subtraction method to extract the foreground in the scene, and then extracts the features of the foreground area, such as the outline of the foreground area, texture, and time of the foreground. Domain features are used as input, and abnormal events are detected by training methods. The effect is also subject to factors such as changes in illumination in the environment, image resolution, and the distance between the event and the camera.

At present, there are methods of using depth information to monitor abnormalities. Most of these methods are based on active light, which can accurately obtain depth values in a small range (generally 1 to 8 meters), and can be used for indoor monitoring.

Contents of the invention

However, the existing methods of using depth information for abnormal monitoring have the disadvantages of being greatly affected by outdoor lighting, small working distance, and small field of view, and cannot be used for outdoor large-scale, wide-angle, and long-distance abnormalities. monitor.

To this end, the present invention provides a multi-camera cooperative monitoring method and device, which are used to overcome the limitation of existing monitoring methods in outdoor monitoring of abnormalities in a large range, wide angle, and long distance.

In the first aspect of the implementation method of the present invention, a multi-camera collaborative monitoring method is provided, including:

Step A, according to the way that at least two cameras form a camera combination, multiple cameras deployed at the monitoring site are combined into multiple camera combinations;

Step B, obtaining the two-dimensional image images collected by each camera combination in each acquisition period, and performing the processing of steps B1 to B3 on the multi-frame two-dimensional image images sequentially acquired by the current camera combination in the current acquisition period;

Step B1, extracting the foreground from the multi-frame two-dimensional image images to obtain a plurality of two-dimensional foreground images; projecting the plurality of two-dimensional foreground images into a three-dimensional space to obtain a plurality of three-dimensional foreground images; wherein, the current acquisition cycle There is a one-to-one correspondence between each acquisition moment, the multi-frame two-dimensional image map, the plurality of two-dimensional foreground images, and the plurality of three-dimensional foreground images;

Step B2, calculating the change of the corresponding positions of the points in the real world in the plurality of two-dimensional foreground images; wherein, the points in the real world are each two-dimensional foreground points in the two-dimensional foreground images A corresponding point in the real world; the two-dimensional foreground point is a pixel in the two-dimensional foreground image;

Step B3, according to the changes in the corresponding positions of points in the real world in the plurality of two-dimensional foreground images, and according to the correspondence between the two-dimensional foreground images and the three-dimensional foreground images, calculate The changes of the corresponding positions of the points in the plurality of three-dimensional foreground images;

Step C: For each collection moment of the current collection cycle, according to the rule of merging all 3D foreground points corresponding to the same point in the real world into a 3D fusion foreground point, combine the corresponding 3D points of each camera at the monitoring site at the same collection moment The 3D foreground points in the foreground image are fused, and all the 3D fused foreground points obtained after fusion are combined to form a 3D fused foreground image at the acquisition moment; wherein, the 3D foreground points are volumes of the 3D foreground image Prime point;

Step D, according to the change of the corresponding positions of the points in the real world in the plurality of 3D foreground images, and according to the corresponding relationship between each 3D fused foreground point and each 3D foreground point fused into each 3D fused foreground point , calculating the change of the corresponding position of the point in the real world in the multiple 3D fusion foreground images at each acquisition moment of the current acquisition cycle;

Step E, according to the multiple 3D fusion foreground maps at each collection moment of the current collection cycle, and the changes in the corresponding positions of points in the real world in the multiple 3D fusion foreground maps at each collection moment of the current collection cycle, determine the target object, and determining the spatial distribution characteristics and operational distribution characteristics of the target object;

Step F, judging whether there is a target scene according to the spatial distribution characteristics and operation distribution characteristics of the target object;

Step G, output the judgment result, and call the police when there is a target situation.

In the second aspect of the implementation method of the present invention, a multi-camera collaborative monitoring device is provided, including:

The camera division module is used to combine multiple cameras deployed at the monitoring site into multiple camera combinations according to the way that at least two cameras form a camera combination;

An image map acquisition module, configured to obtain a two-dimensional image map collected by each camera combination in each acquisition cycle;

The image processing module is used to perform image processing on the multi-frame two-dimensional image images sequentially collected by the current camera combination in the current acquisition cycle;

The image processing module further includes:

The first image processing module is used to extract the foreground from the multi-frame two-dimensional image images to obtain a plurality of two-dimensional foreground images;

The second image processing module is configured to project the plurality of two-dimensional foreground images into a three-dimensional space to obtain a plurality of three-dimensional foreground images; wherein, each acquisition moment of the current acquisition cycle, the plurality of frames of two-dimensional image images, the plurality of The plurality of two-dimensional foreground images and the plurality of three-dimensional foreground images have a one-to-one correspondence;

The third image processing module is used to calculate the change of the corresponding position of the point in the real world in the plurality of two-dimensional foreground images; wherein, the point in the real world is each of the two-dimensional foreground images A point corresponding to a two-dimensional foreground point in the real world; the two-dimensional foreground point is a pixel in the two-dimensional foreground image;

The fourth image processing module is configured to change the corresponding positions of points in the real world in the plurality of two-dimensional foreground images, and according to the corresponding relationship between the two-dimensional foreground image and the three-dimensional foreground image , calculating changes in positions corresponding to points in the real world in the plurality of three-dimensional foreground images;

The fusion processing module is used for combining all the cameras at the monitoring site at the same collection moment according to the rule of merging all 3D foreground points corresponding to the same point in the real world into a 3D fusion foreground point at each collection moment of the current collection cycle The 3D foreground points in the corresponding 3D foreground image are fused, and all the 3D fused foreground points obtained after fusion are combined to form a 3D fused foreground map at the acquisition moment; wherein, the 3D foreground points are the 3D foreground The voxel point of the image;

The displacement calculation module is used for changing the corresponding positions of points in the real world in the plurality of 3D foreground images, and according to each 3D fused foreground point and each 3D foreground point fused into each 3D fused foreground point The corresponding relationship of the points in the real world is calculated at each acquisition moment of the current acquisition cycle and the corresponding position changes in the multiple 3D fusion foreground images;

The target search module is used for multiple 3D fusion foreground maps at each collection moment of the current collection cycle, and changes in the corresponding positions of points in the real world in the multiple 3D fusion foreground maps at each collection moment of the current collection cycle , determining a target object, and determining a spatial distribution feature and an operational distribution feature of the target object;

A judging module, configured to judge whether a target scenario occurs according to the spatial distribution characteristics and operation distribution characteristics of the target object;

The output module is used for outputting judgment results and alarming when a target situation occurs.

With the help of the above technical solution, the present invention synergistically utilizes the images collected by all the cameras on the monitoring site, and makes full use of the three-dimensional space information of the captured scene. In a large-scale scene, the present invention can also provide more accurate detection results, improve the detection accuracy and robustness of outdoor monitoring, and is especially suitable for monitoring fields with few patrol personnel and poor weather conditions.

Description of drawings

The above and other objects, features and advantages of exemplary embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are shown by way of illustration and not limitation, in which:

Fig. 1 is a schematic diagram of the application of the present invention to the oil and gas well equipment safety monitoring scene;

FIG. 2 is a schematic flow diagram of a multi-camera collaborative monitoring method provided by the present invention;

Fig. 3 is a schematic diagram of obtaining a two-dimensional foreground image and a three-dimensional foreground image by performing image processing on a two-dimensional image image collected by a camera combination at each collection moment;

Fig. 4 is a schematic diagram of merging the three-dimensional foreground images combined by various cameras at the same acquisition moment into a three-dimensional fusion foreground image;

Fig. 5 is a schematic diagram of input and output of a multi-camera collaborative monitoring device;

Fig. 6 is a structural block diagram of a multi-camera cooperative monitoring device;

In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

detailed description

The principle and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present invention, rather than to limit the scope of the present invention in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Those skilled in the art know that the embodiments of the present invention can be implemented as a system, device, device, method or computer program product. Therefore, the present disclosure may be embodied in the form of complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

It should be noted that the term "2D foreground point" referred to herein refers to a pixel point of a 2D foreground image, and the term "3D foreground point" refers to a voxel point of a 3D foreground image.

According to an embodiment of the present invention, a multi-camera collaborative monitoring method and device are proposed.

The principle and spirit of the present invention will be explained in detail below with reference to several representative embodiments of the present invention.

Summary of the invention

The present invention makes full use of the existing multi-channel cameras to monitor the abnormal situation in the scene in real time. First, combine the cameras on the monitoring site in pairs to obtain multiple camera combinations, and obtain the two-dimensional image images collected by each camera combination in real time. Secondly, for the 2D images collected by each camera combination, the processing of calculating 2D depth maps, extracting 2D foreground points and projecting them to obtain 3D foreground points, and calculating 3D foreground optical flow are performed sequentially; 3D fused foreground point cloud is obtained by fusion processing of scenic spot clouds; bird's eye view and 3D fused foreground optical flow are obtained according to 3D fused foreground point cloud and 3D foreground optical flow; finally, template matching and pattern classification are performed on the bird's eye view and 3D fused foreground optical flow Wait for processing, judge and output the target scenario.

After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.

Overview of application scenarios

As shown in Figure 1, a schematic diagram of the application of the present invention to the safety monitoring scene of oil and gas well equipment is described, that is, the oil and gas well equipment is monitored through multiple cameras (camera a, camera b, camera c and camera d) that have been deployed on the oil and gas well site and personnel. Within the monitoring range of multiple cameras, personnel and equipment are monitored in real time, and an alarm can be issued once any abnormal behavior occurs. Abnormal behaviors here include but are not limited to someone fighting, someone running, someone stealing oil and gas well equipment, someone placing items on oil and gas well equipment, someone entering an area that should not be entered, someone falling, someone holding dangerous items (machetes, long knives) )Wait.

It should be noted that Fig. 1 only schematically shows an application scenario of the present invention, and the present invention is not limited to the scenario shown in Fig. 1 , and those skilled in the art should understand that the present invention can also be applied to Any other monitoring scenarios, such as logistics warehouse supervision, military border monitoring, etc.

exemplary method

The following describes the multi-camera collaborative monitoring method provided by the present invention with reference to FIG. 2 in combination with the application scenario of FIG. 1 .

It should be noted that the above application scenarios are only shown for the purpose of understanding the spirit and principle of the present invention, and the implementation manners of the present invention are not limited in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable scene.

As shown in Figure 2, the multi-camera collaborative monitoring method provided by the present invention includes:

Step S1, combining multiple cameras deployed at the monitoring site into multiple camera combinations in such a way that at least two cameras form a camera combination;

Step S2, obtaining the two-dimensional image images collected by each camera combination in each acquisition period, and performing the processing of steps S21 to S23 on the multi-frame two-dimensional image images sequentially acquired by the current camera combination in the current acquisition period;

Step S21, extracting the foreground from the multi-frame two-dimensional image images to obtain multiple two-dimensional foreground images; projecting the multiple two-dimensional foreground images into a three-dimensional space to obtain multiple three-dimensional foreground images; wherein, the current acquisition cycle There is a one-to-one correspondence between each acquisition moment, the multi-frame two-dimensional image map, the plurality of two-dimensional foreground images, and the plurality of three-dimensional foreground images;

Step S22, calculating the change of the corresponding positions of the points in the real world in the plurality of two-dimensional foreground images; wherein, the points in the real world are each two-dimensional foreground points in the two-dimensional foreground images A corresponding point in the real world; the two-dimensional foreground point is a pixel in the two-dimensional foreground image;

Step S23, according to the changes in the corresponding positions of the points in the real world in the plurality of two-dimensional foreground images, and the corresponding relationship between the two-dimensional foreground images and the three-dimensional foreground images, calculate the points in the real world Changes of the corresponding positions of the points in the plurality of three-dimensional foreground images;

Step S3, for each collection moment of the current collection cycle, according to the rule of merging all 3D foreground points corresponding to the same point in the real world into a 3D fusion foreground point, combine the corresponding 3D points of each camera at the monitoring site at the same collection moment The 3D foreground points in the foreground image are fused, and all the 3D fused foreground points obtained after fusion are combined to form a 3D fused foreground image at the acquisition moment; wherein, the 3D foreground points are volumes of the 3D foreground image Prime point;

Step S4, according to the change of the corresponding positions of points in the real world in the plurality of 3D foreground images, and according to the corresponding relationship between each 3D fused foreground point and each 3D foreground point fused into each 3D fused foreground point , calculating the change of the corresponding position of the point in the real world in the multiple 3D fusion foreground images at each acquisition moment of the current acquisition cycle;

Step S5, according to the multiple 3D fusion foreground maps at each collection time of the current collection cycle, and the changes in the corresponding positions of the points in the real world in the multiple 3D fusion foreground maps at each collection time of the current collection cycle, determine the target object, and determining the spatial distribution characteristics and operational distribution characteristics of the target object;

Step S6, according to the spatial distribution characteristics and operation distribution characteristics of the target object, it is judged whether there is a target scene;

Step S7, outputting the judgment result, and alarming when a target situation occurs.

The multi-camera cooperative monitoring method shown in FIG. 1 will be introduced below through the image processing process shown in FIGS. 3 to 4 .

Assume that there are three camera combinations of Z1, Z2, and Z3 in the monitoring site, and each acquisition time of the current acquisition cycle is T1, T2, and T3 respectively.

As shown in Fig. 3, the camera combination Z1 collects two-dimensional images sequentially at acquisition times T1, T2, and T3, extracts the foreground from these three frames of two-dimensional image images, and obtains three corresponding two-dimensional foreground images. A two-dimensional foreground image is projected into a three-dimensional space, and three corresponding three-dimensional foreground images are obtained. Among them, the specific process of projecting the 2D foreground image into the 3D space to obtain the 3D foreground image is: project each 2D foreground point in the 2D foreground image into the 3D space to obtain the 3D foreground point, and all the 2D foreground points correspond to The three-dimensional foreground points are combined to form a three-dimensional foreground image. The 3D foreground image is actually a point cloud, which corresponds to the object corresponding to the 2D foreground image in the real world.

For point P and point Q in the real world, their positions in the two-dimensional foreground image at the acquisition time T1 are P1 and Q1 respectively, and their positions in the three-dimensional foreground image at this time are respectively P1' and Q1'; The positions in the 2D foreground image at time T2 are P2 and Q2 respectively, and the positions in the 3D foreground image at this moment are P2' and Q2' respectively; the positions in the 2D foreground image at acquisition time T3 are P3 and Q2 respectively. Q3, the positions in the three-dimensional foreground image at this moment are P3' and Q3' respectively.

According to the positions of P1, P2, and P3, the change of the corresponding position of the point P in the real world in the two-dimensional foreground image at each acquisition moment can be calculated. This change can be calculated by the two-dimensional vector Indicates; according to the positions of Q1, Q2, and Q3, the change of the corresponding position of the point Q in the real world in the two-dimensional foreground image at each acquisition moment can be calculated, and this change can be expressed by a two-dimensional vector express.

According to the corresponding relationship between P1 and P1', the corresponding relationship between P2 and P2', and the two-dimensional vector It is possible to determine the change of the corresponding position of the point P in the real world in the three-dimensional foreground image at the acquisition time T1 and T2, that is, to determine the three-dimensional vector Similarly, according to the corresponding relationship between P2 and P2', the corresponding relationship between P3 and P3', and the two-dimensional vector It is possible to determine the change of the corresponding position of the point P in the real world in the three-dimensional foreground image at the acquisition time T2 and T3, that is, to determine the three-dimensional vector Based on the above, the change of the corresponding position of the point P in the real world in the three-dimensional foreground image at each acquisition time can be determined. Similarly, it is possible to determine the change of the corresponding position of the point Q in the real world in the three-dimensional foreground image at each acquisition moment.

As shown in Figure 4, at the acquisition time T1, the corresponding positions of point X and point Y in the real world in the 3D foreground image of camera combination Z1 are X-Z1-T1 and Y-Z1-T1 respectively, and in the camera combination The corresponding positions in the 3D foreground image of Z2 are X-Z2-T1, Y-Z2-T1, and the corresponding positions in the 3D foreground image of camera combination Z3 are X-Z3-T1, Y-Z3-T1 . At the acquisition time T1, all the 3D foreground points corresponding to point X in the real world are fused into a 3D fused foreground point, denoted as Merge all 3D foreground points corresponding to point Y in the real world into a 3D fused foreground point, denoted as similar to and All such 3D fused foreground points constitute the 3D fused foreground map at the acquisition time T1.

Similarly, at the acquisition time T2, the corresponding positions of point X and point Y in the real world in the 3D foreground image of camera combination Z1 are X-Z1-T2 and Y-Z1-T2 respectively, and in the 3D foreground image of camera combination Z2 The corresponding positions in the foreground image are respectively X-Z2-T2 and Y-Z2-T2, and the corresponding positions in the 3D foreground image of the camera combination Z3 are respectively X-Z3-T2 and Y-Z3-T2. At the acquisition time T2, all the 3D foreground points corresponding to the point X in the real world are fused into a 3D fused foreground point, denoted as Merge all 3D foreground points corresponding to point Y in the real world into a 3D fused foreground point, denoted as similar to and All such 3D fused foreground points constitute the 3D fused foreground map at the acquisition time T1.

Similarly, at the acquisition time T3, the corresponding positions of point X and point Y in the real world in the 3D foreground image of camera combination Z1 are respectively X-Z1-T3 and Y-Z1-T3, and in the 3D foreground image of camera combination Z2 The corresponding positions in the foreground image are respectively X-Z2-T3 and Y-Z2-T3, and the corresponding positions in the 3D foreground image of the camera combination Z3 are respectively X-Z3-T3 and Y-Z3-T3. At the acquisition time T3, all the 3D foreground points corresponding to the point X in the real world are fused into a 3D fused foreground point, denoted as Merge all 3D foreground points corresponding to point Y in the real world into a 3D fused foreground point, denoted as similar to and All such 3D fused foreground points constitute the 3D fused foreground map at the acquisition time T1.

Combining FIG. 3 and FIG. 4, it can be seen that the relationship among the three-dimensional foreground points X-Z1-T1, X-Z1-T2, and X-Z1-T3 is consistent with the relationship among the three-dimensional foreground points P1', P2', and P3'. According to the previous description, the change of the corresponding position of the point P in the real world in the three-dimensional foreground image at the acquisition time T1, T2 and T3 has been calculated, that is, the three-dimensional vector In other words, the changes of the positions of the three-dimensional foreground points X-Z1-T1, X-Z1-T2, and X-Z1-T3 here can be calculated and recorded as three-dimensional vectors

Similarly, the changes in the positions of the three-dimensional foreground points X-Z2-T1, X-Z2-T2, and X-Z2-T3 can also be calculated and recorded as three-dimensional vectors

Similarly, the changes in the positions of the three-dimensional foreground points X-Z3-T1, X-Z3-T2, and X-Z3-T3 can also be calculated and recorded as three-dimensional vectors

At the collection time T1, the point X in the real world corresponds to the 3D foreground points X-Z1-T1, X-Z2-T1, X-Z3-T1; at the collection time T2, the point X in the real world corresponds to the 3D foreground points X-Z1-T2, X-Z2-T2, X-Z3-T2; At the acquisition time T3, the point X in the real world corresponds to the 3D foreground point X-Z1-T3, X-Z2-T3, X-Z3- T3;

Fusion of foreground points according to 3D Correspondence with the three-dimensional foreground points X-Z1-T1, X-Z2-T1, X-Z3-T1, Correspondence with the three-dimensional foreground points X-Z1-T2, X-Z2-T2, X-Z3-T2, Correspondence with the three-dimensional foreground points X-Z1-T3, X-Z2-T3, X-Z3-T3, and according to the three-dimensional vector 3D fusion foreground point The change of the location, you can use the three-dimensional vector Indicates the change of the corresponding position of the point X in the real world in the 3D fusion foreground map at each acquisition moment.

The present invention synergistically utilizes the images collected by all the cameras on the monitoring site, fully utilizes the three-dimensional spatial information of the captured scene, and can provide more accurate detection even in a large-scale scene with poor illumination, large wide-angle and long-distance As a result, the detection accuracy and robustness of outdoor monitoring are improved, and it is especially suitable for monitoring fields with few patrol personnel and poor weather conditions, such as oil field well operation sites, logistics warehouse supervision, and military border monitoring.

Each step in the method is introduced in detail with reference to FIGS. 3 to 4 .

In step S1, multiple cameras deployed at the monitoring site are combined into multiple camera combinations in such a manner that at least two cameras form a camera combination.

The purpose of forming the camera combination in this step is to use the images collected by each camera in the camera combination to calculate the depth value of the captured scene.

During specific implementation, optionally, the cameras on the monitoring site may be grouped in pairs to form several camera combinations. The two 2D images collected by the two cameras in the camera combination are respectively equivalent to the 2D images seen by the human eyes, and can be used to calculate the depth value, that is, the pixel points in the 2D image in the real world. The distance from the corresponding point (point on the surface of the object in the captured scene) to the camera.

Based on the three-dimensional vision principle of the human eye, it is necessary to calculate the depth value from the two-dimensional image captured by the two cameras in the camera combination, which requires that the scenes covered by the two cameras in the same camera combination need to have a certain intersection . In general, the scenes covered by the adjacent cameras will overlap. Based on this consideration, any two cameras at the monitoring site that are less than a preset distance away from each other can be combined into a camera combination during specific implementation. For example, the four cameras a, b, c, and d in Figure 1 are paired according to their installation positions, and finally three camera combinations (a, b), (b, c), (c, d) are obtained.

In this step, a remote terminal unit (Remote Terminal Unit, RTU) can be used to obtain image frames collected by each camera based on a Network Time Protocol (Network Time Protocol, NTP), that is, a two-dimensional image map.

According to different functions of the camera, the collected two-dimensional image may be a grayscale image or a color image (such as an RGB color image).

In the actual monitoring site, the models and parameters of each camera may be different, and the images obtained by shooting may also be different in size and shape. Considering this, during specific implementation, the present invention can also be used for the two-dimensional images collected by each camera. Distortion correction and alignment processing. For example, in this step, a checkerboard-based method can be used to obtain the distortion matrix, camera internal parameters, and external parameters of each camera, and based on the obtained distortion matrix, internal parameters, and external parameters of each camera, perform distortion correction on the two-dimensional image captured by the corresponding camera and alignment processing, the specific implementation can refer to the method provided by OpenCV (Open Source Computer Vision Library), which will not be repeated in this article.

Step S2, obtaining the two-dimensional image images collected by each camera combination in each acquisition cycle, and performing the image processing in steps S21 to S23 on the multi-frame two-dimensional image images sequentially collected by the current camera combination in the current acquisition cycle:

Step S21, extracting the foreground from the multi-frame two-dimensional image images to obtain multiple two-dimensional foreground images; projecting the multiple two-dimensional foreground images into a three-dimensional space to obtain multiple three-dimensional foreground images; wherein, the current acquisition cycle There is a one-to-one correspondence between each acquisition moment of the multiple frames of two-dimensional image images, the multiple two-dimensional foreground images, and the multiple three-dimensional foreground images.

Specifically, when completing the process of extracting the foreground from the two-dimensional image in this step, background modeling (for example, static background modeling or mixed Gaussian background modeling) can be used first, and then the method of background subtraction can be used.

Specifically, in this step, when completing the process of projecting the 2D foreground image into the 3D space to obtain the 3D foreground image, it is necessary to consider the conversion relationship between the 2D space imaged by the camera and the 3D space of the real world.

Optionally, in step S21, the two-dimensional foreground image can be projected into three-dimensional space according to the process of step S211 to step S215:

Step S211, determining the two-dimensional coordinates of each two-dimensional foreground point in the two-dimensional image map corresponding to the two-dimensional foreground image;

Step S212, determining the depth value of each 2D foreground point in the 2D depth map corresponding to the 2D foreground image. Among them, the two-dimensional depth image and the two-dimensional image image collected by the camera use the same coordinate axes. For pixels with the same coordinates, the pixel value in the two-dimensional image image is the image information of the pixel point, while the two-dimensional depth image The pixel value in the figure is the distance from the corresponding point of the pixel in the real world to the camera (that is, the depth value).

When calculating the depth value, traditional block matching, dynamic programming method, graph cut method, semi-global matching and other methods can be used. Optionally, the present invention can also use the following method to calculate the depth value: first, the initial disparity is obtained according to the traditional disparity calculation method; then, a graphical model is constructed for all pixels in the two-dimensional image map, wherein the nodes of the graphical model are pixels. Disparity value, the edge of the graphical model is the similarity measure between pixels; finally, the disparity value of the pixel is propagated through multiple iterations in the graphical model to achieve the global optimum, and the disparity information is converted according to the external and internal parameters of the camera is the depth value.

Step S213, project the two-dimensional foreground point into the camera coordinate system according to the two-dimensional coordinates and the depth value of the two-dimensional foreground point. Wherein, the camera coordinate system refers to the coordinate system of the camera that collects the 2D depth map (from which the 2D foreground image is extracted) in the camera combination.

The camera coordinate system is a coordinate system closely related to the observer. The camera coordinate system is similar to the coordinate system of the 2D image, the difference is that the camera coordinate system is in the 3D space, while the coordinate system of the 2D image is in the 2D space. Among the three coordinate axes of the camera coordinate system, two of them are parallel to the two coordinate axes of the two-dimensional image map, and the other is perpendicular to the two-dimensional image map. The origin of the camera coordinate system is the center of the 2D image.

Optionally, in this step, the following formula can be used to complete the projection process of projecting the two-dimensional foreground point to the camera coordinate system:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}\\ {\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; /</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}\\ {\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; z</span>}\end{array}\right.$

Among them, (u, v) are the two-dimensional coordinates of the two-dimensional foreground point, corresponding to the two coordinate axes of the two-dimensional image map; z is the depth value of the two-dimensional foreground point; c _u and c _v are the two-dimensional image map Center coordinates; f _u and f _v are the focal lengths of the camera that collects the two-dimensional image of the frame in the direction of the two coordinate axes; U _C , V _C , Z _C are the projection points of the two-dimensional foreground point in the camera coordinate system coordinate of.

Step S214, continue to project the projection point of the two-dimensional foreground point in the camera coordinate system to the world coordinate system, and determine the projected point obtained in the world coordinate system as the three-dimensional foreground point.

Since different cameras have different positions, heights, and angles, in order to describe the absolute and relative positions of different cameras at the monitoring site, the world coordinate system is used to describe the position, height, and angle of each camera. The rotation transformation matrix and translation transformation matrix can be used to convert between the camera coordinate system and the world coordinate system.

Optionally, this step can use the following formula to continue projecting the projection point of the two-dimensional foreground point in the camera coordinate system to the world coordinate system:

$\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}}\\ {\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}}\\ {\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}}\end{array}\right]<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; R</span>}\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}\\ {\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}\\ {\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}\end{array}\right]<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; T</span>}$

Among them, (U _W , V _W , Z _W ) are the three-dimensional coordinates of the three-dimensional foreground point, R and T represent the rotation transformation matrix and The translation transformation matrix.

In order to complete the above process, it is necessary to determine the rotation transformation matrix and translation transformation matrix of the camera coordinate system of each camera relative to the world coordinate system.

Optionally, step S214 can use the process of steps S2141 to S2145 to determine the rotation transformation matrix and translation transformation matrix:

Step S2141, determining the points used to display the ground in the two-dimensional image map as ground points.

Specifically, this step is to determine a point whose image information represented in the two-dimensional image map is the ground as a ground point.

Step S2142, determine the two-dimensional coordinates of all ground points in the two-dimensional image map, and determine the depth values of all ground points in the two-dimensional depth map corresponding to the two-dimensional image frame.

Assuming that the two-dimensional coordinates of a certain ground point in the two-dimensional image map are (u', v'), since the two-dimensional image map and its corresponding two-dimensional depth map use the same coordinate axis, the two-dimensional depth map The pixel value (z') of the point whose coordinates are (u', v') is the depth value of the ground point.

During specific implementation, if the texture of the two-dimensional image acquired by the combination of cameras is not clear enough at certain positions, the calculation of the depth value may fail, that is, the depth value of these positions cannot be obtained. If the ground point determined in this step belongs to this case, the ground point is discarded.

Step S2143, using the two-dimensional coordinates of all ground points and their depth values to perform three-dimensional plane fitting.

Step S2144, determining the parameters in the function corresponding to the fitted plane with the largest area as the deployment parameters of the camera combination that collects the two-dimensional image of the frame.

Wherein, the deployment parameters reflect conditions such as the installation position, height and angle of the camera.

Specifically, when using the two-dimensional coordinates of all ground points and their depth values for three-dimensional plane fitting in this step, affected by calculation errors, multiple three-dimensional planes with different areas may be obtained, among which the three-dimensional plane with the largest area Most likely corresponds to the ground in the real world. The ground in the real world captured by the camera can reflect the installation position, height and angle of the camera, while the function parameters of the 3D plane reflect the properties of the 3D plane. Therefore, the 3D plane with the largest area (corresponding to the real world The parameters in the function (reflecting the properties of the ground in the real world) can reflect the installation position, height and angle of the camera, so that the function parameters of the three-dimensional plane can be determined as the deployment parameters of the camera combination.

Assuming that the function corresponding to the largest plane obtained by fitting is Ax+By+Cz+D=0, where x, y, and z are variables, and A, B, C, and D are parameters, then the parameters A, B, C, and D are Deployment parameters for this camera combination.

During specific implementation, this step can adopt the three-dimensional plane fitting method disclosed by the open source PCL (Point Cloud Library, refer to the website http://pointclouds.org/).

It should be noted that the present invention does not limit the three-dimensional plane fitting method used, that is, the above description is only a specific embodiment of the present invention, and is not used to limit the protection scope of the present invention. Within the scope of the invention, any other three-dimensional plane fitting method should be included in the protection scope of the present invention, such as the least square method or methods based on gradient descent.

In step S2145, the deployment parameters are calibrated and calculated based on the world coordinate system to obtain a rotation transformation matrix and a translation transformation matrix of the combination of cameras that collect the two-dimensional image of the frame.

Step S215, combining the 3D foreground points corresponding to all the 2D foreground points in the 2D foreground image to form a 3D foreground image.

In fact, the 3D foreground image is a kind of point cloud, which corresponds to the object corresponding to the 2D foreground image in the real world. Therefore, step S215 is actually a process of gathering point cloud data to form a point cloud.

Step S22, calculating the change of the corresponding position of the point in the real world in the plurality of two-dimensional foreground images collected by the current camera combination in the current collection period. Wherein, the point in the real world is a point corresponding to each 2D foreground point in each 2D foreground image in the real world.

Specifically, changes in the corresponding positions of points in the real world in the multiple frames of two-dimensional foreground images can be obtained by calculating the optical flow for the multiple frames of two-dimensional foreground images.

Optical flow refers to the expression of the motion speed of points on the surface of space objects on the imaging plane of the visual sensor. In the present invention, optical flow refers to the expression of the moving speed of the points on the object surface in the real world on the two-dimensional image map collected by the camera. When the points on the object surface in the real world are limited to be in the two-dimensional foreground image When the two-dimensional foreground point corresponds to the point in the real world, the optical flow refers to the expression of the moving speed of the point on the surface of the object in the real world on the two-dimensional foreground image at different acquisition times.

Therefore, this step can obtain the points in the real world (the points corresponding to each two-dimensional foreground point in each two-dimensional foreground image in the real world) by calculating the optical flow for these multiple two-dimensional foreground images. The change of the corresponding position in a two-dimensional foreground image.

Optionally, in this step, when realizing the process of calculating the optical flow of multiple 2D foreground images collected by the current camera combination in the current acquisition cycle, firstly, the multiple frames of 2D image images corresponding to the multiple 2D foreground images Calculate the dense optical flow, and then use the multiple two-dimensional foreground images to filter the dense optical flow.

Specifically, in this step, the Lucas–Kanade method can be used to calculate the dense optical flow for multiple two-dimensional image images. Considering the calculation speed, GPU (Graphics Processing Unit, graphics processing unit) can also be used to accelerate. Among them, using the two-dimensional foreground image to filter the dense optical flow is to remove the optical flow information of the background part (parts other than the two-dimensional foreground image) in each two-dimensional image image. The filtering process used here can be Gaussian filtering , mean filtering or median filtering and other processing methods.

Step S23, according to the changes of the corresponding positions of the points in the real world in the plurality of two-dimensional foreground images, and according to the corresponding relationship between the two-dimensional foreground images and the three-dimensional foreground images, calculate Changes of the corresponding positions of the points in the plurality of three-dimensional foreground images.

It should be noted that, when the image processing process of step S21 to step S23 is performed on the multi-frame two-dimensional image images sequentially acquired by the current camera combination in the current acquisition period, in order to determine the two-dimensional foreground of the point in the real world at each acquisition time The corresponding positions in the image and the three-dimensional foreground image, preferably, the multi-frame two-dimensional image map is collected by the same camera in the current camera combination, so as to ensure that the scene covered by the multi-frame two-dimensional image map is as large as possible. The same scene, which is conducive to finding the same point in the real world in the 2D foreground image and 3D foreground image at each acquisition moment. In fact, when each camera combination is formed by a combination of multiple cameras with similar geographic locations, even if the multi-frame 2D image maps are collected by different cameras in the current camera combination, the multi-frame 2D image maps covered by There is also a certain intersection of the scenes, which is also conducive to finding the same point in the real world in the 2D foreground image and 3D foreground image at each acquisition moment.

Step S3, for each collection moment of the current collection cycle, according to the rule of merging all 3D foreground points corresponding to the same point in the real world into a 3D fusion foreground point, combine the corresponding 3D points of each camera at the monitoring site at the same collection moment The 3D foreground points in the foreground image are fused, and all 3D fused foreground points obtained after fusion are combined to form a 3D fused foreground image at the acquisition moment.

Since the scene captured by a single camera combination cannot completely cover the entire monitoring site, in order to obtain the 3D scene information of the entire monitoring site, this step integrates the scenes captured by all camera combinations together. The combined 3D foreground images are fused together.

During specific implementation, step S3 can be carried out according to the process of step S31～step S34:

Step S31, taking each collection time of the current collection cycle as the current collection time in turn, selecting each camera combination in turn, and sequentially selecting each 3D foreground point in the 3D foreground image corresponding to the currently selected camera combination at the current collection time as the current 3D Foreground point.

Step S32 , judging whether there is a 3D foreground point corresponding to the same point in the real world as the current 3D foreground point in the 3D foreground image corresponding to other camera combinations at the monitoring site at the current acquisition moment.

Specifically, in this step, when judging whether two 3D foreground points correspond to the same point in the real world, it is judged according to whether the Euclidean distance of the two 3D foreground points in the world coordinate system is less than a given threshold, When it is smaller than the threshold, it is judged that the two three-dimensional foreground points correspond to the same point in the real world; otherwise, it is judged that they do not correspond to the same point in the real world.

During specific implementation, the color difference and the Euclidean distance of the two 3D foreground points can also be weighted for calculation, and then it is judged whether the calculation result is less than a given threshold, and when it is less than the threshold, it is judged whether the two 3D foreground points correspond to the same point in the real world, otherwise the judgment does not correspond to the same point in the real world.

Step S33, if it does not exist, determine the current 3D foreground point as a 3D fused foreground point; if it exists, fuse all 3D foreground points corresponding to the same point in the real world into one 3D fused foreground point according to the following formula:

$\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; u</span>}\\ \mathrm{<span\; class="notranslate">\; V</span>}\\ \mathrm{<span\; class="notranslate">\; Z</span>}\end{array}\right]<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\\ {\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\\ {\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\end{array}\right]<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; weight</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}$

${\mathrm{<span\; class="notranslate">\; weight</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}{\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}$

Among them, U, V, and Z are the three-dimensional coordinates of the three-dimensional fusion foreground point, corresponding to the three coordinate axes of the world coordinate system respectively; each three-dimensional foreground point corresponding to the same point in the real world is determined as the point to be fused, N is the number of all points to be fused corresponding to the same point in the real world; n is the serial number of the point to be fused; (U _Wn , V _Wn , Z _Wn ) is the three-dimensional coordinates of the point to be fused with the serial number n; weight _n is the weight of the point to be fused with the sequence number n; dist _n is the distance from the point to be fused with the sequence number n to the center coordinate of the camera combination corresponding to itself; wherein, the center coordinate of the camera combination is the installation of each camera of the camera combination The coordinates of the central symmetric point of each projected point in the world coordinate system.

When each camera combination is composed of two cameras, the central coordinates of the camera combination are the coordinates of the midpoint of the two projection points of the installation positions of the two cameras in the camera combination in the world coordinate system.

For example, two 3D foreground points corresponding to the same point in the real world are recorded as points to be fused A and B respectively, where point A to be fused comes from the camera combination (a,b), and point B to be fused comes from the camera combination ( b,c), the center coordinates of the camera combination (a,b) are (U ₁ ,V ₁ ,Z ₁ ), the center coordinates of the camera combination (b,c) are (U ₂ ,V ₂ ,Z ₂ ), to be The three-dimensional coordinates of the fusion point A are (U _W1 , V _W1 , Z _W1 ), the three-dimensional coordinates of the fusion point B are (U _W2 , V _W2 , Z _W2 ), the distance between the fusion point A and the camera combination (a, b) The distance between the center coordinates is dist ₁ , and the distance between the point B to be fused and the center coordinates of the camera combination (b,c) is dist ₂ , then:

${\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\sqrt{{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

${\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>\sqrt{{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

The weight of point A to be fused is weight ₁ , and the weight of point B to be fused is weight ₂ , then:

${\mathrm{<span\; class="notranslate">\; weight</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

${\mathrm{<span\; class="notranslate">\; weight</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

In step S34, all the 3D fused foreground points are combined into a 3D fused foreground map corresponding to the current acquisition moment.

Step S4, according to the change of the corresponding positions of points in the real world in the plurality of 3D foreground images, and according to the corresponding relationship between each 3D fused foreground point and each 3D foreground point fused into each 3D fused foreground point , calculating the change of the corresponding position of the point in the real world in the multiple 3D fusion foreground images at each acquisition moment of the current acquisition cycle.

Step S5, according to the multiple 3D fusion foreground maps at each collection time of the current collection cycle, and the changes in the corresponding positions of the points in the real world in the multiple 3D fusion foreground maps at each collection time of the current collection cycle, determine the target Spatial distribution characteristics and operational distribution characteristics of objects and the target object.

Specifically, step S5 can be performed according to the process of step S51-a to step S57-a:

Step S51-a, sequentially select the 3D fusion foreground images at each collection time in the current collection period.

Step S52-a, divide the 3D space where the currently selected 3D fusion foreground image is located into several 3D subspaces, and count one or more of the following information: all 3D fusion foreground points contained in each 3D subspace The number of ; the maximum color information presented by all 3D fusion foreground points contained in each 3D subspace; the maximum height of all 3D fusion foreground points contained in each 3D subspace.

Specifically, this step may be to use cuboids (bins) with preset sizes to divide the three-dimensional space where the currently selected three-dimensional fusion foreground image is located.

During specific implementation, in order to facilitate subsequent use of the statistical results, the statistical results can be expressed in the form of a histogram, that is, the abscissa is the three-dimensional coordinate data of each three-dimensional subspace, and the vertical coordinate is the statistical data.

Step S53-a, according to the statistical results, gather all the 3D fused foreground points contained in the 3D subspace satisfying the clustering condition to form a 3D fused foreground block.

Wherein, the clustering condition is that the spatial distance is smaller than a first preset threshold, and the difference of statistical data is smaller than a second preset threshold. That is to say, the three-dimensional subspace satisfying the clustering condition must be a three-dimensional subspace whose spatial distance is smaller than a first preset threshold and whose statistical data difference is smaller than a second preset threshold.

Specifically, in this step, when all the 3D fused foreground points contained in the 3D subspace satisfying the clustering condition are aggregated to form a 3D fused foreground block, a clustering operation method may be used, such as connected domain analysis, mean shift algorithm, and the like.

Step S54-a, performing template matching on the formed 3D fused foreground block in the world coordinate system, and judging whether the 3D fused foreground block is the target object.

Specifically, corresponding templates may be designed according to the target object, and then these templates are used to match the 3D fused foreground block. When the matching is successful, the 3D fused foreground block is determined as the target object.

During specific implementation, this step can determine the target object according to the characteristics of the monitoring site. For example, for the oil and gas well monitoring site, people, vehicles, crowds, etc. can be used as the target objects.

Step S55-a, when it is determined that the 3D fused foreground block is the target object, according to the change of the corresponding position of the point in the real world in the multiple 3D fused foreground images at each acquisition time of the current acquisition cycle, determine the current acquisition In the 3D fused foreground images at different acquisition moments in the cycle, gather and form each 3D fused foreground point of the target object and the 3D coordinate changes of other 3D fused foreground points corresponding to the same point in the real world, and based on this Determine the position and displacement of the target object at each acquisition moment in the current acquisition cycle.

Step S56-a, determining the position of the target object at each collection moment in the current collection cycle as the spatial distribution feature of the target object.

Step S57-a, determining the displacement of the target object at each collection moment in the current collection cycle as the running distribution feature of the target object.

Specifically, the running distribution characteristics of the target object include but not limited to information such as the motion amplitude, direction, and acceleration of the target object.

The above process is to perform template matching on the formed 3D fusion foreground block in the world coordinate system. Since this template matching method is carried out in 3D space, it will involve a large amount of computation and take a long time. In order to reduce the amount of computation, To improve the calculation speed, optionally, this step can also project the 3D fused foreground map onto a 2D plane to obtain the fused foreground projection map, and then perform template matching in the 2D space. Specifically, step S5 can also follow step S51- The process of b～step S58-b is carried out:

Step S51-b, sequentially selecting the 3D fusion foreground images at each collection time in the current collection period.

Step S52-b, project each 3D fused foreground point in the currently selected 3D fused foreground image onto a two-dimensional plane to obtain fused foreground projection points; The corresponding fused foreground projection points are combined to obtain a fused foreground projection map.

This step can be projected onto a horizontal plane or projected onto a vertical plane.

Step S53-b, dividing the two-dimensional space where the fused foreground projection image is located into several two-dimensional subspaces, and counting one or more of the following information: the fused foreground projection contained in each two-dimensional subspace The number of points; the maximum color information presented by all fused foreground projection points contained in each two-dimensional subspace; the maximum height of all fused foreground projection points contained in each two-dimensional subspace.

Step S54-b, according to the statistical results, gather all the fused foreground projection points contained in the two-dimensional subspace satisfying the clustering condition to form a two-dimensional fused foreground block; wherein, the clustering condition is that the spatial distance is less than the first preset threshold, and the statistical data difference is smaller than a second preset threshold.

Step S55-b, performing template matching on the formed 2D fused foreground block in the 2D space, and judging whether the 2D fused foreground block is a target object.

Step S56-b, when it is determined that the 2D fused foreground block is the target object, determine the current In the fused foreground projection map corresponding to the 3D fused foreground images at different acquisition moments in the acquisition cycle, each fused foreground projection point forming the target object and other fused foreground projection points corresponding to the same point in the real world are gathered together. dimensional coordinate changes, and accordingly determine the position and displacement of the target object at each acquisition moment in the current acquisition cycle.

Step S57-b, determining the position of the target object at each collection moment in the current collection cycle as the spatial distribution feature of the target object.

Step S58-b, determining the displacement of the target object at each collection moment in the current collection cycle as the running distribution feature of the target object.

Step S6, according to the spatial distribution characteristics and operation distribution characteristics of the target object, it is judged whether there is a target scene.

During specific implementation, the target scenario can be determined according to the characteristics of the monitoring site. For example, for an oil and gas well monitoring site, whether someone invades, someone removes an object, someone discards an object, someone runs, someone fights, etc. can be used as the target scenario.

Specifically, in this step, pattern classification may be performed on the spatial distribution characteristics and operation distribution characteristics of the target object; then, whether there is a target scenario is judged according to the result of the pattern classification.

The pattern classification method adopted in this step can be completed based on multiple classifiers, such as random forests, which can be used to directly complete the mapping of features to multiple categories, or can be completed based on multiple single classifiers, such as multiple SVMs (SupportVector Machine, Support Vector Machine) classifier, such as a certain SVM classifier to detect fights, which takes fights as positive examples and other situations as negative examples to complete the training of the fight classifier, and so on.

For example, a schematic diagram of judging multiple abnormal types based on random forest, using the spatial distribution characteristics and motion distribution characteristics of the target object as input, through sample learning, multiple tree models are determined, and prediction and weighting are performed based on multiple tree models Vote to get the abnormal type corresponding to the currently detected spatial distribution feature or motion distribution feature.

Step S7, outputting the judgment result, and alarming when a target situation occurs.

Specifically, in order to facilitate the monitoring personnel to know the situation of the monitoring site in a timely manner, the present invention can directly output the judgment result by means of display or sound, and can also upload the judgment result to the network platform through the network for computers or mobile devices connected to the network platform. Terminal (such as mobile phone, portable notebook computer, etc.) browsing.

It should be noted that although the operations of the multi-camera collaborative monitoring method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all shown operations must be performed to achieve achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be decomposed into multiple steps for execution.

Exemplary device

After introducing the multi-camera cooperative monitoring method provided by the present invention, next, the multi-camera cooperative monitoring device provided by the present invention will be introduced with reference to FIG. 5 to FIG. 6 .

FIG. 5 is a schematic diagram of input and output of the multi-camera collaborative monitoring device. Among them, the input is the two-dimensional image image collected by all the cameras on the monitoring site, and the output is the detected target scene (such as intrusion, fight, running, theft, etc.) and alarm information.

When the present invention is specifically implemented, the multi-camera cooperative monitoring device can monitor and control all cameras on the monitoring site through RTU (Remote Terminal Unit, remote terminal unit), wherein, these cameras all collect images based on the NTP protocol.

As shown in Figure 6, it is a structural block diagram of the multi-camera cooperative monitoring device, and the multi-camera cooperative monitoring device includes:

The camera division module 601 is configured to combine multiple cameras deployed at the monitoring site into multiple camera combinations in such a way that at least two cameras form a camera combination;

An image map acquisition module 602, configured to obtain a two-dimensional image map collected by each camera combination in each acquisition cycle;

The image processing module 603 is used to perform image processing on the multi-frame two-dimensional image images sequentially collected by the current camera combination in the current collection cycle;

The image processing module 603 further includes:

The first image processing module 604 is configured to extract foregrounds from the multi-frame two-dimensional image images to obtain multiple two-dimensional foreground images;

The second image processing module 605 is configured to project the plurality of two-dimensional foreground images into a three-dimensional space to obtain a plurality of three-dimensional foreground images; wherein, each collection moment of the current collection period, the multiple frames of two-dimensional image images, The plurality of two-dimensional foreground images and the plurality of three-dimensional foreground images have a one-to-one correspondence;

The third image processing module 606 is configured to calculate the change of the corresponding positions of the points in the real world in the plurality of two-dimensional foreground images; wherein, the points in the real world are points in the two-dimensional foreground images The point corresponding to each two-dimensional foreground point in the real world; the two-dimensional foreground point is a pixel in the two-dimensional foreground image;

The fourth image processing module 607 is configured to, according to the changes of the corresponding positions of the points in the real world in the multiple 2D foreground images, and according to the correspondence between the 2D foreground images and the 3D foreground images relationship, calculating changes in the corresponding positions of points in the real world in the plurality of three-dimensional foreground images;

The fusion processing module 608 is used for combining all the 3D foreground points corresponding to the same point in the real world into one 3D fused foreground point at each collection moment of the current collection cycle, and combining the various cameras at the monitoring site at the same collection moment Combining the 3D foreground points in the corresponding 3D foreground image for fusion processing, combining all the 3D fused foreground points obtained after fusion to form a 3D fused foreground map at the acquisition moment; wherein, the 3D foreground points are the 3D Voxel points of the foreground image;

The displacement calculation module 609 is configured to change the corresponding positions of points in the real world in the plurality of 3D foreground images, and according to each 3D fused foreground point and each 3D foreground point fused into each 3D fused foreground point Correspondence of scenic spots, calculate the changes of the corresponding positions of points in the real world in the multiple 3D fusion foreground images at each collection moment of the current collection cycle;

The target search module 610 is configured to use multiple 3D fused foreground images at each acquisition moment in the current acquisition period, and changes in the corresponding positions of points in the real world in the multiple 3D fused foreground images at each acquisition moment in the current acquisition period situation, determine the target object, and determine the spatial distribution characteristics and operational distribution characteristics of the target object;

Judging module 611, configured to judge whether there is a target scene according to the spatial distribution characteristics and operation distribution characteristics of the target object;

The output module 612 is used to output the judgment result and give an alarm when a target situation occurs.

The second image processing module 605 further includes: a 2D coordinate calculation submodule, a depth value calculation submodule, a first projection submodule, a second projection submodule, and a 3D foreground point cloud processing submodule.

A two-dimensional coordinate calculation submodule, configured to determine the two-dimensional coordinates of each two-dimensional foreground point in the two-dimensional image map corresponding to the two-dimensional foreground image;

The depth value calculation sub-module is used to determine the depth value of each two-dimensional foreground point in the two-dimensional depth map corresponding to the two-dimensional foreground image; wherein, the two-dimensional depth map uses each camera in the current combination The depth value of each pixel in the corresponding two-dimensional image image calculated by the two-dimensional image image collected by the camera at the same time is formed;

The first projection submodule is configured to project the two-dimensional foreground point into the camera coordinate system according to the two-dimensional coordinates and the depth value of the two-dimensional foreground point;

The second projection sub-module is used to continue projecting the projection point of the two-dimensional foreground point in the camera coordinate system into the world coordinate system, and determine the projection point obtained in the world coordinate system as the three-dimensional foreground point ;

The 3D foreground point cloud processing sub-module is configured to compose the 3D foreground image with the 3D foreground points corresponding to all the 2D foreground points in the 2D foreground image.

The second image processing module 606 calculates the optical flow for the multiple 2D foreground images to obtain the changes of the corresponding positions of the points in the real world in the multiple 2D foreground images.

The fusion processing module 608 further includes: a first polling submodule, a search submodule, a first fusion submodule, a second fusion submodule, and a 3D fusion foreground point cloud processing submodule.

The first polling sub-module is used to sequentially take each collection moment of the current collection cycle as the current collection moment, select each camera combination in turn, and select in turn each of the three-dimensional foreground images corresponding to the camera combination currently selected at the current collection moment. The 3D foreground point is used as the current 3D foreground point;

The search sub-module is used to judge whether there is a 3D foreground point corresponding to the same point in the real world as the current 3D foreground point in the 3D foreground image corresponding to other camera combinations at the monitoring site at the current collection moment; When it exists, trigger the first fusion submodule; when it is judged to exist, trigger the second fusion submodule;

The first fusion submodule is used to determine the current 3D foreground point as a 3D fusion foreground point;

The second fusion sub-module is used to fuse all 3D foreground points corresponding to the same point in the real world into a 3D fusion foreground point according to the following formula:

$\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; u</span>}\\ \mathrm{<span\; class="notranslate">\; V</span>}\\ \mathrm{<span\; class="notranslate">\; Z</span>}\end{array}\right]<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\\ {\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\\ {\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\end{array}\right]<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; weight</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}$

${\mathrm{<span\; class="notranslate">\; weight</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}{\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}$

Among them, U, V, and Z are the three-dimensional coordinates of the three-dimensional fusion foreground point, corresponding to the three coordinate axes of the world coordinate system respectively; each three-dimensional foreground point corresponding to the same point in the real world is determined as the point to be fused, N is the number of all points to be fused corresponding to the same point in the real world; n is the serial number of the point to be fused; (U _Wn , V _Wn , Z _Wn ) is the three-dimensional coordinates of the point to be fused with the serial number n; weight _n is the weight of the point to be fused with the sequence number n; dist _n is the distance from the point to be fused with the sequence number n to the center coordinate of the camera combination corresponding to itself; wherein, the center coordinate of the camera combination is the installation of each camera of the camera combination The coordinates of the central symmetric point of each projected point in the world coordinate system;

The 3D fused foreground point cloud processing sub-module is used to form all 3D fused foreground points into a 3D fused foreground map corresponding to the current acquisition time.

In one embodiment, the target search module 610 further includes: a second polling submodule, a 3D subspace division submodule, a 3D clustering processing submodule, a 3D template matching submodule, a 3D position and displacement calculation submodule, a 3D Spatial feature calculation sub-module and three-dimensional running feature calculation sub-module.

The second polling sub-module is used to sequentially select the three-dimensional fusion foreground map at each acquisition moment of the current acquisition cycle;

The three-dimensional subspace division sub-module is used to divide the three-dimensional space where the currently selected three-dimensional fusion foreground image is located into several three-dimensional subspaces, and count one or more of the following information: all the information contained in each three-dimensional subspace The number of 3D fusion foreground points; the maximum color information presented by all 3D fusion foreground points contained in each 3D subspace; the maximum height of all 3D fusion foreground points contained in each 3D subspace;

The three-dimensional clustering processing sub-module is used to gather all the three-dimensional fusion foreground points contained in the three-dimensional subspace satisfying the clustering condition according to the statistical results to form a three-dimensional fusion foreground block; wherein, the clustering condition is that the spatial distance is less than the first a preset threshold, and the statistical data difference is smaller than a second preset threshold;

The three-dimensional template matching submodule is used to perform template matching on the formed three-dimensional fusion foreground block in the world coordinate system, and judge whether the three-dimensional fusion foreground block is a target object;

The three-dimensional position and displacement calculation sub-module is used for determining the three-dimensional fusion foreground block as the target object, according to the change of the corresponding position of the point in the real world in the multiple three-dimensional fusion foreground images at each acquisition time of the current acquisition cycle Determine the 3D coordinate changes of each 3D fused foreground point that forms the target object and other 3D fused foreground points corresponding to the same point in the real world in the 3D fused foreground image at different acquisition moments in the current acquisition cycle situation, and accordingly determine the position and displacement of the target object at each collection moment in the current collection cycle;

A three-dimensional spatial feature calculation submodule, configured to determine the position of the target object at each acquisition moment in the current acquisition cycle as the spatial distribution feature of the target object;

The three-dimensional operating feature calculation sub-module is used to determine the displacement of the target object at each acquisition moment in the current acquisition cycle as the operating distribution feature of the target object.

In another embodiment, the target search module 610 further includes: a third polling submodule, a two-dimensional projection submodule, a two-dimensional subspace division submodule, a two-dimensional clustering processing submodule, and a two-dimensional template matching submodule , a two-dimensional position and displacement calculation sub-module, a two-dimensional spatial feature calculation sub-module and a two-dimensional running feature calculation sub-module.

The third polling sub-module is used to sequentially select the three-dimensional fusion foreground map at each acquisition moment of the current acquisition cycle;

The two-dimensional projection sub-module is used to project each three-dimensional fusion foreground point in the currently selected three-dimensional fusion foreground image into a two-dimensional plane to obtain the fusion foreground projection point; all three-dimensional fusion foreground points in the currently selected three-dimensional fusion foreground image The fusion foreground projection points corresponding to the fusion foreground points are combined to obtain the fusion foreground projection map;

The two-dimensional subspace division sub-module is used to divide the two-dimensional space where the fused foreground projection image is located into several two-dimensional subspaces, and count one or more of the following information: in each two-dimensional subspace The number of fused foreground projection points included; the maximum color information presented by all fused foreground projection points contained in each 2D subspace; the maximum height of all fused foreground projection points contained in each 2D subspace;

The two-dimensional clustering processing sub-module is used to gather all the fusion foreground projection points contained in the two-dimensional subspace satisfying the clustering condition to form a two-dimensional fusion foreground block according to the statistical results; wherein, the clustering condition is a spatial distance less than the first preset threshold, and the difference of statistical data is less than the second preset threshold;

A two-dimensional template matching submodule, configured to perform template matching on the formed two-dimensional fusion foreground block in the two-dimensional space, and determine whether the two-dimensional fusion foreground block is a target object;

The two-dimensional position and displacement calculation sub-module is used for determining that the two-dimensional fusion foreground block is the target object, according to the corresponding positions of the points in the real world in the multiple three-dimensional fusion foreground images at each acquisition time of the current acquisition cycle Changes in the current acquisition cycle, determine the fusion foreground projection map corresponding to the three-dimensional fusion foreground image at different acquisition moments in the current acquisition cycle, gather and form each fusion foreground projection point of the target object and other points corresponding to the same point in the real world Fuse the change of the two-dimensional coordinates of the foreground projection point, and determine the position and displacement of the target object at each acquisition moment of the current acquisition cycle accordingly;

A two-dimensional spatial feature calculation submodule, configured to determine the position of the target object at each acquisition moment in the current acquisition cycle as the spatial distribution feature of the target object;

The two-dimensional operating feature calculation sub-module is used to determine the displacement of the target object at each acquisition moment in the current acquisition cycle as the operating distribution feature of the target object.

The judging module 611 further includes: a pattern classification processing submodule and a pattern classification judging submodule.

A mode classification processing submodule, configured to perform mode classification on the spatial distribution characteristics and operation distribution characteristics of the target object;

The pattern classification judging sub-module is used to judge whether there is a target scenario according to the result of the pattern classification.

The camera division module 601 is to form a camera combination from any two cameras installed at the monitoring site with a distance less than a preset distance.

When the image processing module 603 performs image processing on the multi-frame 2D image images sequentially collected by the current camera combination in the current acquisition cycle, the multi-frame 2D image images are collected by the same camera in the current camera combination.

In the multi-camera cooperative monitoring device provided by the present invention, the two-dimensional image images collected by the cameras may be grayscale images or color images.

The multi-camera cooperative monitoring device and the multi-camera cooperative monitoring method provided by the present invention are realized based on the same inventive concept, and the specific implementation method can refer to the above-mentioned introduction to the multi-camera cooperative monitoring method, which will not be repeated here.

It should be noted that although several units or subunits of the multi-camera collaborative monitoring device are mentioned in the above detailed description, this division is merely not mandatory. Actually, according to the embodiment of the present invention, the features and functions of two or more units described above may be embodied in one unit. Conversely, the features and functions of one unit described above may be further divided to be embodied by a plurality of units.

Although the spirit and principles of the invention have been described with reference to a number of specific embodiments, it should be understood that the invention is not limited to the specific embodiments disclosed, nor does division of aspects imply that features in these aspects cannot be combined to achieve optimal performance. Benefit, this division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. Protection scope, within the spirit and principles of the present invention, any modification, equivalent replacement, improvement, etc., shall be included in the protection scope of the present invention.

Those skilled in the art can also understand that various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units and steps above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and overall system design requirements. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the protection scope of the embodiments of the present invention.

Various illustrative logic blocks, or units, or devices described in the embodiments of the present invention can be implemented by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to implement or operate the described functions. The general-purpose processor may be a microprocessor, and optionally, the general-purpose processor may also be any conventional processor, controller, microcontroller or state machine. A processor may also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration to accomplish.

The steps of the method or algorithm described in the embodiments of the present invention may be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software modules may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM or any other storage medium in the art. Exemplarily, the storage medium can be connected to the processor, so that the processor can read information from the storage medium, and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be set in the ASIC, and the ASIC can be set in the user terminal. Optionally, the processor and the storage medium may also be set in different components in the user terminal.

In one or more exemplary designs, the above functions described in the embodiments of the present invention may be implemented in hardware, software, firmware or any combination of the three. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media and communication media that facilitate transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a general purpose or special computer. For example, such computer-readable media may include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other device that can be used to carry or store instructions or data structures and Other medium of program code in a form readable by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. In addition, any connection is properly defined as a computer-readable medium, for example, if the software is transmitted from a website site, server, or other remote source via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) Or transmitted by wireless means such as infrared, wireless and microwave are also included in the definition of computer readable media. Disks and discs include compact discs, laser discs, optical discs, DVDs, floppy discs, and Blu-ray discs. Disks usually reproduce data magnetically, while discs usually reproduce data optically with lasers. Combinations of the above can also be contained on a computer readable medium.

Claims (20)

1. A multi-camera collaborative monitoring method, characterized in that, comprising: Step A, according to the way that at least two cameras form a camera combination, multiple cameras deployed at the monitoring site are combined into multiple camera combinations; Step B, obtaining the two-dimensional image images collected by each camera combination in each acquisition period, and performing the processing of steps B1 to B3 on the multi-frame two-dimensional image images sequentially acquired by the current camera combination in the current acquisition period; Step B1, extracting the foreground from the multi-frame two-dimensional image images to obtain a plurality of two-dimensional foreground images; projecting the plurality of two-dimensional foreground images into a three-dimensional space to obtain a plurality of three-dimensional foreground images; wherein, the current acquisition cycle There is a one-to-one correspondence between each acquisition moment, the multi-frame two-dimensional image map, the plurality of two-dimensional foreground images, and the plurality of three-dimensional foreground images; Step B2, calculating the change of the corresponding positions of the points in the real world in the plurality of two-dimensional foreground images; wherein, the points in the real world are each two-dimensional foreground points in the two-dimensional foreground images A corresponding point in the real world; the two-dimensional foreground point is a pixel in the two-dimensional foreground image; Step B3, according to the changes in the corresponding positions of points in the real world in the plurality of two-dimensional foreground images, and according to the correspondence between the two-dimensional foreground images and the three-dimensional foreground images, calculate The changes of the corresponding positions of the points in the plurality of three-dimensional foreground images; Step C: For each collection moment of the current collection cycle, according to the rule of merging all 3D foreground points corresponding to the same point in the real world into a 3D fusion foreground point, combine the corresponding 3D points of each camera at the monitoring site at the same collection moment The 3D foreground points in the foreground image are fused, and all the 3D fused foreground points obtained after fusion are combined to form a 3D fused foreground image at the acquisition moment; wherein, the 3D foreground points are volumes of the 3D foreground image Prime point; Step D, according to the change of the corresponding positions of the points in the real world in the plurality of 3D foreground images, and according to the corresponding relationship between each 3D fused foreground point and each 3D foreground point fused into each 3D fused foreground point , calculating the change of the corresponding position of the point in the real world in the multiple 3D fusion foreground images at each acquisition moment of the current acquisition cycle; Step E, according to the multiple 3D fusion foreground maps at each collection moment of the current collection cycle, and the changes in the corresponding positions of points in the real world in the multiple 3D fusion foreground maps at each collection moment of the current collection cycle, determine the target object, and determining the spatial distribution characteristics and operational distribution characteristics of the target object; Step F, judging whether there is a target scene according to the spatial distribution characteristics and operation distribution characteristics of the target object; Step G, output the judgment result, and call the police when there is a target situation. 2. The multi-camera collaborative monitoring method according to claim 1, wherein in the step B1, projecting the plurality of two-dimensional foreground images into a three-dimensional space to obtain a plurality of three-dimensional foreground images comprises: In the two-dimensional image map corresponding to the two-dimensional foreground image, determine the two-dimensional coordinates of each two-dimensional foreground point; In the two-dimensional depth map corresponding to the two-dimensional foreground image, determine the depth value of each two-dimensional foreground point; wherein, the two-dimensional depth map is a two-dimensional image map simultaneously collected by each camera in the current camera combination The calculated depth value of each pixel in the corresponding two-dimensional image map is formed; Projecting the two-dimensional foreground point into a camera coordinate system according to the two-dimensional coordinates and the depth value of the two-dimensional foreground point; Continue projecting the projection point of the two-dimensional foreground point in the camera coordinate system into the world coordinate system, and determine the projection point obtained in the world coordinate system as the three-dimensional foreground point; The three-dimensional foreground image is composed of all the three-dimensional foreground points corresponding to the two-dimensional foreground points in the two-dimensional foreground image. 3. The multi-camera collaborative monitoring method according to claim 2, characterized in that, the step B2 is to obtain the points in the real world between the multiple two-dimensional foreground images by calculating the optical flow for the multiple two-dimensional foreground images. The change of the corresponding position in the foreground image. 4. The multi-camera collaborative monitoring method according to claim 3, wherein the step C includes: Step C1, taking each collection time of the current collection cycle as the current collection time in turn, selecting each camera combination in turn, and sequentially selecting each 3D foreground point in the 3D foreground image corresponding to the currently selected camera combination at the current collection time as the current 3D Foreground point; Step C2, judging whether there is a 3D foreground point corresponding to the same point in the real world as the current 3D foreground point in the 3D foreground image corresponding to other camera combinations at the monitoring site at the current collection moment; Step C3, if it does not exist, determine the current 3D foreground point as a 3D fused foreground point; if it exists, fuse all 3D foreground points corresponding to the same point in the real world into a 3D fused foreground point according to the following formula: $\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; u</span>}\\ \mathrm{<span\; class="notranslate">\; V</span>}\\ \mathrm{<span\; class="notranslate">\; Z</span>}\end{array}\right]<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\\ {\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\\ {\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\end{array}\right]<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; weight</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}$ ${\mathrm{<span\; class="notranslate">\; weight</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}{\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}$ Among them, U, V, and Z are the three-dimensional coordinates of the three-dimensional fusion foreground point, corresponding to the three coordinate axes of the world coordinate system respectively; each three-dimensional foreground point corresponding to the same point in the real world is determined as the point to be fused, N is the number of all points to be fused corresponding to the same point in the real world; n is the serial number of the point to be fused; (U _Wn , V _Wn , Z _Wn ) is the three-dimensional coordinates of the point to be fused with the serial number n; weight _n is the weight of the point to be fused with the sequence number n; dist _n is the distance from the point to be fused with the sequence number n to the center coordinate of the camera combination corresponding to itself; wherein, the center coordinate of the camera combination is the installation of each camera of the camera combination The coordinates of the central symmetric point of each projected point in the world coordinate system; Step C4, combining all the 3D fused foreground points into a 3D fused foreground map corresponding to the current acquisition moment. 5. The multi-camera collaborative monitoring method according to claim 4, wherein the step E includes: Step E1, sequentially selecting the 3D fusion foreground images at each collection moment of the current collection cycle; Step E2, divide the 3D space where the currently selected 3D fusion foreground image is located into several 3D subspaces, and count one or more of the following information: the number of all 3D fusion foreground points contained in each 3D subspace ; The maximum color information presented by all 3D fusion foreground points contained in each 3D subspace; The maximum height of all 3D fusion foreground points contained in each 3D subspace; Step E3, according to the statistical results, gather all the 3D fused foreground points contained in the 3D subspace satisfying the clustering condition to form a 3D fused foreground block; wherein, the clustering condition is that the spatial distance is less than the first preset threshold, and the statistical The data difference is smaller than a second preset threshold; Step E4, performing template matching on the formed 3D fused foreground block in the world coordinate system, and judging whether the 3D fused foreground block is the target object; Step E5, when it is determined that the 3D fused foreground block is the target object, according to the changes in the corresponding positions of the points in the real world in the multiple 3D fused foreground images at each acquisition time of the current acquisition period, determine In the 3D fused foreground images at different acquisition moments, the 3D coordinate changes of each 3D fused foreground point forming the target object and other 3D fused foreground points corresponding to the same point in the real world are collected, and determined accordingly. The position and displacement of the target object at each collection moment of the current collection cycle; Step E6, determining the position of the target object at each collection moment in the current collection cycle as the spatial distribution feature of the target object; Step E7, determining the displacement of the target object at each collection moment in the current collection cycle as the running distribution feature of the target object. 6. The multi-camera collaborative monitoring method according to claim 4, wherein the step E includes: Step E1, sequentially selecting the 3D fusion foreground images at each collection moment of the current collection cycle; Step E2, projecting each 3D fusion foreground point in the currently selected 3D fusion foreground image onto a two-dimensional plane to obtain the fusion foreground projection point; The fusion foreground projection points are combined to obtain the fusion foreground projection map; Step E3, dividing the two-dimensional space where the fused foreground projection image is located into several two-dimensional subspaces, and counting one or more of the following information: the number of fused foreground projection points contained in each two-dimensional subspace Quantity; the maximum color information presented by all fusion foreground projection points contained in each two-dimensional subspace; the maximum height of all fusion foreground projection points contained in each two-dimensional subspace; Step E4, according to the statistical results, gather all the fused foreground projection points contained in the two-dimensional subspace satisfying the clustering condition to form a two-dimensional fused foreground block; wherein, the clustering condition is that the spatial distance is smaller than the first preset threshold, And the statistical data difference is less than a second preset threshold; Step E5, performing template matching on the formed two-dimensional fusion foreground block in the two-dimensional space, and judging whether the two-dimensional fusion foreground block is a target object; Step E6, when it is determined that the 2D fused foreground block is the target object, determine the current acquisition period according to the changes in the corresponding positions of the points in the real world in the multiple 3D fused foreground images at each acquisition time of the current acquisition period In the fused foreground projection map corresponding to the 3D fused foreground images at different acquisition moments, gather and form each fused foreground projection point of the target object and the two-dimensional coordinates of other fused foreground projection points corresponding to the same point in the real world Changes, and accordingly determine the position and displacement of the target object at each acquisition moment in the current acquisition cycle; Step E7, determining the position of the target object at each collection moment in the current collection cycle as the spatial distribution feature of the target object; Step E8, determining the displacement of the target object at each collection moment in the current collection cycle as the running distribution feature of the target object. 7. The multi-camera collaborative monitoring method according to claim 5 or 6, wherein the step F includes: performing pattern classification on the spatial distribution characteristics and operation distribution characteristics of the target object; According to the results of pattern classification, it is judged whether there is a target scenario. 8. The multi-camera collaborative monitoring method according to claim 1, characterized in that, in the step A, any two cameras installed at the monitoring site that are less than a preset distance apart form a camera combination. 9. The multi-camera collaborative monitoring method according to claim 1, characterized in that, in the step B, the multi-frame two-dimensional images are collected by the same camera in the current camera combination. 10. The multi-camera collaborative monitoring method according to claim 1, wherein the two-dimensional image is a grayscale image or a color image. 11. A multi-camera collaborative monitoring device, characterized in that it comprises: The camera division module is used to combine multiple cameras deployed at the monitoring site into multiple camera combinations according to the way that at least two cameras form a camera combination; An image map acquisition module, configured to obtain a two-dimensional image map collected by each camera combination in each acquisition cycle; The image processing module is used to perform image processing on the multi-frame two-dimensional image images sequentially collected by the current camera combination in the current acquisition cycle; The image processing module further includes: The first image processing module is used to extract the foreground from the multi-frame two-dimensional image images to obtain a plurality of two-dimensional foreground images; The second image processing module is configured to project the plurality of two-dimensional foreground images into a three-dimensional space to obtain a plurality of three-dimensional foreground images; wherein, each acquisition moment of the current acquisition cycle, the plurality of frames of two-dimensional image images, the plurality of The plurality of two-dimensional foreground images and the plurality of three-dimensional foreground images have a one-to-one correspondence; The third image processing module is used to calculate the change of the corresponding position of the point in the real world in the plurality of two-dimensional foreground images; wherein, the point in the real world is each of the two-dimensional foreground images A point corresponding to a two-dimensional foreground point in the real world; the two-dimensional foreground point is a pixel in the two-dimensional foreground image; The fourth image processing module is configured to change the corresponding positions of points in the real world in the plurality of two-dimensional foreground images, and according to the corresponding relationship between the two-dimensional foreground image and the three-dimensional foreground image , calculating changes in positions corresponding to points in the real world in the plurality of three-dimensional foreground images; The fusion processing module is used for combining all the cameras at the monitoring site at the same collection moment according to the rule of merging all 3D foreground points corresponding to the same point in the real world into a 3D fusion foreground point at each collection moment of the current collection cycle The 3D foreground points in the corresponding 3D foreground image are fused, and all the 3D fused foreground points obtained after fusion are combined to form a 3D fused foreground map at the acquisition moment; wherein, the 3D foreground points are the 3D foreground The voxel point of the image; The displacement calculation module is used for changing the corresponding positions of points in the real world in the plurality of 3D foreground images, and according to each 3D fused foreground point and each 3D foreground point fused into each 3D fused foreground point The corresponding relationship of the points in the real world is calculated at each acquisition moment of the current acquisition cycle and the corresponding position changes in the multiple 3D fusion foreground images; The target search module is used for multiple 3D fusion foreground maps at each collection moment of the current collection cycle, and changes in the corresponding positions of points in the real world in the multiple 3D fusion foreground maps at each collection moment of the current collection cycle , determining a target object, and determining a spatial distribution feature and an operational distribution feature of the target object; A judging module, configured to judge whether a target scenario occurs according to the spatial distribution characteristics and operation distribution characteristics of the target object; The output module is used for outputting judgment results and alarming when a target situation occurs. 12. The multi-camera collaborative monitoring device according to claim 11, wherein the second image processing module further comprises: A two-dimensional coordinate calculation submodule, configured to determine the two-dimensional coordinates of each two-dimensional foreground point in the two-dimensional image map corresponding to the two-dimensional foreground image; The depth value calculation sub-module is used to determine the depth value of each two-dimensional foreground point in the two-dimensional depth map corresponding to the two-dimensional foreground image; wherein, the two-dimensional depth map uses each camera in the current combination The depth value of each pixel in the corresponding two-dimensional image image calculated by the two-dimensional image image collected by the camera at the same time is formed; The first projection submodule is configured to project the two-dimensional foreground point into the camera coordinate system according to the two-dimensional coordinates and the depth value of the two-dimensional foreground point; The second projection sub-module is used to continue projecting the projection point of the two-dimensional foreground point in the camera coordinate system into the world coordinate system, and determine the projection point obtained in the world coordinate system as the three-dimensional foreground point ; The 3D foreground point cloud processing sub-module is configured to compose the 3D foreground image with the 3D foreground points corresponding to all the 2D foreground points in the 2D foreground image. 13. The multi-camera collaborative monitoring device according to claim 12, wherein the third image processing module calculates the optical flow of the multiple two-dimensional foreground images to obtain points in the real world in the Changes of corresponding positions in multiple two-dimensional foreground images. 14. The multi-camera collaborative monitoring device according to claim 13, wherein the fusion processing module further comprises: The first polling sub-module is used to sequentially take each collection moment of the current collection cycle as the current collection moment, sequentially select each camera combination, and sequentially select each of the three-dimensional foreground images corresponding to the currently selected camera combination at the current collection moment The 3D foreground point is used as the current 3D foreground point; The search sub-module is used to judge whether there is a 3D foreground point corresponding to the same point in the real world as the current 3D foreground point in the 3D foreground image corresponding to other camera combinations at the monitoring site at the current collection moment; When it exists, trigger the first fusion submodule; when it is judged to exist, trigger the second fusion submodule; The first fusion submodule is used to determine the current 3D foreground point as a 3D fusion foreground point; The second fusion sub-module is used to fuse all 3D foreground points corresponding to the same point in the real world into a 3D fusion foreground point according to the following formula: $\left[\begin{array}{c}\mathrm{<span\; class="notranslate">\; u</span>}\\ \mathrm{<span\; class="notranslate">\; V</span>}\\ \mathrm{<span\; class="notranslate">\; Z</span>}\end{array}\right]<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\\ {\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\\ {\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; W</span>}\mathrm{<span\; class="notranslate">\; no</span>}}\end{array}\right]<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; weight</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}$ ${\mathrm{<span\; class="notranslate">\; weight</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}{\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; dist</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}$ Among them, U, V, and Z are the three-dimensional coordinates of the three-dimensional fusion foreground point, corresponding to the three coordinate axes of the world coordinate system respectively; each three-dimensional foreground point corresponding to the same point in the real world is determined as the point to be fused, N is the number of all points to be fused corresponding to the same point in the real world; n is the serial number of the point to be fused; (U _Wn , V _Wn , Z _Wn ) is the three-dimensional coordinates of the point to be fused with the serial number n; weight _n is the weight of the point to be fused with the sequence number n; dist _n is the distance from the point to be fused with the sequence number n to the center coordinate of the camera combination corresponding to itself; wherein, the center coordinate of the camera combination is the installation of each camera of the camera combination The coordinates of the central symmetric point of each projected point in the world coordinate system; The 3D fused foreground point cloud processing sub-module is used to form all 3D fused foreground points into a 3D fused foreground map corresponding to the current acquisition time. 15. The multi-camera collaborative monitoring device according to claim 14, wherein the target search module further comprises: The second polling sub-module is used to sequentially select the three-dimensional fusion foreground map at each acquisition moment of the current acquisition cycle; The three-dimensional subspace division sub-module is used to divide the three-dimensional space where the currently selected three-dimensional fusion foreground image is located into several three-dimensional subspaces, and count one or more of the following information: all the information contained in each three-dimensional subspace The number of 3D fusion foreground points; the maximum color information presented by all 3D fusion foreground points contained in each 3D subspace; the maximum height of all 3D fusion foreground points contained in each 3D subspace; The three-dimensional clustering processing sub-module is used to gather all the three-dimensional fusion foreground points contained in the three-dimensional subspace satisfying the clustering condition according to the statistical results to form a three-dimensional fusion foreground block; wherein, the clustering condition is that the spatial distance is less than the first a preset threshold, and the statistical data difference is smaller than a second preset threshold; The three-dimensional template matching submodule is used to perform template matching on the formed three-dimensional fusion foreground block in the world coordinate system, and judge whether the three-dimensional fusion foreground block is a target object; The three-dimensional position and displacement calculation sub-module is used for determining the three-dimensional fusion foreground block as the target object, according to the change of the corresponding position of the point in the real world in the multiple three-dimensional fusion foreground images at each acquisition time of the current acquisition cycle Determine the 3D coordinate changes of each 3D fused foreground point that forms the target object and other 3D fused foreground points corresponding to the same point in the real world in the 3D fused foreground image at different acquisition moments in the current acquisition cycle situation, and accordingly determine the position and displacement of the target object at each collection moment in the current collection cycle; A three-dimensional spatial feature calculation submodule, configured to determine the position of the target object at each acquisition moment in the current acquisition cycle as the spatial distribution feature of the target object; The three-dimensional operating feature calculation sub-module is used to determine the displacement of the target object at each acquisition moment in the current acquisition cycle as the operating distribution feature of the target object. 16. The multi-camera collaborative monitoring device according to claim 14, wherein the target search module further comprises: The third polling sub-module is used to sequentially select the three-dimensional fusion foreground map at each acquisition moment of the current acquisition cycle; The two-dimensional projection sub-module is used to project each three-dimensional fusion foreground point in the currently selected three-dimensional fusion foreground image into a two-dimensional plane to obtain the fusion foreground projection point; all three-dimensional fusion foreground points in the currently selected three-dimensional fusion foreground image The fusion foreground projection points corresponding to the fusion foreground points are combined to obtain the fusion foreground projection map; The two-dimensional subspace division sub-module is used to divide the two-dimensional space where the fused foreground projection image is located into several two-dimensional subspaces, and count one or more of the following information: in each two-dimensional subspace The number of fused foreground projection points included; the maximum color information presented by all fused foreground projection points contained in each 2D subspace; the maximum height of all fused foreground projection points contained in each 2D subspace; The two-dimensional clustering processing sub-module is used to gather all the fusion foreground projection points contained in the two-dimensional subspace satisfying the clustering condition to form a two-dimensional fusion foreground block according to the statistical results; wherein, the clustering condition is a spatial distance less than the first preset threshold, and the difference of statistical data is less than the second preset threshold; A two-dimensional template matching submodule, configured to perform template matching on the formed two-dimensional fusion foreground block in the two-dimensional space, and determine whether the two-dimensional fusion foreground block is a target object; The two-dimensional position and displacement calculation sub-module is used for determining that the two-dimensional fusion foreground block is the target object, according to the corresponding positions of the points in the real world in the multiple three-dimensional fusion foreground images at each acquisition time of the current acquisition cycle Changes in the current acquisition cycle, determine the fusion foreground projection map corresponding to the three-dimensional fusion foreground image at different acquisition moments in the current acquisition cycle, gather and form each fusion foreground projection point of the target object and other points corresponding to the same point in the real world Fuse the change of the two-dimensional coordinates of the foreground projection point, and determine the position and displacement of the target object at each acquisition moment of the current acquisition cycle accordingly; A two-dimensional spatial feature calculation submodule, configured to determine the position of the target object at each acquisition moment in the current acquisition cycle as the spatial distribution feature of the target object; The two-dimensional operating feature calculation sub-module is used to determine the displacement of the target object at each acquisition moment in the current acquisition cycle as the operating distribution feature of the target object. 17. The multi-camera collaborative monitoring device according to claim 15 or 16, wherein the judging module further comprises: A mode classification processing submodule, configured to perform mode classification on the spatial distribution characteristics and operation distribution characteristics of the target object; The pattern classification judging sub-module is used to judge whether there is a target scenario according to the result of the pattern classification. 18 . The multi-camera collaborative monitoring device according to claim 11 , wherein the camera division module forms a camera combination from any two cameras installed at the monitoring site with a distance less than a preset distance. 19. The multi-camera cooperative monitoring device according to claim 11, wherein when the image processing module performs image processing on the multi-frame two-dimensional image images sequentially collected by the current camera combination in the current acquisition cycle, the multi-frame The frame 2D image is captured by the same camera in the current camera combination. 20. The multi-camera collaborative monitoring device according to claim 11, wherein the two-dimensional image is a grayscale image or a color image.