CN108009473A - Based on goal behavior attribute video structural processing method, system and storage device - Google Patents

Abstract

The invention discloses a method for structured video processing based on target behavior attributes. The method includes: using the YOLO target detection algorithm to obtain the basic attributes of the target; using a multi-target tracking algorithm to obtain the track information of the detected target; The abnormal behavior analysis algorithm of stream features extracts abnormal video frames; according to the custom-built metadata structure, use the above method to obtain the corresponding target category attributes and target trajectory and other characteristic information; Correct the misdetection data; upload the obtained data to the back-end server for further processing. Through the above method, the present invention can convert unstructured video data into structured data with practical value, which improves the network transmission efficiency of the video monitoring system and reduces the load rate of the back-end server. The invention also provides a real-time processing system and device based on the target behavior attribute.

Description

Video structure processing method, system and storage device based on target behavior attribute

technical field

The invention relates to the field of computer vision, in particular to a video structure processing method, system and storage device based on target behavior attributes.

Background technique

With the development of intelligent monitoring technology, the processing of video is particularly important. In the prior art, the method of image feature detection is often used for video processing, but since the dimensionality of the video will be very high, and there will be a large number of redundant features and irrelevant features, this will cause the pressure of video processing and cannot be realized. The video is processed quickly, and the accuracy of obtaining target features will be reduced. Therefore, in order to meet the needs of the development of intelligent monitoring technology, a video processing method and a video processing system with high real-time performance and high accuracy are required.

Contents of the invention

The technical problem mainly solved by the present invention is to provide a video structured processing method, system and storage device based on target behavior attributes, which can realize high-real-time and high-accuracy processing of video.

In order to solve the above technical problems, the technical solution adopted by the present invention is to provide a method for structured video processing based on target behavior attributes, including the following steps:

Perform target detection and recognition on the single frame picture;

track the target to obtain a tracking result; and/or

Abnormal behavior detection is performed on the target.

In order to solve the above-mentioned technical problems, another technical solution adopted by the present invention is to provide a video structured processing system based on target behavior attributes, including a processor and a memory electrically connected to each other, the processor is coupled to the memory, and the When working, the processor executes instructions to implement the above video processing method, and stores the processing results generated by executing the instructions in the memory.

In order to solve the above technical problems, another technical solution adopted by the present invention is to provide a device with a storage function, storing program data, and implementing the above video processing method when the program data is executed. The beneficial effects of the above technical solutions are: different from the situation of the prior art, the present invention divides the video into single-frame pictures, and performs target detection and recognition on the single-frame pictures, and tracks the recognized targets to obtain tracking results. And detect the abnormal behavior of the recognized target. In this process, the structured data can be extracted from the unstructured video, which can effectively realize the high real-time and high accuracy processing of the video data transmission.

Description of drawings

FIG. 1 is a schematic flow diagram of an embodiment of a method for structured video processing based on target behavior attributes in the present application;

FIG. 2 is a schematic flow diagram of another embodiment of the method for structured video processing based on target behavior attributes of the present application;

FIG. 3 is a schematic flow diagram of another embodiment of the method for structured video processing based on target behavior attributes in the present application;

FIG. 4 is a schematic flow diagram of another embodiment of the method for structured video processing based on target behavior attributes in the present application;

FIG. 5 is a schematic flow diagram of another embodiment of the method for structured video processing based on target behavior attributes in the present application;

FIG. 6 is a schematic flow diagram of another embodiment of the method for structured video processing based on target behavior attributes in the present application;

FIG. 7 is a schematic flow diagram of another embodiment of the method for structured video processing based on target behavior attributes in the present application;

FIG. 8 is a schematic flowchart of an embodiment of step S243 in the embodiment provided in FIG. 7;

FIG. 9 is a schematic diagram of a motion space-time container in an embodiment of a method for structured video processing based on target behavior attributes in the present application;

FIG. 10 is a schematic diagram of an embodiment of a device with a storage function in the present application;

FIG. 11 is a schematic structural diagram of an embodiment of a video structured processing system based on target behavior attributes of the present application.

Detailed ways

Hereinafter, exemplary embodiments of the present application will be described with reference to the accompanying drawings. Well-known functions and constructions are not described in detail for clarity and conciseness. Terms described below, which are defined in consideration of functions in the present application, may vary according to user and operator's intention or implementation. Therefore, the terms should be defined on the basis of the disclosure throughout the specification.

Please refer to FIG. 1 , which is a schematic flowchart of a first embodiment of the video surveillance method based on video structured data and deep learning in the present invention. The method includes:

S10: Read the video.

Optionally, reading the video includes reading real-time video collected by the camera and/or pre-recorded and saved video data. Wherein, the camera for collecting real-time video may be one of a USB camera and a network camera based on rtsp protocol stream, or other types of cameras.

In one embodiment, the read video is a video captured by a USB camera or a network camera based on an rtsp protocol stream in real time.

In another embodiment, the read video is a pre-recorded and saved video, which is input and read from a local memory or external storage devices such as U disk, hard disk, etc., or it can be a video retrieved from the network. This is not detailed one by one.

S20: Perform structured processing on the video to obtain structured data.

Optionally, performing structured processing on the video to obtain structured data specifically refers to converting the unstructured video data read in step S10 into structured data. Specifically, structured data refers to more important data. Optionally, the structured data includes at least one information of the target's location, target category, target attribute, target motion state, target motion trajectory, and target dwell time, where it can be understood that the structured data may also include user Other categories of information required by (a person using the method or system described in the present invention). Other data is not particularly important, or can be obtained through mining related information such as structured data. The specific information included in the structured information depends on different needs. How to process structured data to obtain structured data will be described in detail below.

S30: Upload the structured data to the cloud server, and conduct in-depth analysis on the structured data to obtain preset results.

Optionally, after the structured video is processed in step S20, the obtained structured data is uploaded to the cloud server and stored in the storage area of the cloud server.

In one embodiment, the data obtained through structured processing of the video is directly stored in the storage area of the cloud server, to preserve files and also to improve the database of the system.

Optionally, after the video is processed in step S20, the obtained structured data is uploaded to the cloud server, and the cloud server performs further in-depth analysis on the structured data.

Optionally, the cloud server performs further in-depth analysis on the structured data uploaded from each monitoring node, wherein the in-depth analysis includes target trajectory analysis and target traffic analysis or other required analysis, and the targets include people, vehicles and animals Wait for at least one of these.

In one embodiment, the further in-depth analysis of the structured data uploaded from each monitoring node by the cloud server is trajectory analysis, and the target is further judged according to the law of the trajectory of the uploaded target and the residence time in the scene Whether it is suspicious, whether the target stays in a certain area for a long time, whether abnormal behavior such as area intrusion occurs.

In another embodiment, the further in-depth analysis performed by the cloud server on the structured data uploaded from each monitoring node is target traffic analysis. According to the structured data uploaded by each monitoring point, the Statistics are made on the target, and the traffic of the target in each time period of the monitoring node is obtained through the statistics. The targets can be pedestrians and vehicles, and the peak or low peak periods of the target traffic can be obtained at the same time. By calculating the relevant data of the target traffic, it can be used to reasonably prompt pedestrians and drivers to avoid traffic peak hours, and can also provide a reference for public resources such as lighting.

This method obtains structured data that is critical for in-depth analysis by structured video processing, and then only uploads the structured data to the cloud instead of transmitting the entire video to the cloud, which solves the problem of heavy network transmission pressure and data traffic costs high question.

In one embodiment, according to preset settings, when each monitoring node uploads the structured data processed by the video structured processing system based on the target behavior attribute (hereinafter referred to as: the video processing system) to the cloud server, the cloud server will After saving the structured data, perform in-depth analysis on the structured data.

In another embodiment, when each monitoring node uploads the structured data processed by the video processing system to the cloud server, the server requires the user to choose whether to perform in-depth analysis after saving the structured data.

In yet another embodiment, when the user needs it, the structured data that has already been analyzed in depth at the time of initial upload can be reset for another in-depth analysis.

Optionally, the in-depth analysis of the structured data uploaded by each monitoring node further includes: performing statistics and analysis on the structured data to obtain the behavior type and abnormal behavior of one or more targets, and giving an alarm to the abnormal behavior, etc., Or the content of analysis and processing required by other users.

How to process video structured data to obtain structured data will be described in detail below, that is, the present application also provides a video structured processing method based on target behavior attributes. In one embodiment, the video structured data processing is to use the intelligent analysis module embedded in deep learning algorithms such as target detection and recognition algorithms, multi-target tracking algorithms, abnormal behavior recognition based on motion optical flow features, and read in step S10 Convert unstructured video data into structured data.

Referring to FIG. 2 , it is a schematic flow chart of an embodiment of a method for video structural processing based on target behavior attributes provided by the present application. The method is also step S20 of the above embodiment, including step S22 to step S23.

S22: Perform target detection and recognition on a single frame picture.

Optionally, step S22 is to perform target detection and recognition on a single frame of pictures. Wherein, target detection and recognition objects include pedestrian detection and recognition, vehicle detection and recognition, and animal detection and recognition.

Optionally, the step S22 of performing target detection and recognition on a single frame picture includes: extracting feature information of the target in the single frame picture. Extract the feature information, target category and target location information of all targets in a single frame picture, where the targets can be pedestrians, vehicles and animals.

In an embodiment, when only pedestrians are included in a single frame of pictures, target detection and recognition is detection and recognition of pedestrians, that is, feature information of all pedestrians in the picture is extracted.

In another embodiment, when a single frame of pictures contains multiple types of targets such as pedestrians and vehicles, the target detection and recognition is to detect and identify various types of pedestrians and vehicles, that is, to extract the pedestrians, vehicles, etc. in the single frame of pictures As for the feature information, it can be understood that the type of object recognition can be specified by the user.

Optionally, the algorithm used in step S22 to perform object detection and recognition on a single frame of pictures is an optimized object detection algorithm based on deep learning. Specifically, the YOLOV2 deep learning target detection framework can be used for target detection and recognition. The core of the algorithm is to use the entire image as the network input, and directly return the position of the bounding box and the category to which the bounding box belongs at the output layer.

Optionally, target detection consists of two parts: model training and model testing.

In one embodiment, in terms of model training, 50% of the pedestrian images or vehicle images from the VOC dataset and COCO dataset are used, and the remaining 50% of the data are taken from real streets, indoor passages, and squares. and other monitoring data. It can be understood that the ratio of the data in the public data set (VOC data set and COCO data set) used in model training to the data in the real monitoring data set can be adjusted as needed, wherein when the data in the public data set The higher the proportion of the data, relatively speaking, the accuracy of the obtained data model in the real monitoring scene will be relatively slightly worse. Conversely, when the proportion of the real monitoring data set is higher, the accuracy will be relatively improved. .

Optionally, in one embodiment, after step S22 detects the target in the single frame picture, put the pedestrian target into the tracking queue (hereinafter also referred to as the tracking chain), and then use the target tracking algorithm to Goals are tracked and analyzed by default.

Optionally, before the above step of extracting the feature information of the target in the single frame picture, it further includes: constructing a metadata structure. Optionally, the feature information of the target is extracted according to the metadata structure, that is, the feature information of the target in the single frame picture is extracted according to the metadata structure. In one embodiment, the metadata structure includes basic attribute units of pedestrians, such as at least one of: camera address, time when the target enters and exits the camera, track information of the target at the current monitoring node, color worn by the target, or a screenshot of the target. For example, the metadata structure of pedestrians can be seen in Table 1 below, where the metadata structure can also include information required by other users but not included in the table below.

Optionally, in an embodiment, in order to save resources for network transmission, the metadata structure only contains some basic attribute information, and other attributes can be obtained by mining and calculating relevant information such as target trajectories.

Table 1 Pedestrian metadata structure

attribute name type describe camera ID short Camera node number Target Appearance Time long The time when the target enters the monitoring node target departure time long Time when the target leaves the monitoring node target trajectory point The trajectory of the target at the current node Target ID short target ID number target top color short 10 pre-defined colors target pants color short 5 pre-defined colors screenshot of target image Record the overall screenshot of the target Target head and shoulders screenshot image Record the screenshot of the target's head

In another embodiment, the metadata structure can also include basic attribute information of the vehicle, such as: camera address, time when the target enters and exits the camera, track information of the target at the current monitoring node, appearance color of the target, license plate number or At least one of the screenshots of the target.

It can be understood that the definition of the information contained in the metadata structure and the data type of the metadata is initially set according to the needs, or it can be specially specified in the numerous information that has been set according to the needs of the user after the initial setting The specific attribute information that needs to be obtained.

In one embodiment, the metadata structure is initially set to the camera address, the time when the target enters and exits the camera, the track information of the target at the current monitoring node, the color of the target's clothing, or the screenshot of the target. When performing target recognition, Users can specify the time to acquire the target entering and exiting the camera according to their own needs.

In one embodiment, when the target in the single frame picture is a pedestrian, the feature information of the pedestrian is extracted according to the structure of the metadata of the preset pedestrian, that is, the time when the pedestrian enters and exits the camera, the current camera address where the pedestrian is located, At least one of the time when a pedestrian enters and exits the camera, the trajectory information of the pedestrian at the current monitoring node, the color of the pedestrian's clothing, or the current screenshot of the pedestrian, or other target attribute information specially specified by the user, such as the pedestrian entering and exiting the camera. Time and the color of the pedestrian's clothing, etc.

Optionally, when the target is detected and recognized from a single frame of pictures, while acquiring the feature information of the target, the image of the target is intercepted from the original video frame, and then using A method of target detection and recognition based on deep learning) for model training.

In one embodiment, when the target detection is performed on a single frame picture, the detected target is a pedestrian, then the image of the detected pedestrian is intercepted from the original video frame, and then the head and shoulders, The upper body and lower body detection model segments the pedestrian, judges the clothing color information of the upper and lower body parts, and intercepts the pedestrian's head and shoulders picture.

In another embodiment, when the detected target is a vehicle when performing target detection on a single frame picture, the image of the detected vehicle is intercepted from the original video frame, and then the vehicle detection model is trained using the framework based on yolov2 Detect and recognize the vehicle, judge the appearance color of its body, recognize the license plate information, and intercept the picture of the vehicle. It can be understood that, since the recognized target type can be set and selected by the user, it is up to the manager to decide whether to detect and recognize the vehicle.

In yet another embodiment, when the detected target is an animal when performing target detection on a single frame picture, the image of the detected animal is intercepted from the original video frame, and then the detection model of the animal is trained using a framework based on yolov2 Detect and identify animals, judge their appearance, color, species and other information, and intercept pictures of animals. It can be understood that since the recognized target type can be set and selected by the user, it is up to the user to decide whether to detect and recognize the animal.

Optionally, one single-frame picture may be recognized for each target detection, or multiple single-frame pictures may be performed simultaneously.

In an embodiment, one single frame picture is used for target detection and recognition each time, that is, target detection and recognition is only performed on a target in one single frame picture each time.

In another embodiment, target detection and recognition may be performed on multiple pictures at a time, that is, target detection and recognition may be performed on targets in multiple single-frame pictures at the same time.

Optionally, after model training based on the yolov2 framework, ID (IDentity) labeling is carried out on the detected targets, so as to facilitate association during follow-up tracking. Wherein, the ID numbers of different target categories can be preset, and the upper limit of the ID numbers is set by the user.

Optionally, the ID labeling is automatically performed on the detected and identified targets, or the ID labeling can be performed manually.

In one embodiment, the detected and identified targets are labeled, wherein, depending on the type of the detected target, there will be gaps in the ID numbers of the tags. For example, the ID number of a pedestrian can be set as: number + number, and the vehicle: Uppercase letters + numbers, animals: lowercase letters + numbers, for easy association during follow-up. The setting rules can be set according to the habits and preferences of the user, and will not be repeated here.

In another embodiment, the detected and identified targets are marked, wherein, depending on the category of the detected targets, the ID numbers marked on the targets belong to different intervals. For example, the ID number of the detected pedestrian object is set in the interval 1 to 1,000,000, and the ID number of the detected vehicle object is set in the interval 1,000001 to 2,000,000. Specifically, it can be determined according to the initial setting personnel setting, and can also be adjusted and changed according to needs.

Optionally, the ID labeling of the detected targets can be done automatically by the system in advance, or can be done manually by the user.

In one embodiment, when a pedestrian or a vehicle target is detected in a single frame picture, the system will automatically convert the detected target according to the type of the detected target, and then automatically carry out the ID number that has been marked before. ID number.

In another embodiment, the user manually IDs the objects in the picture. ID labeling can be performed on single-frame image targets that have not been automatically ID-labeled by the system, or missed targets or other targets outside the preset detection target category, and ID labeling can be performed by the user independently.

Optionally, referring to FIG. 3 , in one embodiment, before performing target detection and recognition on a single frame picture in step S22, the method further includes:

S21: Divide the video into single-frame pictures.

Optionally, the step of dividing the video into single-frame pictures is to divide the video read in step S10 into single-frame pictures in preparation for step S22.

Optionally, in an embodiment, the step of dividing the video into single-frame pictures is to divide the video read in step S10 into frames skipped at equal intervals or skipped at unequal intervals.

In one embodiment, the step of dividing the video into single-frame pictures is to divide the video read in step S10 with equidistant frame skipping, and the number of skipped frames is the same, that is, the equidistant skipping is the same The number of frames is divided into single-frame pictures, and the number of skipped frames is the number of frames that do not contain important information, that is, the number of frames that can be ignored. For example, one frame is skipped in the middle of the equidistant interval, and the video is divided, that is, the tth frame, the t+2th frame, and the t+4th frame are taken, and the number of skipped frames is the t+1th frame, the t+th frame 3 frames, the number of skipped frames above is the number of frames that are judged not to contain important information, or the number of frames skipped above is the number of frames that coincide with the number of frames taken or have a high degree of overlap number of frames.

In another embodiment, the step of dividing the video into single-frame pictures is to divide the video read in step S10 into skipped frames at unequal intervals, that is, the number of skipped frames may be different, ranging from The number of skipped frames of the spacing is divided into single-frame pictures. The number of skipped frames is the number of frames that do not contain important information, that is, the number of frames that can be ignored. The number of frames that do not contain important information is determined , and the judgment result is indeed an unimportant number of frames. For example, frame skipping at unequal intervals, that is, take the tth frame, then skip 2 frames and get t+3 frames, then skip 1 frame and get t+5 frames, then skip 3 frames and get t+9 frames, Among them, the number of skipped frames includes t+1 frame, t+2 frame, t+4 frame, t+6 frame, t+7 frame, t+8 frame, etc., and the number of skipped frames mentioned above is The number of frames judged not to contain the information required for this analysis.

In different embodiments, the step of dividing the video into single-frame pictures may be automatically divided into single-frame pictures by the system, or it may be selected by the user whether to divide the video into single-frame pictures, or it may be The user manually inputs a single-frame image that has been pre-segmented.

Optionally, in one embodiment, after the step of dividing the video into single-frame pictures is completed, that is, when the read-in video is divided into single-frame pictures, step S22 is automatically performed on the single-frame pictures obtained by segmentation, namely The target detection and recognition is performed on the single frame of the segmented picture, or the user may decide whether to perform the target detection and recognition described in step S22 on the single frame of the segmented picture.

Optionally, in the process of detecting and identifying the targets, statistical calculations are performed on the detection and identification values of each target according to a certain rule.

In one embodiment, after step S22, the total number of frames (the total number of frames that occur) in the current monitoring node is detected for a certain target, wherein the number of frames whose detection value is A, the number of frames whose detection value is B, etc. etc. (the detection value can have multiple or one type, subject to the detection result), and save the statistical results for calling.

Optionally, correction methods are mainly divided into trajectory correction and target attribute correction.

Optionally, after the structured data of each target is obtained through target detection, the obtained structured data is corrected. That is to correct the false detection data in the structured data. The correction is to vote according to the weight ratio. In the end, the data value of the majority probability is the correct value, and the data value of the minority result is the false detection value.

In one embodiment, after statistical calculation (calling the above-mentioned statistical results), it is found that in step S22, the number of frames in which a certain target is detected and recognized in the current monitoring node is 200 frames, and the top of the target is detected in 180 frames. The color is red, and the coat color of the target detected in 20 frames is black, and votes are made according to the weight ratio, and the accurate value of the target is finally corrected. The coat color is red, and the corresponding value in the structured data is changed to red, and the final completion Correction.

Optionally, the trajectory correction is specifically as follows: Assume that a target appears in a monitoring scene for T frames, so the set of its trajectory points can be obtained as G={p1,p2,...,p _N }, and the calculated trajectory points are at The mean and deviation of the X-axis and Y-axis, and then remove the abnormal and noise track points, the specific expression is:

In one embodiment, during trajectory correction, trajectory points with small deviations or small average values are eliminated to reduce noise point interference.

Optionally, the target attribute correction is specifically as follows: the target attribute correction is to correct the attribute value of the same target based on a weighted determination method. Assume that the color label of a target's shirt is label={"red", "black", "white", ...}, that is, there are T categories for a certain attribute value. First convert it into a digital code L=[m ₁ ,m ₂ ,m ₃ ,...,m _T ]; then calculate the code value x with the highest frequency and its frequency F, and finally directly output the attribute value Y of the target (accurate value). The specific expression is as follows:

F＝T-||Mm _x || ₀

Y=label[m _x ]

The above formula needs to be satisfied,

Optionally, in one embodiment, the present invention combines the YOLO target detection framework for target recognition and positioning, and uses the GoogLeNet network to extract the feature vector of each target for subsequent target matching. GoogLeNet is a 22-layer deep CNN neural network proposed by Google in 2014, which is widely used in image classification, recognition and other fields. Since the feature vectors extracted by the deep-level deep learning network have better robustness and distinguishability, the above steps can better improve the accuracy of subsequent target tracking.

S23: Track the target to obtain a tracking result.

Optionally, in the step of tracking the detected target to obtain the tracking result, the tracked target is the target detected in step S22 or other targets specified by the user, and step S23 further includes: tracking the target, recording The time when the target enters or leaves the monitoring node, and the various positions that the target passes through, to obtain the target's trajectory. Specifically, how to track the target to obtain the tracking result. Based on this, the present application provides an improved multi-target tracking method based on KCF and Kalman, which will be described in detail below.

In another embodiment, the video processing method provided by the present application further includes step S24 on the basis of steps S21, S22 and S23 in the above embodiment, or this embodiment only includes steps S21, S22 and S24, see Fig. 4 and Figure 5. It can be understood that the video structured processing method based on target behavior attributes (referred to as video structured processing) realizes the conversion of video data into structured data, wherein the specific conversion process includes: target detection and recognition, target trajectory tracking and extraction, and target anomalies Behavioral detection. In one embodiment, the video structural processing includes target detection and recognition and target trajectory extraction. In another embodiment, the video structuring process includes object detection, object trajectory extraction, and object abnormal behavior detection.

S24: Perform abnormal behavior detection on the target.

Optionally, step S24 is an operation of performing abnormal behavior detection on the target detected and identified in the above step S21.

Optionally, the abnormal behavior detection includes pedestrian abnormal behavior detection and vehicle abnormal behavior detection, wherein the abnormal behavior of pedestrians includes: running, fighting and rioting, and the abnormal behavior of traffic includes: collision and speeding, etc.

Through the above methods, the video is processed to obtain important data, which can avoid excessive data volume and greatly reduce the pressure of network transmission.

In one embodiment, when the abnormal behavior detection is performed on the pedestrian targets detected in step S21, if it is determined that more than or equal to a preset number of people running in a monitoring node, it can be determined that a crowd riot has occurred. For example, it can be set that when step S24 determines that 10 people are running abnormally, it can be determined that crowd riots have occurred. In other embodiments, the threshold for determining riots depends on specific circumstances.

In another embodiment, it can be set that when step S24 determines that two vehicles have collision abnormalities, it can be determined that a traffic accident has occurred, and when step S24 determines that more than three vehicles have collision abnormal behaviors, it can be determined that a major traffic accident has occurred. It can be understood that the determined number of cars can be set and adjusted according to needs.

In yet another embodiment, when it is detected in step S24 that the speed of the vehicle exceeds the preset speed value, it can be determined that the vehicle is a speeding vehicle, and the corresponding video of the vehicle can be screenshotted and saved. information. The vehicle information includes the license plate number.

Optionally, in an embodiment, when an abnormal behavior is detected in step S24, the monitoring node will perform sound and light alarm processing.

In one embodiment, the content of the sound and light alarm includes broadcasting voice prompt content: such as "Please don't be crowded, pay attention to safety!" or other preset voice prompt content; the content of the sound and light alarm also includes: opening the corresponding monitoring node The warning lights are used to remind passing people and vehicles to pay attention to safety.

Optionally, the severe level of abnormal behavior is set according to the number of people who have abnormal behavior, and different severe levels correspond to different emergency treatment measures. The severe levels of abnormal behavior can be divided into yellow, orange and red. The emergency measure corresponding to the abnormal behavior of the yellow level is to sound and light alarm, the emergency measure corresponding to the abnormal behavior of the orange level is to perform sound and light alarm while connecting the security personnel of the responsible point for monitoring, and the abnormal behavior measure of the red warning is to carry out sound and light alarm The security personnel at the responsible point for alarm and online monitoring will also timely alarm online.

In one embodiment, when the number of abnormal behaviors is 3 or less, the abnormal behavior of the crowd is set as a yellow level; when the number of abnormal behaviors is greater than 3 and less than or equal to 5 people, the orange level of the crowd Abnormal behavior; when the number of abnormal behavior exceeds 5 people, the abnormal behavior of the crowd is set as red level. Wherein, the specific set number of people can be adjusted according to actual needs, and will not be repeated here.

Optionally, in one embodiment, after the step of detecting the abnormal behavior of the target, the following steps are further included: if the abnormal behavior is detected, the screenshot of the current video frame image is saved and combined with the detected characteristic information of the target in which the abnormal behavior occurs Packaged and sent to the cloud server.

Optionally, the feature information corresponding to the target with abnormal behavior may include: camera ID, abnormal event type, abnormal behavior occurrence event, abnormal behavior screenshot, etc., and may also include other types of information required. The information contained in the metadata structure of the abnormal behavior sent to the cloud server includes the structure in Table 2 below, and may also include other types of information.

Table 2 Metadata structure of abnormal behavior

attribute name type of data describe camera ID short Unique camera ID exception type short Two exception behaviors are predefined Abnormal occurrence time long abnormal occurrence time Screenshot of exception image Record abnormal behavior screenshots

In one embodiment, when an abnormal behavior of a target is detected, if an abnormal behavior of a pedestrian sending a fight is detected, the corresponding screenshot of the current video frame image is saved, and the structured data corresponding to the screenshot and the target with the abnormal behavior Package them together and send them to the cloud server. While sending the screenshots of detected abnormal behaviors to the cloud server, the monitoring node performs sound and light alarm processing, and initiates corresponding emergency measures according to the level of abnormal behaviors.

In another embodiment, when the abnormal behavior of the target is detected, when a crowd riot is detected, the screenshot of the current video frame image is saved and sent to the cloud server for further processing by the cloud server. Light alarm, and start corresponding emergency measures according to the level of abnormal behavior.

Specifically, in one embodiment, the step of detecting abnormal behavior of the target includes: extracting optical flow motion information of multiple feature points of one or more targets, and performing clustering and abnormal behavior detection according to the optical flow motion information. Based on this, the present application also provides a method for detecting abnormal behavior based on clustering optical flow features, which will be described in detail below.

The above-mentioned structured video processing method based on target behavior attributes can convert unstructured video data into structured data and improve the real-time performance of food processing analysis.

Referring to FIG. 6 , it is a schematic flowchart of an embodiment of an improved multi-target tracking method based on KCF and Kalman provided by the present application. This method is also step S23 in the above embodiment, specifically including steps S231 to S234. Specifically include the following steps:

S231: Combine the tracking chain and the detection frames corresponding to the first multiple targets in the last frame of the picture to predict the tracking frame of each target in the current frame of the first multiple targets.

Optionally, the tracking chain is calculated based on tracking multiple targets in all single-frame pictures or partial continuous single-frame pictures before the current frame picture, and gathers multiple targets in all previous pictures The trajectory information and experience value of .

In an embodiment, the tracking chain is calculated based on object tracking of all pictures before the current frame picture, and includes all information of all targets in all the frame pictures before the current frame picture.

In another embodiment, the tracking chain is calculated based on object tracking of partially continuous pictures preceding the current frame picture. Among them, the more consecutive pictures are tracked and calculated, the higher the accuracy of the budget will be.

Optionally, combining the feature information of the targets in the tracking chain, and according to the detection frames corresponding to the first multiple targets in the previous frame of pictures, predict the tracking frames of the tracked first multiple targets in the current frame picture, for example Predict where the first multiple objects are likely to appear in the current frame.

In an embodiment, the above steps may predict the positions of the tracking frames of the first multiple targets in the current frame, that is, obtain the predicted values of the first multiple targets.

In another embodiment, the above steps may predict the positions of the tracking frames of the first multiple objects in the frame next to the current frame. Wherein, the predicted positions of the tracking frames of the first multiple targets in the next frame of the current frame have a slightly larger error than the predicted positions of the tracking frames of the first multiple targets in the current frame.

Optionally, the first multiple targets refer to all detected targets in the previous frame of pictures.

S232: Obtain the tracking frames corresponding to the first plurality of targets in the previous frame of the picture in the current frame, and the detection frames of the second plurality of targets in the current frame of the picture.

Specifically, the second plurality of targets refers to all detected targets in the current frame picture.

Optionally, the tracking frames corresponding to the first multiple targets in the previous frame picture in the current frame, and the detection frames of the second multiple targets in the current frame picture are acquired. The tracking frame is a rectangular frame when predicting the position where the first multiple targets will appear in the current frame, or a frame of other shapes, and the frame includes one or more targets.

Optionally, when obtaining the tracking frames corresponding to the first multiple targets in the current frame in the previous frame of pictures, and the detection frames of the second multiple targets in the current frame of pictures, the acquired tracking frames and detection frames contain tracking The feature information of the target corresponding to the frame and the detection frame respectively. For example, the location information, color features and texture features of the target. Optionally, the corresponding feature information can be set by the user according to needs.

S233: Establish an object correlation matrix of the tracking frames of the first multiple targets in the current frame and the detection frames of the second multiple targets in the current frame.

Optionally, according to the corresponding tracking frames in the current frame of the first multiple targets in the previous frame picture acquired in step S232 and the detection frames corresponding to the second multiple targets detected in the current frame picture, establish target incidence matrix.

In one embodiment, for example, the number of the first plurality of targets in the previous frame picture is N, and the number of targets detected in the current frame is M, then a target correlation matrix W of size M×N is established, wherein:

The value of A _ij (0<i≤M;0<j≤N) is determined by dist(i,j), IOU(i,j), m(i,j), specifically, the following formula can be expressed:

Among them, I _W , I _h are the width and height of the image frame; dist(i, j) is the tracking frame of the next frame predicted by the jth target in the tracking chain obtained in the previous frame and the detection and identification in the current frame. The centroid distance of the detection frame of the i-th target, d(i, j) is the centroid distance normalized by the diagonal 1/2 distance of the image frame, and m(i, j) is the two target feature vectors The Euclidean distance of , F _Mi and F _Nj are the feature vectors extracted based on the GoogLeNet network. The feature vectors are extracted using the model of the CNN framework, which is more robust and distinguishable than the traditional manual feature extraction. Among them, the purpose of normalization is mainly to ensure that the influence of d(i, j) and IOU(i, j) on A(i, j) is consistent. IOU(i, j) represents the overlap rate of the tracking frame in the current frame predicted by the jth target in the tracking chain of the previous frame and the detection frame of the jth target detected and recognized in the current frame, that is, the above tracking frame and The intersection of detection boxes is compared to their union. The specific expression of IOU is:

Optionally, the value range of IOU(i, j) is 0≦IOU(i, j)≦1, and the larger the value is, the larger the overlap rate between the tracking frame and the detection frame is.

In one embodiment, when the target is still, the center of mass position detected by the same target in the two frames before and after should be at the same point or the deviation is small, so the value of IOU should be approximately 1, d(i, j) It should also tend to 0, so the value of A _ij is small, and when the target matches, the value of m(i, j) is small, so when matching, the target with ID=j in the tracking chain and the detection chain ID = i is more likely to match the detection target successfully; if the position of the same target detection frame in the two frames before and after is far away and there is no overlap, then the IOU should be 0, and the value of m(i, j) is larger, so The larger the value of d(i, j), the less likely it is that the target with ID=j in the tracking chain will be successfully matched with the detection target with ID=i in the detection chain.

Optionally, the establishment of the target correlation matrix refers to the centroid distance, IOU, and Euclidean distance of the target feature vector, and can also refer to other feature information of the target, such as: color features, texture features, etc. It is understandable that when more indicators are referenced, the accuracy rate will be higher, but the real-time performance will decrease slightly due to the increase in calculation amount.

Optionally, in an embodiment, when a better real-time performance needs to be ensured, in most cases only the position information of the target in the two captured images is used to establish the target correlation matrix.

In one embodiment, the tracking frame corresponding to the first plurality of targets and the detection frame of the current frame corresponding to the second plurality of targets are established with reference to the position information of the target and the clothing color of the target (it may also be the appearance color of the target). Incidence matrix.

S234: Use the target matching algorithm to correct, so as to obtain the actual position corresponding to the first part of the target in the current frame.

Optionally, using the target matching algorithm, the target value is corrected according to the observed value of the actually detected target and the predicted value corresponding to the target detection frame in step S231, so as to obtain the actual values of the first multiple targets in the current frame. The positions, that is, the actual positions of the targets in the current frame of the targets of the second multiple targets that simultaneously appear in the current frame among the first multiple targets in the previous frame. It is understandable that the observation value of the second multiple targets in the current frame will have certain errors due to factors such as the clarity of the segmented image, so the combination of the tracking chain and the first multiple targets in the previous frame is adopted. For the detection frame in the previous frame, the predicted positions of the first multiple targets in the current frame are corrected to the actual positions of the second multiple targets.

Optionally, the target matching algorithm is the Hungarian algorithm (Hungarian), the observed value is the characteristic information of the target obtained when the target is detected and identified in step S22, including the category of the target and the position information of the target, etc., and the predicted value of the target is the target in step S231. Combining the tracking chain and the position of the target in the previous frame to predict the position value of the target in the current frame and other feature information. Among them, the location information of the target is used as the main judgment basis, and other characteristic information is used as the secondary judgment basis.

Optionally, in an embodiment, the detection frame of the second plurality of targets matches the tracking frame of the first plurality of targets in the current frame and is defined as the first part of targets, while the first plurality of targets The tracking frame in the current frame is successfully matched with the detection frames of the second multiple targets in the current frame, which is also defined as the first part of targets, that is, each group of tracking frames and detection frames that are successfully matched are from the same target. Wherein, it can be understood that the successful matching of the detection frames in the second plurality of targets with the tracking frames of the first plurality of targets in the current frame means: one-to-one correspondence between position information and other feature information, or the corresponding item The number is relatively large, that is, the probability of the corresponding item number is relatively high, which means the matching is successful.

In another embodiment, the number of the first part of targets is less than the number of the first multiple targets, that is, only part of the tracking frames of the first multiple targets in the current frame can successfully match with the detection frames of the second multiple targets, and A part cannot be successfully matched according to the feature information of the matching basis in the current frame.

Optionally, in different implementations, the step of successfully matching the detection frames of the second plurality of targets in the current frame with the tracking frames of the first plurality of targets in the previous frame in the current frame includes: according to the The detection frame of the second plurality of targets and the centroid distance and/or overlap rate of the tracking frame of the first plurality of targets in the previous frame in the current frame determine whether the matching is successful.

In one embodiment, the detection frame of one or more targets among the second multiple targets in the current frame and the tracking of one or more targets among the first multiple targets in the previous frame in the current frame When the centroids of the boxes are very close and the overlap rate is high, it is judged that the target matching is successful. It can be understood that the time interval between the segmentation of two adjacent frames of pictures is very short, that is, the target moves a very small distance during this time interval, so it can be determined that the target matching in the two frames of pictures is successful at this time.

Optionally, the second plurality of targets includes a first part of targets and a second part of targets, wherein, as can be seen from the above, the first part of targets is: the detection frame in the second plurality of targets and the detection frame of the first plurality of targets in the current frame The tracking box matches the target successfully. The second part of the target is: the detection frame in the second multiple targets, the target that does not match the tracking frame of the first multiple targets in the current frame, defines the target that is not recorded in the tracking chain in the second part of the target to add a target. It can be understood that in the second part of the target, there may be another type of target besides the newly added target: the target that has not been successfully matched in the first multiple targets but has appeared in the tracking chain.

In an embodiment, the number of the second part of targets may be 0, that is, the detection frames of the second multiple targets in the current frame and the tracking frames of the first multiple targets in the current frame can all match successfully, so at this time The number of targets in the second part is 0.

Optionally, after the step of using the target matching algorithm to perform correction analysis to obtain the actual position corresponding to the first part of the target in the current frame includes: screening out new targets in the second part of targets; adding the new target to the tracking chain. Another embodiment further includes: initializing the corresponding filter tracker with the initial position and/or feature information of the newly added target. In one embodiment, the filter tracker includes a Kalman filter (kalman), a kernelization correlation filter (kcf), and a filter combined with a Kalman filter and a kernelization correlation filter. Kalman filter, correlating correlation filter and the combination of Kalman filter and correlating correlating filter are multi-target tracking algorithms based on programming. Wherein, the filter combining the Kalman filter and the correlating correlation filter refers to a filter structure realized by an algorithm structure combining the structures of the Kalman filter and the correlating correlation filter. In other embodiments, the filter tracker may also be other types of filters, as long as the same function can be realized.

Optionally, the data of the tracking chain is calculated by training data of the previous frame and all frames before the previous frame, and the targets in the tracking chain include the first part of the targets and the third part of the targets described above. Specifically, the first part of targets refers to targets whose tracking frames in the current frame of the first multiple targets successfully match the detection frames of the second multiple targets. The third part of the target refers to: the target in the tracking chain does not successfully match the second multiple targets.

It can be understood that the third part of the targets is essentially all targets in the tracking chain except the first part of targets that successfully match the second plurality of targets.

Optionally, after the step of using the target matching algorithm to perform correction analysis in step S234 to obtain the actual position corresponding to the first part of the target in the current frame, it includes: adding 1 to the count value of the number of lost frames of the target corresponding to the third part of the target, and when the target is lost When the frame count value is greater than or equal to the preset threshold, the corresponding target is removed from the tracking chain. It can be understood that the preset threshold of the count value of the number of lost frames is preset and can be adjusted as required.

In an embodiment, when the lost frame count value corresponding to an object in the third part of objects is greater than or equal to a preset threshold, this object is removed from the current tracking chain.

Optionally, when a target is removed from the current tracking chain, the structured data corresponding to the target is uploaded to the cloud server, and the cloud server will combine the structured data of the target or the experience value in the database, and again Perform in-depth analysis of the trajectory or anomalous behavior of the target.

Wherein, it can be understood that when the structured data corresponding to the target removed from the tracking chain is sent to the cloud server, the system executing the method can choose to trust and interrupt the in-depth analysis of the target by the cloud server.

Optionally, after the step S234 of using the target matching algorithm to perform correction analysis to obtain the actual position corresponding to the first part of the target in the current frame, it includes: adding 1 to the count value of the number of lost frames of the target corresponding to the third part of the target, and adding 1 to the count value When it is less than the preset threshold, the third part of the target is partially tracked to obtain the current tracking value.

Further, in an embodiment, correction is performed according to the current tracking value of the third part of the target and the corresponding predicted value of the third part of the target, so as to obtain the actual position of the third part of the target. Specifically, in one embodiment, the current tracking value is obtained when the third part of the target is partially tracked by a filter combined with a kernelization correlation filter and a Kalman filter and a kernelization correlation filter, and the predicted value is Kalman The filter (kalman) predicts the location value of the third part of the object.

Optionally, the tracking of the target detected in the above step S22 is jointly completed by a filter combined with filters of a Kalman filter tracker (kalman) and a kernelization correlation filter tracker (kcf).

In one embodiment, when the tracked targets are all targets that can be matched, that is, when no target is suspected to be lost, only the Kalman filter tracker (kalman) can be used to complete the target tracking work.

In another embodiment, when there is a target suspected of being lost among the tracked targets, the filter combined with the Kalman filter tracker (kalman) and the kernelization correlation filter tracker (kcf) is called to cooperate to complete the target tracking The tracking work can also be completed successively by cooperation of a Kalman filter tracker (kalman) and a kernelization correlation filter tracker (kcf).

Optionally, in one embodiment, step S234 uses the target matching algorithm to perform correction to obtain the actual position corresponding to the first part of the target in the current frame. The step includes: for each target in the first part of targets, tracking The predicted value corresponding to the frame and the observed value corresponding to the detection frame of the current frame are corrected to obtain the actual position of each target in the first part of targets.

In an embodiment, the prediction value corresponding to the tracking frame of each target in the first part of the target in the current frame can be understood as: combining the experience value in the tracking chain and the position information in the previous frame to predict each target in the first part of the target position information in the current frame, and then combined with the observed actual positions of the first part of the targets in the current frame (ie observed values), the actual positions of the targets in the first part of the targets are corrected. This operation is used to reduce the problem of inaccurate measurement of the actual value of each target caused by the error of the predicted value or the observed value.

Optionally, in one embodiment, the above-mentioned improved multi-target tracking method based on KCF and Kalman can realize tracking and analysis of multiple targets, record the time when the target enters the monitoring node and each A movement position, thereby generating a trajectory chain, which can specifically and clearly reflect the movement information of the target at the current monitoring node.

Referring to FIG. 7 , it is a schematic flowchart of an embodiment of an abnormal behavior detection method based on clustering optical flow features provided by the present application. This method is also step 24 of the above embodiment, including steps S241 to S245. The specific steps are as follows:

S241: Perform optical flow detection on areas where the detection frames of one or more targets are located.

Optionally, before detecting the abnormal behavior of the target, the detection and recognition of the target has been completed based on the preset algorithm, and the detection frame corresponding to each target and the position of the detection frame are obtained when the target in the single frame picture is detected , and then perform optical flow detection on the detection boxes of one or more objects. Among them, the optical flow contains the motion information of the target. Optionally, the preset algorithm may be the yolov2 algorithm, or other algorithms with similar functions.

Understandably, the detection frame corresponding to each target in the acquired single frame picture and the position of the detection frame, because the center of the detection frame will be close to the center of gravity of the target, so it can be used to obtain each pedestrian target in each frame of image Or the location information of other types of targets.

In one embodiment, the essence of performing optical flow detection on the detection frames of one or more objects is to obtain the motion information of the optical flow points in the detection frames corresponding to the objects, including the speed and direction of the motion of the optical flow points.

Optionally, the optical flow detection is to obtain the information of each motion characteristic of the optical flow point, which is completed by the LK (Lucas–Kanade) pyramid optical flow method or other flow methods with the same or similar functions.

Optionally, the optical flow detection can be performed on the detection frame of one target in each frame of the picture at a time, or the optical flow detection can be performed on the detection frames of multiple targets in each frame of the picture at the same time, generally the optical flow detection is performed each time The number of targets is based on the initial settings of the system. It can be understood that this setting can be adjusted and set as needed. When fast optical flow detection is required, it can be set to detect the detection frames of multiple targets in each frame of picture at the same time. When very fine optical flow detection is required, it can be adjusted to perform optical flow detection on the detection frame of a target in each frame of picture at a time.

Optionally, in an embodiment, the optical flow detection is performed on a detection frame of a target in consecutive multiple frames of pictures each time, or may be detected on a detection frame of a target in a single frame of pictures.

Optionally, in another embodiment, the optical flow detection is performed on the detection frames of multiple or all targets in consecutive multi-frame pictures each time, or multiple or all targets in a single frame of pictures each time The detection frame is used for optical flow detection.

Optionally, in one embodiment, before performing optical flow detection on the target, the approximate location area of the target is detected in the above steps, and then directly in the area where the target appears in the two consecutive frames of images (it can be understood is the target detection area) for optical flow detection. Wherein, two consecutive frames of images for optical flow detection are images of the same size.

Optionally, in an embodiment, performing optical flow detection on the region where the detection frame of the target is located may be performing optical flow detection on the region where the detection frame of the target is located in a frame of pictures, and then saving the obtained data and information in the local memory , and then perform optical flow detection on the area where the detection frame of the target in the picture in the next frame or in the preset frame is located.

In one embodiment, the optical flow detection is performed on the detection frame and the region of one object at a time, and the optical flow detection is performed on the detection frames of all objects in the picture one by one.

In another embodiment, the optical flow detection is performed on multiple targets in a picture at the same time, that is, it can be understood that the optical flow detection is performed on all targets or the detection frames of some targets in a single frame of the picture each time.

In yet another embodiment, the optical flow detection is performed on the detection frames of all objects in multiple single-frame pictures each time.

In yet another embodiment, the optical flow detection is performed on specially designated target detection frames of the same category in multiple single-frame pictures each time.

Optionally, after step S241, the obtained optical flow information is added to the space-time model, so that the optical flow vector information of multiple frames of images before and after is obtained through statistical calculation.

S242: Extract the optical flow motion information of the feature points corresponding to the detection frame in at least two consecutive frames of images, and calculate the information entropy of the region where the detection frame is located.

Optionally, step 242 extracts the optical flow motion information of the feature points corresponding to the detection frame in at least two consecutive frames of images, and calculates the information entropy of the area where the detection frame is located, which is the feature corresponding to the detection frame area in at least two consecutive frames of images points, where the optical flow motion information refers to the movement direction and speed of the optical flow point, that is, to extract the movement direction and distance of the optical flow point, and then calculate the movement speed of the optical flow point. The feature point can represent A collection of one or more pixel points of object feature information.

Optionally, after extracting the optical flow motion information of the feature points corresponding to the detection frame in two consecutive frames of images, and calculating the information entropy of the area where the detection frame is located according to the extracted optical flow motion information, it can be understood that the information entropy is calculated based on the optical flow information of all optical flow points in the target detection area.

Optionally, step 242 extracts the optical flow motion information of the feature points corresponding to the detection frame in at least two consecutive frames of images, and calculates the information entropy of the region where the detection frame is located, which is the LK (Lucas–Kanade) pyramid optical flow method (LK pyramid optical flow method (hereinafter referred to as the LK optical flow method) extracts the pixel optical flow feature information in the rectangular frame area containing only pedestrian targets in adjacent frames In addition, a Graphics Processing Unit is used to accelerate the LK optical flow extraction algorithm, so as to realize real-time online extraction of optical flow feature information of pixels. Wherein, the optical flow characteristic information refers to the optical flow vector information, which may be referred to as the optical flow vector.

Optionally, due to the optical flow vector extracted by the optical flow algorithm is composed of two two-dimensional matrix vectors constitute, namely

Wherein, each point in the matrix corresponds to the position of each pixel in the image; Represents the pixel interval of the same pixel point moving on the X axis in adjacent frames, Represents the pixel interval of the same pixel moving on the Y axis in adjacent frames.

Optionally, the pixel interval refers to the moving distance of the feature points in two adjacent frames of images, which can be directly extracted by the LK optical flow extraction algorithm.

In one embodiment, step 242 is to calculate the optical flow motion information of the feature points corresponding to the detection frames of each target in the single-frame image that has completed the target detection and has obtained the detection frame of the target detection . The feature point can also be interpreted as the point where the gray value of the image changes drastically or the point with a large curvature on the edge of the image (ie, the intersection point of two edges). This operation can reduce the calculation amount and improve the calculation efficiency.

Optionally, step S242 can simultaneously calculate the optical flow information of all detection frames or feature points corresponding to some detection frames in two consecutive frames of images, or simultaneously calculate the feature points corresponding to all detection frames in more than two consecutive images For optical flow information, the number of images calculated each time is pre-set in the system and can be set as needed.

In one embodiment, step S242 simultaneously calculates the optical flow information of the feature points corresponding to all the detection frames in two consecutive frames of images.

In another embodiment, step S242 simultaneously calculates the optical flow information of the feature points corresponding to all the detection frames in more than two consecutive images.

Optionally, step S242 may simultaneously calculate the optical flow information of the detection frames corresponding to all targets in at least two consecutive frames of images, or simultaneously calculate the optical flow information of the detection frames of specially designated and corresponding targets in at least two consecutive frames of images stream information.

In one embodiment, step S242 is to simultaneously calculate the optical flow information of the detection frames corresponding to all targets in at least two consecutive frames of images, such as: the detection frames corresponding to all targets in the t-th frame and the t+1-th frame of images optical flow information.

In another embodiment, step S242 is to simultaneously calculate the detection frame of a specially designated and corresponding target in at least two consecutive frames of images, such as: a class A target in the tth frame and a class A' target in the t+1th frame image, The optical flow information of the detection frames corresponding to the targets with ID numbers 1 to 3, that is, to simultaneously extract and calculate the target A ₁ , A ₂ , A ₃ and their corresponding targets A ₁ ′, A ₂ ′, A ₃ ′ The optical flow information of the detection box.

S243: Establish clustering points according to the optical flow motion information and information entropy.

Optionally, clustering points are established according to the optical flow motion information extracted in step S242 and the calculated information entropy. The optical flow motion information is information that reflects the motion characteristics of the optical flow, including the direction of motion and the speed of motion, and may also include other related motion feature information. The information entropy is calculated based on the optical flow motion information.

In one embodiment, the optical flow motion information extracted in step S242 includes at least one of motion direction, motion distance, motion speed, and other related motion feature information.

Optionally, before step S243 establishes clustering points according to the optical flow motion information and the calculated information entropy, the optical flow should be clustered by using the K-means algorithm (k-mean). Among them, the number of clustering points can be determined according to the number of detection frames when the target is detected, and the basis for clustering the optical flow is to establish the optical flow points with the same motion direction and motion speed as cluster points. Optionally, in an embodiment, the value of K ranges from 6 to 9, of course, the value of K may also be other values, which will not be repeated here.

Optionally, the cluster points are a set of optical flow points whose motion directions and motion speeds are the same or approximately the same.

S244: Calculate the kinetic energy of the cluster points or the kinetic energy of the area where the target detection frame is located. Specifically, the kinetic energy of the cluster points established in step S245 is calculated in units of the cluster points established in step S243, or the kinetic energy of the region where the target detection frame is located is calculated at the same time.

In one embodiment, at least one of the kinetic energy of the cluster points established in step S243 or the kinetic energy of the area where the target is located is calculated. It can be understood that in different embodiments, one of the required calculation methods can be configured according to specific needs, and two calculation methods for calculating the kinetic energy of the cluster points or the kinetic energy of the target area can also be configured at the same time. When using one, you can manually choose not to calculate the other. Optionally, according to the position of the clustering point, use the motion vectors of N frames before and after it to establish a motion space-time container, and calculate the information entropy of the optical flow histogram (HOF) of the detection area where each clustering point is located and the clustering point The average kinetic energy of the collection.

Optionally, the formula for the kinetic energy of the area where the target detection frame is located is as follows:

Optionally, i=0,...,k-1 represents the sequence number of the optical flow in the region where the single target detection frame is located, and k represents the total number of clustered optical flows in the single target region. In addition, for the convenience of calculation, let m= 1. Optionally, in an embodiment, the value of K ranges from 6 to 9, of course, the value of K may also be other values, which will not be repeated here.

S245: Judging the abnormal behavior according to the kinetic energy and/or information entropy of the cluster points.

Optionally, according to the kinetic energy of the cluster point calculated in step S244 or the kinetic energy of the area where the target detection frame is located, it is judged whether the target corresponding to the cluster point has abnormal behavior, wherein when the target is a pedestrian, the abnormal behavior includes: Running, fighting and rioting, when the target is a vehicle, abnormal behavior includes hitting and speeding.

Specifically, the two abnormal behaviors of fighting and running are related to the information entropy of the area where the target detection frame is located and the kinetic energy of the cluster points. That is, when the abnormal behavior is fighting, the optical flow information entropy of the area where the target detection frame is located is relatively large, and the kinetic energy of the cluster point corresponding to the target or the kinetic energy of the area where the target is located is also large. When the abnormal behavior is running, the kinetic energy of the cluster point corresponding to the target or the kinetic energy of the area where the target is located is relatively large, and the optical flow information entropy of the area where the target detection frame is located is relatively small. When no abnormal behavior occurs, the optical flow information entropy of the area where the detection frame corresponding to the target is located is small, and the kinetic energy of the cluster point corresponding to the target or the kinetic energy of the area where the target is located is also small.

Optionally, in an embodiment, the step of S245 judging the abnormal behavior according to the kinetic energy and/or information entropy of the cluster points further includes: if the optical flow information entropy of the region where the detection frame corresponding to the target is located is greater than or equal to the first threshold, and If the kinetic energy of the cluster point corresponding to the target or the kinetic energy of the area where the target detection frame is located is greater than or equal to the second threshold, it is judged that the abnormal behavior is fighting.

Optionally, in another embodiment, the step of judging the abnormal behavior according to the kinetic energy and/or information entropy of the cluster points further includes: if the information entropy of the area where the detection frame corresponding to the target is located is greater than or equal to the third threshold and less than the first Threshold, at the same time the kinetic energy of the cluster point corresponding to the target or the kinetic energy of the area where the target detection frame is located is greater than the second threshold. Then it is judged that the abnormal behavior is running.

In one embodiment, for example, information entropy is represented by H, and kinetic energy is represented by E.

Optionally, the judgment formula of the target running behavior is as follows:

In one embodiment, the present invention trains to get the result of running behavior The value range is The value of λ ₁ is 3000, where is used to represent the ratio of the optical flow information entropy H of the area where the target detection frame is located to the kinetic energy E of the area where the target detection frame is located, and λ ₁ is a preset kinetic energy value.

Optionally, the judgment formula for target fighting behavior:

In one embodiment, the present invention trains to obtain the fighting behavior The value range is The value of λ ₂ is 3.0, where is used to represent the ratio of information entropy H to kinetic energy E, and λ ₂ is a preset information entropy value.

Optionally, the judgment formula for normal behavior:

In one embodiment, in the present invention, the normal behavior λ ₃ obtained by training is 1500, λ ₄ is 1.85, λ ₃ is a preset kinetic energy value, and is smaller than λ ₁ , and λ ₄ is a preset information entropy value, and less than λ ₂ .

In an embodiment, when a pedestrian object is running, the optical flow energy of the cluster point corresponding to the pedestrian object is relatively large, and the optical flow information entropy is relatively small.

Optionally, when a crowd riot occurs, multiple pedestrian targets will be detected in a single frame of the picture first, and then when abnormal behavior detection is performed on the detected multiple pedestrian targets, it will be found that all of the multiple targets are running Abnormal, at this time it can be determined that a crowd riot has occurred.

In one embodiment, when abnormal behavior detection is performed on a plurality of targets detected in a single frame picture, when the kinetic energy of the cluster points corresponding to the targets exceeding the preset threshold number is relatively large, the optical flow information entropy is relatively large. Small; at this point, it can be determined that a crowd riot may have occurred.

Optionally, when the target is a vehicle, the determination of abnormal behavior is also based on the distance between most optical flow directions in the detection frame corresponding to the target and the detected vehicle (can be calculated from the position information), Determine if a collision has occurred. It can be understood that when most of the optical flow directions of the detection frames of two vehicle targets are opposite, and the distance between the two vehicles is very close, it can be judged that a collision event is suspected.

Optionally, the result of judging the abnormal behavior in step S245 is saved and sent to the cloud server.

The methods described in steps S241 to S245 above can effectively improve the efficiency and real-time performance of abnormal behavior detection.

Optionally, in one embodiment, step S242 extracts the optical flow motion information of the feature points corresponding to the detection frame in at least two consecutive frames of images, and before the step of calculating the information entropy of the area where the detection frame is located, it also includes: extracting at least two consecutive frames of images feature points.

Optionally, the feature points of at least two consecutive frames of images are extracted, the feature points of the target detection frame of two consecutive frames of images can be extracted each time, or the targets in multiple frames (more than two frames) of continuous images can be extracted each time The feature points of the detection frame, where the number of images extracted each time is set when the system is initialized, and can be adjusted as needed. Among them, the feature point refers to the point where the gray value of the image changes sharply or the point with a large curvature on the edge of the image (ie, the intersection point of two edges).

Optionally, in one embodiment, step S242 extracts the optical flow motion information of the feature points corresponding to the detection frame in at least two consecutive frames of images, and the step of calculating the information entropy of the area where the detection frame is located further includes: using a preset algorithm to calculate two consecutive The matching feature points of the target in the frame image are removed, and the unmatched feature points in two consecutive frames of images are removed.

Optionally, first, call the image processing function (goodFeaturesToTrack()) to extract the feature points (also called Shi-Tomasi corner points) in the target area that have been detected in the previous frame image, and then call the LK-pyramid optical flow The function calcOpticalFlowPyrLK() in the extraction algorithm calculates the feature points that match the target detected in the current frame and the previous frame, and removes the feature points that have not moved in the two frames before and after, so as to obtain the optical flow motion information of the pixels. Wherein, the feature points in this embodiment may be Shi-Tomasi corner points, or corner points for short.

Optionally, in an embodiment, before the step of establishing cluster points according to the optical flow motion information in step S245, the step further includes: drawing the optical flow motion direction of the feature points in the image.

In an embodiment, before the step of establishing the cluster points according to the optical flow motion information, drawing the optical flow motion direction of each feature point in each frame of image.

Optionally, referring to FIG. 8 , in an embodiment, step S243 further includes step S2431 and step S2432 after the step of establishing cluster points according to the optical flow motion information:

S2431: Establish a space-time container based on the position and motion vector of the target detection area.

Optionally, the space-time container is established based on the location information of the target detection area, that is, the position information of the target detection frame, and the motion vector relationship of the cluster points in the detection frame in multiple frames before and after.

Optionally, see FIG. 9 , which is a schematic diagram of a moving space-time container in an embodiment, where AB is the two-dimensional height of the space-time container, BC is the two-dimensional width of the space-time container, and CE is the depth of the space-time container. Among them, the depth CE of the space-time container is the number of video frames, ABCD represents the two-dimensional size of the space-time container, and the two-dimensional size represents the size of the target detection frame during target detection. It can be understood that the model of the space-time container may be other graphs, and when the graph of the target detection frame changes, the model of the space-time container will change accordingly.

Optionally, in an embodiment, when the graphic of the target detection frame changes, the corresponding created space-time container will change according to the graphic change of the target detection frame.

S2432: Calculate the average information entropy and average motion kinetic energy of the optical flow histogram of the detection frame corresponding to each cluster point.

Optionally, calculate the average information entropy and average kinetic energy of the optical flow histogram of the detection frame corresponding to each cluster point. Optical flow histogram HOF (Histogram of Oriented Optical Flow) is a schematic diagram of the probability of statistical distribution of optical flow points in a specific direction.

Optionally, the basic idea of HOF is to project the direction value of each optical flow point into the corresponding histogram bin, and weight it according to the magnitude of the optical flow. In the present invention, the value of the bin is is 12, where the calculation formula of the motion speed and direction of each optical flow point is as follows, and T refers to the time interval between two adjacent frames of images.

Among them, the use of the optical flow histogram can reduce the influence of factors such as the size of the target, the moving direction of the target, and the noise in the video on the optical flow characteristics of the target pixel.

Optionally, the type of abnormal behavior in different embodiments includes one of fighting and running, rioting or abnormal traffic.

In one embodiment, when the target is a pedestrian, the abnormal behavior includes: fighting, running and rioting.

In another embodiment, when the target is a vehicle, the abnormal behaviors are, for example: crashing and speeding.

Optionally, in one embodiment, calculating the average information entropy and average kinetic energy of the optical flow histogram of the detection frame corresponding to each cluster point is essentially calculating the optical flow of each cluster center in the N frame images before and after Average information entropy and average kinetic energy.

The above-mentioned abnormal behavior detection method can effectively improve the intelligence of current security, and at the same time, it can effectively reduce the amount of calculation in the process of abnormal behavior detection, and improve the efficiency, real-time and accuracy of the system for abnormal behavior detection of targets. .

Optionally, after the step of tracking the target to obtain the tracking result, the step further includes: sending the structured data of the target object that has left the current monitoring node to the cloud server.

Optionally, when tracking a target, if the characteristic information of a certain target, especially the location information, is not updated within a preset time, it can be determined that the target has left the current monitoring node, and the structured data of the target is sent to to the cloud server. The preset time can be set by the user, such as 5 minutes or 10 minutes, etc., which will not be described here.

In one embodiment, when the target is tracked, when it is found that the position information of a pedestrian, that is, the coordinate value, has not been updated within a certain preset time, it can be determined that the pedestrian has left the current monitoring node, and the pedestrian corresponds to The structured data is sent to the cloud server.

In another embodiment, when the target is tracked, when the position coordinates of a pedestrian or a vehicle are found to stay at the edge of the viewing angle of the monitoring node, it can be determined that the pedestrian or vehicle has left the current monitoring node, and the The structured data of pedestrians or vehicles is sent to the cloud server.

Optionally, the preset feature information (such as target attribute value, motion track, target screenshot, etc. and other required information) of the target determined to leave the current monitoring node is packaged into a preset metadata structure, and then encoded into The preset format is sent to the cloud server, and the cloud server analyzes the received packaged data, extracts the metadata of the target and saves it to the database.

In one embodiment, the preset feature information of the target that is determined to leave the current node is packaged into a preset metadata structure, and then encoded into a JSON data format and sent to the cloud server through the network, and the cloud server processes the received JSON data packet Analyze, extract the metadata structure, and then save it to the database of the cloud server. It can be understood that the preset feature information can be adjusted and set according to needs, and details will not be described here.

Referring to FIG. 10 , the present invention also provides a device 400 with a storage function, which stores program data. When the program data is executed, the above-mentioned method and implementation of video structured processing based on target behavior attributes are described Methods. Specifically, the above-mentioned device with a storage function may be one of a memory, a personal computer, a server, a network device, or a USB disk.

Please refer to FIG. 11. FIG. 11 is a schematic diagram of an embodiment of a video structured processing system based on target behavior attributes in the present invention. In this embodiment, the video processing system 400 includes: a memory 404 coupled with a processor 402, processing The processor 402 executes instructions during operation to implement the method described in the above-mentioned video processing method and implementation manner, and stores the processing results generated by executing the instructions in the memory 404 .

Optionally, step S23 tracks the target to obtain the tracking result and step S24 detects the abnormal behavior of the target, all based on the target detection and recognition of the single frame picture in step S22, the target can be tracked and Detect abnormal behavior of the target.

Optionally, the abnormal behavior detection of the target in step S24 can be performed directly after step S22 is completed, or can be performed simultaneously with step S23, or can be performed after step S23 and based on the tracking results of step S23.

Optionally, when step S24 detects the abnormal behavior of the target and tracks the target based on step S23 to obtain a tracking result, the detection of the abnormal behavior of the target will be more accurate.

Among them, the video structure processing method based on the target behavior attribute described in step S21 to step S24 can effectively reduce the pressure of network transmission of surveillance video, effectively improve the real-time performance of the surveillance system, and greatly reduce data Traffic fee.

Optionally, the step of performing target detection and recognition on the single-frame picture further includes extracting feature information of the target in the single-frame picture. It can be understood that after the read video is divided into multiple single-frame pictures, target detection and recognition should be performed on the divided single-frame pictures.

Optionally, the feature information of the target in the single-frame picture obtained by segmenting the video is extracted, where the target includes pedestrians, vehicles and animals, and the feature information of buildings or roads and bridges can also be extracted as required.

In an embodiment, when the target is a pedestrian, the extracted feature information includes: pedestrian position, pedestrian clothing color, pedestrian gender, motion state, motion trajectory, residence time and other characteristic information and other available information.

In another embodiment, when the target is a vehicle, the extracted feature information includes: the model of the vehicle, the color of the body, the speed of the vehicle, and the license plate number of the vehicle.

In yet another embodiment, when the target is a building, the extracted feature information includes: basic information of the building: such as building floor height, building height, building appearance color, and the like.

In yet another embodiment, when the target is a road bridge, the extracted feature information includes information such as the width of the road, the name of the road, and the speed limit of the road.

Optionally, the step of detecting the abnormal behavior of the target includes: extracting multi-pixel motion vectors of one or more targets, and performing abnormal behavior detection according to the relationship between the motion vectors.

In an embodiment, for specific details, refer to the above-mentioned method for abnormal behavior detection.

In one embodiment, the structured data initially set to be acquired in the video processing stage includes at least one information of the target's position, target category, target attribute, target motion state, target motion trajectory, and target dwell time. Among them, it can be adjusted according to the user's needs, and only the position information of the target is obtained in the video processing stage, or the position and the target category of the target are obtained at the same time. It can be understood that the information acquired in the video processing stage may be selected by the user as the type of information to be acquired in the video processing stage.

Optionally, after the video structured processing is completed, the obtained structured data is uploaded to the cloud server, and the cloud server will save the structured data uploaded by each monitoring node, and the structured data uploaded by each monitoring node Perform in-depth analysis to get preset results.

Optionally, the step of the cloud server performing an in-depth analysis of the structured data uploaded by each monitoring node may be set to be performed automatically by the system, or manually by the user.

In one embodiment, the basic analysis content included in the in-depth analysis of the cloud server is preset, such as counting the number of pedestrians, target trajectory analysis, whether the target has abnormal behavior, and the number of targets with abnormal behavior. Including other content that needs to be specially selected by the user, such as the proportion of each time period of the target, the speed of the target, and so on.

The above is only the embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process conversion made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technologies fields, all of which are equally included in the scope of patent protection of the present invention.

Claims (10)

1. A kind of 1. method of the video structural processing based on goal behavior attribute, it is characterised in that including：

   Target detection identification is carried out to single frames picture；

   To the target into line trace, to obtain tracking result；And/or

   Unusual checking is carried out to the target.
2. 2. the method for the video structural processing based on goal behavior attribute according to claim 1, it is characterised in that described The step of carrying out target detection identification to the single frames picture includes：

   Extract clarification of objective information described in the single frames picture.
3. 3. the method for the video structural processing based on goal behavior attribute according to claim 2, it is characterised in that described Further comprise before the step of extracting clarification of objective information described in the single frames picture：

   Build metadata structure；

   Wherein, the clarification of objective information is extracted according to metadata structure.
4. 4. the method for the video structural processing according to claim 1 based on goal behavior attribute, it is characterised in that institute State to the target into line trace, to obtain tracking result the step of further comprises：

   To the target into line trace, record the time of the into or out monitoring node of target, and target pass through it is each A position, to form the movement locus of the target.
5. 5. the method for the video structural processing based on goal behavior attribute according to claim 1, it is characterised in that described To the target into line trace, the step of to obtain tracking result after further comprise：Current monitor node will be left The structural data of the destination object is sent to cloud server.
6. 6. the method for the video structural processing based on goal behavior attribute according to claim 1, it is characterised in that described The step of carrying out unusual checking to the target includes：

   The light stream movable information of multiple characteristic points of the one or more targets of extraction, and according to the light stream movable information into Row cluster and unusual checking.
7. 7. the method for the video structural processing based on goal behavior attribute according to claim 1, it is characterised in that described Abnormal behaviour further comprises：Run, fight, at least one of riot or traffic abnormity.
8. 8. the method for the video structural processing based on goal behavior attribute according to claim 1, it is characterised in that described It is further comprising the steps of after the step of carrying out unusual checking to the target：If it is detected that the abnormal behaviour, then will Current video two field picture sectional drawing is preserved and sent to cloud server.
9. 9. a kind of video structural processing system based on goal behavior attribute, it is characterised in that including what is be electrically connected with each other Processor and memory, the processor couple the memory, the processor at work execute instruction to realize such as power Profit requires 1~8 any one of them method, and the handling result that the execute instruction is produced is stored in the memory.
10. 10. a kind of device with store function, it is characterised in that have program stored therein data, and described program data are performed Realize such as claim 1~8 any one of them method.