CN113033504B - A multi-scale video anomaly detection method - Google Patents

Abstract

The invention relates to a multi-scale video anomaly detection method, which comprises the following steps: step S1: acquiring video sample data, and carrying out multi-scale change on the video sample; step S2: constructing an anomaly detection model and carrying out model training; step S3: and testing the multi-scale anomaly detection model. According to the invention, the predicted frame and the real frame are blocked on different scales, so that the monitoring sensitivity of local abnormity is improved.

Description

A multi-scale video anomaly detection method

technical field

The invention belongs to the technical field of security, and in particular relates to a method for detecting abnormality based on multi-scale video.

Background technique

Video anomaly detection refers to the detection of abnormal behaviors that occur in videos. With the increasing popularity of surveillance video, automatic identification of anomalous events in the video becomes more and more necessary, since manual inspection can cause a lot of waste of resources (eg, labor). However, video anomaly detection is a challenging task due to the rarity and diversity of anomalous events. More specifically, unusual events are rare and may be events that have never been seen before. Therefore, it is quite difficult to collect all types of abnormal events, which makes traditional binary classification methods unsuitable. Furthermore, it is difficult to clearly define anomalies. Given that outliers are usually contextual, an anomalous event in one scenario can be treated as a normal event in another scenario. It is generally believed that abnormality refers to the behavior of training samples that do not meet expectations and deviate from the normal distribution. For example, in normal traffic scenarios, cycling on the sidewalk can be considered an abnormal behavior. Based on this fact, many methods for semi-supervised video anomaly detection have been proposed. Specifically, they generally assume that there is only normal data in the training set. Trying to learn the normal distribution. During the testing phase, events that deviate from the norm can be identified as anomalies. In fact, these semi-supervised methods are more in line with the actual situation of video anomaly detection. According to the temporal features of video sequences, they can be roughly divided into two categories: reconstruction model-based methods and predictive model-based methods. For methods based on reconstructed models, they usually feed normal frames into a deep neural network and try to reconstruct those frames with small errors. In addition, some studies also feed trajectory or skeleton features into neural networks for reconstruction. Based on this, in the testing phase, anomalies are expected to exhibit large reconstruction errors due to their deviation from normal visual patterns. However, due to the huge capacity and generality of deep neural networks, the reconstructed models can sometimes even reconstruct anomalous events well, resulting in undetectable anomalies. For predictive model-based methods, they typically feed successive frames into a neural network to try to predict future frames with smaller prediction errors for normal data. Considering that anomaly detection is about identifying events that do not meet expectations, it is natural to exploit the difference between predictions and expectations to detect anomalous events. Based on generative adversarial networks, the prediction of future frames enables greater real-world possibilities, boosting the performance of anomaly detection in videos. These schemes learn normal patterns and detect anomalies in a single space. For example, deep learning-based reconstruction and prediction methods typically compare original and generated frames pixel-by-pixel in the original image space to detect anomalies. We found that the anomaly often occurs in a certain area of the video frame. If you simply calculate a peak signal-to-noise ratio score between the generated frame and the real frame, it will lead to insensitivity to local anomalies and cannot perform well on local anomalies. detection. How to improve the sensitivity to local anomalies is a technical problem to be solved. The present invention improves the detection sensitivity of local anomalies by dividing the predicted frame and the real frame into blocks on different scales. Specifically: Multi-scale changes of samples are made through local changes of samples. While increasing the number of samples, local difference samples are enhanced. While improving the accuracy of local monitoring, the model training efficiency is greatly improved; To limit the error range, scientifically select the trained weight files, and perform line-by-line convolution by setting the window to reduce the computational load of the neural network and greatly speed up the training; reduce local prediction errors through multi-scale testing and eliminate significant local predictions error, which ultimately improves the regional prediction ability of the model.

SUMMARY OF THE INVENTION

In order to solve the above problems in the prior art, the present invention proposes a multi-scale video anomaly detection method, the method includes:

Step S1: acquiring video sample data, and performing multi-scale changes on the video samples; specifically: gridding the video samples, determining grid differences between the gridded video sample data, and tracking the network of the video sample data based on the grid differences. Grid change, make scale changes according to grid changes, select the target scale and make changes to the video samples based on the target scale;

Step S2: constructing an anomaly detection model, and performing model training; specifically: constructing an anomaly detection model, and using video sample data I ₁ to _It , It ₊₁ to perform model training;

Step S3: Test the multi-scale anomaly detection model; use the anomaly detection model to predict and obtain the predicted value I ^' _t+1 of the next frame, and calculate the error value between the predicted value of the next frame and the real value I _t+1 of the next frame ; Obtain the test score based on the error value, and determine whether there is an abnormality according to the test score;

Step S3 specifically includes the following steps:

Step S31: perform test initialization; initialize the test scale Scale to 1; wherein: the test scale is the level or number of scale divisions;

Step S32: Scale the video samples according to the test scale to obtain one or more scale-divided video sample sequences; input the video sample sequences into the anomaly detection model in turn, and obtain the corresponding one or more video sample sequences. Prediction output; calculate the error value of one or more prediction outputs, and obtain the score SC corresponding to the error value;

； 其中：ci为第ci个网格；VN^＇ _ci是第ci个网格的预测值，VN_ci是第ci个网格的真实值，d(VN^＇ _ci, VN_ci)是VN^＇ _ci和VN_ci之间的欧氏距离，Scale是测试尺度，d_max为d(VN^＇ _ci,VN_ci)中的最大值，d_min 为d(VN^＇ _ci,VN_ci)中的最小值；

; where: ci is the ci th grid; VN ^＇ _ci is the predicted value of the ci th grid, VN _ci is the actual value of the ci th grid, d(VN ^＇ _ci , VN _ci ) is the VN ^＇ _ci and Euclidean distance between VN _ci , Scale is the test scale, d _max is the maximum value in d(VN ^' _ci , VN _ci ), and d _min is the minimum value in d(VN ^' _ci , VN _ci );

Step S33: determine whether the division termination condition is satisfied, if so, go to step S34, otherwise, perform an incremental operation on the test scale, and go to step S32;

Step S34: Obtain a total score based on the score, if the total score exceeds the total score threshold, or if there is a score exceeding the score threshold, it is determined that the test video sample is abnormal.

Further, the obtaining of the score corresponding to the error value is specifically: searching for the score corresponding to the error value according to the comparison table of the error value and the score.

Further, the division termination condition is that the test scale is equal to the preset value and/or there is a score threshold that the score exceeds.

Further, the division termination condition is that the test scale is equal to the attention scale plus 1.

Further, the obtaining the total score based on the score is specifically: performing a weighted summation on the above scores to obtain the total score.

Further, step S3 further includes: the grids divided into grids are of the same size.

Further, the anomaly detection model is a neural network model.

The beneficial effects of the present invention include: (1) performing multi-scale changes of samples for local changes of samples, enhancing the local difference samples while increasing the number of samples, and greatly improving the model training efficiency while improving the accuracy of local monitoring; (2) Through the limitation of the number of tests, the number of trainings, and the error range, the weight file that has been trained is scientifically selected, and the line-by-line convolution by setting the window reduces the calculation amount of the neural network and greatly speeds up the training speed; (3) Pass Multi-scale testing reduces local prediction errors and eliminates significant local prediction errors, thereby ultimately improving the model's regional prediction ability.

Description of drawings

The accompanying drawings described here are used to provide a further understanding of the present invention and constitute a part of this application, but do not constitute an improper limitation of the present invention. In the accompanying drawings:

FIG. 1 is a schematic diagram of a multi-scale video anomaly detection method according to the present invention.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments, wherein the exemplary embodiments and descriptions are only used to explain the present invention, but are not intended to limit the present invention.

Video anomaly detection is a challenging task, and it is quite difficult to collect all types of anomalous events, which makes traditional binary classification methods unsuitable. Furthermore, it is difficult to clearly define anomalies. Given that outliers are usually contextual, an anomalous event in one scenario can be treated as a normal event in another scenario.

In the prior art, when performing abnormality detection, the abnormality type is directly found in the binary method through the training of the abnormality type. During the training process of the model, the normal frames are input into the deep neural network, and an attempt is made to reconstruct these frames with a small error. , which also involves feeding the trajectory or skeleton features into a neural network for reconstruction. However, it is obviously impossible to enumerate all anomaly types with this dichotomous prediction method; or to realize anomaly detection directly through normal frame training, but such anomaly detection will not be able to detect anomalies because anomalies often occur in a relatively local range. The invention improves the efficiency of abnormal detection through multi-scale detection;

A method for detecting anomalies based on multi-scale video according to the present invention includes the following steps:

Step S1: acquiring video sample data, and performing multi-scale changes on the video samples; specifically: gridding the video samples, determining grid differences between the gridded video sample data, and tracking the network of the video sample data based on the grid differences. Grid change, make scale changes according to grid changes, select the target scale and make changes to the video samples based on the target scale;

Step S1 specifically includes the following steps:

Step S11: gridding the video sample data when the gridding scale is smaller than the scale threshold, specifically: gridding the video sample data according to the gridding scale;

Preferably: the initial value of the gridding scale is 1; each time the step S11 is entered, the scale value of the gridding scale is increased by 1 or one unit; the increase method can also be increased by the power of 2, For example: J=2 ^JS-1 , where JS is the minimum number of divisions corresponding to the scale value, and the scale value corresponds to the number of grids; when the scale is 2, the video frame can be directly divided into upper and lower parts, or two left and right parts. part;

Preferred: the grids divided into the same size after gridding;

Step S12: Determine the grid difference between the gridded video samples, specifically: for the video samples I ₁ to It , two adjacent video frames I _i and I _{i +} 1 in It ₊₁ , calculate The difference value CRA _i,j of the corresponding grids VN _i,j and VN _i+1,j of adjacent video frames, 1=<j<=J, 1=<i<=t, get the difference value matrix [CRA _{i ,j} ]; CRA _i,j is the jth grid, the difference value between the i+1th and the ith video frame; where: J is the grid scale value;

Step S13: if there is a grid that falls within the preset difference value range in the difference value matrix, the current gridding scale is used as the target scale, and the grid that falls within the preset range is used as the target grid, and step S16 is entered; Otherwise, go to step S14;

Step S14: Calculate the ridge change entropy SH according to formula (1). If the ridge change entropy is less than the preset value, go to step S11; otherwise, go to step S15;

（1）；

(1);

Where: im, jm are the grid and video frame number where the maximum value of CRA _{i, j} is located;

，将网格差异和按照大小分类， 得到的每个分类和其他分类之间的差值都大于预设阈值，如果得到的分类个数大于2，则将 网格差异和最大的分类中的所有网格作为目标网格，将当前网格化尺度作为目标尺度，并 进入步骤S16，否则，进入步骤S11； Step S15: Calculate the grid difference sum

, classify the grid difference sum according to the size, the difference between each classification and other classifications obtained is greater than the preset threshold, if the number of obtained classifications is greater than 2, the grid difference and the largest classification The grid is used as the target grid, and the current gridding scale is used as the target scale, and then go to step S16, otherwise, go to step S11;

Through grid difference and grid distinction, there are obvious differences between each classification obtained, and the fixed part and the strongly changed part are distinguished by the difference classification; Time is used as the weight, while considering the difference, the change trend is considered, and the expansion basis of the sample is selected;

Step S16: making changes to the video samples based on the target scale and the target grid; specifically: generating a video sample that is a video sample, changing the video samples RI ₁ to RI _t , RI _t+1 , where the target grid in RI _i In order to copy the corresponding target grid data in I _i , the non-target grid data in RI _i copy the corresponding non-target grid data in I _i-1 ; on the one hand, the number of samples is increased through the sample change, and the local area in the same video frame is increased simultaneously. The range of change is convenient for subsequent rapid local training;

Alternatively: the target grid data in RI _i is calculated based on the corresponding non-target grid data in t+1 video frames of RI ₁ to RI _t , RI _t+1 ; for example, the pixel value is averaged;

Alternatively, the non-target grid data in RI _i is calculated based on the corresponding non-target grid data in t+1 video frames of RI ₁ to RI _t , RI _t+1 ; for example, pixel values are averaged;

Preferred: determine the number of generated changing video samples based on the target scale, when the difference between the target scale and the attention size is small, for example: the size comparison is determined within the threshold range; generate more video samples, and vice versa; pay attention to the size and abnormality The content targeted by the detection is related, for example: anomaly detection is about the target person, then the size of attention is related to the size of the person in the video sample; in this way, the scale change monitoring efficiency is the highest;

In the prior art, the video content and its changes are not considered when performing sample expansion, let alone local changes, because the segmentation of images is involved in local changes. Obviously, this is not worth the loss for sample expansion. Grid splitting finds a reasonable range of granularity, and makes multi-scale changes of samples to local changes of samples. While increasing the number of samples, it ensures that samples enhance local difference samples, thereby greatly improving the model while improving the accuracy of local monitoring. training efficiency;

Step S2: constructing an anomaly detection model, and performing model training; specifically: constructing an anomaly detection model, and using video sample data I ₁ to _It , It ₊₁ to perform model training;

Preferably: the anomaly detection model is a neural network model;

The building anomaly detection model is to construct an anomaly detection model based on a neural network model, which specifically includes the following steps:

Step SA1: perform input mapping on the video sample data through the mapping layer of the convolutional neural network, and generate an input matrix based on the video sample frame, wherein each sample frame corresponds to an input matrix;

Step SA2: Set a convolution kernel with a window length of W, and perform continuous convolution operations on W input matrices;

Preferably: perform a row-by-row convolution operation on W input matrices, specifically: take a row of elements in the input matrix as an input vector, and perform a convolution operation on the input vector; Ker _k =f(W * V _{k:k+ w} +b); where Ker _k is the convolution kernel of the kth convolution, f() represents the activation function, b is the convolution parameter, W is the window width, and _Vk:k+w is the input vector from the kth Start to the k+w-1th input vector;

Preferred: W≤t;

Step SA3: Perform a pooling operation on the convolution result;

Preferably: the convolution is mean convolution;

Alternative: the neural network model is a U-net model; the present invention has the problems of sparse feature vector and high dimension for video samples at the same time, and the large amount of computations that will be brought about by the neural network will lead to a crash, the present invention is performed by setting a window. The line-by-line convolution reduces the computational complexity of the neural network and improves the efficiency of anomaly detection;

The building anomaly detection model further includes a weight file selection step, specifically:

Step SB1: select a weight file as the current weight file of the model from the weight file set, carry out the training of the model; The training number of times is the first given number of times;

When all the weight files in the weight file set do not meet the requirements, the weight file with the smallest error value in step SB4 is selected from the weight file set as the selected weight file;

Preferably: the weight file is a trained weight file;

Step SB2: use the abnormality detection model to perform the first test and obtain the error value; if the error value is within the first reasonable range, then proceed to step SB3, otherwise, proceed to step SB1;

Preferably: the error value is the average error value of the first number of tests;

Preferably: the first number of tests is 0.1*the first given number of times;

Step SB3: Use the current weight file to continue the training of the model, and the number of training times is the second given number of times;

Step SB4: adopt the model to carry out the second test and obtain the error value; if the error value is in the second reasonable range, the current weight file is used as the selected weight file; otherwise, enter step SB1;

Preferably: the first given number of times NG1 is less than or equal to the second given number of times NG2;

Preferred: As the training times of the model continue to increase, the reasonable range is correspondingly narrowed; specifically:

Specifically: the first reasonable range number D1=[DD1,DU1] satisfies the following relationship:

；

;

The first reasonable range number D2=[DD2,DU2] satisfies the following relationship:

；

;

Preferably: the second number of tests is equal to the first number of tests; although the weight files of other models can be used to increase the training speed, how to select them is rarely involved in the prior art. The present invention passes the number of tests, the number of trainings, The error range is limited, and the trained weight file is scientifically selected, which greatly speeds up the training speed;

Step S3: Test the multi-scale anomaly detection model; use the anomaly detection model to predict and obtain the predicted value I ^' _t+1 of the next frame, and calculate the error value between the predicted value of the next frame and the actual value I _t+1 of the next frame ; Obtain the test score based on the error value, and determine whether there is an abnormality according to the test score;

Preferably: the error value is the peak signal-to-noise ratio; when the error value is large, the test score is high, and when the test score is greater than the abnormal score, it is determined that there is abnormality;

The error is specifically: the intensity loss and/or gradient loss before the predicted value of the next frame and the actual value of the next frame are obtained as the error;

Step S3 specifically includes the following steps:

Step S31: perform test initialization; initialize the test scale Scale to 1; wherein: the test scale is the level or number of scale divisions;

Step S32: Scale the video samples according to the test scale to obtain one or more scale-divided video sample sequences; input the video sample sequences into the anomaly detection model in turn, and obtain the corresponding one or more video sample sequences. Prediction output; calculate the error value of one or more prediction outputs, and obtain the score SC corresponding to the error value;

；

;

Where: ci is the ci-th grid; VN'ci is the predicted value of the ci-th grid, VNci is the real value of the ci-th grid, and d(VN'ci, VNci) is the difference between VN'ci and VNci The Euclidean distance, Scale is the test scale;

The obtaining of the score corresponding to the error value is specifically: searching for the score corresponding to the error value according to the comparison table of the error value and the score;

Step S33: determine whether the division termination condition is satisfied, if so, go to step S34, otherwise, perform an incremental operation on the test scale, and go to step S32;

Preferably: the division termination condition is that the test scale is equal to the preset value and/or there is a score threshold that the score exceeds;

Alternatively: the division termination condition is that the test scale is equal to the attention scale+1;

Step S34: Obtain the total score based on the score. If the total score exceeds the total score threshold, or there is a score exceeding the score threshold, it is determined that the test video sample is abnormal; the present invention proposes to reduce local prediction errors through multi-scale testing and eliminate significant local. Predict errors, thereby ultimately improving the regional prediction ability of the model to achieve the ultimate goal of anomaly detection efficiency;

The obtaining of the total score based on the score is specifically: weighted summation of the above scores to obtain the total score; SCALL=∑SC;

Software environments can be divided into two categories, including system software and application software executing on one or more hardware environments. In one embodiment, the methods and processes disclosed herein may be implemented as system software, application software, or a combination thereof. System software may include control programs, such as an operating system (OS) and information management system, that instruct one or more processors (eg, microprocessors) in a hardware environment how to operate and process information. Application software may include, but is not limited to, program code, data structures, firmware, resident software, microcode, or any other form of information or routines that can be read, analyzed, or executed by a processor.

In other words, application software may be implemented as program code, in the form of a machine-usable or computer-readable storage medium, embedded in a computer program product that provides the program code for use by or in connection with a machine, computer, or any instruction execution system . Furthermore, the application software may include one or more computer programs that execute on top of the system software after being loaded from a storage medium into local memory. In a client-server architecture, application software may include client software and server software. For example, in one embodiment, client software may execute on a client computing system that is distinct and separate from the server computing system that executes the server software.

The software environment may also include browser software to access data provided over a local or remote computing network. Further, the software environment may include a user interface (eg, a graphical user interface (GUI)) for receiving user commands and data. It is necessary to reiterate that the hardware and software architectures and environments described above are for example purposes. Accordingly, one or more embodiments may be implemented on any type of system architecture, functional or logical platform or processing environment.

The above descriptions are only the preferred embodiments of the present invention, so all equivalent changes or modifications made according to the structures, features and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.

Claims (6)

1. A multi-scale video anomaly detection method, the method comprising:

step S1: acquiring video sample data, and carrying out multi-scale change on the video sample; specifically, the method comprises the following steps: gridding the video samples, determining grid difference among gridded video sample data, tracking the grid change condition of the video sample data based on the grid difference, carrying out scale change according to the change condition of the grid, selecting a target scale and changing the video samples based on the target scale;

step S2: constructing an anomaly detection model and carrying out model training; specifically, the method comprises the following steps: constructing an abnormal detection model by adopting video sample data I₁～I_t，I_t+1Carrying out model training;

the method for constructing the anomaly detection model comprises the following steps of:

step SA 1: performing input mapping on video sample data through a mapping layer of a convolutional neural network, and generating an input matrix based on video sample frames, wherein each sample frame corresponds to one input matrix;

step SA 2: setting a convolution kernel with the window length of W, and performing continuous convolution operation on W input matrixes;

performing a line-by-line convolution operation on the W input matrixes, specifically: taking a row of elements in the input matrix as an input vector, and performing convolution operation on the input vector; ker_k=f(W * V_k:k+w+ b); wherein Ker_kFor the convolution kernel of the kth convolution, f () represents the activation function, b is the convolution parameter, W is the window width, V_k:k+wStarting from the kth input vector to the (k + w-1) th input vector;

step SA 3: performing pooling operation on the convolution result;

the method for constructing the anomaly detection model further comprises a weight file selection step, which specifically comprises the following steps:

SB1, selecting a weight file from the weight file set as the current weight file of the model, and training the model; the training times are first given times;

step SB 2: performing a first test by adopting an anomaly detection model and obtaining an error value; if the error value is within the first reasonable range, go to step SB3, otherwise, go to step SB 1;

step SB 3: continuing training the model by using the current weight file, wherein the training times are second given times;

step SB4, performing a second test by using the model and obtaining an error value; if the error value is in the second reasonable range, taking the current weight file as the selected weight file; otherwise, go to step SB 1;

step S3: testing a multi-scale anomaly detection model; predicting by using an anomaly detection model to obtain a predicted value I of a next frame^＇ _t+1Calculating the predicted value and the real value of the next frame_t+1An error value therebetween; obtaining a test score based on the error value, and determining whether an abnormality exists according to the test score;

step S3 specifically includes the following steps:

step S31: carrying out test initialization; initializing a test Scale to be 1; wherein: the testing scale is the level or the times of scale division;

step S32: carrying out scale division on the video samples according to the test scale to obtain one or more video sample sequences subjected to scale division; respectively and sequentially inputting the video sample sequences into an anomaly detection model to obtain prediction outputs corresponding to the one or more video sample sequences; calculating one or more error values of the prediction output, and obtaining scores SC corresponding to the error values;

；

wherein: ci is the ci-th grid; VN^＇ _ciIs the predicted value of the ci-th grid, VN_ciIs the true value of the ci-th mesh，d(VN^＇ _ci,VN_ci) Is VN^＇ _ciAnd VN_ciEuclidean distance between them, Scale is the test Scale, d_maxIs d (VN)^＇ _ci,VN_ci) Maximum value of (1), d_minIs d (VN)^＇ _ci,VN_ci) Minimum value of (1);

step S33: judging whether the dividing termination condition is met, if so, entering a step S34, otherwise, performing incremental operation on the test scale, and entering a step S32;

step S34: and obtaining a total score based on the scores, and determining that the test video sample has abnormality if the total score exceeds a total score threshold value or F is a score threshold value exceeding the F.

2. The method for detecting anomaly based on multi-scale video according to claim 1, wherein the obtaining of the score corresponding to the error value specifically comprises: and searching a score corresponding to the error value according to the comparison table of the error value and the score.

3. The multi-scale video based anomaly detection method according to claim 2, wherein the partition termination condition is that the test scale equals a preset value and/or that there is a score threshold value over which a score exceeds.

4. The method for detecting abnormality based on multi-scale video according to claim 3, wherein the total score is obtained based on the score, and specifically comprises: and weighting and summing the scores to obtain a total score.

5. The method for detecting anomaly based on multi-scale video according to claim 4, wherein the step S3 further comprises: the grids divided after gridding are the same in size.

6. The multi-scale video based anomaly detection method according to claim 5, wherein the anomaly detection model is a neural network model.

Images