KR20240115149A - Apparatus and method for integrated anomaly detection - Google Patents

Abstract

An integrated abnormal situation detection method according to an embodiment of the present invention includes the steps of detecting an object object and a human object from an input image using a first neural network, tracking the human object, and the object detection result and the human object tracking result. It includes a step of detecting an abnormal situation based on.

Description

Integrated abnormal situation detection method and device {APPARATUS AND METHOD FOR INTEGRATED ANOMALY DETECTION}

The present invention relates to a single abnormal situation detection technology that can comprehensively detect multiple abnormal situations.

Specifically, the present invention relates to object detection and tracking technology to comprehensively detect multiple abnormal situations.

Recently, CCTV (Closed-Circuit Television) is being installed not only in security facilities but also throughout the city as a way to reduce damage to life and property due to crimes or accidents. However, as a small number of control personnel monitor the rapidly increasing number of CCTVs, problems with control efficiency have arisen. In addition to securing control personnel, the need for an abnormal situation detection system for automated intelligent video surveillance is gradually increasing. Abnormal situation detection systems can significantly increase control efficiency by detecting accidents or incidents in advance and generating alarm events or quickly searching for situations where an incident has occurred even after the incident has occurred.

Therefore, in order to ensure smooth crisis response capabilities and golden time for accidents, most control centers are trying to introduce abnormal situation detection systems based on artificial intelligence technology. However, current systems have very limited use in actual environments due to various problems. In order to build and secure an advanced abnormal situation detection system applicable to the field, solving the following problems is essential.

First, a unified, integrated framework that can complexly detect multiple abnormal situations is needed. Intelligent CCTV abnormality detection technologies for building a social safety net are being used to detect various situations that may occur in actual sites, such as intrusion, loitering, abandonment, arson, collapse, and fighting. However, most prior technologies only propose an optimized detection method for a single abnormal situation, and do not present an integrated framework that can complexly detect and process different abnormal situations. In order to simultaneously respond to the occurrence of multiple abnormal situations, a unified integrated framework is needed that can secure real-time and reliability of the system by organically combining detection modules with different characteristics.

Additionally, securing stable human detection and tracking technology must take precedence. Human detection and tracking technology is necessary to detect abnormal situations for each of the multiple people appearing in the video. Technologies for detecting and tracking multiple objects in video environments have made rapid progress through publicly competitive challenges. However, in the CCTV environment, it is not easy to ensure the stability of human detection due to frequent changes in lighting, environment, and weather and the diversity of distances and angles between cameras and humans. If human detection and tracking technology fails to properly detect humans or incorrectly detects other objects as humans, the performance of the entire control system is bound to decline sharply. Therefore, technology to accurately detect and track humans in various indoor and outdoor CCTV environments is essential to ensure the reliability of the control system.

Lastly, the versatility to detect abnormal situations even in new, untrained CCTV environments must be secured. Most existing abnormal situation detection studies deal with learning and evaluation using data collected in advance. However, it is very unlikely that a large amount of learning data collected in advance will exist in the target domain where the system will be installed, and collecting it is also very difficult. Therefore, an abnormal situation detection technology that can operate robustly despite changes in various CCTV environments without additional data collection is needed.

Domestic Registered Patent Publication No. 2344606 (Title of Invention: Tracking and Monitoring CCTV System and Tracking and Monitoring Method)

The purpose of the present invention is to provide an integrated abnormal situation detection structure that can simultaneously respond to the occurrence of multiple abnormal situations.

Additionally, the purpose of the present invention is to provide a filtering technique that can reduce false positives and tracking failures of human objects.

Additionally, the purpose of the present invention is to provide a method for detecting abnormal situations that operates robustly in various environments without collecting additional data.

An integrated abnormal situation detection method according to an embodiment of the present invention for achieving the above object includes detecting object objects and human objects from an input image using a first neural network; tracking the human object; and detecting an abnormal situation based on the object detection result and the human object tracking result.

At this time, the step of tracking the human object includes first feature information generated based on an intermediate calculation result of the first neural network; And it may be performed using the human object area corresponding to the final calculation result of the first neural network as input and second feature information extracted through the second neural network.

At this time, the intermediate operation result includes spatial information and texture information of the input image, and the first feature information may be extracted by masking the area of the detected human object with respect to the intermediate operation result.

At this time, the step of tracking the human object may match the same human object between frames using the first characteristic information and the second characteristic information.

At this time, the first feature information may correspond to an M-dimensional feature vector, and the second feature information may correspond to an N-dimensional feature vector.

At this time, in the step of tracking the human object, the human object may be tracked using a (M+N) dimensional feature vector generated based on the first feature information and the second feature information.

At this time, the tracking result of the human object may include information on the occupied area and movement trajectory of the human object.

At this time, the step of tracking the human object may identify a work object that is incorrectly detected as a human object based on movement trajectory information of the human object.

At this time, in the step of tracking the human object, a motion vector corresponding to the movement trajectory of the human object may be calculated, and an object misdetected as a human object may be identified using the result of an external operation between motion vectors for each section.

At this time, the step of detecting the abnormal situation includes visual feature information corresponding to the input image; And the arson situation can be detected using language feature information corresponding to the arson situation text.

At this time, the step of detecting the abnormal situation may detect an arson situation by mapping the visual feature information and the language feature information to the same comparison space and calculating the similarity between the visual feature information and the language feature information.

At this time, the visual characteristic information may be generated based on the input image, the human object area image, and the area image in which the human object is determined to have stayed for more than a preset time.

At this time, the step of detecting the abnormal situation detects human object behavior, sets main behavior for each section based on the frequency of the human object behavior, and calculates the section in which the abnormal situation occurs using the main behavior information for each section. can do.

At this time, the abnormal situation may include intrusion, loitering, arson, abandonment, fighting, and collapse.

In addition, an integrated abnormal situation detection device according to an embodiment of the present invention for achieving the above object includes an object detection unit that detects object objects and human objects from an input image using a first neural network; a human object tracking unit that tracks the human object; and an abnormal situation detection unit that detects an abnormal situation based on the object detection result and the human object tracking result.

At this time, the human object tracking unit includes first feature information generated based on an intermediate calculation result of the first neural network; And the human object area corresponding to the final calculation result of the first neural network is input, and the human object can be tracked using second feature information extracted through the second neural network.

At this time, the intermediate operation result includes spatial information and texture information of the input image, and the first feature information may be extracted by masking the area of the detected human object with respect to the intermediate operation result.

At this time, the human object tracker may match the same human object between frames using the first characteristic information and the second characteristic information.

At this time, the first feature information may correspond to an M-dimensional feature vector, and the second feature information may correspond to an N-dimensional feature vector.

At this time, the human object tracking unit may track the human object using an (M+N)-dimensional feature vector generated based on the first feature information and the second feature information.

At this time, the tracking result of the human object may include information on the occupied area and movement trajectory of the human object.

At this time, the human object tracking unit may identify the object that is incorrectly detected as a human object based on the movement trajectory information of the human object.

At this time, the human object tracking unit may calculate a motion vector corresponding to the movement trajectory of the human object, and use the result of the outer product between motion vectors for each section to identify the object that was incorrectly detected as a human object.

At this time, the abnormal situation detection unit includes visual feature information corresponding to the input image; And the arson situation can be detected using language feature information corresponding to the arson situation text.

According to the present invention, it is possible to provide an integrated abnormal situation detection structure that can simultaneously respond to the occurrence of multiple abnormal situations.

Additionally, the present invention can provide a filtering technique that can reduce false positives and tracking failures of human objects.

Additionally, the present invention can provide a method for detecting abnormal situations that operates robustly in various environments without collecting additional data.

Figure 1 is a flowchart showing an integrated abnormal situation detection method according to an embodiment of the present invention.
Figure 2 is a block diagram showing an abnormal situation detection framework according to an embodiment of the present invention.
Figure 3 is a block diagram showing a human tracking module according to an embodiment of the present invention.
Figure 4 is an example of a movement trajectory detected from a stationary object.
Figure 5 is a block diagram showing in detail a fire prevention recognition module according to an embodiment of the present invention.
Figure 6 is a flowchart showing the abnormal behavior section detection process.
Figure 7 is a block diagram showing an integrated abnormal situation detection device according to an embodiment of the present invention.
Figure 8 is a diagram showing the configuration of a computer system according to an embodiment.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. The present embodiments only serve to ensure that the disclosure of the present invention is complete and that common knowledge in the technical field to which the present invention pertains is not limited. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Although terms such as “first” or “second” are used to describe various components, these components are not limited by the above terms. The above terms may be used only to distinguish one component from another component. Accordingly, the first component mentioned below may also be the second component within the technical spirit of the present invention.

The terms used in this specification are for describing embodiments and are not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” or “comprising” implies that the mentioned component or step does not exclude the presence or addition of one or more other components or steps.

As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof.

Unless otherwise defined, all terms used in this specification can be interpreted as meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components will be assigned the same reference numerals and redundant description thereof will be omitted. .

Figure 1 is a flowchart showing an integrated abnormal situation detection method according to an embodiment of the present invention.

The integrated abnormal situation detection method according to an embodiment of the present invention may be performed in an abnormal situation detection device such as a computing device.

Referring to FIG. 1, the integrated abnormal situation detection method according to an embodiment of the present invention includes detecting object objects and human objects from an input image using a first neural network (S110), and tracking the human object (S110). S120) and detecting an abnormal situation based on the object detection result and the human object tracking result (S130).

At this time, the step of tracking the human object (S120) uses the first feature information corresponding to the intermediate calculation result of the first neural network and the human object area corresponding to the final calculation result of the first neural network as input, and the second neural network It can be performed using the second feature information extracted through .

At this time, the first feature information may represent spatial information and texture information of the input image, and the second feature information may represent appearance information of the human object.

At this time, the step of tracking the human object (S120) may match the same human object between frames using the first characteristic information and the second characteristic information.

At this time, the first feature information may correspond to an M-dimensional feature vector, and the second feature information may correspond to an N-dimensional feature vector. At this time, M and N may each correspond to any natural number.

At this time, in the step of tracking the human object, the human object may be tracked using a (M+N) dimensional feature vector generated based on the first feature information and the second feature information.

At this time, the tracking result of the human object may include information on the occupied area and movement trajectory of the human object.

At this time, in the step of tracking the human object (S120), the object that is incorrectly detected as a human object may be identified based on the movement trajectory information of the human object.

At this time, in the step of tracking the human object (S120), a motion vector corresponding to the movement trajectory of the human object is calculated, and the object that is incorrectly detected as a human object can be identified using the result of the outer product between the motion vectors for each section. there is.

At this time, in the step of detecting the abnormal situation (S130), an arson situation may be detected using visual feature information corresponding to the input image and language feature information corresponding to text generated based on the input image.

At this time, in the step of detecting the abnormal situation (S130), an arson situation can be detected by mapping the visual feature information and the language feature information to the same comparison space and calculating the similarity between the visual feature information and the language feature information. there is.

At this time, the visual characteristic information may be generated based on the input image, the human object area image, and the area image in which the human object is determined to have stayed for more than a preset time.

At this time, the abnormal situation detection step (S130) detects human object behavior, sets main behavior for each section based on the frequency of the human object behavior, and uses the main behavior information for each section to generate an abnormal situation. The section can be calculated.

At this time, the abnormal situation may include intrusion, loitering, arson, abandonment, fighting, and collapse.

Figure 2 is a block diagram showing an abnormal situation detection framework according to an embodiment of the present invention.

Referring to FIG. 2, the integrated abnormal situation detection framework according to an embodiment of the present invention includes a human detection and tracking unit 200, an abnormal situation detection unit focusing on human movement 300, and an abnormal situation detection unit focusing on a target object 400. ), a human behavior attention abnormal situation detection unit 500, and an abnormal situation detection integrated management unit 600.

At this time, the human detection and tracking unit 200 can receive the video stream 100 obtained from CCTV, track the human, and generate the human's area and trajectory within the video.

The abnormal situation detection unit 300, which pays attention to the human movement, receives the human's area and trajectory, analyzes the trajectory, and detects how long the human stayed in the border area to detect abnormal situations related to the human's movement, such as intrusion and wandering. .

The abnormal situation detection unit 400 that focuses on the target object can receive human detection and trajectory information and object detection information to track baggage and recognize arson scenes to detect abandonment and arson, which are abnormal situations related to objects.

The abnormal situation detection unit 500 that pays attention to human behavior can receive human detection and trajectory information, recognize human behavior, detect the section in which the behavior occurred, and detect abnormal situations related to human behavior such as fighting and collapse.

The abnormal situation detection integrated management unit 600 integrates and manages various types of abnormal situations with different characteristics detected by each abnormal situation detection unit 300, 400, and 500, and finally generates an event (700) corresponding to each abnormal situation. ) can occur.

Hereinafter, each module that constitutes the abnormal situation detection framework will be described in detail.

The human detection and tracking unit 200 may detect objects including humans and perform human tracking to collect location information on humans appearing in the video. The human detection and tracking unit 200 may include an object detection module 210 and a human tracking module 220.

The object detection module 210 can estimate the classification label, location information, and probability of an object existing in an image. In one embodiment of the present invention, the object detection model may utilize known models such as RCNN and Yolo, but the scope of the present invention is not limited thereto. The object detection module 210 may obtain location information only for objects classified as humans and transmit it to the human tracking module 220.

The human tracking module 220 can determine whether the human appearing in each frame is the same person, assign the same tracking ID to the same person, and generate a trajectory for the human location. In one embodiment of the present invention, the human tracking model may utilize known models such as Deep SORT and OSNet, but the scope of the present invention is not limited thereto. The human tracking module 220 may transmit the generated human trajectory information to the human trajectory analysis module 310.

There are many large-scale image datasets released for object detection. Object images reflecting various environmental conditions such as lighting, object size, camera angle, and background changes are all included in the large-scale dataset. Therefore, object detection modules learned from large-scale datasets can generally operate stably even in various environmental changes.

However, object tracking datasets, which must be composed of a series of image streams (videos), are overwhelmingly fewer in number than object detection datasets, and most of the object tracking datasets are data that captures the entire body of a human in a CCTV environment. There are three.

Therefore, the conventional human tracking module has the disadvantage of being very vulnerable to untrained environments, which can be a direct factor that reduces the reliability of the entire abnormal situation detection integrated framework.

Specifically, the performance of the human tracking module, which assigns identity using similarity, relies on a feature extractor that expresses features such as the shape, color, and bag of clothes worn by the person in the image as an identity vector to perform tracking. The appearance information that exists in the data distribution learned by the feature extractor can be effective in tracking the same person because it compares features in a high-dimensional space through an identity vector. However, in a distribution environment other than learning, where appearance information that does not exist in the learned dataset is extracted as an identity vector, stable mapping of the identity vector may be difficult.

For example, if the feature extractor does not learn enough information about the appearance of people on a rainy day, there is a possibility that the same person will be recognized as different people when wearing an umbrella and when carrying an umbrella. In other words, the human images sampled from the learning dataset for human tracking are collected in a limited environment and cannot provide various appearance information suitable for the usage environment. Accordingly, the feature extractor performs stable mapping of identity vectors in various CCTV operation situations. There are difficulties.

The importance of the abnormal situation detection system for intelligent video surveillance can be said to be greater at night and under adverse weather conditions, when control becomes difficult. Therefore, securing stable human tracking performance is essential in an abnormal situation detection system.

The abnormal situation detection method according to an embodiment of the present invention provides a method for improving human tracking performance by utilizing feature information extracted from an object detection module for human tracking that is robust to various environmental conditions and adverse weather conditions.

Figure 3 is a block diagram showing a human tracking module according to an embodiment of the present invention.

Referring to FIG. 3 , the human detection and tracking unit 200 may include an object detection module 210 and a human tracking module 220.

The human detection and tracking unit 200 first detects the location of the human in the received video stream 100 through the object detection module 210 and tracks it through a matching process by comparing it with the previously appearing human. The identity recognition process of the human tracking module 220 may consist of a human feature extraction process, a human tracking process, and a trajectory stabilization process. The feature extractor can extract an N-dimensional identity vector using the human area detected in the current frame as input to distinguish individual humans.

The tracker can calculate the similarity between the identity vectors stored in the human information being tracked and the identity vectors extracted from the current frame and assign them to the closest tracking ID. Afterwards, the tracked human movement trajectory information can be predicted and corrected through a stabilization filter.

The human tracking technique according to an embodiment of the present invention is a dimensionality expansion technique for stably obtaining an identity vector that is highly dependent on the environment in various environments.

The deep learning-based object detection module can estimate the type, location information, and probability of existence of objects present in the image. As mentioned above, the object detection module can extract the edge shape and internal texture information of various objects by utilizing large-capacity datasets, and can learn the front-back relationship and interrelated information of objects to accurately infer location information.

Mid-level features acquired during the intermediate calculation process of the object detection module may include spatial information and texture information extracted from the entire image to classify objects and identify their locations.

In the tracking process performed after identifying the human's location, information about individual humans must be selected rather than information extracted from the entire image. To detect the location of a human, location information is preserved, and information about individual humans is obtained by masking the feature information related to object detection obtained from the image with each human region as a region of interest (ROI). Information masked as individual human areas is converted into an M-dimensional object feature vector through an average pooling process. By expanding the N-dimensional identity vector expressed by the feature extractor of the human tracking module with the object feature vector extracted from the object detection module, a (M+N)-dimensional vector can be formed and used for human tracking.

The dimension expansion technique of the identity vector according to an embodiment of the present invention is a technique that uses the feature information of the person's appearance learned separately for tracking by crossing the information expression space of the object detection module and the expression space of the feature extractor. Through dimension expansion, it is possible not only to simultaneously query appearance information extracted from different modules, but also to distinguish and identify different humans that were previously impossible to distinguish due to limitations in the dimensions that can be expressed.

Additionally, because the information obtained from the object detection module is reused, the performance of human tracking can be improved efficiently at low cost.

In addition, since the amount and learning aspect of the dataset used by the object detection module and the human tracking module are different, appearance features can be extracted in a complementary manner and can be applied to various human tracking methods.

The abnormal situation detection unit 300 that pays attention to the human movement can receive the human area and trajectory output by the human detection and tracking unit 200 and analyze the trajectory. In addition, by detecting how long a human has stayed in the border area, it is possible to detect intrusion and wandering, which are abnormal situations related to the human's movements.

The human trajectory analysis module 310 can analyze trajectory information, which is continuous location information of the human being tracked, to verify whether the object tracked as a human is an actual human or an object incorrectly detected as a human.

Multiple object detection datasets used for learning of the object detection module 210 widely use mean Average Precision (mAP) as a performance evaluation standard. The mAP evaluation method, which calculates average precision (AP) individually for each object to be detected and then averages it again to evaluate performance for multiple objects, focuses on the performance of accurately detecting the location rather than the performance of accurately recognizing and classifying objects. has

Specifically, the mAP method extracts only the probability corresponding to the object to be evaluated, calculates each accuracy, and then performs the evaluation by taking the average. For example, even if the object detection module incorrectly predicts that there is a higher probability that another object exists in the location where the chair is, this does not affect the chair detection performance. Even if it is determined that there is a dog with a higher probability than a human in the human area of the actual image, the object classification performance for humans does not decrease. Therefore, conventional high-performance object detection modules are designed with a focus on accurately estimating the location rather than accurately classifying the object.

In this way, object detection modules that do not accurately verify classification performance have the problem of frequently generating errors in classifying non-human objects as human. Since humans and objects are not distinguished in the human tracking process after object detection, objects that are incorrectly tracked as humans may continuously cause unnecessary abnormal situation detection calculations. Excluding incorrectly detected objects from the calculation process of the anomaly detection unit is essential for increasing computation speed and accurately detecting abnormal situations.

Due to the feature of being tracked on a frame-by-frame basis, positional noise is generated in the object detection module even when there is a slight change between images, and the noise is amplified in the prediction calculation of the human tracking process, causing the positional information of a stationary object to move randomly up, down, left, and right. Accordingly, the moving distance measured from the trajectory of a falsely detected object that is stopped in place is measured similarly to the moving distance of a human, making it difficult to identify.

Additionally, even in the case of displacement, which is the amount of change between the first appearing location and the current location, it is difficult to provide reliable human analysis results due to ID allocation failure. In order to reliably measure the movement of an object, it is essential to have a method to measure the movement quantitatively by removing noise from the tracked object trajectory.

The human trajectory analysis method according to an embodiment of the present invention can identify objects incorrectly detected as humans by effectively removing noise from the motion trajectory information of objects detected in an image. Area calculation techniques using the directionality of vectors are used to remove noise that randomly shakes up, down, left, and right and to accurately measure the degree of movement within the image. Equations 1 and 2 below represent a polygon area calculation method using coordinate changes and a polygon area calculation method using a vector cross product operation, respectively.

[Equation 1]

[Equation 2]

Equation 1 calculates the area by using the two-dimensional vertices (x, y) of the polygon and summing the decreasing area and increasing area on the y-axis with opposite signs. Equation 2 is a formula that changes the area of the polygon in Equation 1 so that the sign is determined by the correlation between vectors.

Equation 2 determines whether the rotation is clockwise or counterclockwise through the cross product of the two vectors, and calculates the area by setting the sign to the opposite side. When calculating the area of a polygon using Equation 2, noises that shake up, down, left, and right may not be added to the calculation of the normal area that has actually moved, but may show a mutually cancelling effect.

Figure 4 is an example of a movement trajectory detected from a stationary object.

Referring to FIG. 4, the movement trajectory detected from a stationary object contains noise and has the characteristic of moving randomly. Since vectors A ₀ A ₁ and A ₁ A ₂ have a counterclockwise relationship, the area of triangle A ₀ A ₁ A ₂ is calculated in the negative direction. On the other hand, because vectors A ₂ A ₃ and A ₃ A ₄ have a clockwise relationship, the area of triangle A ₂ A ₃ A ₄ is calculated in the positive direction. In this way, the movement area with noise removed can be calculated by considering the direction of the vector.

In addition, the method for calculating the area of a movement trajectory according to an embodiment of the present invention is capable of calculating the movement area by taking directionality into account even in the case of large moving noise caused by repetitive id-switches, which are a form of human tracking failure. Additionally, by utilizing vector operations, noise that occurs when moving in not only two-dimensional space but also three-dimensional space can be removed. If human movement can be measured quantitatively, incorrectly detected objects can be distinguished using the degree of movement. In addition, by dividing the movement trajectory into short sections and calculating the area, it is possible to identify the human stopped state and identify risk factors and areas.

The human trajectory analysis module 310 filters incorrectly detected results from the human detection and tracking unit, quantitatively measures the human's movement state, and transmits it to other abnormal situation detection units 400 and 500.

The border area intersection detection module 320 quickly and efficiently detects intrusion and loitering situations using only human location information. The degree of overlap between the pre-set boundary zone on each CCTV screen and the area corresponding to the human body is calculated to determine whether the human has entered. The timing of when the human enters and exits the no-trespassing zone is transmitted to the abnormal situation integrated management unit (600).

The abnormal situation detection unit 400 that focuses on the target object receives human detection and trajectory information and object detection information, tracks baggage, recognizes arson scenes, and detects abandonment and arson, which are abnormal situations related to objects.

The baggage tracking module 410 is a module that tracks baggage, identifies the owner in possession, and detects abandonment. From object detection information, it is possible to track handbags, backpacks, travel bags, etc. that correspond to human accessories. Tracked objects are assigned by identifying their owners, and when a human carrying luggage abandons the object and leaves the location, the result is transmitted to the abnormal situation detection integrated management unit 600.

Figure 5 is a block diagram showing in detail a fire prevention recognition module according to an embodiment of the present invention.

Referring to FIG. 5, the arson recognition module 420 is a module that detects a fire situation and identifies arson behavior through text describing the situation. The visual-language information conversion model of Figure 5 can utilize known models such as CLIP, Florence, and Flamingo that learn large-scale image and text pairs, but the scope of the present invention is not limited thereto.

Conventional technology uses a method of detecting the location of a fire through an object detection module that recognizes images containing fire or learns the area where the fire occurred as an object. These methods can operate stably if they learn from large-scale datasets, but the number of datasets taken in consideration of the various appearances and aspects of fire that occur depending on the environment and combustibles is very insufficient due to the high collection cost. Due to the lack of diversity in collected fire images, models learned using existing methods are inevitably very vulnerable to changes in the CCTV environment. Therefore, existing methods require additional data collection and learning processes at new installation locations.

In addition, most of the existing technologies for detecting the location of a fire were developed for the purpose of finding and responding to the location of a large fire, so they do not address arson situations where an arsonist starts a fire. In order to identify arson, in which a person starts a fire for the purpose of threatening a facility or place, recognizing the act of starting a fire is more important than detecting the location of the fire.

As mentioned above, conventional fire detection technology has the disadvantage of being vulnerable to various types of fire situations due to the absence of large-scale datasets, and arson acts that start a fire are impossible to recognize. Therefore, existing fire detection techniques cannot establish a video surveillance system to identify arsonists in a CCTV environment and detect fire situations in the early stages.

The arson situation detection method according to an embodiment of the present invention is an image-text comparison inference technique to solve the data shortage problem and operate stably in an untrained environment.

To enable image-text comparison inference, we utilize a model pre-trained with a large dataset consisting of text describing images. The pre-trained model is trained using hundreds of millions of images to find similar mapping points in the same space for text that best describes the image among hundreds of millions of image descriptions. The pre-trained visual information converter and language information converter can convert images and text into the same space, the image-text comparison space (Contrastive Embedding Space).

Visual-language intersection vectors mapped into comparable dimensional spaces can be used to retrieve textual information describing an image, or conversely, related images can be retrieved through textual information. Existing ontology-based inference is vulnerable to untrained environments and requires retraining to perform new detection goals. However, the image-text comparison inference technique is learned on a large scale, so natural language-based inference is possible using a model that interprets images as language information, and it is possible to recognize by setting a new detection goal.

The arson recognition module 420 recognizes the arson situation by comparing and inferring image information received from CCTV and text information explaining the abnormal situation in the same space. The arson recognition module 420 converts scene information from the entire screen area of the CCTV image and can be used by dividing patches into regional information to analyze the information from various angles. In the example of Figure 5, in order to extract visual information, the detected human area and the image of the area where the human stops are mapped into a comparison space through a visual information converter.

The language information converter maps the text describing the situation to be detected into the same comparison space. Image information and text descriptions mapped to the same space can be used to calculate similarity or recognize arson by comparing it to a normal environment. In Figure 5, recognition of not only the scene but also the action can be performed simultaneously through the description of the scene and the description of the arson action.

The image-language comparative inference technique according to one embodiment has robust versatility in various environments by using a large-capacity learned visual-language information converter. Stable inference based on natural language is possible by learning a conversion process that can understand not only the text that describes the object, but also the human behavior and object interaction in the image through language. The recognition module can receive images as input and express them through human-friendly language, so the inferred results can be interpreted and interaction with people is possible.

The image-language comparative inference technique according to one embodiment is an adaptive method that can adjust the detection goal by changing the description text to suit the environment in which it is used. For example, by adding text such as “I see a white glowing bonfire” to the detection items, it is possible to immediately adapt to the environment without learning. Additionally, detection accuracy can be improved by adding endings such as “at night,” “in the morning,” “on a snowy day,” or “on a rainy day,” using information on the time of shooting or weather information.

The abnormal situation detection unit 500, which pays attention to human behavior, receives human detection and trajectory information, recognizes human behavior, detects the section where the behavior occurred, and detects abnormal situations related to human behavior, such as fighting and falling.

The human behavior recognition module 510 uses the human location and tracking results obtained from the human detection and tracking unit 200 to receive an image of the human area of interest over time, recognizes the human behavior, and determines the type of behavior. It is a module that recognizes continuously. The behavior recognition model in the present invention can utilize known models such as R(2+1)D-18, and continuously classifies the human behavior classification results that appear on the screen to send the classification results to the abnormal behavior section detection module 520. Deliver.

The abnormal behavior section detection module 520 can detect a section of abnormal behavior by processing the previously classified behavior classification results of the human behavior recognition module 510 as time series data. The classification result of the human behavior recognition module 510 can be processed by loading it into an abnormal behavior classification queue (Queue) and an abnormal behavior section detection queue (Queue). The abnormal behavior classification cue is used to determine the type of final abnormal behavior that occurred for each processed image, and the behavior section detection cue can be used to determine the section of the final abnormal behavior determined in the previous abnormal behavior classification cue.

Figure 6 is a flowchart showing the abnormal behavior section detection process.

Referring to FIG. 6, the method for detecting abnormal behavior sections according to an embodiment of the present invention inputs identities appearing in the current frame (S610) and obtains recent behavior for each identity (S620). At this time, the most reliable behavior among the classification results of the most recent behavior for the human identity at each point in time can be defined as the behavior of the human with that identity at that point in time. Identity for which behavioral classification has not yet been performed due to a short appearance time can be treated as normal behavior.

The behavior of all identities present at each point in time is acquired and the most frequent behavior is loaded into the abnormal behavior classification queue (S630). On the other hand, in the abnormal behavior section detection queue, the most common behavior is calculated and loaded only for abnormal behavior (fighting, falling) among the behaviors of all identities present at each time (S640, S650).

The method for detecting abnormal behavior sections largely consists of two steps. First, the most frequently occurring actions are obtained from the actions in the abnormal action classification cue. Obtained behavior is defined as the main behavior occurring in the input image. To detect the main behavior section, only the main behavior is left in the abnormal behavior section detection queue. Afterwards, they are continuously connected into one cluster based on a certain number of frames, and each finally obtained cluster can be defined as the main action section of the input image. The above process (S610 to S650) can be repeated until the video ends (S660).

Figure 7 is a block diagram showing an integrated abnormal situation detection device according to an embodiment of the present invention.

Referring to FIG. 7, the integrated abnormal situation detection device according to an embodiment of the present invention includes an object detector 710 that detects object objects and human objects from an input image using a first neural network, and a human that tracks the human object. It includes an object tracking unit 720 and an abnormal situation detection unit 730 that detects an abnormal situation based on the object detection result and the human object tracking result.

At this time, the human object tracking unit 720 uses the first feature information generated based on the intermediate calculation result of the first neural network and the human object area corresponding to the final calculation result of the first neural network as input, and the second neural network A human object can be tracked using the second feature information extracted through .

At this time, the intermediate operation result includes spatial information and texture information of the input image, and the first feature information may be extracted by masking the area of the detected human object with respect to the intermediate operation result.

At this time, the human object tracking unit 720 may match the same human object between frames using the first characteristic information and the second characteristic information.

At this time, the first feature information may correspond to an M-dimensional feature vector, and the second feature information may correspond to an N-dimensional feature vector.

At this time, the human object tracking unit 720 may track the human object using an (M+N)-dimensional feature vector generated based on the first feature information and the second feature information.

At this time, the tracking result of the human object may include information on the occupied area and movement trajectory of the human object.

At this time, the human object tracking unit 720 may identify the object that was incorrectly detected as a human object based on the movement trajectory information of the human object.

At this time, the human object tracking unit 720 may calculate a motion vector corresponding to the movement trajectory of the human object, and use the result of the outer product between motion vectors for each section to identify the object that was incorrectly detected as a human object.

At this time, the abnormal situation detection unit 730 may detect an arson situation using visual feature information corresponding to the input image and language feature information corresponding to the arson situation text.

Figure 8 is a diagram showing the configuration of a computer system according to an embodiment.

The integrated abnormal situation detection device according to the embodiment may be implemented in a computer system 1000 such as a computer-readable recording medium.

Computer system 1000 may include one or more processors 1010, memory 1030, user interface input device 1040, user interface output device 1050, and storage 1060 that communicate with each other via bus 1020. You can. Additionally, the computer system 1000 may further include a network interface 1070 connected to the network 1080. The processor 1010 may be a central processing unit or a semiconductor device that executes programs or processing instructions stored in the memory 1030 or storage 1060. The memory 1030 and storage 1060 may be storage media that includes at least one of volatile media, non-volatile media, removable media, non-removable media, communication media, and information transfer media. For example, memory 1030 may include ROM 1031 or RAM 1032.

The specific implementations described in the present invention are examples and are not intended to limit the scope of the present invention in any way. For the sake of brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connections or connection members of lines between components shown in the drawings exemplify functional connections and/or physical or circuit connections, and in actual devices, various functional connections or physical connections may be replaced or added. Can be represented as connections, or circuit connections. Additionally, if there is no specific mention such as “essential,” “important,” etc., it may not be a necessary component for the application of the present invention.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the scope of the claims are within the scope of the spirit of the present invention. It will be said to belong to

710: Object detection unit
720: Human object tracking unit
730: Abnormal situation detection unit
1000: computer system 1010: processor
1020: Bus 1030: Memory
1031: Rom 1032: RAM
1040: User interface input device
1050: User interface output device
1060: Storage 1070: Network Interface
1080: Network

Claims (20)

In an integrated abnormal situation detection method performed in an abnormal situation detection device,
Detecting object objects and human objects from an input image using a first neural network;
tracking the human object; and
An integrated abnormal situation detection method comprising the step of detecting an abnormal situation based on an object detection result and a human object tracking result. In claim 1,
The step of tracking the human object is
First feature information generated based on an intermediate calculation result of the first neural network; and
An integrated abnormal situation detection method, characterized in that it is performed using the human object area corresponding to the final calculation result of the first neural network as input and second feature information extracted through a second neural network. In claim 2,
The intermediate operation result includes spatial information and texture information of the input image,
An integrated abnormal situation detection method, characterized in that the first characteristic information is extracted by masking the area of the detected human object with respect to the intermediate calculation result. In claim 2,
The step of tracking the human object is
An integrated abnormal situation detection method, characterized in that matching the same human object between frames using the first characteristic information and the second characteristic information. In claim 2,
The first feature information corresponds to an M-dimensional feature vector, and the second feature information corresponds to an N-dimensional feature vector,
The step of tracking the human object is
An integrated abnormal situation detection method, characterized in that tracking the human object using an (M+N) dimensional feature vector generated based on the first feature information and the second feature information. In claim 1,
The tracking result of the human object is
Contains occupied area and movement trajectory information of the human object,
The step of tracking the human object is
An integrated abnormal situation detection method characterized by identifying an object that is incorrectly detected as a human object based on the movement trajectory information of the human object. In claim 6,
The step of tracking the human object is
An integrated abnormal situation detection method characterized by calculating a motion vector corresponding to the movement trajectory of the human object and identifying an object misdetected as a human object using the result of an outer product between motion vectors for each section. In claim 1,
The step of detecting the abnormal situation is
Visual feature information corresponding to the input image; and
An integrated abnormal situation detection method characterized by detecting an arson situation using language feature information corresponding to the arson situation text. In claim 8,
The step of detecting the abnormal situation is
An integrated abnormal situation detection method, characterized in that the arson situation is detected by mapping the visual feature information and the language feature information to the same comparison space and calculating the similarity between the visual feature information and the language feature information. In claim 8,
The visual feature information is
An integrated abnormal situation detection method, characterized in that it is generated based on the input image, the human object area image, and the area image in which the human object is determined to have stayed for more than a preset time. In claim 1,
The step of detecting the abnormal situation is
An integrated abnormal situation detection method characterized by detecting human object behavior, setting main behavior for each section based on the frequency of the human object behavior, and calculating the section in which the abnormal situation occurs using the main behavior information for each section. . In claim 1,
The above abnormal situation is
An integrated abnormal situation detection method characterized by including intrusion, loitering, arson, abandonment, fighting, and collapse situations. an object detection unit that detects object objects and human objects from an input image using a first neural network;
a human object tracking unit that tracks the human object; and
an abnormal situation detection unit that detects an abnormal situation based on the object detection result and the human object tracking result;
An integrated abnormal situation detection device comprising: In claim 13,
The human object tracking unit
First feature information generated based on an intermediate calculation result of the first neural network; and
An integrated abnormal situation detection device, characterized in that the human object area corresponding to the final calculation result of the first neural network is input and the human object is tracked using second feature information extracted through a second neural network. In claim 14,
The intermediate operation result includes spatial information and texture information of the input image,
An integrated abnormal situation detection device, characterized in that the first characteristic information is extracted by masking the area of the detected human object with respect to the intermediate calculation result. In claim 14,
The human object tracking unit
An integrated abnormal situation detection device, characterized in that matching the same human object between frames using the first characteristic information and the second characteristic information. In claim 14,
The first feature information corresponds to an M-dimensional feature vector, and the second feature information corresponds to an N-dimensional feature vector,
The human object tracking unit
An integrated abnormal situation detection device, characterized in that tracking the human object using an (M+N) dimensional feature vector generated based on the first feature information and the second feature information. In claim 13,
The tracking result of the human object is
Contains occupied area and movement trajectory information of the human object,
The human object tracking unit
An integrated abnormal situation detection device characterized in that it identifies an object that is incorrectly detected as a human object based on the movement trajectory information of the human object. In claim 18,
The human object tracking unit
An integrated abnormal situation detection device that calculates a motion vector corresponding to the movement trajectory of the human object and identifies an object that has been incorrectly detected as a human object using the result of an outer product between motion vectors for each section. In claim 13,
The abnormal situation detection unit
Visual feature information corresponding to the input image; and
An integrated abnormal situation detection device characterized in that it detects an arson situation using language feature information corresponding to the arson situation text.

Images