KR20230077562A - Method and system for detecting complex image-based anomaly event - Google Patents

Abstract

An anomaly detection method based on complex images is provided. The method may include acquiring an RGB image and a depth image through a 2D camera and a depth camera, respectively; calculating a probability value for each deviant behavior class by inputting the RGB image and the depth image to a deep learning model; and outputting an abnormal behavior event based on the probability value for each abnormal behavior class.

Description

Complex image-based anomaly detection method and system

The present invention relates to a method and system for detecting an anomaly based on a composite image, and more particularly, to a technique for detecting an anomaly by using an RGB image and a depth image in combination.

The 2D image-based video security system according to the prior art has a problem in that the reliability of abnormal behavior recognition is greatly affected by environmental factors.

According to the prior art, an abnormal behavior recognition technology based on a 3D depth image has been proposed, but this has a limitation in that it cannot process an input image in real time due to low recognition accuracy and slow processing speed.

Publication No. 10-2006-0031832 (2006.04.13)

An embodiment of the present invention outputs an anomaly event by integrating and analyzing a 2D image (RGB image) and a depth image, and applies it to an intelligent video security system to accurately detect an anomaly event based on a composite image-based anomaly detection method and provide the system.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, a composite image-based anomaly detection method according to a first aspect of the present invention includes acquiring RGB images and depth images through a 2D camera and a depth camera, respectively; calculating a probability value for each deviant behavior class by inputting the RGB image and the depth image to a deep learning model; and outputting an abnormal behavior event based on the probability value for each abnormal behavior class.

In some embodiments of the present invention, the step of inputting the RGB image and the depth image to a deep learning model to calculate a probability value for each deviant behavior class may include inputting the RGB image to a first deep learning model for each first deviant behavior class. Calculating a probability value; and calculating a probability value for each second deviant behavior class by inputting the depth image to a second deep learning model.

In some embodiments of the present invention, in the outputting of the abnormal behavior event based on the probability value for each abnormal behavior class, the abnormal behavior event may be output by ensembling the probability values for each of the first and second abnormal behavior classes.

In some embodiments of the present invention, the outputting of the deviant behavior event based on the probability value for each deviant behavior class may include applying an ensemble technique for averaging the probability values for each of the first and second deviant behavior classes so that the probability value is maximized. Behavior can be output as an anomaly event.

Some embodiments of the present invention may further include transmitting the abnormal behavior event to an intelligent video security system.

Some embodiments of the present invention may further include continuously performing image analysis on the input RGB image and depth image when the abnormal behavior event is not detected.

In addition, an anomaly detection system based on a composite image of an RGB image and a depth image according to a second aspect of the present invention includes a communication module for receiving an RGB image and a depth image captured through a 2D camera and a depth camera, respectively, the RGB image and the depth image. As the program stored in the memory and the program stored in the memory for outputting an anomaly event through image analysis based on the image are executed, the RGB image and the depth image are input to the deep learning model to calculate the probability value for each anomaly class. and a processor for calculating and outputting an abnormal behavior event based on the calculated probability value for each abnormal behavior class.

In some embodiments of the present invention, the processor calculates a probability value for each first anomaly class by inputting the RGB image to a first deep learning model, and inputs the depth image to a second deep learning model to generate a second anomaly behavior. Probability values for each class can be calculated.

In some embodiments of the present invention, the processor may ensemble probability values for each of the first and second abnormal behavior classes to output an abnormal behavior event.

In some embodiments of the present invention, the processor may apply an ensemble technique of averaging probability values for each of the first and second deviant behavior classes to output an behavior having a maximum probability value as the deviant behavior event.

In some embodiments of the present invention, the processor may transmit the output anomaly event to the intelligent video security system through the communication module.

In some embodiments of the present invention, the processor may continuously perform image analysis on the input RGB image and depth image when the abnormal behavior event is not detected.

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium recording a computer program for executing the method may be further provided.

According to one embodiment of the present invention described above, an abnormal behavior can be accurately recognized through integrated analysis of an RGB image and a depth image. In particular, it is possible to recognize anomalies without false negatives and false positives at night or in a complex event environment.

By installing an embodiment of the present invention in a security vulnerable environment and linking it with an intelligent video security system, the effect of maximizing personal safety and preventing criminal acts can be expected.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram for comparing performance of object detection based on 2D and 3D images in a low light environment.
2 is a flow chart of an anomaly detection method based on composite images according to an embodiment of the present invention.
3 is a diagram illustrating a method for detecting an anomaly based on composite images in more detail according to an embodiment of the present invention.
4 is a diagram showing probability values for each RGB image-based abnormal behavior class.
5 is a diagram illustrating probability values for each depth image-based abnormal behavior class.
6 is a block diagram of an anomaly detection system based on composite images according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, in order to help the understanding of those skilled in the art, the background in which the present invention is proposed will be described first, and then the embodiments of the present invention will be described.

1 is a diagram for comparing performance of object detection based on 2D and 3D images in a low light environment.

A video security system based on a 2D image according to the prior art detects and classifies objects such as people and vehicles, and recognizes simple abnormal behaviors such as passing through a specific point, intrusion, and loitering by tracking the object.

However, video security systems that use only existing 2D images are greatly affected by environmental factors such as environments where multiple people cross or overlap each other, lighting, weather, and color, especially in low-light or nighttime situations where sufficient light is not secured. has a disadvantage in that the accuracy of object detection and action recognition is lowered.

In order to solve the problem caused by these external environmental factors, an anomaly behavior recognition technology based on 3D depth image (hereinafter referred to as depth image) is being developed. Depth image is not only an environment where a plurality of people intersect or overlap each other through precise distance data, but also object detection and behavior through stable image acquisition including objects even in a low-light environment such as at night and an environment in which lighting changes are severe as shown in FIG. It has the advantage of high recognition accuracy.

However, since depth images can use slightly less image characteristic information than 2D images, performance is somewhat lower than behavior recognition based on 2D images in daytime or in environments with no lighting changes.

In order to solve this problem, the complex image-based abnormal behavior detection method and system 100 according to an embodiment of the present invention integrates a 2D image, RGB image, and a depth image, and detects assault, fall, object abandonment, etc. By outputting abnormal behavior events and applying them to the intelligent video security system, it is possible to ensure the safety and property of individuals and to contribute to crime prevention by accurately detecting abnormal behavior not only in complex situations but also at night and in various situations.

Hereinafter, an anomaly detection method based on composite images according to an embodiment of the present invention will be described with reference to FIGS. 2 to 5 .

2 is a flow chart of an anomaly detection method based on composite images according to an embodiment of the present invention. 3 is a diagram illustrating a method for detecting an anomaly based on composite images in more detail according to an embodiment of the present invention. 4 is a diagram showing probability values for each RGB image-based abnormal behavior class. 5 is a diagram illustrating probability values for each depth image-based abnormal behavior class.

A composite image-based anomaly detection method according to an embodiment of the present invention includes acquiring RGB images and depth images through a 2D camera and a depth camera (S110), and inputting the RGB images and depth images to a deep learning model. and calculating a probability value for each deviant behavior class (S120) and outputting an anomalous behavior event based on the probability value for each deviant behavior class (S130).

Meanwhile, each step shown in FIG. 2 may be understood to be performed by the compound image-based anomaly detection system 100 to be described later, but is not necessarily limited thereto.

Referring to FIG. 3 , an RGB image is received from a 2D camera (S210), and a depth image is received from a depth camera (S240).

Next, the abnormal behavior is recognized through a deep learning model capable of recognizing the abnormal behavior using spatio and temporal information targeting the acquired RGB image and depth image (S220, S250).

Next, probability values for each deviant behavior class for the recognized deviant behavior are calculated (S230 and S260). At this time, the RGB image is calculated and output as a probability value for each first anomaly class as it is input to the first deep learning model, and the depth image is calculated and output as a probability value for each second anomaly class as it is input to the second deep learning model. do.

4 is a diagram showing probability values for each deviant behavior class through RGB image-based action recognition, and FIG. 5 is a diagram showing probability values for each deviant behavior class through depth image-based action recognition.

Next, the probability value for each first and second abnormal behavior class is ensemble in units of frames to calculate the behavior having the maximum probability value (S270). At this time, as an embodiment of ensemble, the average of probability values for each action may be applied.

Next, in the step of determining whether an abnormal behavior event has occurred (S280), if an abnormal behavior event has occurred, abnormal behavior event information is transmitted to the intelligent video security system, and the intelligent video security system provides a notification (S290). In contrast, when an abnormal behavior event is not detected, image analysis is continuously performed on the input RGB image and depth image.

As such, according to an embodiment of the present invention, an abnormal behavior event may be detected by integrally analyzing an RGB image input from a 2D camera and a depth image input from a depth camera. Through this, an embodiment of the present invention can ensure personal safety and prevent criminal acts by detecting anomalies more accurately and quickly than video analysis technology used in existing video security systems in low-light environments and complex event situations such as at night. can

Meanwhile, in the above description, steps S110 to S290 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. Also, some steps may be omitted if necessary, and the order of steps may be changed. In addition, even if other omitted contents, the contents described in FIGS. 2 to 5 are also applied to the complex image-based anomaly detection system 100 of FIG. 6 .

6 is a block diagram of an anomaly detection system 100 based on composite images according to an embodiment of the present invention.

A composite image-based anomaly detection system 100 according to an embodiment of the present invention includes a communication module 110, a memory 120, and a processor 130.

The communication module 110 receives an RGB image and a depth image respectively photographed through a 2D camera and a depth camera.

The memory 120 stores a program for outputting an abnormal behavior event through image analysis based on an RGB image and a depth image, and the processor 130 executes the program stored in the memory 120 . Here, the memory 120 collectively refers to a non-volatile storage device and a volatile storage device that continuously retain stored information even when power is not supplied.

For example, the memory 120 may include a compact flash (CF) card, a secure digital (SD) card, a memory stick, a solid-state drive (SSD), and a micro SD card. NAND flash memory such as cards, magnetic computer storage devices such as hard disk drives (HDD), and optical disc drives such as CD-ROMs and DVD-ROMs. can

The processor 130 inputs the RGB image and the depth image to the deep learning model to calculate a probability value for each deviant behavior class, and outputs an anomalous behavior event based on the calculated probability value for each deviant behavior class.

The composite image-based anomaly detection method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a server, which is hardware, and stored in a medium.

The aforementioned program is C, C++, JAVA, machine language, etc. It may include a code coded in a computer language of. These codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include memory reference related codes for which location (address address) of the computer's internal or external memory should be referenced for additional information or media required for the computer's processor to execute the functions. there is. In addition, when the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes for whether to communicate, what kind of information or media to transmit/receive during communication, and the like.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the computer or various recording media on the user's computer. In addition, the medium may be distributed to computer systems connected through a network, and computer readable codes may be stored in a distributed manner.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: Complex image-based anomaly detection system
110: communication module
120: memory
130: processor

Claims (12)

In a method performed by a computer,
acquiring an RGB image and a depth image through a 2D camera and a depth camera, respectively;
calculating a probability value for each deviant behavior class by inputting the RGB image and the depth image to a deep learning model; and
Outputting an abnormal behavior event based on the probability value for each abnormal behavior class.
Complex image-based anomaly detection method.
According to claim 1,
In the step of inputting the RGB image and the depth image to a deep learning model to calculate a probability value for each deviant behavior class,
calculating a probability value for each first deviant behavior class by inputting the RGB image to a first deep learning model; and
Calculating a probability value for each second anomaly class by inputting the depth image to a second deep learning model,
Complex image-based anomaly detection method.
According to claim 2,
In the step of outputting an abnormal behavior event based on the probability value for each abnormal behavior class,
An abnormal behavior event is output by ensembling probability values for each of the first and second abnormal behavior classes.
Complex image-based anomaly detection method.
According to claim 3,
In the step of outputting an abnormal behavior event based on the probability value for each abnormal behavior class,
Applying an ensemble technique of averaging probability values for each of the first and second abnormal behavior classes to output an behavior having a maximum probability value as an abnormal behavior event,
Complex image-based anomaly detection method.
According to claim 1,
Further comprising transmitting the abnormal behavior event to an intelligent video security system,
Complex image-based anomaly detection method.
According to claim 1,
Further comprising the step of continuously performing image analysis on the input RGB image and depth image when the abnormal behavior event is not detected.
Complex image-based anomaly detection method.
In the composite image-based anomaly detection system of RGB image and depth image,
A communication module for receiving an RGB image and a depth image captured through a 2D camera and a depth camera, respectively;
A memory storing a program for outputting an abnormal behavior event through image analysis based on the RGB image and the depth image; and
As the program stored in the memory is executed, the RGB image and the depth image are input to the deep learning model to calculate a probability value for each abnormal behavior class, and to output an abnormal behavior event based on the calculated probability value for each abnormal behavior class including a processor,
Complex video-based anomaly detection system.
According to claim 7,
Wherein the processor calculates a probability value for each first deviant behavior class by inputting the RGB image to a first deep learning model, and calculates a probability value for each second deviant behavior class by inputting the depth image to a second deep learning model. ,
Complex video-based anomaly detection system.
According to claim 8,
Wherein the processor outputs an abnormal behavior event by ensembling probability values for each of the first and second abnormal behavior classes;
Complex video-based anomaly detection system.
According to claim 9,
Wherein the processor applies an ensemble technique of averaging probability values for each of the first and second abnormal behavior classes to output an behavior having a maximum probability value as an abnormal behavior event;
Complex video-based anomaly detection system.
According to claim 7,
Wherein the processor transmits the output anomaly event to an intelligent video security system through the communication module;
Complex video-based anomaly detection system.
According to claim 7,
Wherein the processor continues to perform image analysis on the input RGB image and depth image when the abnormal behavior event is not detected.
Complex video-based anomaly detection system.

Images