KR101942808B1 - Apparatus for CCTV Video Analytics Based on Object-Image Recognition DCNN - Google Patents

Abstract

The present invention relates to an object image recognition DCNN-based CCTV image analysis apparatus, and an object image recognition DCNN-based CCTV image analysis apparatus according to an embodiment of the present invention receives a plurality of video frame images provided by a video providing apparatus, And an object image extracting unit for extracting and extracting object images of a plurality of tracking objects for motion tracking in the plurality of video frame images converted into the above format using an object image recognition method DCNN, and classifies the plurality of tracked objects based on the application result. The user interest object obtained by removing the user non-interest object from the plurality of tracked objects, Motion based detection of events based on trace information and assignment rules And an image analysis unit.

Description

[0001] The present invention relates to an object image recognition DCNN-based CCTV image analysis apparatus,

The present invention relates to an object image recognition DCNN-based CCTV image analysis apparatus, and more particularly, to a central processing unit (CPU) capable of detecting / tracking objects of interest from a CCTV image, And more particularly, to a CCTV image analysis apparatus based on an object image recognition DCNN that performs robust and efficient image analysis using a Deep Convolutional Neural Network (DCNN).

Recently, the number of CCTV cameras installed for security / safety in government offices and corporations is increasing explosively. However, compared to the number of installed CCTV cameras, the number of CCTV camera monitoring personnel is insufficient. To solve these problems, the introduction of the intelligent CCTV video surveillance system is actively performed.

The CCTV image analyzer, which is the core of the intelligent CCTV video surveillance system, detects and tracks moving objects by receiving video images from CCTV cameras and automatically detects abnormal situations such as " . The monitoring personnel can effectively monitor multiple CCTV camera images by checking only the CCTV images for which the alarm has occurred without having to constantly watch a large number of (meaningless) CCTV images.

However, most of existing CCTV image analysis devices use motion-based object detection algorithms. In addition to detection of actual objects of interest (eg, people and vehicles), there are various causes (eg, Objects are frequently detected by waves, moving shadows, sudden lighting changes, flashing lights, snow / rain, etc.). This leads to frequent false alarms, making effective monitoring impossible.

Computer Vision Researchers have developed object detection technology based on object shape learning since the mid - 2000s in addition to motion - based object detection technology. In the above technique, object shape features are extracted and learned from various learning images of a specific type of object (for example, a pedestrian), and an area showing shape characteristics similar to the shape features of the learned object is found in the image, . Typically, there are object detection technologies such as Viola-Jones, HOG, ICF, ACF, and DPM. However, it has been difficult to apply it to commercial CCTV image analysis devices due to the detection performance limitations and processing load problems of such object detection techniques.

 In the meantime, the team of Professor G. Hinton of the University of Toronto in Toronto will use DCNN, AlexNet, to win the overwhelming performance difference from the existing image recognition algorithms in ILSVRC (ImageNet Large Scale Visual Recognition Challenge) In the field of vision, Deep Learning technology has been attracting attention, and then attempts have been made to solve various problems of computer vision using deep learning technology.

Object detection technologies based on DCNN have been launched since 2014. These DCNN-based object detection technologies provide detection performance well beyond that of existing object detection techniques. Typically, there are object detection technologies such as Fast / Faster R-CNN, RFCN, SSD and YOLO.

However, there are still several limitations in applying DCNN-based object detection technology to commercial CCTV image analysis devices. A typical limitation is that the hardware cost for real-time video processing using a DCNN-based object detector is very high. Since a typical DCNN-based object detector requires a considerable amount of computation to detect an object from a single video frame, the DCNN-based object detector in the general CPU is used to process the video in real time (for example, It is very difficult to perform frame over object detection). Therefore, to process video in real time using a DCNN-based object detector, a GPU capable of large-scale parallel operation is indispensable. Even if the GPU is used, it is difficult to process several video streams simultaneously in real time in one image analysis device without using an expensive GPU with high performance.

Korean Registered Patent No. 10-1040049 (June 23, 2011) Korean Registered Patent No. 10-1173853 (August 8, 2012) Korean Registered Patent No. 10-1178539 (Aug. 24, 2012) Korean Registered Patent No. 10-1748121 (Jun. Korean Registered Patent No. 10-1789690 (Oct. 18, 2017) Korean Registered Patent No. 10-1808587 (July 07, 2017) Korean Patent Publication No. 10-2018-0072561 (June 29, 2018) Korean Registered Patent No. 10-1850286 (Apr. 13, 2018) Korean Patent Publication No. 10-2018-0107930 (Oct.

"Pedestrian Detection: An Evaluation of the State of the Art" IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, Issue 4, April 2012 "Efficient Processing of Deep Neural Networks: A Tutorial and Survey", Proceedings of the IEEE, Vol. 105, Issue 12, Dec. 2017 "Object Detection with Deep Learning: A Review" arXiv.org, arXiv: 1807.05511 [cs.CV], Jul. 2018

An embodiment of the present invention is a CCTV image analyzing apparatus capable of detecting / tracking interested objects from a CCTV image, for example, in a central processing unit (CPU) The object of the present invention is to provide an object image recognition DCNN-based CCTV image analysis apparatus that performs robust and efficient image analysis.

The object image recognition DCNN-based CCTV image analysis apparatus according to an embodiment of the present invention includes an image recognition DCNN-based CCTV image analysis apparatus that performs image analysis using a DCNN (Deep Convolutional Neural Network) , The first video data being composed of a plurality of first video frames having a first pixel format, a first resolution and a first frame rate, from the video providing apparatus, The first resolution, the first frame rate, and the first pixel format are respectively referred to as a second resolution lower than the first resolution, a second frame rate lower than the first frame rate, An image converter for converting the first image data into a second image data of a different second pixel format having a different format, A plurality of tracking objects are detected and tracked, a plurality of images of the objects being tracked are respectively extracted, and the plurality of tracking objects are identified using recognition results obtained by inputting the extracted object images to the DCNN The method includes the steps of: identifying a user non-interest object that is out of the plurality of tracking objects identified by the user and notifying the user that the user interest object is out of the plurality of tracking objects; And a motion-based image analysis unit for detecting an event based on the first frame rate, wherein the image conversion unit includes a video frame acquisition unit that receives the plurality of video frames input according to the first frame rate, A smaller number of video frames are sampled to generate a plurality of second video frames A video frame scaling unit for converting the plurality of generated second video frames to the second resolution, and a second pixel unit for converting the first pixel format of the plurality of second video frames, And a pixel format conversion unit for providing the second image data generated by converting the second pixel format to the motion-based image analysis unit.

The motion-based image analyzing unit generates a difference image using the second image data and the learned background image, removes noise from the generated difference image, and detects a motion region for the plurality of moving objects An object tracking unit for detecting and tracking the plurality of moving objects by using the detected motion area and the second image data, and an object tracking unit for detecting and tracking the plurality of moving objects using the detected motion area and the second image data, A first recognition result obtained by applying an object image of a tracked object to the DCNN and a second recognition result obtained by applying the plurality of second tracks extracted from at least one other video frame inputted at a predetermined time interval to the one of the plurality of second video frames Based on a second recognition result obtained by applying an object image of an object to the DCNN, A tracking object classifying unit for classifying a plurality of tracking objects, a tracking object classifying unit for classifying objects, a user non-interest object that is out of the plurality of classified tracking objects, And a tracking object-based event detection unit that detects an event in which the tracking information of the user interest object obtained in accordance with the cancellation and the tracking information of the user interest object satisfy the specifying rule.

The tracked object classifier may calculate and accumulate the first recognition result and the second recognition result as scores, respectively, and determine a class having the highest cumulative score as the type of the tracked object.

The non-interest tracking object removal unit may identify a user non-interest tracking object that deviates from the specified reference in the determined type of object and remove the object from the plurality of tracking objects.

The image converter converts the first pixel format having a pixel format of a luminance signal (Y), a difference (U) of a red signal and a difference (V) between a luminance signal and a blue component to RGB (Red-Green-Blue) To a second pixel format having a Gray-Scale pixel format.

The driving method of the object image recognition DCNN-based CCTV image analyzing apparatus according to the embodiment of the present invention includes a DCNN (Deep Convolutional Neural Network) that provides a recognition result for an object in an image, A method for driving a CCTV image analysis apparatus based on a first CCTV image, the method comprising: receiving first video data composed of a plurality of first video frames having a first pixel format, a first resolution and a first frame rate, The first resolution, the first frame rate, and the first pixel format of the received first image data, respectively, to a second resolution lower than the first resolution, a second frame rate lower than the first frame rate, Converting the first video data into second video data of a second different pixel format having a different format from the one pixel format, The method comprising the steps of: detecting and tracking a plurality of moving objects on the basis of motion, extracting images of the plurality of objects being tracked, inputting the extracted object images to the DCNN, The method comprising: identifying a plurality of tracking objects; identifying a user non-interest object that deviates from a specified criterion from the plurality of track objects to remove the identified user non-interest object from the objects of the plurality of tracking objects; And detecting an event based on the tracking information and the assignment rule,

Wherein the converting comprises receiving the plurality of video frames input according to the first frame rate, generating a plurality of second video frames by sampling a lesser number of video frames than the input plurality of first video frames Converting the generated plurality of second video frames to the second resolution and converting the first pixel format of the plurality of second video frames converted to the second resolution into the second pixel format And providing the second image data generated by the conversion to the motion-based image analysis unit.

The motion-based image analyzing unit generates a difference image using the second image data and the learned background image, removes noise from the generated difference image, and detects a motion region for the plurality of moving objects An object tracking unit for detecting and tracking the plurality of moving objects by using the detected motion area and the second image data, and an object tracking unit for detecting and tracking the plurality of moving objects using the detected motion area and the second image data, A first recognition result obtained by applying an object image of a tracked object to the DCNN and a second recognition result obtained by applying the plurality of second tracks extracted from at least one other video frame inputted at a predetermined time interval to the one of the plurality of second video frames Based on a second recognition result obtained by applying an object image of an object to the DCNN, A tracking object classifying unit for classifying a plurality of tracking objects, a tracking object classifying unit for classifying objects, a user non-interest object that is out of the plurality of classified tracking objects, And a tracking object-based event detection unit that detects an event in which the tracking information of the user interest object obtained in accordance with the cancellation and the tracking information of the user interest object satisfy the specifying rule.

The tracked object classifier may calculate and accumulate the first recognition result and the second recognition result as scores, respectively, and determine a class having the highest cumulative score as the type of the tracked object.

The non-interest tracking object removal unit may identify a user non-interest tracking object that deviates from the specified reference in the determined type of object and remove the object from the plurality of tracking objects.

The image converter converts the first pixel format having a pixel format of a luminance signal (Y), a difference (U) of a red signal and a difference (V) between a luminance signal and a blue component to RGB (Red-Green-Blue) To a second pixel format having a Gray-Scale pixel format.

According to the embodiment of the present invention, the problem of the CCTV image analysis apparatus using the existing motion-based object detection algorithm is improved by utilizing the DCNN-based image recognition technology, and DCNN-based image analysis is also possible in a device having a general CPU The cost of purchasing existing expensive equipment will be saved.

Also, by using the DCNN based image recognition technology with deep learning, the accuracy of object detection will be increased, so that alarm sending according to event detection will be accurate.

1 is a block diagram of a CCTV image analysis apparatus based on an object image recognition DCNN according to an embodiment of the present invention;
FIG. 2 is an exemplary diagram showing a processing procedure and a result of a motion region detecting unit according to an embodiment of the present invention;
FIG. 3 is an exemplary view showing a processing procedure and a result of an object tracking unit according to an embodiment of the present invention;
4 is a view for explaining a process of a tracking object classifying unit according to an embodiment of the present invention,
5 is an illustration of sample images for learning an object image recognition DCNN that classifies a given image into one of human, vehicle,
FIG. 6 illustrates an example of classifying trace objects through a trace object classifier according to an embodiment of the present invention. FIG.
FIG. 7 is an exemplary diagram illustrating the processing result of the unattended tracking object removal according to an embodiment of the present invention;
FIG. 8 is a diagram illustrating a processing result of an interest tracking object-based event detection unit according to an embodiment of the present invention;
FIG. 9 is a configuration diagram of an object image recognition DCNN-based CCTV image analyzing device for simultaneously real-time analyzing N CCTV camera images, according to an embodiment of the present invention;
10 is a configuration diagram of a CCTV image analysis system based on an object image recognition DCNN for performing image analysis on a large scale in a CCTV control center or the like according to an embodiment of the present invention;
11 is a block diagram illustrating a structure of an image analysis apparatus according to another embodiment of the present invention, and
12 is a flowchart illustrating a process of driving an image analysis apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an object image recognition DCNN based CCTV image analyzing apparatus according to an embodiment of the present invention.

1, an object image recognition DCNN-based CCTV image analysis apparatus (or an image analysis apparatus) 100 according to a temporal example of the present invention includes an image conversion unit (or an image acquisition unit) 121, An analysis unit 122 and an object image recognition DCNN (unit) 108. [

Here, "including some or all of" means that some components, such as the object image recognition DCNN 108, are omitted so that the CCTV image analysis apparatus 100 is configured or some components such as the motion-based image analysis unit 122 May be integrated with other components such as the image conversion unit 121, and the like, and the description will be made in order to facilitate a thorough understanding of the present invention.

The image converting unit 121 acquires image data to be used by the motion-based image analyzing unit 122. For example, the image conversion unit 121 can acquire image data provided by the CCTV via the communication network of the communication company or image data provided by the server such as a government office. The motion-based image analyzing unit 122 receives the image data provided by the image converting unit 121 and performs motion and DCNN-based image analysis. Here, "image data" may mean a video signal, but may also mean video data. Generally, a video signal includes a video signal and a voice signal, and may further include additional information (e.g., encoding information, caption information, etc.), wherein the video signal represents video data. However, since the processing of video signals is mainly handled in the image processing, those skilled in the art tend to use video data and video data in combination. Therefore, in the embodiment of the present invention, the concept of the above term is not particularly limited. In other words, the video data and the video data can be used in almost the same concept.

However, the image data includes a series of unit frame images. In other words, it consists of a series of video frames. For example, if the frame rate is 60 FPS (Frames Per Second), it means that 60 still images per second are transmitted / received. Therefore, although a video frame, a frame video, or a unit frame is used in a substantially similar concept, the terms used may vary slightly depending on whether the object referred to is a frame or data. Of course, since the unit frame image forms serial data in a communication process, the image data can be regarded as data in which pixel values of pixels constituting a unit frame image are composed of serial data. However, in the embodiment of the present invention, since the above terms are used in various ways by a person skilled in the art, the concept is not particularly limited. However, when classification is necessary, it should be used clearly.

The image conversion unit 121 may include a part or all of the video frame acquisition unit 101, the video frame subsampling unit 102, the video frame scaling unit 103 and the pixel format conversion unit 104, The motion-based image analyzing unit 122 includes a motion region detecting unit 105, an object tracking unit 106, a tracking object classifying unit 107, a non-interest tracking object removing unit 109, an interest tracking object based event detecting unit 110, May include some or all of < RTI ID = 0.0 > Here, "including some or all of them" is the same as the preceding meaning.

The video frame obtaining unit 101 continuously obtains video frame data from a video providing apparatus such as a CCTV camera, a video streaming server, and the like connected to the image analyzing apparatus 100. For example, when the video providing apparatus connected to the image analyzing apparatus 100 is an IP camera, the video frame acquiring unit 101 receives and decodes the encoded video stream data from the IP camera to generate a YUV pixel format (e.g., a YV12 pixel format (Or frame video data) of the video frame. Here, YUV (for example, 16 bits) is a color representing a color by three pieces of information: a difference (U) between a luminance signal (Y) and a red signal, and a difference (V) between a luminance signal and a blue component. Since the Y component is sensitive to errors, it codes more bits than the color components U and V, and the ratio of Y: U: V is generally 4: 2: 2, which is also called YUV422. It is also called YcrCb using the abbreviation of Chroma Red and Chroma Blue combined with Chroma which refers to the color because it uses red component and blue component.

The video frame sub-sampling unit 102 subsamples the frames to be used for image analysis from the (video) frames obtained by the video frame obtaining unit 101. For example, if the frame rate of the video input to the video analysis apparatus 100 is 30 FPS and the video analysis frame rate is set to 6 FPS, the video frame sub-sampling unit 102 extracts And acquires a video frame by one frame for every five frames.

The reason why all the frames of the input video are not used for image analysis and the sub-sampled frame (or the plurality of second video frames) is used is that the processing load of the image analysis apparatus becomes very large when all the frames of the input video are used The effect (for example, object tracking performance improvement, etc.) is usually small. 6 to 10 FPS is sufficient to track a pedestrian or vehicle object in a typical CCTV image.

The video frame scaling unit 103 scales the size (e.g., resolution) of the video frame acquired by the video frame subsampling unit 102 to a predetermined size. In general, the video frame scaling unit 103 reduces a high-resolution input image to a low resolution. For example, if the resolution of the video frame obtained by the video frame obtaining unit 101 is a high resolution of 1920x1080 and the resolution of the image to be used in the image analyzing unit 122 is set to 640x480, the video frame scaling unit 103 obtains a resolution of 1920x1080 resolution And generates an image having a resolution of 640x480 through pixel sub-sampling and linear interpolation from the input image.

The reason for using the reduced low resolution image without using the original high resolution image in the image analysis is that when the high resolution image is used as it is, the processing load of the image analyzing device becomes very large while the effect (for example, Is usually small. To detect a small object in a remote surveillance environment, it may be necessary to process a high-resolution input image without reducing it. However, in a typical short-range surveillance environment, even if a high-resolution input image is reduced, The processing load on the analyzer is much less of a benefit.

The pixel format conversion unit 104 converts an image of the YUV pixel format scaled by the video frame scaling unit 103 into an RGB pixel format or a gray-scale format, which is a pixel format that can be used in the image analysis unit 122, Pixel format.

The motion region detection unit 105 obtains (or recognizes) a basic motion region through the difference (or difference image) between the background image and the input image that are periodically learned, and performs a motion picture region removal method and a morphology filtering Thereby detecting the final motion area. FIG. 2 shows an example of a process and a result of the motion region detection unit 105. The motion region detection unit 105 includes an input image 201, a learned background image 202, a basic motion region detection result 203 based on a difference between an input image and a background image, An example of final motion area detection result 204 by noise foreground pixel removal and morphology filtering.

The object tracking unit 106 performs multiple (or plural) object tracking using the input image 201 and the final motion area detection result 204 of FIG. Specifically, the object tracking unit 106 performs new object detection, inter-frame object tracking by matching, template image of a tracking object and bounding box coordinate update, tracking object list management, trajectory management for each tracking object, and the like . FIG. 3 is an example of the process and result of the object tracking unit 106 of FIG. 1, which is an example of a template image 301 and an object detection / tracking result 302 of some tracking objects. In the example of the object detection / tracking results 302 of FIG. 3, it can be seen that a number of meaningless objects (e.g., objects detected by motion of twigs swaying in the wind) are detected in addition to the human object of interest .

The tracking object classification unit 107 performs object classification using the object image recognition DCNN 108 for the objects being tracked. Details related to the tracking object classifying unit 107 will be described later with reference to FIGS. 4 to 6. FIG.

In addition, the non-interest tracking object removal unit 109 removes the tracking objects classified into the non-interest class (class, class, layer) from the tracking object list of the object tracking unit 106.

The tracking object-based event detection unit 110 detects an event that satisfies a specified rule by using tracking information of objects of interest classes.

FIG. 4 is a diagram for explaining a processing procedure of the tracking object classifying unit of FIG. 1;

Referring to FIG. 4 with reference to FIG. 1, as in the example of FIG. 4, the object image recognition DCNN 108 of FIG. 1 receives input of a normalized object image of size, Unidentified "class. For example, when the object image is input, the object image recognition DCNN 108 may compare the input object image with previously stored images (e.g., sample image or image data) and provide the recognition result.

The tracking object classifying unit 107 of FIG. 1 acquires an image patch (or an object image) corresponding to the object bounding box area from the input image for each object being tracked, and normalizes the size of the image patch And then attempts recognition through the object image recognition DCNN 108.

The tracking object classifying unit 107 periodically tries to perform DCNN recognition not only once but also several times in order to perform more accurate object classification. 4, the DCNN recognition result obtained at a specific point in time may be an unstable recognition result according to the detection state of the object bounding box.

Specifically, as shown in FIG. 4, the tracking object classifying unit 107 acquires an image (or an image patch) of an object from the input image every predetermined period P with respect to a specific tracking object and attempts to recognize the DCNN. Then, the class having the highest cumulative score in the current frame is determined as the class of the trace object. For example, if seven unit frames are input at a predetermined time interval as shown in FIG. 4, the object image patches for the same object are obtained in the first, third, fifth, and seventh unit frames, It can be applied periodically at regular intervals.

The above cycle P value is usually 0.5 to 1 second. The smaller the P value, the greater the processing load of the image analysis apparatus 100. The larger the P value, the lower the object classification performance. When acquiring an image patch of the object, it is preferable to acquire the original resolution image (i.e., a video frame obtained from the video frame subsampling unit 102 in FIG. 1). This is because if the image patch of the object is obtained from the image after the reduction, the quality may be degraded and an image patch which is difficult to recognize may be obtained.

The number of DCNN recognition attempts that are performed periodically may be limited to a total of N circuits. This is to prevent the DCNN recognition attempt from continuing unnecessarily for long-time tracking objects.

5 is an example of sample images for learning an object image recognition DCNN that classifies a given image into one of classes a, b, and c.

For convenience of explanation, referring to Fig. 5 together with Fig. 1, in the case of learning data of a " person " object in particular, as in the example of Fig. 5, a partial image of a person or an image including two or more persons It is noted that it is included in the learning sample. This is because, in the case of the human object area detected by the motion-based image analyzing unit 122 of FIG. 1, a part of a person or more than two persons is included in a case not only of the whole body of a person.

As the model of the object image recognition DCNN 108 of FIG. 1, various types of DCNN models proposed up to now can be used. Examples include AlexNet, VGGNet, GoogLeNet, and ResNet models that have won ILSVRC (ImageNet Large Scale Visual Recognition Challenge). When selecting the actual DCNN model to be used, it is difficult to consider only the recognition performance of DCNN. This is because there is usually a trade-off relationship between the processing time (inference time) and the capacity (for example, the number of parameters) of the DCNN and the recognition performance of the DCNN. Recently, DCNN models that are similar to existing DCNN models and significantly reduce the processing time and the number of parameters have recently been announced. For example, SqueezeNet or MobileNet.

FIG. 6 is an example of classifying the tracking objects through the tracking object classifying unit of FIG. 1, FIG. 7 is an example showing the processing result of the unattended tracking object removing unit of FIG. 1, This is an example showing the processing result of the detection unit.

In FIG. 6, it can be seen that all of the false-detected objects other than the actual human object are normally classified as " Unknown ".

In FIG. 7, when the " unidentified " class is designated as a non-interest class, the processing result of the unattended tracking object removal unit 109 from the object classification result of FIG. 6 is shown.

FIG. 8 shows an example of detecting an event in which a " person " object enters an area designated.

FIG. 9 is a block diagram of an object image recognition DCNN-based CCTV image analyzing apparatus for simultaneously real-time analyzing N CCTV camera images according to an embodiment of the present invention.

9, the CCTV image analysis apparatus 900 according to an embodiment of the present invention includes N video channel processing units 901, an object image recognition processing unit 902, Includes some or all of the object image recognition DCNN 108, wherein "including some or all" is the same as the preceding meaning.

The video channel processing unit 901 may include some or all of the image conversion unit 121 and the motion-based image analysis unit 122 of FIG. The N video channel processing unit 901 and the object image recognition processing unit 902 typically operate on the CPU of the image analysis apparatus 900. [ On the other hand, the object image recognition DCNN 108 may operate on a GPU capable of large scale parallel operations for high speed operation.

The N video channel processing units 901 use "shared" one object image recognition DCNN 108 instead of using N object image recognition DCNNs, respectively. A large amount of system memory is generally required to drive one DCNN. If N video channel processing units 901 individually use DCNN in memory, an insufficient memory problem may occur as N increases to be.

Each of the motion-based image analyzing units 122 in the N video channel processing units 901 performs a classification operation of the tracking objects in the same manner as described with reference to FIG. 4, And transmits the request message to the object image recognition processing unit 902 periodically. The recognition request messages are sequentially stored in a request message queue of the object image recognition processor 902. [ The object image recognition processing unit 902 immediately fetches the request message from the queue and processes it as soon as it finds the recognition request message in the request message queue. That is, the object image recognition processing unit 902 acquires the object image recognition result by inputting the normalized object image patch received together with the recognition request message to the object image recognition DCNN 108, and outputs the corresponding motion-based image analysis And transmits the object image recognition result to the unit 122. The motion-based image analyzing unit 122 receives the object image recognition result and performs an object classification operation as shown in FIG.

10 is a block diagram of an object image recognition DCNN-based CCTV image analysis system for performing image analysis on a large scale in a CCTV control center or the like according to an embodiment of the present invention.

In order to construct a large-scale image analysis system, a system may be constructed simply by using a plurality of image analysis apparatuses 900 shown in FIG. However, high-performance GPUs that can be used in the control center are usually very expensive, so there is a costly problem to install and use the GPUs for individual image analysis devices.

In order to solve the above problem, in the embodiment of the present invention, as shown in FIG. 10, the conventional video channel processing is performed by a plurality of image analysis servers 1001 without GPU, and DCNN-based object image recognition is performed by a high- The image analysis system 1000 may be configured so that a small number of object image recognition servers 1002 loaded with the image analysis system 1002 can perform the image analysis. The image analysis server 1001 performs high-speed network communication with the object image recognition server 1002.

The video channel processing unit of the image analysis server 1001 periodically transmits the normalized object image patch of the tracking object and the recognition request message to the object image recognition server 1002 in order to classify the tracking objects in the same manner as FIG. . At this time, the object image patch data is compressed and delivered in a format such as JPEG. The recognition request messages are sequentially stored in a request message queue of the object image recognition processing unit 1003 of the object image recognition server 1002. [ The object image recognition processing unit 1003 immediately fetches the request message from the queue and processes it as soon as it finds the recognition request message in the request message queue. That is, the object image recognition processing unit 1003 decodes the compressed object image patch data received together with the recognition request message, and inputs the decoded object image patch data to the object image recognition DCNN 108 to acquire the object image recognition result. The object image recognition processing unit 1003 transmits the object image recognition result to the video channel processing unit of the image analysis server 1001 that has issued the recognition request. The video channel processing unit of the image analysis server 1001 receives the object image recognition result and performs the object classification operation as shown in FIG.

11 is a block diagram illustrating a structure of an image analysis apparatus according to another embodiment of the present invention.

11, the image analysis apparatus 1100 according to another embodiment of the present invention includes a communication interface unit 1110, a control unit 1120, a DCNN-based image analysis unit 1130, and a storage unit 1140, Or the like.

Here, "including some or all of" means that the image analyzing apparatus 1100 is constituted by omitting some components such as the communication interface unit 1110 and the storing unit 1140, or the DCNN-based image analyzing unit 1130 But it is to be understood that the same reference numerals are used throughout the drawings to refer to the same or like components, such as the control unit 1120, and the like.

The communication interface unit 1110 communicates with the CCTV via the communication network of the communication company, and can perform operations such as side / demodulation in the course of the communication.

The control unit 1120 performs overall control operations of the communication interface unit 1110, the DCNN-based image analysis unit 1130, and the storage unit 1140 of the image analysis apparatus 1100 of FIG. For example, the control unit 1120 may provide the DCNN-based image analysis unit 1130 with a photographed image of the CCTV provided through the communication interface unit 1110.

The DCNN-based image analyzing unit 1130 can perform the image analyzing operation according to the embodiment of the present invention described above with reference to FIG. 1 to FIG. Typically, the frame rate of the received frame image is converted or a unit frame image is randomly acquired. In the embodiment of the present invention, this is called subsampling. Further, the resolution of the unit frame image acquired through subsampling is changed from high resolution to low resolution. Then, the format of the received frame image can be converted.

In addition, the DCNN-based image analyzer 1130 extracts object image patches from the format-converted sub-sampled frame images, recognizes object images based on DCNN, and accurately classifies the objects. In this case, do. The DCNN-based image analyzing unit 1130 detects an event only for the target object of interest through the removing operation. Then, an alarm can be output based on the event.

The communication interface unit 1110, the control unit 1120, the DCNN-based image analysis unit 1130 and the storage unit 1140 of FIG. 11 have been described above in detail. I would rather.

Meanwhile, as another embodiment of the present invention, the controller 1120 of FIG. 11 may include a CPU and a memory. The CPU may include a control circuit, an operation unit (ALU), an instruction interpretation unit, a registry, and the like, and the memory may include a RAM. Here, the CPU may mean the CPU in Fig. The control circuit is responsible for the control operation, the arithmetic operation unit performs the bit operation, and the instruction interpretation unit can interpret the machine language to determine which instruction is the instruction. The registry can participate in the temporary storage of data. The control unit 1120 may be generally configured as a one-chip. Therefore, the CPU can rapidly increase the processing speed of the CPU by copying the program stored in the DCNN-based image analysis unit 1130 of FIG. 11 at the beginning of operation of the image analysis apparatus 1100, loading the program into the memory, and executing the program .

12 is a flowchart illustrating a process of driving an image analysis apparatus according to an embodiment of the present invention.

11, the image analysis apparatus 1100 according to the embodiment of the present invention receives an image provided by a video providing apparatus (e.g., CCTV or the like), that is, a plurality of video frame images To a format for reducing the load of the image processing (S1200). Here, it is preferable to understand that the format is converted into a video image different from the received video image, rather than the first format. Accordingly, for this purpose, the image analysis apparatus 1100 preferably converts the resolution to a lower resolution than the input image, and to convert the image to be processed into a video image of RGB or G-scale pixel format as much as possible. This can be analyzed through CCTV images even in a general CPU environment so that the control personnel can perform the control.

Then, the image analyzing apparatus 1100 extracts object images of a plurality of object objects for motion tracking in a plurality of formatted video frame images, and applies the extracted object images to a DCNN using an object image recognizing method A plurality of tracking objects are classified based on the application result, and an event is detected based on the tracking information and the designation rule of the user interest object acquired by removing the user non-interest objects out of the plurality of tracking objects that fall within the specified reference (S1210 ).

For example, DCNN can store a sample image and use it to provide recognition results for the input object image. In addition, accumulation results are accumulated by learning to generate cumulative results, and final accumulation result classes are finally confirmed as tracking objects. For example, if the accumulation result for a human object is 2 points and the unknown result is 1 point, it is confirmed as a human object in the corresponding frame period.

Accordingly, for example, a specific object classified as an unidentified object is excluded from a plurality of tracking objects based on the object image of the object. Therefore, the bounding box formed on the trace object on the screen can be removed.

In this way, only the object of interest of the user is tracked, and the trace information detects an event that satisfies the specified rule.

In addition, various operations may be possible, but the details described above are supposed to be replaced.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them. In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. The codes and code segments constituting the computer program may be easily deduced by those skilled in the art. Such a computer program may be stored in a non-transitory computer readable medium readable by a computer, readable and executed by a computer, thereby implementing an embodiment of the present invention.

Here, the non-transitory readable recording medium is not a medium for storing data for a short time such as a register, a cache, a memory, etc., but means a medium which semi-permanently stores data and can be read by a device . Specifically, the above-described programs can be stored in non-volatile readable recording media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

While the invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention.

100, 900, 1100: Image analysis apparatus 101: Video frame acquisition unit
102: video frame sub-sampling unit 103: video frame scaling unit
104: Pixel format conversion unit 105:
106: Object tracking unit 107: Tracked object classification unit
108: object image recognition DCNN 109: non-interest tracking object removal
110: an interest tracking object-based event detection unit 121:
122: motion-based image analysis unit 901: video channel processing unit
902, 1003: object image recognition processing unit 1000: image analysis system
1001: image analysis server 1002: object image recognition server
1110: Communication interface unit 1120:
1130: DCNN image analysis unit 1140:

Claims (4)

The present invention relates to an image recognition DCNN-based CCTV image analysis apparatus for performing image analysis using a DCNN (Deep Convolutional Neural Network)
A method for receiving first video data composed of a plurality of first video frames having a first pixel format, a first resolution and a first frame rate from a video providing apparatus, 1 resolution, the first frame rate, and the first pixel format as a second resolution lower than the first resolution, a second frame rate lower than the first frame rate, and a format different from the first pixel format To a second image data of a second different pixel format having the second pixel format; And
Detecting and tracking a plurality of moving objects based on motion in a video frame of the converted second image data, extracting images of the plurality of objects being tracked, respectively, and outputting the extracted object images to the DCNN The method comprising the steps of: identifying the plurality of tracking objects by using a recognition result obtained by inputting the plurality of tracking objects; And a motion-based image analyzing unit for detecting an event based on the tracking information and the specifying rule of the user interest object obtained by removing the user interest object,
The image converter may include:
A video frame obtaining unit that receives the plurality of video frames input according to the first frame rate;
A video frame subsampling unit for sampling a number of video frames less than the input plurality of first video frames to generate a plurality of second video frames;
A video frame scaling unit for converting the plurality of generated second video frames into the second resolution; And
A pixel format conversion unit for converting the first pixel format of the plurality of second video frames converted into the second resolution into the second pixel format and providing the second image data to the motion-based image analysis unit; Lt; / RTI >
Wherein the motion-
A motion region detection unit for generating a difference image using the second image data and the learned background image, and detecting a motion region for the plurality of moving objects by removing noise from the generated difference image;
An object tracking unit for detecting and tracking the plurality of moving objects using the detected motion area and the second image data;
A first recognition result obtained by applying an object image of the plurality of tracking objects extracted from one of the plurality of second video frames to the DCNN, and a second recognition result obtained by applying the first recognition result obtained by applying the object image of the plurality of tracking objects extracted from one of the plurality of second video frames to the DCNN, A tracking object classifying unit that classifies the plurality of tracking objects based on a second recognition result obtained by applying an object image of the plurality of tracking objects extracted from at least one other video frame inputted at intervals to the DCNN;
An unattended tracking object removing unit that identifies user unattended objects out of the plurality of tracked objects and out of the plurality of tracking objects of the object tracking unit; And
And a tracking object-based event detector for detecting an event that the tracking information of the user interest object obtained according to the removal satisfies a specified rule
Including object image recognition DCNN based CCTV image analysis device. The method according to claim 1,
The tracking object classifier calculates and accumulates the first recognition result and the second recognition result as scores, and determines a class having the highest cumulative score as the type of the tracking object. The DCNN-based CCTV image analysis Device. 3. The method of claim 2,
Wherein the non-interest tracking object removing unit identifies a user non-interest tracking object that deviates from the specified reference in the determined type of object and removes the non-interest tracking object from the objects of the plurality of tracking objects. The method according to claim 1,
The image conversion unit converts the first pixel format having a pixel format of a luminance signal (Y), a difference (U) of a red signal and a difference (V) between a luminance signal and a blue component to RGB (Red-Green-Blue) DCNN-based CCTV image analysis apparatus for converting an object image into a second pixel format having a gray-scale pixel format.

Images