RS20130031A1 - SMART CAMERA FOR TRAFFIC MONITORING AND ANALYSIS - Google Patents

Abstract

Invention herewith described refers to the smart camera for traffic surveillance and analysis. The basic idea of this invention is the development of the procedure and the system in which the collected data from video cameras in the system for traffic surveillance are automatically processed locally on the very surveillance location i.e. in the camera to the semantic level (e.g. detected traffic congestion, crash, etc.). Only the metadata, information of high level of abstraction and the relevant data from the cameras (e.g. several selected pictures or short video clips) are sent to the monitoring centre, significantly reducing the required bandwidth for communication. It is possible to send data wirelessly, and data can be stored locally on the memory card, in case of any problems in communicating with the monitoring centre.The smart camera consists of the visual sensor with the colour acquisition - VS (CCD or CMOS chip with associated electronics that allows image acquisition), which is by “usb” or “Ethernet” interface connected to the PC unit - RJ, that is responsible for the intelligent processing of data from the camera using specialized mathematical algorithms as well as data preparation for sending via wireless communication interface - GPRS which was implemented using the GPRS modem.

Description

Smart camera for traffic monitoring and analysis

(Smart Camera for Traffic Surveillance arrdTamaty»is)

Technical field

The invention belongs to the field of applied information technologies and telecommunications. Specifically, it refers to: processing of the video signal, received from the video surveillance camera, with different algorithms in a portable computer, of minimal dimensions, based on an ARM processor and communication between the monitored location and the monitoring center through a connection established by the GPRS service of the mobile network.

Technical problem:

The technical problem that is solved by the described invention consists in the following: how to improve the system of detecting traffic events through video surveillance, enable greater reliability and robustness of the system with a significant reduction in costs and simpler organization of the system, which at the same time allows greater flexibility when upgrading existing or building new traffic surveillance systems?

State of the art:

The existing state of the art involves the use of video surveillance cameras for traffic surveillance. In the current state of the art, the dominant solution in such systems is to send the complete video stream from surveillance cameras to surveillance centers. This creates high costs due to the lease of expensive telecommunication links and makes it difficult to build such systems because quality telecommunication services and links are often unavailable in remote locations. The large amount of data arriving at the surveillance centers creates a heavy load on processing computers that must simultaneously process video signals from a large number of cameras. In order to automatically process video streams from a large number of video surveillance cameras, a lot of processing power is required. The acquisition of such equipment is very expensive, but it is also necessary to provide special conditions for the said equipment: accommodation space, cooling system, fire protection system and others, which additionally increases the costs of the system. The solution to all these problems is found in a distributed system where data processing is performed locally at the monitoring locations themselves, before sending information of interest to the monitoring center. The development of such distributed systems was made possible only recently by the development of telecommunications and portable computers. The processing power of notebook computers has grown rapidly over the past 3 to 4 years while simultaneously shrinking in size. Thus, mobile platforms are available today whose processing power corresponds to the processing power of personal computers from a few years ago. Such mobile platforms have become an integral part of mobile phones, tablets and netbook computers. They are based mainly on ARM and Intel Atom processors with several gigabytes of operating memory and the ability to run various modern operating systems such as Linux, Android or iOS. These mobile platforms also have the possibility of upgrading with various external modules, especially in order to communicate with other devices or public networks. Thus there are modules for RS-232/RS-422, LAN, Wi-fi or GSM communication. All the mentioned features allow to create complex, distributed networks of data processing devices using the described mobile platforms. Such devices can be the "brain" in systems for automatic control and monitoring of various processes from everyday life. They can also be used to manage various automated systems located at remote locations.

In a modern traffic monitoring system, it is necessary to collect and process information from remote locations such as roads, intersections, traffic loops. In the monitoring center, the operator must have at his disposal as much information as possible in order to quickly and efficiently make the correct conclusions about the traffic situation at the monitored remote location, as well as decisions about the measures to be taken in order to manage traffic and eliminate problematic situations. As video traffic surveillance is expanding, video cameras are being installed in a variety of environments: along open roads and highways, on bridges, in tunnels, at intersections, and throughout cities. The current state of the art implies that data from these cameras are delivered to monitoring centers in "raw" format or with different degrees of compression, which requires a huge network flow, which on the other hand significantly increases the number of communication errors and delays in data delivery. The communication between the cameras and the monitoring center is further complicated by the fact that there is often a need to place the cameras in locations that are very far from the existing telecommunications infrastructure. Setting up such an infrastructure creates high costs for the user and complicates system maintenance. On the other hand, when the appropriate infrastructure exists, considering the flow of large amounts of data, the cost of leasing and maintaining telecommunication links is also high. Another technical problem that exists in practice is the centralization of data collection and processing in the monitoring center. This means that computer equipment is necessary for processing large amounts of data, ie. video material located in one location. This approach creates high costs for the acquisition of equipment for surveillance centers and major problems due to frequent or occasional interruptions in communication with remote video cameras because it is impossible to store large amounts of data locally. When the communication is interrupted, in the current state of the art, between remote video cameras and the surveillance center, there is a complete loss of data because their processing is done in the surveillance center.

The described shortcomings of the existing state of the art can be grouped into two categories:

1. all data from video cameras are continuously collected in the monitoring center (communication problem), 2. intelligent data processing, for system automation, is done on servers in the monitoring center (centralization problem),

Communication problems occur due to the large flow of data between the surveillance location (where the video cameras are installed) and the surveillance center. Large data flows over limited bandwidth result in communication errors, data delays, and high costs of leasing or installing appropriate telecommunication links. Also, a system that requires large bandwidths cannot be easily expanded and upgraded. On the other hand, constant communication between remote monitoring locations and the monitoring center has the disadvantage that large amounts of data are lost due to the interruption of such communication. This requires very quick interventions by maintenance teams to repair telecommunication links, which further increases system costs. The problem of centralization is a direct consequence of the limited processing power of computers at the time when traffic control systems were installed. Until a few years ago, it was necessary to use PCs or servers to solve any demanding computational tasks. Intelligent algorithms for image or video stream analysis are very demanding tasks, and only recently have mobile platforms powerful enough to successfully execute them in real time emerged. The installation of a large number of computers and servers is technically complicated and requires large investments, not only in computer equipment but also in rack cabinets, "data" centers or rooms protected by access control, a cooling system, fire detection and extinguishing, and autonomous power supply systems. As a consequence of the aforementioned problems of communication and centralization, the problem of precision also arises because errors in the detection of events of interest may occur due to errors in communication and the workload of computers that process data. This increases the number of false alarms (false positive events) and the number of undetected events (false negative events). A higher number of false alarms or undetected events reduces user confidence in the system. Reduced trust leads to operators ignoring notifications and alarms and ultimately making the system unusable for its primary purpose. In this way, instead of increasing the operator's efficiency and vigilance, which should be the result of releasing the operator from the obligation to continuously monitor the image from the cameras 24 hours a day, automation gives the completely opposite result.

Presentation of the essence of the invention:

The basic idea of the described invention is the development of a procedure and system in which the data collected from the video cameras in the traffic monitoring system is processed locally at the monitoring location itself, i.e. in the camera itself to the semantic level (eg traffic congestion, collision, etc. was detected). Only metadata, high-level information and relevant data from the cameras themselves (eg a few selected images or short video clips) are sent to the monitoring center, significantly reducing the required bandwidth for communication. The current challenge and the goal of this invention is to process sensor data automatically and to deliver only relevant information to the operator in the monitoring center in real time so that he can make timely decisions. Decisions can be e.g. managing traffic signals or alerting maintenance or emergency services.

The basic elements of a smart camera are (Figure 1): a visual sensor for image acquisition (VS), a computer unit for intelligent data processing and data preparation for sending to the monitoring center (RJ) and a GPRS modem responsible for wireless communication with the monitoring center via a GPRS connection (GPRS). A CCD chip with a resolution of up to 640x480 pixels is used as a visual sensor (VS), which provides a color image during the day, and a black-and-white image during the night, and supports sending a video stream in MJPEG compression format. A 10/100 Mbps ethernet interface is used to connect the visual sensor (VS) to the computer unit (RJ). The computing unit (RJ) is a microprocessor platform, based on the ARM microprocessor. This platform is an integral part of mobile devices of the last generation (mobile phones and computers) and has enough processing power to work with images, sound and video.

The detailed specifications of the computing unit (RJ) are:

• PC performance with very low power consumption,

• it is possible to choose one of 4 performance levels and additionally optimize consumption,

• 1,200 Dhrvstone MIPS (Million Instructions Per Second - measure of software execution speed) using an ARM Cortex-A8 processor with 256KB L2 cache and a clock of up to 600MHz, • OpenGL© ES 2.0 with support for a 2D/3D graphics accelerator capable of rendering 10 million polygons per second,

• TMS320C64x+ DSP specialized for signal processing with a clock of up to 430MHz,

• initial capacity of 128 MB LPDDR and 256 MB Nand Flash memory,

• the possibility of using SD memory cards as data storage space.

The computing unit (RJ) runs the Linux operating system and supports standard C and C++ libraries for programming various applications. Through the USB connection, the computer unit (RJ) is connected to a GPRS modem (GPRS), through which communication with the monitoring center takes place. The GPRS modem (GPRS) used in the system is an external type, powered by 7-30 V, with robust physical characteristics. It is a "quad band" device, which means that it can use the most popular GSM bands of 850/900/1800/1900 MHz. It supports the GPRS class 10 standard, has an integrated SIM card reader, which simplifies the commissioning of the device as much as possible, and has an integrated TCP/IP protocol stack.

The computer unit (RJ) performs a special procedure for analyzing the video stream from the visual sensor and preparing for sending the extracted information organized in three modules: 1) module for extracting the image from the video stream from the visual sensor, 2) module for intelligent image analysis and detection of various traffic events of interest, 3) module for preparing and sending metadata, information of a high level of abstraction and a collection of images or videos to the monitoring center. All modules run in parallel, in separate program threads that are synchronized using semaphores and mutexes. The module for extracting the image from the video stream realizes the first step in the execution of the procedure on the computer unit (RJ). This module performs the process of parsing (searching and analyzing) the video stream that the computer unit (RJ) receives via HTTP communication via the ethernet interface. The visual sensor sends a video stream in binary form, which is organized as a series of JPEG images (the so-called MJPEG video format). Since the JPEG format is well standardized, the module for extracting an image from a video stream works as a reader and parser of an MJPEG video stream. As each JPEG image begins and ends with a specific group of characters (so-called JPEG markers, Table 1), this module performs image extraction by detecting these markers, which can be performed in any type of MJPEG video stream, regardless of the manufacturer of the visual sensor.

The image extraction module itself was implemented using the "cURL" library, which enables very simple processing of HTTP requests and responses ("Request - Response" communications). This module forwards the extracted image to the intelligent image analysis module.

The module for intelligent image analysis and detection of various traffic events of interest receives individual images arriving from the image extraction module from the video stream and performs their processing and analysis. The analysis procedure was implemented using the "OpenCV" library. The analysis procedure itself (Figure 2) contains four important elements: extraction of objects of interest (vehicles) from a static scene (segmentation), analysis of optical flow in the image, tracking of objects of interest and detection of traffic events.

The extraction of objects of interest (segmentation) is based on the difference of the current frame from the background reference image. The difference image is then morphed with a median filter (Median Filter), which results in complete objects. The Approximate median method (AMM) is used to create and refresh the reference background. AMM reduces (or increases) the value of the background image in places where the current frame differs from the background by a certain amount. This creates a background image that is used for the next frame. To track objects, their centroids are used and the position of the object in the image and its speed of movement are taken into account. During the research, object acceleration was also used for state prediction, but it was shown that the system with position and velocity works accurately enough, as well as being more resistant to errors due to imperfect segmentation and selection of tracking features. The position prediction is used in the next frame, where the nearest neighbor method (NN) is used to associate objects from the current frame with objects from the previous one. The cases of joining and separating objects (Split&Merge) were also taken into account. In parallel with the segmentation process, an analysis of the optical flow in the image is performed. The information obtained in this way is important for determining the direction and direction of movement of moving objects (different types of vehicles). Optical flow analysis is performed using a method that minimizes the sum of squares of intensity differences between two consecutive images. Optical flow calculation is used for object tracking correction as well as event detection.

After analyzing the images using segmentation procedures, optical flow calculation and object tracking, the system has the following information: the total number of detected vehicles, data about each vehicle in the image (center, area, number of the image in which it was last detected, number of images in which the specified vehicle was detected, the trajectory from the first image to the last image in which the vehicle was detected and associated optical flow vectors indicating the direction of the vehicle). This information together with information from the scene model is used to detect various traffic events. The scene model is set manually when placing the smart camera at the location to be monitored and involves entering basic information about the monitored location (vehicle movement directions, traffic lanes, pedestrian crossings, prohibited parking and vehicle retention zones). The trajectory of the vehicle is compared to the set traffic lanes and the permitted directions of vehicle movement from the scene model and is thus used for the detection of traffic violations such as: driving in the opposite direction from the permitted and illegal turning of the vehicle. Vehicle counting, for the purpose of statistical analysis or detection of traffic jams, is done using a virtual line that is entered into the scene model. Each crossing of the virtual line by the vehicle is counted, and the direction of the vehicle is detected based on the trajectory and associated optical flow vectors. Information about the vehicle's trajectory and the associated optical flow vectors are also used for the detection of stopping or stopping the vehicle in prohibited zones. For the detection of traffic accidents, the information obtained from the associated optical flow vectors is used. The slope of each associated vector is determined and a histogram is created. During the normal movement of the vehicle, the mode of the histogram is equal to the slope that determines the direction of movement of the vehicle for the observed traffic lane, which is defined in the scene model. This means that the vehicle direction vectors from the scene model and most of the associated optical flow vectors for the observed vehicle are collinear. In case of abnormal vehicle movement

(traffic accident, movements normal to the traffic lanes) associated vectors of the optical flow of the observed vehicle are completely random and consequently differ significantly from the vector of the direction of movement of the vehicle from the scene model (the vectors in this case are not collinear). Information about all detected traffic events is recorded locally and sent to a remote monitoring location, in this way the system is protected from occasional interruptions in communication or from errors in the transmission of a large amount of data, because in this case only the most important information is transmitted, which solves the problems of communication and centralization.

The module for preparing and sending data takes advantage of the HTTP protocol and GPRS technology. The GPRS modem (GPRS) works as an Internet "gateway" through which metadata, information of a high level of abstraction and collections of images or videos are sent to the monitoring center via the HTTP method. First of all, it is necessary to set up the GPRS modem (GPRS), so that an HTTP connection can be made through it. In the "Linux" environment, the modem is set up through script applications, and the parameters used are shown in Table 2. After this setting, it is possible to access the Internet through the modem, so the implementation of the communication client is reduced to an HTTP request for file or data transfer. HTTP methods are used for communication: PUT, POST or GET.

A special application implemented as a Java servlet, which accepts and processes HTTP requests as part of the "Apache Tomcat" web server, is in charge of receiving data from remote locations in the monitoring center. The web server solves the problems of concurrent access and allows simultaneous connection of up to 100 remote sites to the server in the monitoring center. Also, the web server ensures the reliability of system operation and the security of data sent from remote locations.

Inventors:

Claims (4)

1. A device for traffic monitoring and analysis at locations along different types of roads in urban and suburban areas, as a device that performs automatic extraction of data and events of interest from the video stream from the video surveillance camera, their preparation and sending to remote monitoring centers via the HTTP protocol, using the services of the mobile phone network, characterized by fully automatic and autonomous local processing of the video stream from the video surveillance camera by extracting individual JPEG images, and then applying segmentation, optical flow analysis and tracking of moving objects over data and events of interest are separated by separated images, on which analysis and selection of information of interest is then carried out, which is sent to a remote monitoring center via a GPRS connection. 2. The process of extracting individual images from the MJPEG video stream received from the visual sensor, indicated by detecting the beginning and end markers of JPEG images in the binary data stream and thereby extracting successive images received from the visual sensor. 3. The procedure of analyzing images received from the visual sensor for the detection of various traffic events, indicated by the separation of moving objects based on the calculation of the difference between successive images from the video stream, tracking of separated moving objects and logical analysis of events. 4. The procedure for detecting traffic accidents and other extraordinary traffic events based on the analysis of the optical flow vector, indicated by creating a histogram of the optical flow vector for each object of interest and comparing its mode with the vector of the normal and expected direction of traffic defined in the scene model that is set manually by the installer when installing the described traffic monitoring and analysis system.