DE102023201938A1 - Method for detecting smoke in images of an environment and method for training a machine learning algorithm - Google Patents

Abstract

The invention relates to a method for detecting smoke in images of an environment, comprising: providing (220) a plurality of grayscale images (221, 222, 223) of the environment; removing (230) a background of the environment from the plurality of grayscale images to obtain a plurality of smoke images (241, 242, 243); and determining (240) an area (251) with smoke in the environment based on the plurality of smoke images.

Description

The present invention relates to a method for detecting smoke in images of an environment and a method for training a machine learning algorithm as well as a computing unit and a computer program for carrying out the same.

Background of the invention

Detecting fire or smoke in an environment is in many cases a safety-relevant task. For example, video-based systems can be used to detect smoke in images of the environment. EN 10 2014 219 829 A1 For example, a possibility is known to detect smoke based on a sequence of images of an environment.

Disclosure of the invention

According to the invention, a method for detecting smoke in images of an environment and a method for training a machine learning algorithm as well as a computing unit and a computer program for carrying them out are proposed with the features of the independent patent claims. Advantageous embodiments are the subject of the subclaims and the following description.

Both single-image-based methods and methods with temporal processing can be used to detect smoke in video data or images (which are contained in video data). Temporal information that is reflected in the images has proven to be advantageous in this regard. One way of detecting smoke can, for example, be based largely on motion estimation using optical flow. A sufficient smoke density can be important for successful motion estimation. The continuous movement of smoke can therefore be used, for example, as a central feature in the detection of smoke in images. Motion estimation is usually carried out by calculating the optical flow. This then detects the typically upward movement behavior of smoke. However, in order to detect movement using optical flow, the smoke must be sufficiently dense.

Further verification of potential smoke candidates, i.e. areas in images that potentially show smoke, can be done using color features. Here, too, certain requirements must usually be met for reliable detection.

Smoke can be understood in particular as a mixture of a solid phase and a gaseous phase. Smoke can also be understood as an aerosol produced by combustion in a finely divided form consisting of particles such as dust or ash particles and/or liquid droplets (water, oil vapors, acid vapors, liquid combustion residues), in particular contained in exhaust gases.

As has been shown, preprocessing data (i.e. the underlying images) can be an important step in the development of a machine learning-based algorithm. The aim is to retain information that is important for classification and, if necessary, to remove information that potentially has less characteristic features. The second aspect is particularly relevant when the data basis is small, as otherwise so-called overfitting often occurs. This is a phenomenon in which an algorithm makes decisions based on features in the training data that are not general features of the class to be recognized.

A method for preprocessing data presented in the present invention fulfills exactly this purpose by reducing the appearance of smoke in video sections (i.e. images) to characteristic information.

As mentioned, previous investigations of detection algorithms have mostly been limited to clearly visible smoke. In a typical application, the visibility of smoke is limited by the density of the smoke and various environmental influences, such as lighting. In addition, uniformly moving objects such as conveyor belts, cranes or lifting platforms can lead to false alarms if they meet certain color criteria. In addition, color information in the area of thin smoke is usually overlaid with color information from the background. This special property of smoke makes successful detection based on color features significantly more difficult.

Previous algorithms for visual smoke detection can therefore only be used with significant limitations. One aim of the present invention, however, is to significantly reduce the minimum requirements for successful detection and to be able to significantly increase reliability. In particular, pattern recognition of moving smoke edges is used for this purpose.

The present invention provides a way of making moving smoke edges visible using simple mathematical operations. and used to detect smoke. These can then be used as input for machine learning algorithms, especially artificial neural networks. Compared to the use of individual images, these proved to be much more suitable and delivered significantly better results in experiments on previously unseen data.

In particular, convolutional neural networks based on convolutions are one of the most promising approaches in automated image processing, as has been shown. Their functionality is based on the recognition of learned patterns. While clearly visible, dense smoke also shows typical patterns in individual images, thin, poorly lit smoke is difficult or almost impossible to recognize in individual images. However, it can typically still be recognized based on its characteristic movement. The patterns to be recognized in the area of very thin smoke are therefore typically primarily those of the background.

Within the scope of the present invention, it is now proposed that several grayscale images of the environment, i.e. the environment that is being monitored, for example, and in which potentially present smoke is to be detected, are provided. Typically, images (e.g. from a video) are obtained in color, so that, for example, several images, in particular color images, of the environment are initially provided, from which the several grayscale images can then be obtained by converting, in particular by reducing color information in the multiple images. If grayscale images are already provided directly, e.g. from a camera or surveillance camera, the conversion step can be omitted if necessary. Ultimately, stacked grayscale images or grayscale images are then available. The grayscale images can, for example, be those with 256 grayscale levels or gray values, but in principle other types of grayscale images can also be used, e.g. with more or fewer grayscale levels.

A background of the environment is then removed from the several grayscale images, several smoke images are obtained (a smoke image is thus created by removing the background from a grayscale image). Various methods can be used to extract or remove the background model. For example, a continuously updated background model would be conceivable, but a simple subtraction of a reference image, which has been obtained, for example, by averaging the several images or grayscale images, is also conceivable. For this purpose, see, for example, the EN 10 2014 219 829 A1 referred to.

Based on the multiple smoke images, an area with smoke in the surrounding area is then determined. As already mentioned, this is done particularly effectively using a machine learning algorithm. The machine learning algorithm receives the smoke images as input values and outputs the area with smoke as the output value. The machine learning algorithm includes, as also mentioned, in particular an artificial neural network, further in particular a convolutional neural network, which is particularly suitable for image analysis.

Preferably, determining the area with smoke in the environment based on the plurality of smoke images comprises first determining edges of smoke in the smoke images. Based on the edges of smoke, the area with smoke in the environment is then determined. As has been shown, the images always have similar characteristic patterns for the smoke. These patterns correspond to the moving edges of smoke.

In addition, this form of representation with the grayscale images, which can be obtained in particular by means of special preprocessing, as mentioned, has proven to be a particularly suitable input format for a deep learning-based algorithm (machine learning algorithm for detecting smoke). Emerging patterns of rising, spreading and partly turbulent movement of smoke have proven to be particularly characteristic for distinguishing smoke from other moving objects.

To train such a machine learning algorithm or artificial neural network, training data can be provided that include multiple smoke images of one or more different environments, wherein the multiple smoke images have each been obtained from multiple grayscale images of the environment after removing a background. Here, as mentioned, the aforementioned preprocessing (conversion) may again be necessary. The training data can in particular also include annotated smoke images or images, i.e. images for which it is noted whether or that smoke is present. The machine learning algorithm is then adapted based on the training data in such a way that the machine learning algorithm determines an area with smoke in the respective environment based on the respective multiple smoke images. This is therefore in particular a "supervised learning" approach. The training therefore refers to a process consisting of a fed forward step and a backpropagation step.

A computing unit according to the invention, e.g. a computer or a control unit of a surveillance camera, is set up, in particular in terms of programming, to carry out a method according to the invention.

The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, since this causes particularly low costs, in particular if an executing control device is also used for other tasks and is therefore already present. Finally, a machine-readable storage medium is provided with a computer program stored on it as described above. Suitable storage media or data carriers for providing the computer program are in particular magnetic, optical and electrical storage devices, such as hard disks, flash memories, EEPROMs, DVDs, etc. It is also possible to download a program via computer networks (Internet, intranet, etc.). Such a download can be wired or cable-based or wireless (e.g. via a WLAN network, a 3G, 4G, 5G or 6G connection, etc.).

Further advantages and embodiments of the invention emerge from the description and the accompanying drawings.

The invention is illustrated schematically in the drawing using an embodiment and is described below with reference to the drawing.

Short description of the drawings

• 1 shows schematically an environment in which a method according to the invention can be carried out.
• 2 shows schematically a sequence of a method according to the invention in a preferred embodiment.
• 3 shows schematically a sequence of a method according to the invention in a further preferred embodiment.

Embodiment(s) of the invention

In 1 an environment 100 is shown schematically in which a method according to the invention can be carried out. In the environment, two objects 110 and 112 are shown as examples, as well as smoke 114, which can develop in the environment 100, for example due to a fire.

Furthermore, a surveillance camera 120 with a control or computing unit 122 is shown as an example, by means of which the environment 100 can be monitored. In addition, a remote server or computer 130 is shown as an example (this can represent a so-called cloud, for example), which can be connected to the surveillance camera 120 or the control or computing unit 122 there in a data-transmitting manner, e.g. wirelessly (as indicated) or otherwise).

While monitoring the environment 100, the surveillance camera 120 generates video data, which in turn also includes individual images that are captured one after the other. By analyzing these images, potential smoke in the environment, as indicated by 114, can now be detected.

In 2 A sequence of a method according to the invention is shown schematically in a preferred embodiment. Such a method can run completely or partially, e.g. in the control or computing unit 122 of the surveillance camera 120 or in the server or computer 130 or divided between both.

In a step 200, several images 201, 202, 203, in particular color images, of the environment are initially provided. These may have been captured by the surveillance camera; in particular, these images may also have been extracted from video data from the surveillance camera. Although three images 201, 202, 203 are shown here as an example, more than three images or possibly just two images may be used.

In a step 210, these multiple images 201, 202, 203 are converted into multiple (corresponding) grayscale images 221, 222, 223; this can be done, for example, by reducing color information in the multiple images 201, 202, 203. These multiple grayscale images 221, 222, 223 are then provided in step 220.

In a step 230, a background 231 is then removed from each of the multiple grayscale images 221, 222, 223. Such a background (or a background image) can be obtained, for example, from the multiple images 201, 202, 203 or the multiple grayscale images 221, 222, 223, e.g. by averaging or the like. By removing the background, multiple smoke images 241, 242, 243 are obtained from the multiple grayscale images 221, 222, 223. In these smoke images 241, 242, 243, only the smoke is then contained or visible, since this does not belong to the background and therefore nothing has been removed.

In a step 250, an area 251 with smoke in the surrounding area is then determined based on the smoke images 241, 242, 243. For this purpose, for example, edges of smoke in the smoke images 241, 242, 243 can be recognized using pattern recognition and tracked across the smoke images. An edge is designated as 245 in smoke image 243 as an example. This means that the area 251 with smoke can be determined based on the edges of smoke. If an area with smoke has been determined - and smoke has thus been detected - an alarm can be triggered or another measure can be initiated.

As mentioned, it is particularly preferred to use a machine learning algorithm for determining the area with smoke in the environment, and particularly preferably a convolutional neural network. Such a machine learning algorithm is described in 2 shown by way of example and schematically and designated 248. The machine learning algorithm 248 receives the smoke images 241, 242, 243 as input values, recognizes the area with smoke based on this and outputs this area 251 as an output value.

Within the scope of the present invention, it is therefore possible to highlight smoke edge patterns, in particular with the help of special preprocessing - conversion into grayscale images. For example, several images taken at a time interval by a stationary video or surveillance camera are used as input for the conversion. Then, if necessary, the color information of the input images is reduced. This is followed, for example, by a simple subtraction of the background. It was particularly recognized that a targeted reduction of the input information (only grayscale images instead of color images) enables an improved detection result of neural networks on unseen data.

While based on RGB images (i.e. color images), for example, a detection rate of about 50% with a false detection rate of about 25% is achieved, with the proposed grayscale images a detection rate of about 75% with a false detection rate of only about 2% can be achieved.

In 3 is a schematic representation of a sequence of a method according to the invention in a further preferred embodiment, namely a training of a machine learning algorithm, as it can be used - especially after training - to detect smoke in images of an environment, as is the case, for example, with regard to 2 was explained in more detail.

For this purpose, training data 302 is first provided in step 300. The training data 302 comprises several smoke images of one or more different environments (i.e., if there are several environments, several smoke images should be used for each environment). The several smoke images are in turn each obtained from several grayscale images of the environment after removing a background, i.e., e.g., preprocessed as explained above.

These grayscale images can be in the same way as they are in relation to 2 explained and shown; however, it is understood that a significantly higher number of such grayscale images and in particular also of different environments is expedient for training. In addition, the training data 302 can comprise appropriately annotated images.

In a step 310, a machine learning algorithm 312 is then adapted based on the training data 302. This is done in such a way that the machine learning algorithm 312 determines an area with smoke in the respective environment based on the respective multiple smoke images. In this case, for example, weights or so-called neurons (or general parameters) of the machine learning algorithm are adapted, i.e. changed if necessary. In this way, a trained or machine learning algorithm 322 is obtained, which can be made available for further use in a step 320.

The training of the machine learning algorithm can be carried out on a suitable computer, e.g. on the server or computer 130 as in 1 shown.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• DE 102014219829 A1 [0002, 0014]

Claims (11)

A method for detecting smoke in images of an environment, comprising: providing (220) a plurality of grayscale images (221, 222, 223) of the environment; removing (230) a background of the environment from the plurality of grayscale images to obtain a plurality of smoke images (241, 242, 243); and determining (240) an area (251) with smoke (114) in the environment based on the plurality of smoke images. Procedure according to Claim 1 , further comprising: providing (200) a plurality of images (201, 202, 203), in particular color images, of the environment; and converting (210) the plurality of images into the plurality of grayscale images, in particular by reducing color information in the plurality of images, in order to provide the plurality of grayscale images. Procedure according to Claim 1 or 2 , wherein determining (240) the area with smoke in the environment based on the plurality of smoke images is performed using a machine learning algorithm (248), wherein the machine learning algorithm receives the smoke images as input values and outputs the area with smoke as an output value. Procedure according to Claim 3 , wherein the machine learning algorithm (248) comprises an artificial neural network, in particular a convolutional neural network. Procedure according to one of the Claims 3 or 4 , wherein the machine learning algorithm (248) is trained with and adapted to training data (302) comprising a plurality of smoke images each of one or more different environments, the plurality of smoke images each being obtained from a plurality of grayscale images of the environment after removal of a background. The method of any preceding claim, wherein determining the area of smoke in the environment based on the plurality of smoke images comprises: determining edges (245) of smoke in the smoke images; and determining the area of smoke in the environment based on the edges of smoke. Method according to one of the preceding claims, wherein the plurality of grayscale images of the environment correspond to images captured at different times, in particular according to a predetermined time schedule. A method for training a machine learning algorithm (312) for detecting smoke in images of an environment, comprising: providing (300) training data (302) comprising a plurality of smoke images each of one or more different environments, wherein the plurality of smoke images each have been obtained from a plurality of grayscale images of the environment after removal of a background; adapting (310) the machine learning algorithm (312) based on the training data (302) such that the machine learning algorithm determines an area with smoke in the respective environment based on the respective plurality of smoke images; and providing (320) the trained machine learning algorithm (322). Computing unit (122, 130) which is configured to carry out all method steps of a method according to one of the preceding claims. Computer program which causes a computing unit (122, 130) to carry out all method steps of a method according to one of the Claims 1 until 8 when executed on the computing unit (122, 130). Machine-readable storage medium with a computer program stored thereon in accordance with Claim 10 .

Images