DE102011055343A1 - Method for detection of rain from image of camera that is partially focused on disk of vehicle, involves evaluating span width and standard deviation of intensity values inside window with adjacent pixels for determination of edge increase - Google Patents

Abstract

The invention relates to a method for detecting rain from image data of a camera which is at least partially focused on a pane of a vehicle. In particular, the camera can be arranged behind a windshield in the vehicle and focused on it. At least one edge is detected in the image data of the camera focused on the window of the vehicle, e.g. as intensity or color transition of adjacent pixels or pixels. A measure of the slope of the at least one edge is determined, preferably a gradient of the intensity values.

Description

The invention relates to a method for detecting raindrops on a pane by means of a camera.

In the WO2010 / 072198 A1 Rain detection is described with the help of a camera, which is used for automotive driver assistance functions. For rain detection, a bifocal optics is used, which images a portion of the windshield sharply on a portion of the image sensor of the camera.

Edge recognition algorithms are often used to detect raindrops on a pane of image data from a camera. Conventional edge detection algorithms require comparatively little computing power.

One difficulty in detecting rain from edges in the image is that edges are detected in the image, although there is no rain on a dry disk because there may still be edges in the image that have a different origin. This can lead to a false triggering of an automatic windscreen wiper control.

One way to optimize the rain detection is to use complex learning methods for an object recognition from the image data, which can recognize as reliable as possible after a learning phase, whether rain is on the disc or not. A disadvantage of this method is that the need for computing power is very high, which makes the required hardware equipment expensive.

The object of the present invention is to overcome the mentioned disadvantages of the processes known from the prior art.

This object is achieved by a method for detecting rain from image data of a camera which is at least partially focused on a pane of a vehicle. In particular, the camera can be arranged behind a windshield in the vehicle and focused on it. At least one edge is detected in the image data of the camera focused on the window of the vehicle, e.g. as intensity or color transition of adjacent pixels or pixels. In addition, a measure for the rise of the at least one edge is determined, for example a spatial gradient of the intensity values. In the following, only the term intensity value will be used, and it shall at the same time also include color values (for example R-G-B or C-M-Y-K) which may likewise be regarded as individual color intensity values.

Thus, the solution uses edge detection, which, however, is improved over known edge detection algorithms (Canny, Sobel, etc.). Within an area of evaluation or a region-of-interest (ROI) of the image from which rain is to be detected, it is preferable to determine locations at which the probability of detecting rain is high in comparison to adjacent image areas.

A basic idea of the invention is to examine edges in the image for the steepness of the intensity increase. Edges with a steep increase in intensity can cause near-end objects, i. especially raindrops on the disk, and are referred to as relevant edges. Edges with a flat increase in intensity can be assigned to distant objects located in front of the disk and are irrelevant to rain detection in the present invention.

Preferably, an edge with a steep slope is classified as part of a potential raindrop on the disk imaged in the image data. By this is meant that when classifying edge points or objects grouped from edge points, edges with a steep slope are more likely to be classified as raindrops - and thus are relevant edges - than edges having a shallow slope.

According to an advantageous embodiment, to determine the slope of an edge at a pixel position, the span and standard deviation of intensity values within a window will be evaluated with adjacent pixels, with the pixel position in the center of the window. As a window, advantageously, an n × n pixel-sized window can be used, so that the span and the standard deviation of intensity values of the current pixel position and the n ² -1 adjacent pixels are evaluated. For example, the intensity values for a window of 3 × 3 pixels are evaluated, with the currently considered pixel lying in the middle of this window. The span indicates the difference between the maximum and minimum values of these nine individual intensity values. Even windows with n = 5, 7, ... are conceivable.

Advantageously, during the daytime, the average intensity of pixels of the entire evaluation area is additionally evaluated and taken into account. For example, the intensity of the edge pixel and / or the adjacent pixels may be compared to the average intensity of pixels in the evaluation area of the image. During the day raindrops appear darker in the picture on the glass, therefore, a below-average intensity indicates a raindrop.

At night, lighting of at least the rain sensor area of the pane is preferably activated. All strong edges can be classified as relevant at night and further considered for rain detection.

In an advantageous embodiment, object formation takes place from a grouping of the relevant edges. Starting from a first determined (and thus valid) edge pixel, a search is then made in all eight adjacent directions for further valid edge pixels. Found contiguous edge pixels are connected to an object.

A preferred embodiment of the invention provides to analyze an associated rectangular location of each formed object in the image to determine from the number of (valid) edge pixels, aspect ratio of the image detail and / or percentage of saturated pixels, whether the formed object is a candidate for a raindrop or not. Saturated pixels are pixels with a very high or maximum intensity value, e.g. 80 or 90 percent of the maximum intensity value.

Raindrops have different shapes and sizes after they attach to the disc. Small raindrops can be almost transparent in a specific weather situation. In order to detect rain in this situation, it is advantageously possible to adapt a threshold value which is used to extract valid edge pixels, depending on the range of the intensity values of the pixels in the evaluation region (the ROI) and / or on the exposure time of the camera.

In certain cases, the information obtained from the ROI may not be sufficient to distinguish sun-reflections from light raindrops from the image data. This can be remedied preferably by using a second evaluation area to detect the sun in an upper part of the image. A circular Hough transformation can be used to detect the circular solar disk in the image.

The phenomenon of a "surge" occurs when, within a short time, most of the disk is covered with water. This poses a dangerous situation because the driver practically can not see through the window anymore. In order to detect a water surge, the difference of the total number of strong edges between two consecutive images is preferably calculated. In the picture in front of the waterfall edges of individual drops are still visible, while in the picture, in which the water splash first occurs, many of these edges disappear, since a large part of the disc is covered with water surface. The absolute number of relevant edges and the relative change are compared with predetermined thresholds to detect the water surge. Upon detection of a water surge, for example, a signal to the windshield wiper control can be output, so that the disc is wiped at maximum speed.

The invention offers several advantages:
Robustness: Rain detection is based on an analysis of objects on the disk, but also environmental information (sun, shadow, etc.) can be taken into account.

Universality: The method works with any camera that focuses on at least a portion of a vehicle window focused.

Cost savings: The process is not particularly computationally intensive, the rain sensor functionality can be easily integrated into a multifunctional driver assistance camera, the rain detection is more reliable than conventional rain sensors using diodes.

In the following the invention will be explained in more detail with reference to figures and exemplary embodiments.

1 shows a 3 × 3 query window.

2 shows 4 × 5 pixels of a picture detail.

3 shows how the query window can be superimposed over the image detail to query the span r and the standard deviation s of a particular pixel.

The described embodiment comprises several steps for image evaluation:

1. Calculation of the gradient size ("edge strength")

For rain detection, the camera is focused on a near area. Since the disc is transparent, objects in the background or in the far field (such as road, pedestrian, vehicle ahead) are visible, but the edges of these objects are blurred (due to the lack of depth of field of the camera). A selected statistical method is used to calculate the gradient size (see "Fuzzy Models and Algorithms for Pattern Recognition and Image Processing", authors: James C. Bezdek, James Keller, Raghu Krisnapuram, Nikhil Pal, pp. 566-567 ).

The range r and the standard deviation s of pixel intensity values X _i from an n × n kernel (or query window) are, as already explained, relevant quantities for the rise of edges.

1 shows by way of example a 3 × 3 query window, which can be imagined laid over the pixels p _{kl of} the image of the camera. The nine window elements are numbered from w ₁ to w ₉ and the window is used to query the gradient values for the central window element w ₅ (shown in bold font in FIG 1 ). The size of a window element corresponds to the size of an image pixel p _kl .

2 shows a section of the image comprising 20 individual pixels p ₁₁ to p ₄₅ . Each individual pixel is characterized by an intensity value X ₁₁ to X ₄₅ .

3 shows, as for the pixel p ₂₃ off 2 Span r and standard deviation s are determined. The window (thick lines) off 1 is via the pixels p ₁₂ to p _{34 of} the image section (thin lines) off 2 so that the central window element w ₅ lies on the pixel p ₂₃ to be interrogated. For this window w _j , the intensity values I = {X ₁ , X ₂ , X ₃ , X ₄ , X ₅ , X ₆ , X ₇ , X ₈ , X ₉ } correspond to the intensity values {X ₁₂ , X ₁₃ , X ₁₄ , X ₂₂ , X ₂₃ , X ₂₄ , X ₃₂ , X ₃₃ , X ₃₄ } of pixels p ₁₂ to p ₃₄ 2 ,

For a 3 × 3 window, the functions f _r , f _s for calculating the span r and the standard deviation s are given by the following formulas: f _rs (w _j ) = (f _r (w _j ), f _s (w _j ))

The window w _j can be moved in a corresponding manner to query further image pixels p _kl over the image section or the image, whereby the values X _i for the equations given change as a rule.

The feature extractors f _r and f _s thus provide two values for each pixel position, one for the local span and one for the local standard deviation. These values can now be compared with one threshold each. This then results in a statement as to whether a pixel position is assigned to a relevant edge or not. In addition, it is checked during the day whether the pixel or the pixel and its neighboring pixels averaged have a lower intensity than the average of all pixel intensities in the evaluation area. If a pixel fulfills these criteria (steep edge, possibly daytime: pixel (region) dark), it is referred to as a valid pixel, since it can be assumed so far that it is part of a raindrop in the image.

Edges with a steep rise correspond to objects in the vicinity, ie objects in the area of the windshield. Ridges with a shallow rise correspond to objects from the far field or from the background of the windshield. This comparatively simple edge detection method serves to distinguish strong edges from soft edges.

2. Filter edges

Sun or other bright light reflections in the region-of-interest are recognized as possible near-field objects in the calculation of the gradient size due to their pronounced edge strength. During the day, therefore, a further calculation is preferably made for rain detection. A modified averaging segmentation of the ROI is applied: a relevant pixel edge should have a high gradient during the day, and the intensity of the pixel or neighboring pixels should be below the mean ROI intensity. The pixel or "pixel area" should be darker than the average ROI intensity. Strong edges caused by interfering signals can be eliminated by morphological methods: erosion with subsequent dilatation.

For nocturnal rain detection, an infrared illumination of the windshield surface is activated, which is focused on rain detection by the camera. At night, all strong edges can be used as relevant edges. The segmentation by averaging described above is not required here.

3. Object formation / recognition by grouping edges

The valid edge pixels are grouped into objects by means of a method called "object membership map" (OMM).

Each valid pixel is expanded in eight directions. If one of the neighboring pixels is also a valid pixel, this pixel is assigned the same object index as the (relevant) output pixel. In this way, all connected valid pixels of relevant edges are assigned the index of the object to which they correspond. This process is repeated, with the result that groups of connected valid pixels are obtained, each of which is populated with an object index.

4. Filtering objects

Each object is now bounded in the image by a rectangle. The data of the rectangles (occurrences in the picture) are calculated during the evaluation of the OMM matrix. Every rectangular location in the image is evaluated to eliminate false identifications. Irrelevant occurrences in the image have a specific aspect ratio, percentage of saturated pixel intensities (near the maximum value), number of valid edge pixels. One or more of these criteria can be used to distinguish relevant from irrelevant sites in the image. Irrelevant occurrences contain a very small number of relevant edges (or valid pixels) and can be regarded as noise or false signal. The relevant references, on the other hand, include candidates for objects for rain detection. For clarification, irrelevant occurrences could be shown in the picture with a red frame, relevant with a green frame.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• WO 2010/072198 A1 [0002]

Cited non-patent literature

• "Fuzzy Models and Algorithms for Pattern Recognition and Image Processing", authors: James C. Bezdek, James Keller, Raghu Krisnapuram, Nikhil Pal, pp. 566-567 [0027]

Claims (10)

A method for detecting rain from at least one image of a camera which is at least partially focused on a pane of a vehicle, wherein at least one edge is detected in an evaluation region of the camera image and a measure of the increase of the at least one edge is determined. The method of claim 1, wherein an edge with a steep slope is classified as part of a potential raindrop on the disk imaged in the evaluation region. The method of claim 1 or 2, wherein to determine the slope of an edge at a pixel position (p _kl ), the span (r) and the standard deviation (s) of intensity values (X _i ) within a window (w _j ) are evaluated with adjacent pixels , wherein the pixel position (p _kl ) lies in the center of the window (w _j ). Method according to one of the preceding claims, wherein during the day additionally the average intensity of pixels in the entire evaluation range of the image is evaluated. The method of claim 4, wherein the intensity of the edge pixel and / or the adjacent pixels is compared to the average intensity of pixels in the evaluation area of the image. Method according to one of the preceding claims, wherein object formation takes place from a grouping of the relevant edges by searching for further valid edge pixels starting from a first valid edge pixel in all eight adjacent directions. Method according to one of the preceding claims, wherein a rectangular location in the image of each formed object is analyzed to determine from number of edge pixels, aspect ratio of the image detail and / or percentage of saturated pixels whether the object formed represents a candidate for a raindrop or Not. Method according to one of the preceding claims, wherein a threshold value which is used to determine a valid edge pixel is adapted as a function of the span (r) of the intensity values of the pixels in the entire evaluation area and / or of the exposure time of the camera. Method according to one of the preceding claims, wherein in an upper image area a detection of the sun is performed in the image data. Method according to one of the preceding claims, wherein the camera takes at least two consecutive images and the difference of the total number of strong edges between two successive images is calculated to detect a water surge impinging on the pane.

Images