DE102023003206A1 - Method and device for detecting a vehicle environment - Google Patents

Abstract

The invention relates to a method for recording a vehicle environment, wherein event images (B1 to Bn) of the vehicle environment are captured using at least one event camera (3.1 to 3.n). According to the invention, in order to detect a shelling of the vehicle (1) with a projectile, event information (I) aggregated to the captured event images (B1 to Bn) is evaluated by means of a control device (4) of the vehicle (1) and/or by means of a pulsed neural network (5) and environmental suppression. The invention further relates to a device (2) for detecting a vehicle environment.

Description

The invention relates to a method for detecting a vehicle environment according to the preamble of claim 1.

The invention further relates to a device for detecting a vehicle environment according to the preamble of claim 3.

From the GB2605675A a method and a device for detecting air objects are known, in which event images, also referred to as event images, of a scene are captured by means of an event camera, also referred to as an event camera, arranged on a vehicle. Events are detected within the event images and properties of the detected events are calculated to detect possible air objects and calculate their physical properties.

Furthermore, from the EP 3 293 681 A1 a method for temporal encoding of information using a pulsed neural network (English: Spiking Neural Network, short: SNN) in neuromorphic hardware is known.

As described, among other things, in https://en.wikipedia.org/wiki/Event_camera#Algorithms (accessed on July 14, 2023 at 9:20 a.m.), an event camera is also called a neuromorphic camera, silicon Retina (English: silicon retina) or dynamic image sensor (e.g. Dynamic Vision Sensor, DVS for short) or Asynchronous Time-based Image Sensor, ATIS for short. Such an event camera includes an image sensor that reacts to local changes in brightness. Event cameras do not capture images with a shutter like frame-based cameras. Instead, event camera pixels are asynchronous and independent. They generally remain silent, but react independently to changes in brightness in the area around the event camera. To do this, each pixel saves a reference brightness and compares it with a new brightness level. When a brightness difference exceeds a threshold, that pixel sets its reference level to the new brightness value and generates an event, called an event. Event cameras therefore process local relative light intensity changes directly and send information packets as events to control devices connected to them. Here these can be aggregated in time and/or location to form so-called event images. In contrast to frame-based cameras, event cameras typically only provide binary information about relative light intensity changes, with optional information about a direction of change, i.e. an increase or decrease in the relative light intensity changes, as so-called polarity, but tend to have a higher polarity than frame-based cameras temporal resolution and a typically higher dynamic range. This allows, for example, object movements in their surroundings to be tracked very precisely.

The invention is based on the object of specifying a novel method and a novel device for detecting a vehicle environment.

The object is achieved according to the invention by a method which has the features specified in claim 1, and by a device which has the features specified in claim 3.

Advantageous embodiments of the invention are the subject of the subclaims.

In a method for capturing a vehicle environment, event images of the vehicle environment are captured using at least one event camera. According to the invention, in order to detect a bombardment of the vehicle with a projectile, an evaluation of event information aggregated to the captured event images is carried out by means of a control device of the vehicle and/or by means of a pulsed neural network, also referred to as a spiking neural network, and an environmental suppression, also referred to as Surround Supression is carried out.

There are vehicles that are designed to withstand fire from an assault rifle with appropriate hard-core steel ammunition in such a way that both the body and window panes can withstand such fire. For example, these vehicles meet the requirements of the highest civil protection class VR10 according to the VPAM BRV version 3 (Bullet Resistant Vehicles) guideline.

Such vehicles may require additional protective measures, such as: E.g. escape maneuvers etc., can only be taken by a driver or user of the vehicle as soon as such fire, for example due to the impact of a projectile on the body or window, is detected. Due to the armor, the projectile's flying noise is hardly or not perceptible inside the vehicle.

Using the present method, it is possible to automatically detect a vehicle being shot at with a projectile in a simple and reliable manner, without a projectile impacting the body or window of the vehicle must take place. This means that further protective measures can be taken as quickly as possible. It is also possible to automatically issue a warning to vehicle occupants when a projectile is detected. The event camera used has a particularly high temporal resolution, so that projectiles moving at high speeds in the area surrounding the vehicle can be recorded safely and reliably.

If projectiles are detected in the vehicle's surroundings, vehicle occupants can, in addition to the warning, also be informed about the direction of an attack and/or about a type of attack (e.g. pistol or rifle). It is also possible to put the vehicle into an automatic defense mode in which, for example, depending on the terrain, a chassis is lowered to avoid fire (comparable to ducking) or raised to enable escape in the terrain. Window panes can also be automatically darkened or additionally hardened. The speed and trajectory of the vehicle can also be automatically and generically changed, for example to enable so-called hooking and acceleration. The speed and trajectory of the vehicle can also be changed automatically according to the at least approximated shooter position.

Exemplary embodiments of the invention are explained in more detail below with reference to drawings.

• 1 schematisch ein Fahrzeug mit einer Vorrichtung zur Erfassung einer Fahrzeugumgebung,
• 2 schematisch ein Blockschaltbild der Vorrichtung gemäß 1 und
• 3 schematisch eine Verarbeitung von erfassten Eventbildern mittels eines gepulsten neuronalen Netzwerks.

Show:

• 1 schematically a vehicle with a device for detecting a vehicle environment,
• 2 schematically a block diagram of the device according to 1 and
• 3 schematically a processing of captured event images using a pulsed neural network.

Corresponding parts are provided with the same reference numbers in all figures.

In 1 a vehicle 1 is shown with a possible embodiment of a device 2 for detecting a vehicle environment. 2 shows a block diagram of the device 2 according to 1 .

The vehicle 1 can be, for example, a motor vehicle, for example a passenger car or truck, a bus or minibus, or an aircraft, watercraft, emergency vehicle, a construction machine, an agricultural or military vehicle, for example an armored personnel carrier or battle tank.

The device 2 comprises a plurality of event cameras 3.1 to 3.n and a control device 4 coupled to them, with a method for detecting a vehicle environment being carried out by means of the device 2.

Event images B1 to Bn of the vehicle surroundings are captured using the event cameras 3.1 to 3.n. The event cameras 3.1 to 3.n have a low latency, in particular a latency of an individual pixel to changes in intensity is less than or equal to 1 µs. Furthermore, the event cameras 3.1 to 3.n have a high time resolution, which is, for example, 1 µs. In realistic applications, the time resolution can be approximately 10 µs to 100 µs, taking into account technical limitations such as the so-called “timestamping-during-readout scheme” or noise. Furthermore, the event cameras 3.1 to 3.n are characterized by a comparatively high dynamic range, so that recordings are possible at very high and very weak light intensities. Furthermore, the event cameras 3.1 to 3.n have the ability to adapt to changing lighting conditions and a comparatively low energy consumption.

The event cameras 3.1 to 3.n process local relative light intensity changes directly and send the information packets generated therefrom as events E1 to Em to a coupled, temporally asynchronous information processing hardware, for example neuromorphic hardware analogous to pulsed neural networks 5 etc. Alternatively, the events E1 to Em aggregated in event images B1 to Bn and evaluated in digital hardware.

To detect a bombardment of the vehicle 1 with a projectile, an evaluation of event information I aggregated to the captured event images B1 to Bn with events E1 to Em is carried out by means of the control unit 4 of the vehicle 1 and by means of a pulsed neural network 5 and an environment suppression .

For this purpose, the events E1 to Em, represented by dashes and forming the event information I, are sent from the event cameras 3.1 to 3.n directly to the control device 4 and processed there using the pulsed neural network 5.

Alternatively or additionally, at least one event image B1 to Bn is generated from the event information I and sent to the control device 4.

It is also possible for the events E1 to Em to be sent to the control device 4 and at least one event image B1 to Bn to be generated by the control device 4.

Projectiles can be detected in the event information I of the event cameras 3.1 to 3.n due to their exceptionally high and at the same time characteristic speed ranges depending on the type of weapon firing the projectile. For this purpose, characteristic speed ranges to be expected in the pulsed neural network 5 with environmental suppression are parameterized accordingly.

For example, a speed of 500 m/s, for example for a 9 mm pistol, to 1000 m/s, for example for an assault rifle, can be assumed. Projectiles can thus be recorded at a speed in the range of 0.5 m/ms to 1 m/ms in an image plane of the respective event camera 3.1 to 3.n or at a small angle to the image plane of the respective event camera 3.1 to 3.m. As soon as the projectile enters the detection range of the respective event camera 3.1 to 3.n, also referred to as the field of view (FOV for short), events E1 to Em are detected, which move across the image plane following an at least approximately rectilinear, uniform movement. It can be assumed that the projectile is relatively small and that usually only a few pixels of the respective event camera 3.1 to 3.n become active.

At a speed of 0.5 m/ms to 1 m/ms, the event cameras 3.1 to 3.n are used for an exemplary distance of the projectile through the detection range of the respective event camera 3.1 to 3.n of 10 m and an exemplary time Resolution of 100 µs 100 to 200 events E1 to Em detected if the projectile only takes up one pixel at a time. A higher temporal resolution and more activated pixels, for example because the projectile moves past the event camera 3.1 to 3.n at a smaller distance, would lead to correspondingly more events, while a smaller distance would lead to correspondingly fewer events E1 to Em.

If a projectile is detected in the event information I or in the at least one event image B1 to Bn by means of the pulsed neural network 5 or the control device 4, a warning W is issued in the vehicle interior and/or a protective function of the vehicle 1 is activated.

In 3 A possible exemplary embodiment of processing captured event images B1 to Bn or event information I using a pulsed neural network 5 is shown.

For this processing, the events E1 to Em of the event cameras 3.1 to 3.n are sent to individual elements or neurons N1 to Nz of a first layer S of the pulsed neural network 5.

Each incoming event E1 to Em is processed by a neuron Nxy that is uniquely assigned to the position of the pixel in the transverse direction y and longitudinal direction x of the image plane of the respective event camera 3.1 to 3.n, with a neuron Nxy also containing the information from several pixels of the respective event camera 3.1 to 3.n can record or a pixel can be assigned to several neurons Nxy.

If an event E1 to Em occurs at a time t0 (symbolized as a star), a neuron Nxy is informed by its pixel about a change in intensity at position (x, y) and is thus excited or activated (represented by neurons N1 to Nz with dotted line). If the excitation occurs above a threshold value, the other neurons Nij (with i, j not equal to x, y) in the vicinity of the neuron Nxy are either put into an excited state (represented by arrows and neurons N1 to Nz with a dash-dot line ) or put into an inhibited state (represented by arrows and neurons N1 to Nz with a broken dashed line), whereby a strength and orientation (excited/inhibited) depends on a distance of the neuron Nij to the neuron Nxy. This allows ambient suppression to be realized. For this purpose, the neurons N1 to Nz are strongly connected in a recurrent manner.

At a later time t1 > t0, the projectile is in a different location and another neuron N1 to Nz is activated with the events E1 to Em of the respective event camera 3.1 to 3.n (dotted border). Recently active neurons N1 to Nz are in a temporarily deactivated state or a refractory period (shown hatched).

If a neuron Nij is in an excited state, an event E1 to Em at a position ij of the respective event camera 3.1 to 3.n can trigger a response or a spike from the neuron Nij more easily and quickly.

If a neuron in Nij is in an inhibited state, an event E1 to Em at a position ij of the respective event camera 3.1 to 3.n cannot trigger a response or a spike from the neuron Nij.

Due to the ambient suppression, all neurons N1 to Nz are inhibited, except for those that could become active in the future when a projectile is detected. This causes hyperactivity of the pulsed neural network werks 5 is prevented and the detection is directed selectively at objects in high speed ranges or at a certain speed. This results in a temporal-spatial pattern M with the coordinates transverse direction y and longitudinal direction x of the image plane and the time t in the first layer S of the pulsed neural network 5, which can be evaluated by a further layer or layers.

In the simplest case, a coincidence detector is used in a second layer, which can detect delayed activity propagation in the first layer S. This means that such a coincidence detector detects the characteristic time-delayed occurrence of several events E1 to Em of the first layer S, which depends on the speed of the projectile. The movement of a projectile is thus detected in the image plane of the respective event camera 3.1 to 3.n.

A parameterization of the surrounding suppression is determined by the expected projectile speed with regard to a strength and distance as well as the excitation and inhibition or generally to the two-dimensional function which describes the strength of the excitation and inhibition from each neuron Nij.

A parameterization of the time-delayed coincidence detector with regard to different synaptic delays, which depends on the distance of the connected neurons Nij, is also determined by the expected projectile speed

Alternatively or additionally, the event information I, as already described, can be temporally aggregated directly in the control device 4 to form event images B1 to Bn or converted into so-called “discretized event streams”, using various methods for this purpose, for example histograms, binary histograms , event volume etc., can be used. These event images B1 to Bn can be generated with high temporal resolution, as it were, by a frame camera with a high image frequency and evaluated using the known method of computer vision, for example artificial neural networks in the control device 4. In particular, such artificial neural networks are trained so that they can recognize projectiles in the event images B1 to Bn.

In addition, a position of a shooter in the control device 4 can at least be approximated by information about which of the event cameras 3.1 to 3.n detected the projectile and which direction of flight was recognized.

In addition, information about a type of fire can be determined from the speed of the projectile and thus a type of threat can be at least approximated in the control device 4. For this purpose, different pulsed evaluation networks with different parameterizations can be used in parallel according to the speed to be detected.

In a possible embodiment, the artificial neural network used can also be designed as an anomaly detector, since projectiles do not occur in normal road traffic due to characteristics such as size, shape and, above all, their speed.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• GB 2605675 A [0003]
• EP 3293681 A1 [0004]

Claims (3)

Method for detecting a vehicle environment, wherein event images (B1 to Bn) of the vehicle environment are captured by means of at least one event camera (3.1 to 3.n), characterized in that in order to detect that the vehicle (1) has been shot at with a projectile, an evaluation is carried out event information (I) aggregated from the captured event images (B1 to Bn) is carried out by means of a control device (4) of the vehicle (1) and/or by means of a pulsed neural network (5) and an environmental suppression. Procedure according to Claim 1 , characterized in that a position of a shooter is determined from a position of the event camera (3.1 to 3.n) capturing the projectile and a direction of flight of the projectile. Device (2) for detecting a vehicle environment with at least one event camera (3.1 to 3.n), which is designed to capture event images (B1 to Bn) of the vehicle environment, characterized by - one for detecting a fire on the vehicle (1). a projectile by means of an evaluation of event information (I) aggregated to the captured event images (B1 to Bn) and/or - a control device (4) designed to detect a bombardment of the vehicle (1) with a projectile by means of an evaluation of a projectile Pulsed neural network (5) with ambient suppression is formed with event information (I) aggregated from the captured event images (B1 to Bn).

Images