DE102020203814A1 - Training an artificial neural network - Google Patents

Abstract

The present invention relates to a device (16) for autonomous or semi-autonomous operation of a vehicle (12), having: an input interface (22) for receiving camera data from a camera sensor (14), the camera data providing information on objects (18) in the vicinity of the Vehicle (12) include; an evaluation unit (24) for determining a vehicle reaction based on an evaluation of the camera data in a pre-trained artificial neural network; and a control unit (26) for determining a control signal for an actuator (20) of the vehicle (12) for executing the vehicle reaction, wherein the artificial neural network is based on environment observation data with information about objects (18) in a field of view of the driver (31 ), simultaneous driver observation data with information on a direction of view of the driver (31) within the field of view of the driver (31) and reaction data with information on an action of the driver (31) is pre-trained. The present invention also relates to a system (10) and a method for autonomous or semi-autonomous operation of a vehicle (12) as well as a training device (28) and a training method.

Description

The present invention relates to a device for the autonomous operation of a vehicle. The invention further relates to a method and a system for autonomous operation of a vehicle, as well as a training device and a training method for an artificial neural network.

Modern vehicles (cars, vans, trucks, motorcycles, etc.) comprise a large number of systems that provide the driver with information and control individual functions of the motor vehicle in a partially or fully automated manner. The surroundings of the motor vehicle and other road users are recorded via sensors. Based on the recorded data, a model of the vehicle environment can be generated and changes in this vehicle environment can be reacted to. As a result of the advancing development in the field of autonomous and semi-autonomous vehicles, the influence and scope of such driver assistance systems (Advanced Driver Assistance Systems, ADAS) are increasing. With the development of ever more precise sensors, it is possible to record the environment and the traffic and to control individual functions of the motor vehicle completely or partially without the intervention of the driver. Driver assistance systems can contribute in particular to increasing traffic safety and improving driving comfort.

In this context, artificial neural networks are particularly relevant for data evaluation and further processing. Applications are, for example, in the area of object recognition based on camera and other sensor data and in the derivation of relevant findings and recommendations for action. One challenge is that a large amount of training data is required to train a neural network. These training data are mostly annotated manually in order to enable an association between sensor data as input training data and corresponding objects as output training data or a correct classification. In addition, there is a challenge in using suitable input data on the basis of which an adequate and appropriate reaction of an autonomous or semi-autonomous vehicle control system can be determined.

In this context, the DE 10 2018 117 380 A1 discloses an approach to braking prediction and braking intervention. A computing device in a vehicle is programmed to predict a collision risk by comparing an acquired occupant facial expression with a plurality of stored occupant facial expressions and to brake a vehicle based on the collision risk.

A relevant problem is to achieve a high level of reliability when deriving an adequate vehicle reaction. The autonomous or semi-autonomous vehicle is intended to carry out a reaction to a recognized situation with the highest possible reliability, which is appropriate to the recognized situation and, as far as possible, corresponds to a reaction of a human driver.

On the basis of this, the present invention has the task of ensuring a reaction of the vehicle to a situation that is as appropriate as possible to the situation in the case of an autonomous or partially autonomous operation of a vehicle. In particular, the safety of one's own vehicle and the occupants as well as other road users should be guaranteed.

• einer Eingangsschnittstelle zum Empfangen von Kameradaten eines Kamerasensors, wobei die Kameradaten Informationen zu Objekten in einer Umgebung des Fahrzeugs umfassen;
• einer Auswerteeinheit zum Ermitteln einer Fahrzeugreaktion basierend auf einer Auswertung der Kameradaten in einem vortrainierten künstlichen neuronalen Netzwerk; und
• einer Steuereinheit zum Ermitteln eines Steuersignals für einen Aktor des Fahrzeugs zum Ausführen der Fahrzeugreaktion, wobei
• das künstliche neuronale Netzwerk basierend auf Umfeld-Beobachtungsdaten mit Informationen zu Objekten in einem Sichtfeld des Fahrers, simultanen Fahrer-Beobachtungsdaten mit Informationen zu einer Blickrichtung des Fahrers innerhalb des Sichtfelds des Fahrers und Reaktionsdaten mit Informationen zu einer Aktion des Fahrers vortrainiert ist.

To achieve this object, the present invention relates in a first aspect to a device for the autonomous or semi-autonomous operation of a vehicle, with:

• an input interface for receiving camera data from a camera sensor, the camera data including information on objects in the vicinity of the vehicle;
• an evaluation unit for determining a vehicle reaction based on an evaluation of the camera data in a pre-trained artificial neural network; and
• a control unit for determining a control signal for an actuator of the vehicle for executing the vehicle reaction, wherein
• the artificial neural network is pre-trained based on environment observation data with information on objects in a driver's field of vision, simultaneous driver observation data with information on a direction of view of the driver within the driver's field of vision and reaction data with information on an action by the driver.

• einem Kamerasensor; und
• einer Vorrichtung wie zuvor beschrieben.

In a further aspect, the invention relates to a system for the autonomous or semi-autonomous operation of a vehicle, with:

• a camera sensor; and
• a device as described above.

• einer Eingangsschnittstelle zum Empfangen von Umfeld-Beobachtungsdaten mit Informationen zu Objekten in einem Sichtfeld eines Fahrers eines Fahrzeugs, simultanen Fahrer-Beobachtungsdaten mit Informationen zu einer Blickrichtung des Fahrers innerhalb des Sichtfelds des Fahrers und Reaktionsdaten mit Informationen zu einer Aktion des Fahrers;
• einer Trainingseinheit zum Eingeben der Umfeld-Beobachtungsdaten und der Fahrer-Beobachtungsdaten als Eingabe-Trainingsdaten sowie der Reaktionsdaten als Ausgabe-Trainingsdaten in ein künstliches neuronales Netzwerk.

In a further aspect, the invention relates to a training device of an artificial neural network, with:

• an input interface for receiving environment observation data with information on objects in a field of vision of a driver of a vehicle, simultaneous driver observation data with information on a direction of view of the driver within the field of vision of the driver and reaction data with information on an action of the driver;
• a training unit for inputting the environment observation data and the driver observation data as input training data and the reaction data as output training data into an artificial neural network.

Further aspects of the invention relate to methods embodied in accordance with the devices and a computer program product with program code for performing the steps of the method when the program code is executed on a computer and a storage medium on which a computer program is stored that when it is executed on a computer , causes the methods described herein to be carried out.

Preferred embodiments of the invention are described in the dependent claims. It goes without saying that the system and the method for autonomous or semi-autonomous operation of the motor vehicle as well as the training device and the training method of the artificial neural network and the computer program product can be designed in accordance with the configurations defined for the device for autonomous or semi-autonomous operation of the vehicle.

Provision is made for camera data to be received via an input point. This camera data is evaluated in order to determine a vehicle reaction based on it. The vehicle reaction corresponds to a driving maneuver that is to be carried out by the vehicle in order, based on a current situation, to avoid endangering the own vehicle or another road user as much as possible. For this purpose, a control signal is determined in a control unit based on the determined vehicle reaction. An actuator of the vehicle can be controlled by means of this control signal in order to cause the vehicle reaction to be carried out. The vehicle reaction is determined by means of an artificial neural network that is pre-trained based on environment observation data, simultaneous driver observation data and reaction data. In particular, these data have been collected over a long period of time and processed in a training phase in the artificial neural network.

In previous approaches, it was proposed to use the driver's line of sight as an input signal for an artificial neural network to determine a vehicle reaction. In contrast, the approach described provides for such driver observation data to be used only for training the artificial neural network, but not during the actual operation of the vehicle. For the decision regarding the vehicle reaction in (later) operation, only camera data is used. In this situation, a reaction can be triggered during operation, even if the driver does not recognize the danger in this case. It is also possible not to trigger a reaction if the driver observes a certain area for other reasons. The result is greater reliability when determining the adapted vehicle reaction to a current situation.

In the training phase, both environment observation data and simultaneously recorded driver observation data are used as input training data in a training device of an artificial neural network according to the invention. The driver's actions in response to the surroundings of the vehicle are used as output training data. Because the environment observation data and the driver observation data are used for the training, information about an image area relevant to the action carried out by the driver can be included. In particular, a relevant object can be identified. In other words, the artificial neural network can be trained to recognize which objects cause or require a specific reaction in which context. The approach according to the invention can in this respect improve the reliability when determining a suitable vehicle reaction. In addition, costs for manually annotating data can be saved, so that more efficient training of an artificial neural network is made possible. In this way, costs can be reduced on the one hand, and the volume of training data can be increased on the other.

In a preferred embodiment, the input interface of the device is designed to receive the camera data from a camera aimed at a planned trajectory of the vehicle. The area that the vehicle intends to cross in the near future is particularly relevant. Objects within this area are relevant. A camera observes the area in which the vehicle will move in the near future. Based on the data obtained, a vehicle reaction can then be determined and a control signal can be generated for an actuator of the vehicle.

In a preferred embodiment, the environment observation data include image data of a surrounding area directed towards the environment of the vehicle. Observation camera. Additionally or alternatively, the driver observation data comprise image data of a driver observation camera sensor directed at the driver. Furthermore, additionally or alternatively, the reaction data include sensor data of a sensor of the vehicle and / or operating data of an actuator of the vehicle. The environment observation data are, in particular, data from an environment camera that is attached to the vehicle. The driver observation data are, in particular, data from a camera that is directed at the driver and, in particular, his eyes. The reaction data can be information from an actuator with regard to its activation as well as information from a sensor, for example an inertial sensor or a sensor attached directly to an actuator of the vehicle. A reliable determination of an adequate vehicle reaction results.

In a preferred embodiment, the control unit is designed to control the actuator with the determined control signal. In particular, it is possible for an actuator to be controlled directly via the control unit. In this respect, the control unit makes a decision as to which action is triggered. For example, a pressure unit of a brake cylinder can be activated in order to increase a pressure in the brake cylinder as a precaution. It is also possible for the pressure to be increased to such an extent that braking is triggered directly. The braking force can be appropriate for the situation. Furthermore, for example, a steering or an acceleration unit of a vehicle can be controlled directly.

In a preferred embodiment, the evaluation unit is designed to determine the vehicle reaction without taking into account current driver observation data with information on a direction of view of the driver within a field of vision of the driver as input data in the pre-trained artificial neural network. The artificial neural network was pre-trained based on previously collected driver observation data. The current driver observation data are not taken into account when determining the vehicle reaction. This can further increase the reliability in determining the adequate response. The driver may be distracted or inattentive. The system recognizes the necessary reaction of the vehicle to the surroundings without taking into account the current direction of view of the driver.

A field of view or a field of view of a camera sensor or also of a driver corresponds to an area that can be seen by the camera sensor or by the driver. In particular, the field of view is defined by specifying an angle in the vertical direction and an angle in the horizontal direction. A vehicle reaction can in particular correspond to a combination of braking, steering and acceleration processes. The environment observation data and the driver observation data are recorded simultaneously. In particular, a corresponding coordinate transformation takes place in order to map the direction of view of the driver in the environment observation data. Such a coordinate transformation makes it possible to assign a viewing direction to a position within an image or within the driver observation data. The reaction data correspond in particular to information about braking, steering and acceleration processes carried out by the driver as a direct result of the environment observation data and the driver observation data. In this respect, the reaction data depict a reaction of the driver to a state of the surroundings. The artificial neural network is trained in particular before the device for autonomous or semi-autonomous operation of the vehicle is operated.

• 1 eine schematische Darstellung eines Fahrzeugs mit einem erfindungsgemäßen System;
• 2 eine schematische Darstellung einer erfindungsgemäßen Vorrichtung zum autonomen oder teilautonomen Betreiben eines Fahrzeugs;
• 3 eine schematische Darstellung eines Fahrzeugs mit einer Trainingsvorrichtung eines künstlichen neuronalen Netzwerks;
• 4 eine Trainingsvorrichtung eines künstlichen neuronalen Netzwerks;
• 5 eine Fahrsituation aus der Perspektive eines Fahrzeugs bzw. eines Fahrers des Fahrzeugs; und
• 6 eine schematische Darstellung eines erfindungsgemäßen Verfahrens.

The invention is described and explained in more detail below with the aid of a few selected exemplary embodiments in connection with the accompanying drawings. Show it:

• 1 a schematic representation of a vehicle with a system according to the invention;
• 2 a schematic representation of a device according to the invention for autonomous or partially autonomous operation of a vehicle;
• 3 a schematic representation of a vehicle with a training device of an artificial neural network;
• 4th an artificial neural network training device;
• 5 a driving situation from the perspective of a vehicle or a driver of the vehicle; and
• 6th a schematic representation of a method according to the invention.

In the 1 is schematically a system according to the invention 10 in a vehicle 12th shown. The representation is to be understood as a schematic lateral sectional view. The system 10 includes a camera sensor 14th as well as a device 16 for autonomous or semi-autonomous operation of the vehicle 12th . With the camera sensor 14th becomes an object 18th within a field of view 19th of the camera sensor 14th in the vicinity of the vehicle 12th recorded. A pre-trained artificial neural network is used in the device 16 an adapted vehicle reaction to the object 18th or the situation in the vicinity of the vehicle 12th determined. To the determined vehicle reaction of the vehicle 12th perform includes the vehicle 12th in particular an actuator 20th , for example a brake cylinder or an acceleration unit etc.

In the illustrated embodiment, the camera sensor 14th , the device 16 as well as the actuator 20th in the vehicle 12th integrated. It is also possible that the device 16 and / or the camera sensor 14th are carried out separately. For example, the device 16 and / or the camera sensor 14th can also be integrated into a mobile device.

It is proposed here to evaluate camera data in a pre-trained artificial neural network in order to determine a vehicle reaction to a situation in the vicinity of an autonomous or partially autonomous vehicle. The artificial neural network is based on previously collected environment observation data, which includes information on objects in the driver's field of vision, simultaneously recorded driver observation data with information on a direction of view of the driver within the driver's field of vision and reaction data with information on an action of the Driver pre-trained. In this respect, the neural network has previously been trained in which, on the one hand, camera data of the environment (surroundings observation data) and, on the other hand, information on a direction of view of the driver (driver observation data) were used. In order to learn the appropriate reaction, the action taken by the driver in response to the situation was taken into account. In this respect, the neural network is pre-trained based on the direction in which the driver is looking. However, the driver's line of sight is then no longer used during operation to determine the vehicle reaction.

In the 2 is schematically a device according to the invention 16 shown for autonomous or semi-autonomous operation of a vehicle. The device 16 includes an input interface 22nd , an evaluation unit 24 as well as a control unit 26th . The various units and interfaces can be implemented separately or in combination in hardware and / or software. In particular, it is possible for the device to be implemented in software that is executed on a control unit of a vehicle. The device 16 can in particular be designed as part of a high-level assistance system. In such an assistance system, a driver guides the vehicle, and the system can intervene by activating various actuators of the vehicle.

Via the input interface 22nd camera data or an image signal from a camera sensor are received. The input interface 22nd be tied in particular to a bus system of the vehicle and connected to the camera sensor via this bus system. In particular, camera data from a camera sensor that is attached to the vehicle can be received. The camera sensor can correspond, for example, to a camera that is fastened in the area of a bumper of the vehicle. The camera data correspond in particular to image data of an area in front of the vehicle. In particular, this area can include a planned trajectory of the vehicle. The planned trajectory is the path that the vehicle will cover in the immediate future. In particular, the lane in front of the vehicle is in the field of view of the camera. The camera data usually include information on several objects. In particular, there are several objects in the recorded image signal.

In the evaluation unit 24 a vehicle reaction is determined based on the camera data. A pre-trained artificial neural network is used for this. The determination of the vehicle reaction can in particular include a determination of a necessary braking, acceleration and / or steering movement of the vehicle. In this respect, a vehicle reaction corresponds in particular to a driving maneuver that is intended to be carried out in response to the objects in the vicinity of the vehicle as recognized by the camera sensor. An adequate reaction of the vehicle to the situation in its surroundings is determined in the evaluation unit by means of the pre-trained artificial neural network. The artificial neural network is pre-trained based on environment observation data and simultaneous driver observation data on the one hand and reaction data with information on an action by the driver on the other hand. The training data was collected earlier. During operation, the vehicle reaction is only determined based on the current camera data.

In the control unit 26th a control signal for an actuator of the vehicle is determined on the basis of the determined vehicle reaction. The actuator of the vehicle, for example a steering system, a braking unit or also an acceleration unit, can be controlled by means of the control signal. The control unit 26th be connected directly to the corresponding actuator. It is also possible that only the control signal for the actuator is determined in the control unit, so that this can be checked again before being applied to the corresponding actuator and, if necessary, further processed. In particular, a control signal for a pressure unit for increasing a pressure in a brake cylinder of the vehicle can be output via the control unit.

A training device is used for training the artificial neural network 28 used. In the 3 is a vehicle 12 ' shown that is a training device 28 includes. On the one hand, environment observation data are recorded and used as input training data. On the other hand, driver observation data are simultaneously recorded and used as input training data. Furthermore, reaction data are used as output training data. The environment observation data are in particular in the form of camera data from an environment observation camera 29 at the vehicle 12 ' recorded. The driver observation data are in particular sent to the driver by means of a 31 directional driver observation camera 30th recorded. The driver's action is monitored by a corresponding sensor 32 determined. As a sensor 32 For example, a pressure sensor in a brake cylinder or an acceleration sensor, etc. can be used. The environment observation camera 29 who have favourited Driver Observation Camera 30th as well as the sensor 32 record data simultaneously. A direction of view of the driver can be determined on the basis of the driver observation data. It goes without saying that a corresponding coordinate transformation must be carried out in order to connect to the environment observation data. At a point in time t + 1, the neural network then tries to predict an action by the driver (also referred to as a driver's request) at a subsequent point in time t2. The action of the driver in the form of reaction data is communicated to the neural network as feedback during the training phase. Based on this, the training device 28 then learn over time to link scenarios based on the environment observation data and the driver observation data with actions to be carried out. Such a continuous labeling of relevant image areas based on the direction of view of the driver enables behavioral patterns to be learned. It is possible that the system is already available to support you during the training phase. It is then advantageous for both the training device 28 as well as for the device 16 a common camera sensor is used.

In the 4th is schematically a training device according to the invention 28 for training an artificial neural network. The training device 28 includes an input interface 34 as well as a training session 36 . The units and interfaces can in turn be implemented separately or in combination in hardware and / or software. In particular, it is possible that the training device 28 and the device 16 are designed to be combined for autonomous or semi-autonomous operation of the vehicle and, in this respect, provide both functions.

In the 5 is a schematic of a usage scenario of a training device according to the invention 28 shown. The representation of the 5 is a situation representation from a driver's perspective of the driver of the vehicle 12 ' to understand. The driver takes off the road 38 in front of the vehicle 12 ' an object 18th true at the roadside. The object can be a deer, for example. The deer is on the side of the road and the vehicle 12 ' drives towards the deer. The driver can assess the situation well and brakes the vehicle or evades the animal. In the training phase, the neural network can then estimate the situation based on the direction in which the vehicle is looking. During operation at a later point in time, the driver can then be supported in similar situations. For example, a pressure in a brake pressure tank can be increased so that braking can be carried out more quickly. By additionally taking into account the direction of view of the driver, it can be mapped in the neural network that this image section contains relevant information regarding an action by the driver. For example, if the driver decides to brake after the animal has turned around, the artificial neural network can learn a link between the movement of the animal and the braking action.

In the 5 a method according to the invention for autonomous or partially autonomous operation of a vehicle is shown schematically. The method comprises the steps of receiving S10 camera data, determining S12 a vehicle reaction and determining S14 a control signal. The method can in particular be implemented in software that is executed on a vehicle control device of a vehicle. In particular, it can be software that is implemented in a driver assistance system.

The invention has been comprehensively described and explained with reference to the drawings and the description. The description and explanation are to be understood as an example and not restrictive. The invention is not limited to the disclosed embodiments. Other embodiments or variations will become apparent to those skilled in the art using the present invention and a careful analysis of the drawings, the disclosure, and the following claims.

In the claims, the words “comprising” and “having” do not exclude the presence of further elements or steps. The undefined article “a” or “an” does not exclude the presence of a plural. A single element or a single unit can perform the functions of several of the units mentioned in the patent claims carry out. An element, a unit, an interface, a device and a system can be implemented partially or completely in hardware and / or in software. The mere mention of some measures in several different dependent patent claims should not be understood to mean that a combination of these measures cannot also be used advantageously. A computer program can be stored / distributed on a non-volatile data carrier, for example on an optical memory or on a semiconductor drive (SSD). A computer program can be distributed together with hardware and / or as part of hardware, for example by means of the Internet or by means of wired or wireless communication systems. Reference signs in the patent claims are not to be understood as restrictive.

List of reference symbols

10

system

12, 12 '

vehicle

14th

Camera sensor

16

contraption

18th

object

19th

Field of view

20th

Actuator

22nd

Input interface

24

Evaluation unit

26th

Control unit

28

Exercise device

29

Environment observation camera

30th

Driver observation camera

31

driver

32

sensor

34

Input interface

36

Training session

38

roadway

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 102018117380 A1 [0004]

Claims (11)

Device (16) for the autonomous or semi-autonomous operation of a vehicle (12), with: an input interface (22) for receiving camera data from a camera sensor (14), the camera data including information on objects (18) in the vicinity of the vehicle (12); an evaluation unit (24) for determining a vehicle reaction based on an evaluation of the camera data in a pre-trained artificial neural network; and a control unit (26) for determining a control signal for an actuator (20) of the vehicle (12) for executing the vehicle reaction, wherein the artificial neural network based on environment observation data with information on objects (18) in a field of vision of the driver (31), simultaneous driver observation data with information on a direction of view of the driver (31) within the field of vision of the driver (31) and reaction data Information on an action by the driver (31) is pre-trained. Device (16) after Claim 1 wherein the input interface (22) is designed to receive the camera data from a camera aimed at a planned trajectory of the vehicle (12). Device (16) according to one of the preceding claims, wherein the environment observation data comprise image data of an environment observation camera (29) aimed at the environment of the vehicle (12); the driver observation data comprise image data of a driver observation camera (30) directed at the driver (31); and or the reaction data include sensor data of a sensor of the vehicle (12) and / or operating data of an actuator (20) of the vehicle (12). Device (16) according to one of the preceding claims, wherein the control unit (26) is designed to control the actuator (20) with the determined control signal. Device (16) according to one of the preceding claims, wherein the control unit (26) is designed to determine a control signal for a pressure actuator for increasing a pressure in a brake cylinder of the vehicle (12). Device (16) according to one of the preceding claims, wherein the evaluation unit (24) for determining the vehicle reaction without taking into account current driver observation data with information on a direction of view of the driver (31) within a field of view of the driver (31) as input data in the pre-trained artificial neural network is formed. System (10) for autonomous or semi-autonomous operation of a vehicle (12), comprising: a camera sensor (14); a device (16) according to one of the Claims 1 until 6th . Method for the autonomous or semi-autonomous operation of a vehicle (12), with the steps: Receiving (S10) camera data from a camera sensor (14), the camera data including information on objects (18) in the vicinity of the vehicle (12); Determining (S12) a vehicle reaction based on an evaluation of the camera data in a pre-trained artificial neural network; and Determination (S14) of a control signal for an actuator (20) of the vehicle (12) for executing the vehicle reaction, wherein the artificial neural network based on environment observation data with information on objects (18) in a field of vision of the driver (31), simultaneous driver observation data with information on a direction of view of the driver (31) within the field of vision of the driver (31) and reaction data Information on an action by the driver (31) is pre-trained. Training device (28) of an artificial neural network, comprising: an input interface (34) for receiving environment observation data with information on objects (18) in a field of vision of a driver (31) of a vehicle (12 '), simultaneous driver observation data with information on a direction of view of the driver (31) within the field of vision of the driver (31) and reaction data with information on an action by the driver (31); a training unit (36) for inputting the environment observation data and the driver observation data as input training data and the reaction data as output training data into an artificial neural network. Training method of an artificial neural network, with the steps: Receiving environment observation data with information on objects (18) in a field of vision of a driver (31) of a vehicle (12), simultaneous driver observation data with information on a direction of view of the driver (31) within the field of vision of the driver (31) and reaction data with information on an action by the driver (31); Inputting the environment observation data and the driver observation data as input training data and the reaction data as output training data into an artificial neural network. Computer program product with program code for performing the steps of one of the Procedure according to Claims 8 and 10 when the program code is executed on a computer.

Images