DE102014201036A1 - Image-based classification of driver status and / or driver behavior - Google Patents

Abstract

The invention relates to a method and a corresponding device for determining the condition of a driver of a vehicle. A control unit (101) for a vehicle (100) is described. The control unit (101) is configured to receive image data from one or more image sensors (103). The one or more image sensors (103) are configured to detect a driver position (102) of the vehicle (100). The control unit (101) is further configured to use the image data to assign the viewing direction of a driver of the vehicle (100) to a viewing sector from a multiplicity of viewing sectors (201,..., 213) for a sequence of times, and thus temporally To determine course s (t) of viewing sectors as a representation of a temporal evolution of the driver's line of sight. Furthermore, the control unit (101) is set up to evaluate the temporal course s (t) of gaze sectors.

Description

The invention relates to a method and a corresponding device for determining the condition of a driver of a vehicle.

In the areas of intelligent driver assistance systems (FAS) and / or highly automated driving (HAF), it makes sense to include driver-relevant parameters such as the driver's attention, distraction of the driver from the driving task, driver intent, tiredness of the driver, and drowsiness of the driver and / or a takeover ability of the driver, in particular if the driver is driven highly automated, to determine. The driver-relevant parameters can then be correlated with takeover times and the various driving systems (FAS and / or automatic driving systems) can be adaptively adapted to the determined driver state.

Today's driving systems typically have a relatively limited view of the driver's condition. For example, the simplified assumption is based in part on the fact that the driver is attentive to the driving task when the driver has his hands on the steering wheel. On the other hand, the ability to take over in automated driving (for example in a train driver) can be checked with a so-called "dead man" switch, which has to be actuated periodically by the driver in order to ensure that it can be taken over.

The measures described above for determining the condition of the driver are insufficient. On the one hand, simplified assumptions (such as the assumption that the driver has his hands on the steering wheel can be closed, that the driver is alert) lead to misjudgements regarding the driver's actual condition and the ability of the driver to take over active driving. On the other hand, verification routines (such as the periodic operation of a "dead man" switch) cause the driver to be confronted with a secondary task that additionally distracts the driver from the traffic.

The present document is concerned with the technical task of reliably determining the driver's condition and / or a driver's ability to take over, without the driver having to carry out an active checking routine which would distract the driver from the traffic situation and / or the comfort for the driver the driver would be reduced.

The object is solved by the independent claims. Advantageous embodiments are u. a. in the dependent claims.

In one aspect, a control unit for a vehicle (eg, for a two-lane vehicle such as a passenger car or a lorry) is described. The control unit may be used to provide an assistance function to the driver of the vehicle. In this connection, a driver's condition or a driver's takeover ability can be determined. Alternatively or additionally, it can be determined in this context whether or not a driver has perceived an object or an obstacle in an environment of the vehicle.

The control unit is set up to receive image data from one or more image sensors (eg cameras). The one or more image sensors are configured to detect a driver's position of the vehicle. In particular, the one or more image sensors can be set up in such a way that the angle of vision of the driver of the vehicle in a predefined viewing angle range (possibly up to 360 degrees) can be determined from the acquired image data. The image data for this purpose typically includes information about predefined and / or distinctive facial features (such as the eyes) and / or facial expression of the driver.

The control unit is further configured, on the basis of the image data and for a sequence of points in time, to assign the viewing direction of the driver of the vehicle (which results from the image data) to a viewing sector from a plurality of viewing sectors. The gaze sectors of the plurality of gaze sectors may be assigned to a coordinate system of the vehicle or may be defined in a coordinate system of the vehicle. In particular, the viewing sectors of the plurality of viewing sectors may each comprise different (defined) areas of a front area and / or the side area of the vehicle. In particular, the viewing sectors may include areas of the vehicle that indicate the driver is focused on the traffic (eg, a gaze sector comprising a windshield and / or rearview mirror and / or side mirrors of the vehicle). In addition, other gaze sectors may include areas of the vehicle that indicate that the driver is engaged in secondary tasks (eg, includes a gaze sector of the vehicle's controls). The entire driver's field of view may be divided into a limited number N of viewing sectors. Typical values for N are 5, 10, 15 or 20 viewing sectors.

By assigning the line of sight for the sequence of times, a temporal course s (t) of gaze sectors can be determined as representing a temporal evolution of the driver's line of sight.

The control unit can also be set up to evaluate the time profile s (t) of view sectors. The evaluation can be carried out in particular with regard to a provided assistance function. As part of the evaluation, values for one or more statistical parameters can be determined. For example, the control unit may be set up to determine portions p _{i of} the individual view sectors from the plurality of viewing sectors in the driver's line of sight in a predefined time interval on the basis of the time profile s (t) of view sectors. Alternatively or additionally, the control unit may be configured to determine frequencies h _i for transitions between the individual view sectors from the plurality of view sectors in the predefined time interval. These or other statistical parameters can be used to determine a condition of the driver. In addition, it can be determined on the basis of these or other statistical parameters whether the driver has perceived a specific obstacle or an object in the vicinity of the vehicle. This can be achieved, for example, by matching data with regard to the viewing sectors (for example, by adjusting the time profile s (t) of view sectors) with an environment model. In the environment model, an object detected by a driver assistance system (FAS) in the surroundings of the vehicle, for example another road user, can be spatially detected.

As already explained, the control unit can be set up to determine a driver's condition on the basis of the time profile s (t) of view sectors. For this purpose, the control unit can be set up, in particular, for the predefined time interval, one or more current parameter values for one or more statistical parameters (eg for the portions p _i or for the frequencies h _i ) from the time course s (t ) of viewing sectors. Furthermore, the control unit can provide a predefined cluster algorithm or classification algorithm that is set up to assign different parameter values of the one or more statistical parameters to different predefined state classes. Examples of condition classes are: "Driver makes a lane change", "Driver makes a controlled straight-ahead drive", "Driver is distracted", and / or "Driver stares unconsciously".

The control unit can also be set up to determine a current condition class as the driver's condition based on the one or more current parameter values and on the basis of the predefined cluster algorithm / classification algorithm. In other words, the current parameter values can be used to determine the current state of the driver.

The control unit is thus set up to determine the condition of the driver in a reliable and autonomous manner. In particular, to determine the driver state, no action of the driver is required by which the driver could be distracted from driving.

The control unit may be further configured to determine that a reliability of the driver's state determined based on the time course s (t) of gaze sectors is less than or equal to a predefined threshold value. In other words, it can be recognized that the condition of the driver can not be reliably determined from the temporal history s (t) of gaze sectors (for example, because the driver wears sunglasses). The control unit may, in response, cause the driver of the vehicle to be prompted to actuate a switch of the vehicle (possibly periodically) to check the condition of the driver. This ensures that the condition of the driver can be reliably determined even with insufficient image data.

The control unit may be configured to receive environment data from one or more environment sensors (eg a camera and / or a radar sensor) of the vehicle and / or position data from a positioning unit (eg a navigation device) of the vehicle. On the basis of the environment data and / or the position data, an object can be detected in an environment of the vehicle. The detected object can be assigned to a gaze sector from the plurality of gaze sectors. Alternatively or additionally, the determined eye direction vectors of the driver can be cut directly to the intersection recognition with the detected object to determine whether the object has been detected in an environment of the vehicle.

The size and / or position of an o. G. Object dependent gaze sector may change in time (due to the change in the position of the object relative to the vehicle). Thus, the gaze sectors of the plurality of gaze sectors may change dynamically with time. Alternatively or additionally, the positions of the objects in the vehicle environment may vary due to object or vehicle inherent motions. Because of this, the assignment of objects in the vehicle environment to gaze sectors can also vary dynamically over time.

The control unit may be configured to consider the detected object in determining the driver's condition. Thereby, an accuracy of determining the condition of the driver can be increased.

Alternatively or additionally, the control unit can be set up on the basis of the time History s (t) of Blicksektoren to determine that the driver does not perceive the detected object. In response, issuance of a warning regarding the detected object to the driver of the vehicle may be initiated. Thus, the safety of the vehicle in road traffic can be increased.

In another aspect, a method of providing an assistance function in a vehicle is described. The method includes receiving image data from one or more image sensors. The one or more image sensors are configured to detect a driver position of the vehicle (in particular a head of a driver of the vehicle).

The method further comprises assigning, based on the image data and for a sequence of times, the direction of view of a driver of the vehicle to a gaze sector from a plurality of gaze sectors, a temporal course s (t) of gaze sectors representing a temporal evolution of the gaze direction to determine the driver. Furthermore, the method comprises the evaluation of the time profile s (t) of gaze sectors for providing the assistance function.

In another aspect, a vehicle (eg, a passenger car, a truck, or a motorcycle) is described that includes a control unit described in this document.

In another aspect, a software (SW) program is described. The SW program may be set up to run on a processor (eg, on a controller) and thereby perform the method described in this document.

In another aspect, a storage medium is described. The storage medium may include a SW program that is set up to run on a processor and thereby perform the method described in this document.

It should be understood that the methods, devices and systems described herein may be used alone as well as in combination with other methods, devices and systems described in this document. Furthermore, any aspects of the methods, apparatus, and systems described herein may be combined in a variety of ways. In particular, the features of the claims can be combined in a variety of ways.

Furthermore, the invention will be described in more detail with reference to exemplary embodiments. It shows

1 a block diagram with exemplary components of a vehicle;

2 an exemplary classification of a vehicle in view sectors; and

3 a flowchart of an exemplary method for determining the state of a driver.

As set forth above, the present document is concerned with determining the condition of the driver of a vehicle (eg, a passenger car or a truck). This is particularly important when using driver assistance systems (FAS) or in partially or highly automated driving. Through such assistance functions, the driver is partially or completely relieved of active driving. Nevertheless, it should be ensured that the driver can take over active driving again if necessary. The present document describes in this connection methods and devices for determining the condition of the driver. From the condition of the driver can also be typically concluded on the ability of the driver to take over the active driving of the vehicle (the so-called takeover capability). Furthermore, from the driver state, a period can be determined, which the driver needs to take over the active driving operation (the so-called takeover time).

• • Level 1: Fahrer ist aufmerksam (Fahrer partizipiert aktiv am Verkehrsgeschehen). Der Fahrer kann somit jederzeit und mit einer minimalen Übernahmezeit den aktiven Fahrbetrieb übernehmen.
• • Level 2: Fahrer ist kurzfristig abgelenkt von der Fahraufgabe (durch sekundäre Aufgaben wie Telefonieren, Bahnplanung, Bedienung von Entertainment- und Komfortfunktionen). Durch die kurzfristige Ablenkung ergibt sich eine erhöhte Übernahmezeit.
• • Level 3: Fahrer ist müde/erschöpft. Hier ergibt sich eine eingeschränkte Fahrperformance und eine weiter erhöhte Übernahmezeit.
• • Level 4: Fahrer ist schläfrig (starke Müdigkeitserscheinungen treten auf, z. B. hochfrequenter Liedschlag, Augen fallen zu, keine bewusste Szenenwahrnehmung, Sekundenschlaf). Dies führt zu einer weiter eingeschränkten Fahrperformance und einer weiter erhöhten Übernahmezeit.
• • Level 5: Fahrer ist nicht übernahmefähig (z. B. eingeschlafen, physisch nicht präsent).
• • Level 6: Fahrer ist tot.

The driver state can be classified into a variety of predefined states. These predefined driver states can also be referred to as takeover capability levels. Exemplary takeover levels are:

• • Level 1: driver is attentive (driver participates actively in traffic). The driver can thus take over active driving at any time and with a minimum takeover time.
• • Level 2: driver is distracted from the driving task at short notice (due to secondary tasks such as telephoning, path planning, operation of entertainment and comfort functions). The short-term distraction results in an increased takeover time.
• • Level 3: driver is tired / exhausted. This results in a limited driving performance and a further increase takeover time.
• • Level 4: Driver is sleepy (severe fatigue symptoms occur, eg high frequency song punch, eyes fall, no conscious scene perception, microsleep). this leads to a further limited driving performance and a further increase takeover time.
• • Level 5: driver is not eligible (eg asleep, physically not present).
• • Level 6: driver is dead.

• • Spurwechselabsicht, d. h. der Fahrer hat die Absicht eine Fahrspur zu wechseln.
• • Der Fahrer verhält sich als idealer Regler, der typischerweise durch eine aufmerksame Geradeausfahrt und bewusste Szenenwahrnehmung gekennzeichnet ist.
• • Der Fahrer ist durch sekundäre Aktivitäten abgelenkt.
• • Der Fahrer führt ein unbewusstes Starren durch.

In the present document, a camera-based system and method is described, which are adapted to perform a classification of the driver state, for. For example, in the above-mentioned takeover level. Different driver behavior can be determined. Exemplary driver behaviors are:

• • Lane change intent, ie the driver intends to change lanes.
• • The driver behaves as an ideal controller, which is typically characterized by attentive straight-ahead driving and deliberate scene perception.
• • The driver is distracted by secondary activities.
• • The driver performs an unconscious stare.

In order to determine the condition of the driver, a system is proposed with a multiplicity of cameras which are arranged such that at least one of the plurality of cameras can detect the driver's eyes in any desired orientation of the head in order to determine a viewing direction of the driver. The determined viewing direction can then be subdivided into a plurality of viewing sectors in order to carry out a statistical evaluation of the viewing direction over time. The statistical evaluation of the driver's line of sight over time can provide information about the condition of the driver.

1 shows a block diagram of exemplary components of a vehicle 100 , In particular shows 1 a variety of image sensors 103 (eg, cameras) that are set up image data regarding the driver of the vehicle 100 capture. The variety of image sensors 103 is on a driver's position 102 of the vehicle 100 directed. In particular, the plurality of image sensors 103 arranged such that a head of the driver of the vehicle 100 can be captured. Furthermore, the large number of image sensors 103 such in the vehicle 100 arranged such that for a predefined range of angles the driver's eyes are each at least one image sensor of the plurality of image sensors 103 be detected to determine the driver's line of sight. The predefined range of viewing angles may include a range of azimuth angles and / or a range of elevation angles.

bezeichnet werden, wobei α(t) der zeitliche Verlauf des Azimutwinkels ist und wobei γ(t) der zeitliche Verlauf des Höhenwinkels ist.The image data of the plurality of image sensors 103 can be connected to a control unit 101 be transmitted. The control unit 101 is set up, based on the image data, at a variety of times the perspective of the driver of the vehicle 100 to investigate. The image data may in particular be video data with a specific frame rate (eg 30 frames / second). The control unit 101 can be configured to determine the time course of the driver's angle of view with a temporal resolution that depends on the frame rate of the captured image data. Typically, the temporal resolution of the temporal course of the viewing angle corresponds to the frame rate of the image data. The temporal course of the angle can with

where α (t) is the time course of the azimuth angle and where γ (t) is the time profile of the elevation angle.

• • einen Blicksektor 201 für ein Fahrerseitiges Fenster des Fahrzeugs 100 (ohne Außenspiegel);
• • einen Blicksektor 202 für den Fahrerseitigen Außenspiegel des Fahrzeugs 100;
• • einen Blicksektor 203 für Fahrerseitige seitliche Bedienelemente/Ablagen des Fahrzeugs 100;
• • einen Blicksektor 204 für einen Fahrerseitigen Teil der Windschutzscheibe;
• • einen Blicksektor 205 für ein Lenkrad und ein Armaturenbrett des Fahrzeugs 100;
• • einen Blicksektor 206 für einen Rückspiegel des Fahrzeugs 100;
• • einen Blicksektor 207 für ein Infotainmentsystem des Fahrzeugs 100;
• • einen Blicksektor 208 für einen Gangwahlhebel und/oder eine zentrale Ablage des Fahrzeugs 100;
• • einen Blicksektor 209 für einen Beifahrerseitigen Teil der Windschutzscheibe;
• • einen Blicksektor 210 für ein Handschuhfach des Fahrzeugs 100;
• • einen Blicksektor 211 für eine Beifahrerposition des Fahrzeugs 100;
• • einen Blicksektor 212 für den beifahrerseitigen Außenspiegel des Fahrzeugs 100; und/oder
• • einen Blicksektor 213 für ein Beifahrerseitiges Fenster des Fahrzeugs 100 (ohne Außenspiegel).

Furthermore, the control unit 101 be configured to classify the driver's viewpoint B (t) into a plurality of predefined view sectors, where S = {s _i ) where i = 1, ..., N and N> 1 represents the set of predefined view sectors s _i . 2 shows an exemplary division of the front portion of the vehicle 100 into a large number of (possibly static) viewing sectors 201 , ..., 213 , The exemplary variety of viewing sectors 201 , ..., 213 includes:

• • a gaze sector 201 for a driver-side window of the vehicle 100 (without exterior mirrors);
• • a gaze sector 202 for the driver side exterior mirror of the vehicle 100 ;
• • a gaze sector 203 for driver side side controls / shelves of the vehicle 100 ;
• • a gaze sector 204 for a driver side portion of the windshield;
• • a gaze sector 205 for a steering wheel and a dashboard of the vehicle 100 ;
• • a gaze sector 206 for a rearview mirror of the vehicle 100 ;
• • a gaze sector 207 for an infotainment system of the vehicle 100 ;
• • a gaze sector 208 for a gear selector and / or a central shelf of the vehicle 100 ;
• • a gaze sector 209 for a passenger-side part of the windshield;
• • a gaze sector 210 for a glove compartment of the vehicle 100 ;
• • a gaze sector 211 for a passenger position of the vehicle 100 ;
• • a gaze sector 212 for the passenger-side exterior mirror of the vehicle 100 ; and or
• • a gaze sector 213 for a passenger side window of the vehicle 100 (without outside mirror).

The control unit 101 may be arranged to assign the determined viewing angle B (t) to one of the plurality of viewing sectors s _i . This results in a time course of viewing sectors s (t), where s (t) Can accept values from the set S of possible view sectors. Alternatively or additionally, the control unit 101 be configured to determine the time course of gaze sectors s (t) directly from the captured image data. By classifying the driver's perspective in a limited number of viewing sectors 201 , ..., 213 a statistically relevant evaluation of the driver's perspective is made possible.

The driver state can now be determined on the basis of the time course s (t) of view sectors. For this purpose, the course s (t) can be statistically evaluated. In particular, a distribution of the driver's line of sight (i.e., the gaze sectors) may be determined in a predefined time period or time interval. The distribution of the viewing direction can then provide indications of the condition of the driver.

For example, two-dimensional (2D) and / or three-dimensional (3D) bar histograms of the visual distributions can be used for driving behavior estimation. The gaze distribution can reflect the gaze behavior of the driver over sliding time windows / time intervals. The time windows may have a configurable time length.

An exemplary bar histogram includes on a first axis the possible plurality of predefined view sectors 201 , ..., 213 , On a second axis, the time duration (ie the time proportion of the sliding time window) can be plotted, in which the driver in the respective Blicksektoren 201 , ..., 213 looked. In a third dimension, the frequency with which the view into the respective eye sector can be recorded 201 , ..., 213 is curly. Based on the thus calculated dynamic gaze distribution histograms, classifier and / or template matching can be used to classify different driver behaviors.

für die Blicksektoren s_i, mit i = 1, ..., N, ermittelt werden, wobei T die Zeitdauer des vordefinierten Zeitfensters ist, und wobei T_i die kumulierte Zeitdauer auf dem vordefinierten Zeitfenster ist, in der sich die Blickrichtung des Fahrers in dem entsprechenden Blicksektor s_i befindet. Die zeitlichen Anteile p_i sind ein Indiz dafür, worauf der Fahrer derzeit seine Aufmerksamkeit gerichtet hat.In other words, on the basis of the time course s (t) of the determined viewing sectors 201 , ..., 213 may be a relative proportion p _{i of} the individual view sectors 201 , ..., 213 be determined in the driver's line of sight in a predefined time window. In particular, shares

for the gaze sectors s _i , where i = 1, ..., N, where T is the time duration of the predefined time window, and where T _{i is} the accumulated time on the predefined time window in which the driver's line of sight is in the corresponding eye sector s _i . The temporal portions p _i are an indication of what the driver is currently paying attention to.

von Übergängen der Blickrichtung in den Blicksektor s_i erfasst werden. Die Häufigkeiten h_i können ein Indiz für eine Grad der Konzentration des Fahrers sein.Based on the time course s (t) of the determined viewing sectors 201 , ..., 213 Furthermore, a frequency h _i can be determined with which the driver's gaze was directed to the corresponding gaze sector s _i in the predefined time window / time interval. The frequency h _i can z. B. as an absolute number Z _i of transitions of the viewing direction in the gaze sector s _i during the predefined time window or as a relative number

be detected by transitions of the viewing direction in the eye sector s _i . The frequencies h _i can be an indication of a degree of concentration of the driver.

• • Spurwechselabsicht: Bei einer Spurwechselabsicht erscheinen bei korrektem Fahrverhalten mit hohem Anteil Spiegel- und Schulterblicke (z. B. Blicksektoren 201, 202, 206, 212) sowie Blicke in Fahrtrichtung (z. B. Blicksektor 204). Andererseits sind Blicke auf Bedienteile und/oder auf ein Display typischerweise gering (z. B. Blicksektoren 203, 207).
• • Idealer Regler: Dies beschreibt eine Person die überwiegend aufmerksam in Fahrtrichtung (d. h. Blicksektor 204) schaut, mit einem geringen Anteil von Spiegelblicken (Blicksektoren 202, 206, 212) und Schulterblicken. Es liegt somit eine bewusste Szenenwahrnehmung vor. Der Anteil von Blicken auf Bedienteile oder Displayanzeigen (z. B. Blicksektoren 203, 207) ist typischerweise gering.
• • Abgelenkter Fahrer: In diesem Fall blickt der Fahrer für einen hohen prozentualen Anteil der Zeit innerhalb eines gleitenden Zeitfensters auf Bedienteile oder Displayelemente (z. B. Blicksektoren 203 und/oder 207).
• • Unbewusstes Starren liegt z. B. vor, wenn der Blick des Fahrers auf einen einzelnen Blicksektor fokussiert ist. Eine Streuung der Blickrichtungsvektoren ist dabei typischerweise gering (bspw. liegt eine Varianz von 1° oder weniger vor), wodurch auf Unaufmerksamkeit durch Starren gefolgert werden kann.

From the statistical analysis of the time course s (t) of the identified viewing sectors 201 , ..., 213 , in particular from the portions p _{i of} the gaze sectors and / or from the frequencies h _{i of} the gaze sectors, the driving behavior of the driver can be assigned to individual driving behavior classes:

• • Lane change intent: In the case of a lane change intention, with correct driving behavior with a high proportion, mirror and shoulder views (eg, view sectors 201 . 202 . 206 . 212 ) and looks in the direction of travel (eg Blicksektor 204 ). On the other hand, glances at keypads and / or on a display are typically low (eg, viewing sectors 203 . 207 ).
• • Ideal controller: This describes a person's mostly attentive in the direction of travel (ie gaze sector 204 ) looks, with a small proportion of mirror views (Blicksektoren 202 . 206 . 212 ) and shoulder looks. There is thus a conscious scene perception. The proportion of views of keypads or display displays (eg, view sectors 203 . 207 ) is typically low.
• • Deflected driver: In this case, the driver looks at keypads or display elements (eg, viewing sectors) for a high percentage of the time within a sliding time window 203 and or 207 ).
• • Unconscious staring is z. For example, when the driver's gaze is focused on a single gaze sector. A scattering of the line of sight vectors is typically low (for example, there is a variance of 1 ° or less), which can lead to inattention by stare.

Thus, on the basis of the time course s (t) of the determined viewing sectors 201 , ..., 213 a condition of the driver can be determined. For this purpose, one or more statistical parameters from the time course s (t) of the determined Blicksektoren 201 , ..., 213 be calculated. In particular, one or more statistical parameters can be determined via a (possibly sliding) predefined time window / time interval. Examples of such statistical parameters are the proportions p _{i of} the individual view sectors 201 , ..., 213 in the direction of the driver and the frequencies h _i for the transitions between the individual viewing sectors 201 , ..., 213 ,

The one or more statistical parameters may then be used to determine the driver's condition. For this purpose a cluster algorithm can be used. In particular, in a learning phase, a plurality of contiguous pairs of "values of the one or more statistical parameters" and "driver states" may be recorded. From this multitude of contiguous pairs, a cluster algorithm can then be derived which recognizes relationships between the "values of the one or more statistical parameters" and the "driver states". In particular, the cluster algorithm so trained can be used to determine the current driver state in normal operation from the determined current values of the one or more statistical parameters.

The clustering algorithm may be determined based on a learning group with a plurality of different drivers (eg, to provide basic functionality for a new vehicle). Alternatively or additionally, the cluster algorithm can be trained or adapted driver-specific (for example, in order better to be able to take into account driver-specific habits).

An extension of the classification procedure, via the evaluation of the gaze sectors 201 . 213 In addition, it may be to capture additional knowledge about objects in the vehicle environment and to take into account in the evaluation of driver behavior. In particular, knowledge about a 3D position of vehicles and / or knowledge about persons in the vehicle environment of the vehicle 100 , and / or knowledge about the position and orientation of lanes.

For this purpose, the vehicle 100 one or more environmental sensors 105 include, which are set up, an environment of the vehicle 100 capture. The one or more environment sensors 105 can z. B. include a camera and / or a radar sensor. Based on the one or more environment sensors 105 provided environment data, the control unit 101 For example, the presence of objects and / or obstacles in the environment of the vehicle 100 determine.

Furthermore, the vehicle can 100 a positioning unit 106 include, which is adapted to determine position data. The positioning unit 106 In particular, a navigation system with digital maps of the environment of the vehicle 100 include. Based on the position data can from the control unit 101 an absolute position of the vehicle 100 and / or a relative position of the vehicle 100 relative to a detected object in the environment of the vehicle 100 be determined.

A detected object can be used in the evaluation of the time course s (t) of the determined viewing sectors 201 , ..., 213 be taken into account. In particular, it can be determined and taken into account whether a viewing direction of the driver (ie a determined gaze sector and / or eye vector) correlates with a position of the detected object. As an alternative or in addition, it is possible with knowledge of the road course to determine whether the driver is looking in the direction of travel (ie, in a viewing sector that correlates with the course of the road).

Furthermore, as an alternative or in addition to the Blicksektoren 201 , ..., 213 the one or more detected objects and / or the road course are taken into account as classes for the assignment of the driver's line of sight. In particular, instead of or in addition to a classification in the Blicksektoren 204 . 209 the windshield, information about consciously perceived scene objects in the visual distribution histograms for classification, ie in the statistical analysis, are taken into account. This typically improves the classification quality, since it can be discriminated whether the driver looks out the window (and thus tends to be unfocused) or whether the driver is looking at a relevant scene object.

In order to take into account detected objects and / or to take account of the course of the road in the statistical analysis, for example, the histogram bars of certain viewing sectors (for example the viewing sectors 204 . 209 ) are replaced by a bar with the viewing time and viewing frequency of one or more actively perceived scene objects. More generally, the one or more detected objects and / or the known road course may be taken into account as (possibly additional) view sectors s _i , wherein a position and possibly a size of these view sectors may change over time (eg the vehicle 100 passing the detected object). The statistical analysis may then be performed as set forth above to determine a condition of the driver.

The described object lists with viewing times and viewing frequencies can, alternatively or in addition to the classification of Driver behavior in a driver assistance system used to warn of unperceived objects (eg cyclists crossing lanes). In other words, the analysis of the time course s (t) of the determined viewing sectors described in this document 201 , ..., 213 can also be used to warn the driver of unperceived objects.

• • Level 6: Ob der Fahrer lebt oder tot ist kann beispielsweise über eine bildbasierte Puls schlagerkennung, bei der minimale Hautverfärbungen im Videobild erkannt werden, bestimmt werden. Alternativ oder ergänzend können nichtinvasive Sensoren zur Pulsmessung im Sitz verbaut werden.
• • Level 5: Die Anwesenheit des Fahrers am Fahrersitz kann über Bildverarbeitung erkannt werden, da der Kopf sich im Fahrzeug innerhalb einer validen Kopfposition befinden sollte. So fällt beispielsweise auf, wenn der Fahrer beim hochautomatisierten Fahren den Fahrersitz verlässt oder eine ungewöhnliche Sitzposition einnimmt. Ob der Fahrer eingeschlafen ist, kann ebenfalls bildbasiert ermittelt werden, da dann die Augen des Fahrers für ein längeres Zeitintervall geschlossen sind.
• • Level 3/4: Zur Abgrenzung des Levels 3 von Levels 4 können Verhaltensmerkmale der Augen (Liedschlagfrequenz, Geschwindigkeit, Augenöffnungsmaß) analysiert werden.
• • Level 2: Durch Analyse der beschriebenen statistischen Analyse (z. B. anhand von Blickverteilungshistogrammen) kann dieser Level typischerweise eindeutig festgestellt werden.
• • Level 1: Durch Analyse der beschriebenen statistischen Analyse (z. B. anhand von Blickverteilungshistogrammen) kann dieser Level typischerweise eindeutig festgestellt werden.

In order to determine from the classified driver behavior and / or from the classified driver state a takeover ability level of the driver, additional information may possibly be analyzed:

• • Level 6: Whether the driver is alive or dead can be determined, for example, by means of an image-based pulse beat detection, in which minimal skin discoloration in the video image is detected. Alternatively or additionally, non-invasive sensors for pulse measurement in the seat can be installed.
• • Level 5: The presence of the driver on the driver's seat can be detected by image processing, as the head should be in the vehicle within a valid head position. For example, it is noticeable when the driver leaves the driver's seat in highly automated driving or assumes an unusual seating position. Whether the driver has fallen asleep, can also be determined image-based, since then the eyes of the driver are closed for a longer time interval.
• • Level 3/4: To level 3 Levels 4, eye behavioral characteristics (song beat frequency, speed, eye opening) can be analyzed.
• • Level 2: By analyzing the described statistical analysis (eg, using gaze distribution histograms), this level can typically be uniquely determined.
• • Level 1: By analyzing the described statistical analysis (eg using gaze distribution histograms), this level can typically be uniquely determined.

For highly automated drivers, it may be useful to be able to differentiate between takeover skill levels 3 and 5. For the adaptation of driver assistance systems or partially automated driving, a differentiation of takeover capability levels 1 and 2 can be realized in particular. The acceptance level should be recognized with a high degree of reliability. Also in situations in which a camera-based system may not work properly (eg driver wears mirrored glasses, eyes are not visible due to concealment, optics of the camera is misted, etc.), the driver's condition and / or To be able to reliably determine the takeover level, the vehicle can 100 a "dead man" switch 104 which can then be used when using the plurality of image sensors 103 no reliable gradients of the driver's line of sight can be determined. The control unit 101 can be set up to prompt the driver to switch the "dead man" switch 104 when it is not possible to determine reliable courses of sight of the driver, or when it is recognized that the driving condition can not be detected with sufficient reliability. So a high reliability can be achieved at the same time high comfort of use.

3 shows a flowchart of an exemplary method 300 , The procedure 300 In particular, the provision of an assistance function in a vehicle 100 (eg a driver assistance function or to provide a partially automated or highly automated driving function). The procedure 300 includes receiving 301 of image data from one or more image sensors 103 , The one or more image sensors 103 are set up, a driver position 102 of the vehicle 100 capture. In particular, the one or more image sensors 103 be arranged so that different viewing directions of the driver (in a certain viewing angle range) can be detected and analyzed based on the image data.

The procedure 300 further comprises assigning 302 based on the image data and for a sequence of times, the line of sight of the driver of the vehicle 100 to one eye sector from a large number of viewing sectors 201 , ..., 213 , By such an assignment for the sequence of times, a temporal course s (t) of gaze sectors can be determined as representing a temporal development of the driver's line of sight.

The procedure 300 includes further evaluation 303 the time course s (t) of viewing sectors to provide the assistance function. For example, the temporal course s (t) of gaze sectors with regard to a partially automated or highly automated driving function can be evaluated to determine the driver's condition. Alternatively or additionally, the time profile s (t) of gaze sectors can be evaluated with regard to a driver assistance function, in which the driver has no perceived obstacles or objects in an environment of the vehicle 100 is informed.

This document describes a classification procedure for the reliable determination of driver status. The classification method can without explicit intervention, i. H. without distraction, the driver made. Furthermore, the underlying driver behavior analyzes and the environment object lists enriched with eye-sight and frequency of views result in possibilities for implementing improved and new driver assistance functions (in particular with regard to warning the driver against unperceived obstacles).

The present invention is not limited to the embodiments shown. In particular, it should be noted that the description and figures are intended to illustrate only the principle of the proposed methods, apparatus and systems.

Claims (10)

Control unit ( 101 ) for a vehicle ( 100 ), the control unit ( 101 ), image data from one or more image sensors ( 103 ) to recieve; wherein the one or more image sensors ( 103 ), a driver position ( 102 ) of the vehicle ( 100 ) capture; On the basis of the image data, for a sequence of times the line of vision of a driver of the vehicle ( 100 ) one eye sector from a large number of view sectors ( 201 , ..., 213 ), and thus to determine a temporal course s (t) of viewing sectors as a representation of a temporal evolution of the driver's line of sight; and - evaluate the time course s (t) of viewing sectors. Control unit ( 101 ) according to claim 1, wherein the control unit ( 101 ) is arranged to determine a state of the driver based on the time course s (t) of gaze sectors. Control unit ( 101 ) according to claim 2, wherein the control unit ( 101 ) is arranged to: - determine, for a predefined time interval, one or more current parameter values for one or more statistical parameters from the time course s (t) of view sectors; To provide a predefined cluster algorithm that is set up to assign different parameter values of the one or more statistical parameters to different predefined state classes; and - based on the one or more current parameter values and on the basis of the predefined cluster algorithm, to determine a current state class as the state of the driver. Control unit ( 101 ) according to one of claims 2 to 3, wherein the control unit ( 101 ) is arranged to determine that a reliability of the driver's state determined on the basis of the time profile s (t) of gaze sectors is less than or equal to a predefined threshold value; and - in response to causing the driver of the vehicle ( 100 ), a switch ( 104 ) of the vehicle ( 100 ) to check the condition of the driver. Control unit ( 101 ) according to any preceding claim, wherein the control unit ( 101 ) environment data from one or more environment sensors ( 105 ) of the vehicle ( 100 ) and / or position data from a positioning unit ( 106 ) of the vehicle ( 100 ) to recieve; On the basis of the environment data and / or the position data, an object in an environment of the vehicle ( 100 ) to detect; and the detected object is a gaze sector from the plurality of gaze sectors ( 201 , ..., 213 ). Control unit ( 101 ) according to claim 5 with respect to claims 2 or 4, wherein the control unit ( 101 ) is arranged to take into account the detected object in the determination of the driver's condition. Control unit ( 101 ) according to one of claims 5 to 6, wherein the control unit ( 101 ) is arranged to - determine on the basis of the time course s (t) of viewing sectors that the driver does not perceive the detected object; and - in response, the issuing of a warning of the detected object to the driver of the vehicle ( 100 ). Control unit ( 101 ) according to any preceding claim, wherein the control unit ( 101 ) is set up on the basis of the time profile s (t) of viewing sectors, portions p _{i of} the individual view sectors from the plurality of viewing sectors ( 201 , ..., 213 ) in the driver's line of sight at a predefined time interval; and / or on the basis of the time profile s (t) of viewing sectors, frequencies h _i for transitions between the individual viewing sectors from the large number of viewing sectors ( 201 , ..., 213 ) in the predefined time interval. Control unit ( 101 ) according to any preceding claim, wherein the view sectors of the plurality of view sectors ( 201 , ..., 213 ) different areas of a front area and / or a side area of the vehicle ( 100 ). Procedure ( 300 ) for providing an assistance function in a vehicle ( 100 ), the process ( 300 ), - receiving ( 301 ) of image data from one or more image sensors ( 103 ); where the one or several image sensors ( 103 ), a driver position ( 102 ) of the vehicle ( 100 ) capture; - To assign ( 302 ), based on the image data and for a sequence of times, the line of sight of a driver of the vehicle ( 100 ) to one eye sector each from a large number of view sectors ( 201 , ..., 213 ) to determine a time course s (t) of viewing sectors as a representation of a temporal evolution of the driver's line of sight; and - evaluation ( 303 ) of the time course s (t) of view sectors to provide the assistance function.

Images