DE102024200317A1 - Method, device and computer program for predicting fatigue of a driver of a vehicle - Google Patents

Abstract

A method for predicting fatigue of a driver of a vehicle comprises obtaining (110) environmental perception data of the vehicle from at least one environmental perception sensor of the vehicle, wherein the environmental perception data represents at least one environment of the vehicle while driving. The method further comprises processing (120) the environmental perception data to determine (150) a prediction variable that reflects whether the driver of the vehicle is fatigued. The method further comprises providing (160) a warning based on the prediction variable.

Description

The invention relates to a method, a device and a computer program for predicting fatigue of a driver of a vehicle.

Drowsy and fatigued driving is a major problem for road safety, and as the level of automation in autonomous vehicles increases, it becomes increasingly difficult for drivers to remain focused and alert when required to take control of the vehicle. The biggest challenge is detecting the driver's condition early enough before they largely lose control of the system. In general, methods for detecting fatigue fall into two categories.

On the one hand, there are methods based on monitoring the driver's physical state (usually with an interior camera or other biometric sensors), which some people perceive as an invasion of privacy. On the other hand, there are methods based on monitoring vehicle-related conditions that correlate with the driver's physical state. Due to the increasing overlay of driver activity by interventions from ADAS (Advanced Driver-Assistance Systems) systems, this is becoming increasingly difficult, as the driver's actions are smoothed and filtered, and there is no direct information from the road geometry or the environment.

Information about such methods can be found, for example, in the Handbook of Intelligent Vehicles by Azim Eskandarian, as well as in the publications US2011284304A1 , US5900819A , US8519853B2 and US20100039249A1 be taken.

There is a need for an improved method for predicting driver fatigue of a vehicle that overcomes the disadvantages of existing systems.

This need is taken into account by the present independent claims.

The present invention is based on the finding that the disadvantages of the two aforementioned categories can be overcome by predicting driver fatigue based on monitoring environmental conditions. By utilizing an environmental model from external vehicle sensors (e.g., radar, lidar, camera, acoustic sensors, thermal imaging cameras, etc.), direct correlations can be established between the state of a vehicle in the driving environment and the state of the driver. This creates an additional category for determining driver fatigue. In particular, a vehicle-independent approach to estimating driver state in general and detecting fatigue in particular is enabled.

One aspect of the present invention relates to a method for predicting fatigue of a driver of a vehicle. The method comprises obtaining environmental perception data of the vehicle from at least one environmental perception sensor of the vehicle. The environmental perception data represents at least one environment of the vehicle while driving. The method comprises processing the environmental perception data to determine a predictive variable that reflects whether the driver of the vehicle is fatigued. The method comprises providing a warning based on the predictive variable. By using the environmental perception data, conclusions can be drawn about the driver's fatigue without violating the driver's privacy. Furthermore, minor interventions by the driver assistance systems have only a limited influence on the state of the vehicle represented by the environmental perception data. Furthermore, this approach can be transferred to a variety of vehicle models without the need for adaptation to the sensor hardware used there.

Many of the aspects that indicate driver fatigue can be quantified, which allows both the prediction to be traced and the definition of precise limits. The method may, for example, include determining one or more numerical parameters based on the environmental perception data. The method may include determining the predicted value based on the one or more numerical parameters. The numerical parameters may, for example, be compared to threshold values to assess whether the driver is fatigued.

For example, the one or more numerical parameters may be at least one of a numerical parameter about driving the vehicle in a central area of the lane, a numerical parameter about a deceleration between a braking of a vehicle ahead and a braking of the vehicle driven by the driver, a numerical parameter about a braking intensity, a numerical parameter for steering intensity and a numerical parameter for the deviation between an optimal steering trajectory and an actual steering trajectory in a curve. These numerical parameters allow conclusions to be drawn about the driver's fatigue.

In some implementations, determining at least one numerical characteristic may include processing at least a portion of the environmental perception data using a machine learning model to determine the at least one numerical characteristic or an intermediate result for determining the at least one numerical characteristic. Machine learning models can efficiently evaluate a variety of different environmental perception data and determine a corresponding numerical characteristic.

Machine learning methods can be used not only to determine the numerical parameter(s), but also to process them. For example, determining the predictive parameter can involve processing the one or more numerical parameters using a machine learning model that is trained to predict the driver's fatigue based on the one or more numerical parameters. The predictive parameter can be based on an output of the machine learning model. By using a machine learning model, a large number of input values (such as numerical parameters) can be combined. These input values, in isolation, are not sufficient to predict fatigue, but are sufficiently meaningful when combined. For this purpose, so-called training of the model is sufficient; threshold values for the individual numerical parameters do not necessarily have to be defined.

For example, the machine learning model outputting the predicted variable can be a regressor machine learning model or a classifier machine learning model. Regressors output a value on a scale and are therefore useful when subsequently adjusting thresholds for classification into "fatigued" and "not fatigued." Classifiers directly output classifications, such as "fatigued" and "not fatigued," which cannot offer the aforementioned flexibility of subsequent threshold adjustment. However, they facilitate training the machine learning model, as classifier machine learning models converge better in many cases, and specifying labels as training output data is easier (since only labels for the various desired classifications are required).

In some cases, determining the predictor may involve comparing one or more numerical metrics with one or more thresholds. For example, the predictor may be based on the comparison. Using thresholds makes predictions more transparent, and calculating the predictor is less complex than using a machine learning model to predict the predictor. However, the use of thresholds reaches its limits when many different numerical metrics are used, which are not sufficiently meaningful in isolation.

For example, the method may include processing the environmental perception data and/or data derived from the environmental perception data using a machine learning model. The machine learning model may be trained to predict the driver's fatigue based on the environmental perception data and/or data derived from the environmental perception data as input data. The prediction variable may be based on an output of this machine learning model. This allows for the implementation of a prediction that is independent of numerical parameters and therefore requires fewer intermediate steps. On the other hand, such an approach requires complex training of the machine learning model and a correspondingly large amount of training data.

Such a machine learning model can, for example, further process additional sensor data from the vehicle, which relates to at least one driving characteristic of the vehicle, such as vehicle speed, as additional input data. Using such input data can, on the one hand, improve prediction accuracy and, on the other hand, enable faster convergence of the model during training.

In some cases, the presented approaches can be combined by using the one or more numerical parameters as additional input data for the machine learning model. The method can thus comprise determining one or more numerical parameters based on the environmental perception data. The machine learning model can further process the one numerical parameter or the several numerical parameters as additional input data. By using the numerical parameter(s), on the one hand, the prediction accuracy can be improved and, on the other hand, faster convergence of the model during training can be enabled.

In modern vehicles, environmental perception data is used by many different systems, such as traffic sign recognition or the implementation of adaptive cruise control. To reduce the computational effort required to predict drowsiness, data generated by these systems can be repurposed. The method can comprise processing the data derived from the environmental perception data using the machine learning model, wherein the data derived from the environmental perception data originates from at least one of the vehicle's adaptive cruise control system, traffic sign recognition, and lane departure warning system. This avoids the additional effort required for multiple processing of the environmental perception data.

For example, the environmental perception data may include at least one of camera sensor data, radar sensor data, lidar sensor data, ultrasonic sensor data, accelerometer data, and audio recording data. These sensors are suitable for assessing how the vehicle is being moved by the driver within the environment.

The predictive value may include a classification of predicted driver fatigue into one of at least two classes, such as "fatigue" and "not fatigued." Such a predictive value may be used directly to determine whether a driver warning is necessary. Alternatively or additionally, the predictive value may include a numerical value. When outputting a numerical value, a subsequently adjustable threshold may be used to classify driver fatigue.

In some cases, the method may further include restricting vehicle functionality based on the predicted value. This can increase driving safety by forcing the driver to take a break or at least forgo assistance systems that promote drowsiness.

Driver fatigue is a cause of many accidents. Driver fatigue detection should be performed regardless of whether a driver reacts late even when not fatigued, is not driving in the center of the lane, etc. For example, the prediction value can be determined regardless of the driver. Accordingly, drivers who react late or are generally inattentive can be warned sooner, even if they may not yet be fatigued. This can have an educational effect and thus increase driving safety in the medium term.

Using this method, the prediction variable can be determined independently of the vehicle type. This allows the method to be used in a variety of different vehicles without the need for costly adaptation to the specific vehicle.

A further aspect of the present invention relates to a program comprising a program code for performing the method described above when the program code is executed on a computer, a processor, a control module or a programmable hardware component.

A further aspect of the present invention relates to a device comprising a memory, machine-readable instructions, and at least one processor circuit for executing the machine-readable instructions for carrying out the method described above.

Another aspect of the present invention relates to a vehicle comprising this device.

• 1a zeigt ein Flussdiagramm eines Verfahrens zum Vorhersagen von Müdigkeit eines Fahrers eines Fahrzeugs;
• 1b zeigt ein schematisches Diagramm einer Vorrichtung zum Vorhersagen von Müdigkeit eines Fahrers eines Fahrzeugs;
• 2 zeigt ein Datenflussdiagramm eines Flusses von Daten zum Vorhersagen von Müdigkeit eines Fahrers eines Fahrzeugs; und
• 3 zeigt ein Beispiel eines segmentierten Bilds eines Kamerasensors.

Some examples of devices and/or methods are explained in more detail below with reference to the accompanying figures. They show:

• 1a shows a flowchart of a method for predicting fatigue of a driver of a vehicle;
• 1b shows a schematic diagram of an apparatus for predicting fatigue of a driver of a vehicle;
• 2 shows a data flow diagram of a flow of data for predicting fatigue of a driver of a vehicle; and
• 3 shows an example of a segmented image from a camera sensor.

Some examples will now be described in more detail with reference to the accompanying figures. However, further possible examples are not limited to the features of these detailed embodiments.

1a shows a flowchart of a method for predicting fatigue of a driver of a vehicle. The method comprises a Obtaining 110 environmental perception data of the vehicle from at least one environmental perception sensor of the vehicle. The environmental perception data represents at least one environment of the vehicle while driving. The method includes processing 120 the environmental perception data to determine 150 a predictive value that reflects whether the driver of the vehicle is fatigued. The method includes providing 160 a warning based on the predictive value.

1b shows a schematic diagram of a device 10 for predicting fatigue of a driver of a vehicle. The device 10 comprises an (optional) interface 12, a processor circuit 14, and a memory 16, wherein the processor circuit 14 is coupled to the optional interface 12 and the memory. The device 10 further comprises machine-readable instructions, such as program code or a compiled computer program. These can be stored, for example, in the memory 16. The processor circuit 14 is configured to execute the machine-readable instructions for carrying out the method of 1a For example, the device can be used in a vehicle. Accordingly, the present invention also relates to a vehicle with the device 10.

The present invention relates to predicting the fatigue of a vehicle driver. In the context of the present disclosure, the term "prediction" does not refer to a prediction in the temporal sense, but rather to a prediction in the sense of an estimate or a derivation, since in the present invention, the driver's fatigue is derived from environmental perception data.

Environmental perception data is data that represents at least the vehicle's surroundings while driving. Environmental perception data is generally collected by the vehicle's external sensors, i.e., sensors that detect the environment outside the vehicle. This includes, for example, camera sensor data (at least one outward-facing camera), radar sensor data (at least one radar sensor), lidar sensor data (at least one lidar sensor), ultrasonic sensor data (at least one ultrasonic sensor), accelerometer data (at least one accelerometer or gyroscope), and audio recording data (from an outward-facing microphone). Based on the environmental perception data, it is possible to determine how the driver drives the vehicle within the environment, which allows conclusions to be drawn about the driver's fatigue. To give an example, if the driver drifts from the center of the lane, brakes late, or performs numerous corrective steering maneuvers, this is a sign of driver fatigue. Indications of such driving behavior can be found in the environmental perception data.

In the following, 2 Three possible approaches are presented to predict fatigue based on environmental perception data. 2 shows a data flow diagram of a flow of data for predicting fatigue of a driver of a vehicle. In 2 The environmental sensors 210 are shown, which provide the environmental perception data 215. The environmental perception data can additionally be processed by one or more other vehicle systems 220, such as an adaptive cruise control system, traffic sign recognition, or a lane departure warning system, which provide data 225 derived from the environmental perception data. Furthermore, one or more additional sensors 230, such as a speed sensor, can be used to provide additional sensor data 235. In a first approach, the environmental perception data 215 or numerical parameters 245 derived therefrom are compared with threshold values to determine the predictive variable 270. In a second approach, the environmental perception data 215 are converted into numerical parameters, for example with the aid of a first machine learning model 240, which are in turn processed by a second machine learning model 250. The output of the second machine learning model is in turn used to determine the predictive variable. In a third approach, the environmental perception data 215 and/or the data 225 derived from the environmental perception data, and optionally the numerical parameters and/or the additional sensor data 235, are processed by a third machine learning model 260. The output of the third machine learning model 260 is then used to determine the predictive variable. In this context, the term "predictive variable" was chosen because the predictive variable represents a prediction of the driver's fatigue (or lack of fatigue). The aforementioned three approaches are explained below.

The first approach is based on determining numerical parameters based on the environmental perception data and then comparing these with threshold values. Accordingly, the method in this approach comprises determining 130 one or more numerical parameters based on the environmental perception data, and determining the predicted value based on the one or more several numerical parameters. The one or more numerical parameters can comprise at least one of a numerical parameter relating to the vehicle's travel in a central region of the lane, a numerical parameter relating to a deceleration between a braking of a vehicle traveling ahead and a braking of the vehicle driven by the driver, a numerical parameter relating to a braking intensity, a numerical parameter relating to a steering intensity, and a numerical parameter relating to a deviation between an optimal steering trajectory and an actual steering trajectory in a curve. At least some of these parameters can be determined, for example, using machine learning. In other words, determining 130 at least one numerical parameter can comprise processing at least a portion of the environmental perception data using the first machine learning model 240 to determine the at least one numerical parameter or an intermediate result for determining the at least one numerical parameter. This is briefly explained below for some of the aforementioned numerical parameters.

Before that, however, we will briefly discuss machine learning itself. Machine learning refers to algorithms and statistical models that computer systems can use to perform a specific task without using explicit instructions, instead of relying on models and inference. For example, instead of a rule-based transformation of data, machine learning can use a transformation of data that can be derived from an analysis of historical and/or training data. For example, the content of images can be analyzed using a machine learning model or using a machine learning algorithm. In order for the machine learning model to analyze the content of an image, the machine learning model can be trained using training images as input and training content information as output. By training the machine learning model with a large number of training images and/or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize the content of the images, allowing the content of images not included in the training data to be recognized using the machine learning model. The same principle can be used for other types of sensor data as well: By training a machine learning model using training sensor data and a desired output, the machine learning model "learns" a conversion between the sensor data and the output, which can be used to provide an output based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and/or image data) can be preprocessed to obtain a feature vector, which is used as input to the machine learning model.

Machine learning models, such as the first, second, and third machine learning models, can be trained using training input data. The examples above use a training method called supervised learning. In supervised learning, the machine learning model is trained using a plurality of training samples, where each sample can include one or more input values (also called training input data) and a plurality of desired output values, i.e., each training sample is associated with a desired output value (also called training output data). By specifying both training input values and desired output values, the machine learning model "learns" which output value to provide based on an input value that is similar to the input values provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some of the training samples lack a desired output value.

Supervised learning is generally based on a supervised learning algorithm (e.g., a classification algorithm, a regression algorithm, or a similarity learning algorithm). Classification algorithms can be used when the outputs are restricted to a limited set of values (categorical variables), meaning the input is classified as one of the limited set of values. Regression algorithms can be used when the outputs indicate any numerical value (within a range). Similarity learning algorithms can be similar to both classification and regression algorithms, but are based on learning from examples using a similarity function that measures how similar or related two objects are.

In addition to supervised learning or semi-supervised learning, unsupervised learning can be used to train the machine learning model. In unsupervised learning, (only) input data may be provided and an unsupervised learning algorithm can used to find structure in the input data (e.g., by grouping or clustering the input data, finding commonalities in the data). Clustering is the assignment of input data comprising a plurality of input values into subsets (clusters) such that input values within the same cluster are similar according to one or more (predefined) similarity criteria, while they are dissimilar to input values comprised in other clusters.

Reinforcement learning is a third group of machine learning algorithms. In other words, reinforcement learning can be used to train the machine learning model. In reinforcement learning, one or more software agents are trained to perform actions in an environment. A reward is calculated based on the actions performed. Reinforcement learning is based on training the one or more software agents to select actions in such a way that the cumulative reward is increased, resulting in software agents that become better at the task given to them (as evidenced by increasing rewards).

Machine learning algorithms are typically based on a machine learning model. In other words, the term "machine learning algorithm" may refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" may refer to a data structure and/or a set of rules that represents the learned knowledge (e.g., based on the training performed by the machine learning algorithm). In embodiments, the use of a machine learning algorithm may imply the use of an underlying machine learning model (or a plurality of underlying machine learning models). The use of a machine learning model may imply that the machine learning model and/or the data structure/set of rules that is/are the machine learning model is trained by a machine learning algorithm.

For example, the machine learning model, such as the first, second, and third machine learning models, can be an artificial neural network (ANN). ANNs are systems inspired by biological neural networks, such as those found in a retina or brain. ANNs comprise a plurality of interconnected nodes and a plurality of connections, called edges, between the nodes. There are typically three types of nodes: input nodes that receive input values, hidden nodes that are connected (only) to other nodes, and output nodes that provide output values. Each node can represent an artificial neuron. Each edge can send information from one node to another. The output of a node can be defined as a (nonlinear) function of the inputs (e.g., the sum of its inputs). The inputs of a node can be used in the function based on a "weight" of the edge or the node providing the input. The weight of nodes and/or edges can be adjusted during the learning process. In other words, training an artificial neural network may involve adjusting the weights of the nodes and/or edges of the artificial neural network, i.e., to achieve a desired output for a given input. If one or more layers of hidden nodes are present, it is also referred to as a deep neural network. The machine learning model used here may be an artificial neural network, and in particular, a deep neural network.

In the present case, the first machine learning model is used to process at least a portion of the environmental perception data in order to determine the at least one numerical characteristic or an intermediate result for determining the at least one numerical characteristic. The first machine learning model is therefore trained to use the environmental perception data as input data and, based on the environmental perception data as input data, to determine the at least one numerical characteristic or an intermediate result for determining the at least one numerical characteristic. Preferably, different first machine learning models are used if different numerical characteristics are to be determined.

To give an example - to determine the numerical parameter about the vehicle driving in the middle of the lane, the machine learning model can be trained to perform image segmentation of camera sensor data. Such machine learning models are well known and widespread, which is why the training of such a model will not be discussed further below. From the image segmentation, the image coordinates of a vehicle's current lane can now be determined. If it is known which x-coordinates of the image correspond to the lateral center of the vehicle, it can be determined based on the image coordinates of a vehicle's current lane whether the vehicle is driving in the middle of the lane, and a numerical parameter The magnitude of a deviation from the center of the lane can be calculated. Similarly, the numerical parameter for the deviation between the optimal steering trajectory and the actual steering trajectory in a curve can be determined by determining and quantifying the deviation between the x-coordinates of the lateral center of the vehicle and the x-coordinates of the center of the roadway, which corresponds to the optimal steering trajectory.

In another example, the numerical parameter can be based on the delay between the braking of a vehicle in front and the braking of the vehicle driven by the driver. The first machine learning model can be trained to determine whether or when the vehicle in front brakes based on camera, radar, lidar and/or ultrasonic sensor data and/or data from an acceleration sensor (for the own vehicle). To this end, the first machine learning model can be trained, for example, using supervised learning, with the respective environmental perception data being used as training input data and a classification of “vehicle in front brakes” or “vehicle in front does not brake” being used as training output data. In addition, it can be determined whether or when the own vehicle brakes based on the camera, radar, lidar and/or ultrasonic sensor data and/or data from an acceleration sensor. The delay between braking events can be used to determine the numerical parameter. The numerical parameter for braking intensity can be derived directly from the accelerometer data or from the camera sensor data using the first machine learning model. The numerical parameter for steering intensity can also be derived from the accelerometer data or from the camera sensor data using the first machine learning model.

In the first approach, the one or more numerical parameters are then compared with one or more threshold values to predict whether the driver is fatigued. Accordingly, as in 1a As further shown, determining 150 the predictive value may comprise comparing the one or more numerical characteristic values with one or more threshold values, wherein the predictive value is based on the comparison. The comparison may be used to distinguish between the driver states of "fatigue" and "not fatigued." Accordingly, the predictive value may also comprise classifying a predicted fatigue of the driver into one of at least two classes, namely "fatigue" and "not fatigued."

In the second approach, the one or more numerical parameters are also used to determine the predictive value. In this case, however, instead of (or in addition to) the one or more threshold values, the second machine learning model 250 is used to determine the predictive value. Accordingly, determining 150 the predictive value can comprise processing the one or more numerical parameters using the second machine learning model. The second machine learning model can be trained to predict the driver's fatigue based on the one or more numerical parameters. Accordingly, the predictive value can be based on an output of the machine learning model. Depending on whether the second machine learning model is a regressor machine learning model or a classifier machine learning model, training of the machine learning model can take place. In both cases, supervised learning can be used to train the second machine learning model. Samples with sets of one or more numerical parameters can be used as training input data. In the case of a classifier, appropriate labels can be used as training output data to indicate whether the driver is fatigued or not. In the case of a regressor, training output data can be used to indicate the driver's fatigue on a scale (approximately between 0 and 1). The respective training output data can be defined, for example, based on self-reporting by one or more test drivers or based on the output of another fatigue prediction system, such as the output of a fatigue prediction system that uses a camera to monitor the driver's attention, and thus fatigue. For example, during test drives, corresponding numerical parameters (as training input data) and training output data for training the second machine learning model can be generated. Depending on the type and training of the second machine learning model (regressor or classifier), the prediction variable can include a classification of the predicted driver fatigue into one of at least two classes or a numerical value.

In the third approach, the environmental perception data and/or data derived from the environmental perception data are processed directly by the third machine learning model, which is trained to output a prediction of the driver's fatigue. Accordingly, the method may include processing 140 the environmental perception data and/or the data derived from the environmental perception data using the third machine learning model 260. The third machine learning model is now trained to predict the driver's fatigue based on the environmental perception data and/or data derived from the environmental perception data as input data. Accordingly, the prediction variable may be based on an output of the machine learning model. If, in addition to or instead of the environmental perception data, the data 225 derived from the environmental perception data is processed using the machine learning model, the data derived from the environmental perception data may be generated by other vehicle systems of the vehicle that already perform this task to fulfill their respective functionality. Thus, the data derived from the environmental perception data may originate from at least one of an adaptive cruise control system, a traffic sign recognition system, and a lane departure warning system (etc.) of the vehicle. The data derived from the environmental perception data may, for example, contain data already segmented using image segmentation or object lists from the images or radar/lidar sensor data.

The third machine learning model can further receive additional input data to improve the precision of the prediction and to support convergence during training. Thus, the third machine learning model can further process additional sensor data 235 of the vehicle relating to at least one driving characteristic of the vehicle, such as a vehicle speed, a steering angle, a braking effect, a recuperation effect, etc., as additional input data. Additionally (or alternatively), the third machine learning model can further process the one or more numerical parameters 245 as additional input data.

Training is similar to the training of the second machine learning model, whereby in this case the training input data is the data used in production operation (environmental perception data, data derived from the environmental perception data, additional sensor data, and/or numerical parameter(s)), and the same training output data that were previously mentioned for the second machine learning model 250 are used as training output data. Since in the case of the third machine learning model, the output of other machine learning models and processing steps is also partially used (such as from the additional vehicle component(s) 220) or the first machine learning model 240, end-to-end training can be carried out in this case, in which the entire pipeline is included, starting from the environmental perception data and the additional sensor data. The weightings of the other machine learning models can, for example, be frozen to avoid unwanted changes. Depending on the type and training of the third machine learning model (regressor or classifier), the predictive variable can also include a classification of the predicted driver fatigue into one of at least two classes or a numerical value.

Based on the predictive value, which indicates whether (in the case of classification) or to what extent (in the case of regression) the driver appears to be fatigued, a warning based on the predictive value is then provided 160, for example via a screen, a warning light, a projection, or an audio output. Additionally, the method may further comprise restricting 170 a functionality of the vehicle based on the predictive value. For example, the use of a driver assistance system that relies on the driver being able to regain control of the vehicle within a very short time may be prevented.

As can be seen from the previous description, the presented method is independent of how the driver normally drives the vehicle and which driver is driving the vehicle. In other words, the determination of the predicted variable is performed regardless of which driver is driving the vehicle. Furthermore, provided the environmental perception data are consistent or similar, which is often the case in vehicles based on modular systems, no or only minor adaptation of the method to a specific vehicle type is necessary. Accordingly, the determination of the predicted variable can be performed regardless of the vehicle type.

The interface 12 may, for example, correspond to one or more inputs and/or one or more outputs for receiving and/or transmitting information, such as in digital bit values, based on a code, within a module, between modules, or between modules of different entities.

In embodiments, the processor circuit 14 may correspond to any controller or processor or a programmable hardware component. For example, the processor circuit 14 may also be implemented as software. which is programmed for a corresponding hardware component. In this respect, the processor circuit 14 can be implemented as programmable hardware with correspondingly adapted software. Any processors, such as digital signal processors (DSPs), can be used. Embodiments are not limited to a specific type of processor. Any processor or even multiple processors are conceivable for implementation. For example, the processor circuit 14 can correspond to at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an accelerator for performing machine learning, and a Field-Programmable Gate Array (FPGA).

The memory 16 may, for example, comprise at least one member of the group of computer-readable storage medium, magnetic storage medium, optical storage medium, hard disk, flash memory, floppy disk, random access memory (also known as random access memory), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), and network storage.

More details and aspects of the procedure from 1a , the device 10 of 1b and the corresponding computer program are mentioned in connection with the concept or examples that are given afterwards (e.g. 3 ). The method, apparatus and computer program may comprise one or more additional optional features corresponding to one or more aspects of the proposed concept or the described examples, as described before or after.

The present invention relates to a non-invasive perception-based estimation of driver state.

3 shows an example of a segmented image from a camera sensor. Different elements are displayed differently in the segmented image. For example, the segmentation distinguishes between the vehicle's own lane 310, an oncoming lane 320, and a turning lane 330. The segmentation also distinguishes between other vehicles 340, indicators 350 (such as a traffic light or a warning column) and people 360, as well as background buildings 370 and sky 380. Based on a perceptual environment model (derived from environmental perception data), such as derived from lanes (the vehicle's own lane 310, the oncoming lane 320) and vehicle segmentation (the vehicles 330), features of the vehicle state in the environment over time can be extracted.

Different approaches can be used to predict driver fatigue based on environmental perception data. In a first approach, manually defined features can be used that are related to the 1a to 2 "numerical parameters." Possible features that can be extracted include the robustness of lane keeping (how centered the vehicle is within the segmented lane), the start of braking (relative to the deceleration of the vehicle in front), the deceleration rate of braking, and the start and speed of the steering process (e.g., related to a deviation from the optimal lane in curves). Using a data-driven model, these features can then be mapped to a driver state and estimated by a model.

In a third approach, driver fatigue can be estimated from large available datasets. An end-to-end approach can be used, in which the driver status is derived directly from the segmentation mask using machine learning.

The basic workflow of the method includes three core steps - obtaining an environment model (e.g., road segmentation or clearance estimation), feature estimation (not necessary if the environment model is directly processed by a machine learning model), and estimation of the driver's state.

The present invention presents a methodology that aims to improve the ability to detect fatigue by monitoring environmental conditions and correlating them with the driver's state. The method is non-intrusive and does not compromise the driver's privacy. Furthermore, in many implementations, the method is vehicle-independent and can therefore be used across entire fleets, eliminating the need to adapt to different vehicle models. The method is also capable of assessing the driver's state in general if the machine learning models used are appropriately trained, with a focus on detecting fatigue. The method can be implemented using a system with a modular design, which enables functionality with different data availabilities and the three approaches explained above. a handcrafted feature approach to an end-to-end approach.

The aspects and features described in connection with a particular one of the previous examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the feature into the further example.

Examples may further include or relate to a (computer) program having program code for performing one or more of the above methods when the program is executed on a computer, processor, or other programmable hardware component. Steps, operations, or processes of various of the methods described above may therefore also be performed by programmed computers, processors, or other programmable hardware components. Examples may also cover program storage devices, e.g., digital data storage media, that are machine-, processor-, or computer-readable and encode or contain machine-executable, processor-executable, or computer-executable programs and instructions. The program storage devices may include, for example, digital memories, magnetic storage media such as magnetic disks and magnetic tapes, hard disk drives, or optically readable digital data storage media. Further examples may also cover computers, processors, control units, field-programmable logic arrays ((F)PLAs = (Field) Programmable Logic Arrays), field-programmable gate arrays ((F)PGA = (Field) Programmable Gate Arrays), graphics processor units (GPU = Graphics Processor Unit), application-specific integrated circuits (ASIC = application-specific integrated circuit), integrated circuits (IC = Integrated Circuit) or system-on-a-chip (SoC = System-on-a-Chip) programmed to perform the steps of the methods described above.

It is further understood that the disclosure of multiple steps, processes, operations, or functions disclosed in the specification or claims should not be construed as necessarily being in the described order, unless explicitly stated in the individual case or absolutely necessary for technical reasons. Therefore, the foregoing description does not limit the performance of multiple steps or functions to any particular order. Furthermore, in further examples, a single step, function, process, or operation may include and/or be broken down into multiple sub-steps, functions, processes, or operations.

If some aspects in the preceding sections were described in connection with a device or system, these aspects are also to be understood as a description of the corresponding method. For example, a block, a device, or a functional aspect of the device or system can correspond to a feature, such as a method step, of the corresponding method. Accordingly, aspects described in connection with a method are also to be understood as a description of a corresponding block, a corresponding element, a property, or a functional feature of a corresponding device or system.

Reference symbol

10

device

12

interface

14

Processor circuit

16

memory

110

Obtaining environmental perception data

115

Obtaining additional sensor data

120

Processing the environmental perception data and/or additional sensor data

130

Determining one or more numerical parameters

140

Process environmental perception data or data derived from it using an ML model

150

Determining a predictor

160

Providing a warning

170

Restricting the functionality of a vehicle

210

Environmental sensors

215

Environmental perception data

220

Vehicle component

225

data derived from environmental perception data

230

Additional sensor

235

Additional sensor data

240

First ML model

245

Numerical parameters

250

Second ML model

260

Third ML model

270

Predictor

310

Own lane

320

oncoming lane

330

turning lane

340

Other vehicles

350

Whistleblower

360

persons

370

Background building

380

Heaven

QUOTES CONTAINED IN THE DESCRIPTION

This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 2011284304A1 [0004]
• US 5900819A [0004]
• US 8519853B2 [0004]
• US 20100039249A1 [0004]

Claims (10)

A method for predicting fatigue of a driver of a vehicle, the method comprising: Receiving (110) environmental perception data of the vehicle from at least one environmental perception sensor of the vehicle, wherein the environmental perception data represents at least one environment of the vehicle while driving; Processing (120) the environmental perception data to determine (150) a predictive variable representing whether the driver of the vehicle is fatigued; and Providing (160) a warning based on the predictive variable. The procedure according to Claim 1 , wherein the method comprises determining (130) one or more numerical characteristics based on the environmental perception data, and determining the prediction value based on the one or more numerical characteristics. The procedure according to Claim 2 , wherein the one or more numerical parameters comprise at least one of a numerical parameter about driving the vehicle in a central region of the lane, a numerical parameter about a deceleration between braking of a vehicle driving ahead and braking of the vehicle driven by the driver, a numerical parameter about a braking intensity, a numerical parameter about a steering intensity and a numerical parameter about a deviation between an optimal steering trajectory and an actual steering trajectory in a curve. The procedure according to one of the Claims 2 or 3 , wherein determining (150) the predictive value comprises processing the one or more numerical metrics by means of a machine learning model trained to predict the driver's fatigue based on the one or more numerical metrics, the predictive value being based on an output of the machine learning model. The procedure according to one of the Claims 2 or 3 , wherein determining (150) the predictive value comprises comparing the one or more numerical characteristics to one or more threshold values, the predictive value being based on the comparison. The procedure according to one of the Claims 1 until 5 , wherein the method comprises processing (140) the environmental perception data and/or data derived from the environmental perception data by means of a machine learning model, wherein the machine learning model is trained to predict the driver's fatigue based on the environmental perception data and/or data derived from the environmental perception data as input data, wherein the prediction variable is based on an output of the machine learning model. The procedure according to Claim 6 , wherein the machine learning model further processes additional sensor data of the vehicle relating to at least one driving characteristic of the vehicle, such as a speed of the vehicle, as further input data, and/or wherein the method comprises determining (130) one or more numerical characteristics based on the environmental perception data, wherein the machine learning model further processes the one numerical characteristic or the plurality of numerical characteristics as further input data. The procedure according to one of the Claims 6 or 7 , wherein the method comprises processing (140) the data derived from the environmental perception data by means of the machine learning model, wherein the data derived from the environmental perception data originate from at least one of an adaptive cruise control system, a traffic sign recognition system and a lane departure warning system of the vehicle. A program comprising program code for carrying out the method according to any one of the preceding claims when the program code is executed on a computer, a processor, a control module or a programmable hardware component. A device (10) comprising a memory (16), machine-readable instructions, and at least one processor circuit (14) for executing the machine-readable instructions for carrying out the method according to one of the Claims 1 until 8 .