DE102024003569B3 - Method for visualizing an assistance graphic - Google Patents

Abstract

The invention relates to a method for visualizing an assistance graphic on a display device in a motor vehicle, wherein the assistance graphic is represented as a sequence of individual images which are arranged one another at a frame rate. The method according to the invention is characterized in that the frame rate is dynamically adaptively adjusted, for which purpose a person using the assistance graphic is detected by a driver observation camera and their gaze direction is determined, after which the resulting field of view (14) of the person is assigned to an object (12) shown in the assistance graphic, after which it is detected whether the field of view (14) follows a trajectory (11) of the shown object (12) to which the field of view (14) has been assigned, wherein in the case that the field of view (14) follows the trajectory (11), the fixation of the object (12) along the trajectory (11) is detected, after which an evaluation of the scatter of the fixation is carried out, and after which the frame rate is adjusted depending on whether the scatter exceeds or falls below a predetermined threshold value.

Description

The invention relates to a method for visualizing an assistance graphic according to the type defined in more detail in the preamble of claim 1.

Display devices in a motor vehicle are known from the prior art. It is also known to monitor a person driving a motor vehicle using one or more cameras, for example, to determine their direction of gaze. DE 10 2015 015 136 A1 In this context, a head-up display is described as a display device that adapts the information shown to the actual usage by the person driving the vehicle. Essentially, the idea is that some people prefer more information, while others prefer less. By tracking the person's gaze direction, it's possible to analyze which information the person regularly looks at and uses, and which information they don't. The display can then be controlled accordingly, based on the individual's preferences.

The DE 10 2011 114 059 A1 describes a device for displaying moving images in a motor vehicle, comprising a screen and a control unit that switches between two modes depending on the vehicle speed and the distance to objects ahead, whereby the frame rate is reduced in the second mode and the thresholds for the modes are adjusted by an isotonic function of the distance.

Furthermore, the JP 2015 - 101 189 A A vehicle display that detects when the driver's gaze is not directed at the display and then reduces the light flicker component to improve the visibility of the vehicle display.

It is well known that displaying constantly changing content on a display device is comparatively resource-intensive. This can be particularly problematic in a vehicle, which inherently has certain limitations in its data processing resources. This is especially true for so-called assistance graphics, which, by definition, depict moving objects, such as other road users, and stationary objects, such as roads and buildings, in the vehicle's surroundings. The individual objects are detected, in whole or in part, by the vehicle's environmental sensors and displayed to the driver, for example, on an instrument cluster or a head-up display. The display is presented as several consecutive frames, similar to a video display. The initial refresh rate of the frames is typically specified at 60 Hz. In practice, however, it becomes apparent, for example, in very dynamic scenarios with many moving objects, such as in city traffic, that the display device can no longer display some of the objects due to resource limitations. Then either the number of visualized projects must be reduced in order to be able to display the rest, or the frame rate must be reduced.

Reducing the refresh rate can, however, lead to a jerky display for the person viewing or using the assistive graphics. This is due to the so-called flicker fusion frequency of the affected person, above which they perceive the sequence of images not as a continuous film but as successive individual frames. In practice, this flicker fusion frequency varies considerably, as it is both an individual characteristic of each person and also depends on the time of day, fatigue, and other factors, even for the same person.

The object of the present invention is to provide an improved method for visualizing an assistance graphic in which the refresh rate can be throttled if necessary, but always remains above the current critical flicker fusion frequency of the person viewing or using the assistance graphic.

According to the invention, this problem is solved by a method with the features in claim 1, and in particular in the characterizing part of claim 1. Advantageous embodiments and further developments are described in the dependent claims.

The inventive method provides for the dynamic adaptive adjustment of the frame rate to visualize an assistance graphic as defined above. To prevent a jerky image impression for the person viewing the display, the person using the assistance graphic is detected by a driver observation camera, and their gaze direction is determined. Based on this gaze direction, a viewing area of the person can then be assigned to an object depicted in the assistance graphic. This can also be described as a static correspondence. Subsequently, it is detected whether the viewing area corresponds to a trajectory. The system follows the path of the depicted object to which the gaze has been assigned. This would then be the dynamic correspondence. If this is the case, then the fixation, i.e., the focused viewing of the object along this trajectory, is recorded. Subsequently, the variance of the fixation is evaluated. The variance of the fixation allows conclusions to be drawn as to whether the person using the assistive graphic perceives the current image sequence as a smooth flow, or, with a sufficiently high variance, perceives this sequence as a series of individual images. This leads to a jerkiness in the perceived image sequence. Since the person cannot always directly follow the movements of the object but is sometimes still in the previous position with their gaze when the object has already moved further along the trajectory, this results in a greater variance of the fixation.

According to the invention, the frame rate can now be adjusted depending on whether the variation exceeds or falls below a predetermined threshold. This ensures that the user always sees the assistance graphic in a smooth sequence of images, while simultaneously reducing the frame rate as much as possible, at least for resource-intensive displays, to save resources and to be able to depict as many captured objects as possible with sufficient dynamic range.

The advantage of such adaptive adjustment of the refresh rate to the needs and current situation of the individual user enables exceptionally efficient resource utilization. In particular, resources can be freed up when a low flicker fusion frequency is detected. No human intervention is required; the adaptive dynamic adjustment occurs without any active participation from the person using the assistive graphics.

This method allows for highly realistic assistance graphics, as typically no objects need to be deleted due to resource constraints. The method is independent of the existing scene and automatically takes lighting and weather conditions into account if these affect the current flicker fusion frequency of the person.

The process can be repeated cyclically within a control loop and can be applied to virtually all dynamic display devices, such as screens, head-up displays, and the like.

A particularly advantageous embodiment of the method according to the invention provides that, prior to the method, the driver observation camera is calibrated in a higher-level coordinate system, so that the coordinate systems of the driver observation camera and the vehicle are calibrated relative to each other. If the location of the display device in this higher-level coordinate system is known, e.g., that of the vehicle, it is then possible to determine simply and precisely which of the objects, whose position on the display device and thus in the space of the vehicle is known, the person's gaze is directed at.

Another very advantageous design option involves displaying the assistance graphics at the start of the procedure in an instrument cluster or head-up display using a predefined refresh rate. This predefined refresh rate could, for example, be set at a typical refresh rate of 60 Hz, at which most people perceive the image sequence as smooth.

A further highly advantageous embodiment of the method according to the invention provides that the person's gaze direction is projected onto the display device in order to assign the person's field of vision to one of the displayed objects. Particularly when, due to the calibrated driver observation camera and the matching coordinate systems of the vehicle and the display device on the one hand, and the driver observation camera on the other, the spatial position of the objects on the surface of the display device is known, such a projection of the gaze direction onto the display device, which can be visualized but does not necessarily have to be, allows for the very simple and efficient identification of the object to which the person's gaze is directed.

Another highly advantageous approach involves determining the standard deviation between the object's position and the person's field of vision as a measure of dispersion. According to a particularly beneficial further development, a threshold value on the order of 1 can be used for the standard deviation, allowing for appropriate action depending on whether the determined dispersion value is above or below this threshold.

According to an exceptionally advantageous further development of the inventive method, it is now provided that, in the case where the variation is below the predetermined threshold, the frame rate is maintained or is reduced. Referring to the particularly favorable configuration using the standard deviation with a threshold of 1, this would mean that the described case, in which the refresh rate is maintained or reduced, always occurs when the standard deviation is less than 1.

Maintaining the frame rate is correspondingly simple. However, to free up resources, possibly for subsequent operations, or in case of a current resource shortage, the frame rate can also be reduced. In particular, such a reduction can be performed iteratively, according to a very advantageous advanced technique, in order to gradually approach the threshold and determine the currently minimum possible frame rate in several iterative steps.

Conversely, a highly advantageous embodiment provides that if the dispersion exceeds a predetermined threshold, or, in the described embodiment with the standard deviation, is greater than 1, the frame rate is increased. In principle, it would of course be conceivable to increase it directly to the initially predetermined value or a maximum value to be on the safe side in any case. However, it is more resource-efficient if, according to a highly advantageous embodiment of the inventive method, the increase is carried out iteratively in order to approach the minimum required frame rate for displaying a smooth motion sequence, as already explained above – but now from the other direction.

In particular, but not exclusively, in the case of iterative adjustment, a highly advantageous further development may entail repeating the process with the newly adjusted frame rate as the new initial frame rate. The process is thus repeated to ensure continuous adaptation to the current conditions. What does not need to be repeated is the upstream step of calibrating the driver monitoring camera.

Further advantageous embodiments and developments of the method according to the invention also become apparent from the exemplary embodiment, which is described in more detail below with reference to the figures.

• 1 ein schematisches Ablaufdiagramm einer möglichen Ausführungsform des erfindungsgemäßen Verfahrens;
• 2 eine Darstellung gemäß des Verfahrensschritts VS 7 aus der Darstellung der 1; und
• 3 eine Darstellung der Detektion gemäß des Verfahrensschritts VS 8 aus der Darstellung gemäß 1.

This shows:

• 1 a schematic flowchart of a possible embodiment of the method according to the invention;
• 2 a representation according to procedure step VS 7 from the representation of the 1 ; and
• 3 a representation of the detection according to procedure step VS 8 from the representation according to 1 .

The core of this description is to adapt the visualization of an assistance graphic in the aforementioned sense to the resource load in such a way that a comfortable visualization is still guaranteed, taking into account the individual critical flicker fusion frequency.

As shown in the depiction of the 1 As illustrated by an exemplary flow diagram, in a first upstream process step VS 1 a driver observation camera is calibrated in a higher-level coordinate system, in particular the coordinate system of the vehicle, whereby the intrinsic and extrinsic position in relation to a coordinate origin within the vehicle is known.

In a further process step, VS 2, the assistance graphic is generated based on data from the environmental sensors and map data and displayed on a display device. This makes it known which objects, with which position and movement trajectory, are visualized on the display device, for example, an instrument cluster, a head-up display, or the like.

In the subsequent third process step, VS 3, the gaze direction of a person using the assistance graphic is recorded and analyzed. This person is typically the driver of the vehicle. The recording and analysis of gaze directions is a well-established technique, so the specific eye-tracking algorithms used (Purkinje Reflection, Iris Silhouette Tracking, etc.) need not be discussed further. This allows for the detection of saccades (gaze movements) and fixations (gaze fixations). Based on the three-dimensional relationship between the driver observation camera and the vehicle's coordinate system, established during calibration, the person's field of view can be projected onto the display device in a fifth process step, VS 5. The position and trajectory of the objects in this display were previously stored in a fourth process step, VS 4.

This allows the saccade, and especially the person's fixation, to be assigned to a specific visualized object, which, as mentioned above, is also referred to as static correspondence. This is done in the representation of the 1 In process step VS 6, the object's movement, i.e., its trajectory, is compared with the eye movements for a predetermined number of frames in the subsequent process step VS 7. If the object's movement vectors and the person's eye movements are similar, a dynamic correspondence is assumed. The person then observes the identified visualized object and follows its trajectory with their gaze. If no similar movement vectors are detected, the assignment according to process step VS 6 is checked again using a different object. Ultimately, process steps VS 6 and VS 7 are repeated until sufficiently similar movement vectors between the object's trajectory and the person's eye movements are identified.

In the subsequent crucial step of the procedure, VS 8, the scatter of the object's fixation is evaluated along its trajectory in relation to the object's trajectory. A high scatter suggests that the person—as described above—does not perceive the visualization as a continuous sequence of images, but rather at least partially resolves individual frames, resulting in a jerky display.

If this variation exceeds a defined threshold, a control unit for the display device is instructed to increase the refresh rate in iterative steps, as the current refresh rate setting is potentially below the flicker fusion frequency currently present in the individual. This allows for adaptive adjustment of the refresh rate to the individual and highly variable flicker fusion frequency of each person at any given time. This enables resource-efficient visualization that is perceived as good by the individual. This approach also allows for the visualization of highly realistic assistive graphics, as increased resource load does not necessarily lead to a reduction in the number of objects displayed. As explained above, resources can be freed up by reducing the refresh rate down to the minimum refresh rate at which the individual still perceives the process as smooth.

The aforementioned threshold comparison is visualized in the representation of the 1 The process step VS 9 and the process step VS 10 show the reduction or increase of the frame rate.

The entire process between steps VS 2 and VS 10 is then repeated cyclically, ensuring continuous adjustment. For this purpose, step VS 2 is started with the previously determined new frame rate as the initial frame rate for the next process run.

In the presentation of the 2 Two exemplary cases will now be used to demonstrate how the dispersion of fixation can behave. In the representation of the 2a) The trajectory 11 of an object labeled 12 is shown. The object 12 is depicted multiple times along its trajectory 11. In the vertical direction, various so-called frames of the representation are schematically indicated and labeled with the corresponding times T0 to T4. Within each of the multiple depictions of object 12, a filled dot labeled 13 is visible at the bottom of the object 12, representing the center of the object 12 on its trajectory 11. Additionally, an unfilled circle marks the gaze area 14 of the person, i.e., the area on which the person is currently looking in the current frame or subframe. The distance shown by the thin line between the center 13 of object 12 and the gaze area 14 symbolizes the dispersion of the fixation of the gaze area 14 relative to the actual position of object 12. 2b) This is shown again on the right.

This is in the 2a) It can be seen that the gaze areas 14 largely follow the objects 12. The dispersion of fixation is low. In the representation of the 2b) It can be seen, however, that several viewing areas 14 are assigned to one object position, while no viewing area 14 is assigned to other positions of the object 12. One of the two viewing areas 14 at the second object position from the bottom therefore actually belongs to the third object position above it. The dispersion between the viewing area 14 on the one hand and the center 13 of the object 12 on the other is thus significantly higher here.

The presentation of 3 This is addressed accordingly. The unfilled bars, which represent the time axis on the x-axis, indicate the... 2a) The case shown illustrates how the dispersion, which is represented on the y-axis by the distance between the center 13 of the object 12 and the viewpoint 14, is varied. is equally low. In each of the individual frames, the case of the hatched bars is also shown. 2b) shown, whereby it can be seen that the deviation here is analogous to the representation in 2b) , between the viewing area 14 and the center 13 of the object 12 in the third and fifth representations is very large.

This graphic can therefore be used to illustrate the first in 2a) In the case presented, a rather low dispersion is very easily perceived; for the second case in the 2b) a very high dispersion. In the embodiment shown here, the standard deviation for the case according to 2a) at approximately 0.4, while the standard deviation for the case according to 2b) 1.9.

In the case according to 2a) The refresh rate can be reduced, whereas in the case according to 2b) The level needs to be increased to visualize a smooth-looking representation of the assistance graphic for the person.

Claims (13)

A method for visualizing an assistance graphic on a display device in a motor vehicle, wherein the assistance graphic is displayed as a sequence of individual images which follow one another at a specific frame rate, characterized in that the frame rate is dynamically adaptively adjusted, for this purpose a person using the assistance graphic is detected by a driver observation camera and their gaze direction is determined, after which the resulting field of view (14) of the person is assigned to an object (12) displayed in the assistance graphic, after which it is detected whether the field of view (14) follows a trajectory (11) of the displayed object (12) to which the field of view (14) has been assigned, wherein, in the case that the field of view (14) follows the trajectory (11), the fixation of the object (12) along the trajectory (11) is detected, after which an evaluation of the scatter of the fixation is carried out, and after which the frame rate is adjusted depending on whether the scatter exceeds or falls below a predetermined threshold value. Procedure according to Claim 1 , characterized in that the method is preceded by a calibration of the driver observation camera in a higher-level coordinate system of the vehicle. Procedure according to Claim 1 or 2 , characterized in that a visualization of the assistance graphic at the beginning of the procedure takes place in a combination instrument or a head display as a display device using an initially specified refresh rate. Procedure according to Claim 3 characterized in that the initial refresh rate is set to approximately 60 Hz. Procedure according to one of the Claims 1 until 4 , characterized in that a projection of the person's gaze direction into the visualization of the assistance graphic on the display device takes place in order to assign the person's field of vision (14) to one of the depicted objects (12). Procedure according to one of the Claims 1 until 5 , characterized in that the standard deviation between the position of the object (12) and the viewing area (14) of the person is determined as a measure of dispersion. Procedure according to Claim 6 , characterized in that the threshold for the standard deviation is set at approximately 1. Procedure according to one of the Claims 1 until 7 , characterized in that, in the event that the variation is below the specified threshold, the refresh rate is maintained or reduced. Procedure according to Claim 8 characterized in that the reduction only occurs if the available resources are utilized beyond a predetermined limit. Procedure according to Claim 8 or 9 characterized by the fact that the reduction takes place in iterative steps. Procedure according to one of the Claims 1 until 10 , characterized in that , in the event that the variation is above the specified threshold, the refresh rate is increased. Procedure according to Claim 11 characterized by the fact that the increase occurs in iterative steps. Procedure according to one of the Claims 1 until 12 , characterized in that the procedure is repeated with the adapted frame rate as the new initial frame rate.