DE102023003826A1 - Method for displaying hidden road users and vehicles - Google Patents

Abstract

The invention relates to a method for displaying hidden road users (1), wherein at least one road user (1) hidden by an object (2) for a vehicle's own within-the-line-of-sight detector is detected by at least one vehicle's own outside-the-line-of-sight detector. Detector (3) is detected in the surroundings of a vehicle (4), and at least the current relative position of the road user (1) relative to the vehicle (4) is determined. The method according to the invention is characterized in that the road user (1) is displayed in contact with his current location on an augmented reality display (5) used in the vehicle (4).

Description

The invention relates to a method for displaying hidden road users according to the type defined in more detail in the preamble of claim 1 and to a vehicle for carrying out the method.

A person driving the vehicle must observe their surroundings while driving in order to identify static and dynamic objects, such as other road users, and to avoid collisions with them. If the person driving the vehicle is distracted and/or there are adverse visibility conditions, the detection ability of the person driving the vehicle may suffer, which increases the potential for an accident. The solution here is driver assistance systems based on sensor-based environmental detection, which are also able to detect surrounding objects and, if necessary, issue information or warnings to the person driving the vehicle or even intervene automatically in the control of the vehicle. For this purpose, sensors are used to generate depth information such as ultrasonic sensors, radar sensors, laser scanners such as LiDARe and cameras.

Such sensors have a certain detection range or a certain field of vision. Only objects within the field of vision can be detected, which is why such sensors could also be referred to as “within-line-of-sight detectors”.

In road traffic, however, situations are often encountered in which road users are obscured by objects such as parked cars, corners of houses, hedges, walls and the like, so that they cannot be detected by the person driving the vehicle or by corresponding within-line-of-sight detectors can be recognized. For example, a child may suddenly run onto the road or a cyclist may drive across the street at excessive speed. This leads to traffic situations with a particularly high risk of accidents.

It is therefore desirable to provide procedures and means with which the risk of accidents can be reduced even in such situations.

So-called “outside-the-line-of-sight detectors”, also referred to as none-line off-side detectors, are known from the prior art. With the help of such detectors it is possible to “see around corners” and thus also be able to recognize road users hidden by objects.

An out-of-sight detector formed by a Doppler radar is known, for example, from: Scheiner, Nikolas, et al.; Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition (CVPR); 2020; pp. 2068-2077. The document describes a method for detecting hidden road users, which can also be used in complex and dynamic traffic situations and is therefore suitable for integration into vehicles such as cars. Road users behind objects can not only be recognized, but also tracked. For this purpose, radar waves are thrown onto objects in the surrounding area, such as walls or parked vehicles, which in turn reflect the radar waves back into the environment. These radar echoes then hit the road user that is actually being recognized, which in turn causes radar echoes that are reflected back to the radar via multiple reflections. The radar used was a so-called frequency-modulated continuous-wave (FMCW) Doppler radar with a multiple-input-multiple-output (MIMO) array configuration. The radar sensor data generated in this way is evaluated using machine learning. However, no detailed structures of an obscured road user can be seen, which would allow a reconstruction of his appearance. However, statements can be made about the approximate dimensions of the road user, which allows a classification into one of the groups, for example: pedestrians, cyclists, cars, trucks or the like.

Furthermore, it is known to use cameras and correspondingly adapted algorithms to analyze the camera images as outside-of-the-line-of-sight detectors, which is known, for example, from: Stanford camera can watch moving objects around corners, July 29, 2019, https://news.stanford .edu/press/view/29312.

Due to the smaller wavelength used compared to radar, detailed information can be obtained, which also makes it possible to determine the specific appearance of an obscured road user.

Furthermore, from the DE 10 2016 220 894 A1 a method and a device for generating a vehicle surroundings view in a vehicle are known. The publication describes a vehicle that records its surroundings with cameras arranged around the circumference. Areas of the captured environment can be obscured by obstacles such as posts. A first camera cannot view a certain area hen, but a second camera, the camera images recorded by the second camera are used to represent the environment. However, if it is not possible to see a specific surrounding area, the corresponding surrounding area can be reconstructed by extrapolating camera images generated by the cameras. The disadvantage, however, is that objects located within a correspondingly covered area cannot be recognized.

The present invention is based on the object of specifying an improved method for displaying hidden road users, with which traffic safety can be further increased.

According to the invention, this object is achieved by a method for displaying hidden road users with the features of claim 1. Advantageous refinements and further developments as well as a vehicle for carrying out the method result from the dependent claims.

A generic method for displaying hidden road users, wherein at least one road user hidden by an object for a vehicle's own within-the-line-of-sight detector is detected in the surroundings of a vehicle by detection by at least one vehicle's own outside-the-line-of-sight detector, and wherein at least the current relative position of the road user relative to the vehicle is determined, is further developed according to the invention in that the road user is displayed in contact with his current position on an augmented reality display used in the vehicle. This makes it possible to improve traffic safety because the person driving the vehicle is already informed about the presence of the hidden road user, even though the person driving the vehicle cannot see the hidden road user even with the naked eye. The person driving the vehicle can adapt their driving behavior accordingly and, for example, reduce the speed of movement of the vehicle, steer it to the side, take their foot off the accelerator pedal and place it on the brake pedal, and the like.

The representation of the hidden road user on an augmented reality display has the further advantage that the road user is displayed directly in the field of vision of the person driving the vehicle and in a contact analogue to his real position. This means that the person driving the vehicle is not only informed about the presence of the hidden road user, but can also locate him relative to the vehicle. This means that the person driving the vehicle can react even more appropriately to the hidden road user.

The vehicle does not necessarily have to have detectors within the line of sight, but the hidden road user only stays in areas that would not be visible through such a detector potentially carried by the vehicle. Of course, the vehicle can also have within-line-of-sight detectors. Ultrasonic sensor systems, radar sensor systems, LiDARe and cameras are particularly suitable for this.

The installation position of the outside-of-line-of-sight detector on the vehicle as well as the installation position or position and orientation of the augmented reality display in the vehicle are also known. Taking into account the head position of the person driving the vehicle and their line of sight, the respective hidden road user can then be displayed in contact with their actual position on the augmented reality display.

By analyzing the sensor data generated by the out-of-sight detector, the dimensions of the road user can at least be estimated. In a preferred embodiment, this makes it possible to adapt the size of the representation of the road user on the augmented reality display to the actual dimensions of the road user in reality, taking into account their distance from the vehicle. For example, a hidden truck can be shown larger than a cyclist and/or a car hidden at a large distance is smaller than a car hidden at a short distance.

The representation of the hidden road user on the augmented reality display can be selected differently depending on different boundary conditions. In particular, the color, contrast, transparency and the like of different parts of the representation can be different or changed depending on the elapsed time or the occurrence of an event. To better identify the hidden road user, for example, the representation can be distorted using one or more filters. For example, by imposing transparency, a ghostly appearance could be simulated.

Sensor data generated by the at least one outside-of-the-line-of-sight detector and optionally sensor data generated by the within-the-line-of-sight detector are processed by a vehicle-internal computing unit tet. Depending on this processing, the respective computing unit then controls the augmented reality display in order to display the hidden road user in a contact-analogous manner to their actual position relative to the vehicle.

An advantageous embodiment of the method according to the invention provides that a Doppler radar is used as an outside-of-the-line-of-sight detector. Accordingly, a program code or algorithm suitable for processing the sensor data generated by the Doppler radar is implemented on the computing unit. This can preferably be based on artificial intelligence, in particular using machine learning models. With the help of a Doppler radar and corresponding sensor evaluation, hidden road users can be detected through reflections of the radar waves from walls, other vehicles and even curbs.

Sensor data generated by the Doppler radar is preferably analyzed using an appropriately trained machine learning model to classify the road user and the road user is represented on the augmented reality display by a class-specific symbolic image. An artificial neural network is preferably used as the machine learning model. Using an artificial neural network trained specifically for this purpose, it is reliably possible to identify characteristic geometric features of the road user based on the available radar signals and to assign the road user to certain classes based on these geometric features, such as a pedestrian, a cyclist, a car, a truck and the like. Using radar waves, the dimensions of the hidden road user cannot be determined exactly, but can only be determined to a comparatively rough approximation. However, this is sufficient for classification into appropriate groups. By assigning a respective group and adapting the representation by selecting the class-specific symbol image, the person driving the vehicle can be informed even more extensively about the hidden road user, since the person driving the vehicle now knows what type of hidden road user it is. For example, if a child is depicted as a road user, the person driving the vehicle would be more likely to expect the child to run into the street than if a truck was depicted, since trucks typically only slowly make their way into an intersection.

The class-specific symbol image can be changed. This means that different symbol images can be kept on the computing unit for one and the same class. A respective symbol image can be selected at random. However, the person driving the vehicle can also make a manual control input to select a specific symbol image in accordance with their preferences. For example, the car could always be a silver Mercedes 300 SL and the bicycle could be a green mountain bike with a male driver in a gray suit and yellow helmet.

A further advantageous embodiment of the method according to the invention further provides that a camera is used as an outside-of-the-line-of-sight detector. Here too, an algorithm specially adapted to evaluate camera images generated by the camera is implemented in the computing unit, which enables hidden road users to be detected. Corresponding algorithms are sufficiently known from the prior art cited at the beginning. Visible light, UV light or infrared light can be used. The scene can be actively illuminated by an ultra-bright light source, such as a laser, to create corresponding shadows/reflections. The shorter wavelength compared to radar waves allows finer structures to be detected, which not only enables road users to be classified into object classes, but also allows the dimensions of the road user to be recorded with sufficient precision so that the road user can be represented in this way on the augmented reality display , what it actually looks like in reality. This only refers to the dimensions of the road user, since the color of the respective surface areas of the road user cannot necessarily be identified from corresponding light reflections.

Radar sensors and cameras can therefore be used both as within-the-line-of-sight detectors and as outside-the-line-of-sight detectors. However, certain adjustments are necessary, such as the use of a specially adapted evaluation algorithm to evaluate the sensor data, and the respective detectors must also have sufficient detection accuracy. In this regard, reference is made again to the sources from the prior art cited at the beginning.

In general, it would be possible for one and the same radar system and one and the same camera to be used simultaneously as an within-line-of-sight detector and an outside-of-line-of-sight detector.

The vehicle may also have multiple Doppler radars as well as multiple cameras. It can also be a Doppler radar with a camera acting as an out-of-line-of-sight detector cooperate. Corresponding sensor data can be merged, allowing even more reliable detection of hidden road users.

According to a further advantageous embodiment of the method according to the invention, the at least one outside-the-line-of-sight detector continuously detects the surroundings of the vehicle and the relative position of the road user is updated after each scanning of the surroundings. This means that the hidden road user can be located and represented not just once in the area around the vehicle, but several times over time. If the position of the vehicle and/or the hidden road user changes, the person driving the vehicle can be informed about the changed relative position.

A movement trajectory for the road user is preferably determined. For this purpose, the difference between the detected relative position of the hidden road user to the vehicle can be tracked over time, and from this in turn it can be determined in which direction and at what speed the hidden road user is moving. For example, it can be determined whether the hidden road user will soon enter the road the vehicle is traveling on.

According to a preferred embodiment of the method according to the invention, this can be used to compare the movement trajectory of the road user and the vehicle and to display only those road users on the augmented reality display for whom there is a risk of a collision with the vehicle. In general, the cognitive processing ability of the person driving the vehicle is limited. Therefore, if possible, only relevant information should be given to the person driving the vehicle in order not to overload the person driving the vehicle mentally. For example, there could be several people and cyclists on a sidewalk obscured by parked vehicles, with the majority of these road users moving parallel to the road. A representation of these road users could quickly overload the augmented reality display and excessively distract the person driving the vehicle. In addition, a representation of these road users is not relevant as there is no potential risk of an accident. Therefore, only those hidden road users who are potentially in danger of colliding with the vehicle are shown. The vehicle and the road user do not necessarily have to meet at the same point in time on their respective trajectories. It may also be sufficient that only the trajectories of the vehicle and the hidden road user intersect so that a respective hidden road user is displayed.

The representation of the hidden road users can also be adapted depending on different boundary conditions. For example, the absolute number of hidden road users could be used as a boundary condition. If, for example, only a few hidden road users are identified, then those hidden road users who are not at risk of colliding with the vehicle can also preferably be shown, since there is no risk that the person driving the vehicle will be mentally overloaded by the representation. This means that the person driving the vehicle is still informed about the general presence of hidden road users, which could lead the person driving the vehicle to adopt a safer or more defensive driving style.

The display could also be adapted depending on the class of road user. For example, pedestrians and cyclists could always be shown, and cars and trucks could only be shown if there was a risk of a collision with the vehicle.

• - die Darstellung des Verkehrsteilnehmers auf der Erweiterte-Realität-Anzeige eingestellt wird;
• - der nicht mehr durch das Objekt verdeckte Teil des Verkehrsteilnehmers von der Darstellung auf der Erweiterte-Realität-Anzeige ausgespart wird und der noch verdeckte Teil des Verkehrsteilnehmers weiterhin dargestellt wird; oder
• - die Darstellung des Verkehrsteilnehmers auf der Erweiterte-Realität-Anzeige fortgeführt wird.

A further advantageous embodiment of the method according to the invention further provides that as soon as the road user enters the field of vision of the person driving the vehicle:

• - the representation of the road user on the augmented reality display is adjusted;
• - the part of the road user that is no longer covered by the object is omitted from the representation on the augmented reality display and the part of the road user that is still covered continues to be shown; or
• - the representation of the road user continues on the augmented reality display.

If the hidden road user comes out from behind the object so that the person driving the vehicle can now see the road user, it is generally no longer necessary to display the road user on the augmented reality display. However, the representation could also be continued to highlight the road user for the person driving the vehicle. For example, the road user could be poorly visible, for example because it is a driving situation at night in a dark environment and the road user is wearing dark clothing. This reduces the risk that the person driving the vehicle will overlook the road user, even if the road user is now in the field of vision of the person driving the vehicle. In one In a particularly preferred embodiment of the method according to the invention, parts of the road user that are already visible are no longer shown on the augmented reality display, but the areas that are still hidden are still shown. This enables a transition from the object into the field of vision, which in particular conveys to the person driving the vehicle that the road user is no longer obscured.

According to a further advantageous embodiment of the method according to the invention, a head-up display, augmented reality glasses and/or a display device is used as an augmented reality display, the camera image stream recorded by a vehicle's own camera being output on the display device, which is overlaid with the representation of the road user in a contact-like manner. A wide variety of output means are therefore available for displaying hidden road users. With the help of head-up displays, a particularly convenient recognition of both the real vehicle surroundings and the virtual representations for the person driving the vehicle is possible. In contrast to using augmented reality glasses, the person driving the vehicle does not have to wear any additional components on their body, which improves comfort. However, using augmented reality glasses or a correspondingly adapted display device also makes it possible for the other vehicle occupants to be able to recognize hidden road users at an early stage. Corresponding display devices can be installed anywhere in the vehicle. For example, the display of an instrument cluster, the display of the head unit, a dedicated passenger display or the like can be used for this purpose.

In a vehicle comprising at least one outside-of-the-line-of-sight detector, an augmented reality display and a computing unit for evaluating sensor data generated by the outside-of-the-line-of-sight detector and for controlling the augmented reality display, according to the invention the outside-the-line-of-sight detector, the augmented reality display and the computing unit are set up to carry out a method described above. The vehicle can be any road vehicle such as a car, truck, van, bus or the like. In general, it could also be a rail vehicle, watercraft or aircraft.

Further advantageous embodiments of the method according to the invention for displaying hidden road users and the vehicle according to the invention also result from the exemplary embodiments, which are described in more detail below with reference to the figures.

• 1 eine schematische Darstellung des der außerhalb-der-Sichtlinie-Detektion zugrundeliegenden Funktionsprinzips; und
• 2 eine schematisierte Darstellung eines Anzeigeinhalts einer Erweiterte-Realität-Anzeige in einer Verkehrssituation zu zwei aufeinanderfolgenden Zeitpunkten.

Show:

• 1 a schematic representation of the functional principle underlying out-of-line-of-sight detection; and
• 2 a schematic representation of a display content of an augmented reality display in a traffic situation at two consecutive times.

1 serves to explain the functional principle of out-of-line-of-sight detection known from the prior art. Shown is a vehicle 4 comprising at least one outside-the-line-of-sight detector 3 and a road user 1 obscured by an object 2. The outside-of-the-line-of-sight detector 3 is, for example, a Doppler radar or a camera. To scan the environment, the outside-of-line-of-sight detector 3 throws electromagnetic beams 8 into the environment. These are reflected by a surface 9 in the surrounding area. The electromagnetic rays 8 are radar waves or light, for example visible light or infrared light or UV light. When using a camera as an out-of-line-of-sight detector 3, a light source can also be provided that throws corresponding electromagnetic radiation 8 into the environment, such as a comparatively strong laser. The reflections thrown back from the surface 9 hit the hidden road user 1 and are in turn reflected by him. The reflections thrown back by the road user 1 hit the surface 9 again and from there return to the vehicle 4 or to the out-of-line-of-sight detector 3. By scanning the environment several times in a row, not only can the whereabouts of the road user 1 be identified, but also its speed and direction of movement. This allows a movement trajectory 7 for the hidden road user 1 to be calculated.

With the help of out-of-line-of-sight detection, hidden road users 1 can be reliably identified. In order to use this function profitably in the vehicle 4, according to the invention the hidden road user 1 is displayed on an augmented reality display 5 (see 2 ), for example in the form of a head-up display, augmented reality glasses or a typical display device. The typical display device shows a camera image stream recorded by the vehicle 4, which is superimposed with the hidden road user 1.

2 shows a traffic situation in which the vehicle 4 drives past parked cars 10. 2a) shows the representation on the augmented reality display 5 at the time while the road user 1 is still completely hidden behind the parked cars 10. 2 B) However, shows the time at which the road user 1 partially emerges from behind the object 2 in the form of a parked car 10.

Using a Doppler radar as an out-of-line-of-sight detector 3, the road user 1 can be assigned to an object class with the aid of an artificial neural network, for example as in 2 shown embodiment of the “cyclist” class. An accurate representation of the road user 1 is not possible due to the resolution of the out-of-line-of-sight detector 3. Instead, a class-specific symbol image 6 is used to represent the hidden road user 1.

At the in 2 B) In the exemplary embodiment shown, the class-specific symbol image 6 is only displayed for that part of the road user 1 that is covered by the object 2. The part of the road user 1 that has emerged behind the object 2 is no longer overlaid with the class-specific symbol image 6. This ensures improved visibility and cognitive processing of the road user 1 entering the field of vision of the person driving the vehicle.

With the help of the method according to the invention and the vehicle according to the invention, traffic safety can be improved, since the person driving the vehicle is reliably made aware of the presence of hidden road users 1. The use of an augmented reality display 5 to display the hidden road user 1 allows a contact-analog representation of the position of the hidden road user 1 at exactly the position where the road user 1 is in reality. This means that the person driving the vehicle is particularly reliably able to correctly assess the traffic situation. This improves road safety even further.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102016220894 A1 [0010]

Claims (10)

Method for displaying hidden road users (1), wherein at least one road user (1) hidden by an object (2) for a vehicle's own within-the-line-of-sight detector is detected by at least one vehicle's own outside-the-line-of-sight detector (3) is recognized in the environment of a vehicle (4), and at least the current relative position of the road user (1) relative to the vehicle (4) is determined, characterized in that the road user (1) is based on an augmented reality used in the vehicle (4). -Display (5) is displayed in a manner analogous to its current position. Procedure according to Claim 1 , characterized in that a Doppler radar is used as an out-of-line-of-sight detector (3). Procedure according to Claim 2 , characterized in that sensor data generated by the Doppler radar are analyzed using a correspondingly trained machine learning model to classify the road user (1) and the road user (1) is represented on the augmented reality display (5) by a class-specific symbolic image (6 ) is pictured. Procedure according to one of the Claims 1 until 3 , characterized in that a camera is used as an out-of-line-of-sight detector (3). Procedure according to one of the Claims 1 until 4 , characterized in that the at least one outside-the-line-of-sight detector (3) continuously detects the surroundings of the vehicle (4) and the relative position of the road user (1) is updated after each scanning of the surroundings. Procedure according to Claim 5 , characterized in that a movement trajectory (7) is determined for the road user (1). Procedure according to Claim 6 , characterized in that the movement trajectory (7) of the road user (1) and the vehicle (4) are compared with one another and only those road users (1) are displayed on the augmented reality display (5) for whom a collision with the Vehicle (4) threatens. Procedure according to one of the Claims 1 until 7 , characterized in that as soon as the road user (1) enters the field of vision of the person driving the vehicle: - the representation of the road user (1) is set on the augmented reality display (5); - the part of the road user (1) that is no longer covered by the object (2) is omitted from the representation on the augmented reality display (5) and the part of the road user (1) that is still covered continues to be shown; or - the representation of the road user (1) is continued on the augmented reality display (5). Procedure according to one of the Claims 1 until 8th , characterized in that a head-up display, augmented reality glasses and/or a display device is used as an augmented reality display (5), the camera image stream recorded by a vehicle's own camera being output on the display device, which is superimposed with the representation of the road user (1) in a contact-like manner. Vehicle (4), comprising at least one outside-of-the-line-of-sight detector (3), an augmented reality display (5) and a computing unit for evaluating sensor data generated by the outside-of-the-line-of-sight detector (3) and for controlling it the augmented reality display (5), characterized in that the outside-of-line-of-sight detector (3), the augmented reality display (5) and the computing unit for executing a method according to one of Claims 1 until 9 are set up.

Images