DE102023107897A1 - METHOD AND SYSTEM FOR ASSISTANCE TO A DRIVER - Google Patents

Abstract

A method for assisting a driver (4) when driving a vehicle (1) with impaired visibility (7). The method comprises detecting an object (6) in an area of a path ahead of the vehicle (1) by means of at least one detection device (2, 3) of the vehicle (1), recognizing the object (6) and determining a graphic representation (8) of the object (6) which represents the recognized object (6), and displaying the graphic representation (8) of the object (6) on a display device (5), wherein the graphic representation (8) is assigned to the actual view of the object (6) in reality in order to visually highlight the object (6) in the area of the path ahead. The display can take place as an augmented reality representation in a mobile terminal, such as AR glasses (5).

Description

The present invention relates to a method and a system for assisting a driver when driving a vehicle. In particular, driving-related information is displayed in a display device, for example in an augmented reality display. The display device can be part of a mobile terminal, such as AR glasses.

In various situations while driving, i.e. while driving a vehicle, it may happen that objects or conditions in the environment, in particular the path ahead of the vehicle, are not perceived by the driver, are not perceived sufficiently, or are perceived too late to be able to react appropriately. This may be the case, for example, if other road users, such as other vehicles, pedestrians and the like, are not perceived by the driver, are only perceived weakly, or are perceived too late due to poor visibility (e.g. darkness or bad weather conditions such as fog or rain) or because they are at least partially obscured by other objects or road users. Other circumstances, such as environmental conditions, may not be recognizable to a driver at all. This can therefore lead to a dangerous situation in road traffic.

Modern vehicles have a range of driver assistance systems with corresponding sensor devices, such as radar systems or optical systems (cameras), which can detect such situations better or earlier than the driver (or even at all). The vehicle can then either react automatically, e.g. brake or steer, which can be irritating for some drivers. A warning can also appear, for example, on a display of the infotainment system or on the dashboard, or be given acoustically. However, such warnings are often not immediately understandable or difficult to assign to the driver and he may have to take his eyes off the road to see the message on a display, which in turn can pose a safety risk. This can also result in a longer reaction time.

Independent of vehicles, there are various widely used (mobile) devices that are intuitively operated and mastered by users in everyday life, such as smartphones. However, content and information from vehicle systems, especially driver assistance systems, are typically not suitable for playback on such devices, not least because in many cases the connection between the vehicle and the device is not sufficiently compatible, which is often limited to other non-driving-related content such as telephone or music playback.

Another well-known way of displaying digital content is augmented reality (AR). Using AR, virtual objects can be shown so that they appear to be in certain positions in reality. The display of virtual objects is also used in vehicles, for example to display navigation instructions, such as turning information on the road in front of the vehicle. Usually, however, the virtual objects are simply superimposed on reality, be it on a video image or in so-called head-up displays, where the virtual object is displayed in the windshield of a vehicle. If the virtual object is not only to be superimposed on reality, but also to appear in a position that is assigned to a position in reality, measures are required that allow reality to be captured with depth information. In this way, a realistic impression of the position of the virtual object can then be created.

AR content can also be displayed on a user's device, such as a smartphone or special glasses ("AR glasses"). However, as mentioned above, this requires a compatible transmission of data from, for example, a driver assistance system to the device. On the other hand, the user's AR-capable device must be located or localized ("tracked") in the vehicle in order to enable a meaningful AR display of driving-related information on the device, which is often not possible with smartphones, for example. With AR glasses, tracking seems to be possible, for example, in the interior of a vehicle.

The present invention is based on the object of providing an improved method for assisting a driver when driving a vehicle. In particular, driving-related information is to be displayed in a display device, in particular in an augmented reality display.

This object is achieved according to the teaching of the independent claims. Various embodiments and developments of the invention are the subject of the subclaims.

A first aspect of the invention relates to a, in particular computer-implemented, method for assisting a driver when driving a vehicle. The method comprises detecting an object in an area of a path of travel ahead of the vehicle by means of at least one detection device of the vehicle, recognizing the object and determining a graphical representation of the object which represents the recognized object, and displaying the graphical representation of the object on a display device, wherein the graphical representation is assigned to the actual view of the object in reality in order to visually highlight the object in the area of the path of travel ahead.

The aforementioned method according to the first aspect is therefore based in particular on the fact that an object which is located in the area of a path ahead of the vehicle, i.e. in front of the vehicle in the direction of travel, is detected by means of a detection device of the vehicle. Modern vehicles are often equipped with such detection devices as part of driver assistance systems in order to, as explained at the beginning, detect possible collision risks in good time, for example when visibility is poor and the driver himself would not detect the object, would only detect it poorly or too late. The object is then detected in order to highlight it for the driver in the area of the path ahead by displaying a graphic representation or even to make it recognizable in the first place.

When the graphic representation is displayed, it is assigned to the object to improve comprehensibility. This gives the driver the opportunity to react appropriately to a potentially dangerous situation, so that the procedure can increase safety when driving a vehicle. In particular, by displaying the graphic representation, the driver receives a clear and understandable overview of the current situation in road traffic.

The term “vehicle” as used here refers in particular to a passenger car, including all types of motor vehicles, hybrid and battery-powered electric vehicles, as well as vehicles such as vans, buses, trucks, delivery vans and the like, as well as two-wheelers.

The term “road ahead” used here refers in particular to an area of a route in the direction of travel in front of the vehicle during the journey, which the vehicle will travel on if it continues to travel with the currently prevailing driving parameters or also with changed parameters (such as acceleration, braking, steering, etc.). Accordingly, this does not only apply to the section of a route directly ahead of the vehicle, but can also include curves, turning, lane changes and the like. The term “area of the road ahead” is to be understood accordingly, which also refers in particular not only to the same lane in front of the vehicle, but also adjacent lanes, the roadside, sidewalks or cycle paths in the area of the road ahead and the like. This also includes areas in front of the vehicle that could possibly be reached, for example, if the vehicle leaves the road or the like.

The term "augmented reality" (AR) used here refers in particular to the visual representation of digital content which is superimposed on or displayed in a view of actual reality as a virtual extension or supplement. In particular, the virtual displays appear at a specific location in reality and can therefore be linked to a point in reality. For this purpose, a display with depth information can be recorded. In other words, the geometry and position in space of the environment with its objects, in particular in front of the vehicle, is recorded. This can also be referred to as a depth map and at least partially represents a three-dimensional model of the environment of the motor vehicle. The depth information can be used to determine the position at which the virtual object should be placed in the augmented reality.

The term "graphical representation" as used here refers in particular to a representation of an object of reality. The representation can be graphic, for example a simplified view of the actual view, a symbol, an animation, a (colored) marking, a border, a highlight and the like. The graphical representation allows a quick and understandable perception of an actual object of reality.

The term "object" as used here refers in particular to an object in road traffic, for example another road user. The term "movable object" as used here, if applicable, then refers in particular to an object in the environment that is not stationary and is therefore only temporarily or not permanently present in the environment. This includes, for example, vehicles, both in moving traffic and in stationary traffic (traffic jams, traffic lights, etc.) as well as parked vehicles. Pedestrians can also be moving objects. In contrast to moving objects, the term "not The term "movable object" refers in particular to stationary objects. These can be, in particular, buildings, roads, public transport stops, walls, fences, trees, bushes, hedges and the like.

The term "scanning device" used here refers in particular to a device that uses suitable technology, such as sending and receiving electromagnetic radiation, to scan the surroundings of a vehicle, in particular the area of the vehicle's path ahead. Radar or lidar systems can be used for this purpose, for example. The scanning device can also be able to create a map with depth information in order to enable an AR representation as described above.

The term "image capture device" used here refers in particular to a camera, in particular a digital camera. The camera can take still images (photos) or moving images (videos). The image capture device can take such an image and capture corresponding image data.

The term "display device" used here refers in particular to a device that is set up to display content digitally, such as a display, a monitor or any other screen. The display device can also be at least partially transparent so as not to block a view of reality through the display device. For example, AR glasses are known which have transparent lenses but which simultaneously function as a display device in order to be able to display content using augmented reality.

As used herein, the terms "comprises," "includes," "has," "includes," "comprising," "having," "with," or any other variation thereof are intended to cover non-exclusive inclusion. For example, a method or apparatus that comprises or has a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or that are inherent in such method or apparatus.

Furthermore, unless explicitly stated to the contrary, "or" refers to an inclusive "or" and not an exclusive "or". For example, a condition A or B is satisfied by one of the following conditions: A is true (or exists) and B is false (or absent), A is false (or absent) and B is true (or exists), and both A and B are true (or exists).

As used herein, the terms "a" or "an" are defined to mean "one or more". The terms "another" and "another" and any other variation thereof are defined to mean "at least one other".

The term “plurality” or “multiplicity” as used here should be understood to mean “two or more”.

Measure words such as “first”, “second” (and variations thereof), as used here, serve only to distinguish, for example, different points in time, and do not necessarily indicate an order, for example a chronological order.

The term "configured" or "set up" to perform a specific function (and respective modifications thereof) is to be understood in the sense of the invention that the corresponding device is already in a design or setting in which it can perform the function or that it is at least adjustable - i.e. configurable - so that it can perform the function after the corresponding setting. The configuration can be carried out, for example, by setting parameters of a process sequence or switches or the like to activate or deactivate functionalities or settings. In particular, the device can have several predetermined configurations or operating modes, so that configuration can be carried out by selecting one of these configurations or operating modes.

Preferred embodiments of the method are described below, which can be combined with each other as well as with the other aspects of the invention described, unless this is expressly excluded or is technically impossible.

In some embodiments, the detection device comprises a scanning device of the vehicle, which at least partially scans the vehicle's path ahead in order to detect the object in the area of the path ahead and to determine at least one position of the object relative to the vehicle. In contrast to optical systems, the scanning device can detect objects even in poor visibility conditions or at a great distance. For example, the position, size and/or distance (relative) to the vehicle It can also be detected whether a detected object is moving and, if necessary, a movement relative to the vehicle can be determined. In this way, a view with depth information can be created, which can be used for an augmented reality representation. The scanning device can in particular comprise at least one radar sensor. Alternatively or additionally, the scanning device can also comprise at least one lidar sensor.

In some embodiments, the at least one detection device further comprises at least one image capture device. The image capture device and optionally a scanning device can be provided in one unit or as separate devices in the vehicle. The method then further comprises capturing image data on the area of the road ahead using the image capture device, evaluating the image data in order to recognize the object in the image data and to determine the recognizability of the object, how strongly the object stands out visually from its surroundings, and displaying the graphic representation of the object in the event that the recognizability of the object is less than a predetermined threshold value.

In other words, the graphic representation should only be displayed if it is highly likely that the object is not visible to the driver or can only be seen with difficulty. This can be determined, for example, if an object detected by the scanning device has little or no visual contrast with its surroundings, for example if a vehicle ahead is not well lit in fog. The graphic representation should not, however, be displayed if the detectability is greater than or equal to the specified threshold value, i.e. if the object can be seen well or at least sufficiently with the naked eye. In this case, displaying the graphic representation would be unnecessary or even disruptive. The driver therefore only receives support when necessary and is not disturbed, for example, in normal or good visibility conditions.

In some embodiments, the graphic representation is displayed in an augmented reality representation. As explained above, the graphic representation is superimposed on the actual reality at a suitable point. It is therefore easy for the driver to perceive it without, for example, having to take his eyes off the road. A representation of the surroundings with depth information can be obtained in different ways. As mentioned, the depth information can be captured by means of the scanning device that scans the surroundings. Alternatively or additionally, depth information can be determined from a video image using computer vision and machine learning, such as deep learning, in particular based on an artificial neural network. A combination of different methods for determining the depth information is also conceivable.

In some related embodiments, the graphical representation symbolizes the actual view of the object in reality, wherein the graphical representation is displayed superimposed on the actual view of the object in the augmented reality representation. This improves comprehensibility for the driver and thus increases safety when driving. For example, a symbol of a vehicle can be superimposed on a vehicle driving ahead, adjusted accordingly in terms of size and position, possibly in a highlighting color to improve recognizability.

In some embodiments, the display device is part of a mobile terminal that is not part of the vehicle. The mobile terminal can be AR glasses that are worn by the driver while driving. This creates a connection between the vehicle sensors and a mobile terminal that the driver may also use outside the vehicle in other contexts and is therefore familiar with its operation. The method thus simplifies driver support because no special equipment that can only be used in the vehicle is required. Alternatively or additionally, the display device can also be part of the vehicle. This can be a display of an infotainment system in which the graphic representation is superimposed on a video image, or other displays, for example in the windshield (so-called head-up displays), which allow the graphic representation to be displayed in the driver's field of vision of the road ahead.

In some embodiments, the object comprises a moving or immobile object in road traffic in the area of the vehicle's path ahead. This can in particular be a vehicle driving ahead. Pedestrians, trees, buildings and the like can also be considered objects within the meaning of the present disclosure (see above).

In some embodiments, the vehicle further comprises at least one communication device for direct communication with at least one other vehicle in the surrounding area. The method then further comprises receiving a signal by means of the communication device from at least one vehicle in front. vehicle in the area of the road ahead, which contains at least information about the position of the vehicle in front relative to the vehicle and/or the absolute position of the vehicle in front in world coordinates. Such communication is also known as V2V ("vehicle-to-vehicle") or more generally as V2X. A vehicle, such as a vehicle in front, can send information directly to another vehicle, such as a vehicle behind, in particular wirelessly, such as via WLAN or cellular network. In this way, even more precise information about the area of the road ahead can be obtained. A vehicle can send various data, such as its speed, type of vehicle and the like, to its surroundings, in particular to other vehicles in the area.

A second aspect of the invention relates to a system for data processing, comprising at least one processor configured to carry out the method according to the first aspect of the invention.

In some embodiments, the system further comprises a mobile terminal comprising a display device configured to display the graphical representation of the object.

In some embodiments of the system, the mobile terminal is configured to be located in the interior of the vehicle, and the display device is configured to display the graphical representation of the object in an augmented reality representation. The mobile terminal can in particular be AR glasses that the driver can wear while driving.

A third aspect of the invention relates to a computer program with instructions which, when executed on a system according to the second aspect, cause the system to carry out the method according to the first aspect.

The computer program can in particular be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloaded via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. In addition, the computer program can have a plurality of interacting individual program modules.

The system according to the second aspect can accordingly have a program memory in which the computer program is stored. Alternatively, the system can also be set up to access an external computer program, for example on one or more servers or other data processing units, via a communication connection, in particular to exchange data with it that is used during the execution of the method or computer program or that represents outputs of the computer program.

The features and advantages explained with respect to the first and second aspects of the invention also apply accordingly to the other aspects of the invention.

Further advantages, features and possible applications of the present invention will become apparent from the following detailed description taken in conjunction with the drawings.

• 1 schematisch ein Fahrzeug mit einem vorausfahrenden Fahrzeug bei schlechten Sichtverhältnissen;
• 2 die Situation aus 1 aus Sicht des Fahrers; und
• 3 die Sicht des Fahrers aus 2 in einer AR-Ansicht mit eingeblendeter grafischer Darstellung des vorausfahrenden Fahrzeugs.

It shows:

• 1 schematic of a vehicle with a vehicle ahead in poor visibility conditions;
• 2 the situation 1 from the driver's perspective; and
• 3 the driver's perspective 2 in an AR view with a graphic representation of the vehicle in front.

Throughout the figures, the same reference numerals are used for the same or corresponding elements of the invention.

Using the method of the present invention, an object in the area of a vehicle's path ahead can be highlighted by displaying a graphic representation to increase safety when driving. Moving and stationary objects can be detected and recognized, which are difficult or impossible for the driver to see, for example, due to poor visibility. Poor visibility can occur due to weather conditions such as fog or rain or other factors such as darkness or glare.

In 1 A vehicle 1 with a driver 4 is shown. In front of the vehicle 1 a vehicle 6, which is difficult to see due to fog 7. The vehicle 1 is equipped with a scanning device 3, which scans the area of the road ahead and can thereby create a view with depth information. The scanning device 3 can be a radar system, for example. Furthermore, an image capture device 2 is provided in the vehicle 1, which supplies image data on the road ahead and thus also on the vehicle 6 driving ahead.

The in 1 The traffic situation shown is in 2 shown from the perspective of the driver 4. The vehicle 6 is difficult to recognize. Here, the data recorded by the scanning device 3 and the image capture device 2 can be compared. In particular, it can be determined that the scanning device 3 detects the vehicle 6, whereas the image capture device 3 hardly perceives the vehicle 6, for example due to low contrast.

In order to highlight the vehicle 6 for the driver 4, a graphic representation 8 of the vehicle 6 is superimposed on the actual reality, ie the field of vision of the driver 4. As in 3 shown, the graphic representation 8 is displayed in the form of a vehicle symbol. This is done in an augmented reality representation, ie the graphic representation 8 is superimposed on the actual view of the vehicle 6 in terms of position and size. This can be done in a suitable display device, such as AR glasses 5 (see 1 ) that the driver wears while driving. The graphic representation 8 can be displayed in a suitable color to distinguish the representation 8 from the actual reality. This improves the warning function of the display of the graphic representation 8 and allows the driver 4 to better assess the traffic situation. In order to correctly display the augmented reality, the AR glasses 5 are located ("tracked") in the interior of the vehicle 1.

While at least one exemplary embodiment has been described above, it should be noted that a wide variety of variations thereon exist. It should also be noted that the exemplary embodiments described are merely non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide one skilled in the art with guidance for implementing at least one exemplary embodiment, it being understood that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from the subject matter set forth in the appended claims, as well as their legal equivalents.

REFERENCE SYMBOL LIST

1

vehicle

2

image capture device

3

scanning device

4

driver

5

AR glasses

6

vehicle ahead

7

Fog

8

graphical representation

Claims (15)

Method for assisting a driver (4) when driving a vehicle (1), the method comprising: - detecting an object (6) in an area of a path ahead of the vehicle (1) by means of at least one detection device (2, 3) of the vehicle; - recognizing the object (6) and determining a graphic representation (8) of the object (6) which represents the recognized object (6); and - displaying the graphic representation (8) of the object (6) on a display device (5), the graphic representation (8) being assigned to the actual view of the object (6) in reality in order to visually highlight the object (6) in the area of the path ahead. procedure according to Claim 1 , wherein the detection device comprises a scanning device (3) of the vehicle (1), which at least partially scans the path of travel ahead of the vehicle (1) in order to detect the object (6) in the region of the path of travel ahead and to determine at least one position of the object (6) relative to the vehicle (1). procedure according to claim 2 , wherein the scanning device (3) comprises at least one radar sensor. Method according to one of the preceding claims, wherein the at least one detection device further comprises at least one image detection device (2) and the method further comprises: - detecting image data on the area of the roadway ahead by means of the image detection device (2); - evaluating the image data in order to detect the object (6) in the to recognize image data and to determine a recognizability of the object (6), how strongly the object (6) stands out optically from its surroundings; and - displaying the graphic representation (8) of the object (6) in the event that the recognizability of the object (6) is less than a predetermined threshold value. Method according to one of the preceding claims, wherein the display of the graphical representation (8) takes place in an augmented reality representation. procedure according to claim 5 , wherein the graphical representation (8) symbolizes the actual view of the object (6) in reality and is displayed superimposed on the actual view of the object (6) in the augmented reality representation. Method according to one of the preceding claims, wherein the display device is part of a mobile terminal (5) which is not part of the vehicle. procedure according to claim 7 , wherein the terminal is an AR glasses (5) which is worn by the driver while driving. Method according to one of the preceding claims, wherein the object (6) comprises a moving or immovable object in road traffic in the region of the path ahead of the vehicle (1). Method according to one of the preceding claims, wherein the vehicle (1) further comprises at least one communication device for direct communication with at least one other vehicle in the surroundings, wherein the method further comprises: - receiving a signal by means of the communication device from at least one preceding vehicle in the area of the route ahead, which signal contains at least information about the position of the preceding vehicle relative to the vehicle and/or the absolute position of the preceding vehicle in world coordinates. A data processing system comprising at least one processor configured to carry out the method according to any one of the preceding claims. system according to claim 11 , further comprising a mobile terminal (5) which comprises a display device which is configured to display the graphical representation (8) of the object (6). system according to claim 11 or 12 , wherein the mobile terminal is configured to be located in the interior of the vehicle (1), and the display device is configured to display the graphical representation (8) of the object (6) in an augmented reality representation. System according to one of the Claims 11 until 13 , wherein the mobile device is AR glasses (5). Computer program containing instructions which, when executed on a system, execute one of the Claims 11 until 14 cause the procedure to be carried out in accordance with one of the Claims 1 until 10 to execute.

Images