DE102022107297A1 - Method and device for operating data glasses in a mobile device for displaying environment-related display objects - Google Patents

Abstract

The invention relates to a method for operating a display system (1) with data glasses (3) in a vehicle (5), with the following steps: - Detecting (S1) a vehicle pose based on an environmental coordinate system (U); - Determining (S2) a Glasses pose in a vehicle coordinate system (F), - Determining (S3) an environment-related glasses pose in the environment coordinate system (U); - Transmitting (S4) the environment-related glasses pose to the data glasses (3).

Description

The invention relates to display systems with data glasses, in particular for use in a vehicle. The invention further relates to measures for operating data glasses, in particular for displaying display objects depending on the viewing direction.

Data glasses, also called head-mounted displays, are known which, using a display device, can display an image on one or two display surfaces in the field of vision of the wearer of the data glasses. The display surfaces can correspond to reflective surfaces that direct images into the eyes of the wearer of the data glasses. The viewing openings of the data glasses are transparent, so that the real surroundings can also be perceived in the usual way through the display surfaces of the data glasses. The display surfaces are located in the viewing openings so that information to be displayed, such as text, symbols, graphics, video displays and the like, can be displayed overlaying the perception of the surroundings.

The information can usually be presented to the wearer of the data glasses in a contact-like manner, i.e. H. be displayed in such a way that they are superimposed as display objects on a specific assigned environmental object in the real environment or are oriented to this or that the display object to be displayed is displayed in a specific orientation of the data glasses or their wearer. Furthermore, the display object to be displayed in a contact-analog manner can be displayed in such a way that it appears perspectively correct in relation to the surrounding object in the real environment, i.e. H. The illusion is created that the environmental object of the real environment has actually been supplemented by the additional feature of the visual display object.

However, in order to display the display object in a contact-like manner on the display surfaces of the data glasses, it is necessary to know the position of the surrounding object in the environment and the user's viewing direction. When wearing the data glasses, the user's viewing direction is permanently assigned to their pose, i.e. H. the 3D position and the 3D orientation of the data glasses correspond to six degrees of freedom (6 DoF, Degrees of Freedom).

In particular, when using data glasses in a vehicle, a pose recognition unit can be provided in the data glasses to determine the pose of the data glasses. The pose recognition unit usually has a data glasses camera and a computing device, e.g. B. in the form of a microprocessor. With the help of images recorded by the data glasses camera in the field of vision of the wearer of the data glasses, the pose of the data glasses in the vehicle interior, i.e. with respect to a vehicle coordinate system, can be determined based on markers of a known position or other structures of a known position. The markers or structures are permanently arranged outside the glasses in the interior of the vehicle or are displayed there. This process is also called inside-out tracking.

Furthermore, the pose of a data glasses can also be determined by an external pose recognition unit (outside-in tracking method), in which an interior camera records the head of the wearer of the data glasses and, by evaluating the camera image, either determines the pose of the head and from that the pose of the Data glasses are derived or the pose of the data glasses is determined directly. With these so-called outside-in tracking systems, one difficulty lies in transmitting the absolute pose information determined outside the data glasses to the data glasses with a sufficiently low latency, especially in the case of a wireless communication connection, so that the data glasses provide corresponding contact-analog representations without delay or only with a non-disturbing delay.

Vehicle-fixed contact-analog display objects can be displayed in the data glasses with low latency using glasses poses determined using camera-based inside-out tracking methods. However, inside-out tracking systems can only track on previously known markers fixed to the vehicle, since tracking on surrounding objects would only be possible with very complex object recognition processes due to the high variability of the environment. However, the computing capacity in the data glasses and the equipment of the data glasses is generally not sufficient to determine data glasses poses based on a world-fixed environmental coordinate system/world coordinate system through pattern recognition of environmental objects. Therefore, previous inside-out tracking methods are not suitable for determining a glasses pose of the data glasses in relation to a globally fixed environmental coordinate system or in relation to environmental objects in the vehicle surroundings.

In order to display global contact-analog display objects in the data glasses, it makes sense to localize the display objects in the environmental coordinate system for display with the data glasses.

It is the object of the present invention to provide an improved method for operating a display system with which a consistent Clock-analog representation of a display object with respect to an environmental position in a world-fixed environmental coordinate system is possible in an improved manner. In particular, tracking of the display object to be displayed with reference to the surrounding position should be possible with as little latency as possible.

Disclosure of the invention

This object is achieved by the method for operating a display system in a vehicle according to claim 1 as well as a display system, data glasses and an assistance system according to the independent claims.

Further refinements are specified in the dependent claims.

• - Erfassen einer Fahrzeugpose bezogen auf ein Umgebungskoordinatensystem;
• - Ermitteln einer Brillenpose in einem Fahrzeugkoordinatensystem,
• - Bestimmen einer umgebungsbezogenen Brillenpose im Umgebungskoordinatensystem;
• - Übermitteln der umgebungsbezogenen Brillenpose an die Datenbrille.

According to a first aspect, a method for operating a display system with data glasses in a vehicle is provided, with the following steps:

• - Detecting a vehicle pose based on an environmental coordinate system;
• - Determining a glasses pose in a vehicle coordinate system,
• - Determine an environment-related glasses pose in the environment coordinate system;
• - Transmit the environment-related glasses pose to the data glasses.

• - Erfassen einer Bewegungsinformation mithilfe einer Inertialsensorik,
• - Bereitstellen einer Objektinformation für ein Anzeigeobjekt, die mit einer Umgebungsposition und einem Zeitstempel versehen ist,
• - Erzeugen eines anzuzeigenden Anzeigebilds abhängig von der umgebungsbezogenen Brillenpose entsprechend der Objektinformation für das Anzeigeobjekt;
• - ausgehend von einem Zeitpunkt, der durch den Zeitstempel vorgegeben ist, Durchführen eines Warpings basierend auf der Bewegungsinformation, um eine Anpassung des von der Objektinformation abhängigen Anzeigebilds vorzunehmen.

The following steps can also be carried out in the data glasses:

• - Detecting movement information using inertial sensors,
• - Providing object information for a display object, which is provided with an environmental position and a time stamp,
• - Generating a display image to be displayed depending on the environment-related glasses pose according to the object information for the display object;
• - starting from a point in time that is predetermined by the timestamp, performing warping based on the movement information in order to adapt the display image dependent on the object information.

The contact-analog representation of a display object on a display surface of the data glasses with a fixed reference to an environmental position, such as the position of an environmental object or another position in the area surrounding the vehicle, requires precise knowledge of the glasses pose of the data glasses in relation to an environmental coordinate system (environment-related glasses pose). The environmental coordinate system corresponds to a reference coordinate system that refers to fixed positions in the environment of the vehicle, i.e. H. is world festival. Glasses-based tracking methods usually only use inertial sensors and can therefore only record the glasses position differentially in relation to the environmental coordinate system, so that constant updating through time integration is necessary to determine the current position. Such a procedure is complex and error-prone, so that the resulting glasses pose must always be corrected in the surrounding coordinate system. In addition, due to the movement of the vehicle while driving in the world, in particular the oscillating pitching, tilting and yaw movements, the accelerations recorded via the inertial sensors in the data glasses are always influenced by these disruptive movements, which lead to an additional error in determining the glasses pose in the surrounding coordinate system.

A glasses pose in the vehicle coordinate system can be determined by integrating rotational and translational accelerations detected by an inertial sensor system and/or a data glasses camera. Since the direct detection of the environment-related glasses pose in the data glasses is limited due to limited computing capacity, an assistance system provided in the vehicle is used according to the above method, which supports the provision of the environment-related glasses pose in the data glasses. For this purpose, a vehicle pose of the vehicle is determined in relation to the surrounding coordinate system using a camera-based and/or geopositioning system. This can be done, for example, by recognizing prominent surrounding objects by comparing them with high-precision map data and, if necessary, by merging them with a e.g. B. geoposition of the mobile device recorded via a geopositioning system (such as GPS). Furthermore, an outside-in tracking system is provided which, for example, uses an interior camera that is aligned or can be aligned with the data glasses to determine the glasses pose of the data glasses in relation to a vehicle coordinate system.

The assistance system can now easily determine an environment-related glasses pose by vector addition based on the glasses pose of the data glasses in the vehicle coordinate system and the vehicle pose within the environmental coordinate system. The glasses pose in the environmental coordinate system can now be transmitted to the data glasses in conjunction with a time stamp and there by fusing the relative changes An updated environment-related glasses pose can be determined since the time of the time stamp with the environment-related glasses pose transmitted by the assistance system.

With the help of object information provided, which specifies a display object to be displayed and a corresponding environmental position at which the display object to be displayed in a contact-analog manner is to be displayed, the display object can be displayed at a display position in the data glasses depending on a glasses pose determined in the environmental coordinate system. This gives the user of the data glasses the impression that the display object is displayed firmly in the surrounding position. This is done using known rendering methods, which, based on the pose of the glasses, determine a field of view and the display location at which the user perceives the display object on the display surface.

It can be provided that the vehicle pose is recorded in relation to the environmental coordinate system using a geoposition system, in particular GPS, and/or using a localization system based on object recognition using map data as a geoposition and an orientation in the vehicle environment (world).

With the help of the information provided about the environmental position at which a display object to be displayed in a contact-analog manner is to be displayed and which is specified in relation to an environmental coordinate system preferably defined for a geoposition, the display object can be displayed at a display position in the data glasses depending on a glasses pose determined in the environmental coordinate system, so that the display object is displayed firmly at the surrounding position.

However, the transmission of information that enables or includes a conversion of the environmental position (based on a vehicle coordinate system) into the glasses coordinate system is subject to latency, so that a quasi real-time contact-analog representation of the display object with respect to the world-fixed environmental position is not possible.

In order to display display objects at specific environmental positions in real time, it is necessary that all temporal movements, including the higher-frequency movement of the mobile device, which superimpose the deterministic smooth movement trajectory of the vehicle in the vehicle environment, be latency-corrected. In this regard, a warping functionality can be provided. The warping functionality provides for creating a display image corresponding to the environment-related glasses pose last received by the assistance system (with a time stamp) and for carrying out a two-dimensional shift in the image plane according to the movements of the data glasses recorded by the inertial sensor system since the time of the time stamp at which the environment-related glasses pose has been recorded. This allows the latency between receiving the environmental glasses pose and the actual glasses pose to be compensated for a short period of time with little computational effort, without this being noticeably noticeable to the user.

The above method describes an architecture for using smart glasses in a vehicle. The integration of the data glasses can be done very easily, since no intervention in the glasses architecture is necessary in order to use them in a vehicle according to the method described above. In particular, the above method makes it possible to carry out warping in a simple manner, since a glasses pose of the data glasses is available at any time in an environmental coordinate system, with a low latency at the actual time.

• - eine Positionsbestimmungseinrichtung, die ausgebildet ist, um eine Fahrzeugpose bezogen auf ein Umgebungskoordinatensystem zu erfassen;
• - eine Steuereinheit, die ausgebildet ist, um eine Brillenpose in einem Fahrzeugkoordinatensystem zu erfassen und eine umgebungsbezogene Brillenpose im Umgebungskoordinatensystem zu bestimmen;
• - eine Kommunikationseinheit, die ausgebildet ist, um die umgebungsbezogene Brillenpose an eine Datenbrille zu übermitteln.

According to a further aspect, a display system with an assistance system is provided in a vehicle, comprising:

• - a position determination device which is designed to detect a vehicle pose based on an environmental coordinate system;
• - a control unit which is designed to detect a glasses pose in a vehicle coordinate system and to determine an environment-related glasses pose in the environment coordinate system;
• - a communication unit that is designed to transmit the environment-related glasses pose to data glasses.

• - eine Kommunikationseinrichtung zum Erfassen der umgebungsbezogenen Brillenpose;
• - eine Inertialsensorik, die ausgebildet ist, um zum Bestimmen einer Bewegungsinformation beizutragen,
• - eine Steuereinheit, die ausgebildet ist,
  • ◯ um eine Objektinformation für ein Anzeigeobjekt bereitzustellen oder zu empfangen, die mit einer Umgebungsposition und einem Zeitstempel versehen ist,
  • ◯ um ein anzuzeigendes Anzeigebild abhängig von der umgebungsbezogenen Brillenpose entsprechend der Objektinformation für das Anzeigeobjekt zu erstellen;
  • ◯ um ausgehend von einem Zeitpunkt, der durch den Zeitstempel vorgegeben ist, ein Warping basierend auf der Bewegungsinformation durchzuführen, um das von der Objektinformation abhängige Anzeigebild anzupassen.

Furthermore, the display system can include:

• - a communication device for detecting the environment-related glasses pose;
• - an inertial sensor system that is designed to contribute to determining movement information,
• - a control unit that is designed
  • ◯ to provide or receive object information for a display object, which is provided with an environmental position and a time stamp,
  • ◯ to create a display image to be displayed depending on the environmental glasses pose according to the object information for the display object;
  • ◯ to carry out warping based on the movement information, starting from a point in time specified by the time stamp, in order to adapt the display image dependent on the object information.

• 1 eine schematische Darstellung eines Anzeigesystems in einem Fahrzeug mit einer Datenbrille und einem Assistenzsystem;
• 2 eine Darstellung eines Fahrzeugs mit einer Datenbrille in einer Fahrzeugumgebung zur Veranschaulichung der relativen Beziehungen zwischen dem Umgebungskoordinatensystem und dem Fahrzeugkoordinatensystem; und
• 3 ein Flussdiagramm zur Veranschaulichung eines Verfahrens zum Betreiben des Anzeigesystems zur kontaktanalogen Darstellung von Informationsobjekten bezogen auf eine feste Umgebungsposition.

Embodiments are explained in more detail below with reference to the accompanying drawings. Show it:

• 1 a schematic representation of a display system in a vehicle with data glasses and an assistance system;
• 2 a representation of a vehicle with data glasses in a vehicle environment to illustrate the relative relationships between the environmental coordinate system and the vehicle coordinate system; and
• 3 a flowchart to illustrate a method for operating the display system for the contact-analog display of information objects based on a fixed environmental position.

Description of embodiments

1 shows a schematic representation of a display system 1, particularly for use in a vehicle. The display system 1 includes an assistance system 2, which is in communication connection 4 with data glasses 3. The communication connection 4 is designed as a data transmission channel, e.g. B. in the form of a wireless communication connection or a wired communication connection. The communication connection 4 is able to transmit any type of data and information between the assistance system 2 and the data glasses 3, for example based on packet-bound data transmission. The communication connection 4 can be based, for example, on WiFi, Bluetooth, Bluetooth Low Energy or a comparable standardized radio protocol.

The assistance system 2 can be part of a vehicle assistance system and, in particular, can be provided in a stationary manner in the vehicle. The assistance system 2 can be equipped with a communication unit 21, which enables the communication connection 4 between data glasses 3 and the assistance system 2.

The assistance system 2 can also be connected to an environment detection system 22, which has one or more cameras. The environment detection system 22 can detect the surroundings of the vehicle. The one or more cameras can z. B. include an RGB, IR, fisheye camera, a dynamic vision sensor and the like.

The assistance system 2 can have a control unit 23, through which at least one virtual display object for display in the data glasses 3 is generated and/or determined in a manner known per se, depending on a geographical position of the vehicle and depending on an environmental image captured by the environment detection system 22. The environmental position of the at least one virtual display object is specified with respect to a fixed, predefined environmental coordinate system.

The assistance system 2 also has a position determination device 28. The position determination device 28 may include a GPS system for detecting a geoposition of the vehicle and/or odometry for providing information about a movement trajectory (speed, steering angle) of the vehicle with respect to the surroundings. Additionally (by fusion of the position information) or alternatively, a geoposition of the vehicle can be determined using an environmental image captured by the environment detection system 22 and comparisons with highly accurate map data. The geoposition corresponds to a vehicle pose in the environmental coordinate system

An interior camera 25 can be provided, which is aimed at the data glasses 3. The assistance system 2 can be used to track the data glasses in relation to the vehicle coordinate system using methods known per se in order to obtain a glasses pose in the vehicle coordinate system.

The data glasses 3 include two transparent lenses 32, which are enclosed in a frame 31 in a manner known per se. The frame 31 is, for example, provided with temples 33, so that the data glasses 3 can be worn on the head of a user in a manner known per se.

One or both viewing panes 32 (glass lenses) are furthermore provided with a transparent display surface 35, through which a display image for displaying virtual display objects is brought into the eye of the wearer of the data glasses 3 by a suitable device, such as a display device 36 arranged on the frame 31 can be projected. The display device 36 can have a microprocessor or a comparable computing unit and a display unit, such as. B. have a projection device or the like. The Display unit can be designed to direct the electronically generated display image onto the display surface 35 and to image/display it there.

Due to the transparent design of the display surface 35, the electronically generated image can be superimposed on the real environment that can be perceived through the display surface 35. Using the display device 36, a virtual display object, such as text, a symbol, video information, a graphic or the like, can be displayed on one or both display surfaces 35.

The data glasses 3 can be worn on the user's head like a typical visual aid, wherein the data glasses 3 with the frame 31 can rest on the user's nose and the brackets 33 can rest laterally on the user's head. The user's viewing direction in a straight-ahead direction then occurs through the transparent display surfaces 35 of the viewing windows 32, so that the user's viewing direction, which is predetermined by an eye position and an optical viewing axis (eye axis), has a fixed relationship to the orientation of the data glasses 3. This reference depends individually on the wearer of the data glasses 3 and is specified by calibration information.

To display display objects, corresponding object information in the form of object data is transmitted from the assistance system 2 to the data glasses 3. The object data indicates the type of display object, such as: B. a text object, an icon or other identification of a display area, the viewing angle range or viewing angle ranges in which the display object is to be displayed on the display area and the object position on the display area at which the display object is to be displayed.

An inertial sensor system 38 can be provided, which z. B. is designed in the form of a 6-DOF intertial sensor. This provides movement information in the form of translational accelerations and angular accelerations, which can be converted into a change in position and orientation through the respective double integration.

With the help of a control unit 37, object information is received and processed by the assistance system 2 via a communication device 39, so that it is displayed in a contact-like manner in the respective viewing angle area into which the user of the data glasses 3 looks and in which surrounding objects can be perceived. This means that the display object specified by the object information is displayed on the display surface 35 if its surrounding position is in the viewing angle range.

The function of the control unit 37 and/or the communication unit 39 can be accommodated partially or completely on a companion device (e.g. a smart phone) that is physically separate from the data glasses 3, which in turn is connected to the actual data glasses 3, for example wirelessly or by cable.

The data glasses 3 has a data glasses camera 34, which is preferably aligned in the viewing direction of the wearer of the data glasses 3. The data glasses camera 34 can be used to determine the glasses pose of the data glasses 3 in relation to a vehicle coordinate system in the control unit 37 of the data glasses 3 using a marker or other structures that are arranged in the interior of the vehicle.

The data glasses camera 34 can work in the visible spectrum and/or in the invisible spectrum, for example in the infrared spectrum. The latter can be, for example, a data glasses camera that is used for a LIDAR-based system on data glasses.

2 shows schematically the definitions of the coordinate systems regarding the vehicle 5 and the surroundings. The environmental coordinate system U is defined globally and can, for example, be defined in geocoordinates based on a fixed origin point.

Essentially interesting for the present invention are the rotational deviations between the vehicle coordinate system F and the environmental coordinate system U.

The user of the data glasses 3 is a passenger in the vehicle 5 and can change the glasses pose of the data glasses 3 relative to the vehicle coordinate system F by moving his head. In order to display a display object in a contact-analog manner in relation to an environmental position, i.e. the corresponding object information is provided with an environmental position, on the display surface of the data glasses 3, there must be a glasses pose in the data glasses 3 in relation to the environmental coordinate system.

In 3 is a flowchart to illustrate a method for operating the display system 1 with the data glasses 3 described in more detail.

The method is implemented in the assistance system 2 and the data glasses 3, whose timers are synchronized with one another. The aim of the method is to display something in the data glasses 3 Display object, the environmental position at which it is to be displayed, is defined in relation to an environmental coordinate system, to be displayed in the data glasses 3 in a contact-like manner. The object information, which indicates one or more display objects with their respective environmental positions, can be generated in the data glasses 3 or previously provided by the assistance system 2 via the communication connection 4.

In step S1, the positioning device 28 of the assistance system 2 detects a geoposition of the vehicle 5 and its orientation in the environment, in particular using an inertial sensor system, the GPS system and/or odometry. The geoposition and orientation are provided as a vehicle pose. In particular, camera data can additionally or alternatively be used to recognize mapped surrounding objects and to locate the vehicle in the surrounding coordinate system based on their positions, which is known, for example, from high-precision map data.

In step S2, a glasses pose of the data glasses 3 in the vehicle is now determined using the interior camera 25 or a comparable sensor system. This glasses pose is first determined based on the vehicle coordinate system F and provided in the assistance system 2.

In step S3, the data glasses 3 can now be located in the environmental coordinate system U by simple vector addition by calculating the glasses pose and the vehicle position in the environmental coordinate system U. In this way, an environment-related glasses pose of the data glasses 3 can be determined.

In step S4, the environment-related glasses pose is transmitted to the data glasses 3.

In a subsequent step S5, the environment-related glasses pose in the data glasses 3 can be updated.

With the help of a movement trajectory, which results from the acceleration data of the inertial sensor system in the data glasses 3, warping can now be carried out in step S6, in which the display image to be displayed, which relates to the last determined or provided environment-related glasses pose, is translated translationally, i.e. H. two-dimensionally in the image plane, is moved without carrying out a perspective correction. The display image is shifted in such a way that a movement of the data glasses 3 in relation to the environment is compensated for since the time of a time stamp of the last received environment-related glasses pose. In this way, the display image can be tracked in real time by the movement of the data glasses 3 in the environmental coordinate system, so that the latency in providing the environmental-related glasses pose by the assistance system 2 can be compensated.

The process is carried out cyclically so that the environment-related glasses pose in the data glasses is continuously updated.

Reference symbol list

1

Display system

2

Assistance system

3

Smart glasses

4

Communication link

21

Communication unit

22

Environment detection system

23

Control unit

25

Interior camera

28

Positioning device

31

Frame

32

viewing windows

33

Glasses temples

34

Smart glasses camera

35

Display area

36

Display device

37

Control unit

38

Inertial sensors

39

Communication facility

5

vehicle

Claims (7)

Method for operating a display system (1) with data glasses (3) in a vehicle (5), with the following steps: - Detecting (S1) a vehicle pose based on an environmental coordinate system (U); - determining (S2) a glasses pose in a vehicle coordinate system (F), - Determine (S3) an environment-related glasses pose in the environment coordinate system (U); - Transmit (S4) the environment-related glasses pose to the data glasses (3). Procedure according to Claim 1 , whereby the following steps are carried out in the data glasses (3): - Detecting movement information using an inertial sensor system (38), - Providing object information for a display geobject, which is provided with an environmental position and a time stamp, - generating a display image to be displayed depending on the environment-related glasses pose according to the object information for the display object; - starting from a point in time that is predetermined by the time stamp, performing (S6) warping based on the movement information in order to adapt the display image dependent on the object information. Procedure according to Claim 2 , whereby the adjustment is carried out by a two-dimensional displacement of the display image in the image plane of the display image. Procedure according to one of the Claims 1 until 3 , wherein the vehicle pose is recorded in relation to the environmental coordinate system using a geopositioning system, in particular GPS, and/or using a localization system based on object recognition using map data. Display system (1) with an assistance system (2) in a vehicle (5), comprising: - a position determination device which is designed to detect a vehicle pose based on an environmental coordinate system; - a control unit (23) which is designed to detect a glasses pose in a vehicle coordinate system (F) and to determine an environment-related glasses pose in the environment coordinate system (U); - A communication unit (21) which is designed to transmit the environment-related glasses pose to data glasses (3). display system Claim 5 with data glasses (3), the data glasses (3) comprising: - a communication device (39) for detecting the environment-related glasses pose; - an inertial sensor system (38), which is designed to contribute to determining movement information, - a control unit (37), which is designed to provide or receive object information for a display object, which is provided with an environmental position and a time stamp , o to create a display image to be displayed depending on the environmental glasses pose according to the object information for the display object; o to carry out warping based on the movement information, starting from a point in time that is predetermined by the time stamp, in order to adapt the display image dependent on the object information. display system Claim 6 , whereby the data glasses (3) are designed to be an augmenting display.

Images