DE102023203522B4 - System for validating a component of a means of transport, method and computer program for operating such a system - Google Patents

Abstract

System (1) for the validation of a component (2) of a means of transport, with
- a movement platform (3) with at least three degrees of freedom for a dynamic movement simulation of a cockpit (4) of the means of transport in which the component (2) to be validated is installed;
- a mixed reality display device (5) for displaying an augmented environment for a user (6) located in the cockpit (4); and
- a graphics system (7) for calculating (11) the augmented environment, wherein the graphics system (7) is configured to calculate (11) the augmented environment in such a way that at least the user's hands and the component (2) to be validated are actually perceivable within a virtual environment, and wherein the cockpit (4) is color-prepared and the graphics system (7) is configured to calculate (11) the augmented environment using a chroma key technique and using a virtual mask (VM), wherein the color preparation of the cockpit (4) is limited to those areas of the cockpit (4) that are not covered by a virtual mask (VM).

Description

The present invention relates to a system for validating a component of a means of transportation. The invention further relates to a method and a computer program with instructions for operating such a system.

Dynamic driving simulations are often used to verify or validate functions in automotive development. Initially, driving simulators primarily used projectors for visualization. In recent years, a combination of dynamic driving simulators with virtual reality technologies has become established. In virtual reality (VR), the user is immersed in a fully virtual, computer-generated world. Head-mounted display devices (HMDs), such as VR headsets, are typically used for this purpose. The advantage of using VR headsets is that no hardware components are required for driving simulation. Changes in design, concept, or function can therefore be implemented quickly and virtually.

However, VR headsets have the disadvantage that realistic, haptic interaction options are very limited in virtual reality. A major problem is that the test subject is isolated from the real world by the VR headset. In particular, the test subject does not see their own hands, but only virtual representations of them. Due to the positional inaccuracy of these virtual representations of the hands, functions in automotive development, for example, that require realistic, haptic interactions between driver and vehicle cannot be evaluated in early development phases in virtual reality. These functions are therefore only evaluated in later development phases, with the costs for changes and optimizations during these development phases rising rapidly.

Experiments have been conducted using multiple hand tracking devices simultaneously to reduce the positional inaccuracy of the virtual representations of the hands. However, sufficient accuracy has not been achieved so far. Even approaches based on data gloves have not yet achieved the desired accuracy.

As an alternative to VR technologies, augmented reality technologies can be used in combination with driving simulators.

Augmented Reality (AR) involves enhancing the perception of the real world with virtual elements that are accurately registered in three-dimensional space and allow real-time interaction. Data glasses, for example, can be used to display AR displays. Data glasses are worn like regular glasses, but have one or more projection units or displays that can project information in front of the wearer's eyes or directly onto the retina. The glasses are designed so that the wearer can also perceive their surroundings.

Against this background, WO 2022 / 070 033 A1 A device for creating and managing prototypes comprising a plurality of physical structures and a digital medium, comprising a CPU and a display and control device. The physical structures are movable and adjustable by means of a plurality of respective actuators. The CPU is configured to control the actuators to adjust the positioning of the physical structures to predetermined positions or to display the position of the physical structures.

AU 2018 200 822 A1 describes a simulator for a vehicle. The simulator uses green screening to allow a user to see their hands or body in an augmented environment.

KR 2017 0 005 971 A Describes a driving simulator. The driving simulator uses no or only a limited number of physical components. A sense of operation, a driving environment, and an environmental situation that are identical or similar to reality are provided as augmented reality images.

WO 2012 / 113 686 A1 Describes a simulator for training a person, comprising a training room, a partially transparent head-mounted display, and a simulation processing unit. The inner boundary surface of the training room is provided with a color suitable for color-based image isolation. The head-mounted display is equipped with two video cameras for stereoscopic recordings and a head tracker that enables the determination of the wearer's gaze direction. The simulation processing unit comprises two image generators for a stereoscopic display of virtual images on one head-mounted display.

US 2019 / 0 392 639 A1 Describes a head-mounted display with a frame and a display system connected to it. A stereo camera connected to the frame generates information that includes image data and depth data. The depth data indicates a distance to a simulator cockpit. A processor uses this information to identify a cockpit model corresponding to the simulator cockpit and generates computer-generated images based on the cockpit model.

The article J. Hönig et al.: “Mixed-Reality-in-the-Loop Simulation,” atp magazine, Issue 6-7 2021 , describes a solution approach for combining hardware-in-the-loop simulations with modern visualization methods to create a mixed-reality-in-the-loop simulation.

With the solutions described, the test subject can see their hands, enabling improved haptic interaction between the test subject and real objects. However, further improved solutions are desirable for validating components that require interaction during driving.

It is an object of the invention to provide improved solutions for the validation of a component of a means of transport which requires interactions during driving operation.

This object is achieved by a system having the features of claim 1, by a method according to claim 7, and by a computer program with instructions according to claim 8. Preferred embodiments of the invention are the subject of the dependent claims.

According to a first aspect of the invention, a system for validating a component of a means of transport comprises a motion platform, a mixed-reality display device, and a graphics system. The motion platform has at least three degrees of freedom for dynamic motion simulation of a cockpit of the means of transport in which the component to be validated is installed. The mixed-reality display device is configured to display an augmented environment for a user located in the cockpit. The graphics system is configured to calculate the augmented environment. The augmented environment is calculated such that the user can at least perceive his or her hands and the component to be validated in a real way within a virtual environment. The cockpit is color-prepared, and the graphics system is configured to calculate the augmented environment using a chroma key technique and a virtual mask, wherein the color-prepared cockpit is limited to those areas of the cockpit that are not obscured by a virtual mask.

• - Ansteuern einer Bewegungsplattform des Systems entsprechend einer zu simulierenden Bewegung;
• - Berechnen einer augmentierten Umgebung zur Anzeige durch ein Mixed-Reality-Anzeigegerät des Systems, wobei die augmentierte Umgebung derart berechnet wird, dass für einen Anwender zumindest seine Hände und die zu validierende Komponente real innerhalb einer virtuellen Umgebung wahrnehmbar sind; und
• - Übermitteln von Grafikdaten der augmentierten Umgebung an das Mixed-Reality-Anzeigegerät.

According to a further aspect of the invention, a method for operating a system according to the invention for validating a component of a means of transport comprises the steps:

• - Controlling a motion platform of the system according to a movement to be simulated;
• - Calculating an augmented environment for display by a mixed reality display device of the system, wherein the augmented environment is calculated in such a way that a user can perceive at least his hands and the component to be validated in a virtual environment; and
• - Transmitting graphic data from the augmented environment to the mixed reality display device.

A cockpit of the system is color-prepared and the augmented environment is calculated using a chroma key technique and a virtual mask, whereby the color preparation of the cockpit is limited to those areas of the cockpit that are not covered by a virtual mask.

• - Ansteuern einer Bewegungsplattform des Systems entsprechend einer zu simulierenden Bewegung;
• - Berechnen einer augmentierten Umgebung zur Anzeige durch ein Mixed-Reality-Anzeigegerät des Systems, wobei die augmentierte Umgebung derart berechnet wird, dass für einen Anwender zumindest seine Hände und die zu validierende Komponente real innerhalb einer virtuellen Umgebung wahrnehmbar sind; und
• - Übermitteln von Grafikdaten der augmentierten Umgebung an das Mixed-Reality-Anzeigegerät.

According to a further aspect of the invention, a computer program comprises instructions which, when executed by a computer, cause the computer to perform the following steps for operating a system according to the invention for validating a component of a means of transport:

• - Controlling a motion platform of the system according to a movement to be simulated;
• - Calculating an augmented environment for display by a mixed reality display device of the system, wherein the augmented environment is calculated in such a way that a user can perceive at least his hands and the component to be validated in a virtual environment; and
• - Transmitting graphic data from the augmented environment to the mixed reality display device.

A cockpit of the system is color-prepared and the augmented environment is calculated using a chroma key technique and a virtual mask, whereby the color preparation of the cockpit is limited to those areas of the cockpit that are not covered by a virtual mask.

The term "computer" should be understood broadly. It specifically includes workstations, distributed systems, and other processor-based data processing devices.

The computer program may, for example, be made available for electronic retrieval or stored on a computer-readable storage medium.

The solution according to the invention combines mixed reality technologies with a dynamic driving simulator. Mixed reality (MR) combines elements of virtual reality and augmented reality, i.e., the real world is connected to virtual environments to create a new environment. The user interacts simultaneously with the real and a virtual environment. This makes it possible to integrate the desired real components into the virtual simulation. This allows the simulations to be designed so that the real components are used simultaneously with virtual components. A concrete example is a combination of a real steering wheel, a display, the driver's body, and an otherwise virtual interior. The mixed reality display device can be designed, for example, as a head-mounted display. The component to be validated can be either a physical component or a functionality or software, so that, for example, the content displayed on a display can also be validated.

The combination of mixed reality and dynamic driving simulation opens up numerous new possibilities for functional validation. In a static setup, the evaluation content is significantly limited because many vehicle functions must be tested while driving. The spectrum of assessable content, particularly in automotive development, is significantly expanded by the inventive solution, as the driving simulator enables the driving task to be performed. The intended number of degrees of freedom for the dynamic motion simulation ensures that the user does not experience simulation sickness or that this is significantly reduced. Simulation sickness, often referred to as motion sickness in the literature, frequently occurs without suitable motion simulation because the user drives in the simulation but their body does not move in the real world. A major advantage over dynamic driving simulators that use VR technologies is that the direct haptic interaction with control elements enables a significantly wider spectrum of assessable content, for example, in automotive development.

According to the invention, the cockpit is color-prepared, and the graphics system is configured to calculate the augmented environment using a chroma key technique. To apply chroma key techniques, the cockpit can be completely or partially covered with a suitable color. Alternatively, the cockpit components that are to be replaced by a virtual environment in the augmented environment can be manufactured directly in the selected color. Another possibility is to conceal parts of the cockpit with appropriately colored covers. The most suitable colors for chroma key techniques are shades of green and blue, depending on human skin color. The mixed reality display device shows the parts of the cockpit that are executed, painted, or covered in the selected color as virtual content. The control elements that are not executed, painted, or covered in this color, e.g., the steering wheel and touch displays, are displayed in real life.

According to the invention, the graphics system is configured to calculate the augmented environment using a virtual mask. With a virtual mask, real and virtual content can be separated using virtual boundaries that can be defined in various two- or three-dimensional forms. This solution eliminates the need for extensive cockpit preparation, as the preparation can be limited to those areas of the cockpit that are not obscured by a virtual mask.

According to one aspect of the invention, the motion platform is a hexapod. A hexapod is characterized by six degrees of freedom and high dynamics. This enables a highly realistic driving simulation, largely avoiding the occurrence of simulation sickness. At the same time, a distinctive overall experience is achieved for the user and thus a very good evaluation of the component to be validated and its function.

According to one aspect of the invention, the mixed reality display device has a depth sensor, and the graphics system is configured to calculate the augmented environment using depth data from the depth sensor. For example, the depth sensor can be a LIDAR sensor. The depth sensor is preferably built into the mixed reality display device. The use of the depth data enables objects within a specific distance range preferred by the user to be viewed as real and thus not replaced by a virtual environment. It is particularly advantageous that that no extensive preparation of the cockpit is required.

According to one aspect of the invention, the system comprises a tracking system for tracking the cockpit. The tracking system can be an active or passive tracking system or can be configured to determine a cockpit pose from motion data of the motion platform. Such a tracking system is particularly advantageous when virtual masks are used. To use virtual masks in the dynamic simulator, they should move with the simulator.

Preparing the cockpit for the use of chroma key techniques requires a certain amount of effort. This is especially true when validating different vehicles in the dynamic driving simulator, as the cockpit components must be moved into new positions each time. This may make it impossible to attach a cover for the chroma key techniques due to the different layout. In such cases, it may be advantageous to combine the approaches described above. This way, it may be sufficient to color-prepare only a limited portion of the cockpit for the chroma key techniques.

A system according to the invention is particularly advantageously used for the validation of a component of a motor vehicle, in particular a passenger car or a commercial vehicle. The solution according to the invention allows vehicle functions, e.g., autonomous driving functions, to be validated and developed in very early development phases without physical prototypes.

• 1 zeigt schematisch ein System für die Validierung einer Komponente eines Fortbewegungsmittels;
• 2 zeigt schematisch ein Verfahren zum Betreiben des Systems aus 1;
• 3 zeigt schematisch ein Systemdiagramm einer erfindungsgemäßen Lösung;
• 4 zeigt schematisch ein farblich präpariertes Cockpit;
• 5 zeigt schematisch die Verwendung eines Tiefensensors; und
• 6 zeigt schematisch die Verwendung von virtuellen Masken.

Further features of the present invention will become apparent from the following description and the appended claims taken in conjunction with the figures.

• 1 shows schematically a system for the validation of a component of a means of transport;
• 2 shows schematically a method for operating the system from 1 ;
• 3 shows schematically a system diagram of a solution according to the invention;
• 4 shows schematically a color-prepared cockpit;
• 5 shows schematically the use of a depth sensor; and
• 6 shows schematically the use of virtual masks.

To better understand the principles of the present invention, embodiments of the invention are explained in more detail below with reference to the figures. It should be understood that the invention is not limited to these embodiments and that the described features may also be combined or modified without departing from the scope of the invention as defined in the appended claims.

1 shows a schematic of a system 1 for the validation of a component 2 of a means of transport. The system 1 has a motion platform 3 with at least three degrees of freedom. In the example shown, the motion platform 3 is a hexapod with six degrees of freedom. If necessary, the hexapod can also be mounted on a rotation platform (not shown here). The motion platform 3 enables dynamic motion simulation of a cockpit 4 of the means of transport arranged on the motion platform 3, in which the component 2 to be validated is installed. The cockpit 4 is preferably adaptable in order to be able to set the desired ergonomics for different means of transport. In the example shown, the means of transport is a motor vehicle, and the component 2 to be validated is a steering wheel of the motor vehicle. In the cockpit 4 there is a user 6 wearing a mixed reality display device 5. The mixed reality display device 5 shows the user 6 an augmented environment and can, for example, be designed as a head-mounted display. A graphics system 7, shown here as part of a control computer 9, is used to calculate the augmented environment. The graphics system 7 is configured to calculate the augmented environment in such a way that the user 6 can at least perceive his hands and the component 2 to be validated in reality within a virtual environment. For this purpose, the cockpit 4 is color-coded. The graphics system 7 is configured to calculate the augmented environment using a chroma key technique. In addition, the mixed reality display device 5 can have a depth sensor (not shown here), e.g., a LIDAR sensor. In this case, the graphics system 7 is configured to calculate the augmented environment using depth data from the depth sensor. Furthermore, the graphics system 7 is configured to calculate the augmented environment using virtual masks, wherein the color-coded preparation of the cockpit is limited to those areas of the cockpit that are not covered by a virtual mask. For this purpose, it is advantageous if a tracking system 8 generates a The cockpit 4 is tracked so that the virtual masks move with the cockpit 4 or the motion platform 3. The tracking system 8 can be designed as an active or passive tracking system 8. Alternatively, it is possible to determine the pose of the cockpit 4 from movement data of the motion platform 3.

2 shows schematically a method for operating the system from 1 In the method, a motion platform of the system is controlled according to a movement to be simulated 10. In addition, an augmented environment is calculated for display by a mixed reality display device of the system 11. The augmented environment is calculated 11 in such a way that a user can actually perceive at least his hands and a component to be validated within a virtual environment. For this purpose, a cockpit of the system is color-prepared. The augmented environment is calculated using a chroma key technique 11. In addition, the mixed reality display device can have a depth sensor, e.g. a LIDAR sensor. In this case, the augmented environment can be calculated using depth data from the depth sensor 11. Furthermore, the augmented environment is calculated using a virtual mask 11, wherein the color preparation of the cockpit is limited to those areas of the cockpit that are not covered by a virtual mask. For this purpose, it is advantageous to use a tracking system to track the cockpit so that the virtual masks move with the cockpit or the motion platform. The tracking data from the tracking system is then used to calculate the augmented environment. The various approaches can also be combined. The resulting graphic data of the augmented environment is finally transmitted to the mixed reality display device 12 and displayed by it 13.

3 shows a schematic system diagram of a solution according to the invention. The system 1 comprises a motion platform 3 with the cockpit 4 arranged thereon, a mixed reality display device 5, and a graphics system 7. The graphics system 7 can be designed as a standalone system or as part of a control computer 9 and generates graphics data GD for the mixed reality display device 5. The control computer 9 generates control data SD for the movement of the motion platform 3. The mixed reality display device 5 can have a depth sensor 50 that provides depth data TD for the graphics system 7. The system 1 can further comprise a tracking system 8 for tracking the cockpit 4, which provides tracking data TR. Alternatively or additionally, the control computer 9 can provide movement data BD of the motion platform 3 for the graphics system 7. The graphics system 7 uses the provided data BD, TD, TR when calculating an augmented environment to be displayed by the mixed reality display device 5.

4 shows schematically a color-prepared cockpit 4. In order to be able to calculate the augmented environment using a chroma key technique, the cockpit in the example shown is partially covered with a suitable color, which in 4 is indicated by hatching. Alternatively, the components of the cockpit 4 that are to be replaced by a virtual environment in the augmented environment can be manufactured directly in the selected color. Furthermore, parts of the cockpit 4 in the example shown are covered by a correspondingly colored cover FA. The most suitable colors for the chroma key technique are shades of green and blue, depending on human skin color. The mixed reality display device 5 shows the parts of the cockpit 4 that are executed or painted in the selected color or the covered parts as virtual content. Other components 2 that are not executed, painted, or covered in this color, e.g. the steering wheel and touch displays, are displayed in real life.

5 schematically shows the use of a depth sensor 50 of the mixed reality display device 5. With this approach, a depth range TB is defined that is actually visible to the user 6. In addition, the cockpit 4 in the example shown is again color-prepared. Preparing the cockpit 4 for the use of chroma key techniques requires a certain amount of effort. By using the defined depth range TB, it is sufficient to color-prepare only a limited part of the cockpit for the chroma key techniques. The required effort can thus be significantly reduced.

6 schematically shows the use of virtual masks (VM) in the calculation of the augmented environment. Virtual masks (VM) are defined for specific areas. These allow real and virtual content to be separated using virtual boundaries. The boundaries can be defined in various two- or three-dimensional forms. By using virtual masks (VM), the effort required for color-preparing the cockpit (4) can be reduced. The preparation can be limited to those areas of the cockpit (4) that are not covered by a virtual mask (VM).

List of reference symbols

1

system

2

component

3

Movement platform

4

cockpit

5

Mixed reality display device

50

Depth sensor

6

user

7

Graphics system

8

Tracking system

9

Tax calculator

10

Controlling a motion platform

11

Calculating an augmented environment

12

Transmitting graphic data

13

Displaying the augmented environment

BD

Movement data

FA

Colored cover

GD

Graphic data

SD

Tax data

TB

Depth range

TD

Depth data

TR

Tracking data

VM

Virtual mask

Claims (8)

A system (1) for validating a component (2) of a means of transport, comprising: - a motion platform (3) with at least three degrees of freedom for dynamic motion simulation of a cockpit (4) of the means of transport in which the component (2) to be validated is installed; - a mixed-reality display device (5) for displaying an augmented environment for a user (6) located in the cockpit (4); and - a graphics system (7) for calculating (11) the augmented environment, wherein the graphics system (7) is configured to calculate (11) the augmented environment such that the user (6) can actually perceive at least his hands and the component (2) to be validated within a virtual environment, and wherein the cockpit (4) is color-prepared and the graphics system (7) is configured to calculate (11) the augmented environment using a chroma key technique and using a virtual mask (VM), wherein the color-preparedness of the cockpit (4) is limited to those areas of the cockpit (4) that are not covered by a virtual mask (VM). System (1) according to Claim 1 , wherein the movement platform (3) is a hexapod. System (1) according to Claim 1 or 2 , wherein the mixed reality display device (5) has a depth sensor (50) and the graphics system (7) is configured to calculate (11) the augmented environment using depth data (TD) of the depth sensor (50). System (1) according to Claim 3 , wherein the depth sensor (50) is a LIDAR sensor. System (1) according to one of the preceding claims, with a tracking system (8) for tracking the cockpit (4). System (1) according to Claim 5 , wherein the tracking system (8) is an active or passive tracking system (8) or is configured to determine a pose of the cockpit (4) from movement data (BD) of the movement platform (3). Method for operating a system (1) according to one of the Claims 1 until 6 , comprising the steps of: - controlling (10) a movement platform (3) of the system (1) according to a movement to be simulated; - calculating (11) an augmented environment for display by a mixed reality display device (5) of the system (1), wherein the augmented environment is calculated (11) such that a user (6) can actually perceive at least his hands and a component (2) to be validated within a virtual environment, wherein a cockpit (4) of the system (1) is color-prepared and the augmented environment is calculated (11) using a chroma key technique and using a virtual mask (VM), wherein the color preparation of the cockpit (4) is limited to those areas of the cockpit (4) that are not covered by a virtual mask (VM); and - transmitting (12) graphic data (GD) of the augmented environment to the mixed reality display device (5). A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to Claim 7 for operating a system (1) according to one of the Claims 1 until 6 initiate.

Images