DE102023132324A1 - Device and method for operating a user interface of a vehicle - Google Patents

Abstract

A device (100) for operating a user interface of a vehicle (102) comprises an output unit (106) of the vehicle (102), which is designed to output at least one element of the user interface of the vehicle (102) to a user (104). The device (100) further comprises a detection unit (108), which is designed to detect at least one piece of information corresponding to a user input of the user (104) and/or environmental information related to the surroundings of the vehicle (102), and to generate input data corresponding to the detected information. The device (100) further comprises a control unit (110), which is designed to control the output unit (106) of the vehicle (102) and to process the input data using at least one machine learning method. The control unit (110) is further configured to generate, on the basis of the input data and using the machine learning method, at least one element of the user interface of the vehicle (102) that can be output by the output unit (106), and to control the output unit (106) to output the generated element.

Description

The invention relates to a device for operating a user interface of a vehicle. The invention further relates to a method for operating a user interface of a vehicle.

The user interface of a modern vehicle comprises a multitude of output units, in particular visual output units for outputting visual information. Examples of visual output units are an instrument cluster, a central information display (CID), a projector, or a head-up display. Lighting elements arranged inside or on the outside of the vehicle can also represent visual output units. Vehicles also have auditory output units, such as speakers of a vehicle's media playback system. Elements output by the output units, such as elements of a graphical user interface or voice output, typically follow a uniform design. The design is predefined at the factory, and a user can choose between at most a few designs. However, many users have the need to personalize the design of the user interface. However, storage space and bandwidth are limited, so the large number of user interface elements cannot simply be provided in a variety of designs.

From the DE 10 2021 206 537 A1 A method is known in which graphical user interfaces for various devices and platforms are generated using a neural network. In the method, a second user interface for a second platform is generated based on the functional and visual features of a first user interface of a first platform. The second user interface can be generated, in particular, taking into account the limitations of the second platform.

The US 10452902 B1 discloses a method in which image components of a patent drawing are generated using a generative adversarial network. One example is a patent drawing of a graphical user interface.

The object of the invention is to provide a device and a method for operating a user interface of a vehicle, which can have a variety of designs and use little memory space and/or bandwidth.

This object is achieved by a device having the features of claim 1 and by a method having the features of the independent method claim. Advantageous further developments are specified in the dependent claims.

The proposed device for operating a user interface of a vehicle comprises an output unit of the vehicle, which is designed to output at least one element of the user interface of the vehicle to a user. The device further comprises a detection unit, which is designed to detect at least one piece of information corresponding to a user input of the user and/or environmental information related to the surroundings of the vehicle, and to generate input data corresponding to the detected information. The device further comprises a control unit, which is designed to control the output unit of the vehicle and to process the input data using at least one machine learning method. The control unit is further designed to generate, based on the input data and using the machine learning method, at least one element of the user interface of the vehicle, which can be output by the output unit, and to control the output unit to output the generated element.

The output unit can, for example, comprise a display surface facing the vehicle driver and/or another occupant, or a loudspeaker of the vehicle. The detection unit comprises, in particular, an input unit for detecting the user input and/or an environmental sensor system of the vehicle for detecting the environmental information. Further examples of the output unit and the detection unit are mentioned in the following description in connection with embodiments. At least part of the control unit can be formed by a processing unit remote from the vehicle. For example, the machine learning method can be executed on the remote processing unit, while the control of the output unit takes place locally in the vehicle.

The device first captures the information on the basis of which the user interface element is to be generated. The information can be user input, for example, a selection from predefined text modules or a prompt that the user freely enters. The information can also be environmental information, for example, the time of day, the season, or the geographical region in which the vehicle is currently located. Furthermore, the captured information can also be a combination of the user input input and the environmental information. The input data is generated from the captured information, which serves as input to the machine learning process. The input data can be modified, for example, or the input can be restricted to a predefined framework, for example, to create a brand-appropriate aesthetic framework for the generation of the element. The user interface element is then generated using the machine learning process. The machine learning process can be a generic model or a model fine-tuned based on a small training dataset. A generic model has the advantage that a very large number of very different designs can be realized. A fine-tuned model has the advantage that, for example, it can be ensured that the generated elements are designed in accordance with the design principles of a brand identity. Machine learning processes are inherently noisy, so the output varies even with the same input. For the device, this means that each generated element is unique, making the user interface hyper-personalizable. This allows the device to satisfy the user's need for personalization of the user interface. However, because the user interface elements are generated and not deployed, the device uses less storage space and/or bandwidth for this personalization than known personalizations.

In one embodiment, the output unit is configured to output visual information to the user, and the generated element is a visual element. The control unit is configured, in particular, to generate an image as the visual element. In such an embodiment, the output unit can, in particular, comprise a display unit of the vehicle, for example, a graphical instrument cluster, a central information display (CID), a head-up display, in particular a panoramic head-up display, or other display surfaces facing the occupants, for example, projection surfaces belonging to a projector. However, the output unit can also comprise a controllable lighting element, which is configured, for example, to generate light of an adjustable color and/or an adjustable sequence of light of different colors. In such an embodiment, the generated visual element is the light color or the sequence of the light of different colors. In particular, by generating exterior lighting, an individualized welcome scenario can thus be realized, for example. The local provision of various visual elements requires a particularly large amount of storage space. Accordingly, the transmission of visual elements from a remote storage location requires a high amount of bandwidth. Thus, creating visual elements can save a lot of storage space and bandwidth.

In a further embodiment, the control unit is configured to generate a first image with a first resolution using the machine learning method and, based on the first image, to generate a second image with a second resolution as the visual element using a further machine learning method, wherein the second resolution is higher than the first resolution. For example, a background image for a display surface of the vehicle can be initially generated with a low resolution based on a prompt entered by the user and then upscaled to obtain a high-resolution background image. Generating images is computationally more complex than increasing the resolution. Thus, in this embodiment, the computing power used to generate the visual element can be optimally utilized.

In a further embodiment, the control unit is configured to generate at least one visual element, which is part of a graphical user interface of the vehicle, based on the input data and using the machine learning method. The visual element can be, for example, an operating element such as a button, a functional element such as a warning notice, or a decorative element such as a background image. The visual element can be static or dynamic, for example, animated. The user interacts with the vehicle primarily via the graphical user interface. The elements of the graphical user interface are therefore particularly influential on the impression the user interface leaves on the user. Accordingly, the ability to customize the graphical user interface offers significant added value. On the other hand, a typical graphical user interface comprises a large number of visual elements. By generating these visual elements, significant added value can be created for the user with little expenditure on storage space or bandwidth.

In a further embodiment, the control unit is configured to generate at least one further element of the user interface based on the generated visual element. The further element can be generated, in particular, based on a template in order to maintain a uniform design of the graphical user interface despite individualization, for example, in accordance with the design principles of a brand identity. In one example, the control unit first generates a background image as the visual element. Using a clustering algorithm, such as median cut, the control unit then extracts dominant colors from the generated background image. The control unit then generates the remaining elements of the graphical user interface based on the extracted color; for example, the control unit colors the remaining elements according to the extracted color. Since, in this embodiment, only one visual element is initially generated using the machine learning method, this embodiment is particularly resource-efficient.

In a further embodiment, the control unit is configured to generate, based on the input data and using the machine learning method, a plurality of visual elements that are part of a uniform skin of the graphical user interface. A skin, or theme, refers to the uniform design of the elements of the graphical user interface. In other words, in this embodiment, for example, the layout, typography, colors, patterns, as well as icons and buttons, are generated uniformly and in a coordinated manner according to predefined specifications. The visual elements generated in this way adhere to a common scheme and create a consistent impression for the user.

In a further embodiment, the acquisition unit is configured to provide the input data in the form of natural language text. The control unit is configured to process the input data using a text-to-image model as the machine learning method to generate the visual element. The input data can, for example, be a prompt that the user has freely entered or composed based on predefined text modules. Alternatively or additionally, the natural language text can also be generated using a further machine learning method based on the environmental information. For example, the further machine learning method can be used to classify the immediate surroundings. In one example, the control unit determines, based on the environmental information and using the further machine learning method, that the vehicle is traveling through an alpine landscape. The control unit then generates the prompt "alpine landscape" as input for the text-to-image model to generate the visual element. A number of generic text-to-image models exist in the prior art that can be easily adapted to the requirements of the device. This embodiment is therefore particularly simple to implement.

In a further embodiment, the output unit is configured to output auditory information to the user, and the element of the user interface is an auditory element. Audible elements, such as short pieces of music (jingles) or voice outputs, are also part of the look and feel of the user interface. These audible elements can also be generated using the machine learning method to enable greater customization.

In a further embodiment, the detection unit is configured to detect, as the information, navigation information relating to the vehicle and/or weather information relating to the vehicle's surroundings. For example, the detection unit can receive the position of the vehicle from a global satellite navigation system. The navigation information is, for example, the geographical region in which the vehicle is currently located. The weather information can, for example, be the current weather or the time of year detected or received from a corresponding service. Based on the navigation information and/or the weather information, the element of the user interface can be adaptively adjusted to the vehicle's surroundings without having to provide many elements that are memory- or bandwidth-intensive. In one example, the control unit determines, based on the weather information, that it is snowing in the vehicle's surroundings. The control unit then generates visual elements with a snow theme using the machine learning method.

In a further embodiment, the detection unit comprises an image detection unit of the vehicle, which is designed to capture at least one image with a depiction of an area outside the vehicle as the information and to generate image data corresponding to the image, and the input data comprises the image data. The image detection unit is in particular an external camera of the vehicle and is designed to capture a two-dimensional image of an area outside the vehicle. The image detection unit captures in particular light in the optical spectrum and/or in the infrared spectrum. As an alternative to an optical camera, the environment detection unit can also comprise a RADAR or LIDAR system that generates a three-dimensional image of the area outside the vehicle, in particular in the form of a point cloud. A series of information items can be extracted from the image data, on the basis of which the element of the user interface can be generated. In one example, the control unit determines The control unit determines the predominant color in the vehicle's surroundings based on the image data and generates a visual element of the graphical user interface based on the predominant color. In another example, the control unit determines, based on the image data, that the vehicle is currently driving through an alpine landscape and generates visual elements of the graphical user interface with an alpine theme that corresponds to the landscape currently being driven through.

In a further embodiment, the detection unit comprises a mobile terminal device that is designed to detect the information, in particular the user input of the user. The terminal device can be, for example, a smartphone or a tablet computer running an application that communicates with the control unit. The user input can be detected particularly easily for the user via the mobile terminal device, in particular when the user is not in the vehicle interior. In order to create a particularly uniform look and feel, the control unit can also be designed to generate elements of a user interface of the mobile terminal device corresponding to the generated elements of the user interface of the vehicle. For example, the control unit can generate visual elements of the graphical user interface of the application corresponding to the visual elements of the graphical user interface of the vehicle.

In another embodiment, the machine learning method is a generative model, in particular a generative adversarial network. Generative models, in particular diffusion models and generative adversarial networks, exist in the prior art and can be easily adapted to the requirements of the device. The device is thus easy to implement. In particular, the generative model can be fine-tuned based on a small training dataset in order to create the elements of the user interface in accordance with predefined design principles.

In a further embodiment, the control unit is configured to control the output unit to output at least one predefined question to the user. The acquisition unit is configured to acquire the user's response in the form of a user input as the information and to generate the input data of the response accordingly. The predefined question can, for example, ask the user about their favorite artist, favorite vacation destination, and/or favorite color. This guides the user through the creation of the element and facilitates use of the device.

In a further embodiment, the detection unit is configured to detect user feedback related to the generated element and to generate feedback data corresponding to the feedback. The control unit is configured to modify the machine learning method based at least on the feedback data. The feedback may include, for example, whether the user is satisfied with the generated element. Based on this feedback, the machine learning method can then be modified until an element satisfactory to the user is generated. This increases the user-friendliness of the device.

The invention further relates to a method for operating a user interface of a vehicle. In the method, at least one piece of information is acquired that corresponds to a user input from the user and/or environmental information related to the vehicle's surroundings. Input data corresponding to the acquired information is generated. Based on the input data and using a machine learning method, at least one element of the vehicle's user interface is generated. The generated element is output by an output unit of the vehicle.

The method has the same advantages as the claimed device. In particular, the method can be further developed with the features of the dependent claims directed to the device. Furthermore, the device described above can be further developed with the features described in this document in connection with the method.

• 1 in schematischer Darstellung eine Vorrichtung zum Betreiben einer Benutzerschnittstelle eines Fahrzeugs; und
• 2 einen Ablaufplan des Verfahrens zum Betreiben der Benutzerschnittstelle des Fahrzeugs.

Embodiments of the invention are explained in more detail below with reference to the figures, in which:

• 1 a schematic representation of a device for operating a user interface of a vehicle; and
• 2 a flowchart of the procedure for operating the vehicle's user interface.

1 shows a schematic representation of a device 100 for operating a user interface of a vehicle 102.

The device 100 serves to personalize the user interface of the vehicle 102 for a user 104. For this purpose, the device 100 generates at least one element of the user interface, which can be output by an output unit 106, based on a user input and/or environmental information related to the surroundings of the vehicle 102.

The element generated by the device 100 can, for example, be an operating element or a decorative element of a graphical user interface that is output to the user 104 via a screen and/or a head-up display of the vehicle 102 as the output unit 106. Audible elements, such as a short piece of music or a voice output, can also be generated by the device 100 and output, for example, via a loudspeaker of the vehicle 102 as the output unit 106. Furthermore, the element can also be abstract, such as a light color or a sequence of light colors that can be output by lighting elements of the vehicle 102 as the output unit 106.

To capture information on the basis of which the user interface element is generated, the device 100 comprises a capture unit 108. The capture unit 108 processes the captured information and generates input data therefrom, which the capture unit 108 transmits to a control unit 110 of the device 100. The input data can be generated, in particular, in the form of a text in natural language, which is particularly easy to further process. However, the input data can also be generated in the form of any token that can be further processed by the control unit 110.

In the 1 In the embodiment shown, the detection unit 108 comprises an input unit 112, which is configured to detect a user input from the user 104. The input unit 112 can be a haptic input unit, for example an ego commander, a keyboard, or a touchscreen of the vehicle 102. However, the input unit 112 can also comprise a microphone and be configured to receive the user input in spoken form from the user 104. Furthermore, the input unit 112 can also be configured to receive the user input in the form of a gesture from the user 104.

The user input can, for example, be a prompt composed by the user 104 from predefined text modules or freely formulated, which describes the aesthetic qualities of the element to be created. In particular, the device 100 can also be configured to motivate the user 104 to enter the prompt by providing a corresponding output. For example, the device 100 can ask the user 104 about their favorite vacation destination, their favorite artist, and/or their favorite color. This output can be visual, for example, through text output via a screen and/or a head-up display of the vehicle 102. However, the output can also be audible, for example, in the form of a voice output generated by the device 100.

In the exemplary embodiment shown, the detection unit 108 also includes an environmental sensor system of the vehicle 102, which is configured to detect the environmental information. The environmental sensor system is shown purely by way of example as an external camera 114 of the vehicle 102. Using the external camera 114, for example, a predominant color tone in the surroundings of the vehicle 102 can be determined as the environmental information. The predominant color tone can be determined, for example, by the detection unit 108 processing an image captured by the external camera 114 using a clustering algorithm. Furthermore, the detection unit 108 can be configured to determine a time of day, a season, and/or the weather in the surroundings of the vehicle 102 as the environmental information based on the image.

In the exemplary embodiment shown, the detection unit 108 further comprises a navigation system 116 configured to determine or receive the position of the vehicle 102. The detection unit 108 may, for example, be configured to determine the geographical region in which the vehicle 102 is currently located as the environmental information based on the position of the vehicle 102.

The control unit 110 of the device 100 is configured to process the input data using a machine learning method to generate the user interface element. The machine learning method is, in particular, a generative model, for example, a generative adversarial network. If a visual element is to be generated, the control unit 110 can, in particular, use a text-to-image model to process the input data in the form of text in natural language. The generation of the user interface element is described below using the 2 described in more detail. In the 1 In the embodiment shown, the control unit 110 is part of the vehicle 102 itself. In other embodiments, at least part of the control unit 110 may be formed by a processing unit remote from the vehicle 102.

For example, the machine learning process can be executed on the remote processing unit. This makes it easier for the provider to update the machine learning process and does not use any storage space of the vehicle 102.

The 1 The exemplary device 100 shown further comprises a mobile terminal 118 of the user 104, which is shown purely as an example as a smartphone. The mobile terminal 118 is connected to the device 100, for example via a wireless data connection such as ^Bluetooth® , WLAN, or near-field communication (NFC). An application can run on the mobile terminal 118, which enables the user 104 to make user inputs to the device 100 via the mobile terminal 118. Furthermore, the control unit 110 can be configured to generate at least one element of a user interface of the mobile terminal 118 corresponding to the generated element of the user interface of the vehicle 102 and to transmit it to the mobile terminal 118. As a result, the user interfaces of the vehicle 102 and the mobile terminal 118 have a uniform look and feel.

2 shows a flowchart of a method for operating the user interface of the vehicle 102.

The method is started in step S200. The method can be started, in particular, based on a corresponding user input from user 104. In step S202, control unit 110 controls acquisition unit 108 to acquire the information on the basis of which the user interface element is to be generated. In a first example, device 100 requests user 104, via a corresponding output, to formulate a prompt from a set of predefined text modules. In a second example, device 100 asks user 104, via a corresponding output, about user 104's favorite vacation destination and favorite artist in order to generate the input data from user 104's response. In a third example, acquisition unit 108 acquires an image of the surroundings of vehicle 102 and, based on the image and using an image classification method, determines that vehicle 102 is currently located in an alpine environment. Based on this environmental information, the acquisition unit 108 generates the input data, for example as text in natural language: “alpine landscape”.

In step S204, the control unit 110 generates the element of the user interface of the vehicle 102 based on the input data and using the machine learning method. In the first example, the control unit 110 generates operating elements and decorative elements of a graphical user interface of the vehicle 102 using a text-to-image model based on the prompt formulated by the user 104. The control unit 110 further generates a short piece of music based on the prompt, which is output to the user 104 as part of a welcome scenario. In the second example, the control unit 110 generates a background image for the graphical user interface of the vehicle 102 using a generative adversarial network. This background image corresponds to an image of a landscape of the user's 104 favorite vacation destination in the style of the user's 104 favorite artist. In the third example, the control unit 110 generates controls and decorative elements of the graphical user interface of the vehicle 102 with an alpine theme.

In the optional step S206, the control unit 110 generates further elements of the user interface based on the already generated element. In the second example, the control unit 110 first extracts predominant colors from the generated background image using a clustering algorithm. The control unit 110 then colors the controls and decorative elements of the graphical user interface of the vehicle 102 in the extracted colors. The control unit 110 further generates instructions for lighting elements of the vehicle 102 to generate light in at least one of the predominant colors of the background image in order to illuminate the vehicle interior in these colors.

In step S208, the control unit 110 controls the output unit 106 to output the generated element(s). In the examples, the control unit 110 controls, for example, a screen of the vehicle 102 to display the graphical user interface with the generated controls and decorative elements. In the first example, the control unit 110 further controls a speaker of the vehicle 102 as the output unit 106 to play the generated piece of music when the user 104 enters the vehicle 102. In the second example, the control unit 110 further controls the lighting elements to illuminate the vehicle interior in the predominant colors of the background image.

The method is then terminated in step S210.

In the 1 and 2 In the embodiment described above, at least the output unit 106, the detection unit 108 and the control unit 110 form the device 100 for operating the user interface of the vehicle 102. Further 1 and 2 shown and in the previous Elements and features mentioned in the above description may be part of the device 100. Likewise, method steps described with reference to the device 100 may be part of the claimed method.

List of reference symbols

100

device

102

vehicle

104

user

106

Output unit

108

Registration unit

110

Control unit

112

Input unit

114

Image acquisition unit

116

navigation system

QUOTES CONTAINED IN THE DESCRIPTION

This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 10 2021 206 537 A1 [0003]
• US 10452902 B1 [0004]

Claims (15)

Device (100) for operating a user interface of a vehicle (102), with an output unit (106) of the vehicle (102) configured to output at least one element of the user interface of the vehicle (102) to a user (104); with a detection unit (108) configured to detect at least one piece of information corresponding to a user input of the user (104) and/or environmental information related to the surroundings of the vehicle (102), and to generate input data corresponding to the detected information; and with a control unit (110) configured to control the output unit (106) of the vehicle (102) and to process the input data using at least one machine learning method, wherein the control unit (110) is further configured to generate, based on the input data and using the machine learning method, at least one element of the user interface of the vehicle (102) that can be output by the output unit (106), and to control the output unit (106) to output the generated element. Device (100) according to Claim 1 , wherein the output unit (106) is configured to output visual information to the user (104), and the generated element is a visual element. Device (100) according to Claim 2 , wherein the control unit (110) is configured to generate a first image having a first resolution using the machine learning method and to generate a second image having a second resolution as the visual element based on the first image using a further machine learning method, wherein the second resolution is higher than the first resolution. Device (100) according to Claim 2 or 3 , wherein the control unit (110) is configured to generate at least one visual element, which is part of a graphical user interface of the vehicle (102), based on the input data and using the machine learning method. Device (100) according to Claim 4 , wherein the control unit (110) is designed to generate at least one further element of the user interface based on the generated visual element. Device (100) according to one of the Claims 2 until 5 , wherein the control unit (110) is configured to generate, based on the input data and using the machine learning method, a plurality of visual elements that are part of a uniform skin of the graphical user interface. Device (100) according to one of the Claims 2 until 6 , wherein the detection unit (108) is configured to provide the input data in the form of a text in natural language, and the control unit (110) is configured to process the input data using a text-to-image model as the machine learning method to generate the visual element. Device (100) according to one of the preceding claims, wherein the output unit (106) is configured to output auditory information to the user (104), and the element of the user interface is an auditory element. Device (100) according to one of the preceding claims, wherein the detection unit (108) is designed to detect as the information navigation information related to the vehicle (102) and/or weather information related to the surroundings of the vehicle (102). Device (100) according to one of the preceding claims, wherein the detection unit (108) comprises an image detection unit (114) of the vehicle (102) which is designed to detect at least one image with a depiction of an area outside the vehicle (102) as the information and to generate image data corresponding to the image, and the input data comprises the image data. Device (100) according to one of the preceding claims, wherein the detection unit (108) comprises a mobile terminal (118) which is designed to detect the information, in particular the user input of the user (104). Device (100) according to one of the preceding claims, wherein the machine learning method is a generative model, in particular a generative adversarial network. Device (100) according to one of the preceding claims, wherein the control unit (110) is designed to control the output unit (106) to output at least one predefined question to the user (104), and the detection unit (108) is designed to detect the response of the user (104) in the form of a user input as the information and to generate the input data of the response accordingly. Device (100) according to one of the preceding claims, wherein the detection unit (108) is designed to detect feedback from the user (104) related to the generated element and to generate feedback data corresponding to the feedback, and the control unit (110) is designed to modify the machine learning method at least on the basis of the feedback data. A method for operating a user interface of a vehicle (102), comprising: capturing at least one piece of information corresponding to a user input from the user (104) and/or environmental information related to the surroundings of the vehicle (102); generating input data corresponding to the captured information; generating at least one element of the user interface of the vehicle (102) based on the input data and using a machine learning method; and outputting the generated element by an output unit (106) of the vehicle (102).

Images