DE102022002701A1 - Method for operating a voice assistant and vehicle with a voice assistant - Google Patents

Abstract

The invention relates to a method for operating a voice assistant (1). The method according to the invention is characterized in that a decision module (9) comprises a first database (11), with intention parameters (7) sorted by functional groups (12) being stored in the first database (11); and the following method steps are carried out: Analyzing the intention parameter (7) by the decision module (9) and determining a functional group (12) for which the intention parameter (7) applies; - Determining a functional framework (10) to which the intention parameter (7) is to be applied by the decision module (9) depending on the determined functional group (12); - forwarding the intention parameter (7) to a switching module (13) higher than the determined functional framework (10); - determining an application (APP) for which the intention parameter (7) is to be applied by the switching module ( 13);- Forwarding the intention parameter (7) and the specific application (APP) from the switching module (13) to a respective control module (14) downstream of the switching module (13);- Translating the intention parameter (7.) to be used for the specific application (APP). ) in at least one control command by the control module (14) that can be interpreted by the respective functional framework (10); and- controlling at least the specific application (APP) depending on the at least one control command.

Description

The invention relates to a method for operating a voice assistant according to the type defined in more detail in the preamble of claim 1 and to a vehicle with a voice assistant.

Voice assistants are familiar in many ways from everyday life. For example, a user can interact verbally with a smartphone in order to use functionality provided by the smartphone. Using a voice command, the user can cause a contact from a phone book to be called, a short message to be sent to a contact, an Internet search to be started or navigation guidance to be started. Voice assistants are also known as stationary devices for the household. You can interact with the voice assistant using voice, for example to check a weather report, set an alarm clock or place an order for an item via an online shop. Voice assistants can also be installed in vehicles and enable a person driving the vehicle to interact, for example with the vehicle's infotainment system.

One challenge that arises when using voice assistants is the correct interpretation by the voice assistant of the underlying intent of the voice command issued by a user. Correct interpretation is made more difficult when the voice commands for different functionalities are similar. In particular, if similar or the same functionalities are provided by multiple applications running on a corresponding device, there is a risk of misinterpretation of the voice command issued by the user, which is accompanied by a reduced user experience. For example, the voice assistant selects the wrong application to provide a functionality and/or executes a functionality that does not correspond to the intention underlying the voice command. This creates a need to provide methods and means that increase user comfort when using a voice assistant in an environment in which the voice assistant can control multiple applications.

A voice assistant, for example, is from the US 7,505,910 B2 known. The voice assistant enables a user to simultaneously interact with the applications running on a computing unit in the foreground and in the background. In the foreground this means that the user can interact with an application via a path other than voice, for example by operating operating devices, and in the background it means that the user cannot interact with the application via the corresponding path other than language. The voice commands used to interact with the voice assistant are divided into global and local voice commands, whereby only an application running in the foreground can be controlled using a local voice command and both an application running in the foreground and an application running in the background can be controlled via a global voice command can be controlled. Each application has its own library with interpretable voice commands. This can lead to conflicts if voice commands collide, i.e. multiple applications use the same voice command.

From the US 8,577,424 B2 A method and a device for communication between a vehicle computer system and an application running externally to the vehicle are also known. The application can, for example, be executed on a mobile device such as a smartphone, connect to the vehicle computer system and thereby read out vehicle data. Voice control of the application is possible.

Another voice assistant is also from the US 9,548,066 B2 known. The voice assistant runs on a voice interface device. The voice interface device can execute multiple applications or use the functionalities of applications executed on a central computing device. With the help of a command interpreter, an application suitable for executing a voice command is selected. However, the document does not go into detail at any point about the criteria used to select the most suitable application for executing a command. There is therefore a risk that voice commands will be misinterpreted and therefore incorrect functionality will be carried out or an application that is not desired by the user will be used to carry out the functionality.

The present invention is based on the object of specifying an improved method for operating a voice assistant, with the help of which the user experience when using a voice assistant with access to several applications is improved.

According to the invention, this object is achieved by a method for operating a voice assistant with the features of claim 1. Advantageous designs and further developments as well as a vehicle with a voice assistant result from the dependent requirements.

• - das Entscheidungsmodul eine erste Datenbank umfasst, wobei in der ersten Datenbank Absichtsparameter nach Funktionsgruppen sortiert gespeichert sind;
• - der Sprachassistent für jedes Funktionsrahmenwerk ein individuelles Vermittlungsmodul umfasst; und
• - für jedes Vermittlungsmodul ein individuelles Ansteuerungsmodul zur bidirektionalen Informationsweitergabe zwischen dem jeweiligen Vermittlungsmodul und dem jeweiligen Funktionsrahmenwerk vorgesehen ist; und folgende Verfahrensschritte ausgeführt werden:
• - Analysieren des Absichtsparameters durch das Entscheidungsmodul und Ermitteln einer Funktionsgruppe für die der Absichtsparameter gilt;
• - Ermitteln eines Funktionsrahmenwerks auf das der Absichtsparameter anzuwenden ist durch das Entscheidungsmodul in Abhängigkeit der ermittelten Funktionsgruppe;
• - Weiterleiten des Absichtsparameters an das dem ermittelten Funktionsrahmenwerk übergeordnete Vermittlungsmodul;
• - Bestimmen einer Applikation für die der Absichtsparameter anzuwenden ist durch das Vermittlungsmodul;
• - Weiterleiten des Absichtsparameters und der bestimmten Applikation vom Vermittlungsmodul an das jeweilige dem Vermittlungsmodul nachgelagerte Ansteuerungsmodul;
• - Übersetzen des für die bestimmte Applikation anzuwendenden Absichtsparameters in wenigstens einen durch das jeweilige Funktionsrahmenwerk interpretierbaren Steuerungsbefehl durch das Ansteuerungsmodul; und
• - Ansteuern zumindest der bestimmten Applikation in Abhängigkeit des wenigstens einen Steuerungsbefehls.

A generic method for operating a voice assistant, wherein the voice assistant detects an acoustic signal and converts it into an audio signal, a voice recognition module reads the audio signal and converts it into a character string, a natural language recognition module reads the character string and therein at least one intention parameter for controlling a a computing unit recognizes the application running in the foreground or in the background and a decision module reads the intention parameter and, depending on the intention parameter, causes the application to be controlled and at least one control command used to control the application to be determined, with control commands that can be issued being defined by at least one functional framework according to the invention further developed in that

• - the decision module comprises a first database, wherein intention parameters are stored sorted by functional groups in the first database;
• - the language assistant includes an individual mediation module for each functional framework; and
• - an individual control module is provided for each switching module for bidirectional information transfer between the respective switching module and the respective functional framework; and the following procedural steps are carried out:
• - Analyzing the intent parameter by the decision module and determining a functional group to which the intent parameter applies;
• - Determining a functional framework to which the intention parameter is to be applied by the decision module depending on the determined functional group;
• - forwarding the intention parameter to the switching module higher than the determined functional framework;
• - Determining an application for which the intent parameter is to be applied by the switching module;
• - forwarding the intention parameter and the specific application from the switching module to the respective control module downstream of the switching module;
• - Translating the intention parameter to be used for the specific application into at least one control command that can be interpreted by the respective functional framework by the control module; and
• - Controlling at least the specific application depending on the at least one control command.

With the help of the method according to the invention, the user comfort when using a voice assistant in an environment in which the voice assistant can control several applications can be reliably improved and thus the user experience can be increased. By means of the method according to the invention, a type of translator is created which allows voice commands to be formulated according to a standard pattern and translates the voice commands into the format that can be processed for the respective application for which the voice command is intended. This enables reliable control of a wide variety of applications and makes it easier for a user to operate, since the user only has to remember the standard pattern for formulating the voice commands and does not have to rethink for each application to be controlled. The structure of the language assistant described above also enables a particularly computationally efficient provision of the functionalities. Furthermore, the method according to the invention enables the automated processing of voice commands with a wide variety of applications without the individual functions of the applications having to be explicitly specified in the decision module, which means that the addition of new applications is accompanied by greatly reduced effort.

The method according to the invention allows applications running on the computing unit in the foreground as well as in the background to be controlled. The computing unit is able to control a display device for displaying a graphical user interface, also referred to as a graphical user interface (GUI), in order to display information relating to applications on the display device and in a display device set up to record operating actions, for example a Touch-sensitive display display device, capable of receiving corresponding operating actions for controlling the applications via the graphical user interface. In the following, an application is understood to be running in the foreground, for which relevant information is displayed on the display device and which can be controlled by entering operating actions. Applications which are executed on the computing unit, but for which no or only redundant information is displayed on the display device and which cannot be controlled directly by entering operating actions, are understood to be executed in the background. The possibility of using language both in the foreground and Being able to control applications that are running in the background also improves the user experience even further. This means that the user does not have to first bring an application to the foreground so that the user can operate the corresponding application.

To detect acoustic signals, the voice assistant can receive information from detection means, for example from one or more microphones. The audio signal therefore corresponds to a coding of the noises described by the acoustic signal. The speech recognition module is able to recognize speech in the audio signal and convert corresponding words into a character string. The character string therefore represents a text string. This character string is analyzed by the natural language recognition module. Proven algorithms can be used for this. The computing unit is therefore able to recognize the intention with which a corresponding voice command was issued by a user. Multiple intentions can also be contained in a voice command.

An application can also be referred to as a program or software. An application that can be run on the computing unit can be executed by the computing unit, whereby the functionalities that can be provided by the corresponding application are used or are available. The application can be, for example, an operating system of the computing unit, an application for controlling the graphical user interface displayed on the display device, an application for playing media content such as video content, for example films, series, music videos and the like, audio content such as audio books, pieces of music, Podcasts and the like. It can also be an application for carrying out communication, such as a telephone client, a chat program or a video conference program. The functional framework, also known as a framework, defines the rules according to which interaction with the respective applications is possible. The functional framework therefore specifies which information is output by applications according to which pattern and which control commands are to be used and how to control the corresponding applications. Depending on the type of executable application, several functional frameworks can be present, for example a first functional framework for interacting with communication applications such as one or more applications for conducting a web meeting and one or more applications for chatting, as well as a second functional framework for interacting with an application for controlling the graphical user interface and a third functional framework for interacting with applications for playing media content.

An intention parameter contains at least implicit information about which application is to be operated by the voice command and how exactly the operation is to be carried out, i.e. which functionality is desired by the user. By dividing the process flow into the individual modules, a particularly computationally efficient and reliable interpretation of the voice commands is made possible. The decision module makes an initial preselection as to which groups of applications come into question using the corresponding voice command. For this purpose, the decision module selects the respective functional framework that is suitable for executing the voice command and thus the respective switching module upstream of the selected functional framework.

Individual applications can also be assigned to several function groups. For example, an application that provides a chat function can also enable the transmission of voice messages and/or the provision of video telephony. The corresponding application can then be assigned to the telephony function group, the text messaging function group and the video telephony or conference function group.

An assignment of intent parameters to function groups based solely on a command contained in the intent parameter can only be unique to a limited extent. For example, a voice command can contain: “Increase volume”. This command can be used, for example, by an application to provide video telephony, to play media content, to conduct a video conference, to provide barrier-free operation of a vehicle infotainment system or the like.

For the ambiguous cases described above, further information can be included in the voice command according to the standard pattern for formulating voice commands, which enables the decision module to clearly decide, even in such a scenario, for which functional group the respective intention parameter is valid. This will be discussed in more detail below.

However, the decision module itself does not yet select an application for which the control command should apply. This is carried out by the respective switching module. The intention parameter is then transferred to the functional framework determined by the decision module ordered switching module and the corresponding application to be controlled is determined by the switching module. The application can be determined by the switching module based on rules. The individual switching modules can have different rules. These rules describe which application is selected for control based on the information contained in an intention parameter.

The respective control modules then have their own rules, which specify how control commands can be derived from the respective intention parameters, or the information contained therein explicitly or implicitly, which can be interpreted by a respective functional framework. These rules define how the respective control module “translates” commands and are specified by programmers. If new applications come onto the market that are intended to be executed on the computing unit, the corresponding set of rules must be tailored to the new applications. This enables a reliable option for controlling a wide variety of applications, including new ones, using voice commands formulated according to a standard pattern.

• - die Bezeichner sämtlicher auf der Recheneinheit lauffähigen Applikationen; und
• - eine Priorisierungsliste, wobei die Priorisierungsliste umfasst, welche Funktion welchen Applikation bevorzugt von der Recheneinheit ausgeführt werden soll, für den Fall, dass die gleichzeitige Ausführung wenigstens zweier Funktionen durch wenigstens eine Applikation von der Recheneinheit nicht möglich ist.

An advantageous development of the method provides that the switching modules access a second database in which at least the following is stored:

• - the identifiers of all applications that can run on the computing unit; and
• - a prioritization list, wherein the prioritization list includes which function of which application should preferably be executed by the computing unit, in the event that the simultaneous execution of at least two functions by at least one application of the computing unit is not possible.

The identifier of an application can be a name of the respective application, an identification number or the like. With the help of the identifier, applications can be clearly referenced. In this context, executable means that corresponding applications are generally intended to be executed on the computing unit. There can be another database, also referred to as an app store, from which applications that can run on the computing unit can be reloaded for execution on the computing unit. If a user wants to use a certain function, it may happen that the corresponding application cannot be provided by the computing unit because the respective application is not installed on the computing unit. However, if the respective application can run on the computing unit, i.e. is available for download to the computing unit from the app store, for example, then the user of the voice assistant can be offered the option of having an application that has not yet been installed loaded onto the computing unit and on the computing unit is installed. By accessing the second database, the corresponding switching modules therefore have information about which applications are generally available for use on the computing unit.

Using the prioritization list, program collisions can be avoided. An application can, for example, be an application for displaying a video stream to be displayed on the display device by a rear-view camera of a vehicle and another application can be a program for using video telephony. It is then stored in the prioritization list that the display of the video stream from the rear view camera on the display device is to be given priority over all other image content. The corresponding video stream from the rear-view camera is displayed on the display device and the video telephony is carried out on the computing unit in the background and the corresponding video image of a conversation partner is only displayed on the display device when no more video stream is received from the vehicle's rear-view camera or the computing unit receives a signal to stop displaying the video stream from the rear view camera.

According to a further advantageous embodiment of the method, the control modules access a third database in which at least the identifiers of the applications that can currently be executed on the computing unit are stored. In this context, currently executable means that the respective applications are installed on the computing unit. Executable also means that the respective applications can be executed, i.e. installed, on a further computing unit or operating system emulated or simulated by the computing unit or on a physical further computing unit coupled to the computing unit. This provides the computing unit with particularly simple and reliable information about which applications can be controlled directly and do not have to be reloaded from the app store.

In general, it is also possible for a control module to be able to access the second database and/or a switching module to be able to access the third database.

A further advantageous embodiment of the method according to the invention further provides that the control modules are in communication connection with one another and are set up to read out a status of the respective applications accessible via the functional framework downstream of the respective control module and to exchange it with one another via the communication connection. The status of an application can be the corresponding installation status of the application on the computing unit. The status can also be a state of an application that is currently running on the computing unit in the foreground or in the background. Such a state describes various properties of the applications. The following exemplary states are mentioned for an application for playing media content: the media content is being played, the media content is paused, a playback point of the media content is at position minute 3 and second 37, media content number 7 from a list of 15 media contents is being played, one Expression of a playback volume and the like. The following exemplary states are mentioned for an application for composing text messages: a window with a chat history with contact XY is displayed on the display device using the graphical user interface, a text message is currently being written, a voice message is currently being recorded, an image is being downloaded and the like . The status also describes, for example, whether a corresponding application is currently running, in particular whether the application is running in the foreground or in the background. For example, the status can describe that no video stream is currently being received from a vehicle rear view camera and can be displayed on the display device.

The individual control modules can then exchange a corresponding status with one another via the communication connection. This enables a control module to also process information from an application that is accessible through a different functional framework than the functional framework downstream of the respective control module. This enables even more extensive options for interaction between users and applications as well as the automatic derivation of control commands by the control modules.

Thus, a control module preferably determines at least one additional control command for controlling at least one application accessible to the control module via the functional framework downstream of the control module, depending on a status received from a further control module. In other words, this makes it possible to adapt the control behavior of a respective application by a control module depending on another application currently being executed by the computing unit. For example, if “increase volume by three points” is received as a voice command, the playback volume of a program for playing audio content can be increased if the corresponding application is running in the foreground or background. However, if no application for playing media content is running, a system volume can be increased instead to acoustically output vehicle information, such as warning tones. This can also be expressed as the computing unit intelligently deciding, depending on system behavior, how a respective voice command should be optimally implemented. Here too, there is a set of rules according to which criteria which additional control commands should be issued by the control modules depending on the respective status. This set of rules can be stored in the respective control modules themselves or externally, for example in the third database.

A further advantageous embodiment of the method according to the invention further provides that behavioral information, which describes a result of the activity of the control module or the switching module, is generated by a respective control module and / or switching module and behavioral information is transmitted from a control module to a switching module upstream of the respective control module and are transmitted from a switching module to the decision module, the behavior of the switching module being adaptable depending on the behavioral information received from the downstream control module and the behavior of the decision module depending on the behavioral information received from the switching module and / or the behavioral information generated by the control module and forwarded via the switching module is customizable. The behavioral information generated by the control module and forwarded by the switching module can be forwarded unchanged by the switching module or can also be adapted by the switching module. A switching module and control module arranged one behind the other in the information flow direction can also simultaneously generate behavioral information and forward it to the decision-making module. The behavioral information from the control module can be combined with the behavioral information from the switching module, i.e. the behavioral information from the control module communication module can be included in the behavioral information of the switching module.

Forwarding the behavioral information to the decision module enables the decision module to monitor the activities of the switching modules and the control modules at any time and, depending on a potentially occurring result, to control the respective switching modules and / or control modules depending on the respective activities in order to ensure that they change their behavior. This gives the decision-making module the ability to function as a central decision-making body.

According to a further advantageous embodiment of the method according to the invention, the decision module is executed partly on the computing unit and partly on a central computing device which is in communication connection with the computing unit. The central computing device can be a cloud server, often also referred to as a backend. Such a cloud server often has comparatively powerful hardware, which allows computationally intensive procedures to be processed quickly and reliably. Particularly powerful algorithms can thus be implemented on the central computing device, which enable reliable assignment of functional groups depending on intention parameters regardless of the complexity of a voice command. However, this provision and use of so-called cloud services relies on an active communication connection between the computing unit and the central computing device. The computing unit can, for example, be connected to the Internet via a telecommunications unit via mobile radio and can exchange information with the central computing device. If the telecommunications unit has limited or no mobile phone reception, for example because it is in a dead zone, the corresponding functionalities of the cloud service cannot be used. For this reason, it is advantageous to also execute at least part of the decision module locally on the computing unit. Depending on the performance of the computing unit, the allocation ability or decision-making ability of the decision module can then be reduced. However, continued usability of the voice assistant is ensured when the telecommunications unit is in a dead zone.

A further advantageous embodiment of the method further provides that the natural language recognition module uses logic to recognize intention parameters in the character string, the logic providing that a voice command represented by the character string contains at least one activator and one command as well as a supplementary one Target includes, and wherein the activator represents a defined term for activating the functionality provided by the decision module, the command represents a term identifying at least one control command and the target represents a term identifying an application to be controlled. The logic applied by the natural language recognition module thus defines the standard pattern according to which voice commands are formulated. Every voice command contains at least the activator and at least one command. Additionally, a destination can also be included in a voice command. In the following, a term is understood to mean a single word or a chain of words. For example, the following can be used as an activator: “Hey Mercedes”, “OK voice assistant”, or the like. In other words, the activator serves to enable a user to inform the voice assistant that a voice command is now being entered, which is to be carried out by the voice assistant. This enables a particularly efficient method for operating the voice assistant, since not all words issued by a user have to be examined for the content of commands, but only those words or word chains that follow an activator.

Various keywords or keyword chains can be used as commands. The commands are stored in the first database and are fixed by programmers according to the standard pattern. Thus, for example, to increase a volume regardless of a respective target application, a user only needs to remember the command “Increase volume by x” instead of, for example, “Volume up by x steps” for a first application and “Increase volume” for a second application and the same. This standardizes and therefore simplifies the control of different applications using voice commands.

By specifying the target, the user is able to clearly tell the computing unit or decision module which application is intended to be controlled. The target can therefore be, for example, the name of a corresponding application.

If an application to be controlled by the user is not running on the computing unit, the user can, for example, enter by voice: “Start application”, which starts the corresponding application. To further improve the user experience, the separate Starting an application is also no longer necessary. The user can also directly issue a command via voice for an inactive, i.e. not running, application, whereupon the application is started and executed in the foreground or background and controlled according to the command. Such a command could, for example, be: “Hey Mercedes, compose a message to Michael via SMS”. If the application for writing SMS is running in the foreground, the text message can be displayed on the display device during voice input by the user. However, if the application for writing SMS is running in the background, there is no display on the display device or only a letter symbol is displayed as redundant information during the voice recording of the text message and during sending. The successful writing and sending of the text message can then be confirmed, for example, by an acoustic indication, for example a confirmation of the type “Message sent successfully” read out by a computer voice.

According to a further advantageous embodiment of the method, if the natural language recognition module does not recognize a target in a character string, the decision module causes the application to be controlled to be the application currently running in the foreground on the computing unit, the application that was last executed in the foreground or a most frequently used application The application running on the computing unit is selected. It is irrelevant whether the application most frequently executed on the computing unit is or was executed in the foreground or in the background on the computing unit. “Most” can be defined in different ways, for example as the application that was executed on the computing unit for the longest period of time during the period of use of the computing unit or with which the user interacted for the longest time. In this way, user comfort can be increased even further, since the need to include a destination in a voice command for the user can be eliminated. The corresponding decision is also based on fixed rules.

Preferably, at least one application to be controlled is executed in a virtual machine running on the computing unit or on an external mobile computing device coupled to the computing unit. This enables even more in-depth control of applications via voice commands. For example, an operating system on which a first number of applications are executed can be executed on the computing unit. The operating system can be, for example, the operating system of a vehicle control unit. Various applications can be executed on this, such as a radio application, a navigation application, an application for setting vehicle functions, an application for reading out vehicle data and the like. A virtual machine can be executed on the computing unit, which in turn can emulate or simulate another computing unit with another operating system. This operating system can be, for example, Android, Windows, Linux or iOS. This means that corresponding Android apps, Windows apps or Apple apps can be run on the computing unit. A mobile device such as a smartphone, tablet computer, laptop or the like can be coupled to the computing unit as an external mobile computing device. The coupling can be wired, for example via USB cable or Ethernet cable, or wireless, for example via Bluetooth, WIFI, NFC or the like. This means that a corresponding user's Android phone or iPhone can also be operated using a voice command. The voice commands natively specified by the respective operating system, for example Android, are overwritten by the voice commands defined according to the standard pattern. This enables a uniform user experience for the voice assistant user.

In a vehicle with a voice assistant, according to the invention, the voice assistant is set up to be operated according to a method described above. The vehicle can be any vehicle such as a car, truck, van, bus or the like. This uniform user experience is then provided in such a vehicle.

Further advantageous embodiments of the method according to the invention for operating a voice assistant also result from the exemplary embodiments, which are described in more detail below with reference to the figures.

• 1 einen schematisierten Systemaufbau eines gemäß eines erfindungsgemäßen Verfahrens betreibbaren Sprachassistenten;
• 2 eine schematisierte Darstellung einer ersten Datenbank; und
• 3 eine Darstellung eines in einer Zeichenkette gewandelten Sprachbefehls.

Show:

• 1 a schematic system structure of a voice assistant that can be operated according to a method according to the invention;
• 2 a schematic representation of a first database; and
• 3 a representation of a voice command converted into a character string.

1 shows the system structure of a voice assistant 1 that can be operated according to a method according to the invention. The voice assistant 1 is executed on a computing unit 8 and allows APP applications to be controlled via voice command. An acoustic detection means 19, for example a microphone, detects an acoustic signal 2. This is converted into an audio signal 3, which is read in by a speech recognition module 4. The speech recognition module 4 converts the audio signal 3 into a character string 5. The character string 5 is in turn read by a natural language recognition module 6, which recognizes at least one intention parameter 7 therein. This is in turn read in by a decision module 9. The decision module 9 has access to an in 2 First database 11 shown. In the first database 11 possible intention parameters 7 are stored sorted according to functional groups 12. The decision module 9 analyzes corresponding intention parameters 7 and determines which functional group 12 the respective intention parameter 7 can be assigned to.

An operating system OS is executed in an application environment 20. The application environment 20 can be the computing unit 8 itself, a virtual machine running on the computing unit 8, or an external mobile computing device coupled to the computing unit 8, for example a mobile terminal such as a smartphone. The operating system OS can be, for example, an Android or iOS operating system. However, the operating system OS can also be the operating system of a vehicle control unit, for example an infotainment system. Said applications APP are executed on the operating system OS. The operating system OS can also be referred to as an application APP. Depending on the functionalities provided, the individual APP applications can be addressed via different functional frameworks 10. This makes it possible to read out information and transmit control commands via a respective functional framework 10 to the respective APP applications.

The decision module 9 determines a functional framework 10 to which the respective intention parameter 7 is to be applied depending on the determined functional group 12. The intention parameter 7 is then forwarded to a switching module 13 that is higher-level than the determined functional framework 10. The respective switching module 13 determines an application APP for which the intention parameter 7 is to be applied. The switching module 13 then forwards the intention parameter 7 together with the specific application APP to a respective control module 14 downstream of the switching module 13. The individual switching modules 13 and control modules 14 can also be referred to as so-called “handlers”.

The respective control module 14 translates the intention parameter 7 to be used for the specific application APP into at least one control command that can be interpreted by the respective functional framework 10, which is then used for control by the respective specific application APP.

The individual control modules 14 are in communication connection with one another, which enables the status of a respective application APP to be read out and shared across the control modules 14.

The respective control modules 14 and switching modules 13 can also generate behavioral information and transmit it to the respective higher-lying entity in the information flow direction in order to adapt the respective behavior of the higher-lying entity. This enables the decision-making module 9 to form a central, decision-making body.

Parts of the decision module 9 can also be outsourced to a central computing device 15, for example a cloud server.

2 shows a representation of the first database 11. In the exemplary embodiment in 2 the first database 11 is symbolized by a table, with the respective function groups 12 listed in a first column of the table and the respective function groups 12 available for a corresponding function group 12 and included in the intention parameters 7 in the second column 3 commands shown 17.

Individual intention parameters 7, or the commands 17 contained therein, as well as different applications APP can also be assigned to several function groups 12.

3 shows an example of a voice command converted into a character string 5. To process voice commands, the voice commands follow a defined logic. For this purpose, a respective voice command includes an activator 16, a command 17 and, in addition, a target 18. Each voice command is based on an intention 21 of a user to control at least one application APP. This intention 21 is to be understood abstractly and is formulated in an understandable way for the voice assistant by the part of the voice command attached to the activator 16. In the in 3 In the exemplary embodiment shown, the user's intention is to select randomly selected music Play a piece by Michael Jackson on the music streaming service “Deezer”.

The voice command can contain further information 22. The additional information 22 can also be omitted. For example, another voice command without the additional information 22 could also be “Hey Mercedes, open the email program”.

1. 1: Verfügt die Recheneinheit 8 generell über die Fähigkeit bzw. Autorisierung zur Ausführung von Applikationen APP? ist dies nicht der Fall, so wird der Nutzer darüber informiert, dass die Applikation APP nicht genutzt werden kann.
2. 2: Können die anzusteuernde Applikation APP und/oder der durch den Absichtsparameter 7 beschriebene Befehl 17 aufgefunden werden? In diesem Falle prüft das Vermittlungsmodul 13, ob die entsprechende Applikation APP aus einem App-Store auf die Recheneinheit 8 heruntergeladen und installiert werden kann. Ist dies der Fall, so wird der Nutzer vorher um Erlaubnis gefragt. Existiert hingegen die entsprechende Applikation APP nicht im App-Store, so wird der Nutzer informiert, dass die Applikation APP auch nicht installiert werden kann. Ist hingegen die entsprechende Applikation APP installiert, kann jedoch den dem Absichtsparameter 7 zugrunde liegenden Befehl 17 nicht ausführen, so wird ebenfalls der Nutzer darüber informiert.
3. 3: Kann die anzusteuernde Applikation APP in einer aktuellen Situation genutzt werden? Dies ist beispielsweise der Fall, wenn der Nutzer ein Fahrzeug steuert und das Verwenden der Applikation APP die Sicherheit im Straßenverkehr gefährden würde, wie durch das Abspielen von Filmen auf der Anzeigevorrichtung einer Headunit während der Fahrt. Ebenfalls können Funktionalitäten einer Applikation APP geblockt werden, beispielsweise weil die Funktionalität einer anderen Applikation APP bevorzugt bzw. priorisiert wird. So wird auf eine Anzeigevorrichtung im Fahrzeug beispielsweise der Videostrom einer Rückfahrkamera dargestellt und auf eine Darstellung einer entsprechenden Telefonieapplikation verzichtet.

Another command could be “Hey Mercedes, call Johannes on WhatsApp”. The decision module 9 determines “telephony” as function group 12, since the command 17 of the corresponding intention parameter 7 is assigned to this function group 12. The decision module 9 then determines the functional framework 10 available for telephony, whereby the switching module 13 and control module 14 to be used for the further process are determined. The respective switching module 13 receives the intention parameter 7 and recognizes which application APP is to be controlled, in particular taking into account the goal 18. Before the corresponding application APP is controlled, the following checks are carried out by the switching module 13, with the downstream control module 14 providing the necessary information can:

1. 1: Does the computing unit 8 generally have the ability or authorization to execute APP applications? If this is not the case, the user will be informed that the APP application cannot be used.
2. 2: Can the APP application to be controlled and/or the command 17 described by the intent parameter 7 be found? In this case, the switching module 13 checks whether the corresponding application APP can be downloaded from an app store onto the computing unit 8 and installed. If this is the case, the user will be asked for permission beforehand. However, if the corresponding APP application does not exist in the App Store, the user is informed that the APP application cannot be installed either. However, if the corresponding application APP is installed but cannot execute the command 17 on which the intention parameter 7 is based, the user will also be informed of this.
3. 3: Can the APP application to be controlled be used in a current situation? This is the case, for example, if the user is driving a vehicle and using the APP application would endanger road safety, such as playing films on the display of a head unit while driving. Functionalities of an application APP can also be blocked, for example because the functionality of another application APP is preferred or prioritized. For example, the video stream from a rear-view camera is displayed on a display device in the vehicle and a corresponding telephony application is not displayed.

After all tests have been carried out successfully, the APP application is controlled in accordance with the control command translated by the control module 14.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 7505910 B2 [0004]
• US 8577424 B2 [0005]
• US 9548066 B2 [0006]

Claims (11)

Method for operating a voice assistant (1), wherein the voice assistant (1) detects an acoustic signal (2) and converts it into an audio signal (3), a voice recognition module (4) reads the audio signal (3) and converts it into a character string (5). , a natural language recognition module (6) reads the character string (5) and recognizes therein at least one intention parameter (7) for controlling an application (APP) running on a computing unit (8) in the foreground or in the background, and a decision module (9) reads the intention parameter (7) and, depending on the intention parameter (7), causes the application to be controlled (APP) and at least one control command used to control the application (APP) to be determined, control commands that can be issued being defined by at least one functional framework (10). , characterized in that - the decision module (9) comprises a first database (11), wherein in the first database (11) intention parameters (7) are stored sorted by functional groups (12); - the language assistant (1) comprises an individual switching module (13) for each functional framework (10); and - an individual control module (14) is provided for each switching module (13) for bidirectional information transfer between the respective switching module (13) and the respective functional framework (10); and the following method steps are carried out: - analyzing the intention parameter (7) by the decision module (9) and determining a functional group (12) for which the intention parameter (7) applies; - Determining a functional framework (10) to which the intention parameter (7) is to be applied by the decision module (9) depending on the determined functional group (12); - Forwarding the intention parameter (7) to the switching module (13) higher than the determined functional framework (10); - Determining an application (APP) for which the intention parameter (7) is to be applied by the switching module (13); - forwarding the intention parameter (7) and the specific application (APP) from the switching module (13) to the respective control module (14) downstream of the switching module (13); - Translating the intention parameter (7) to be used for the specific application (APP) into at least one control command that can be interpreted by the respective functional framework (10) by the control module (14); and - controlling at least the specific application (APP) depending on the at least one control command. Procedure according to Claim 1 , characterized in that the switching modules (13) access a second database in which at least: - the identifiers of all applications (APP) that can be run on the computing unit (8) are stored; and - a prioritization list, wherein the prioritization list includes which function of which application (APP) should preferably be executed by the computing unit (8), in the event that the simultaneous execution of at least two functions by at least one application (APP) is carried out by the computing unit (8) is not possible. Procedure according to Claim 1 or 2 , characterized in that the control modules (14) access a third database in which at least the identifiers of the applications (APP) that can currently be executed on the computing unit (8) are stored. Procedure according to one of the Claims 1 until 3 , characterized in that the control modules (14) are in communication connection with one another and are set up to read out a status of the respective applications (APP) accessible via the functional framework (10) downstream of the respective control module (14) and to exchange them with one another via the communication connection. Procedure according to Claim 4 , characterized in that a control module (14) has at least one additional control command for controlling at least one application (APP) accessible to the control module (14) via the functional framework (10) downstream of the control module (14) depending on a further control module (14). received status. Procedure according to one of the Claims 1 until 5 , characterized in that behavioral information, which describes a result of the activity of the control module (14) or the switching module (13), is generated by a respective control module (14) and / or switching module (13) and behavioral information from a control module (14). a switching module (13) upstream of the respective control module (14) are transmitted and transmitted from a switching module (13) to the decision module (9), the behavior of the switching module (13) being adaptable depending on the behavioral information received from the downstream control module (14). is and the behavior of the decision module (9) depending on the behavioral information received from the switching module (13) and/or that generated and transmitted by the control module (14). the switching module (13) can be adapted to the behavioral information forwarded. Procedure according to one of the Claims 1 until 6 , characterized in that the decision module (9) is executed partly on the computing unit (8) and partly on a central computing device (15), which is in communication connection with the computing unit (8). Procedure according to one of the Claims 1 until 7 , characterized in that the natural language recognition module (6) applies logic to recognize intent parameters (7) in the character string (5), the logic providing that a voice command represented by the character string (5) at least one activator (16) and a command (17) as well as a target (18), and wherein the activator (16) represents a defined term for activating the functionality provided by the decision module (9), the command (17) represents at least one control command represents an identifying term and the target (18) represents a term identifying an application (APP) to be controlled. Procedure according to Claim 8 , characterized in that if the natural language recognition module (6) does not recognize a target (18) in a character string (5), the decision module (9) causes the application (APP) currently on the computing unit (8) to be controlled. Application (APP) running in the foreground, the application (APP) running most recently in the foreground or an application (APP) running most frequently on the computing unit (8) is selected. Procedure according to one of the Claims 1 until 9 , characterized in that at least one application (APP) to be controlled is executed in a virtual machine running on the computing unit (8) or on an external mobile computing device coupled to the computing unit (8). Vehicle with a voice assistant (1), characterized in that the voice assistant (1) according to a method according to one of Claims 1 until 10 is operable.

Images