DE102019007797A1 - Align voice commands while testing voice-assisted app prototypes for languages with non-phonetic alphabets - Google Patents

Abstract

It describes the matching of voice commands during the testing of voice-supported app prototypes for languages with non-phonetic alphabets. A visual page of an app prototype is displayed during a test phase of the app prototype. A spoken-to-text service converts a non-phonetic voice command that is spoken in a language with a non-phonetic alphabet and is picked up by at least one microphone during the test phase of the app prototype into a non-phonetic text string of the voice command given in the non-phonetic alphabet . A phonetic language translator translates the non-phonetic text string of the voice command into a phonetic text string of the voice command given in a phonetic alphabet. A comparison module compares the phonetic text string of the voice command with phonetic text strings of stored voice commands given in the phonetic alphabet, which are assigned to the app prototype, in order to identify a suitable voice command. An implementation module performs an action that is assigned to the appropriate voice command.

Description

Related registrations

Pursuant to 35 USC § 119 (e), the present application claims priority from the provisional application filed on January 18, 2019 U.S. Patent Application No. 62 / 794,466 with the title "Voice Command Matching During Testing of Voice-Assisted Application Prototypes for Languages with Non-Phonetic Alphabets", the entire disclosure of which is hereby incorporated by reference.

background

Users interact with voice assist devices to accomplish an increasing number of different voice assisted tasks. In general, such voice assist devices are configured with one or more microphones that receive voice commands from a user. Examples of special voice support devices include Amazon® Echo and Google® Home, which enable interactions with the voice support platforms Amazon® Alexa and Google® Assistant, respectively. However, other devices, such as cell phones, desktop computers, laptop computers, game systems, and the like, may also be configured as voice support devices that can use the functionality of a voice support platform. For example, an AppleⓇ iPhone (cell phone) can serve as a voice support device that enables a user to interact with the SiriⓇ voice support platform.

Such voice assist devices are configured to respond to a user's voice commands in a variety of different ways, such as by outputting an acoustic (voice) response through one or more speakers, displaying visual content, providing other visual effects (e.g., information about light emitting diodes (LEDs)), providing tactile feedback and the like. The voice support platform may also cause other devices, such as a thermostat, speakers, lights, and the like, to respond to the user's voice command. Some examples of responses to household appliances include changing the temperature of a thermostat to adjust the temperature of a house, starting playing music from a music service through speakers, turning certain lights on or off in a house, and the like.

As voice interaction becomes increasingly prevalent, developers are beginning to add voice interaction functionality to digital interfaces, for example, by adding voice functionality to visual pages of an app or webpage user interface. However, conventional development apps fail to support the development and testing of voice functionality for app prototypes. It is difficult per se to develop app prototypes and to test them with voice support. In addition, in some cases, users want to interact with app prototypes using languages with non-phonetic alphabets such as Chinese or Japanese, resulting in a high rate of hit errors.

Summary

To solve these problems, matching voice commands during testing of voice-assisted app prototypes for languages with non-phonetic alphabets is described. A voice interaction tool for voice-assisted app prototypes is used in a digital media environment. A development app provides a development interface that enables a user to develop an app prototype for an app. In general, the app prototype contains a sequence of visual pages as well as various triggers and corresponding actions that are assigned to the visual pages. In order to enable the development of the app prototype, at least one visual page of the app prototype is displayed in the development interface. The development interface is controlled to provide an interaction interface that provides a trigger and an associated action for the at least one visual page of the app prototype. The trigger may correspond to one of a voice command, a user gesture, or a time delay, while the action may correspond to one of a spoken reply, a page transition to an additional visual page of the app prototype, or playing a media file. User input is received through the development interface to provide the trigger and action. In response to the user input, assigned interaction data are generated, which include the trigger, the action and the visual page of the app prototype. The associated interaction data is saved to enable testing of the trigger and action during a test phase of the app prototype.

At least one visual page of the app prototype is displayed in the test phase and a trigger is detected. The trigger can be a voice command received by a user, a user gesture or the expiration of a Time delay, to name just a few, correspond. The trigger is compared to the saved triggers associated with the app prototype to identify a suitable trigger. If the trigger corresponds to a voice command, the voice command is provided for a voice-to-text service which converts the voice command into a text string in real time. The text string of the voice command is then compared to stored text strings of voice commands associated with the app prototype to identify a suitable voice command.

In response to identifying the appropriate trigger, an action associated with the appropriate trigger is performed during the test phase. The action may be a spoken reply, a page transition to another visual page, or playing a media file. If the action corresponds to a spoken reply, a text string of the spoken reply can be provided for a text-to-voice service which converts the spoken reply into a sound file in real time. The spoken reply is then output by playing the sound file.

In one or more implementations, the development app uses a phonetic language translator to convert text strings of voice commands from languages with non-phonetic alphabets into phonetic versions of the text strings. In one example, the phonetic language translator Mandarin and a combination of katakana, hiragana and kanji came to translate into Pinyin and Romaji, respectively. In the test phase, a non-phonetic voice command that is spoken in a language with a non-phonetic alphabet is recorded by at least one microphone. The non-phonetic voice command is communicated in real time to a spoken-to-text service to convert the non-phonetic voice command into a non-phonetic text string of the voice command given in the language of the non-phonetic alphabet. The non-phonetic text string of the voice command is then communicated in real time to a phonetic language translator who translates the non-phonetic text string into a phonetic text string of the voice command given in a phonetic alphabet. The phonetic text string of the voice command is compared with phonetic text strings of stored voice commands given in the phonetic alphabet, which are associated with the app prototype, in order to identify a suitable voice command. Then an action that is assigned to the appropriate voice command is carried out.

This summary introduces a selection of concepts in a simplified form, which are further described below in the detailed description. As such, the present summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Figure list

• 1 ist eine Darstellung einer Umgebung bei einer exemplarischen Implementierung, die dafür betreibbar ist, ein Stimminteraktionswerkzeug für hier beschriebene stimmunterstützte App-Prototypen einzusetzen.
• 2 zeigt eine exemplarische Implementierung, bei der das Stimminteraktionswerkzeug von 1 Zuordnungen zwischen Interaktionsdaten und visuellen Pages eines App-Prototyps generiert und die Interaktionsdaten und visuellen Pages vorhält.
• 3 zeigt eine exemplarische Implementierung, bei der das Testmodul der Entwicklungs-App von 1 das Testen des App-Prototyps vereinfacht.
• 4 zeigt eine exemplarische Implementierung, bei der das Testmodul der Entwicklungs-App von 1 einen phonetischen Sprachübersetzer nutzt, um Stimmbefehle abzugleichen, die während des Testens des App-Prototyps aufgenommen werden.
• 5A bis 5C zeigen ein Beispiel für eine Entwicklungsschnittstelle einer Entwicklungs-App, die dafür benutzt wird, einen stimmunterstützten App-Prototyp zu entwickeln.
• 6A und 6B zeigen ein Beispiel für eine Entwicklungsschnittstelle einer Entwicklungs-App während einer Testphase eines stimmunterstützten App-Prototyps.
• 7 zeigt eine exemplarische Prozedur, bei der ein stimmunterstützter App-Prototyp entwickelt wird.
• 8 zeigt eine exemplarische Prozedur, bei der ein stimmunterstützter App-Prototyp getestet wird.
• 9 zeigt eine exemplarische Prozedur, bei der ein phonetischer Sprachübersetzer benutzt wird, um Stimmbefehle abzugleichen, die während des Testens des App-Prototyps aufgenommen worden sind.
• 10 zeigt ein exemplarisches System, das verschiedene Komponenten einer exemplarischen Vorrichtung beinhaltet, die als beliebiger Typ von Rechenvorrichtung, wie sie anhand 1 bis 9 beschrieben und/oder eingesetzt wird, implementiert sein kann, um Ausführungsformen der hier beschriebenen Techniken zu implementieren.

The detailed description is based on the accompanying figures.

• 1 is a representation of an environment in an exemplary implementation that is operable to use a voice interaction tool for voice-assisted app prototypes described here.
• 2nd shows an exemplary implementation in which the voice interaction tool of 1 Maps between interaction data and visual pages of an app prototype are generated and the interaction data and visual pages are kept.
• 3rd shows an exemplary implementation in which the test module of the development app from 1 testing the app prototype simplified.
• 4th shows an exemplary implementation in which the test module of the development app from 1 uses a phonetic language translator to match voice commands that are recorded while testing the app prototype.
• 5A to 5C show an example of a development interface of a development app that is used to develop a voice-assisted app prototype.
• 6A and 6B show an example of a development interface of a development app during a test phase of a voice-supported app prototype.
• 7 shows an exemplary procedure in which a voice-assisted app prototype is developed.
• 8th shows an exemplary procedure in which a voice-assisted app prototype is tested.
• 9 Figure 4 shows an exemplary procedure using a phonetic speech translator to match voice commands that were recorded during testing of the app prototype.
• 10th FIG. 12 shows an exemplary system that includes various components of an exemplary device that are considered any type of computing device as used 1 to 9 is described and / or used, can be implemented to implement embodiments of the techniques described herein.

Detailed description

Overview

Users interact with voice assist devices to accomplish an increasing number of different voice assisted tasks. In order to accomplish these tasks through voice-supported interaction, a functionality for this must be provided through the development, implementation and use of a software app, a firmware, a service, a plug-in, a script and the like. However, traditional tools for developing app prototypes limit the ability of users to develop and test voice-assisted functionality for app prototypes.

To solve these problems, a voice interaction tool for voice-assisted app prototypes is used in a digital media environment. A development app, such as AdobeⓇ XD, provides a development interface that enables a user (e.g. a developer) to develop an app prototype for an app. In general, the app prototype includes visual pages of the app as well as various triggers and corresponding actions associated with the visual pages. As part of the development of visual pages, the development app provides tools and functionalities that enable the development and layout of user interface controls or regulations, graphics and text for the app prototype. A development interface enables, for example, the development of a sequence of visual pages of the app prototype (for example a home screen page as well as various other visual pages) that can be navigated to while interacting with the app prototype.

In order to enable the development of a functioning app prototype, a voice interaction tool of the development app enables the direct integration of both voice commands and spoken feedback into the development of the app prototype. For this purpose, the voice interaction tool controls the development interface in such a way that an interaction interface is provided which contains a control system for providing a trigger and an associated action for a visual page of the app prototype. In the sense of the present, the term “trigger” denotes a command that can trigger a corresponding action, while an “action” is a process that is carried out in response to the detection of the corresponding trigger. Examples of triggers include voice commands, user gestures (such as typing, dragging, swiping, pressing buttons) and time delays, for example and not limitation. Examples of actions include, by way of example, and not by way of limitation, spoken responses, page transitions to other visual pages of the app prototype, and playing media files (such as a sound or video file).

The interaction interface enables the user to select the trigger and the associated action from a list of defined triggers and actions. As soon as the selection has been made, the user provides an input which describes the selected trigger and the action. For example, to associate a voice command with a particular visual page of the app prototype, the voice interaction tool can allow the user to select the particular visual page (e.g., using a touch functionality) and to use the interaction interface ( for example by tapping the commands or replies, speaking the commands or replies into a microphone or a device coupled to communication technology and the like). The voice commands can be configured as sound data or text data that indicate what the user speaks for a voice-supported app. The spoken reply represents the acoustic output of the voice-assisted app (for example by a voice supporter or assistant) in response to the receipt of the voice command.

In particular, the interaction interface enables the user to assign various other types of triggers and actions to visual pages of the app prototype. For example, the user can assign voice command triggers to corresponding spoken responses. The development app, however, enables the user to mix and match voice interactions with other triggers and actions, for example by assigning a user gesture trigger to a spoken reply or by assigning a voice command to a transition to another visual page of the app prototype. In response to a voice command, such as "What is the weather like?", The app For example, go to a visual page that presents a summary of future weather.

The development app then assigns the interaction data of the trigger and the action to the corresponding visual page. This procedure enables the interaction data associated with the respective visual pages to be called up on the basis of an identification of the trigger during a test phase of the app prototype. In some cases, the interaction data and associated visual pages can be stored locally on the client device. For example, the voice interaction tool can generate a file for the app prototype that contains the interaction data and the associated visual pages and cause the file to be stored in a memory of the client device. In this way, the file can be easily transferred to other developers who continue to develop. Additionally or alternatively, the interaction data and the assigned visual pages can be stored in a remote memory.

The development app is also implemented with a test module that enables the testing of a voice-supported app prototype that is created using the development app. If the test phase is initiated, the test module displays a first visual page (e.g. the homepage) of the app prototype. If the user interacts with the app prototype, for example by saying a voice command or tapping the visual page, the test module causes the relevant actions in real time, which are assigned to the triggers, for example by giving a spoken response to another visual page passes, media content is initiated, or a combination of these. In this way, the described system enables the user to interact with the app prototype during the test phase to ensure that the app prototype works as intended. This approach enables the developer to identify and address any problems in developing voice-assisted interactions before the implementation phase. After the app prototype has been developed and tested, the development app can generate a development specification that contains instructions and data that an app developer can use to implement the app according to the development of the app prototype. Alternatively, the development app can be implemented to generate an executable app to complete the development and testing process.

The techniques described thus simplify the development of voice-assisted app prototypes by allowing problem-free direct integration of both voice commands and spoken feedback into app prototype developments, while the test phase enables the user to provide voice commands and output spoken feedback, which in turn enables the user to simulate the interaction with the app prototype.

In particular, when testing voice commands, the test module of the development app can select a suitable voice command based on the number of letters that the text string of the voice command has in common with the text strings of the stored voice commands. This approach works for languages with phonetic alphabets such as English, since matching the common characters comes close to saying the words. For example, the words “house” and “mouse” each have four common letters of five in English and therefore sound similar. In a common scenario where the voice-to-text service incorrectly converts a voice command into a text string that is similar to the intended voice command (for example, "house" instead of "mouse" or "red chair" ( red chair) instead of “ride share”), the matching techniques described are able to identify the appropriate stored voice command with a high degree of probability, which is due to the letter commonality between words that sound the same in phonetic alphabets. However, this matching technique does not work for languages with "non-phonetic alphabets" such as Mandarin Chinese and Japanese with a view to the Kanji alphabet, since words with the same sound cannot easily be compared letter by letter.

To solve this problem, the development app uses a phonetic language translator that converts text strings from voice commands in languages with non-phonetic alphabets into phonetic versions of the text strings. In one example, the phonetic language translator can translate Mandarin and a mixture of Hiragana, Katakana and Kanji into Pinyin and Romaji, respectively. In the test phase, a microphone records the sound of a voice command in a non-phonetic language and provides the sound of the voice command in the non-phonetic language for a spoken-to-text service. The spoken-to-text service converts the tone of the voice command into a text string of the voice command given in the non-phonetic alphabet. The text string of the voice command in the non-phonetic alphabet is then provided for the phonetic language translator who converts the text string of the voice command in the non-phonetic alphabet into a phonetic version of the text string of the voice command. Simultaneously with that During testing or before that, the phonetic translator translates text strings of stored voice commands, which are assigned to the app prototype, into non-phonetic alphabets into phonetic versions of the text strings of the stored voice commands.

The translated phonetic version of the text string of the recorded voice command is then compared with the translated phonetic versions of the text strings of the stored voice commands. A suitable voice command is identified based on a determination of the number of common characters between the phonetic version of the text string of the recorded voice command and the phonetic versions of the text strings of the stored voice commands. By first converting the non-phonetic text strings into phonetic versions, this comparison based on the number of common characters is effective in identifying the appropriate voice command with a reasonable degree of reliability. Therefore, the techniques of phonetic speech translation described here increase the accuracy of matching voice commands while testing voice-assisted app prototypes.

As used herein, the term "non-phonetic text string" refers to a text string written in a non-phonetic alphabet such as Mandarin Chinese or Kanji, while a non-phonetic voice command refers to sound data that is recorded by a microphone when a user sends a voice command speaks in a language with a non-phonetic alphabet. In contrast, a “phonetic text string” or a “translated phonetic text string” refers to a text string in a non-phonetic alphabet that has been translated into a phonetic representation, such as Pinyin or Romaji.

The following discussion first describes an exemplary environment in which the techniques described here can be used. Exemplary details and procedures of implementation that can be performed in the exemplary environment as well as in other environments are then described. Therefore, the performance of the exemplary procedures is not limited to the exemplary environment, and the exemplary environment is not limited to the performance of the exemplary procedures.

Exemplary environment

1 is a representation of an environment 100 in an exemplary implementation that is operable to match voice commands while testing voice-assisted app prototypes for languages with non-phonetic alphabets, as described here. The environment shown 100 includes a computing device 102 , a voice interaction development system 104 and a voice support platform 106 as well as various voice support devices 108 that over a network 110 are coupled with each other in terms of communication technology.

Devices that can be used for this purpose, the computing device 102 , the voice interaction development system 104 and the voice support platform 106 implement can be configured in a variety of ways. The devices can be configured, for example, as a desktop computer, laptop computer, mobile device (for example, assuming a manual configuration such as with a tablet or cell phone) and the like. The devices can range from full resource devices with significant memory and processor resources (e.g. PCs, game consoles) to low resource devices with limited memory and / or processing resources (e.g. mobile devices). In addition, one device can represent multiple different devices, such as multiple servers used by a company to perform operations "through the cloud", as shown below 8th is still described.

The voice support devices 108 can be configured in a variety of ways. General are the voice support devices 108 configured with one or more microphones, the voice commands 112 from a user 114 receive. Examples of special voice support devices 108 include AmazonⓇ Echo and Google® Home, which enable interactions with the voice support platforms Amazon® Alexa and Google® Assistant. Other devices, such as cell phones, desktop computers, laptop computers, game systems, and the like, can also be used as voice assist devices 108 configured to be able to use the functionality of a voice support platform. For example, an AppleⓇ iPhone (cell phone) can be used as a voice support device 108 serve the user 114 allows you to interact with the SiriⓇ voice support platform. For this purpose, the voice support devices 108 configured to do so in addition to receiving voice commands 112 receive user input through interfaces (e.g., touch, camera, and the like).

In any case, the voice support devices interact 108 with their respective voice support platform 106 to click on the voice commands 112 to reply. The voice support platform 106 is shown as being a voice command processing module 116 having. The voice command processing module 116 represents functionality that is one of the voice command 112 receives initiated request, determines how to respond to the request, and then receives a response for the voice support device 108 or any other device by the voice command 112 may be affected, for example for connected devices within a house or household, for service provider devices for providing one or more respective services (for example to simplify the delivery of goods through an e-commerce platform) and the like. Accordingly, the voice support devices 108 configured to respond to user voice commands 114 to respond in a variety of ways, such as using an acoustic spoken reply 115 output via one or more loudspeakers, display visual content, provide other visual effects (for example information by means of light-emitting diodes (LEDs)), provide tactile feedback and the like. In addition, the voice support platform 106 cause other devices, such as a thermostat, speakers, lights, and the like, to respond to the voice command 112 of the user 114 to reply. A few examples of responses to household appliances include changing the temperature of a thermostat to adjust the temperature of a house, starting playing music from a music service through speakers, turning certain lights on or off in a house, and the like. The voice commands 112 can be used with a variety of settings (e.g., household, work, and the like) to respond to one or more devices, including but not limited to the voice support device 108 , to initiate.

In connection with the presented environment 100 provides a development app 118 functionality available to a user of the computing device 102 (e.g. a developer) enables an app prototype 120 for an app, such as a software app, firmware, services, plug-ins, scripts and the like, to develop. In some cases, the app prototypes 120 for a voice support platform 106 , for example AmazonⓇ Show, which includes an integrated display screen. The app prototypes 120 however, can be designed for any type of app that includes voice-supported functionality. The development app 118 can be implemented as a development app, such as AdobeⓇ XD or AdobeⓇ Sketch. The development app 118 may be implemented as a variety of different third party content creation or editing apps, voice app development apps, development environments, and the like.

The development app 118 provides various interfaces and controls that allow the user a sequence of visual pages 122 of the app prototype 120 to develop. As part of the development of the visual pages 122 provides the development app 118 Tools ready, the various user interface controls or regulations (for example selectable buttons and dropdown lists), graphics, text and the like for the app prototype 120 develop and lay out. For example, the user can have a homepage for an app as well as other visual pages 122 that an end user can navigate to.

According to the techniques described, the development app includes 118 also a voice interaction tool 126 which is the mapping of interaction data 124 in the form of voice commands and spoken responses to various visual pages 122 of the app prototype 120 simplified. For this purpose, the voice interaction tool controls 126 a development interface of the development app 118 to provide an interaction interface that has a trigger and associated action for a visual page 122 of the app prototype 120 receives. In the sense of the present case, triggers contain various user commands which can trigger a corresponding action. Examples of triggers include voice commands, user gestures (e.g. tapping, dragging, swiping, key presses) and time delays, to name just a few. Examples of actions include spoken responses, transitions to other visual pages of the app prototype, and playing media files (such as a video file) to name a few.

In some cases, the interaction interface allows the user to select the trigger and an associated action from a list of defined triggers and actions. As soon as the selection has been made, the user provides an input which describes the selected trigger and the action. For example, to order a voice command from a specific visual page 122 of the app prototype 120 can assign the voice interaction tool 126 enable the user to view the particular visual page 122 (e.g. using a touch functionality) and select different voice commands or spoken ones Specify responses that correspond to the particular visual page via the interaction interface (for example, by typing the commands or responses, speaking the commands or responses into a microphone or a communication device, and the like). In the sense of the present, contain the interaction data 124 Voice commands and spoken responses, as well as other types of triggers (e.g. user gestures and time delays) and actions (e.g. page transitions and media playback).

In particular, the development app enables 118 the user, various other types of triggers and actions the visual pages 122 of the app prototype 120 assign. For example, the user can assign voice command triggers to corresponding spoken responses. The development app 118 enables the user, however, to mix and match voice interactions with other triggers and actions, for example by assigning a user gesture trigger to a spoken response or by a voice command moving to another visual page 122 of the app prototype 120 is assigned.

As part of simplifying the development of voice-based interactions for the app prototype 120 can use the voice interaction tool 126 the functionality of the voice interaction development system 104 deploy. In the environment shown 100 is the voice interaction development system 104 with a voice-to-text service 128 and a text-to-speech service 130 shown. Generally speaking, the spoken-to-text service 128 provides functionality to identify spoken words and phrases in the voice data picked up by the one or more microphones and convert the spoken words and phrases into a text string. The text-to-speech service 130 represents a functionality for processing text data (for example text data that is received via a user input for a user interface of the development app) for a spoken reply. The spoken-to-text service 128 and the text-to-speech service 130 therefore allow the user both acoustic and text input for the development app 118 while developing and testing the app prototype 120 to provide. Although the voice interaction development system 104 depicted with these particular components, it should be appreciated that it may include, or have access to, more, fewer, and other components that provide the functionality described herein without departing from the spirit or scope of what is described herein. In some cases, the spoken-to-text service 128 or the text-to-speech service 130 additionally on the computing device 102 or implemented on separate third party servers. Are the spoken-to-text service 128 and the text-to-speech service 130 implemented as third party services, the voice interaction tool 126 Use authorization tokens to provide secure communication channels to the spoken-to-text service 128 and the text-to-speech service 130 build up.

The development app 118 is also with a test module 132 implemented, which enables testing of the voice-assisted app prototype 120 that using the development app 118 is created, made possible. For this purpose the test module shows 132 a first visual page (e.g. the homepage) of the app prototype 120 in a test phase. A user interacts with the app prototype 120 , for example by saying a voice command or the visual page 122 taps, the test module initiates 132 in real time that the appropriate actions associated with the triggers are performed, for example by spoken response, transitioning to another visual page, initiating media play, or a combination thereof. In this way, the system described enables the user with the app prototype 120 interact during the testing phase to ensure that the app prototype is working properly. This approach enables the user to identify and address problems in developing voice-assisted interactions before the development phase.

The voice interaction development system 104 is also with a phonetic language translator 134 implemented on the computing device 102 or remote from the computing device 102 (for example as a cloud service) can be implemented. In general, the phonetic voice or language translator 134 configured to get text strings in a non-phonetic alphabet and convert the non-phonetic text string to a phonetic text string in another language or font. For example, the phonetic language translator can use romanization techniques to convert text strings in non-phonetic alphabets to Roman / Latin script. To this end, the phonetic speech translator can use transliteration techniques to represent written text and transcription techniques to represent spoken words. Transcription methods can be divided into the phonemic transcription, which records the phonemes or units with semantic meaning in the spoken, and the stricter phonetic transcription that accurately records spoken sounds.

The phonetic language converter or language translator 134 can be used to convert text strings in a variety of different types from non-phonetic languages to phonetic versions of the text strings. In one example, the phonetic language translator can be configured to translate Mandarin Chinese, which is often written using Chinese characters, into the Romanized version of Chinese known as "Pinyin". In a further example, the phonetic language translator can be configured to translate Japanese, which is usually written in a combination of logographic characters (Kanji) originally derived from Chinese and syllable-based fonts (Kana) originally derived from Chinese characters, into Romaji. For Japanese, there are several different types of romanization systems that include Hepburn Romanization (most commonly used), Kunrei-shiki Romanization, and Nihon-shiki Romanization. In particular, the voice interaction development system 104 use a variety of different types of romanization techniques to convert text received in a variety of different languages into a standard phonetic version of the text.

After the app prototype 120 has been developed and tested, the development app 118 Generate a development specification that is provided for an app developer so that the app can be implemented. Alternatively, the development app 118 implemented to generate an executable app to complete the development and test process. In this scenario, the app prototype can 120 automatically into an executable app that displays the visual pages 122 and the interaction data 124 of the app prototype 120 includes, be compiled (or provided for a third-party implementation tool that generates the executable app). In this way, the development app enables 118 Users, fully functional voice-supported apps using the development app 118 even in cases where the user has very little knowledge of how voice-supported apps are implemented. By enabling voice interactions for app prototypes, the techniques described enable developers to develop voice-assisted apps that include visual interfaces.

After considering an exemplary environment, a discussion of some exemplary details of the techniques for a voice interaction tool for voice-assisted app prototypes in a digital media environment follows in accordance with one or more implementations.

2nd shows an exemplary implementation 200 where the voice interaction tool from 1 Mappings between interaction data and visual pages of an app prototype are generated and the interaction data and the visual pages are kept. The illustrated example 200 includes the voice interaction tool 126 the development app 118 out 1 .

In example 200 is the voice interaction tool 126 presented such that there is user input 202 receives. The user input 202 can from the voice interaction tool 126 via a user interface of the development app 118 , for example a user interface that is similar to the one based on 4th and 5 is to be received. In some cases, user input 202 correspond to a voice input that is recorded by one or more microphones. Generally, the user input identifies a trigger and a corresponding action that a visual page 122 of the app prototype 120 assigned. As described here, the triggers contain various user commands that can trigger a corresponding action.

Based on user input 202 generates the voice interaction tool 126 associated interaction data 204 that the mapping between the interaction data 124 (e.g. triggers and actions) and respective visual pages 122 describe. This associated interaction data 204 can from the development app 118 are provided in such a way that those of the respective visual page 122 assigned interaction data based on a detection of the trigger from a memory 206 can be accessed.

The memory 206 can be local to the computing device 102 be implemented. For example, the voice interaction tool can be a file for the app prototype 120 generate that the interaction data 124 and the associated visual pages 122 and cause the file to be in memory 206 the computing device 102 is saved. This procedure enables the interaction data 124 that correspond to the respective visual pages 122 are assigned, based on an identification of the trigger during a test phase of the app prototype 120 be retrieved from memory. Alternatively or additionally, the interaction data 124 and the associated visual pages 122 in a remote memory of the voice interaction development system 104 be held up. In one or more implementations For example, the files that are stored locally on the client device can be synchronized with the associated interaction data that is stored remotely in the memory of the voice interaction development system.

The interaction data 124 include triggers 208 and corresponding actions 210 . In the present example, the triggers include 208 Voice commands 212 , User gestures 214 (such as tapping, dragging, swiping, or pressing a key) and time delays 216 while the actions are spoken responses 218 , Page transitions 220 and a media issue 222 include. In some cases, the development app provides 118 an interaction interface that enables the user to select the trigger and the associated action from a list of defined triggers and actions. As soon as the selection has been made, the user provides an input which describes the selected trigger and the action. For example, to order a voice command from a specific visual page 122 of the app prototype 120 can assign the voice interaction tool 126 enable the user to view the particular visual page 122 (e.g. using a touch functionality) and select different voice commands 212 or spoken responses 218 that correspond to the particular element, via the interaction interface (for example, by typing the commands or responses, speaking the commands or responses into a microphone or a communication device, and the like). The voice commands can be configured as sound data or text data indicating what a user speaks for a voice support device. The spoken reply represents the acoustic output in response to the receipt of the voice command, for example of the acoustically spoken voice. It becomes a voice command 212 received as voice input from the user via a microphone, so the spoken-to-text service 128 can be used to convert the recorded sound into a text string. In one or more implementations, the voice interaction tool can 126 several variations of the voice command 212 to generate. For example, the voice command 212 "How will the weather be received tomorrow," is how the voice interaction tool can 126 Generate several variations of this voice command, such as "How will the weather be tomorrow", "Will it rain tomorrow", "Will the sun shine tomorrow", "How hot will it be tomorrow" and the like.

In one or more implementations, the voice interaction tool can 126 Voice commands 212 received as text input in a non-phonetic alphabet, such as Mandarin Chinese or Kanji. In this scenario, the voice interaction tool 126 the voice commands 212 as non-phonetic text strings in memory 206 save until the test phase of the app prototype. Alternatively, the non-phonetic text strings of the voice commands 206 in some cases by the phonetic translator 134 before saving the text string in memory 206 can be converted into phonetic versions of the text string. In this scenario, the phonetic versions of the text strings are in memory 206 saved.

The techniques described also allow the user to prototype the app 120 interacts during a test phase. In this context, consider 3rd where an exemplary implementation 300 is shown, in which the test module of the development app from 1 testing the app prototype simplified. The illustrated example 300 contains the test module 132 , the spoken-to-text service 128 and the text-to-speech service 130 out 1 .

The test module initiates in the test phase 132 the display of a first visual page 122 (e.g. a homepage) of the app prototype 120 . While displaying the first visual page 122 a user interacts with the app prototype 120 , for example by saying a voice command or tapping the visual page 122 , and the test module 132 receives a trigger based on real-time user interaction 302 . The test module 132 is shown such that it is a processing module 304 that represents functionality that processes user interactions around the trigger 302 , such as a voice command 306 , a user gesture 308 or a time delay 310 , to identify.

The processing module 304 compares the identified trigger 302 with the assigned interaction data 204 to identify a suitable trigger. The request processing module 304 compares the voice command 306 or the user gesture 308 for example with the saved voice commands 212 or the user gestures 214 the associated interaction data 204 . The processing module can be used to identify the suitable trigger 304 use a variety of techniques, such as unique identifiers, hashes, text-based searches, and the like. In addition, the processing module 304 continuously monitor time delay triggers when a new visual page is displayed.

Becomes the trigger 302 as a voice command 306 received, so the processing module 304 the sound data of the voice command 306 for the Spoken-to-text service 128 ready. The spoken-to-text service 128 processes the sound data in real time to the sound data of the voice command 306 convert to a text string. The processing module 304 receives the text string of the voice command 306 from the spoken-to-text service 128 and compares the text string of the voice command 306 with the stored text strings of the voice commands 212 to identify a suitable voice command.

The processing module 304 can identify a suitable voice command by typing the letters of the text string of the received voice command 306 with letters of the text strings of the saved voice commands 212 compares. A higher number of matching letters indicates a higher probability of a hit. For example, if the command contains the word "mouse", the processing module can 304 identify the word "house" as matching "mouse" because the words have four matching characters. In some cases, the spoken-to-text service includes 128 a reliability score with the text string that indicates a degree of reliability that the text string is correct. The processing module supports the reliability parameter 304 when identifying a suitable voice command. In one or more implementations, the processing module is 304 Configured to select the appropriate low tolerance text string to allow the user to more easily interact with the app prototype.

In response to a detection of that the trigger 302 to the saved trigger 208 the associated interaction data 204 matches, the processing module identifies 304 the corresponding action 210 that the trigger in the associated interaction data 204 assigned. An action 312 having a spoken reply 314 , a page transition 316 or a media issue 318 corresponds to an implementation module 320 forwarded. The implementation module 320 causes the appropriate action 312 associated with the identified trigger is performed, for example, by outputting the spoken response 314 , Make a page transition 316 to another visual page, output of sound or video content or a combination of these.

Corresponds to the action 312 a spoken reply 314 , the implementation module can use the text string of the spoken reply 314 in real time for the text-to-speech service 130 provide. The text-to-speech service 130 converts the text string of the spoken reply 314 in sound data, that of the spoken reply 314 correspond to. The implementation module 320 then causes the output of the sound data, that of the spoken reply 314 correspond.

In this way, the system described enables the user to test the app prototype during the test phase 120 to interact to make sure the app prototype 120 works on purpose. This approach enables the user to have problems developing the voice-assisted app prototype 120 to identify and address before the implementation phase.

In particular, the test module 132 when matching voice commands, select a suitable voice command based on the number of letters that the text string of the voice command has in common with the text strings of the stored voice commands. This approach works for languages with phonetic alphabets such as English, since matching the common characters comes close to saying the words. In languages with "non-phonetic" alphabets such as Mandarin Chinese and Japanese with a view to the Kanji alphabet, this matching technique no longer works because words that sound the same cannot simply be compared letter by letter.

In this context, consider 4th which is an exemplary implementation 400 where the test module of the development app from 1 uses a phonetic language translator to match voice commands that are recorded while testing the app prototype. The illustrated example 400 contains the test module 132 , the spoken-to-text service 128 and the phonetic language translator 134 out 1 .

The test module initiates in the test phase 132 the display of a first visual page 122 (e.g. a homepage) of the app prototype 120 . While displaying the first visual page 122 a user interacts with the app prototype 120 by speaking a non-phonetic voice command 402 . As described here, the nonphonetic voice command corresponds 402 a voice command spoken in a language with a non-phonetic alphabet, such as Mandarin Chinese or Japanese with a view of the Kanji alphabet. The nonphonetic voice command 402 is recorded by at least one microphone as sound data.

The test module 132 communicates the sound data of the recorded non-phonetic voice command 402 to the spoken-to-text service 128 . The spoken-to-text service 128 processes the sound data of the non-phonetic in real time Voice command 402 to the sound data of the non-phonetic voice command 402 in a non-phonetic text string given in the language of the non-phonetic alphabet 404 convert the recorded voice command.

The test module 132 communicates the non-phonetic text string 404 to the phonetic translator 134 . The phonetic language translator 134 translates the non-phonetic text string 404 the recorded voice command in a phonetic text string 406 the recorded voice command, which is written in a phonetic alphabet such as Pinyin or Romaji. In addition, the phonetic language translator receives 134 non-phonetic text strings given in the non-phonetic alphabet during the test phase 410 the stored voice commands 408 . The saved voice commands 408 can, for example, the voice commands 212 that are in the store 206 be held up to correspond. The phonetic language translator 134 translates the non-phonetic text strings 410 the stored voice commands 408 in phonetic text strings given in the phonetic alphabet 412 the stored voice commands 408 . Alternatively, the phonetic language translator 134 the phonetic text strings 410 the stored voice commands 408 before the test phase in phonetic text strings 412 the stored voice commands 408 translate. In this case, the phonetic text strings 412 are held in the memory.

A phonetic comparison module 414 of the test module 132 receives the phonetic text string 406 of the recorded voice command 402 and the phonetic text strings 412 the stored voice commands 408 . The phonetic comparison module 414 compares the phonetic text string 406 then with the phonetic text strings 412 to a matching voice command 416 the phonetic text strings 412 to identify according to the stored voice commands. The phonetic comparison module 414 can the matching voice command 416 identify it by the letters of the phonetic text string 406 with those of the phonetic text strings 412 the stored voice commands 408 compares. Because the phonetic text strings 406 and 412 The phonetic comparison module can be translated into a phonetic script 414 the appropriate voice command 416 based on the number of letters that match the voice command 416 in common with the recorded voice command. A higher number of matching letters indicates a higher probability of a hit, since both text strings are written in a phonetic font.

As soon as the appropriate voice command 416 from the phonetic comparison module 414 has been identified, the test module initiates the execution of the assigned action, as described here.

5A to 5C FIG. 5 shows an example 500 of a development user interface of a development app that is used to develop a voice-assisted app prototype.

In 5A becomes a development interface 502 the development app 118 by means of a display device 504 displayed. The development interface 502 is shown in such a way that it provides a first visual page 506 and a second visual page 508 includes. The first visual page 506 and the second visual page 508 can by a user who uses the development app 118 uses, developed or imported from another development app. In particular, the development interface 502 used by the user to generate any number of different visual pages.

The illustrated example 500 also includes a cursor 510 , which represents a functionality that enables a user, an input for interaction with various controls or regulations and functionalities of the development interface 502 by the development app 118 provided. Although the cursor 510 is shown, no cursor may be displayed in one or more implementations. In some cases, the various controls or regulations and functionalities of the development interface 502 selected or used for interaction in other ways, such as via touch (or other gesture), keyboard, pen, voice, and the like.

The visual pages 506 and 508 provide visual pages or screens of the app prototype 120 , which developers often refer to as "art boards". In the present example, the first visual page corresponds 506 a "homepage" of the app prototype 120 and includes a text that indicates a current time, a current temperature and a graphic that describes the current weather as "sunny".

The second visual page 508 corresponds to a calendar page of the app prototype in this example 120 and includes a text that indicates that the user has three upcoming events and that the user has an appointment corresponding to "Brunch with Judy in the Cafe Presse tomorrow at 10am". In particular, the first page contains 506 a text that indicates that the user can say “what's on my calendar”. This text provides a visual indication for the User ready to go to a calendar page by saying the voice command.

In order to enable the transition to the calendar page based on this voice command, the development interface enables the user to create the first visual page 506 with the second visual page 508 to link. In this example, the user links the visual pages 506 and 508 by using a handle control on the first visual page 506 using the cursor 510 selects and the cursor to the second visual page 508 using dragging. This approach causes the development interface to be a visual linking element 512 between the first visual page 506 and the second visual page 508 generated.

In response to linking the first visual page 506 with the second visual page 508 the development interface enables the user to provide input that identifies a trigger for this transition, as in 5B is shown. In 5B the development app presents an interaction interface 514 , which contains controls for the provision of a trigger and an associated action for a visual page of the app prototype. In this example, the interaction interface includes 514 a trigger control or regulation 516 that receives an input that defines a type of trigger. In some cases, trigger control or regulation is possible 516 the interaction interface 514 the user to select the type of trigger from a list of defined triggers, for example by using the down arrow next to the trigger control 516 selects to present a list of triggers. As described here, the list of predefined triggers that can be used in the interaction interface 514 appears, include a voice command, a user gesture, or a time delay, to name just a few. In this example, the user provided input to select the voice command as the trigger.

After the voice command has been selected as the trigger, the interaction interface can 514 be modified to a trigger content control or regulation 518 to provide. In 5B shows the interaction interface 514 for example a trigger content control or regulation 518 as a command prompt configured to receive an input that defines the words of the voice command. If the trigger is alternatively selected as a user gesture or a time delay, the trigger content control or regulation 518 modified to receive an input that describes the type of user gesture or the duration of the time delay.

In this case, the user has provided an input that defines the voice command as "what's on my calendar". This user input can be used as text input for trigger content control or regulation 518 be received. In other words, the user can directly insert the words of the voice command into the trigger content control 518 the interaction interface 514 type in. Alternatively, the user can speak the words of the voice command into a microphone or a device coupled in terms of communication technology, which causes the spoken words to be converted into text input, as described here. If the voice command is spoken by the user, the interaction interface can 514 the converted text in the interaction interface 514 Show to enable the user to confirm that the text is correctly recognized and to make changes if necessary.

The interaction interface 514 is further shown in such a way that it controls action or regulation 520 and a content control intended for an assigned action 522 includes. The action control or regulation 520 is configured to receive an input that defines the type of action to be taken in response to the detection of the corresponding trigger. In some cases, action control or regulation enables 520 the interaction interface 514 the user to select the type of action from a list of defined actions, for example by clicking the down arrow next to the action control 520 to cause a list of actions to be presented. As described here, the actions may include a spoken reply, a page transition, or an output content or content, to name just a few. In this example, the user provided input to select a page transition as an action for the corresponding voice command trigger. Alternatively, the page transition action can be automatically selected for the user based on the input to the first visual page 506 the second visual page 508 assign.

After the page transition has been selected as an action, the interaction interface can 514 be modified to control action content or regulation 522 to provide. In 5B shows the interaction interface 514 for example action content control or regulation 522 as a command prompt (command prompt) that is configured to receive input that defines the destination of the page transition. If the action is alternatively selected as a spoken reply or content output, the action content control or regulation can 522 are modified to receive an input that describes the words of the spoken reply or defines the content for output (for example a file name of a video file).

In this case, the user has provided an input that defines the page transition for the transition to the calendar page of the app prototype. Again, in some cases the location of the page transition can be automatically selected for the user based on the input to link the first visual page to the second visual page. In this example, the development interface enables the user to further define characteristics of the page transition, such as easing, a transition effect or a duration of the page transition.

In particular, according to 5B the trigger and the associated action that is entered into the interaction interface 514 have been defined, the first visual page 506 and assigned to the second visual page. The development interface can also enable the user to define a trigger and a corresponding action that is associated with a single visual page in such a way that an action in response to the trigger is carried out without a transition to another visual page.

In this context, consider 5C . In 5C the development app presents an interaction interface 514 , which contains controls for the provision of a trigger and an associated action for a visual page of the app prototype. In this example, the interaction interface 514 dynamic in response to user input of the second visual page 508 presented according to the calendar page. In this case, instead of selecting a voice command, the user selects a time delay trigger via the trigger control or regulation 516 and then issues the trigger content control or regulation 518 an input ready to define the duration of the time delay as 0 s. The user also has an input for selecting a spoken response via the action control or regulation 520 provided and then provides the action content control or regulation 522 an input ready that defines the words of the act of the spoken reply. The input that defines the spoken reply can be used as text input for the action content control or regulation 522 be received. In other words, the user can directly insert the words of the spoken reply into the action content control or regulation 522 the interaction interface 514 type in. Alternatively, the user can speak the words of the spoken reply into a microphone or a device coupled to communication technology, which causes the spoken words to be converted into text input, as described here. If the spoken reply is spoken by the user, the interaction interface can 514 the converted text in the action content control or regulation 522 the interaction interface 514 Show to enable the user to confirm that the text is correctly recognized and to make changes if necessary.

In particular, the user interface enables 514 the user to assign various other types of triggers and actions to visual pages of the app prototype. For example, the developer can associate voice command triggers with corresponding spoken responses. However, the developer can also mix and match voice interactions with other triggers and actions, for example, by associating a user gesture trigger with a spoken reply or by associating a voice command with a transition to another visual page of the app prototype.

6A and 6B FIG. 5 shows an example 600 for a development user interface of a development app during a test phase of a voice-assisted app prototype. Once the user sees the visual pages and user interactions of the app prototype, as in 5A-4C a test phase can be initiated by selecting a test control or regulation of the development interface. In the example 600, the test module shows 132 the development app 118 a first visual page 602 of the app prototype on the display device 504 . In particular, the first visual page corresponds 602 the first visual page 506 , as in 5A-4C is shown. In the test phase, the test module controls 132 the developed app prototype 120 such that it functions like an executable app instead of displaying a layout of visual pages and associated triggers and actions.

If the user interacts with the app prototype during the test phase, for example by saying a voice command or tapping the visual page, the test module initiates it 132 in real time that the corresponding actions associated with the trigger are carried out, for example by outputting acoustic content, outputting visual content, skipping to another visual page or a combination of these. In 6A for example, the user speaks a voice command 604 that contains the words "What's on my calendar?" A microphone detects the spoken voice command and, and the spoken-to-text service 128 then converts the audio data into a text string that contains the words "What's on my calendar". This text string is then compared with the associated interaction data that were provided by the user during the development of the app prototype.

In this case the test module recognizes 132 the text string of the voice command 504 as suitable for the voice command that is used for trigger content control or regulation 518 the interaction interface 514 in 5B provided. Based on the detection of a hit, the test module initiates 132 the implementation of the corresponding action, previously carried out by the user in the controls 520 and 522 the interaction interface 514 from 5B has been entered. In this case, the corresponding action is a page transition to the calendar page (e.g. visual page 508 from 5 ) detected.

6B shows the display of a second visual page 606 according to the calendar page after the test module 132 based on the action associated with the detected trigger from the first visual page 602 to the second visual page 608 has passed. If the calendar page contains a time delay trigger of 0 s with a corresponding spoken reply, the test module initiates 132 (App) additionally that a loudspeaker or an associated device speaks the reply 608 outputs acoustically, which in this example corresponds to the words "You have a brunch with Judy in the Cafe Presse tomorrow at 10 am". For this purpose, a text string containing the words of the spoken reply is sent to the text-to-speech service 130 forwarded. The text-to-speech service 130 then converts the text string of the spoken reply into a sound file which is then output in real time on the computing device. In this way, the described system enables the user to interact with the app prototype during the test phase to ensure that the app prototype works as intended.

After discussing exemplary details of the techniques for a voice interaction tool, some exemplary procedures to illustrate additional aspects of the techniques are now considered.

Exemplary procedures

This section describes exemplary procedures for developing and testing voice-assisted app prototypes in one or more implementations. Aspects of the procedures can be implemented in hardware, firmware or software, or in a combination thereof. The procedures are shown as a set of blocks that specify operations performed by one or more devices and that are not necessarily limited to the orders shown for performing the operations by the respective blocks.

7 shows an exemplary procedure 700 , in which a voice-assisted app prototype is developed.

At least one visual page of an app prototype is displayed in a development interface on a display of a computing device (block 702 ). For example, the development app shows 118 the visual page 122 of the app prototype 120 in the development interface 502 on the display of the computing device 102 on.

The development interface is controlled or regulated in such a way that it provides an interaction interface for receiving a trigger and an associated action for the at least one visual page of the app prototype (block 704 ). According to the principles explained here, the trigger corresponds to one of a voice command, a user gesture or a time delay, while the action corresponds to one of a spoken reply, a transition to an additional visual page of the app prototype or a playback of a media file. For example, the voice interaction tool controls 126 the development interface 502 to the interaction interface 514 to provide the triggers 208 and the associated action 210 for at least one visual page 122 of the app prototype 120 to recieve.

User input is received over the interaction interface to provide the trigger and action (block 706 ). For example, the voice interaction tool receives 126 the user input 202 via the development interface 502 to the trigger 208 and the plot 210 to provide.

Assigned interaction data are generated, which contain the trigger, the action and the visual page of the app prototype, and the assigned interaction data are stored in order to enable testing of the app prototype during a test phase (block 708 ). For example, the voice interaction tool saves 126 the trigger 208 and the plot 210 with an indication of the visual page 122 as associated interaction data 204 in a store 206 , so that the detection of the trigger during a test phase causes the action to be performed.

8th shows an exemplary procedure 800 , in which a voice-assisted app prototype is tested.

A test phase for an app prototype is initiated (block 802 ), and at least one visual page of the app prototype is displayed in the test phase (block 804 ). As an example, the test module initiates 132 the test phase for the app prototype 120 and shows at least one visual page during the test phase 122 for the app prototype 120 on.

A trigger is detected based on user input that is received while the at least one visual page is displayed (block 806 ). The trigger can correspond to a voice command, a user gesture, a time delay, to name just a few. For example, a user interacts while displaying the first visual page 122 in real time with the app prototype 120 , for example by saying a voice command or tapping the visual page 122 , and it receives the test module 132 the trigger based on user interaction 302 . The processing module 304 of the test module 132 processes the user input to the trigger 302 , for example the voice command 306 , the user gesture 308 or the time delay 310 , to identify.

The trigger is compared with saved triggers that are assigned to the app prototype in order to identify a suitable trigger (block 808 ). For example, the processing module compares 304 the identified trigger 302 with the assigned interaction data 204 to identify a suitable trigger. The request processing module 304 compares the voice command 306 or the user gesture 308 for example with the saved voice commands 212 or the user gestures 214 the associated interaction data 204 . The processing module can be used to identify the suitable trigger 304 use a variety of techniques such as unique identifiers, hashes, text-based searches, and the like. In addition, the processing module 304 constantly monitor time delay triggers when a new visual page is displayed.

Becomes the trigger 302 as a voice command 306 received, so the processing module 304 the sound data of the voice command 306 for the spoken-to-text service 128 ready. The spoken-to-text service 128 processes the sound data in real time and converts the sound data of the voice command 306 into a text string. The processing module 304 receives the text string of the voice command 306 from the spoken-to-text service 128 and compares the text string of the voice command 306 with the stored text strings of the voice commands 212 to identify a suitable voice command.

The processing module 304 can identify a suitable voice command by typing the letters of the text string of the received voice command 306 with letters of the text strings of the saved voice commands 212 compares. A higher number of matching letters indicates a higher probability of a hit. For example, if the command contains the word "mouse", the processing module can 304 Recognize the word "house" as matching "mouse" because the words have four matching characters. In some cases, the spoken-to-text service includes 128 a reliability score with the text string indicating the degree of reliability that the text string is correct. The processing module supports the reliability parameter 304 when identifying a suitable voice command. In one or more implementations, the processing module is 304 configured to select the appropriate, low tolerance text string to allow the user to interact with the app prototype more easily.

An action that is assigned to the appropriate trigger is carried out during the test phase (block 810 ). In accordance with the principles explained here, the act may correspond to a spoken reply, a page transition to another visual page, or the playing of a media file. For example, the processing module identifies 304 in response to a detection of that the trigger 302 to a saved trigger 208 the associated interaction data 204 fits, the corresponding action 210 associated with the trigger in the associated interaction data 204 . An action 312 having a spoken reply 314 , a page transition 316 or a media issue 318 corresponds to the implementation module 320 forwarded. The implementation module 320 causes the appropriate action 312 associated with the identified trigger is performed, for example, by outputting the spoken response 314 , Performing a page transition 316 to another visual page, output of sound or video content or a combination of these.

Corresponds to the action 312 a spoken reply 314 , the implementation module can use the text string of the spoken reply 314 in real time for the text-to-speech service 130 provide. The text-to-speech service 130 converts the text string of the spoken reply 314 in sound data, that of the spoken reply 314 correspond to. The implementation module 320 then causes the output of the sound data, that of the spoken reply 314 correspond.

9 shows an exemplary procedure 900 , which uses a phonetic language translator to match voice commands that were recorded while testing the app prototype.

Interaction data that are assigned to an app prototype are stored in a memory (block 902 ). In accordance with the principles explained here, the interaction data include stored voice commands, actions that are to be performed in response to the detection of respective stored voice commands, and at least one visual page. The at least one visual page is displayed on a display device during a test phase for the app prototype (block 904 ), and a non-phonetic voice command spoken in a language with a non-phonetic alphabet is recorded (block 906 ). As an example, the test module initiates 132 in the test phase the display of a first visual page 122 (e.g. a homepage) of the app prototype 120 . While displaying the first visual page 122 a user interacts with the app prototype 120 by issuing a nonphonetic voice command 402 speaks. The nonphonetic voice command 402 spoken in the language with a non-phonetic alphabet is recorded by at least one microphone as sound data.

The non-phonetic voice command is communicated in real time to a spoken-to-text service in order to convert the non-phonetic voice command into a non-phonetic text string of the voice command given in the language of the non-phonetic alphabet (block 908 ). For example, the test module communicates 132 the sound data of the recorded non-phonetic voice command 402 to the spoken-to-text service 128 . The spoken-to-text service 128 processes the sound data of the non-phonetic voice command 402 in real time to the sound data of the non-phonetic voice command 402 into a non-phonetic text string 404 convert the recorded voice command.

The non-phonetic text string of the voice command is communicated in real time to a phonetic language translator who translates the non-phonetic text string into a phonetic text string of the voice command given in a language with a phonetic alphabet (block 910 ). The test module 132 communicates, for example, the non-phonetic text string 404 to the phonetic translator 134 . The phonetic language translator 134 translates the non-phonetic text string given in the language with the non-phonetic alphabet 404 the recorded voice command into a phonetic text string given in the language with the phonetic alphabet 406 of the recorded voice command. In one or more implementations, the phonetic language translator receives 134 non-phonetic text strings during the test phase 410 the stored voice commands 408 . The saved voice commands 408 can, for example, the voice commands 412 that are in the store 406 be held up to correspond. The phonetic language translator 134 translates the non-phonetic text strings 410 the stored voice commands 408 in phonetic text strings 412 the stored voice commands 408 . Alternatively, the phonetic language translator 134 the phonetic text strings 410 the stored voice commands 408 before the test phase in phonetic text strings 412 the stored voice commands 408 translate. In this case, the phonetic text strings 412 are held in the memory.

The phonetic text string of the voice command is compared with phonetic text strings of the stored voice commands that are associated with the app prototype in order to identify a suitable voice command (block 912 ). As an example, the phonetic comparison module receives 414 of the test module 132 the phonetic text string 406 of the recorded voice command 402 and the phonetic text strings 412 the stored voice commands 408 . The phonetic comparison module 414 compares the phonetic text string 406 then with the phonetic text strings 412 to a matching voice command 416 the phonetic text strings 412 to identify according to the stored voice commands. The phonetic comparison module 414 can the matching voice command 416 identify it by the letters of the phonetic text string 406 with those of the phonetic text strings 412 the stored voice commands 408 compares. Because the phonetic text strings 406 and 412 have been translated into a phonetic script is the phonetic comparison module 414 able to get the appropriate voice command 416 based on the number of letters that match the voice command 416 has in common with the recorded voice command. A higher number of matching letters indicates a higher probability of a hit, since both text strings are written in a phonetic font.

An action is carried out that is assigned to the appropriate voice command (block 914 ). As soon as the appropriate voice command 416 from the phonetic comparison module 414 has been identified, the test module initiates, for example, the execution of the assigned action, as described here.

Having described exemplary procedures according to one or more implementations, consider an exemplary system and apparatus that can be used to implement the various techniques described herein.

Exemplary system and device

10th Figure 900 generally shows an exemplary system that includes an exemplary computing device 1002 includes one or more computing systems and / or one or more devices that can implement the various techniques described herein. This is through the inclusion of the development app 118 and the voice interaction development system 104 shown. The computing device 1002 can be, for example, a server of a service provider, a device which is assigned to a client (for example a client device), an on-chip system and / or any other suitable computing device or such a computing system.

The exemplary computing device 1002 includes a processing system as shown 1004 , one or more computer-readable media 1006 and one or more I / O interfaces 1008 that are coupled to one another in terms of communication technology. Although not shown, the computing device can 1002 further include a system bus or other data and command transmission system that couples the various components together. A system bus can contain any bus structure or a combination of different bus structures, for example a memory bus or a memory controller, a peripheral bus, a universal serial bus and / or a processor or local bus that uses a large number of bus architectures. A variety of other examples, such as control or regulation and data lines, are also included.

The processing system 1004 represents a functionality for performing one or more operations using hardware. The processing system is corresponding 1004 represented such that there are hardware elements 1010 includes that can be configured as processors, functional blocks, and the like. This may include implementation in hardware as an application specific integrated circuit or as another logic device formed using one or more semiconductors. The hardware elements 1010 are not limited by the materials from which they are formed or by the processing mechanisms that are used. The processors can consist, for example, of a semiconductor / semiconductors and / or transistors (for example electronic integrated circuits (ICs)). In this context, instructions that can be executed on the processor side can be instructions that can be carried out electronically.

The computer readable storage media 1006 are shown such that they have a memory 1012 include. The memory 1012 offers a memory capacity that is assigned to one or more computer-readable media. The memory component 1012 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read only memory (ROM), flash memory, optical disks, magnetic disks, and the like). The memory component 1012 may include fixed media (e.g. RAM, ROM, a hard disk drive and the like) as well as removable media (e.g. a flash memory, a removable hard disk drive, an optical disk and the like). The computer readable media 1006 can be configured in a variety of ways, as described below.

An input / output interface / input / output interfaces 1008 provides functionality that enables a user to enter commands and information into the computing device 1002 inputs, and which also enables the user and / or other components or devices to be presented with information using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone, a scanner, a touch functionality (e.g., capacitive or other sensors configured to detect physical touch), a camera (the For example, visible or invisible wavelengths, such as infrared frequencies, can be used to recognize movements as gestures that do not imply touch), and the like. Examples of output devices include a display device (e.g., a monitor or projector), speakers, a printer, a network card, a tactile-reactive device, and the like. Therefore, the computing device 1002 in many ways, like to be described below, can be configured to support user interaction.

Various techniques have been described in the general context of software, hardware elements, or program modules. In general, such modules include routines, programs, objects, elements, components, data structures and the like, which perform certain tasks or implement certain abstract data types. The terms "module", "functionality" and "component" generally refer to software, firmware, hardware, or a combination thereof, as used herein. The features of the techniques described here are platform-independent, which means that the techniques can be implemented on a large number of commercially available computing platforms with a large number of processors.

An implementation of the modules and techniques described can be stored on or transmitted via a specific form of computer-readable media. The computer readable media may include a variety of media on which the computing device is based 1002 can be accessed. By way of example and not limitation, computer readable media may include "computer readable storage media" and "computer readable signal media".

"Computer-readable storage media" can refer to media and / or devices that enable permanent and / or non-temporary storage of information in contrast to mere signal transmission, carrier waves or signals per se. Computer-readable storage media therefore refer to non-signal-carrying media. Computer-readable storage media contain hardware, for example volatile and non-volatile, removable and non-removable media and / or storage devices, which are implemented in a method or a technology which is suitable for storing information, for example as computer-readable instructions, data structures, program modules, logic elements / Circuits or other data. Examples may include computer readable storage media including RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, DVD or other optical storage, hard drives, magnetic cartridges, magnetic tapes, magnetic disk storage or other magnetic storage devices or other storage devices, physical Media or objects that are suitable for storing the desired information and that can be accessed by a computer.

“Computer-readable signal media” can refer to signal-carrying media that are configured to provide instructions to the hardware of the computing device 1002 for example, to transmit over a network. Signal media can typically embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves, data signals, or another transport mechanism. Signal media also contain any information distribution media. The term “modulated data signal” denotes a signal in which one or more properties are set or changed in such a way that information is encoded in the signal. By way of example and not limitation, communication media include wired media, such as a wired network or a direct wired connection, and wireless media, such as acoustic, radio frequency, infrared, and other wireless media.

As described above, the hardware elements 1010 and the computer readable media 1006 Modules, programmable device logic, and / or fixed device logic implemented in the form of hardware that, in some embodiments, can be used to implement at least some aspects of the techniques described herein, for example, to perform one or more instructions. The hardware can include components of an integrated circuit or on-chip system, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD) and other implementations in silicon or other hardware. In this context, hardware can act as a processing device that performs program tasks defined by instructions and / or logic embodied by the hardware, as well as hardware used to store instructions for execution, such as those previously described computer readable storage media.

Combinations of the foregoing can also be used to implement various techniques described here. Accordingly, software, hardware, or executable modules may be implemented as one or more instructions and / or logic implemented in some form of computer readable storage media and / or by one or more hardware elements 1010 is embodied. The computing device 1002 can be configured to implement certain instructions and / or functions according to software and / or hardware modules. Correspondingly, an implementation of a module, given by software, can be carried out by the computing device 1002 is executable, at least partially can be achieved in hardware, for example using computer-readable storage media and / or hardware elements 1010 of the processing system 1004 . The instructions and / or functions can be of one or more objects (for example one or more computing devices 1002 and / or one or more processing systems 1004 ) be executable / operable to implement the techniques, modules and examples described here.

The techniques described here can vary in various configurations of the computing device 1002 are supported and are not limited to the specific examples of the techniques described here. The functionality can also be implemented entirely or in parts using a distributed system, for example via a "cloud" 1014 using a platform 1016 as described below.

The cloud 1014 includes a platform 1016 for resources 1018 and / or represents it. The platform 1016 abstracts the underlying functionality of the hardware (e.g. server) and software resources of the cloud 1010 . The resources 1018 may include applications and / or data that can be used while computer processing is performed on servers that are run by the computing device 1002 are removed. The resources 1018 may also include services that are provided over the Internet and / or over a subscriber network, such as a cell-based or a Wi-Fi network.

The platform 1016 can abstract resources and functions to the computing device 1002 to connect with other computing devices. The platform 1016 can also serve to abstract the scaling of resources in order to provide a corresponding degree of scaling for existing needs for the resources 1018 that over the platform 1016 are implemented to provide. Accordingly, in an embodiment with mutually connected devices, the implementation of the functionality described here can be implemented via the system 1000 be distributed. The functionality can also be used on the computing device, for example 1002 as well as through the platform 1016 that the functionality of the cloud 1010 be abstracted, implemented.

Final note

Although the invention has been described in a language specific to structural features and / or methodological processes, it should be appreciated that the invention as defined in the appended claims is not necessarily limited to the specific features or processes described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed invention.

QUOTES INCLUDE IN THE DESCRIPTION

This list of documents listed by the applicant has been generated automatically and is only included for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 62794466 [0001]

Claims (20)

A method implemented by at least one computing device in a digital media environment that enables testing of an app prototype with voice functionality, the method comprising: by the at least one computing device, holding interaction data associated with the app prototype in a memory, the interaction data comprising stored voice commands, actions that are to be performed in response to the detection of respective stored voice commands, and at least one visual page; on a display device of the at least one computing device, displaying the at least one visual page during a test phase for the app prototype; recording at least one microphone of a non-phonetic voice command spoken in a language with a non-phonetic alphabet; communicating in real time the non-phonetic voice command to a voice-to-text service to convert the non-phonetic voice command into a non-phonetic text string of the voice command given in the non-phonetic alphabet; communicating in real time the non-phonetic text string of the voice command to a phonetic language translator for translating the non-phonetic text string into a phonetic text string of the voice command given in a phonetic alphabet; Comparing the phonetic text string of the voice command with phonetic text strings of the stored voice commands given in the phonetic alphabet associated with the app prototype to identify a suitable voice command; and Perform an action associated with the appropriate voice command in the memory. Procedure according to Claim 1 , wherein the stored voice commands are received before the test phase as non-phonetic text strings given in the non-phonetic alphabet. Procedure according to Claim 2 , wherein the stored voice commands are held in the memory as non-phonetic text strings of the stored voice commands and the phonetic language translator translates the non-phonetic text strings of the stored voice commands into the phonetic text strings of the stored voice commands given in the phonetic alphabet during the test phase. Procedure according to Claim 2 or 3rd , wherein the phonetic language translator translates the non-phonetic text strings of the stored voice commands into the phonetic text strings of the stored voice commands given in the phonetic alphabet before the test phase, and wherein the stored voice commands are stored in the memory as phonetic text strings of the stored voice commands. The method of any preceding claim, wherein the comparing comprises: Compare a number of characters that the non-phonetic text string of the voice command has in common with the non-phonetic text strings of the stored voice commands. Procedure according to Claim 5 wherein the matching voice command is identified based at least in part on the phonetic text string of the matching voice command with a maximum number of matching characters with respect to the phonetic text string of the recorded voice command. Method according to one of the preceding claims, wherein the non-phonetic text string of the voice command is written in Mandarin Chinese and wherein the translated phonetic text string of the voice command is written in Pinyin. Method according to one of the preceding claims, wherein the non-phonetic text string of the voice command is written in Kanji and wherein the translated phonetic text string of the voice command is written in Romaji. Method according to one of the preceding claims, wherein the phonetic text string of the voice command and the phonetic text strings of the stored voice commands are written in the same phonetic alphabet. Method according to one of the preceding claims, wherein the phonetic language translator translates non-phonetic text strings into languages with non-phonetic alphabets by romanizing the non-phonetic text strings in Latin script. A system comprising: a test module for displaying at least one visual page of an app prototype on a display device during a test phase of the app prototype; a spoken-to-text service for converting a non-phonetic voice command spoken in a language with a non-phonetic alphabet and picked up by at least one microphone during the test phase of the app prototype into one in a non-phonetic Alphabet given non-phonetic text string of the voice command; a phonetic speech translator for translating the non-phonetic text string of the voice command into a phonetic text string of the voice command given in a phonetic alphabet; a comparison module for comparing the phonetic text string of the voice command with phonetic text strings given in the phonetic alphabet of stored voice commands that are associated with the app prototype, to identify a suitable voice command; and an execution module for performing an action associated with the appropriate voice command. System according to Claim 11 , wherein the comparison module is configured to identify the appropriate voice command by: comparing a number of characters that the non-phonetic text string of the voice command has in common with the non-phonetic text strings of the stored voice commands; and identifying the matching voice command based at least in part on the phonetic text string of the matching voice command with a maximum number of matching characters with respect to the phonetic text string of the recorded voice command. System according to Claim 11 or 12th , with the non-phonetic text string of the voice command written in Mandarin Chinese, and the translated phonetic text string of the voice command written in Pinyin. System according to one of the Claims 11 to 13 , where the non-phonetic text string of the voice command is written in Kanji and the translated phonetic text string of the voice command is written in Romaji. System according to one of the Claims 11 to 14 wherein the execution module is configured to perform the action by: identifying a spoken response associated with the appropriate voice command; Providing a spoken reply text string for a text-to-speech service; receiving a spoken response audio file from the text-to-speech service; and outputting the spoken response by playing the sound file. System according to one of the Claims 11 to 15 , wherein the implementation module is configured to perform the action by: identifying a page transition associated with the appropriate voice command, the page transition identifying an additional visual page of the app prototype; and display the other visual page of the app prototype. System according to one of the Claims 11 to 16 , wherein the stored voice commands are received before the test phase as non-phonetic text strings given in the non-phonetic alphabet. System according to Claim 17 , wherein the stored voice commands are held in a memory as non-phonetic text strings of the stored voice commands and wherein the phonetic speech translator is further configured to translate the non-phonetic text strings of the stored voice commands into the phonetic text strings of the stored voice commands during the test phase. System according to Claim 17 or 18th , wherein the phonetic speech translator is further configured to translate the non-phonetic text strings of the stored voice commands into the phonetic text strings of the stored voice commands given in the phonetic alphabet prior to the test phase, and wherein the stored voice commands are stored in a memory as phonetic text strings of the stored voice commands . System according to one of the Claims 11 to 19th , wherein the phonetic language translator is configured to translate non-phonetic text strings into languages with non-phonetic alphabets by romanizing the non-phonetic text strings in Latin script.

Images