US20220214775A1

US20220214775A1 - Method for extracting salient dialog usage from live data

Info

Publication number: US20220214775A1
Application number: US17/703,308
Authority: US
Inventors: Rushin N. SHAH; Devang K. Naik
Original assignee: Apple Inc
Current assignee: Apple Inc
Priority date: 2013-12-06
Filing date: 2022-03-24
Publication date: 2022-07-07
Also published as: US11314370B2; US20190034040A1; US10296160B2; US20150161521A1

Abstract

Systems and processes are disclosed for virtual assistant request recognition using live usage data and data relating to future events. User requests that are received but not recognized can be used to generate candidate request templates. A count can be associated with each candidate request template and can be incremented each time a matching candidate request template is received. When a count reaches a threshold level, the corresponding candidate request template can be used to train a virtual assistant to recognize and respond to similar user requests in the future. In addition, data relating to future events can be mined to extract relevant information that can be used to populate both recognized user request templates and candidate user request templates. Populated user request templates (e.g., whole expected utterances) can then be used to recognize user requests and disambiguate user intent as future events become relevant.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No. 16/144,871, filed Sep. 27, 2018, which is a continuation of U.S. application Ser. No. 14/099,776, filed Dec. 6, 2013, the entire contents of which are hereby incorporated by reference.

FIELD

This relates generally to virtual assistants and, more specifically, to mechanisms for using live data for recognition of requests provided to virtual assistants.

BACKGROUND

Electronic devices are able to access an increasingly larger and more diverse quantity of functions, services, and information, both via the Internet and from other sources. Functionality for such devices continues to improve rapidly, with advances in both hardware and software applications of consumer devices, smartphones, tablet computers, and the like. In many instances, each software application, function, website, or other feature can have its own user interface and operational models, which can be difficult to learn and even overwhelming for some novice users. Moreover, many users may overlook or be unaware of the extensive device functionality and information available to them. Such users may be unable to use certain resources effectively, and some users may become frustrated or overwhelmed by the growing capabilities of consumer devices.
In addition to novice users, a variety of other individuals may find it difficult to effectively utilize the many features available on consumer devices. For example, individuals who are impaired, disabled, elderly, busy, distracted, operating a vehicle, engaged in certain activities, or the like may have difficulty interfacing with their electronic devices safely and effectively. Such users can also be particularly likely to find it difficult to navigate the large number of diverse functions, applications, websites, features, and the like that may be available.
An intelligent automated assistant (or virtual assistant) can beneficially provide an improved interface between a human user and an electronic device that addresses the needs of certain users while also providing enhanced functionality for even expert users. For example, a virtual assistant can facilitate effective use of the varied functions, applications, websites, features, and the like that may be available.
In some examples, a virtual assistant can function by recognizing and responding to known requests in predetermined ways. A virtual assistant, however, may not yet recognize and may not yet be trained to respond to a variety of new requests from users that may change over time. For example, users can request information from a new source, request information from a known source in a new way, request a certain function using as-yet unrecognized terminology, request a new function for a new software application, or the like. In addition, recognized or known source information can become outdated over time, and the virtual assistant may not recognize new terminology employed by users to refer to updated information. For example, users may request information related to future events (e.g., a movie premier), but the virtual assistant may not yet recognize related terminology (e.g., the new movie title). Accordingly, a virtual assistant can receive user requests that it is unable to effectively handle as, for example, source information changes over time and as users make new requests or make certain requests in new ways.

SUMMARY

Systems and processes are disclosed for virtual assistant request recognition using live usage data and anticipated or future data. In one example, a user request received by a virtual assistant can be compared to known request templates to determine how to respond. If the received request is not recognized, the received request can be used to develop candidate request templates that can be stored and tracked to determine the salience of each of the candidate request templates. For example, candidate templates formed from unrecognized user requests can be stored in a database. As new candidate templates are received, they can be compared to candidate templates already in the database. When matching candidate templates are identified, a count associated with matched candidate templates can be incremented to reflect how frequently the candidate template has been identified from received requests. When a count associated with a candidate template reaches a threshold level, the corresponding candidate template can be considered salient and can be used to train the virtual assistant to recognize similar requests in the future (e.g., an associated language model can be trained with the candidate templates).
In some examples, a virtual assistant can be trained with data that is expected to appear in future user requests. Data relating to future events can be received, and names, dates, addresses, and like information can be extracted from the received data. The extracted data can be used to populate or seed recognized user request templates to generate new populated request templates (e.g., whole user utterances). The populated request templates can be used to train a virtual assistant, such that when the future data becomes relevant, the virtual assistant can be prepared to recognize and respond to related user requests.
In other examples, data relating to future events can be combined with candidate request templates that a virtual assistant may not yet recognize. For example, names, dates, addresses, and the like extracted from data relating to future events can be used to populate or seed candidate request templates that have been deemed salient based on how frequently they have been received. The populated candidate request templates can be used to train a virtual assistant along with training the virtual assistant to recognize and respond to the corresponding unpopulated candidate request templates.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary system for request recognition for a virtual assistant.

FIG. 2 illustrates an exemplary process for request recognition for a virtual assistant.

FIG. 3 illustrates exemplary virtual assistant request templates.

FIG. 4 illustrates an exemplary process for training a virtual assistant to recognize anticipated future requests.

FIG. 5 illustrates an exemplary process for facilitating user interactions with a virtual assistant associated with a user device.

FIG. 6 illustrates an exemplary system that can be used for request recognition for a virtual assistant.

FIG. 7 illustrates an exemplary personal device that can be configured to provide a virtual assistant interface according to various examples.

FIG. 8 illustrates another exemplary personal device that can be configured to provide a virtual assistant interface according to various examples.

DETAILED DESCRIPTION

In the following description of examples, reference is made to the accompanying drawings in which it is shown by way of illustration specific examples that can be practiced. It is to be understood that other examples can be used and changes can be made without departing from the scope of the various examples.
This relates to virtual assistant request recognition using live usage data and anticipated or future data. In one example, a received user request can be compared to known or recognized request templates to determine how to respond. If the received request is not recognized (e.g., a matching known request template is not found), the received request can be used to develop candidate request templates that can be stored and tracked to determine the salience of each of the candidate request templates. For example, a user request can be parsed into a variety of candidate templates based on combinations and sub-combinations of parsed phrases (or n-grams) of the request. Each of the candidate templates can be compared to previously stored candidate templates to determine whether each template has been developed from a prior user request from the same or a different user. When matching candidate templates are identified, a count associated with matched candidate templates can be incremented to reflect how frequently the candidate template has been identified from received requests.
As the same candidate template is identified multiple times from various as yet unrecognized user requests, the count associated with a particular candidate template can increase significantly. When the count associated with a particular candidate template reaches a threshold level, a notification can be generated including the candidate request template to indicate that the candidate request template may be salient and useful for language models used to recognize and respond to user requests. In some examples, when a count reaches a threshold level, an associated candidate request template can be automatically included in virtual assistant language models or otherwise used to train virtual assistants to recognize the user request in the future. By training virtual assistants (or training language models associated with virtual assistants) with candidate request templates that appear frequently enough to be considered salient, virtual assistants can more effectively recognize and respond to user requests and adapt as user requests change over time.
In another example, virtual assistants can be trained with anticipated or future data (e.g., data relating to a future event). Anticipated or future data can be received or mined from a variety of sources, such as news feeds, blogs, websites, or the like. A variety of names, event details, dates, and other information can be extracted from the received data. The extracted data can then be used to generate new populated request templates or request enumerations (e.g., complete expected user utterances) by populating or seeding known user request templates with the future data (e.g., inserting future data into request templates). For example, new populated request templates can be specific user requests, including entities or defined variables that a virtual assistant can use to formulate a particular response (e.g., detailed requests including specific search terms that a virtual assistant can recognize in order to provide a response directly related to the specific search terms).
The new populated request templates can be used to train a virtual assistant, such that when the future data becomes relevant, the virtual assistant can be prepared to recognize and respond to related user requests. For example, request templates relating to movies can be populated with details for a movie that will be released at a future date (e.g., a new movie title, new actors, new director, etc.). The populated request templates based on the future movie release date can then be used to train a virtual assistant in anticipation of future user requests relating to the movie. By training a virtual assistant based on anticipated or future data, the virtual assistant can more effectively recognize and respond to user requests related to current events, and in general can more effectively anticipate user needs that change over time.
In yet another example, anticipated or future data can be used to populate candidate request templates that a virtual assistant may not yet recognize. As described above, frequently appearing user requests that may not yet be recognized can be used to identify salient candidate request templates that can be useful for training a virtual assistant. Instead of or in addition to training a virtual assistant with new candidate request templates, anticipated or future data can be used to generate populated requests templates (e.g., complete expected user utterances) based on the identified salient candidate request templates. The populated candidate request templates can then be used to train a virtual assistant to recognize anticipated user requests based on newly identified request templates. In this manner, a virtual assistant can be made more robust by adapting to new and changing user request templates as well as anticipating and preemptively adapting to accommodate expected future user needs.
Accordingly, identifying new salient request templates from live usage data and anticipating future user needs based on data relating to future events can advantageously improve virtual assistant request recognition and overall virtual assistant utility. It should be understood, however, that still many other advantages can be achieved according to the various examples discussed herein.
FIG. 1 illustrates exemplary system 100 for request recognition for a virtual assistant. In one example, system 100 can include user device 102 that can provide an interface for interacting with a virtual assistant. User device 102 can include any of a variety of devices, such as a cellular telephone (e.g., smartphone), tablet computer, laptop computer, desktop computer, portable media player, wearable digital device (e.g., digital glasses, wristband, wristwatch, brooch, armband, etc.), or the like. In some examples, user device 102 can include microphone 106 that can record spoken user requests. In other examples, user device 102 can include a variety of other mechanisms for receiving input from a user, such as a touchscreen, keyboard, mouse, optical sensor, camera, gesture recognition sensor, proximity sensor, ambient light sensor, or the like. Although microphone 106 is illustrated in the example of system 100 for receiving spoken user requests, user device 102 can receive user requests for a virtual assistant using any available input mechanisms (e.g., receiving text via a keyboard, text via a touchscreen, gestures via a camera, instructional barcodes via a camera, instructional quick response (QR) codes via a camera, instructions via a near field communication sensor, etc.).
User device 102 can also include processor 104, which can receive user requests and process the requests in any number of ways. For example, processor 104 can cause an audio file of a spoken user request to be transmitted to server 110 through network 108. Network 108 can include any of a variety of networks, such as a cellular telephone network, WiFi network, wide area network, local area network, the Internet, or the like. In another example, processor 104 can cause a spoken user request to be transcribed into a textual request or translated into a different format useable by the virtual assistant for processing the request, and can cause the transcribed request to be transmitted to server 110 through network 108. Processor 104 can also run some or all of the instructions for providing the interface for the virtual assistant (e.g., prompting a user for a request, causing audio to be played, causing information to be displayed, etc.).
In some examples, server 110 can include a language processor for performing speech recognition. For example, server 110 can be configured to recognize a speech sequence by decoding the acoustics that represent speech sounds and using a language model to place constraints on the acoustic sequence to estimate the most likely spoken word sequence that makes up a user request. In system 100, language model database 112 can include various constraints, user request templates, interpretation likelihoods, domain boundaries, and a variety of other information to facilitate accurate speech recognition and ultimately accurate determination of a user request. The spoken word sequence can be parsed by a natural language engine (not shown) that can derive user intent (e.g., determine the functional request that a virtual assistant can process and act upon to respond to the request). In some examples, acoustic models can be trained on speech audio while language models can be trained on recognized text data to robustly estimate the likelihoods of particular n-gram sequences (e.g., parsed portions of a request). It should be understood that training a virtual assistant can include training an associated acoustic model, training an associated language model, or both.
Other aspects relating to virtual assistant technology are disclosed in the following references: U.S. Patent Publication No. 2012/0016678 for “Intelligent Automated Assistant,” the disclosure of which is incorporated herein by reference; and U.S. Patent Publication No. 2012/0265528 for “Using Context Information to Facilitate Processing of Commands in a Virtual Assistant,” the disclosure of which is incorporated herein by reference.
In some examples, speech recognition accuracy can be enhanced by training both acoustic and language models on large sets of real world usage data. As speech changes (e.g., different word choice, altered phrasing, etc.), and as new terminology emerges to describe current ideas, events, and the like, models trained on older data can become outdated. For example, models trained on only text-based data sources can miss changes over time in spoken dialogue. To improve accuracy, live usage data can be employed as part of the training for virtual assistants in general and for acoustic and language models in particular. As discussed below with reference to process 220 of FIG. 2, candidate database 114 of system 100 can be used to track live usage data, and in particular, track the salience of unrecognized user requests to determine whether such requests should be used for training a virtual assistant or language model.
Although FIG. 1 illustrates server 110, language model database 112, and candidate database 114 as being separated from user device 102 by network 108, it should be appreciated that, in other examples, the functions of server 110 can be performed by processor 104 on user device 102, and databases 112 and 114 can likewise be stored on user device 102. In such examples, speech recognition and other language processing functions can be performed directly on user device 102, and the techniques discussed herein for request recognition can similarly be performed on user device 102.
It should likewise be understood that many variations are possible for a system that can be used according to the examples herein for virtual assistant request recognition. For example, although FIG. 1 illustrates databases 112 and 114 as separate storage entities, both can be stored on a single storage device, each can be distributed across multiple storage devices, some or all of the databases can be stored within user device 102, and many other variations are also possible.
FIG. 2 illustrates exemplary process 220 for request recognition for a virtual assistant. Process 220 can, for example, be executed on server 110 of system 100 utilizing language model database 112 and candidate database 114 discussed above with reference to FIG. 1. At block 222, a textual representation of user speech can be received. Although not shown in FIG. 2, prior to block 222, a spoken user request can be received in audio format and transcribed into a textual format according to any of a variety of speech recognition methods. Such transcription can be performed on a user device (e.g., user device 102 of FIG. 1), on a server (e.g., server 110 of FIG. 1), or on another device. In some examples, speech recognition can be performed using an acoustic model.
User speech can be directed to a virtual assistant via an interface on a user device, and can include any of a variety of user requests. For example, user requests can include a command for the virtual assistant to perform a certain function (e.g., compose an email, send a text message, initiate a phone call, launch an application, play an audio file, etc.). In other examples, user requests can include a request for information (e.g., sports scores, directions, restaurant recommendations, theater schedules, weather, stocks, internet searches, encyclopedia entries, etc.). In still other examples, user requests can include dialogue directed to the virtual assistant or requests relating to the virtual assistant (e.g., statements personifying the virtual assistant, queries of the functional capabilities of the virtual assistant, statements defining preferences for virtual assistant interaction, etc.). It should be appreciated that user requests need not be limited to queries or commands, but can include any interaction between a user and the virtual assistant. It should likewise be understood that user speech and user requests of any type can be received, and the virtual assistant can be trained to provide meaningful responses to any type of user speech or request.
At block 224, the received textual representation of user speech can be compared to recognized user request templates (e.g., exemplars or use cases identifying a particular user request to which a virtual assistant is trained to respond). In some examples, recognized user request templates can form part of a language model, such as that stored in language model database 112 of FIG. 1. In general, a language model associated with a virtual assistant can define how the virtual assistant responds to known user requests (e.g., can enumerate the requests to which a virtual assistant is trained to respond). The received textual representation of user speech can be compared to all or some subset of the recognized user request templates in a language model, database, or the like. FIG. 3 illustrates exemplary virtual assistant request templates, including recognized request templates stored in language model database 112. Although not shown, recognized user request templates can be stored along with corresponding functions that a virtual assistant can perform in response to receiving a particular user request. For example, a recognized user request template can be stored along with an instruction, set of steps, phrase, command, or the like that a virtual assistant can execute or otherwise use to respond to a recognized request.
In some examples, user request templates can include variables, entities, operands, search terms, or the like that a virtual assistant can use in performing a corresponding function. For example, as illustrated in FIG. 3, one exemplary recognized user request template could include “Identify new <MOVIE_TYPE> movies.” The variable <MOVIE_TYPE> can indicate that whatever entity appears in a user request in that position could be used to perform the function corresponding to the request. In this example, a user could request that the virtual assistant identify new horror movies (e.g., “horror” may be the <MOVIE_TYPE> entity). In response, the virtual assistant may execute a function that searches for recently-released movies by genre, using the entity “horror” to narrow the search results to horror films and thereby provide a meaningful response to the user request.
In some examples, variables or entities can be implied in a user request. For example, another exemplary user request template could include “Locate the nearest <RESTAURANT_TYPE> restaurant.” In this example, the virtual assistant could perform a corresponding search function that incorporates both the explicitly stated restaurant type as well as the implicit variable of the user's current location to find nearby restaurants of a particular type. For example, a user can request that the virtual assistant identify the nearest Thai restaurant. In response, the virtual assistant can perform a corresponding search function that locates Thai restaurants and filters them for proximity.
Although not shown in process 220 of FIG. 2, in some examples, text received at block 222 can be analyzed and used to identify a domain or a set of domains corresponding to the user request to limit the field of potential request templates for comparison at block 224. As illustrated in FIG. 3, recognized request templates can be segmented into different domains (e.g., Domain A, Domain B, etc.). A domain can indicate a subject, genre, area of interest, group of similar requests, or the like. For example, Domain A of FIG. 3 can represent a movie domain that includes requests related to movies, theater schedules, theater locations, movie facts, videos available for viewing on a user device, and the like. Domain B, on the other hand, can represent a restaurant domain that includes requests related to restaurants, foods, restaurant locations, restaurant types, restaurant reviews, reservation systems, and the like.
In some examples, segmenting database 112 into different domains can improve speech recognition and user request interpretation. For example, words appearing in a user request can be used to narrow the field of potential user request matches to a corresponding domain or a subset of all domains. For a movie domain, for example, words like “movies,” “theater,” “showing,” “playing,” “starring,” “director,” “actor,” or the like could be used during speech recognition to narrow a template search to the movie domain. Domain segmentation can also help disambiguate user intent when, for example, entities or request terms may be confusing and difficult to accurately recognize or interpret.
Referring again to process 220 of FIG. 2, at block 226, a determination can be made of whether or not a matching recognized user request template has been found at block 224 from comparing the received textual representation of user speech to recognized user request templates (e.g., those request templates forming part of a language model associated with a virtual assistant). It should be appreciated that a “matching” template need not be identical to the textual representation of user speech. A template can be recognized as a positive match even though users may employ different terminology in formulating a request. For example, a template may include the word “identify,” but synonyms of “identify” can be considered positive matches, such as “find,” “locate,” “list,” “get,” “show,” and the like. A template can also be recognized as a positive match even though the order of words in a request may differ from a template. For example, “Locate Thai restaurants near me” and “Find nearby Thai restaurants” could both match a common request template, such as “Locate the nearest <RESTAURANT_TYPE> restaurant.” Many other request variations are also possible that can still be recognized as positively matching a request template.
If a matching recognized user request template is found (e.g., the “yes” branch of block 226), a function corresponding to the recognized user request template can be performed at block 228. For example, a virtual assistant can be trained to identify movies playing near a particular location in response to the user request template “Identify movies playing in <LOCATION>.” The request template can be included in a language model associated with the virtual assistant, and the template could be identified as matching a corresponding user request, such as “Find movies playing in Sacramento, Calif.” After recognizing that the user request matches the template “Identify movies playing in <LOCATION>,” the virtual assistant can perform the corresponding function of searching for movies playing in theaters located in or near Sacramento, Calif. It should be appreciated that functions attributed to the virtual assistant can be performed by software executing on a server, a user device, and/or another device. For example, virtual assistant software executing on a server can perform the search function and send the results to the user device; virtual assistant software executing on the user device can then cause the results to be displayed, read out, or otherwise provided to the user.
If no matching recognized user request template is found (e.g., the “no” branch of block 226), candidate templates can be generated at block 230 based on the received textual representation of user speech. In some examples, candidate request templates can be generated based on various combinations of the words in a received request. In other examples, candidate request templates can be generated by removing an entity or removing different words or portions of a user request. In still other examples, candidate request templates can be generated by parsing a received request into different n-grams and forming new candidate templates from different numbers and orders of the n-grams. It should be appreciated that a received request can be maintained whole and/or broken up in a variety of ways to generate candidate request templates.
In one example, an unrecognized user request could include “Show movies playing nearby.” At block 230, various candidate request templates could be generated from such a request, such as “Show <>,” “Show <> nearby,” “Show movies <>,” “Show movies playing <>,” “Show <> playing,” “Show <> playing nearby,” “Show <> playing <>,” etc., along with the unmodified request itself. Various candidate request templates can thus be generated from a received request.
Referring again to process 220 of FIG. 2, at block 232, generated candidate request templates can be compared to existing candidate request templates. In one example, the various candidate request templates formed at block 230 from a received request can each be compared to previously generated and stored candidate request templates. As illustrated in FIG. 3, candidate request templates can be stored in candidate database 114. In some examples, candidate request templates can be organized by expected usage domain, such as Domain A shown in FIG. 3 (which may be a movie or theater domain). The expected usage domain can be determined based on words in a received request, contextual cues from prior virtual assistant interaction, contextual cues from subsequent virtual assistant interaction (e.g., repeating a request, correcting a transcribed request, etc.), or in other ways. In some examples, the expected usage domain can be used to narrow the field of search for matching candidate request templates.
Although FIG. 3 illustrates candidate request templates as including a defined entity placeholder (e.g., <BOOK_SERIES>, <DIRECTOR_NAME>), in other examples, a generic entity placeholder can be included in candidate templates (e.g., “Identify movies based on <>”), candidate templates might include a defined entity (e.g., “Identify movies based on ‘The Lord of the Rings’”), or candidate templates might be constructed in other ways for tracking as desired.
Referring again to process 220 of FIG. 2, at block 234, a determination can be made as to whether a matching candidate request template has been found in a database of previously-generated and stored candidate request templates (e.g., candidate database 114). In some examples, “matching” can occur without requiring identical similarity (e.g., candidate templates can be considered matching despite minor variations, such as synonyms, pluralities, connecting words, or the like). If no already-stored matching candidate request template has been found (e.g., the “no” branch of block 234), the unmatched candidate template can be added to a candidate database at block 242. For example, if no match has been found after comparing a generated request template to already-stored candidate templates in a candidate database, the generated request template can be added to the candidate database, such as candidate database 114.
If, however, an already-stored matching candidate request template has been found (e.g., the “yes” branch of block 234), a count associated with a matched candidate template can be incremented. Incrementing the count can reflect, for example, that a particular candidate template has once again been generated from a received and unrecognized user request. FIG. 3 illustrates a count associated with various candidate request templates stored in candidate database 114. For example, candidate request template “Identify movies based on <BOOK_SERIES>” has an associated count of 152, which can indicate that the candidate template has been generated from received and unrecognized user requests 152 times. Similarly, candidate request template “Identify <DIRECTOR_NAME> movies” has an associated count of 57, which can indicate that the candidate template has been generated from received and unrecognized user requests 57 times.
Referring again to process 220 of FIG. 2, at block 238, a notification can be generated when a count associated with a candidate template reaches a threshold level. In some examples, a salience threshold can be defined such that when a count reaches the threshold level, the associated candidate template can be considered salient and useful for training a virtual assistant to recognize the template in the future. In other words, counts associated with candidate templates can be used to determine whether particular candidate templates are received so frequently as to warrant training a virtual assistant to recognize the candidate template in the future. A generated notification can include the associated candidate template. In some examples, the notification can also include additional information that can be useful for training a virtual assistant to recognize the candidate template, such as contextual information relating to the original unrecognized user requests (e.g., dates, times, near-in-time requests, user profile information, etc.). The generated notification can be directed to administrators of the virtual assistant software, to a user, to software associated with a virtual assistant, or the like.
In some examples, the most frequently generated candidate template can yield a request template that corresponds to and can be responsive to a wide variety of specific user requests. For example, several candidate templates can be related and can correspond to the same or a similar function, and selecting the most frequently generated candidate template can yield a request template that satisfies a large number of potential user requests. In addition, in some examples, generating candidate request templates and tracking the frequency of receipt can be used to generate a ranked list of candidate templates similar to that illustrated in FIG. 3. Such a ranked list can be useful for system administrators, users, or others to recognize potential areas for improving a virtual assistant, for understanding current request trends, for understanding user dialogue, for monitoring requests received in a particular domain, or the like. It should be understood that many other benefits are also possible.
Referring again to process 220 of FIG. 2, at block 240, a virtual assistant can be trained with salient candidate templates having associated counts exceeding a threshold level. For example, after a notification is generated at block 238 including a particular candidate template, the candidate template can be used at block 240 to train a virtual assistant to recognize the candidate template in the future and respond appropriately. Training a virtual assistant to recognize the candidate template can include incorporating the candidate template into a language model along with an associated function that can be an appropriate response to receiving a corresponding request. For example, the candidate template and an associated function can be added to language model database 112 of system 100 in FIG. 1. Training can also be performed in other ways to enable a virtual assistant to recognize and respond to a new template.
The function associated with a salient candidate template can be defined in any number of ways. In one example, a system administrator, user, or other person can define a particular function that a virtual assistant can use in responding to receipt of a candidate request template. In other examples, contextual information relating to received requests can be used to determine a desired functionality (e.g., using near-in-time requests to determine a likely desired function for an unrecognized request). Various other approaches can also be used to determine a corresponding function with which to train a virtual assistant to respond to a candidate request template.
FIG. 4 illustrates exemplary process 440 for training a virtual assistant to recognize anticipated future requests. As available information, news, events, and the like change over time, user interactions with a virtual assistant can also change. Process 440 can be used in some examples to prepare a virtual assistant for new anticipated user requests. At block 442, data relating to future events can be received. Such data can come from a variety of sources and relate to a variety of events. For example, data can be received from a variety of news feeds, blogs, announcement pages, corporate websites, RSS feeds, information aggregators, social media sites, or the like. Data can relate to any of a variety of future events, such as a movie premiering, a restaurant opening, a business relocating, a new product being released, a meeting taking place, a speech being given, a holiday approaching, a politician taking office, a sports event, or the like.
Received data can include any of a variety of information relating to future events. For example, for an upcoming movie premier, received data can include a movie title, actor names, plot information, director names, producer names, filming locations, premier/release date, theater locations, or the like. For a restaurant opening, received data can include a restaurant name, location, business hours, head chef, owner, opening date, associated restaurants, job opportunities, or the like. For a sports game being played, received data can include team names, event location, event date, event time, ticketing information, arena information, related statistics, or the like. Thus, a wide variety of information can be received relating to a wide variety of future events, and it should be appreciated that the examples enumerated herein are not limiting of the types of data and types of future events that can be used in executing process 440.
At block 444, entity names can be extracted from the data received at block 442. In one example, received information can be parsed or categorized into recognizable entities or variables. For example, a location or address can be extracted from received data (e.g., recognized as a location or address and delineated) and designated as the location or address associated with a future event; a release date, opening date, event date, or the like can be extracted from received data (e.g., recognized as a date) and designated as a significant date associated with a future event; a person's name and any associated title, role, position, or the like can be extracted from received data and correlated with a future event; and a variety of other information can be extracted from received data and categorized. In some examples, extracted entities can be stored in a database, table, or the like that can indicate how an extracted entity name corresponds to a future event or to what variables an entity name might relate. For example, for a new movie, an actor's name can be extracted from a data feed and designated as relating to a variety of variables or entities that can appear in request templates, such as “actor,” “star,” “role,” or the like. Extracted entity names can also be stored and categorized in a variety of other ways that can be useful for subsequent virtual assistant training.
At block 446, recognized request templates can be populated or seeded with entity names, thereby generating populated request templates (e.g., particular utterances, example uses, example phrases, etc.). In one example, entity names relating to a future event can be used to populate or seed some or all related request templates in a corresponding domain (e.g., insert entity names where there are variables in request templates). For example, for a new movie, extracted entity names can be inserted into some or all of the recognized request templates in a movie domain: an extracted movie title can be used to populate any request template with a related movie title variable, such as <MOVIE_TITLE> in the template “Where is <MOVIE_TITLE> playing;” an extracted movie type can be used to populate any request template with a related movie type variable, such as <MOVIE_TYPE> in the template “Identify new <MOVIE_TYPE> movies,” an actor's name can be used to populate any request template with a related actor variable, such as <ACTOR> in the template “Identify new movies starring <ACTOR>;” and so on. In other examples, a predetermined subset of recognized request templates in a domain can be populated with new entity names.
Some example populated request templates are illustrated in FIG. 3 in language model database 112 including “Identify new ‘horror’ movies,” “Identify new ‘thriller’ movies,” and “Identify new ‘comedy’ movies.” The examples shown correspond to the recognized request template “Identify new <MOVIE_TYPE> movies.” Entity names “horror,” “thriller,” and “comedy” have been populated in the variable <MOVIE_TYPE> to generate the illustrated populated request templates.
Referring again to process 440 of FIG. 4, at block 448, a virtual assistant can be trained with populated request templates. In some examples, populated request templates can improve virtual assistant request recognition by providing expected example utterances a virtual assistant can employ as a reference during request recognition. When a request is received, virtual assistant recognition software can compare a received request to a populated request template to recognize a user's request and disambiguate the user's intent. For example, where requests are ambiguous and can be perceived as different strings of words, the request can be compared to known populated request templates to determine whether one of the potential interpretations is expected or more likely to be correct. The movie title “Argo,” for instance, might sound like “hour go,” “are go,” “our go,” “argh oh,” “Argo,” or other words. A populated request template like “Where is ‘Argo’ playing,” however, can be used to recognize that a user is requesting information relating to the movie title “Argo” as opposed to one of the other possible interpretations.
Training at block 448 can be performed in a variety of ways. In some examples, populated request templates from block 446 can be stored in a database associated with a virtual assistant, such as a language model database or an acoustic model database. As illustrated in FIG. 3, for example, populated request templates can be stored in language model database 112 along with recognized request templates. To recognize what a user wants and determine what action a virtual assistant should perform, a received user request can be compared to recognized request templates, populated request templates, or both. In other examples, populated request templates can be employed in other ways to train a virtual assistant to correctly recognize requests and avoid errors of misunderstanding user intent.
In some examples, entity names extracted at block 444 from newly received data can be used in other ways for virtual assistant request recognition. At block 450, for example, candidate request templates can be populated with entity names in a manner similar to block 446. As discussed above, candidate request templates can include user requests that a virtual assistant may not yet recognize and to which a virtual assistant may not yet be trained to respond. At block 450, candidate request templates can be populated with new entity names to generate populated candidate request templates. As illustrated in FIG. 3, for instance, candidate request templates stored in candidate database 114 can be populated with entity names to generate populated candidate request templates (not shown). For example, data relating to a new movie based on a book series can be received, corresponding entity names can be extracted from the data, and the entity names can be used to populate variables in corresponding candidate request templates, such as “Identify movies based on <BOOK_SERIES>” and “Identify <BOOK_SERIES> movies” shown in FIG. 3. Resulting populated candidate request templates might include, for instance, “Identify movies based on ‘The Lord of the Rings’” and “Identify ‘The Lord of the Rings’ movies.”
At block 452, a virtual assistant can be trained with populated candidate request templates in a manner similar to block 448. In some examples, a virtual assistant can be trained with both new candidate request templates and corresponding populated candidate request templates. For example, at block 240 in process 220 of FIG. 2, in addition to training a virtual assistant with new salient candidate request templates, the virtual assistant can be trained with populated candidate request templates generated according to process 440. In some examples, such training can include adding new request templates and/or new populated request templates to a language model associated with the virtual assistant. Process 440 can thus be used in multiple ways for virtual assistant recognition of anticipated future user requests.
FIG. 5 illustrates exemplary process 560 for facilitating user interactions with a virtual assistant associated with a user device. In some instances, process 560 can be performed by a user device, such as user device 102 of system 100 in FIG. 1. In other instances, process 560 can be performed by a user device in conjunction with a server, such as server 110 of system 100 in FIG. 1. Moreover, in some examples, process 560 can be performed together in a coordinated manner with other processes—such as with process 220 of FIG. 2 and process 440 of FIG. 4—for virtual assistant request recognition.
At block 562, a user query or user request can be received at a user device. A user query can be directed toward a virtual assistant and can include any command, request, question, statement, or the like. A user query can also be received in any form, including text, voice, gestures, images, or the like. At block 564, a response can be provided indicating that the virtual assistant is as yet untrained to respond to the received user query. For example, as discussed above with respect to blocks 224 and 226 of process 220, a received request may not yet be recognized by a virtual assistant, or a virtual assistant may otherwise not yet be trained to provide a response to a received query. In some examples, the user query can be transmitted to a server for processing, and the server can indicate to the user device that a trained response is unavailable. Providing a response at block 564 can include causing text to be displayed, causing audio to be played, causing text to be read out, causing an image to be displayed, or the like. It should be appreciated that the response need not be a particular message, but might include a tone, image, word, or the like that can indicate to a user that the virtual assistant is untrained to respond substantively to the particular request. In other examples, the virtual assistant can prompt the user for additional information (e.g., repeat the request, restate the request, confirm interpretation, query whether a related web search is desired, etc.).
At block 566, the user query can be transmitted to a server. In some examples, an audio file of the user query can be transmitted to a server. In other examples, contextual information relating to the user query can be transmitted to the server along with the user query (e.g., near-in-time requests, user profile information, date, time, etc.). In some examples, transmitting the user query to the server at block 566 can be done prior to providing a response at block 564 (e.g., the user query can be transmitted to a server, the server can indicate that no trained response is available, and a corresponding response can then be provided). In other examples, a user query can be transmitted to a first server for processing to determine whether a trained response is available, and can be transmitted to a second server upon determination that no trained response is available.
At block 568, the same user query can again be received at the user device. As at block 562, the user query can be received in any of a variety of ways and can again be directed to a virtual assistant associated with the device. At block 570, in contrast to the response at 564, a trained response to the user query can be provided (e.g., a response that appropriately/substantively responds to the user query, provides the requested information, performs the desired function, etc.). In some examples, the trained response to the user query can be provided by a server associated with the virtual assistant. In some instances, the virtual assistant can learn such a trained response according to process 220 of FIG. 2 after receiving the user query at block 562 and transmitting the user query to a server at block 566.
In some examples, the steps at blocks 572, 574, and 576 can be performed to train the virtual assistant to recognize and provide a response to the user query received at block 562, such that the virtual assistant is able to provide a trained response at block 570 after receiving the same user query again at block 568. At block 572, a plurality of request templates can be generated based on the user query received at block 562. For example, the plurality of request templates can be generated in a similar manner as described above with reference to block 230 of process 220 in FIG. 2. At block 574, a count can be maintained of the number of times each request template is received. For example, such a count can be maintained in a similar manner as described above with reference to blocks 232, 234, and 236 of process 220 in FIG. 2. At block 576, a trained response can be generated for the user query when an associated count reaches a predetermined amount. For example, a trained response can be generated and incorporated into a virtual assistant in a similar manner as described above with reference to blocks 238 and 240 of FIG. 2. In other examples, a notification can be generated indicating that an associated count has reached a threshold level, and a system administrator, user, or the like can provide an appropriate response that can be used to train the virtual assistant to respond to the corresponding query.
One or more of the functions described above relating to virtual assistant request recognition can be performed by a system similar or identical to system 600 shown in FIG. 6. System 600 can include instructions stored in a non-transitory computer readable storage medium, such as memory 603 or storage device 601, and executed by processor 605. The instructions can also be stored and/or transported within any non-transitory computer readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a “non-transitory computer readable storage medium” can be any medium that can contain or store the program for use by or in connection with the instruction execution system, apparatus, or device. The non-transitory computer readable storage medium can include, but is not limited to, a semiconductor system, apparatus, or device, a portable computer diskette (magnetic), a random access memory (RAM) (magnetic), a read-only memory (ROM) (magnetic), an erasable programmable read-only memory (EPROM) (magnetic), a portable optical disc such as CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW, or flash memory such as compact flash cards, secured digital cards, USB memory devices, memory sticks, and the like.
The instructions can also be propagated within any transport medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a “transport medium” can be any medium that can communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The transport medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, or infrared wired or wireless propagation medium.
System 600 can further include touch sensitive display 607 coupled to processor 605. Touch sensitive display 607 can be configured for users to interact with a virtual assistant along with other components (e.g., a microphone).
It is to be understood that the system is not limited to the components and configuration of FIG. 6, but can include other or additional components in multiple configurations according to various examples. Additionally, the components of system 600 can be included within a single device, or can be distributed between multiple devices. In some examples, processor 605 can be located within touch sensitive display 607.
FIG. 7 illustrates an exemplary personal device 700, such as a tablet, that can be configured to provide a virtual assistant interface according to various examples.
FIG. 8 illustrates another exemplary personal device 800, such as a mobile phone, that can be configured to provide a virtual assistant interface according to various examples.
Therefore, according to the above, some examples of the disclosure are directed to a method for request recognition for a virtual assistant, the method comprising: receiving a textual representation of user speech; generating a plurality of request templates based on the textual representation; associating a count with a request template of the plurality of request templates based on a number of times the request template is received; and in response to a determination that the count is more than a predetermined amount, generating a notification including the request template. Additionally or alternatively to one or more of the examples disclosed above, in some examples a method for request recognition for a virtual assistant can further comprise: in response to a determination that the count is more than the predetermined amount, training a language model with the request template, the language model associated with the virtual assistant. Additionally or alternatively to one or more of the examples disclosed above, in some examples a method for request recognition for a virtual assistant can further comprise: in response to a determination that the count is more than the predetermined amount, generating a plurality of populated templates, each of the plurality of populated templates comprising one of the plurality of generated request templates populated with an entity. Additionally or alternatively to one or more of the examples disclosed above, in some examples a method for request recognition for a virtual assistant can further comprise: training a language model with the plurality of populated templates. Additionally or alternatively to one or more of the examples disclosed above, in some examples each of the generated plurality of request templates comprises at least one word or phrase that indicates a language domain and at least one entity related to the language domain. Additionally or alternatively to one or more of the examples disclosed above, in some examples a method for request recognition for a virtual assistant can further comprise: receiving contextual data relating to the textual representation of user speech; wherein the plurality of request templates is generated based on the textual representation and the contextual data relating to the textual representation.
According to the above, other examples of the disclosure are directed to a system for request recognition for a virtual assistant, the system comprising: a memory; and a processor capable of: receiving a textual representation of user speech; generating a plurality of request templates based on the textual representation; associating a count with a request template of the plurality of request templates based on a number of times the request template is received; and in response to a determination that the count is more than a predetermined amount, generating a notification including the request template.
According to the above, other examples of the disclosure are directed to a method for facilitating user interactions with a virtual assistant associated with a user device, the method comprising: receiving a first user query at the user device; providing a response indicating that the virtual assistant is untrained to respond to the first user query; transmitting the first user query to a server associated with the virtual assistant; receiving a second user query at the user device, wherein the second user query is the same as the first user query; and in response to receiving the second user query, providing a trained response. Additionally or alternatively to one or more of the examples disclosed above, in some examples the trained response is determined by: generating a plurality of request templates based on the first user query; associating a count with a request template of the plurality of request templates based on a number of times the request template is received; and in response to a determination that the count is more than a predetermined amount, generating the trained response. Additionally or alternatively to one or more of the examples disclosed above, in some examples the trained response comprises a search result associated with the second user query.
According to the above, other examples of the disclosure are directed to a method for request recognition for a virtual assistant, the method comprising: receiving data comprising a reference to a future event; extracting an entity name from the received data, wherein the entity name describes the future event; generating a plurality of populated request templates based on the extracted entity name; and training a language model of the virtual assistant with the plurality of populated request templates. Additionally or alternatively to one or more of the examples disclosed above, in some examples generating the plurality of populated request templates based on the extracted entity name comprises: inserting the extracted entity name into a plurality of request templates recognizable by the virtual assistant. Additionally or alternatively to one or more of the examples disclosed above, in some examples a method for request recognition further comprises: inserting the extracted entity name into a first request template recognizable by the virtual assistant; receiving a second request template as yet unrecognized by the virtual assistant; and generating a candidate populated request template based on the extracted entity name by inserting the extracted entity name into the second request template. Additionally or alternatively to one or more of the examples disclosed above, in some examples a method for request recognition further comprises: training the language model of the virtual assistant with the candidate populated request template.
According to the above, other examples of the disclosure are directed to a method for request recognition for a virtual assistant, the method comprising: receiving a textual transcription of user speech; comparing the textual transcription to one or more first request templates recognizable by the virtual assistant; in response to a first match between the textual transcription and the first request template being found, causing an action to be performed corresponding to the first match; and in response to no first match being found: generating a plurality of second request templates based on the textual transcription; comparing each of the plurality of second request templates to one or more third request templates as yet unrecognized by the virtual assistant; incrementing a count associated with a second request template in response to a second match between the second request template and a third request template being found; and storing a second request template in response to a matching third request template not being found. Additionally or alternatively to one or more of the examples disclosed above, in some examples a method for request recognition further comprises: in response to a determination that the incremented count is more than a predetermined threshold, generating a notification including the second request template. Additionally or alternatively to one or more of the examples disclosed above, in some examples a method for request recognition further comprises: in response to no first match being found, training a language model with at least one of the plurality of generated second request templates. Additionally or alternatively to one or more of the examples disclosed above, in some examples a method for request recognition further comprises: in response to no first match being found, generating a plurality of populated request templates, each of the plurality of populated request templates comprising one of the plurality of generated second request templates populated with an entity; and training a language model with the plurality of populated request templates. Additionally or alternatively to one or more of the examples disclosed above, in some examples the action to be performed comprises causing a search result associated with the first match to be displayed. Additionally or alternatively to one or more of the examples disclosed above, in some examples each of the plurality of second request templates comprises at least one word or phrase that indicates a language domain. Additionally or alternatively to one or more of the examples disclosed above, in some examples a method for request recognition further comprises: receiving contextual data relating to the textual transcription of user speech; wherein the plurality of second request templates is generated based on the textual transcription and the contextual data relating to the textual transcription.
According to the above, other examples of the disclosure are directed to a system for request recognition for a virtual assistant, the system comprising: a memory; and a processor capable of: receiving a textual transcription of user speech; comparing the textual transcription to one or more first request templates recognizable by the virtual assistant; in response to a first match between the textual transcription and the first request template being found, causing an action to be performed corresponding to the first match; and in response to no first match being found: generating a plurality of second request templates based on the textual transcription; comparing each of the plurality of second request templates to one or more third request templates as yet unrecognized by the virtual assistant; incrementing a count associated with a second request template in response to a second match between the second request template and a third request template being found; and storing a second request template in response to a matching third request template not being found. Additionally or alternatively to one or more of the examples disclosed above, in some examples the processor is further capable of: in response to a determination that the incremented count is more than a predetermined threshold, generating a notification including the second request template. Additionally or alternatively to one or more of the examples disclosed above, in some examples the processor is further capable of: in response to no first match being found, training a language model with at least one of the plurality of generated second request templates. Additionally or alternatively to one or more of the examples disclosed above, in some examples the processor is further capable of: in response to no first match being found, generating a plurality of populated request templates, each of the plurality of populated request templates comprising one of the plurality of generated second request templates populated with an entity; and training a language model with the plurality of populated request templates.
Although examples have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the various examples as defined by the appended claims.

Claims

What is claimed is:

1. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for:

receiving a first user query at the user device;

providing a response indicating that the virtual assistant is untrained to respond to the first user query;

transmitting the first user query to a server associated with the virtual assistant;

receiving a second user query at the user device, wherein the second user query and the first user query are separate user queries, and wherein the second user query is substantially the same in substance as the first user query; and

in response to receiving the second user query, providing a trained response.

2. The non-transitory computer-readable storage medium of claim 1, wherein the trained response is determined by:

generating a plurality of request templates based on the first user query;

associating a count with a request template of the plurality of request templates based on a number of times the request template is received; and

in response to a determination that the count is more than a predetermined amount, generating the trained response.

3. The non-transitory computer-readable storage medium of claim 1, wherein the trained response comprises a search result associated with the second user query.

4. The non-transitory computer-readable storage medium of claim 1, the one or more programs further including instructions for:

transmitting contextual information related to the user query to the server.

5. The non-transitory computer-readable storage medium of claim 1, wherein the first user query is transmitted to the server prior to providing a response indicating that the virtual assistant is untrained to respond to the first user query.

6. The non-transitory computer-readable storage medium of claim 5, wherein an indication the virtual assistant is untrained to respond to the first user query is received from the server.

7. The non-transitory computer-readable storage medium of claim 1, wherein the second user query is transmitted to the server.

8. The non-transitory computer-readable storage medium of claim 1, wherein the trained response is received from the server.

9. The non-transitory computer-readable storage medium of claim 8, wherein the trained response is received from the server prior to receiving the second user query.

10. An electronic device comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving a first user query at the user device;

in response to receiving the second user query, providing a trained response.

11. The electronic device of claim 10, wherein the trained response is determined by:

generating a plurality of request templates based on the first user query;

12. The electronic device of claim 10, wherein the trained response comprises a search result associated with the second user query.

13. The electronic device of claim 10, the one or more programs further including instructions for:

transmitting contextual information related to the user query to the server.

14. The electronic device of claim 10, wherein the first user query is transmitted to the server prior to providing a response indicating that the virtual assistant is untrained to respond to the first user query.

15. The electronic device of claim 14, wherein an indication the virtual assistant is untrained to respond to the first user query is received from the server.

16. The electronic device of claim 10, wherein the second user query is transmitted to the server.

17. The electronic device of claim 10, wherein the trained response is received from the server.

18. The electronic device of claim 17, wherein the trained response is received from the server prior to receiving the second user query.

19. A method for facilitating user interactions with a virtual assistant associated with a user device, the method comprising:

receiving a first user query at the user device;

in response to receiving the second user query, providing a trained response.

20. The method of claim 19, wherein the trained response is determined by:

generating a plurality of request templates based on the first user query;

21. The method of claim 19, wherein the trained response comprises a search result associated with the second user query.

22. The method of claim 19, further comprising:

transmitting contextual information related to the user query to the server.

23. The method of claim 19, wherein the first user query is transmitted to the server prior to providing a response indicating that the virtual assistant is untrained to respond to the first user query.

24. The method of claim 23, wherein an indication the virtual assistant is untrained to respond to the first user query is received from the server.

25. The method of claim 19, wherein the second user query is transmitted to the server.

26. The method of claim 19, wherein the trained response is received from the server.

27. The method of claim 26, wherein the trained response is received from the server prior to receiving the second user query.