[go: up one dir, main page]

WO2019060468A1 - Systèmes et procédés de traitement et de classification en langage naturel - Google Patents

Systèmes et procédés de traitement et de classification en langage naturel Download PDF

Info

Publication number
WO2019060468A1
WO2019060468A1 PCT/US2018/051827 US2018051827W WO2019060468A1 WO 2019060468 A1 WO2019060468 A1 WO 2019060468A1 US 2018051827 W US2018051827 W US 2018051827W WO 2019060468 A1 WO2019060468 A1 WO 2019060468A1
Authority
WO
WIPO (PCT)
Prior art keywords
message
training
model
models
campaign
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2018/051827
Other languages
English (en)
Inventor
Alex Terry
Werner KOEPF
James Harriger
Will Webb-Purkis
Joseph M. SILVERBEARS
Macgregor S. GAINOR
Ryan Ginstrom
Siddhartha Reddy JONNALAGADDA
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Conversica Inc
Original Assignee
Conversica Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/019,382 external-priority patent/US11301632B2/en
Application filed by Conversica Inc filed Critical Conversica Inc
Publication of WO2019060468A1 publication Critical patent/WO2019060468A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/01—Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology

Definitions

  • the present invention relates to systems and methods for improved natural language processing and classification of documents, primarily in the context of the generation and management of a dynamic messaging campaign.
  • Such systems and methods provide marketers and sales people more efficient tools for client management and outreach, and also improve computer functioning as it relates to processing documents for meaning.
  • Such system and methods enable more productive business conversations.
  • An example scenario discussed is sales where the system enhances sales activity, increased profits, and more efficient allocation of sales resources.
  • the systems and methods include a communication editor dashboard which receives the message, performs natural language processing to divide the message into component parts. Each component part has a discrete meaning.
  • the message is analyzed for sentiment, insights for the component parts, confidence for the insights, and generation of actions based on these insights.
  • the system displays the message in a first pane with each of the component parts overlaid with a different color, and displaying in a second pane the insights, the confidence scores associated with each insight, the sentiment and the actions.
  • the systems and methods include combining outputs from multiple machine learned AI models into a unified output.
  • This includes receiving feedback from multiple machine learned AI models.
  • the feedback is a classification for a message processed using natural language processing.
  • a scenario is also received for the message.
  • a credibility score for each of the machine learned AI models is determined according to the scenario in a credibility matrix.
  • a weight is applied to each of the machine learned AI models based on the credibility score. The scores are then averaged using these weights to generate the unified output.
  • the systems and methods include responding to simple question using natural language processing. This includes receiving, from a campaign manager, a set of training questions linked to facts answering the associated training question. The facts are stored in a third party database. Each of the training questions is linked to a fundamental question category. A concept for a message is received from the machine learning AI model. The machine learning AI model utilize a dataset independent from the third party database. The concept for the message is referenced to the fundamental question categories, which causes the fact associated with the fundamental question category that the message concept references to be retrieved from the third party database. A response template for the fundamental question category that the message concept references to is also retrieved from the independent dataset, and populated with the fact. This final response can then be outputted. [0012] Note that the various features of the present invention described above may be practiced alone or in combination. These and other features of the present invention will be described in more detail below in the detailed description of the invention and in conjunction with the following figures.
  • Figure 1 is an example logical diagram of a system for generation and implementation of messaging campaigns, in accordance with some embodiment
  • Figure 2 is an example logical diagram of a dynamic messaging server, in accordance with some embodiment
  • Figure 3 is an example logical diagram of a user interface within the dynamic messaging server, in accordance with some embodiment
  • Figure 4 is an example logical diagram of a message generator within the dynamic messaging server, in accordance with some embodiment
  • Figure 5A is an example logical diagram of a message response system within the dynamic messaging server, in accordance with some embodiment
  • Figure 5B is an example logical diagram of a classification engine within the message response system, in accordance with some embodiment
  • Figure 6 is an example flow diagram for a dynamic message campaign, in accordance with some embodiment
  • Figure 7 is an example flow diagram for the process of on-boarding a business actor, in accordance with some embodiment
  • Figure 8 is an example flow diagram for the process of building a business activity such as campaign, in accordance with some embodiment
  • Figure 9 is an example flow diagram for the process of generating message templates, in accordance with some embodiment.
  • Figure 10 is an example flow diagram for the process of implementing the campaign, in accordance with some embodiment;
  • Figure 11 is an example flow diagram for the process of preparing and sending the outgoing message, in accordance with some embodiment
  • Figure 12 is an example flow diagram for the process of processing received responses, in accordance with some embodiment
  • Figure 13 is an example flow diagram for the process of AI classification, in accordance with some embodiment
  • Figure 14 is an example flow diagram for the process of AI council reconciliation, in accordance with some embodiment
  • Figure 15 is an example flow diagram for the process of populating a robust actor profile for response tailoring, in accordance with some embodiment
  • Figure 16 is an example flow diagram for the process of generating a transparency interface, in accordance with some embodiment
  • Figure 17 is an example illustration of a possible message exchange between the message response system and a lead, in accordance with some embodiment
  • Figure 18 is an example illustration of the message being overlaid with transparency labels, in accordance with some embodiment
  • Figure 19 is an example illustration of a transparency output information associated with the message exchange between the message response system and a lead, in accordance with some embodiment
  • Figure 20 is an example flow diagram for the process of analyzing non-text message elements, in accordance with some embodiment
  • Figure 21 is an example flow diagram for the process of simple question response, in accordance with some embodiment.
  • Figure 22 is an example flow diagram for the process of populating and utilizing a decision tree tool for complex classification efforts, in accordance with some embodiment.
  • Figures 23A and 23B are example illustrations of a computer system capable of embodying the current invention. DETAILED DESCRIPTION
  • the present invention relates to cooperation between business actors such as human operators and AI systems. While such systems and methods may be utilized with any AI system, such cooperation systems particularly excel in AI systems relating to the generation of automated messaging for marketing and other sales functions. While the following disclosure is applicable for other combinations, we will focus upon mechanisms of cooperation between human operators and AI marketing systems as an example, to demonstrate the context within which the cooperation system excels.
  • the following systems and methods for dynamic messaging a campaign relies upon an interplay of user interaction, and sophisticates artificial intelligence (AI) processing of received messages.
  • AI artificial intelligence
  • the goal of the message campaign it to enable a logical dialog exchange with a recipient, where the recipient is not necessarily aware that they are communicating with an automated machine as opposed to a human actor. This may be most efficiently performed via a written dialog, such as email, text messaging, chat, etc. However, it is entirely possible that given advancement in audio and video processing, it may be entirely possible to have the dialog include audio or video components as well.
  • an AI system is employed within an AI platform within the messaging system to process the responses and generate conclusions regarding the exchange.
  • These conclusions include calculating the context of a document, insights, sentiment and confidence for the conclusions. Given that these terms are not readily familiar outside of the field of natural language processing, a series of definitions are provided to clarify the terminology:
  • accuracy the quantitative evaluation of how reliable and comprehensive the system classifications are. We formally define it as the sum of all instances of inputs where the system classification is correct as annotated by a business actor (sales person, linguistic annotator, etc.) divided by the total number of instances.
  • (AI) algorithm - a method employed to calculate the weight of a document in a particular category.
  • aspect - a specific AI algorithm.
  • AI Trainer term for the tool used to classify a document that the aspects were not confident scoring.
  • campaign a set of possible messaging designed to be sent out to a lead over the course of a conversation depending on the receipt and classification of responses (or lack thereof).
  • categorization the process in which ideas and objects are recognized, differentiated, and understood, generally into categories.
  • classification another word for categorization.
  • context a collection of documents that have some commonality.
  • document - a set of words in a specific order used to convey a meaning.
  • Hard-rule - an AI algorithm that dictates a category based on a heuristic match involving tokens, regular expression patterns and discourse-level intents and entity classifications.
  • Hard-rule term an example of a string in the response that is used by the Hard-rule aspect as part of the overall heuristics.
  • insight a collection of categories used to answer some question about a document. Example: “What does this person mean?”, “How does this person feel?”, “Should we keep emailing this person?”
  • knowledge set - a set of tokens with their associated category weights used by an aspect during classification.
  • actor - a person or another entity like an AI system that represents a person who is placed into the system at a certain time for a certain purpose.
  • a lead is placed in our system using for sales conversion under a predefined campaign
  • Alternate terms for 'actor' may include "lead”, “contact”, “business actor” or the like depending upon use case.
  • actor (event) history the notable information for an actor coming into the system, messages sent to that actor, responses received and alerts sent out, in the chronological order of their occurrences.
  • ngram - denotes the number of words used to make a token.
  • token "yes it is” is a tri-gram or an ngram of 3.
  • normalization - removing characters / tokens to reduce the complexity of the document without changing the accuracy of classifications.
  • question - an inquiry included in a message designed to limit the response to a subset of the target language.
  • sentiment classifier - an AI algorithm that is used to gauge how strongly a category expresses itself in a document.
  • series - a subset of a campaign designed to be sent out until a response is received for that subset of messages. Based on the classification of the response, the system may continue to another series of messaging in that same campaign.
  • score - a set of probabilities or a positive number between 0 and 1 associated with the set of classifications made by the different aspects for different insights.
  • the (AI) Platform the system that allows interaction with, setup, score, and modify the AI algorithms as need be. This also includes the code, databases and servers used for this specific purpose.
  • training set - a set of classified documents used to calculate knowledge sets.
  • word - a combination of characters used to denote meaning in a language.
  • FIG. 1 is an example logical diagram of a system for generating and implementing messaging campaigns, shown generally at 100.
  • several users 102a-n are illustrated engaging a dynamic messaging system 108 via a network 106.
  • messaging campaigns may be uniquely customized by each user 102a-n in some embodiments.
  • users may be part of collaborative sales departments (or other collaborative group) and may all have common access to the messaging campaigns.
  • the users 102a-n may access the network from any number of suitable devices, such as laptop and desktop computers, work stations, mobile devices, media centers, etc.
  • the network 106 most typically includes the internet, but may also include other networks such as a corporate WAN, cellular network, corporate local area network, or combination thereof, for example.
  • the messaging server 108 may distribute the generated messages to the various message delivery platforms 112 for delivery to the individual recipients.
  • the message delivery platforms 1 12 may include any suitable messaging platform. Much of the present disclosure will focus on email messaging, and in such embodiments the message delivery platforms 1 12 may include email servers (Gmail, yahoo, Hotmail, etc.). However, it should be realized that the presently disclosed systems for messaging are not necessarily limited to email messaging. Indeed, any messaging type is possible under some embodiments of the present messaging system.
  • the message delivery platforms 1 12 could easily include a social network interface, instant messaging system, text messaging (SMS) platforms, or even audio telecommunications systems. While audio is possible with the given messaging system, it is often desirable for the recipient to have a seamless experience where the automated messages are virtually indistinguishable from messages authored by a sales associate. Due to inherent difficulties in generating realistically human sounding automated audio (much less imitating a specific sales associate), much of the present disclosure will focus on the generation of written textual messages.
  • One or more data sources 1 10 may be available to the messaging server 108 to provide user specific information, message template data, knowledge sets, insights, and lead information. These data sources may be internal sources for the system's utilization, or may include external third-party data sources (such as business information belonging to a customer for whom the campaign is being generated). These information types will be described in greater detail below.
  • FIG. 2 provides a more detailed view of the dynamic messaging server 108, in accordance with some embodiment.
  • the server is comprised of three main logical subsystems: a user interface 210, a message generator 220, and a message response system 230.
  • the user interface 210 may be utilized to access the message generator 220 and the message response system 230 to set up messaging campaigns, and manage those campaigns throughout their life cycle.
  • the user interface 210 includes APIs to allow a user's device to access these subsystems.
  • the user interface 210 may include web accessible messaging creation and management tools, as will be explored below in some of the accompanying example screenshots.
  • FIG. 3 provides a more detailed illustration of the user interface 210.
  • the user interface 210 includes a series of modules to enable the previously mentioned functions to be carried out in the message generator 220 and the message response system 230. These modules include a campaign builder 310, a campaign manager 320 an AI manager 330, an insight manager 340, and a knowledge base manager 350.
  • the campaign builder 310 allows the user to define a campaign, and input message templates for each series within the campaign.
  • a knowledge set and lead data may be associated with the campaign to allow the system to automatically effectuate the campaign once built.
  • Lead data includes all the information collected on the intended recipients, and the knowledge set includes a database from which the AI can infer context and perform classifications on the responses received from the recipients.
  • the campaign manager 320 provides activity information, status, and logs of the campaign once it has been implemented. This allows the user 102a to keep track of the campaign's progress, success and allows the user to manually intercede if required.
  • the campaign may likewise be edited or otherwise altered using the campaign manager 320.
  • the AI manager 330 allows the user to access the training of the artificial intelligence which analyzes responses received from a recipient.
  • One purpose of the given systems and methods is to allow very high throughput of message exchanges with the recipient with relatively minimal user input. To perform this correctly, natural language processing by the AI is required, and the AI (or multiple AI models) must be correctly trained to make the appropriate inferences and classifications of the response message.
  • the user may leverage the AI manager 330 to review documents the AI has processed and has made classifications for.
  • the insight manager 340 allows the user to manage insights.
  • insights are a collection of categories used to answer some question about a document. For example, a question for the document could include "is the lead looking to purchase a car in the next month?" Answering this question can have direct and significant importance to a car dealership. Certain categories that the AI system generates may be relevant toward the determination of this question. These categories are the 'insight' to the question, and may be edited or newly created via the insight manager 340.
  • the knowledge base manager 350 enables the management of knowledge sets by the user.
  • a knowledge set is set of tokens with their associated category weights used by an aspect (AI algorithm) during classification. For example, a category may include “continue contact?", and associated knowledge set tokens could include statements such as "stop", “do no contact", "please respond” and the like.
  • the knowledge base manager 350 enables the user to build new knowledge sets, or edit exiting ones.
  • the message generator 220 utilizes context knowledge 440 and lead data 450 to generate the initial message.
  • the message generator 220 includes a rule builder 410 which allows the user to define rules for the messages.
  • a rule creation interface which allows users to define a variable to check in a situation and then alter the data in a specific way. For example, when receiving the scores from the AI, if the insight is Interpretation and the chosen category is 'good', then have the Continue Messaging insight return 'continue'.
  • the rule builder 410 may provide possible phrases for the message based upon available lead data.
  • the message builder 420 incorporates those possible phrases into a message template, where variables are designated, to generate the outgoing message. This is provided to the message sender 430 which formats the outgoing message and provides it to the messaging platforms for delivery to the appropriate recipient.
  • Figure 5A is an example logical diagram of the message response system 230.
  • the contextual knowledge base 440 is utilized in combination with response data 560 received from the lead.
  • the message receiver 520 receives the response data 560 and provides it to the AI interface 510, objective modeler 530, and classifier engine 550 for feedback.
  • the AI interface 510 allows the AI platform (or multiple AI models) to process the response for context, insights, sentiments and associated confidence scores.
  • classification engine 550 includes a suite of tools that enable better classification of the messages. Based on the classifications generated by the AI and classification engine 550 tools lead objectives may be updated by the objective modeler 530. [0089] The message receiver 520 can then determine whether there are further objectives that are still pending, or whether there has been a request to discontinue messaging the lead. If there has been a termination request, or if all objectives have been fulfilled, the message receiver may deactivate the campaign for the given lead. If not, a scheduler 540 may be employed to assist in scheduling the next step of the campaign.
  • This component of the message response system may include modules for the generation of user profiles 551.
  • the user profile generator 551 used information gained from the user's communications to append a profile with information related to education levels, language fluency, sentiments and behavioral drivers. Language fluency and education level may directly impact the language and sentence complexity used to communicate with the user.
  • Sentiment and behavioral driver profiles may be utilized to select from different messaging sentiments based upon likelihood of success at meeting the objectives. Examples of one framework for utilizing behavioral drivers include identifying a dominant function for the lead. Functions traditionally are thought of as the elements of a personality type, and these include sensation, intuition, thinking and feeling.
  • a sensation focused lead may react well to messages that include details and concrete facts.
  • intuition focused leads may be more influenced by abstractions and theory based reasoning.
  • feeling individuals tend to be influenced by more rational arguments, whereas feeling individuals may be more heavily influenced by empathy and needs of people involved.
  • a message campaign with a social justice element may be more effective for a 'feeling' focused lead, whereas a lead focused on sensation may react best to a message with details regarding a sale.
  • An additional component of the classification engine 550 is the AI council (a set of business rules, another ML model that takes the outputs of other models, or a combination of these) 552, which directly interfaces with more than one AI models that each independently reviews an incoming document for insights, classification and sentiments.
  • AI council a set of business rules, another ML model that takes the outputs of other models, or a combination of these
  • a unified AI model would be employed to analyze a given document as having non-redundant processing of the message results in reduced processing demands and hence lower system overhead.
  • the present system may employ an ensemble (combination through simple voting, weighted voting and other approaches) of AI models to analyze the document in parallel. This often results in differences in classification, sentiment scores, and confidence levels.
  • the impact on the final score may be minimal, but sometimes these differences may become material, and the AI council 552 may be utilized to reconcile these scores.
  • At the heart of the AI counsel 552 is a weighted average of the outputs from the different AI models.
  • the AI counsel 552 is unique in the manner that the different weights are calculated. These weights may vary over time, and may be situationally dependent.
  • the AI council 552 may perform a lookup on historical accuracy levels for the given models, and weight them accordingly.
  • This accuracy table may include a number of entries that are situationally dependent. For example, the default situation may average three models, in one example. Model A is accurate 87% of the time, model B is accurate 95% of the time, and model C is accurate 70% of the time.
  • the weights applied to these models would be 0.3, 0.6 and 0.1 respectively.
  • these weights are driven by historical data on which model might be more accurate and confident for particular conversations and/or industries and are user configured, in this example, but could be based upon the standard deviation of the accuracy levels or use the outputs of subordinate models as inputs to higher-level ML models, which may use any number of statistical means to determine the best output based on the given inputs.
  • the standard deviation is 10.42.
  • model A and model B are within one standard deviation of one another, whereas model C is a full four standard deviations from the highest model accuracy. Obviously, with this example of three models, determining standard deviation is prone to exaggeration due to the low number of samples.
  • x is the accuracy of model i
  • " is the mean of the model accuracies
  • s x is the standard deviation of the model accuracies
  • N is the number of models utilized
  • C is a configurable constant.
  • C may be set to a value of 3.
  • Lower values for constant C increase the degree of impact the difference between the accuracy and the mean influences the weight, whereas larger values for constant C decrease the differences in weights afforded the models.
  • the model accuracies are recorded in the accuracy table as 87%, 95% and 70%.
  • these accuracy values may also be recorded in the table differently given different situational factors. For example, if another language is utilized, it is possible alternate models may become more accurate.
  • the accuracies for models A, B and C respectively may change to 85%, 80% and 90%.
  • the models may again score differently: 92%, 85% and 70% for example.
  • Other situational factors that may be incorporated into the accuracy tables for models may also include industry segment, nature of the conversation/campaign type, lead personality type, campaign vehicle (e.g., text message versus emails) and any other relevant factor.
  • FAQAA "Frequently Asked Questions and Approved Answers” module
  • the system will elevate the campaign to a human user.
  • questions a lead may ask which are considered “simple” or "routine" that the system may be designed to handle without human intervention.
  • These include questions such as store hours, presence of merchandise at a given store location, business location, extensions, etc.
  • These questions may be generated by the campaign manager and input into the simple question handler 553 as a training set.
  • the campaign manager then links the training question to customer specific information located in a third-party database belonging to the customer. For example, if the customer building campaigns is a car dealership, the campaign manager may input a training question of "Do you have a [brand,make,model] in stock?".
  • the manager may then couple the message response system, which is implemented on its own servers, to the car dealership inventory database through the network.
  • the system parses the training questions into n-grams, and applies a synonym database to each n-gram of the training question. This results in the generation of many hundreds or even thousands of permutations of the training questions. Given the above example, the following would all be generated by the synonym replacement: "Do you a [brand,make,model] in inventory?”, "Is there a [brand,make,model] in stock?”, "Do you have a [brand,make,model] in store?”, "Do you keep a
  • These multitudes of training questions may be used by the machine learning AI to model the simple questions and generate a result including a feature/template answer populated with data derived from the third-party database.
  • the system transparency handler 554 includes a user interface populated with a number of features that allows a user to see and modify a number of details regarding the AI performance and function. These may include information regarding upstream nodes, primary questions being asked at any given node, lists of actions the AI or operator can take at a given node, insights, usage volumes, and performance metrics for the given node.
  • insights includes, triggering events that the AI relies upon before performing a particular action. These insights may vary from one node to another.
  • Performance metrics may include information for the node related to the percentage of the time the node result in a handoff to a human operator (expressing a confidence of the AI at the given node), the percent audit occurred (expressing an error percentage when the AI thought it was indeed confident), and an percent audit for the human operator (expressing the "difficulty" of delivering an appropriate response at the node).
  • the system transparency module 554 provides a graphical display to the campaign manager that allows them to better understand what the system is actually doing, rather than appearing as a "black box" where messages responses from a lead go in and return messages come back out. By allowing the campaign manager to "peek under the hood" of the system, the campaign manager can gain confidence in the systems functioning, and when an error occurs, enables the campaign manager to better identify the breakdown.
  • the output of the transparency module 554 may include an overlay of the insights gained from a leas message. If one insight is grossly incorrect, the campaign manager can determine which portion of the message was relied upon to make the erroneous insight, and the model may be updated to prevent this sort of error.
  • embodiments of the classification engine 550 may also include non-text analyzers 555 that take in non-textual information and generate outputs.
  • non-text analyzers 555 that take in non-textual information and generate outputs.
  • the inputs that are non-textual are often embedded emoticons/emoji, or embedded images or occasionally movies.
  • Emoticons in particular have proliferated within text and email messaging, and are used almost exclusively to convey an emotion. This information is extremely helpful for the calculation of the lead's sentiment, and the non-text analyzer 555 is able to leverage these elements to more accurately classify a response.
  • Emoticon analysis may include a basic table lookup of emoticons against sentiment values and/or insights.
  • the "clapping hands” emoticon and variants of the "smiley face” may all be used to modify the sentiment score to be more positive.
  • a “thumbs up” emoticon may be construed as being an affirmation statement.
  • a “frowny face” or “mad face” may be utilized to downgrade the sentiment score.
  • Video information may have the audio components isolated and analyzed in much the same manner as text. This analysis may additionally include the incorporation of third party data regarding source and/or content/context/meaning of the video file or image.
  • the video may be also analyzed for ambient color and brightness, which are often associated with a mood or sentiment. Images may also be analyzed for any embedded text, facial information, etc.
  • Third party applications can be leveraged to extend the power of multimedia classification. Additional video, audio or text classifiers can be integrated to response- analysis to further inform our Inference Engine downstream.
  • the system may be configured to ignore images embedded in signature lines however, since these are frequently used as an automatic feature and provide the system with minimal additional information.
  • the final element of the classification engine 550 is a tree builder 556 which is an internal tool to enable system configuration.
  • the Tree builder 556 is used by an admin user to build a classification tree for each state in a conversation.
  • the decision tree may have a knowledge base that is different from the classifiers used previously, as well as different thresholds required for the classification.
  • Decision trees enable the integration of machine-leaming with business logic.
  • Business decisions are applied based on classifier results and confidence thresholds.
  • An admin user can use the tree-builder to select classifiers to associate with nodes in a conversation. The user can click on nodes to add, edit or delete a classifier from the tree. For each node in a tree, the user can apply a confidence threshold. To ensure quality, an admin user can test a tree's performance against a validation set before publishing.
  • the decision tree enables developers to integrate the machine learning with business logic.
  • Business decisions may be configured to be based on classifier results and confidence thresholds.
  • the decision tree is built after the machine learned model has been populated.
  • the developer selects classifiers to associate with nodes in the decision tree.
  • the developer may add, delete or select a node in the tree. For a selected node, the developer sets a confidence threshold. If the AI model is unable to achieve this confidence threshold, the class assigned is "uncertain" and the message is routed for disambiguation by a human operator.
  • Validation sets provide expected accuracy and confidence for the tree by comparing the tree's predicted classification against known answers.
  • the decision tree may produce a confident classification or initiate human review for "uncertain" situations.
  • a confident classification occurs when each node in the tree meets or exceeds the node's confidence threshold.
  • a single decision tree is capable of generating a single classification describing the final leaf node.
  • FIG. 6 an example flow diagram for a dynamic message campaign is provided, shown generally at 600.
  • the process can be broadly broken down into three portions: the on-boarding of a user (at 610), campaign generation (at 620) and campaign implementation (at 630).
  • the following figures and associated disclosure will delve deeper into the specifics of these given process steps.
  • Figure 7 provides a more detailed look into the on-boarding process, shown generally at 610.
  • a user is provided (or generates) a set of authentication credentials (at 710). This enables subsequent authentication of the user by any known methods of authentication. This may include username and password combinations, biometric identification, device credentials, etc.
  • the lead data associated with the user is imported, or otherwise aggregated, to provide the system with a lead database for message generation (at 720).
  • context knowledge data may be populated as it pertains to the user (at 730).
  • This may be as simple as selecting dashboard layouts, to configuring confidence thresholds required before alerting the user for manual intervention.
  • FIG. 8 is the example flow diagram for the process of building a campaign, shown generally at 620.
  • the user initiates the new campaign by first describing the campaign (at 810).
  • Campaign description includes providing a campaign name, description, industry selection, and service type.
  • the industry selection and service type may be utilized to ensure the proper knowledge sets are relied upon for the analysis of responses.
  • the message templates in the campaign are generated (at 820). If the series is populated (at 830), then the campaign is reviewed and submitted (at 840). Otherwise, the next message in the template is generated (at 820).
  • Figure 9 provides greater details of an example of this sub-process for generating message templates. Initially the user is queried if an existing campaign can be leveraged for templates, or whether a new template is desired (at 910).
  • the new message templates are generated by populating the templates with existing templates (at 920). The user is then afforded the opportunity to modify the message templates to better reflect the new campaign (at 930). Since the objectives of many campaigns may be similar, the user will tend to generate a library of campaign that may be reused, with or without modification, in some situations. Reusing campaigns has time saving advantages, when it is possible.
  • the user may opt to write the message templates from scratch using the Conversation Editor (at 940).
  • a message template is generated, the bulk of the message is written by the user, and variables are imported for regions of the message that will vary based upon the lead data.
  • Successful messages are designed to elicit responses that are readily classified. Higher classification accuracy enables the system to operate longer without user interference, which increases campaign efficiency and user workload.
  • Lead data is uploaded (at 1010).
  • Lead data may include any number of data types, but commonly includes lead names, contact information, date of contact, item the lead was interested in, etc.
  • Other data can include open comments that leads supplied to the lead provider, any items the lead may have to trade in, and the date the lead came into the lead provider's system.
  • lead data is specific to the industry, and individual users may have unique data that may be employed.
  • An appropriate delay period is allowed to elapse (at 1020) before the message is prepared and sent out (at 1030).
  • the waiting period is important so that the lead does not feel overly pressured, nor the user appears overly eager. Additionally, this delay more accurately mimics a human correspondence (rather than an instantaneous automated message).
  • Figure 1 1 provides a more detailed example of the message preparation and output.
  • the message within the series is selected based upon which objectives are outstanding (at 1 110).
  • the messages will be presented in a set order; however, if the objective for a particular lead has already been met for a given series, then another message may be more appropriate.
  • the recipient didn't respond as expected, or not at all, it may be desirous to have alternate message templates to address the lead most effectively.
  • the lead data is parsed through, and matches for the variable fields in the message templates are populated (at 1120).
  • the populated message is output to the appropriate messaging platform (at 1130), which as previously discussed typically includes an email service, but may also include SMS services, instant messages, social networks, or the like.
  • the process waits for a response (at 1040). If a response is not received (at 1050) the process determines if the wait has been timed out (at 1060). Allowing a lead to languish too long may result in missed opportunities; however, pestering the lead to frequently may have an adverse impact on the relationship. As such, this timeout period may be user defined. Often the timeout period varies from a few days to a week or more. If there has not been a timeout event, then the system continues to wait for a response (at 1050). However, once sufficient time has passed without a response, it may be desirous to return to the delay period (at 1020) and send a follow-up message (at 1030). Often there will be available reminder templates designed for just such a circumstance.
  • the process may continue with the response being processed (at 1070). This processing of the response is described in further detail in relation to Figure 12.
  • the response is initially received (at 1210) and the document may be cleaned (at 1220).
  • Document cleaning may include a
  • classification using the knowledge sets (at 1230). Classification is described in relation to Figure 13 in considerably more detail.
  • the system initially determines if the content being classified is text or not (at 1305).
  • non-textual content may be processed through a separate analysis for meaning and sentiment. The process for analyzing non-text information will be discussed further below.
  • Figure 14 discusses in greater detail this sub process for the AI council reconciling the inputs from multiple machine learned models.
  • the feedback from the various models is received (at 1410).
  • processing a document through multiple models in parallel has a processing cost; it simply requires more resources to process multiple algorithms compared to a single model.
  • the advantage gained through this technique in terms of improved accuracy of the computer's ability to classify the document makes the processing overhead cost worthwhile.
  • the system may look up accuracy scores for the models in a credibility score matrix (at 1420).
  • the credibility score matrix is populated, over time, by comparing classification outputs of a model against Audit Desk output where correct classifications are already known.
  • the credibility index includes a multitude of credibility scores for a given model, each based upon external factors. These may include situation type, industry, language used, and the personality type for a lead being analyzed. For example, responses to informational messages may be classified differently than responses to questions, and some models may be better at classification of questions versus another model.
  • the system may weight the models (at 1430). This weighting may include manual application of weights based upon accuracy values, or may employ a weighting algorithm that takes into account the standard deviation of the accuracy scores, such as provided as an example previously.
  • the AI council may perform a weighted average of the models' results to render a single classification (at 1440). This single interpretation score is then output for further processing (at 1450).
  • the system renders insights from the message (at 1330).
  • insights are categories used to answer some underlying question related to the document.
  • the classifications may map to a given insight based upon the context of the campaign message.
  • a confidence score is then generated for the insight (at 1340).
  • the AI algorithm assigns the confidence score for the classification.
  • Sentiment analysis is a determination of how strongly the category applies to the document. Sentiment may be computed using the same AI algorithms that were used for classifying insights. In this case, the classes would be the polarity (positive/negative) of opinion as expressed in the response in relation to an object (product being sold, for example).
  • the process updates the user's profile (at 1360) using the information garnered from the analysis of the documents received by the lead.
  • This process is described in greater detail in relation to Figure 15.
  • the analysis of the user's profile starts with the parsing of the language used into specific categories (at 1510). These categories include emotion, perception, complex cognitive processes, and function words. Function words include pronouns, articles, conjunctions and numerals. Words belonging to complex cognitive processes category include those indicating a causal relationship and those that convey vagueness and "balancing words" (such as 'except', 'but', 'however', etc.). These categories are further broken down into subcategories. For example, in the emotional category, is the emotion 'happy' or 'scared' .
  • a useful subcategory is between 'individual' versus 'group' (e.g., "I” and “me” versus “us” and “we”).
  • Tools such as Linguistic Inquiry and Word Count (LIWC), or the like, may be adapted to perform this parsing.
  • the process may compile character counts per category (at 1520) and determine the number of words in each of the categories (at 1530).
  • the complexity of the vocabulary may also be determined (at 1540) by comparing the words used by the lead against listings of vocabulary words by educational expectations.
  • a complexity score may also be generated based upon the number of discrete concepts expressed per character count and per sentence (at 1550). All this information may be used to estimate the user's education level and/or IQ.
  • vocabulary complexity maps very well to education achievement.
  • the presence of complex thought processes words, and the complexity score may map well to IQ level, and may also be indicative of education level.
  • an education level estimate may be generated by weighted averaging of the vocabulary complexity, computed complexity score and the presence of words indicative of complex though processes (such as balancing words). In some embodiments, each of these factors may be given equal weight.
  • the predictive measure of each of these factors may be compared against a control training set, where education level is known, and weights may be generated accordingly.
  • the weight of vocabulary usage is given greater weight, complexity score an intermediate weight, and complex though process word count a lower weight.
  • the person's language fluency is analyzed (at 1570). This process used vocabulary used, complexity score and semantic accuracy to identify what language the user is familiar with (based upon vocabulary use), and degree of fluency (based upon complexity and semantic accuracy). This is important, as a fluent user of a language many be more influenced by eloquent and complex messages, but a less fluent user may merely be frustrated, and thus simpler messaging is preferable.
  • the user's tone and emotional sentiment may next be determined (at 1580).
  • Sentiment in this context, relates to the disposition of the lead.
  • the emotional words used by the user are the primary clue as to user tone/sentiment.
  • function wording may also provide insight into the lead's sentiment. For example, users who use more "individual” language such as “I” and “me” tend to be more insecure, threatened and defensive, compared to greater usage of "group” wording ("us” and “we”). It should be noted that analysis of word usage can likewise shed light on a number of other traits of the lead. For example, males tend to use more articles, prepositions and big words than women, who are more likely to use pronouns and reference other people. Likewise, word usage may indicate lead age, as older individuals tend to refer to themselves less over time, and use more positive emotion words and fewer negative emotion words. Older individuals also use more future tense verbs and fewer past tense verbs. Honest is another factor that may be analyzed for. When telling the truth, people more often use first person singular pronouns, and also use words such as "except” and "but”.
  • the behavioral drivers for the individual may be analyzed for (at 1370).
  • These drivers may be personality traits, such as thinking, feeling, emotion, and perception, as discussed previously. Certain word frequency can indicate what personality type dominates a particular lead. For example, a lead with above average emotional category of words may have an 'emotional' personality type, whereas an individual which utilized a high number of prepositions and exclusionary words ("unless", “but”, “except”, etc.) may be more of a 'thinking' personality type.
  • behavioral drivers/personality types may be used to tailor campaign messages to the lead in a manner that is more likely to influence that individual.
  • Jungian personality types other behavioral drivers/personality schemes may likewise be leveraged.
  • Theme may include enneagram traits, and traditional behavioral economic drivers.
  • All of the information generated for the lead may be stored within a user profile for the lead. As additional messaging occurs, this information may be updated over time. This allows tracking of volatility to also be measured. This volatility may indicate accuracy of the profile. For example, if an individual is determined to be a high school graduate level education and subsequently a graduate level education, clearly there is a disconnect in the measurements. It could be that the user is in a different context and is choosing to write differently, or it may be that the system is operating using too little information to accurately determine the user's education level. Regardless, this volatility in the profile metric will indicate that the estimate for this feature of the lead is suspect, and may be inaccurate. In contrast, consistent estimates for any given profile element may act as reinforcement that the proper vales were accurately ascertained. Volatility in the sentiment/tone is to be expected, and does not denote inaccuracy of the measurement.
  • the system may inquire if the document received is a "simple" question for which the system is capable of auto-generating a response (at 1370). If so, a separate process for automated response generation is initiated, as will be discussed below. This inquiry, as discussed previously, may depend upon a set of training questions for specific answers supplied by the campaign manager. These training questions may be subjected to a synonym replacement to generate a very large set of permutations for training questions. These are then fed to a machine learning algorithm to teach the model what questions should retrieve specific information. The retrieved information is stored in a separate database from the knowledge set used by the AI models.
  • This separate database is typically a third party database of inventory, employees/extensions, store locations, store hours, etc.
  • the system may apply business logic to the classifications to determine the best action to take (at 1380).
  • Campaign objectives as they are updated, may be used to redefine the actions collected and scheduled. For example, 'skip-to-follow-up' action may be replaced with an 'informational message' introducing the sales rep before proceeding to 'series 3' objectives.
  • 'Do Not Email' or ' Stop Messaging' classifications should deactivate a lead and remove scheduling at any time during a lead's life-cycle.
  • the system is also capable of populating a transparency interface (at 1390).
  • the system breaks down the message into component chunks (at 1610), with each component having a discrete meaning, insight, or conveying a piece of information.
  • These insights are have been previously extracted from the message and confidence scores have been computed for the given insights.
  • Sentiment has also been determined, typically on a 0-100 score.
  • actions are decided upon using the computed insights. All of these activities have been previously discussed, and are essential to the process of generating a dynamic messaging system.
  • the system may layer a "mouse over" description of the color coded component to help assist the user understand what the chunking signifies (at 1630). For example, if the message component states “Send me the details", by moving the cursor over this message component the system may provide, at a minimum, that this component relates to an insight of "interested in learning more". Other information that may be provided to the user may include the confidence level for the insight, and even the rules and/or NPL algorithms involved in making this conclusion, based upon implementation.
  • the last element in the transparency interface process may generate is a separate information pane that includes top insights, other derived attributes, sentiment, confidence levels for the insights, and actions to be taken (at 1640).
  • This pane provides a summary of critical information derived from the given message node.
  • the pane described here, and shown in subsequent figures, may be regarded as a 'basic' view with the most critical information for the given message. More advanced views are also possible, which may include far more information, which can vary by communication channel. For example for email communications, IP address and other header information could be included in the summary pane. For audio communications, the routing or call setup details from the mobile tower involved (including location data) could be included.
  • the details provided may include locally derived information (such as insights) and information received from 3rd part sources (such as cellular tower data). Attributes for a given lead may be tracked to assist in future predictive analysis. Further, some attributes, such as the sentiment, may be a basic compilation score (0-100 scale for example) or may be multi-faceted. For sentiment for example, this score may be broken into a continuum for happy /unhappy, and also a score for clear/confused. Likewise, it may be possible to go beyond sentiment for the given message node, and provide the previously populated behavioral profile for the lead, which becomes more robust as the lead interacts with the AI system.
  • This profile can indicate how patient/impatient the lead is, decision making tendencies (analytical versus emotional), temperament, education level, likes, etc., as previously discussed.
  • the speed of which the AI sends a given message to a human operator may be influenced by this profile, as well as response tactics.
  • Figure 17 provides an example screenshot 1700 for a message string between a lead and the AI system.
  • the responses from the lead are illustrated for the user's benefit.
  • These message nodes are analyzed as discussed above, and a color-coded display of the messages is provided to the user, as seen in reference to Figure 18 at 1800.
  • the message components are shaded at different levels to comport to patent office restrictions regarding color in the figures. Regardless, it can be seen that different components of the message such as "Hi Lindsey ! " and "Yes I am interested! " are determined to be separate, and convey discrete meanings. If the user were to move the cursor over the different components, the system would present what information is being derived from the message component, and the confidence level for the insight being generated.
  • the system derives a summary pane with the insights, confidence levels, other details, and actions all illustrated in an easily understood manner for the user's consumption, as seen in Figure 19 at 1900.
  • the attributes derived include the message language, estimates for the user's fluency level in the language and education level, sentiment and a behavior type.
  • additional or alternate information such as IP address, physical location, and other 3 rd party data (e.g., credit scores, age, job history, etc.) can be included in this summary pane.
  • the actions derived from the insights are listed in the bottom of the summary panes. For example, when the insight is "interested in learning more" the action may be to continue messaging with a template that provides more information. In contrast when the insight is to "propose a meeting" the action may be to continue messaging with an introduction of a representative and a schedule meeting template.
  • the Inference Engine combines statistical decisions with rule-based business logic to determine the best set of actions to take. A determination is made whether there is an action conflict (at 1250) or unsatisfactory confidence in action decisions. Manual review may be needed when in both cases, (at 1270). Otherwise, the actions may be executed by the system (at 1260).
  • a determination is made whether to deactivate the lead (at 1075). Such a deactivation may be determined as needed when the lead requests it. If so, then the lead is deactivated (at 1090). If not, the process continues by determining if the campaign for the given lead is complete (at 1080). The campaign may be completed when all objectives for the lead have been met, or when there are no longer messages in the series that are applicable to the given lead. Once the campaign is completed, the lead may likewise be deactivated (at 1090).
  • the process may return to the delay period (at 1020) before preparing and sending out the next message in the series (at 1030).
  • the process iterates in this manner until the lead requests deactivation, or until all objectives are met. This concludes the main process for a comprehensive messaging campaign. Attention will now be focused on the sub-processes that further enhance message classification.
  • the system initially undergoes a query to identify if the non-textual element is a movie or an image (at 2010).
  • the file type or extension may be utilized to determine if the element is a video or image. If the element is a movie, the system may separate out any audio elements to the video (at 201 1) and then perform a speech to text conversion (at 2012). The textual output can then be run though the same textual analysis described above (at 2013). Lighting of the video image frames may also be scored (at 2015) as lower lighting tends to indicate a reduced sentiment, as it is associated with intimacy, fear, and melancholy. Brighter frames may indicate happiness and other positive emotions. As such the lighting score may be utilized to modulate the sentiment score (at 2015). Likewise, embedded pure audio files may also be extracted, converted to text, and processed in a similar manner.
  • the system may make a determination if the image is an emoji/emoticon (at 2020). This may be performed by first looking at the image pixel size, as emoticons fall within standard sizes for fitting into text lines. If the image is within the parameters of an emoti con's size constraints, the emoticon may be compared against a repository of emoticons (at 2021).
  • This repository may be a database that includes known emoji's, each annotated with a meaning. For example, a "party horn" emoji may indicate celebration and joy. A “poo” emoji may indicate disapproval, disappointment or disgust. These annotated meanings may be added to the classification of the document being processed.
  • the system may determine if there is a face present (at 2030). If a face is present the face may be analyzed with standard facial recognition tools, and emotion in the face may be extracted (at 2031). Likewise, if there is text in the image, as is common in meme type images, this text may be extracted and processed as described above.
  • Figure 21 provides an example process for simple message handling, at 2100.
  • the first step is to determine if the message concept matches a fundamental question category (at 2110).
  • These fundamental questions are uploaded by a campaign manager and linked back to an external dataset which includes information answering the question.
  • the campaign manager provides a training question, which may be combined with similar training questions belonging to the same category as the question posed by the campaign manager. For example, the campaign manager may submit the following training question: "what are your store hours?". This question belongs to the category related to "store hours”. The campaign manager links this training question to a listing of hours on the store's website.
  • the system may then pull up other questions that are known to fulfill the category of "store hours” and likewise link them to the provided dataset with the answer.
  • An example of such an alternate known question could include: "when are you open?" or "When can I come in?”.
  • the system may perform a synonym replacement in the training set to generate an even greater expanded set of training questions. By replacing words in each training question with all known synonyms, it is possible to generate many hundreds or even thousands of permutations. Each of these permutations may be leveraged by the AI model to determine when a message comes in whether it relates to a fundamental question.
  • the system uses vector similarity and instance classification approaches.
  • neural network, LSI and other word embedding approaches such as random projection are used to learnt the semantic vector representation of first the words, then individual questions, then question categories and finally the input from actor in online mode.
  • the input semantic vector is compared to the semantic vector of the question categories and the most similar question category is chosen as the matched question if the similarity value (as measured by dot product and other approaches) is more than the minimum similarity value between all pairs of the variations in the question category.
  • the similarity value as measured by dot product and other approaches
  • the AI model may utilize the training questions to populate a definition table including an ID for each question, name of the question category, description, active status, insert date and last modified date.
  • the factual data answering the question may be looked up in accordance with the training provided from the campaign manager (at 2120). Again, this typically includes looking up the factual data in an external third party dataset.
  • the recovered data is then populated into a human-mimicking response template (at 2130) which is output as a message to the lead (at 2140) without the need for any human intervention.
  • simple questions include business hours, business location inquiry, "how are you” inquiries, questions pertaining to contacting a specific employ ee/department/extension, questions regarding inventory availability, etc.
  • the response template may be a singular template, multiple templates that can be used interchangeably, or a template with variable features that may be psudo-randomly replaced to ensure the response is as "human sounding” as possible.
  • the response template could include "[Salutation], I [verb] [status] today.”
  • the salutation could be randomly, or pseudo-randomly selected from the following: “Thanks for asking”, “Hi”, “Hey”, “You are so sweet” or the like.
  • the verb could include the following: “am”, “am feeling”, “feel”, etc.
  • the status could include: “happy”, “fine”, “great”, etc. This allows a total of at least 36 possible outputs for this question. This is important since the lead may send more than one simple question over the course of a message exchange, and having a static answer may appear "robotic" to the lead over time.
  • FIG. 22 an example process is presented for the generation and leveraging of decision trees to infer the best action to take, shown at 2200.
  • the campaign manager or other administrator is provided a Tree Builder interface for populating the decision tree (at 2210) (see section [00106]).
  • a document is processed through a standardized classification with an initial knowledge set (at 2220). In the case of difficult documents, this may result in a classification for the document with two or more categories attributed to the document.
  • This document is then reclassified through the decision tree using a different dataset and different classification thresholds (at 2230). Each decision tree can either produce a confident classification or indicate that human review is required via Training Desk.
  • Confident classifications apply when each decision node in a classification tree meets or exceeds the node's confidence threshold.
  • a single decision tree can produce a single classification describing the best action for the system to apply. Decision trees are run within the Inference Engine to infer the best action to take. Lastly, the action for the document is determined based upon the decision tree output (at 2240).
  • Figures 23A and 23B illustrate a Computer System 2300, which is suitable for implementing embodiments of the present invention.
  • Figure 23A shows one possible physical form of the Computer System 2300.
  • the Computer System 2300 may have many physical forms ranging from a printed circuit board, an integrated circuit, and a small handheld device up to a huge super computer.
  • Computer system 2300 may include a Monitor 2302, a Display 2304, a Housing 2306, a Disk Drive 2308, a Keyboard 2310, and a Mouse 2312.
  • Disk 2314 is a computer-readable medium used to transfer data to and from Computer System 2300.
  • Figure 23B is an example of a block diagram for Computer System 2300.
  • Processor(s) 2322 (also referred to as central processing units, or CPUs) are coupled to storage devices, including Memory 2324.
  • Memory 2324 includes random access memory (RAM) and read-only memory (ROM).
  • RAM random access memory
  • ROM read-only memory
  • RAM random access memory
  • ROM read-only memory
  • Both of these types of memories may include any suitable of the computer-readable media described below.
  • a Fixed Disk 2326 may also be coupled bi- directionally to the Processor 2322; it provides additional data storage capacity and may also include any of the computer-readable media described below.
  • Fixed Disk 2326 may be used to store programs, data, and the like and is typically a secondary storage medium (such as a hard disk) that is slower than primary storage. It will be appreciated that the information retained within Fixed Disk 2326 may, in appropriate cases, be incorporated in standard fashion as virtual memory in Memory 2324.
  • Removable Disk 2314 may take the form of any of the computer-readable media described below.
  • Processor 2322 is also coupled to a variety of input/output devices, such as
  • an input/output device may be any of: video displays, track balls, mice, keyboards, microphones, touch- sensitive displays, transducer card readers, magnetic or paper tape readers, tablets, styluses, voice or handwriting recognizers, biometrics readers, motion sensors, brain wave readers, or other computers.
  • Processor 2322 optionally may be coupled to another computer or telecommunications network using Network Interface 2340. With such a Network Interface 2340, it is contemplated that the Processor 2322 might receive information from the network, or might output information to the network in the course of performing the above-described classification and message processing.
  • method embodiments of the present invention may execute solely upon Processor 2322 or may execute over a network such as the Internet in conjunction with a remote CPU that shares a portion of the processing.
  • Software is typically stored in the non-volatile memory and/or the drive unit.
  • a software program is assumed to be stored at any known or convenient location (from non-volatile storage to hardware registers) when the software program is referred to as “implemented in a computer-readable medium.”
  • a processor is considered to be “configured to execute a program” when at least one value associated with the program is stored in a register readable by the processor.
  • the computer system 2300 can be controlled by operating system software that includes a file management system, such as a disk operating system.
  • a file management system such as a disk operating system.
  • operating system software with associated file management system software is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Washington, and their associated file management systems.
  • Windows® from Microsoft Corporation of Redmond, Washington
  • Linux operating system and its associated file management system is the Linux operating system and its associated file management system.
  • the file management system is typically stored in the non-volatile memory and/or drive unit and causes the processor to execute the various acts required by the operating system to input and output data and to store data in the memory, including storing files on the non-volatile memory and/or drive unit.
  • the machine operates as a standalone device or may be connected (e.g., networked) to other machines.
  • the machine may operate in the capacity of a server or a client machine in a client-server network environment or as a peer machine in a peer-to-peer (or distributed) network environment.
  • the machine may be a server computer, a client computer, a virtual machine, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, an iPhone, a Blackberry, a processor, a telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
  • PC personal computer
  • PDA personal digital assistant
  • machine-readable medium or machine-readable storage medium is shown in an exemplary embodiment to be a single medium, the term “machine-readable medium” and “machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions.
  • the term “machine- readable medium” and “machine-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the presently disclosed technique and innovation.
  • routines executed to implement the embodiments of the disclosure may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions referred to as "computer programs.”
  • the computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and when read and executed by one or more processing units or processors in a computer, cause the computer to perform operations to execute elements involving the various aspects of the disclosure.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Business, Economics & Management (AREA)
  • Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Development Economics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

L'invention concerne des systèmes et procédés de traitement et de classification en langage naturel. Dans certains modes de réalisation, les systèmes et procédés intègrent un tableau de bord d'éditeur de communication qui reçoit le message et effectue un traitement en langage naturel de façon à diviser le message en parties constitutives. Le système affiche le message, chacune des parties constitutives étant superposée avec une couleur différente, dans un premier volet. Il affiche les opinions, les scores de confiance associés à chaque opinion, le sentiment et les actions dans un second volet. Dans un autre mode de réalisation, les systèmes et procédés intègrent la combinaison de sorties provenant de multiples modèles IA appris par machine en une sortie unifiée. Dans un autre mode de réalisation, les systèmes et procédés intègrent la réponse à une question simple à l'aide d'un traitement en langage naturel.
PCT/US2018/051827 2017-09-20 2018-09-19 Systèmes et procédés de traitement et de classification en langage naturel Ceased WO2019060468A1 (fr)

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US201762561194P 2017-09-20 2017-09-20
US62/561,194 2017-09-20
US16/019,382 2018-06-26
US16/019,382 US11301632B2 (en) 2015-01-23 2018-06-26 Systems and methods for natural language processing and classification
US16/129,722 2018-09-12
US16/129,729 2018-09-12
US16/129,722 US20190079920A1 (en) 2015-01-23 2018-09-12 Systems and methods for a communication editor dashboard
US16/129,729 US11010555B2 (en) 2015-01-23 2018-09-12 Systems and methods for automated question response

Publications (1)

Publication Number Publication Date
WO2019060468A1 true WO2019060468A1 (fr) 2019-03-28

Family

ID=65809856

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/051827 Ceased WO2019060468A1 (fr) 2017-09-20 2018-09-19 Systèmes et procédés de traitement et de classification en langage naturel

Country Status (1)

Country Link
WO (1) WO2019060468A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021046541A1 (fr) * 2019-09-06 2021-03-11 Digital Asset Capital, Inc. Détermination de résultat de graphe dans un environnement d'exécution spécifique au domaine
US10990879B2 (en) 2019-09-06 2021-04-27 Digital Asset Capital, Inc. Graph expansion and outcome determination for graph-defined program states
WO2023279038A1 (fr) * 2021-06-30 2023-01-05 Pricewaterhousecoopers Llp Plateforme de vérification à ia augmentée comprenant des techniques d'adjudication automatique de substance commerciale, de parties associées et de collectabilité
US12339904B2 (en) 2019-09-06 2025-06-24 Digital Asset Capital, Inc Dimensional reduction of categorized directed graphs

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070233692A1 (en) * 2006-04-03 2007-10-04 Lisa Steven G System, methods and applications for embedded internet searching and result display
US20130346067A1 (en) * 2012-06-26 2013-12-26 International Business Machines Corporation Real-time message sentiment awareness
US20150350144A1 (en) * 2014-05-27 2015-12-03 Insidesales.com Email optimization for predicted recipient behavior: suggesting changes in an email to increase the likelihood of an outcome
US20150365527A1 (en) * 2014-06-12 2015-12-17 Sriram Chakravarthy Methods and systems of an automated answering system
US20160217500A1 (en) * 2015-01-23 2016-07-28 Conversica, Llc Systems and methods for management of automated dynamic messaging

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070233692A1 (en) * 2006-04-03 2007-10-04 Lisa Steven G System, methods and applications for embedded internet searching and result display
US20130346067A1 (en) * 2012-06-26 2013-12-26 International Business Machines Corporation Real-time message sentiment awareness
US20150350144A1 (en) * 2014-05-27 2015-12-03 Insidesales.com Email optimization for predicted recipient behavior: suggesting changes in an email to increase the likelihood of an outcome
US20150365527A1 (en) * 2014-06-12 2015-12-17 Sriram Chakravarthy Methods and systems of an automated answering system
US20160217500A1 (en) * 2015-01-23 2016-07-28 Conversica, Llc Systems and methods for management of automated dynamic messaging

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021046541A1 (fr) * 2019-09-06 2021-03-11 Digital Asset Capital, Inc. Détermination de résultat de graphe dans un environnement d'exécution spécifique au domaine
US10990879B2 (en) 2019-09-06 2021-04-27 Digital Asset Capital, Inc. Graph expansion and outcome determination for graph-defined program states
US11132403B2 (en) 2019-09-06 2021-09-28 Digital Asset Capital, Inc. Graph-manipulation based domain-specific execution environment
US11526333B2 (en) 2019-09-06 2022-12-13 Digital Asset Capital, Inc. Graph outcome determination in domain-specific execution environment
US11853724B2 (en) 2019-09-06 2023-12-26 Digital Asset Capital, Inc. Graph outcome determination in domain-specific execution environment
US12299036B2 (en) 2019-09-06 2025-05-13 Digital Asset Capital, Inc Querying graph-based models
US12339904B2 (en) 2019-09-06 2025-06-24 Digital Asset Capital, Inc Dimensional reduction of categorized directed graphs
US12379902B2 (en) 2019-09-06 2025-08-05 Digital Asset Capital, Inc. Event-based entity scoring in distributed systems
WO2023279038A1 (fr) * 2021-06-30 2023-01-05 Pricewaterhousecoopers Llp Plateforme de vérification à ia augmentée comprenant des techniques d'adjudication automatique de substance commerciale, de parties associées et de collectabilité
US12217318B2 (en) 2021-06-30 2025-02-04 PwC Product Sales LLC AI-augmented auditing platform including techniques for automated document processing

Similar Documents

Publication Publication Date Title
US11010555B2 (en) Systems and methods for automated question response
US11336539B2 (en) Support ticket summarizer, similarity classifier, and resolution forecaster
US11574026B2 (en) Analytics-driven recommendation engine
US11641330B2 (en) Communication content tailoring
US20200143247A1 (en) Systems and methods for improved automated conversations with intent and action response generation
US11663409B2 (en) Systems and methods for training machine learning models using active learning
US20200143115A1 (en) Systems and methods for improved automated conversations
US20190180196A1 (en) Systems and methods for generating and updating machine hybrid deep learning models
WO2021138020A1 (fr) Systèmes et procédés pour des améliorations d'intelligence artificielle dans des conversations automatisées
US10803479B2 (en) Systems and methods for management of automated dynamic messaging
WO2020139865A1 (fr) Systèmes et procédés pour des conversations automatisées améliorées
US20190179903A1 (en) Systems and methods for multi language automated action response
US20190286711A1 (en) Systems and methods for message building for machine learning conversations
WO2019113122A1 (fr) Systèmes et procédés d'apprentissage automatique amélioré pour des conversations
US20190286712A1 (en) Systems and methods for phrase selection for machine learning conversations
US11106871B2 (en) Systems and methods for configurable messaging response-action engine
US11042910B2 (en) Systems and methods for processing message exchanges using artificial intelligence
US20190122236A1 (en) Systems and methods for message cadence optimization
US20190286713A1 (en) Systems and methods for enhanced natural language processing for machine learning conversations
WO2019060468A1 (fr) Systèmes et procédés de traitement et de classification en langage naturel
US10026037B2 (en) Systems and methods for configuring knowledge sets and AI algorithms for automated message exchanges
Raju et al. Advancing AI-Driven Customer Service with NLP: A Novel Bert-Based Model for Automated Responses
WO2019191337A1 (fr) Systèmes et procédés pour un traitement amélioré de langage naturel pour des conversations par apprentissage automatique
US11100285B2 (en) Systems and methods for configurable messaging with feature extraction
WO2019084321A1 (fr) Systèmes et procédés pour moteur de réponse-action à message configurable

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18858527

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18858527

Country of ref document: EP

Kind code of ref document: A1