WO2024259168A1

WO2024259168A1 - Systems and methods for efficiently determining minimum amounts of data for accurate model validation

Info

Publication number: WO2024259168A1
Application number: PCT/US2024/033906
Authority: WO
Inventors: Muhammad Umair MUSTAFA; Muhammad Abdullah ALI
Original assignee: Afiniti Ltd; Afiniti Inc
Current assignee: Afiniti Ltd; Afiniti Inc
Priority date: 2023-06-15
Filing date: 2024-06-13
Publication date: 2024-12-19
Anticipated expiration: 2025-12-15

Abstract

Systems and methods for determining the minimum amount of data needed to accurately validate models. The system obtains historical contact-agent interaction data that comprises outcomes associated with the contact-agent interactions at a contact center over a first time period. Using this historical contact-agent interaction data, the system validates a first model (e.g., a current model determined to have a threshold level of efficiency) using a subset of the historical contact-agent interaction data having a predetermined size (e.g., a predetermined number of days) based on characteristics of the contact center (e.g., how many agents, how many contact groups, how many contact-agent interactions per day, economic conditions, product/resource offerings associated with the contact center, etc.) using different subsets of the historical contact-agent interaction data with different sizes (e.g., differing amounts of days) to generate mean measurements and standard error measurements.

Description

SYSTEMS AND METHODS FOR EFFICIENTLY

DETERMINING MINIMUM AMOUNTS OF DATA FOR ACCURATE

MODEL VALIDATION

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63/508,500, filed on June 15, 2023. The content of the foregoing application is incorporated herein in its entirety by reference.

BACKGROUND

[0002] Systems for connecting parties in a communication, irrespective of whether the format of the communication comprises in-person discussions, electronic transmissions, and/or other forms through which information is exchanged, are increasingly focusing on streamlining the connection process and/or increasing the efficiency of the resulting communications. To achieve this outcome, systems increasingly rely on one or more models for selecting the parties for the communication. These models may invoke one or more algorithms and/or pairing strategies which comprise a deliberate approach or plan for combining or matching items, parties, and/or entities for a particular purpose or desired outcome. In some cases, these algorithms may also involve the use of artificial intelligence, including, but not limited to, machine learning, deep learning, etc. (referred to collectively herein as artificial intelligence models, machine learning models, or simply models). [0003] Regardless of the algorithms and/or pairing strategies used, effectively pairing parties to communications involves overcoming several technical challenges inherent to the pairing process. For example, pairing parties with complementary skills and knowledge is crucial for achieving intended outcomes; however, finding the right pairing can be challenging, especially when there is a wide range of expertise and/or when specific domain knowledge is required. Moreover, even if skill sets are known, coordinating the schedules of those parties to facilitate pairing may be difficult, particularly when the parties have different commitments and/or work on different projects. Failure of the system to account for misaligned availability may lead to delays and interruptions, reducing the effectiveness of the pairing strategy. These technical problems may be further exacerbated when attempting to use a model-based solution in dynamic environments featuring parties with a diverse spectrum of skills, requests featuring unknown skill sets, and parties with varying schedules, such as a contact center.

SUMMARY

[0004] Systems and methods are described herein for novel uses and/or improvements to modeling applications. As one example, systems and methods are described herein for improvements to modeling applications used in dynamic environments such as contact centers. In particular, the systems and methods provide improvements for determining the minimum amount of data needed to accurately validate models. For example, the systems and methods may provide improvements in determining the minimum number of days of data needed to accurately validate models corresponding to contact center pairing strategies.

[0005] Vox example, to efficiently connect contacts and agents at a contact center, the contact center may use a pairing strategy based on a model. To maintain efficiency, the model may be periodically updated based on newly received data (e.g., results from previous contact-agent interactions). Prior to updating a pairing strategy based on the updated model, the updated model needs to be validated to ensure its efficiency. However, validating this model creates a technical problem in terms of how much data (e.g., how many days of data) should be used for validation.

[0006] If the model is trained based on data that is too outdated, then the model will not accurately predict contact-agent pairings for a current contact center environment because contacts calling the contact center may have different issues, new product releases may render agents’ knowledge obsolete, general economic conditions may have changed market conditions, etc. In contrast, if too small an amount of data is used (e.g., too few days of data), the model may not be adequately trained to anticipate and react to the current contact center environment. To further exacerbate this problem, the amount of data available each day may vary.

[0007] To overcome these technical deficiencies in adapting models for this practical benefit, the systems and methods obtain historical contact-agent interaction data that comprises outcomes associated with the contact-agent interactions at a contact center over a first time period. Using this historical contact-agent interaction data, the system validates a first model (e.g., a current model determined to have a threshold level of efficiency) using a subset of the historical contactagent interaction data having a predetermined size (e.g., a predetermined number of days) based on characteristics of the contact center (e.g., how many agents, how many contact groups, how many contact-agent interactions per day, economic conditions, product/resource offerings associated with the contact center, etc.) using different subsets of the historical contact-agent interaction data with different sizes (e.g., differing amounts of days) to generate mean measurements and standard error measurements.

[0008] The system then determines a set of numbers, wherein each number of the set of numbers corresponds to one of a plurality of subset sizes for the set of historical contact-agent interaction data (e.g., each number corresponds to a number of days of data). The system then determines a number of validations (with sampling, as is necessary) for each number in the set such that the number of days generated by the data is equal for the validations. The system performs the validations to determine a plurality of mean measurements and a plurality of standard error measurements for comparison to the results of the first model. By doing so, the system may account for the variability in both the amount of data available each day and any statistical difference in the composition of the data (e.g., particular contacts, agents, contact-agent interactions) and any other statistical noise (e.g., market fluctuations, etc.). Based on the comparison, the system may determine a subset size to use for validation of future models; for example, the determined subset size may be one number of the set of numbers.

[0009] In some aspects, systems and methods for efficiently determining minimum amounts of data needed in order to accurately validate contact center pairing strategy models are described. For example, the system may obtain a first set of historical contact-agent interaction data. The system may determine a set of numbers, wherein each number of the set of numbers corresponds to one of a plurality of subset sizes for the first set of historical contact-agent interaction data. The system may validate a first model of a plurality of models based on a folding process and the set of numbers to determine a plurality of mean measurements and a plurality of standard error measurements to generate a first validation result, wherein one of the plurality of mean measurements and one of the plurality of standard error measurements corresponds to each number of the set of numbers. The system may determine a first subset size of the plurality of subset sizes for validation based on a first mean measurement of the plurality of mean measurements and a first standard error measurement of the plurality of standard error measurements, wherein the first mean measurement corresponds to the first subset size, and wherein the first standard error measurement corresponds to the first subset size. The system may validate a second model of the plurality of models based on the folding process and the first subset size to generate a second validation result. The system may select the second model of the plurality of models based on the second validation result. The system may output the second model for use in a contact center.

[0010] Various other aspects, features, and advantages of the invention will be apparent through the detailed description of the invention and the drawings attached hereto. It is also to be understood that both the foregoing general description and the following detailed description are examples and are not restrictive of the scope of the invention. As used in the specification and in the claims, the singular forms of “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. In addition, as used in the specification and the claims, the term “or” means “and/or” unless the context clearly dictates otherwise. Additionally, as used in the specification, “a portion” refers to a part of, or the entirety of (i.e., the entire portion), a given item (e.g., data) unless the context clearly dictates otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1A shows an example communication system, in accordance with one or more embodiments.

[0012] FIG. IB shows an example communication system, in accordance with one or more embodiments.

[0013] FIG. 1C shows an example communication system, in accordance with one or more embodiments.

[0014] FIG. ID shows an example communication system, in accordance with one or more embodiments.

[0015] FIG. 2 shows a system featuring a model configured to facilitate pairing strategies, in accordance with one or more embodiments.

[0016] FIG. 3 shows graphical representations of artificial neural network models for facilitating pairing strategies, in accordance with one or more embodiments.

[0017] FIG. 4 shows a representation of a graphic for facilitating pairing strategies, in accordance with one or more embodiments.

[0018] FIG. 5 shows a flowchart for efficiently determining minimum amounts of data needed in order to efficiently validate contact center pairing strategy models, in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE DRAWINGS

[0019] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the invention. It will be appreciated, however, by those having skill in the art that the embodiments of the invention may be practiced without these specific details or with an equivalent arrangement. In other cases, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the embodiments of the invention.

[0020] FIG. 1A illustrates an example communication system 100A. In this example, communication system 100A may comprise a contact center to facilitate communications between a contact 102 and agents (e.g., agent 130A and agent 130B) via one or more user interfaces 104. For example, a contact center may comprise a centralized facility or department within an organization that handles incoming and outgoing communications with users, customers, and/or other parties (e.g., referred to herein as “contacts”). As referred to herein, a “user interface” may comprise a human-computer interaction and communication in a device, and may include display screens, keyboards, a mouse, and the appearance of a desktop. For example, a user interface may comprise a way for an agent to interact with an application, website, or contact, and/or to communicate content. As referred to herein, “content” should be understood to mean an electronically consumable and/or transmittable data such as Internet content (e.g., streaming content, downloadable content, Webcasts, etc.), video clips, audio, content information, pictures, rotating images, documents, playlists, websites, articles, books, electronic books, blogs, advertisements, chat sessions, social media content, applications, games, and/or any other media or multimedia and/or combination of the same.

[0021] For example, a contact center may comprise a primary point of contact for contacts (e.g., contact 102) to interact with one or more agents (e.g., agent 130A and agent 130B) to ask questions, seek support, or make inquiries. As referred to herein, an agent may comprise a representative of a contact center, which may include a human and/or non-human entity that may interact with a contact during a contact-agent interaction.

[0022] As described herein, contact centers may use various communication channels such as phone calls, emails, webchat, social media, video calls, text chat, text messages, and/or other communication means to facilitate interactions between contacts and agents (e.g., contact-agent interactions). In some embodiments, contact centers often utilize advanced technologies like interactive voice response (IVR) systems, customer relationship management (CRM) software, call routing and queuing systems, knowledge bases, and analytics tools to streamline operations, improve efficiency, and enhance the overall customer experience.

[0023] As described herein, a contact-agent interaction may comprise a distinct session or a series of communications between a contact (e.g., contact 102) and an agent (e.g., agent 130A). A given contact-agent interaction may be distinguished from another contact-agent interaction based on one or more characteristics. As described herein, “interaction data” may comprise a feature or quality (whether quantitative or qualitative) used to identify a contact-agent interaction and/or distinguish the contact-agent interaction from another contact-agent interaction.

[0024] In some embodiments, interaction data may comprise: a time or date (or other temporal characteristic) related to the contact-agent interaction; content at issue and/or discussed at the contact-agent interaction; an identity (or identifier) for an agent and/or contact (e.g., such as a Billing Telephone Number (BTN)), including profile data related to the agent and/or contact involved in the contact-agent interaction; a model and/or pairing strategy, including one or more metrics thereof, used to connect parties in the contact-agent interaction; a result or outcome of a contact-agent interaction related to the contact-agent interaction; a contact center state before, during, and/or after the contact-agent interaction; a contact account state, including types of data and/or values thereof, before, during, and/or after the contact-agent interaction; and/or a relationship of the contact-agent interaction to another contact-agent interaction, including whether or not the contact-agent interaction, and/or any interaction data thereof, corresponds to (e.g., is equal to, is within a proximity to, is within a range of, matches to a certain degree and/or confidence, and/or involves similar interaction data) another contact-agent interaction, and/or any interaction data thereof.

[0025] In some embodiments, interaction data may also include transcript data that may include information that was sent between a contact/agent (e.g., email s/texts). For example, interaction data may comprise a written or printed version of material originally presented in another medium. In some embodiments, interaction data may include time data that may indicate an interaction start time, interaction end time, estimated wait time, handle time, etc.

[0026] For example, the system may record interaction data for one or more contact-agent interactions. In some embodiments, the system may use the interaction data (and/or its relationship to one or more contact-agent interactions) to generate historical contact-agent interaction data (e.g., which may be used for modeling purposes).

[0027] In some embodiments, the interaction data may include profile data. The system may monitor for and/or record contact-agent interaction characteristics to generate profile data. As referred to herein, “a profile” and/or “profile data” may comprise actively and/or passively collected data about a contact-agent interaction, contact, agent, model, contact center, and/or collection (or subset thereof). For example, the profile data may comprise interaction data generated by, at, and/or in response to the contact-agent interaction.

[0028] In some embodiments, interaction data may comprise time-series data. As described herein, “time-series data” may include a sequence of data points that occur in successive order over some period of time. In some embodiments, time-series data may be contrasted with cross-sectional data, which captures a point in time. A time series can be taken on any variable that changes over time (e g., a value in a contact account). The system may use a time series to track the variable (e.g., account status) of an account over time (e.g., across multiple contact-agent interactions). This can be tracked over the short term, such as whether an account was “saved” during a contact-agent interaction, or the long term, such as whether the account continued to be saved over a given period of time and/or after successive contact-agent interactions. In some embodiments, the system may generate a time-series analysis and/or may use the time-series analysis for one or more metrics (e.g., as described below).

[0029] As shown in FIG. 1A, the communication system 100A may include a central switch 110. The central switch 110 may receive incoming contacts (e.g., callers) or support outbound connections to contacts via a telecommunications network (not shown). The central switch 110 may include contact routing hardware and software for helping to route contacts among one or more contact centers, or among one or more contact center switches 120 A and 120B. For example, the central switch 110 is a load balancer. Contact center switches 120A and 120B may include a Private Branch Exchanges (PBXs) and/or Automatic Call Distributors (ACDs) or other queuing or switching components, including other Internet-based, cloud-based, or otherwise networked contact-agent hardware or software-based contact center solutions.

[0030] In should be noted that the central switch 110 or load balancer may not be necessary such as if there is only one contact center, or if there is only one switch or PBX/ACD routing component, in the communication system 100A (not shown). If more than one contact center is part of the communication system 100A, each contact center may include at least one contact center switch (e.g., contact center switches 120A and 120B). For the purposes of this example embodiment, contact center switches 120A and 120B are both routed to the same contact center system. The contact center switches 120 A and 120B may be communicatively coupled to the central switch 110. In embodiments, various topologies of routing and network components may be configured to implement the contact center system.

[0031] Each contact center switch for each contact center may be communicatively coupled to a plurality (or “pool”) of agents (e.g., agent 130A and agent 130B). Each contact center switch may support a certain number of agents (or “seats”) to be logged in at one time. At any given time, a logged-in agent may be available and waiting to be connected to a contact, or the logged-in agent may be unavailable for any of a number of reasons, such as being connected to another contact, performing certain post-call functions such as logging information about the call, or taking a break. [0032] In some embodiments, a contact may be included in a contact group. For example, a contact group or contact type may be a partition or grouping of contacts based on historical contactagent interaction data. As one example, contacts who arrive at a contact center may be assigned to or associated with a specific contact group or type. In some embodiments, a contact group may be defined as a contact percentile range. For example, percentiles may comprise a division of a ranked data set into a set number of (e.g., 100) equal parts. In one example, each (ranked) data set has 99 percentiles that divide it into 100 equal parts. For example, the kth percentile is denoted by Pk, where k is an integer in the range 1 to 99. In some embodiments, the contact percentile range may include a group of percentiles and/or these groups may be ranked. For example, a percentile rank of a score refers to the percentage of metrics in its frequency distribution that are lower than it or equal to. For example, a metric that is greater than 75% of the metrics of results of contacts in contact-agent interactions may be at the 75th percentile. For example, having a sufficient number of contact groups ensures that there is enough diversity in the data to test behavioral pairing strategies in the contact center environment.

[0033] In the example of FIG. 1A, the central switch 110 routes contacts via contact center switch 120A and contact center switch 120B, respectively. The contact center switches 120A and 120B are shown and may be connected with two agents each. Agents 130A and I30B may be logged in to contact center switch 120 A, and agents 130C and BOD may be logged in to contact center switch 120B. [0034] The communication system 100A may also be communicatively coupled to an integrated service from, for example, a third-party vendor. In the example of FIG. 1A, a pairing node 140 may be communicatively coupled to one or more switches in the switch system of the communication system 100A, such as central switch 110, contact center switch 120A, and/or contact center switch 120B. In some embodiments, switches of the communication system 100A may be communicatively coupled to multiple pairing nodes (e.g., each switch is connected to a different pairing node). In some embodiments, pairing node 140 may be embedded within a component of a contact center system (e.g., embedded in or otherwise integrated with a switch). The pairing node 140 may receive information from a switch (e g., contact center switch 120A) about agents logged in to the switch (e.g., agents 130A and 130B) and about incoming contacts via another switch (e.g., central switch 110) or, in some embodiments, from a network (e.g., the Internet or a telecommunications network) (not shown).

[0035] A contact center may include multiple pairing nodes. In some embodiments, one or more pairing nodes may be components of pairing node 140 or one or more switches such as central switch 110 or contact center switches 120A and 120B. In some embodiments, a pairing node may determine which pairing node may handle pairing for a particular contact. For example, a first pairing node may alternate between enabling pairing via a Behavioral Pairing (BP) strategy via the first pairing node and enabling pairing with a First-in-First-out (FIFO) strategy via a second pairing node. In other embodiments, one pairing node (e.g., the BP pairing node) may itself be configured to emulate other pairing strategies.

[0036] FIG. IB illustrates a second example communication system 100B. As shown in FIG. IB, the communication system 100B may include one or more agent endpoints 151A, 151B and one or more contact endpoints 152A, 152B. The agent endpoints 151A, 151B may include an agent terminal and/or an agent computing device (e.g., laptop, cellphone). The contact endpoints 152A, 152B may include a contact terminal and/or a contact computing device (e.g., laptop, cellphone). Agent endpoints 151 A, 15 IB and/or contact endpoints 152A, 152B may connect to a Contact Center as a Service (CCaaS) 170 through either the Internet or a public switched telephone network (PSTN) 160, according to the capabilities of the endpoint device.

[0037] FIG. 1C illustrates an example communication system 100C with an example configuration of a CCaaS 170. For example, a CCaaS 170 may include multiple data centers 180A, 180B. The data centers 180A, 180B may be separated physically, even in different countries and/or continents. The data centers 180A, 180B may communicate with each other. For example, one data center is a backup for the other data center, so that, in some embodiments, only one data center 180A or 180B receives agent endpoints 151 A, 15 IB and contact endpoints 152A, 152B at a time.

data center 180A, 180B includes web demilitarized zone (DMZ) equipment 171A and 171B, respectively, which is configured to receive the agent endpoints 151A, 15 IB and contact endpoints 152A, 152B, which are communicatively connecting to CCaaS via the Internet. Web DMZ equipment 171A and 171B may operate outside a firewall to connect with the agent endpoints 151A, 151B and contact endpoints 152A, 152B while the rest of the components of data centers 180A, 180B may be within said firewall (besides the telephony DMZ equipment 172A, 172B, which may also be outside said firewall). Similarly, each data center 180A, 180B includes telephony DMZ equipment 172A and 172B, respectively, which is configured to receive agent endpoints 151 A, 15 IB and contact endpoints 152A, 152B, which are communicatively connecting to CCaaS via the PSTN. Telephony DMZ equipment 172A and 172B may operate outside a firewall to connect with the agent endpoints 151 A, 15 IB and contact endpoints 152A, 152B while the rest of the components of data centers 180A, 180B (excluding web DMZ equipment 171 A, 171B) may be within said firewall.

[0039] Further, each data center 180A, 180B may include one or more nodes 173A, 173B and 173C, 173D, respectively. All nodes 173A, 173B and 173C, 173D may communicate with web DMZ equipment 171A and 171B, respectively, and with telephony DMZ equipment 172A and 172B, respectively. In some embodiments, only one node in each data center 180 A, 180B may be communicating with web DMZ equipment 171A, 171B and with telephony DMZ equipment 172A, 172B at a time. 0040/ Each node 173A, 173B, 173C, 173D may have one or more pairing modules 174A, 174B, 174C, 174D, respectively. Similar to pairing node 140 of communication system 100A of FIG. 1A, pairing modules 174A, 174B, 174C, 174D may pair contacts to agents. For example, the pairing module may alternate between enabling pairing via a BP module and enabling pairing with a FIFO module. In other embodiments, one pairing module (e.g., the BP module) may be configured to emulate other pairing strategies.

[0041] Turning now to FIG. ID and example communication system 100D, the disclosed CCaaS communication systems (e.g., FIGs. IB and/or 1C) may support multi-tenancy such that multiple contact centers (or contact center operations or businesses) may be operated on a shared environment. That is, each tenant may have a separate, non-overlapping pool of agents. CCaaS 170 is shown in FIG. ID as comprising two tenants 190A and 190B. Turning back to FIG. 1C, for example, multi-tenancy may be supported by node 173A supporting tenant 190A while node 173B supports tenant 190B. In another embodiment, data center 180 A supports tenant 190A while data center 180B supports tenant 190B. In another example, multi-tenancy may be supported through a shared machine or shared virtual machine such that node 173 A may support both tenants 190A and 190B, and similarly for nodes 173B, 173C, and 173D. In other embodiments, the system may be configured for a single tenant within a dedicated environment such as a private machine or private virtual machine.

[0042] FIG. 2 shows a system featuring a model configured to facilitate pairing strategies, in accordance with one or more embodiments. In some embodiments, system 200 may use one or more models, including artificial intelligence-based and non-artificial intelligence-based, to select a model for pairing one or more contacts and/or agents, pair one or more contacts and/or agents, and/or evaluate (or validate) another model (e.g., used to pair one or more contacts and/or agents). For example, as shown in FIG. 2, system 200 may generate a prediction and/or determination using model 202. The determination may be output shown as output on a user interface (e.g., interface 104 (FIG. 1 A)) and/or may direct a pairing using one or more components (e.g., pairing node 140 (FIG. 1A)) or one or more switches (such as central switch 110 (FIG. 1A) or contact center switches 120A and 120B (FIG. 1A)). The system may include one or more pairing strategies, neural networks (e.g., as discussed in relation to FIG. 3), and/or other models.

[0043] As an example, with respect to FIG. 2, model 202 may take inputs 204 and provide outputs 206. The inputs may include multiple datasets such as a training dataset and a test dataset (which may be based on historical interaction data). The datasets may represent one or more contact-agent interactions and/or interaction data related thereto. In some embodiments, outputs 206 may be fed back to model 202 as input to train model 202 (e.g., alone or in conjunction with user indications of the accuracy of outputs 206, labels associated with the inputs, or other reference feedback information). Alternatively or additionally, model 202 may update its configurations (e.g., weights, biases, or other parameters) based on its assessment of its prediction (e.g., outputs 206) and reference feedback information (e.g., user indication of accuracy, reference labels, additional interaction data, or other information). Alternatively or additionally, where model 202 is a neural network, connection weights may be adjusted to reconcile differences between the neural network’s prediction and the reference feedback. In a further use case, one or more neurons (or nodes) of the neural network may require that their respective errors be sent backward through the neural network to them to facilitate the update process (e.g., b ackpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the model 202 may be trained to generate better predictions and/or determinations.

[0044] In some embodiments, model 202 may comprise one or more versions. For example, different versions of model 202 (and/or other models referred to herein) may refer to variations or iterations of a particular model that have been developed over time. These versions may involve changes to the model’s architecture, parameters, and/or training data (e.g., as a result of newly available data, improvements to the model’s performance, and/or updates to address specific requirements). Here are a few common types of model versions. In some embodiments, the system may use an initial baseline model, which may be used by the system as a starting point for further improvements and/or comparisons of the model and/or its results. The versions of model 202 may also have one or more iterative versions as a result of the model undergoing a series of incremental changes and/or iterations. In some embodiments, system 200 may store a plurality of models. This plurality of models may include multiple versions of the same model as well as different models. The models may differ based on algorithms used for the model, pairing strategies corresponding to the model, and/or datasets upon which the model was trained,

[0045] Model 202 may be trained to select a model for pairing one or more contacts and/or agents, pair one or more contacts and/or agents, and/or evaluate (or validate) another model (e.g., used to pair one or more contacts and/or agents). For example, model 202 may receive a first dataset (e.g., historical interaction data) and/or one or more classifications (e.g., available pairing strategies, available agents for assigning a contact-agent interaction, a metric for a model, etc.). Model 202 is then trained based on a first dataset to classify a second dataset (e.g., contacts awaiting assignment to an agent, etc.).

[0046] In some embodiments, model 202 may generate and/or record one or more metrics for a model. As referred to herein, a model metric may comprise data about a model and/or its results. In some embodiments, metrics may comprise evaluation metrics that are measures used to assess the performance and effectiveness of the model and/or a version thereof. For example, the metrics may provide quantitative information about how well the model is performing and/or may be used to compare different models and/or evaluate the same model under different conditions.

[0047] In some embodiments, the metrics may comprise one or more values based on statistical and/or mathematical operations. For example, the metric may reflect an accuracy of a model (e.g., a proportion of correct predictions made by the model over the total number of predictions), precision (e.g., a proportion of true positive predictions over the total number of positive predictions), recall (e.g., a proportion of true positive predictions over the total number of actual positive instances), Fl score (e.g., a harmonic mean of precision and recall), specificity (e.g., a proportion of true negative predictions over the total number of actual negative instances), mean squared error (e.g., an average squared difference between predicted and actual values), mean absolute error (e.g., an average absolute difference between predicted and actual values), R- squared (e.g., a proportion of the variance in the dependent variable that is predictable from the independent variables), contact (or agent) percentile or ranking (e g., a percentile or ranking at which a contact or agent is assigned), standard deviation (e.g., a measure of how dispersed the data is in relation to the mean), and/or mean absolute deviation (e.g., a measure of variability that indicates the average distance between each prediction and a mean of the total number of predictions). In some embodiments, the metrics may be determined using specific formulas and/or criteria. For example, a contact group may be assigned to a percentile range (e.g., 10-20%). A contact associated to that contact group may then be assigned within that range randomly.

[0048] Additionally or alternatively, the metrics for one or more models (or versions thereof), samples of the one or more models’ results, and/or results from running one or more models may be aggregated. The system may aggregate the models to determine means, medians, modes, and/or other averaging functions to determine an average metric for one or more models.

[0049] Additionally or alternatively, the metrics for a model may be based on, a derivative of, and/or related to a contact account (or value thereof) as well as one or more contact-agent interactions. For example, a metric may relate to an average amount, a number of, and/or a percentile of contact accounts that comprise a particular value for a particular category and/or a change in the value. For example, a metric may comprise an average amount, a number of, and/or a percentile of contact accounts that maintain, change, and/or gain a value before, during, and/or after a client interaction. For example, in the context of a call center, a metric for a model may be based on the number of contact-agent interactions that resulted in an account being maintained, an account making a purchase, or any indicator of a successful outcome. For example, in the context of a call center, a metric for a model may be based on the number of contact-agent interactions that resulted in an account being cancelled.

[0050] Additionally or alternatively, the metric for a model may be in relation to another model. For example, a metric for a model may relate to a lift or gain experienced as the result of a model. Lift may be a measure of an effectiveness of a predictive model calculated as the difference between the results of a first model versus a second model (e.g., the ratio of contact accounts that were lost using a first model for pairing strategies subtracted from the ratio of client accounts that were lost using a second model for pairing strategies). Gain may be the ratio between the lift and data measured ratio of a given pairing strategy (e.g., the ratio of contact accounts that were lost using the first model). For example, the system may determine the lift or gain attributed to one model in relation to another model.

[0051] Additionally or alternatively, the system may determine weights and/or effects on an evaluation and/or a validation of a model (e.g., in relation to another model) based on additional factors. For example, a metric corresponding to gain or lift may be increased (or decreased) based on whether multiple contacts, agents, contact-agent interactions, etc. corresponded to a given model.

[0052] For example, a metric may be based on a difference or lack of a difference between a respective outcome (e.g., a respective outcome of a contact-agent interaction under a first model) and a contact center state (e.g., a contact center state after contact-agent interaction under the same or different model). The system may determine groupings of outcomes and contact center states based on common contacts and/or contact-agent interactions within a given time period.

[0053] In one example, the metric may correspond to difference (or lack thereof) in a respective outcome (e.g., a change or lack thereof in an account) under a first model and a contact center state (e.g., a change or lack thereof in an account) under a different model. For example, the metric may be based on whether an account was eventually canceled under a second model after initially being saved under a first model. Alternatively, the metric may be based on whether an account was eventually saved under a second model after initially being canceled under a first model.

[0054] In another example, the metric may correspond to difference (or lack thereof) in a respective outcome (e.g., a change or lack thereof in an account) under a first model and a contact center state (e.g., a change or lack thereof in an account) under the same model. For example, the metric may be based on whether an account was eventually canceled under a first model after initially being saved under the first model. Alternatively, the metric may be based on whether an account was eventually saved under a first model after initially being canceled under the first model.

[0055] In some embodiments, the metric may be based on instances in which the same model provided particular results (e.g., two contact-agent interactions that resulted in a save, cancelation, etc.). The metric in such instances may be based on additional weighting and/or emphasis. In some embodiments, the metric may be based on instances in which the same model provided particular frequencies of subsequent contact-agent interactions (e g., a model that generates additional contact-agent interactions). The metric may in such instances be based on additional weighting and/or emphasis.

[0056] For example, when determining a metric for a first model during a period of time, the first model may be associated with various sets of calls, e.g.,: (i) a first set of calls, wherein each call in the first set of calls is a first touch, and has a value associated with a particular outcome; (ii) a second set of calls, wherein each call is a second touch, wherein the first touch for each call in the second set of calls was associated with a second model, and wherein the second touch for each call in the second set of calls was associated with the first model, and wherein each call in the second set of calls is associated with a value associated with a particular outcome after the second touch; (iii) a third set of calls, wherein each call is a second touch, wherein the first touch for each call in the third set of calls was associated with the first model, and wherein the second touch for each call in the third set of calls was associated with the second model, and wherein each call in the third set of calls is associated with a value associated with a particular outcome after the second touch; (iv) a fourth set of calls, wherein each call is a second touch, wherein the first touch for each call in the fourth set of calls was associated with the first model, and wherein the second touch for each call in the fourth set of calls was associated with the first model, and wherein each call in the fourth set of calls is associated with a value associated with a particular outcome after the second touch. For example, the metric may be based on a ratio/percentage/count of the first set of calls, the second set of calls, the third set of calls, and the fourth set of calls. In some examples, the fourth set of calls may be associated with a weighting to increase its weight relative to the other sets of calls. For example, the outcome may be successful (e.g., the metric may be associated with a save rate or a conversion rate) or the outcome may be unsuccessful (e.g., the metric may be associated with a cancel rate). In another example, a gain or lift metric may be determined based on at least one of the first set of calls, the second set of calls, the third set of calls, and the fourth set of calls. In another example, such a gain or lift metric may further include a weighting on the fourth set of calls to increase its weight relative to the other sets of calls. For example, such a weight is 1.25, 1.5, 1.75, 2, 2.25, 2.5, 2.75, 3, etc.

[0057] In another example, the metric may be based on a callback rate for each model (e.g., callback rate corresponds to a number/percentage/ratio of contacts who returned to the contact center within a threshold period of time after a first interaction of that contact with the contact center system).

[0058] In some embodiments, the system may generate a time-series analysis and/or may use the time-series data for one or more metrics. For example, a time-series analysis may be useful to see how a given contact account or value therein changes (or does not change) over time and/or subsequent contact-agent interactions. The system may also use a time-series analysis to examine how the changes associated with the chosen data point compare to shifts in other variables (e.g., model, agent, contact center performance) over the same time period and/or subsequent contactagent interactions. For example, with regard to accounts saved or gained, the system may receive time-series data for the various sub-segments indicating daily values for agents, contact centers, products, accounts, contacts, and/or groups thereof.

[0059] The time-series analysis may determine various trends such as secular trends (which describe movements along the term), seasonal variations (which represent seasonal changes), cyclical fluctuations (which correspond to periodic but not seasonal variations), and irregular variations (which are other nonrandom sources of series variations). The system may maintain correlations for this data during modeling. In particular, the system may maintain correlations through non-normalization as normalizing data inherently changes the underlying data, which may render correlations, if any, undetectable and/or lead to the detection of false positive correlations. For example, modeling techniques (and the predictions generated by them), such as rarefying (e.g., resampling as if each sample has the same total counts), total sum scaling (e.g., dividing counts by the sequencing depth), and others, and the performance of some strongly parametric approaches, depends heavily on the normalization choices. Thus, normalization may lead to lower model performance and more model errors. The use of a non-parametric bias test alleviates the need for normalization, while still allowing the methods and systems to determine a respective proportion of error detections for each of the plurality of time-series data component models.

[0060] FIG. 3 shows graphical representations of artificial neural network models for facilitating pairing strategies, in accordance with one or more embodiments. Model 300 illustrates an artificial neural network. Model 300 includes input layer 302. Input layer 302 may receive contacts and/or agents waiting to be paired, metrics used to select a model for pairing one or more contacts and/or agents, etc. Model 300 also includes one or more hidden layers (e.g., hidden layer 304 and hidden layer 306). Model 300 may be based on a large collection of neural units (or artificial neurons). Model 300 loosely mimics the manner in which a biological brain works (e.g., via large clusters of biological neurons connected by axons). Each neural unit of a model 300 may be connected with many other neural units of model 300. Such connections can be enforcing or inhibitory in their effect on the activation state of connected neural units. In some embodiments, each individual neural unit may have a summation function which combines the values of all of its inputs together. In some embodiments, each connection (or the neural unit itself) may have a threshold function that the signal must surpass before it propagates to other neural units. Model 300 may be selflearning and trained, rather than explicitly programmed, and can perform significantly better in certain areas of problem solving, as compared to traditional computer programs. During training, output layer 308 may correspond to a classification of model 300 (e.g., contacts awaiting assignment to an agent, etc.), and an input known to correspond to that classification may be input into input layer 302. In some embodiments, model 300 may include multiple layers (e.g., where a signal path traverses from front layers to back layers). In some embodiments, backpropagation techniques may be utilized by model 300 where forward stimulation is used to reset weights on the “front” neural units. In some embodiments, stimulation and inhibition for model 300 may be more free-flowing, with connections interacting in a more chaotic and complex fashion. Model 300 also includes output layer 308. During testing, output layer 308 may indicate whether or not a given input corresponds to a classification of model 300 (e.g., pair one or more contacts and/or agents, select a model for pairing one or more contacts and/or agents, and/or evaluate (or validate) another model (e.g., used to pair one or more contacts and/or agents)).

[0061] FIG. 3 also includes model 350, which is a convolutional neural network. The convolutional neural network is an artificial neural network that features one or more convolutional layers. Convolutional layers extract features from an input. Convolution preserves the relationship between the data input by learning features using partitions of the input data. As shown in model 350, input layer 352 may proceed to convolution blocks 354 and 356 before being output to a convolutional output (e.g., output 360). In some embodiments, model 350 may itself serve as an input to model 300.

[0062] In some embodiments, model 350 may implement an inverted residual structure where the input and output of a residual block (e.g., block 354) are thin bottleneck layers. A residual layer may feed into the next layer and directly into layers that are one or more layers downstream. A bottleneck layer (e.g., block 358) is a layer that contains few neural units compared to the previous layers. Model 350 may use a bottleneck layer to obtain a representation of the input with reduced dimensionality. An example of this is the use of autoencoders with bottleneck layers for nonlinear dimensionality reduction. Additionally, model 350 may remove nonlinearities in a narrow layer (e.g., block 358) in order to maintain representational power. In some embodiments, the design of model 350 may also be guided by the metric of computational complexity (e.g., the number of floating point operations). In some embodiments, model 350 may increase the feature map dimension at all units to involve as many locations as possible instead of sharply increasing the feature map dimensions at neural units that perform downsampling. In some embodiments, model 350 may decrease the depth and increase the width of residual layers in the downstream direction. Model 350 may then generate output 360.

[0063] FIG. 4 shows a representation of a graphic for facilitating pairing strategies, in accordance with one or more embodiments. For example, the system may generate a graphic (e.g., graphic 400) in order to facilitate the selection of a model and/or pairing strategy. For example, the system may use a graphic and/or other information to select subset sizes. For example, the system may generate a color map that may show visual data to the user to help the user select the number of days of data that should be used. The graphic may show multiple types of data such that the user may select the row (e.g., corresponding to a subset size) in the color map which has the best overall measurements. For example, when determining the first subset size of the plurality of subset sizes for validation, the system may generate for display, on a user interface, a graphic (e.g., graphic 400) comprising a plurality of mean measurements and a plurality of standard error measurements that correspond to each number of the set of numbers. The system may receive a user input selecting the first subset size based on the graphic. [0064] For example, the system may generate for display graphic 400. As shown in graphic 400, the system may determine respective color codings for each of the plurality of mean measurements and the plurality of standard error measurements. The system may generate a color map based on the respective color codings. As shown in graphic 400, the system may generate various statistical metrics corresponding to different validation results (e.g., referred to herein both individually and collectively as validation results).

[0065] As shown in graphic 400, the system may generate a table with color codings corresponding to the metrics. The various metrics may further be organized according to fold size and include metrics corresponding to mean lift, mean absolute deviation (MAD), MAD lift, and other metrics as discussed herein.

[0066] For example, the system may obtain historical contact-agent interaction data that comprises outcomes associated with the contact-agent interactions at a contact center over a first time period. [0067] Using this historical contact-agent interaction data, the system may validate a first model (e.g., a current model determined to have a threshold level of efficiency). For example, the system may select a model which has produced results that have achieved a predetermined level of accuracy and/or precision, wherein the model comprises a sufficient number of contact groups for evaluation. For example, contact groups a sufficient number of contact groups may comprise 10, 15, 20, 25, 30, 35, etc. contact groups.

[0068] The system may validate the first model using a set of the historical contact-agent interaction data having a predetermined size (e.g., a predetermined number of days) based on characteristics of the contact center (e.g., how many agents, how many contact groups, how many contact-agent interactions per day, economic conditions, product/resource offerings associated with the contact center, etc.) using different subsets of the set of historical contact-agent interaction data with different sizes (e.g., differing amounts of days) to generate mean measurements and standard error measurements.

[0069] For example, the system may use Ni days of data (X) and run multiple validations (e.g., 10) without any replacement sampling. These multiple validations may capture variations coming from random contact percentile (cp) selection between the minimum and maximum cp range for a given contact in a given contact type/group. For example, the system may use random cp selection to assign each contact within a contact group a random placement within a percentile range associated with the contact group, said percentile range having a minimum percentile and a maximum percentile. These multiple validations ensure coverage of the contact being placed throughout the percentile range space associated with the contact group. The system may then calculate mean lift and MAD.

[0070] or example, Ni may be 15, 20, 30, 35, 40, 45, 50, 55, 60, 75, 90, etc. The system may select this number based on a contact center state (e.g., how many agents, how many contact groups, how much throughput (contact-agent interactions) per day in the contact center), economic conditions, product/resource offerings associated with the contact center, etc. The system may then perform the validation (e.g., using k-fold cross-validation). The system may then use the validation results as a ground truth and/or baseline.

[0071] The system may then determine a set of numbers, wherein each number of the set of numbers corresponds to one of a plurality of subset sizes for the set of historical contact-agent interaction data. For example, the system may determine a set of numbers N2, N3, N4, Ns, Ne and a length of the set, where each number is smaller than Ni. For example, the set may comprise values such as 5, 7, 10, 15, 20.

[0072] The system may determine a number of validations (with sampling, as is necessary) for each number in the set such that the number of days generated by the data is equal for the validations. For example, the system may determine a number of validations (with sampling, as is necessary) for each number in the set (e.g., a set of Mi, M2, M3, M4, Ms), such that the number of days generated by the data is equal for the validations of each N2, N3, N4, Ns, Ne. For example, to perform at least 10 validations (e.g., Ms = 10) on Ne where Ne is 20, the system may select the remaining numbers in the sets to equal approximately 200 days’ worth of data. Accordingly, if N2 = 5, the system may perform 40 validations (e.g., Mi = 40).

[0073] The system may perform the validations to determine a plurality of measurements in order to compare the results of each validation to the results of the first model. For example, if the measurements performed during validation of the first model were mean measurements and standard error measurements, then the plurality of measurements may comprise a plurality of mean measurements and a plurality of standard error measurements for comparison to the results of the first model. For example, the system may sample N2 days of data from X and run Mi validations on the sample by sampling with replacement and calculate mean lift and MAD (e.g., as shown in graphic 400). The system may sample N3 days of data from X and run M2 validations on the sample by sampling with replacement and calculate mean lift and MAD. The system may sample N4 days of data from X and run M3 validations on the sample by sampling with replacement and calculate mean lift and MAD. The sample may sample Ns days of data from X and run M4 validations on the sample by sampling with replacement and calculate mean lift and MAD. The system may sample Ns days of data from X and run Ms validations on the sample by sampling with replacement and calculate mean lift and MAD.

[0074] The system may then obtain the validation results for comparison to the validation of other models. For example, graphic 400 shows the results of the aforementioned validations. The system may then select a best-performing number of days for future validations. For example, the system may select one of N2, N3, N4, Ns, or Ne. The system may then determine to validate a current contact center model using the selected number of days of data (e.g., the selected one of N2, N3, N4, Ns, or Ne). Optionally, this selection is used for a set period of time (e.g., one month, three months, six months).

[0075] In one example of this comparison, the system may select a mean lift of the selected number of days of data that is stable (e.g., has results with a predetermined level of accuracy), and is also stable in terms of deviation. The system may determine deviation as being relative to the mean lift (MAD lift = absolute deviation/lift). For example, if MAD is higher than the lift for a number of days of data, then the measured lift may be determined by the system to be clouded by noise. As such, the system may not select that number of days.

[0076] In some examples, the inventors contemplate a color map that may show visual data to the user to help select the number of days of data. The user would want to select the row in the color map which has the best overall measurements.

[0077] FIG. 5 shows a flowchart for efficiently determining minimum amounts of data needed in order to validate contact center pairing strategy models, in accordance with one or more embodiments. For example, process 500 may provide an efficient approximation of the accuracy of a model. The system may use process 500 (e.g., as implemented on one or more system components described above) in order to account for the variability in the amount of data available during each day as well as any statistical difference in the composition of the data (e.g., particular contacts, agents, contact-agent interactions) in a contact center. For example, process 500 obtains historical contact-agent interaction data that comprises outcomes associated with the contact-agent interactions at a contact center over a first time period. Using this historical contact-agent interaction data, process 500 validates a first model (e.g., a current model determined to have a threshold level of efficiency) using a subset of the historical contact-agent interaction data having a predetermined size (e.g., a predetermined number of days) based on characteristics of the contact center (e.g., how many agents, how many contact groups, how many contact-agent interactions per day, economic conditions, product/resource offerings associated with the contact center, etc.) using different subsets of the historical contact-agent interaction data with different sizes (e.g., differing amounts of days) to generate mean measurements and standard error measurements. Process 500 then determines a set of numbers, wherein each number of the set of numbers corresponds to one of a plurality of subset sizes for the set of historical contact-agent interaction data. Process 500 then determines a number of validations (with sampling, as is necessary) for each number in the set such that the number of days generated by the data is equal for the validations. Process 500 performs the validations to determine a plurality of mean measurements and a plurality of standard error measurements for comparison to the results of the first model. Based on the comparison, process 500 may determine a subset size to use for validation of future models.

[0078] At step 502, process 500 (e.g., using one or more components described above) obtains historical contact-agent interaction data. For example, the system may obtain a first set of historical contact-agent interaction data. For example, the system obtains historical contact-agent interaction data that comprises outcomes associated with the contact-agent interactions at a contact center over a first time period.

[0079] In some embodiments, these outcomes may comprise one or more metrics. In some embodiments, the metrics may include determining which contact-agent interactions corresponded to the same contact and/or contact account. For example, the system may determine that a retention metric for agent performance should be adjusted irrespective of whether the contact was “saved” on an initial interaction with an agent but then was canceled on a subsequent interaction with an agent. For example, performing a respective validation for each model of the plurality of models based on the metric may comprise the system assigning a first contact center state associated with a third contact-agent interaction to the first model in response to determining that a first contact corresponds to both the first contact-agent interaction and the third contact-agent interaction. The system may then determine the metric for a first model based on determining that the first contact corresponds to both the first contact-agent interaction and the third contact-agent interaction. [0080] In some embodiments, the system may measure a contact center state. The contact center state may be a qualitative or quantitative metric. In some embodiments, the contact center state many correspond to a metric used for an outcome and/or result of a contact-agent interaction. The system may determine an initial value (or change in value) from an initial interaction and determine whether that initial value (or the change in value) is maintained during a subsequent interaction. For example, the system may determine whether a change to an account occurred during a first threshold proximity (e.g., during a first interaction). The system may also determine whether a change occurred during a second threshold proximity (e.g., during a second interaction). By determining whether the change occurred during either proximity, the system may determine whether a change (or lack thereof) was permanent or temporary. For example, by measuring the contact center state after each contact-agent interaction, the system may retrieve the respective contact account and determine whether the respective contact account state for the respective contact account changes from the first value to the second value within a second threshold proximity of the respective first contact time point. The system may perform a respective second validation for each model of the plurality of models based on a second metric, wherein the second metric is based on measuring a contact center state after each contact-agent interaction or further in time from the first contact-agent interaction (e.g., an outcome subsequently following a second contact-agent interaction corresponding to the first). For example, there may not be a second contact-agent interaction, but the second metric may still be taken 5 or 30 days after the initial interaction.

[0081] In some embodiments, the system may measure the contact center state based on changes to an account state of a contact. For example, the system may measure the contact center state based on whether the account state has increased in value or decreased in value. For example, the system may measure the contact center state after each contact-agent interaction by determining a respective contact account corresponding to each contact-agent interaction. The system may then determine a respective first contact time point corresponding to each contact-agent interaction. The system may then determine a first value of a respective contact account state for the respective contact account at the respective first contact time point. The system may then determine a second value of the respective contact account state for the respective contact account at a second contact time point. The system may then determine whether the first value equals the second value. [0082] In some embodiments, the system may measure the contact center state based on comparing outcomes from a first contact-agent interaction and a second contact-agent interaction. For example, when measuring the contact center state after each contact-agent interaction, the system may determine whether each contact-agent interaction corresponds to a subsequent contact-agent interaction and determine a subsequent respective outcome associated with the subsequent contactagent interaction.

[0083] In some embodiments, the system may perform the model validations based on the one or more metrics. These metrics may be based on determining a value (e.g., whether an account was saved or canceled) after a contact-agent interaction. For example, the system may determine a first value of a contact account after a first contact-agent interaction, wherein the first contact-agent interaction occurred under a first model of the plurality of models. The system may then determine the first metric for the first model based on the first value. The system may determine a second value of the contact account after a second contact-agent interaction. The system may determine the second metric for the first model based on the second value.

[0084] In some embodiments, the system may increase (or decrease) the weight of a value based on whether or not multiple contact-agent interactions occurred under a model (e.g., the same pairing strategy) for the same contact. For example, the system may include additional weights (or penalties) to reflect whether a pairing strategy succeeded (or failed) multiple times. For example, determining the second metric for the first model based on the second value may comprise the system determining whether the second contact-agent interaction occurred under the first model. The system may then determine a weight (or other effect) of the second value on the second metric based on determining that the second contact-agent interaction occurred under the first model.

[0085] For example, a metric may be based on a difference or lack of a difference between a respective outcome (e g., a respective outcome of a contact-agent interaction under a first model) and a contact center state (e.g., a contact center state after contact-agent interaction under the same or different model). The system may determine groupings of outcomes and contact center states based on common contacts and/or contact-agent interactions within a given time period.

[0086] In one example, the metric may correspond to difference (or lack thereof) in a respective outcome (e.g., a change or lack thereof in an account) under a first model and a contact center state (e.g., a change or lack thereof in an account) under a different model. For example, the metric may be based on whether an account was eventually canceled under a second model after initially being saved under a first model. Alternatively, the metric may be based on whether an account was eventually saved under a second model after initially being canceled under a first model.

[0087] In another example, the metric may correspond to difference (or lack thereof) in a respective outcome (e.g., a change or lack thereof in an account) under a first model and a contact center state (e.g., a change or lack thereof in an account) under the same model. For example, the metric may be based on whether an account was eventually canceled under a first model after initially being saved under the first model. Alternatively, the metric may be based on whether an account was eventually saved under a first model after initially being canceled under the first model.

[0088] In some embodiments, the metric may be based on instances in which the same model provided particular results (e.g., two contact-agent interactions that resulted in a save, cancelation, etc.). The metric may in such instances be based on additional weighting and/or emphasis. In some embodiments, the metric may be based on instances in which the same model provided particular frequencies of subsequent contact-agent interactions (e.g., a model that generates additional contact-agent interactions). The metric may in such instances be based on additional weighting and/or emphasis.

[0089] At step 504, process 500 (e.g., using one or more components described above) determines a set of numbers. For example, the system may determine a set of numbers, wherein each number of the set of numbers corresponds to one of a plurality of subset sizes for the first set of historical contact-agent interaction data. For example, the system may determine a subset of the historical contact-agent interaction data having a predetermined size (e.g., a predetermined number of days) based on characteristics of the contact center (e.g., how many agents, how many contact groups, how many contact-agent interactions per day, economic conditions, product/resource offerings associated with the contact center, etc.) using different subsets of the historical contact-agent interaction data with different sizes (e.g., differing amounts of days).

[0090] In some embodiments, the set of numbers may comprise a number of days of data related to contact-agent interactions. This number may be selected based on contact center state (e g., how many agents, how many contact groups, how much throughput (contact-agent interactions) per day in the contact center), economic conditions, product/resource offerings associated with the contact center, etc. For example, when determining the set of numbers, the system may determine a number of contact-agent interactions in the first set of historical contact-agent interaction data, wherein the first set of historical contact-agent interaction data comprises outcomes associated with contact-agent interactions over a first time period. The system may partition the first time period into a plurality of partitions. The system may determine a number of contact-agent interactions in each of the plurality of partitions, wherein the set of numbers is based on the number of contact-agent interactions in each of the plurality of partitions.

[0091] In some embodiments, the set of numbers may comprise a number of days of data related to contact-agent interactions. This number may be selected based on the age of the data. For example, when determining the set of numbers, the system may determine an age of the first set of historical contact-agent interaction data and select each number of the set of numbers based on the age.

[0092] In some embodiments, the set of numbers may comprise a number of days of data related to contact-agent interactions. This number may be selected based on requirements that are specific to a location or feature of a contact center environment at which the data is generated. For example, when determining the set of numbers, the system may determine a location or feature of a contact center environment at which the first set of historical contact-agent interaction data is generated. The system may determine a specific requirement based on the location or feature of a contact center environment. The system may select each number of the set of numbers based on the specific requirement.

[0093] At step 506, process 500 (e.g., using one or more components described above) validates a first model of a plurality of models. For example, the system may validate a first model of a plurality of models based on a folding process and the set of numbers to determine a plurality of mean measurements and a plurality of standard error measurements to generate a first validation result, wherein one of the plurality of mean measurements and one of the plurality of standard error measurements corresponds to each number of the set of numbers. For example, using this historical contact-agent interaction data, the system validates a first model (e.g., a current model determined to have a threshold level of efficiency) using the subset of the historical contact-agent interaction data having the predetermined size to generate mean measurements and standard error measurements.

[0094] In some embodiments, the folding process may comprise a k-fold cross-validation. In k- fold cross-validation, an original sample is randomly partitioned into k equal-sized subsamples (e.g., corresponding to the set of numbers). Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k - 1 subsamples are used as training and/or comparison data. For example, the system may partition the set of historical contact-agent interaction data based on the set of numbers. The system may determine a first partition of the set of historical contact-agent interaction data corresponding to the first subset size. The system may determine a second partition of the set of historical contact-agent interaction data corresponding to a second subset size. The system may generate the first validation results based on the first partition and not the second partition. For example, when validating the first model of the plurality of models based on the folding process and the set of numbers, the system may partition the first set of historical contact-agent interaction data based on the set of numbers. The system may do so by determining a first plurality of partitions of the first set of historical contact-agent interaction data corresponding to a first number in the set of numbers and generating the first validation result based on the first plurality of partitions. For example, partitioning the first set of historical contactagent interaction data based on the set of numbers may comprise the system determining a second plurality of partitions of the first set of historical contact-agent interaction data corresponding to a second number in the set of numbers.

[0095] At step 508, process 500 (e.g., using one or more components described above) determines a first subset size. For example, the system may determine a first subset size of the plurality of subset sizes for validation based on a first mean measurement of the plurality of mean measurements and a first standard error measurement of the plurality of standard error measurements, wherein the first mean measurement corresponds to the first subset size, and wherein the first standard error measurement corresponds to the first subset size. For example, the system may determine a set of numbers, wherein each number of the set of numbers corresponds to one of a plurality of subset sizes for the set of historical contact-agent interaction data.

[0096] In some embodiments, the system may determine a MAD, a measure of variability that indicates the average distance between each observation in a dataset and the mean of the dataset. For example, the system may determine a first partition of the first set of historical contact-agent interaction data based on the first subset size. The system may determine a first mathematical mean for the first partition. The system may determine a plurality of average distances between points in the first partition and the first mathematical mean. The system may determine the first mean measurement based on the plurality of average distances between points in the first partition and the first mathematical mean. [0097] In some embodiments, the system may determine a standard error. The standard error of a statistic is the standard deviation of its sampling distribution or an estimate of that standard deviation. For example, for a given sample size, the standard error equals the standard deviation divided by the square root of the sample size. For example, the system may determine a first partition of the first set of historical contact-agent interaction data based on the first subset size. The system may determine a first mathematical mean for the first partition. The system may determine a standard deviation between points in the first partition and the first mathematical mean. The system may determine the first standard error measurement based on multiplying the standard deviation by a square root of a number of points in the first partition.

[0098] In some embodiments, the system may use various formulas, variables, and/or priorities to determine a subset size. For example, instead of, or in addition to, using MAD, the system may use standard error. In some embodiments, the system may further determine a threshold value for standard error and select the subset size for validation based on the threshold value. For example, the system may ensure that the standard error exceeds the threshold value (e.g., standard error is roughly two sigma). For example, when determining the first subset size of the plurality of subset sizes for validation based on the first standard error measurement, the system may determine a threshold value for a standard error and compare the first standard error measurement to the threshold value to select the first subset size.

[0099] In some embodiments, the system may further generate values and/or conditions for selecting the subset size. For example, the system may select a subset size based on determining a value corresponding to a mean divided by standard error that is higher than the threshold value. For example, in some embodiments, a mean measurement may include any measurement based on the mean of a category of data. For example, a mean measurement may comprise a mean lift, which may comprise a mean of measurements of lift (e.g., a measure of the performance of the model at predicting or classifying cases as having an enhanced response (with respect to the population as a whole), measured against a random choice of the model). For example, when determining a first subset size based on the first mean measurement and the first standard error measurement, the system may determine a value corresponding to the first mean measurement divided by the first standard error measurement and select the first subset size based on the value. [0100] At step 510, process 500 (e.g., using one or more components described above) validates a second model of the plurality of models. For example, the system may validate a second model of the plurality of models based on the folding process and the first subset size to generate a second validation result.

[0101] At step 512, process 500 (e.g., using one or more components described above) selects the second model based on a second validation result. For example, the system may select the second model of the plurality of models based on the second validation result. The validation results may then be used by the system to determine a subset size to use for validation of future models. Additionally or alternatively, the system may output the second model for use in a contact center (e.g., generate a recommendation to use the second model). For example, the system may generate graphic 400 (FIG. 4) to facilitate a selection of the second model. Additionally or alternatively, the system may establish a connection between communication equipment of a contact (e.g., contact 102 (FIG. 1A)) and communication equipment of an agent (e.g., agent 130A (FIG. 1A)) based on one of the plurality of models.

[0102] In some embodiments, the system may periodically and/or in response to a user input receive a request to revalidate the models linked to one or more pairing strategies. For example, the system may trigger the revalidating of models based on events that indicate the models need to be validated more extensively, including (1) if a client associated with the contact center launches agent training, so agents are expected to perform differently (e.g., different rankings), (2) a client releases different product offerings, so the contact groups are expected to perform differently (e.g., different ordering and/or different contact groups entirely), (3) the economic market changes and the volume of expected purchases grows or shrinks, (4) the routing logic of the contact center system changes, and/or (5) any other economic shift that suggests historical trends may be more or less accurate.

[0103] For example, the system may receive a request to determine a new subset size. The system may validate the second model of the plurality of models based on the folding process and the set of numbers to determine a new plurality of mean measurements and a new plurality of standard error measurements to generate a second validation result, wherein one of the new plurality of mean measurements and one of the new plurality of standard error measurements corresponds to each number of the set of numbers. The system may determine a second subset size of the plurality of subset sizes for validation based on a second mean measurement of the new plurality of mean measurements and a second standard error measurement of the new plurality of standard error measurements, wherein the second mean measurement corresponds to the second subset size, and wherein the first standard error measurement corresponds to the second subset size. The system may validate a third model of the plurality of models based on the folding process and the first subset size to generate a third validation result. The system may select the third model of the plurality of models based on the third validation result. The system may output the third model for use in the contact center.

[0104] It is contemplated that the steps or descriptions of FIG. 5 may be used with any other embodiment of this disclosure. In addition, the steps and descriptions described in relation to FIG. 5 may be done in alternative orders or in parallel to further the purposes of this disclosure. For example, each of these steps may be performed in any order, in parallel, or simultaneously to reduce lag or increase the speed of the system or method. Furthermore, it should be noted that any of the components, devices, or equipment discussed in relation to the figures above could be used to perform one or more of the steps in FIG. 5.

[0105] The above-described embodiments of the present disclosure are presented for purposes of illustration and not of limitation, and the present disclosure is limited only by the claims which follow. Furthermore, it should be noted that the features and limitations described in any one embodiment may be applied to any embodiment herein, and flowcharts or examples relating to one embodiment may be combined with any other embodiment in a suitable manner, done in different orders, or done in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the systems and/or methods described above may be applied to, or used in accordance with, other systems and/or methods.

[0106] The present techniques will be better understood with reference to the following enumerated embodiments:

1. A method for efficiently determining minimum amounts of data needed in order to accurately validate contact center pairing strategy models.

2. The method of the preceding embodiment, further comprising: obtaining a first set of historical contact-agent interaction data; determining a set of numbers, wherein each number of the set of numbers corresponds to one of a plurality of subset sizes for the first set of historical contact-agent interaction data; validating a first model of a plurality of models based on a folding process and the set of numbers to determine a plurality of mean measurements and a plurality of standard error measurements to generate a first validation result, wherein one of the plurality of mean measurements and one of the plurality of standard error measurements corresponds to each number of the set of numbers; determining a first subset size of the plurality of subset sizes for validation based on a first mean measurement of the plurality of mean measurements and a first standard error measurement of the plurality of standard error measurements, wherein the first mean measurement corresponds to the first subset size, and wherein the first standard error measurement corresponds to the first subset size; validating a second model of the plurality of models based on the folding process and the first subset size to generate a second validation result; selecting the second model of the plurality of models based on the second validation result; and outputting the second model for use in a contact center.

3. The method of embodiment 2, wherein the first mean measurement is based on: determining a first partition of the first set of historical contact-agent interaction data based on the first subset size; determining a first mathematical mean for the first partition; determining a plurality of average distances between points in the first partition and the first mathematical mean; and determining the first mean measurement based on the plurality of average distances between points in the first partition and the first mathematical mean.

4. The method of embodiment 2, wherein the first standard error measurement is based on: determining a first partition of the first set of historical contact-agent interaction data based on the first subset size; determining a first mathematical mean for the first partition; determining a standard deviation between points in the first partition and the first mathematical mean; and determining the first standard error measurement based on multiplying the standard deviation by a square root of a number of points in the first partition.

5. The method of embodiment 2, wherein determining the first subset size of the plurality of subset sizes for validation based on the first standard error measurement further comprises: determining a threshold value for a standard error; and comparing the first standard error measurement to the threshold value to select the first subset size.

6. The method of embodiment 5, wherein determining the first subset size based on the first mean measurement and the first standard error measurement further comprises: determining a value corresponding to the first mean measurement divided by the first standard error measurement; and selecting the first subset size based on the value.

7. The method of embodiment 2, wherein determining the set of numbers further comprises: determining a number of contact-agent interactions in the first set of historical contact-agent interaction data, wherein the first set of historical contact-agent interaction data comprises outcomes associated with contact-agent interactions over a first time period; partitioning the first time period into a plurality of partitions; and determining a number of contact-agent interactions in each of the plurality of partitions, wherein the set of numbers is based on the number of contactagent interactions in each of the plurality of partitions.

8. The method of embodiment 2, wherein determining the set of numbers further comprises: determining an age of the first set of historical contact-agent interaction data; and selecting each number of the set of numbers based on the age.

9. The method of embodiment 2, wherein determining the set of numbers further comprises: determining a location at which the first set of historical contact-agent interaction data is generated; determining a location-specific requirement based on the location; and selecting each number of the set of numbers based on the location-specific requirement.

10. The method of embodiment 2, wherein determining the first subset size of the plurality of subset sizes for validation further comprises: generating for display, on a user interface, a graphic comprising the plurality of mean measurements and the plurality of standard error measurements that correspond to each number of the set of numbers; and receiving a user input selecting the first subset size based on the graphic.

11. The method of embodiment 10, wherein generating for display the graphic further comprises: determining respective color codings for each of the plurality of mean measurements and the plurality of standard error measurements; and generating a color map based on the respective color codings.

12. The method of embodiment 2, wherein validating the first model of the plurality of models based on the folding process and the set of numbers further comprises: partitioning the first set of historical contact-agent interaction data based on the set of numbers, comprising: determining a first plurality of partitions of the first set of historical contact-agent interaction data corresponding to a first number in the set of numbers; and generating the first validation result based on the first plurality of partitions.

13. The method of embodiment 12, wherein partitioning the first set of historical contact-agent interaction data based on the set of numbers further comprises determining a second plurality of partitions of the first set of historical contact-agent interaction data corresponding to a second number in the set of numbers. 14. The method of embodiment 2, further comprising: receiving a request to determine a new subset size; validating the second model of the plurality of models based on the folding process and the set of numbers to determine a new plurality of mean measurements and a new plurality of standard error measurements to generate a second validation result, wherein one of the new plurality of mean measurements and one of the new plurality of standard error measurements corresponds to each number of the set of numbers; determining a second subset size of the plurality of subset sizes for validation based on a second mean measurement of the new plurality of mean measurements and a second standard error measurement of the new plurality of standard error measurements, wherein the second mean measurement corresponds to the second subset size, and wherein the second standard error measurement corresponds to the second subset size; validating a third model of the plurality of models based on the folding process and the first subset size to generate a third validation result; selecting the third model of the plurality of models based on the third validation result; and outputting the third model for use in the contact center.

15. A non-transitory, computer-readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising those of any of embodiments 1-14.

16. A system comprising one or more processors; and memory storing instructions that, when executed by the processors, cause the processors to effectuate operations comprising those of any of embodiments 1-14.

17. A system comprising means for performing any of embodiments 1-14.

Claims

WHAT IS CLAIMED IS:

1. A system for efficiently determining minimum amounts of data needed in order to accurately validate contact center pairing strategy models, the system comprising: one or more processors; and a non-transitory, computer-readable medium having instructions recorded thereon that when executed by the one or more processors cause operations comprising: obtaining a first set of historical contact-agent interaction data, wherein the first set of historical contact-agent interaction data comprises outcomes associated with contact-agent interactions over a first time period; determining a set of numbers, wherein each number of the set of numbers corresponds to one of a plurality of subset sizes for the first set of historical contact-agent interaction data; validating a first model of a plurality of models based on a folding process and the set of numbers to determine a plurality of mean measurements and a plurality of standard error measurements to generate a first validation result, wherein the first model corresponds to a first pairing strategy of a plurality of pairing strategies, and wherein one of the plurality of mean measurements and one of the plurality of standard error measurements corresponds to each number of the set of numbers; determining a first subset size of the plurality of subset sizes for validation based on a first mean measurement of the plurality of mean measurements and a first standard error measurement of the plurality of standard error measurements, wherein the first mean measurement corresponds to the first subset size, and wherein the first standard error measurement corresponds to the first subset size; obtaining a plurality of models, wherein each model of the plurality of models corresponds to a respective pairing strategy of a plurality of pairing strategies for contacts and agents; performing a validation process for each model of the plurality of models, wherein the validation process is based on the first subset size; selecting one of the plurality of models based on the validation process; and establishing a connection between communication equipment of a contact and communication equipment of an agent based on the one of the plurality of models.

2. A method for efficiently determining minimum amounts of data needed in order to accurately validate contact center pairing strategy models, the method comprising: obtaining a first set of historical contact-agent interaction data; determining a set of numbers, wherein each number of the set of numbers corresponds to one of a plurality of subset sizes for the first set of historical contact-agent interaction data; validating a first model of a plurality of models based on a folding process and the set of numbers to determine a plurality of mean measurements and a plurality of standard error measurements to generate a first validation result, wherein one of the plurality of mean measurements and one of the plurality of standard error measurements corresponds to each number of the set of numbers; determining a first subset size of the plurality of subset sizes for validation based on a first mean measurement of the plurality of mean measurements and a first standard error measurement of the plurality of standard error measurements, wherein the first mean measurement corresponds to the first subset size, and wherein the first standard error measurement corresponds to the first subset size; validating a second model of the plurality of models based on the folding process and the first subset size to generate a second validation result; selecting the second model of the plurality of models based on the second validation result; and outputting the second model for use in a contact center.

3. The method of claim 2, wherein the first mean measurement is based on: determining a first partition of the first set of historical contact-agent interaction data based on the first subset size; determining a first mathematical mean for the first partition; determining a plurality of average distances between points in the first partition and the first mathematical mean; and determining the first mean measurement based on the plurality of average distances between points in the first partition and the first mathematical mean.

4. The method of claim 2, wherein the first standard error measurement is based on: determining a first partition of the first set of historical contact-agent interaction data based on the first subset size; determining a first mathematical mean for the first partition; determining a standard deviation between points in the first partition and the first mathematical mean; and determining the first standard error measurement based on multiplying the standard deviation by a square root of a number of points in the first partition.

5. The method of claim 2, wherein determining the first subset size of the plurality of subset sizes for validation based on the first standard error measurement further comprises: determining a threshold value for a standard error; and comparing the first standard error measurement to the threshold value to select the first subset size.

6. The method of claim 5, wherein determining the first subset size based on the first mean measurement and the first standard error measurement further comprises: determining a value corresponding to the first mean measurement divided by the first standard error measurement; and selecting the first subset size based on the value.

7. The method of claim 2, wherein determining the set of numbers further comprises: determining a number of contact-agent interactions in the first set of historical contactagent interaction data, wherein the first set of historical contact-agent interaction data comprises outcomes associated with contact-agent interactions over a first time period; partitioning the first time period into a plurality of partitions; and determining a number of contact-agent interactions in each of the plurality of partitions, wherein the set of numbers is based on the number of contact-agent interactions in each of the plurality of partitions.

8. The method of claim 2, wherein determining the set of numbers further comprises: determining an age of the first set of historical contact-agent interaction data; and selecting each number of the set of numbers based on the age.

9. The method of claim 2, wherein determining the set of numbers further comprises: determining a location at which the first set of historical contact-agent interaction data is generated; determining a location-specific requirement based on the location; and selecting each number of the set of numbers based on the location-specific requirement.

10. The method of claim 2, wherein determining the first subset size of the plurality of subset sizes for validation further comprises: generating for display, on a user interface, a graphic comprising the plurality of mean measurements and the plurality of standard error measurements that correspond to each number of the set of numbers; and receiving a user input selecting the first subset size based on the graphic.

11. The method of claim 10, wherein generating for display the graphic further comprises: determining respective color codings for each of the plurality of mean measurements and the plurality of standard error measurements; and generating a color map based on the respective color codings.

12. The method of claim 2, wherein validating the first model of the plurality of models based on the folding process and the set of numbers further comprises: partitioning the first set of historical contact-agent interaction data based on the set of numbers, comprising: determining a first plurality of partitions of the first set of historical contact-agent interaction data corresponding to a first number in the set of numbers; and generating the first validation result based on the first plurality of partitions.

13. The method of claim 12, wherein partitioning the first set of historical contact-agent interaction data based on the set of numbers further comprises determining a second plurality of partitions of the first set of historical contact-agent interaction data corresponding to a second number in the set of numbers.

14. The method of claim 2, further comprising: receiving a request to determine a new subset size; validating the second model of the plurality of models based on the folding process and the set of numbers to determine a new plurality of mean measurements and a new plurality of standard error measurements to generate a second validation result, wherein one of the new plurality of mean measurements and one of the new plurality of standard error measurements corresponds to each number of the set of numbers; determining a second subset size of the plurality of subset sizes for validation based on a second mean measurement of the new plurality of mean measurements and a second standard error measurement of the new plurality of standard error measurements, wherein the second mean measurement corresponds to the second subset size, and wherein the second standard error measurement corresponds to the second subset size; validating a third model of the plurality of models based on the folding process and the first subset size to generate a third validation result; selecting the third model of the plurality of models based on the third validation result; and outputting the third model for use in the contact center.

15. A non-transitory, computer-readable medium having instructions recorded thereon that when executed by one or more processors cause operations comprising: obtaining a first set of historical contact-agent interaction data; determining a set of numbers, wherein each number of the set of numbers corresponds to one of a plurality of subset sizes for the first set of historical contact-agent interaction data; validating a first model of a plurality of models based on a folding process and the set of numbers to determine a plurality of mean measurements and a plurality of standard error measurements to generate a first validation result, wherein one of the plurality of mean measurements and one of the plurality of standard error measurements corresponds to each number of the set of numbers; determining a first subset size of the plurality of subset sizes for validation based on a first mean measurement of the plurality of mean measurements and a first standard error measurement of the plurality of standard error measurements, wherein the first mean measurement corresponds to the first subset size, and wherein the first standard error measurement corresponds to the first subset size; validating a second model of the plurality of models based on the folding process and the first subset size to generate a second validation result; and selecting the second model of the plurality of models based on the second validation result.

16. The non-transitory, computer-readable medium of claim 15, wherein the first mean measurement is based on: determining a first partition of the first set of historical contact-agent interaction data based on the first subset size; determining a first mathematical mean for the first partition; determining a plurality of average distances between points in the first partition and the first mathematical mean; and determining the first mean measurement based on the plurality of average distances between points in the first partition and the first mathematical mean.

17. The non-transitory, computer-readable medium of claim 15, wherein the first standard error measurement is based on: determining a first partition of the first set of historical contact-agent interaction data based on the first subset size; determining a first mathematical mean for the first partition; determining a standard deviation between points in the first partition and the first mathematical mean; and determining the first standard error measurement based on multiplying the standard deviation by a square root of a number of points in the first partition.

18. The non-transitory, computer-readable medium of claim 15, wherein determining the first subset size of the plurality of subset sizes for validation based on the first standard error measurement further comprises: determining a threshold value for a standard error; and comparing the first standard error measurement to the threshold value to select the first subset size.

19. The non-transitory, computer-readable medium of claim 18, wherein determining the first subset size based on the first mean measurement and the first standard error measurement further comprises: determining a value corresponding to the first mean measurement divided by the first standard error measurement; and selecting the first subset size based on the value.

20. The non-transitory, computer-readable medium of claim 15, wherein determining the set of numbers further comprises: determining a number of contact-agent interactions in the first set of historical contactagent interaction data, wherein the first set of historical contact-agent interaction data comprises outcomes associated with contact-agent interactions over a first time period; partitioning the first time period into a plurality of partitions; and determining a number of contact-agent interactions in each of the plurality of partitions, wherein the set of numbers is based on the number of contact-agent interactions in each of the plurality of partitions.