HK1262894B

HK1262894B - Techniques for estimating expected performance in a task assignment system

Info

Publication number: HK1262894B
Application number: HK19122768.5A
Authority: HK
Inventors: Chishti Zia; Kan Ittai; Khatri Vikash
Original assignee: Afiniti, Ltd.
Priority date: 2017-07-10
Filing date: 2018-04-05
Publication date: 2021-02-05

Description

Techniques for estimating expected performance in a task distribution system

Cross Reference to Related Applications

This international patent application claims priority from U.S. patent application No.15/645,277 filed on 10.7.2017 and U.S. patent application No.15/648,788 filed on 13.7.2017, the entire contents of which are incorporated herein by reference as if fully set forth herein.

Technical Field

The present disclosure relates generally to performance of task distribution systems, and more particularly to techniques for estimating expected performance of task distribution strategies in task distribution systems.

Background

A typical task distribution system distributes a limited number of tasks to a limited number of workers ("agents") over a period of time. One example of a task distribution system is a contact center (e.g., a call center). At a call center, a limited number of agents are available during a given shift or other time period, and a limited number of callers call the call center during the shift. Each caller with various needs and reasons for the call represents a task assigned to one of the call center agents.

Typical task allocation policies determine which tasks are allocated to which agents. In general, task allocation policies result from the insight that certain types of agents perform better on certain types of tasks, and based on these insights, the agents are allocated specific tasks. In the call center example, it is believed that agents proficient in sales should be preferentially assigned to sales queues of callers attempting to purchase, while agents proficient in technical support should be preferentially assigned to technical support queues of callers seeking solutions to technical problems.

While typical task allocation strategies may be effective in improving the performance of typical task allocation systems in some cases, in other cases they may have no substantial impact on performance or at worst reduce performance. In general, examples where a typical task allocation strategy may be inefficient are those where the comparative advantages of agents allocated to different types of tasks are not considered.

In view of the foregoing, it can be appreciated that there is a need for a system that can estimate the expected performance of different task allocation strategies for allocating a limited number of tasks to a limited number of agents over a period of time in a task allocation system.

Disclosure of Invention

Techniques for estimating the expected performance of a task allocation strategy in a task allocation system are disclosed. In one particular embodiment, the techniques may be implemented as a method comprising: receiving, by at least one computer processor communicatively coupled to a task allocation system, a plurality of historical agent task allocations; determining, by the at least one computer processor, a sample of the plurality based on a policy pairing the agent with the task; determining, by the at least one computer processor, an expected performance of the policy based on the samples; outputting, by at least one computer processor, the expected performance; and optimizing, by the at least one computer processor, performance of the task distribution system based on the expected performance.

In accordance with other aspects of this particular embodiment, the task assignment system may be a contact center, and wherein the policy may assign contacts to contact center agents.

According to other aspects of this particular embodiment, the method further comprises oversampling, by the at least one computer processor, the plurality of historical agent task allocations by determining a plurality of samples comprising at least one overlapping historical agent task allocation of the plurality of historical agent task allocations.

According to other aspects of this particular embodiment, the method further comprises determining, by the at least one computer processor, a deviation in the sample; and calculating, by the at least one computer processor, a deviation in the expected performance.

In accordance with other aspects of this particular embodiment, the deviation may be due to a too high proportion of a subset of agents in the sample or a too high proportion of a subset of task types in the sample.

In accordance with other aspects of this particular embodiment, determining the expected performance may include determining, by the at least one computer processor, a plurality of samples of a plurality of historical agent task assignments.

According to other aspects of this particular embodiment, the method may further include dividing, by the at least one computer processor, the plurality of historical agent task assignments into a first subset of historical agent task assignments and a maintained subset of historical agent task assignments different from the first subset; and generating, by the at least one computer processor, a policy based on the first subset, wherein the sample is a subset of the retention subset, and wherein determining the expected performance is based on the retention subset.

According to other aspects of this particular embodiment, the method may further comprise: such that a historical agent task allocation in the retention subset attributable to a second policy for pairing agents and tasks that is different from the policy is prorated.

In accordance with other aspects of this particular embodiment, determining the expected performance may be based on a plurality of retention subsets.

In accordance with other aspects of this particular embodiment, the method may further comprise determining a standard error associated with the expected performance based on the plurality of retention subsets.

In accordance with other aspects of this particular embodiment, the sample may be associated with an amount of performance difference between the expected performance of the sample and the peak performance of the strategy.

In another particular embodiment, the techniques may be realized as a system comprising at least one computer processor communicatively coupled to a task distribution system, wherein the at least one computer processor is configured to perform the steps of the above-described method.

In another particular embodiment, the techniques may be implemented as an article of manufacture comprising a non-transitory computer readable medium; and instructions stored on the medium; wherein the instructions are configured to be readable from the medium by at least one computer processor communicatively coupled to the task distribution system, thereby causing the at least one computer processor to operate to perform the steps of the above-described method.

In another particular embodiment, the techniques may be realized as a method comprising: receiving, by at least one computer processor communicatively coupled to a task allocation system, a plurality of historical agent task allocations; determining, by the at least one computer processor, a weighting of at least one of the plurality based on a policy pairing the agent with the task; determining, by the at least one computer processor, an expected performance of the policy based on the weighting; and outputting, by the at least one computer processor, the expected performance, wherein the expected performance of the strategy demonstrates that the performance of the task distribution system can be optimized if the task distribution system is configured to use the strategy.

In accordance with other aspects of this particular embodiment, the at least one weight may be 0 or a minimum value.

According to other aspects of this particular embodiment, the method may further comprise determining, by the at least one computer processor, a deviation in the sample; and calculating, by the at least one computer processor, a deviation in the expected performance.

In accordance with other aspects of this particular embodiment, the deviation may be due to an overly high proportion of a subset of agents or an overly high proportion of a subset of task types.

In accordance with other aspects of this particular embodiment, determining the expected performance includes combining at least two weighted results corresponding to at least two weighted pairs in the sample.

The present disclosure will now be described in more detail with reference to specific embodiments as illustrated in the accompanying drawings. Although the present disclosure is described below with reference to specific embodiments, it should be understood that the present disclosure is not limited thereto. Those of ordinary skill in the art having access to the teachings herein will recognize additional implementations, modifications, and embodiments, as well as other fields of use, which are within the scope of the present disclosure as described herein, and with respect to which the present disclosure may be of significant utility.

Drawings

For a more complete understanding of this disclosure, reference is now made to the drawings, wherein like elements are designated with like numerals. These drawings should not be construed as limiting the present disclosure, but are intended to be exemplary only.

FIG. 1 shows a block diagram of a task allocation system according to an embodiment of the present disclosure.

Fig. 2 illustrates a block diagram of a contact center according to an embodiment of the present disclosure.

FIG. 3A illustrates a schematic representation of a task assignment model according to an embodiment of the present disclosure.

Fig. 3B illustrates a schematic representation of a contact pairing model according to an embodiment of the disclosure.

Fig. 4A shows a schematic representation of task allocation strategy expected performance estimation according to an embodiment of the present disclosure.

Fig. 4B illustrates a schematic representation of task allocation strategy expected performance estimation in accordance with an embodiment of the present disclosure.

Fig. 4C shows a schematic representation of task allocation strategy expected performance estimation according to an embodiment of the present disclosure.

Fig. 4D shows a schematic representation of task allocation strategy expected performance estimation according to an embodiment of the present disclosure.

Fig. 4E illustrates a schematic representation of task allocation strategy expected performance estimation in accordance with an embodiment of the present disclosure.

Fig. 4F illustrates a schematic representation of task allocation strategy expected performance estimation in accordance with an embodiment of the present disclosure.

Fig. 4G illustrates a schematic representation of task allocation strategy expected performance estimation in accordance with an embodiment of the present disclosure.

Fig. 4H illustrates a schematic representation of task assignment strategy expected performance estimation in accordance with an embodiment of the present disclosure.

Fig. 4I illustrates a schematic representation of task assignment strategy performance expectancy estimation in accordance with an embodiment of the present disclosure.

Fig. 4J illustrates a schematic representation of task allocation strategy expected performance estimation in accordance with an embodiment of the present disclosure.

Fig. 4K shows a schematic representation of task allocation strategy expected performance estimation according to an embodiment of the present disclosure.

FIG. 5 shows a schematic representation of a task allocation spending matrix according to an embodiment of the present disclosure.

Fig. 6A illustrates a flow chart of an expected performance estimation method in accordance with an embodiment of the present disclosure.

Fig. 6B illustrates a flow chart of an expected performance estimation method in accordance with an embodiment of the present disclosure.

Fig. 7 illustrates a flow chart of an expected performance estimation method in accordance with an embodiment of the present disclosure.

Fig. 8 illustrates a flow chart of an expected performance estimation method in accordance with an embodiment of the present disclosure.

Fig. 9 illustrates a flow chart of a method of expected performance estimation in accordance with an embodiment of the present disclosure.

Detailed Description

While typical task allocation strategies may be effective in improving the performance of typical task allocation systems in some cases, in other cases they may have no substantial impact on performance or at worst reduce performance. In general, examples where a typical task allocation strategy may be inefficient are those that do not take into account the comparative advantages of agents allocated to different types of tasks. A general description of the comparative advantages can be found, for example, in Cowen and Tabarok, Modern Principles: Microeconomics,2d ed. (2011), pages 16-19, the entire contents of which are incorporated herein by reference.

FIG. 1 shows a block diagram of a task allocation system 100 according to an embodiment of the present disclosure. The description herein describes network elements, computers, and/or components of systems and methods for estimating the expected performance of a task distribution system that may include one or more modules. As used herein, the term "module" may be understood to refer to computing software, firmware, hardware, and/or various combinations thereof. However, a module should not be construed as software that is not implemented on hardware, firmware, or recorded on a non-transitory processor-readable recordable storage medium (i.e., the module is not software itself). Note that the modules are exemplary. Modules may be combined, integrated, separated, and/or duplicated to support various applications. Further, functions described herein as being performed at a particular module may be performed at one or more other modules and/or by one or more other devices in place of, or in addition to, functions performed at the particular module. Further, modules may be implemented across multiple devices and/or other components, local or remote to each other. Additionally, a module may be removed from one device and added to another device, and/or may be included in both devices.

As shown in FIG. 1, the task assignment system 100 may include a task assignment module 110. The task distribution system 100 may include switches or other types of routing hardware and software to help distribute tasks among the various agents, including queuing or switching components or other internet, cloud, or network based hardware or software solutions.

The task assignment module 110 can receive an incoming task. In the example of FIG. 1, task distribution system 100 receives m tasks, tasks 130A-130m, in a given period of time. Each of the m tasks may be assigned to an agent of the task distribution system 100 for service or other type of task processing. In the example of FIG. 1, n agents are available, agents 120A-120n, within a given time period. m and n may be any large finite integer greater than or equal to 1. In a real-world task distribution system such as a contact center, there may be tens, hundreds, etc. of agents logged into the contact center to interact with the contact during the shift, and the contact center may receive tens, hundreds, thousands, etc. of contacts (e.g., calls) during the shift.

In some environments, referred to as an "L0" environment, one of the m tasks (e.g., task 130A) may be ready to be assigned to an agent, and one of the n agents (e.g., agent 120A) may be ready to receive the assigned task. In the L0 environment, there is no choice of tasks or agents available, and the task allocation policy may allocate the task 130A to the agent 130A.

In other environments, referred to as "L1" environments, one of the m tasks (e.g., task 130A) may be ready to be assigned to an agent, and multiple agents (e.g., agents 120A and 120B) may be ready to receive the assigned tasks. In the L1 environment, there is a choice between multiple available agents, and a task allocation policy may allocate a task 130A to an agent 120A or 120B.

In other environments, referred to as "L2" environments, multiple tasks (e.g., tasks 130A and 130B) may be prepared to be assigned to an agent, and one of the n agents (e.g., agent 120A) may be prepared to receive the assigned task. In the L2 environment, there is a choice between multiple available tasks, and a task allocation policy may allocate a task 130A or 130B to an agent 120A.

In other environments, referred to as "L3" environments, multiple tasks (e.g., tasks 130A and 130B) may be prepared to be assigned to an agent, and multiple agents (e.g., agents 120A and 120B) may be prepared to receive the assigned tasks. In an L3 environment, a selection may be made between multiple available agents and tasks, and a task allocation policy may pair some or all of the available tasks to some or all of the available agents (e.g., in an order of allocation or a single batch allocation).

These environments L0-L3 and the various transitions between them are described in detail for a contact center context, for example, in U.S. patent application No.15/395,469, the entire contents of which are incorporated herein by reference.

In some embodiments, the task allocation policy module 140 may be communicatively coupled and/or configured to operate in the task allocation system 100. The task allocation policy module 140 may implement one or more task allocation policies (or "pairing policies") for allocating individual tasks to individual agents (e.g., pairing contacts with contact center agents).

Various different task allocation policies may be designed and implemented by the task allocation policy module 140. In some embodiments, a first-in/first-out ("FIFO") policy may be implemented in which, for example, the longest waiting agent receives the next available task (in the L1 environment), or assigns the longest waiting task to the next available task (in the L2 environment). Other FIFOs and FIFO-like strategies may be allocated without relying on information specific to a single task or a single agent.

In other embodiments, a performance-based routing (PBR) policy may be used to optimize the higher performance agents for task allocation. For example, under PBR, the next available task is received with the highest performing agent among the available agents. Other PBR and PBR-like policies may be assigned using information about a particular agent, but do not necessarily rely on information about a particular task or agent.

In other embodiments, a Behavioral Pairing (BP) policy may be used to optimally assign tasks to agents using information about specific tasks and specific agents. Various BP policies may be used, such as a diagonal model BP policy or a network flow BP policy. These and other task assignment strategies are described in detail in the context of contact centers, for example, in U.S. patent No.9,300,802 and U.S. patent application No.15/582,223, the entire contents of which are incorporated herein by reference.

In some embodiments, the history allocation module 150 may be communicatively coupled to the task allocation system 100 and/or configured to operate in the task allocation system 100 via other modules such as the task allocation module 110 and/or the task allocation policy module 140. The history allocation module 150 may be responsible for various functions, such as monitoring, storing, retrieving and/or outputting information regarding the allocation of agent tasks that have been performed. For example, the historical allocation module 150 may monitor the task allocation module 110 to collect information about task allocations in a given period of time. Each record of historical task assignments may include information such as an agent identifier, a task and/or task type identifier, and result information.

In some embodiments and for some contexts, additional information may be stored. For example, in a call center context, the history allocation module 150 may also store information about the time the call started, the time the call ended, the dialed phone number, and the caller's phone number. As another example, in a dispatch center (e.g., "door service (truck roll)") context, the history allocation module 150 may also store information about the time the driver (i.e., the on-site seat) left the dispatch center, the recommended route, the route taken, the estimated travel time, the actual travel time, the time spent handling the customer tasks at the customer site, and the like.

In some embodiments, the historical allocation module 150 may generate a pair model or similar computer processor-generated model based on a set of historical allocations over a period of time (e.g., the past week, the past month, the past year, etc.), which may be used by the task allocation policy module 140 to issue task allocation suggestions or instructions to the task allocation module 110. In other embodiments, the historical allocation module 150 may send the historical allocation information to another module, such as the task allocation policy module 140 or the expected performance estimation module 160, to generate the pairing model and/or pairing policy based on the pairing model.

In some embodiments, the expected performance estimation module 160 may be communicatively coupled to the task distribution system 100 and/or configured to operate in the task distribution system 100 via other modules such as the task distribution module 110 and/or the history distribution module 150. The expected performance estimation module 160 may estimate the expected performance of the task allocation strategy (e.g., in conjunction with the pairing model) using, for example, historical allocation information received from the historical allocation module 150. Techniques for estimating expected performance and other functions performed by the expected performance estimation module 160 for various tasks assigned policies and various contexts are described throughout subsequent portions of this disclosure.

In some embodiments, the expected performance estimation module 160 may output or otherwise report or use the estimated expected performance. The estimated expected performance may be used to evaluate the quality of the task allocation strategy to determine, for example, that a different task allocation strategy (or a different pairing model) should be used, or to predict an expected overall performance (or performance gain) achieved within the task allocation system 100 when optimized or otherwise configured to use the task allocation strategy.

As described above, various contexts may use embodiments similar to task assignment system 100, including but not limited to contact centers and dispatch centers. One such example for a contact center is described below with reference to fig. 2.

Fig. 2 illustrates a block diagram of a contact center 200 according to an embodiment of the disclosure. The contact center 200 is similar to the task assignment system 100 (fig. 1) as long as it is a dedicated context for assigning tasks (i.e., "contacts") to agents in the contact center. In an incoming environment, a call is contacted or otherwise connected to a switch or other component of the contact center 200 (via, for example, real-time text chat, video chat, email, social media). In an outgoing environment, contacts may be called (or called back) or otherwise connected via an outgoing dialer or other component of the contact center 200 and assigned to an agent simultaneously or subsequently.

Similar to task assignment system 100, contact center 200 has n agents 220A-220n and m contacts 230A-230m, which are assigned to agents in a given time period. A switch 210 or similar routing component, such as a PBX/ACD or load balancer, may connect the contacts to the agents.

Similar to the task allocation policy module 140, the contact pairing policy module 240 (e.g., the BP module and/or the benchmarking module) may issue pairing suggestions or instructions to the switch 210 according to the contact pairing policy.

Similar to the historical assignment module 150, the historical contact pairing module 250 may monitor, store, retrieve, and/or output information regarding agent contact pairings that have been made. The historical contact pairing module 250 can generate a pairing model that can be used by the contact pairing policy module 240 to issue task assignment suggestions or instructions to the switch 210. In other embodiments, the historical contact pairing module 250 may send historical assignment information to another module, such as the contact pairing policy module 240 or the expected performance assessment module 160, to generate a pairing model for use with the pairing policy. Fig. 3A and 3B, described in detail below, illustrate examples of such pairing models for a simplified task distribution system.

FIG. 3A illustrates a schematic representation of a task assignment model 300A according to an embodiment of the present disclosure. The task assignment model 300A simulates, for exemplary purposes, three agents a₀-a₂And three task types t₀-t₂A simple task distribution system. In some embodiments, the agents and/or task types may be ordered according to some information about the agents or task types (e.g., a diagonal BP model). In other embodiments, the agents and task types may appear in a model without a particular ordering (e.g., a spending matrix or a network flow BP model).

In the task assignment model 300A, each unit representation has a particular task type t_iAnd a specific seat a_kIs possible. Each cell contains an interaction term (function) g (a) of agent and task type_k,t_i). For example, showing the task type t₁Assigned to agent a₂To have an interactive item g (a)₂,t₁). The particular function definition for any given interaction item depends on the task allocation policy to be used, the context of the task allocation system, the data available to a given agent, and any of the tasks used to construct the task allocation model 300AType of service, etc.

In some embodiments, the interaction term may represent a cost or value of a particular pairing (e.g., an expected conversion rate in a sales queue, an expected customer satisfaction rating in a customer support queue, an expected cost of top-door services in a dispatch center, etc.). In each case, the expected value may be estimated or otherwise determined using a combination of information about the agent and task type.

FIG. 3B illustrates a schematic representation of a contact pairing model 300B according to an embodiment of the disclosure. Similar to the task assignment model 300A (FIG. 3A), the contact pairing model 300B simulates, for exemplary purposes, having three agents a₀-a₂And three contact types c₀-c₂Which may be a task in the context of a contact center. Each element of the contact pairing model 300B indicates a value for an interaction term for each possible pairing of an individual agent and contact type. The contact pairing model 300B may be adapted for use with a diagonal BP strategy as described in U.S. Pat. No.9,300,802, the entire contents of which are incorporated herein by reference.

Consistent with the diagonal model, the preferred pair falls along the equivalent of a 45 ° diagonal through "y ═ x" linking the pair model 300B, i.e.: a is₀And c₀，a₁And c₁，a₂And c₂. Each of these cells found along the diagonal is shown with an interactive item evaluating 1.

In some cases, optimal selection of an agent (L1) or contact (L2) is not always available for selecting a preferred agent for each contact (L1) or an optimal contact for each agent (L2). In cases where the selection is limited, the pairing strategy may select the best available option. For example, the next best pair as shown in the connected pairings model 300B is in a unit relatively close to the y-x diagonal. Each of these cells is shown with an interaction term that evaluates to 0.5. The farther away from the ideal diagonal is the less preferred pairing: a is₀And c₂And a₂And c₀Both have an interaction term that evaluates to 0.

A pairing model, such as contact pairing model 300B for a contact center, is one exemplary context for a task allocation system. Other contexts mentioned above include service technicians who dispatch home services from dispatch centers, and consultants who consult companies to be responsible for a particular project. Further examples of contexts include case assignment (e.g., insurance claims) to agents (e.g., insurance claim adjusters) and identifying retail customers for pairing with individual store personnel in a retail store. These contexts are used as example task allocation systems, and embodiments of the present disclosure are not limited to these contexts.

Unlike task assignment model 300A and contact pairing model 300B, typical work assignment strategies assign workers (or "agents") to tasks based on a potentially naive intuition. For example, in a contact center, tasks may be assigned to agents based on agent performance (e.g., PBR). Similarly, a service technician may be assigned a truck route calculated to minimize the length of the truck route. As another example, the consulting firm may assign employees to particular projects based on their tenure.

In all of these cases, what is lacking is to assess the impact of assigning individual workers to individual tasks on the remaining workers and tasks within the overall system. For example, in a contact center environment, assigning one task to a high performance agent may require assigning a different task to a low performance agent without proper consideration of the types of tasks assigned to the high and low performance agents. Overall, overall contact center performance is reduced. Similarly, assigning a particular route to a service technician may result in assigning other routes to other service technicians, thereby reducing overall system performance.

As a result, while a single agent does perform the assigned task well, another agent assigned the remaining tasks may complete quite poorly. Counter to intuition, attempting to optimize each task assignment independently may actually decrease the overall performance of the task assignment system rather than increase it.

Embodiments of the present disclosure provide techniques for more accurately estimating performance of a task allocation strategy. These techniques assign tasks to agents in a manner that accounts for the cumulative impact of subsequent assignments over a period of time to optimize overall system performance, rather than optimizing performance at any individual assigned task. For example, a task allocation policy may take advantage of the use of certain agents for certain types of tasks, while using other agents for other types of tasks.

Fig. 4A-4K show schematic representations of expected performance estimations for stepping through task allocation strategies for several techniques for estimating performance estimations. These examples mainly illustrate the estimation of the expected performance of a diagonal model BP strategy in a task distribution system under different conditions. These techniques use historical task allocation data to estimate expected performance and validate policies.

In particular, as described in more detail below, FIGS. 4A-4C illustrate a validation technique that uses historical allocation data to estimate the expected performance of a diagonal BP model. Fig. 4E and 4F additionally illustrate techniques for improving the accuracy of the estimation techniques described with reference to fig. 4A-4C when an agent selection bias is present in the basic historical allocation data. Fig. 4G and 4H additionally illustrate techniques for improving the accuracy of the estimation techniques described with reference to fig. 4A-C when there is a task selection bias in the basic historical allocation data. Fig. 4I and 4J additionally illustrate techniques for comparing the estimated expected performance of a validated task allocation policy with a task allocation policy that results in basic historical allocation data. Finally, FIG. 4K additionally illustrates a technique for visualizing the discretionary distributions found in the underlying historical allocation data to improve the realistic expectation of the estimated expected performance of the task allocation strategy being validated.

Fig. 4A illustrates a schematic representation of a task allocation strategy expected performance estimate 400A in accordance with an embodiment of the present disclosure. In the task distribution system, there may be any finite number of agents and task types. Under the BP policy diagonal model, each agent may be assigned an agent percentile between 0 and 1, and each task type may be assigned a task percentile between 0 and 1. The ideal or optimal pairing that the expected diagonal BP strategy operates at peak performance under ideal selection is shown as the dashed diagonal along y ═ x (i.e., the line where the task percentile is equal to the seat percentile).

Historical allocation data may be received. In the schematic representation of the expected performance estimate 400A, each historical allocation is shown on the graph. The dots indicated with the letter "O" represent pairs with desired results, and the dots indicated with the letter "X" represent pairs with undesired results. For example, in a contact center sales queue, "O" may indicate a historical task that sells, and "X" may indicate a historical task that does not sell (a binomial result variable). In other examples, multiple or continuous result variables may be used.

If the task allocation system uses a FIFO pairing strategy or another substantially random or relatively evenly distributed pairing strategy, a set of historical allocations may be distributed relatively evenly throughout a graph, such as the expected performance estimate 400A. In other words, under a FIFO, there is an equal probability of allocating a task for a given type to any agent. Some history assignments appear to be near the diagonal (preferred pairs under the diagonal BP strategy), while other history assignments appear to be far from the diagonal (non-preferred pairs), and so on.

In the expected performance estimate 400A, a complete set of 29 historical assignments is shown. The set contains 12 pairs with desired outcome (O) and 17 pairs with undesired outcome (X). In some embodiments, the estimated expected performance of a baseline or basic task allocation policy (e.g., FIFO in this example) may be calculated as the proportion of expected results found in the set: 12/29 ≈ 41%.

The diagonal BP strategy attempts to improve other pairing strategies by pairing tasks preferentially to the most similar agents in the percentile ranking, so that if these pairs are plotted on a graph, they will be as close as possible to the y-x diagonal (i.e., the task percentile-the agent percentile).

In some embodiments, historical assignment data may be used to estimate (e.g., verify) the expected performance of a task assignment strategy. In some embodiments, verification may be performed using the same historical assignments used to build the pairing model. In other embodiments, a pairing model may be built using a set of historical assignments, and the model may be validated using a different set of historical assignments.

The insight to validate the task allocation policy using historical allocation data is to sample (or weight, etc.) the set of historical allocations based on the likelihood of the task allocation system running the validated pairing policy, rather than the underlying pairing policy that produced the historical allocation, occurring.

In the case of the diagonal BP strategy, there is a convenient geometric representation of the authentication technique used in some embodiments and illustrated by fig. 4B and 4C. That is, the closer a history pair is to an ideal diagonal, the more likely it is that such a history pair will occur using a verified diagonal BP policy. Partitions may be established to exclude historical assignments from the sample that exceed the diagonal by a certain distance.

For a good diagonal pairing model, there should be proportionally more desirable results within the sample because the acceptable distance (threshold distance or threshold "closeness of fit") of the sample is closer to the diagonal. Fig. 4A shows the entire verification set of history pairings, fig. 4B shows relatively large samples (relatively far acceptable/threshold distances from the diagonal), and fig. 4C shows relatively narrow samples (relatively short acceptable/threshold distances from the diagonal).

Fig. 4B illustrates a schematic representation of a task allocation strategy expected performance estimate 400B in accordance with an embodiment of the present disclosure. In the expected performance estimate 400B, a map of the historical tasks is again shown, except that two regions beyond a specified distance from the diagonal have been grayed out. In the expected performance estimate 400B, the remaining samples contain 21 historical assignments, including 10 pairs with the expected outcome (O). Thus, the estimated expected performance of the diagonal BP strategy given this distance by the ideal peak performance may be determined to be 10/21 ≈ 48%.

Comparing the expected performance of 48% with the basic performance of 41% as shown by the reference expected performance estimate 400A (fig. 4A), the diagonal BP strategy provides an expected improvement (or expected "gain") of about 17% relative to the basic strategy. The expected gain estimates assume that the typical (limited) number of choices available in the mission distribution system tends to produce pairings distributed throughout the relatively wide pairing band.

Fig. 4C illustrates a schematic representation of a task allocation strategy expected performance estimate 400C in accordance with an embodiment of the present disclosure. The expected performance estimate 400C illustrates a narrower band (i.e., a shorter acceptable distance from the diagonal to include in the sample) than the expected performance estimate 400B (fig. 4B) and a larger exclusion area. In the expected performance estimate 400C, a subset of the 12 historical assignments contains 8 pairs with the expected result (O). Thus, for this (less restrictive) selection amount, the estimated expected performance of the diagonal BP strategy can be determined as 8/12 ≈ 67%, or about 63% gain relative to the basic strategy, which tends to produce pairings distributed throughout this relatively narrow band.

In the example of fig. 4A-4C, as the sample band narrows, the expected performance increases, indicating that the diagonal BP model may be effective to optimize or otherwise increase the performance of the task allocation system as compared to the basic task allocation strategy (e.g., FIFO).

In some embodiments, any number of samples may be measured at different distances from the diagonal, starting with the full set and ending with an infinitesimal distance from the diagonal.

As the bands narrow, infinitesimal bands (representing the desired, best-chosen peak performance for each pairing) near the diagonal are closer, excluding more and more historical tasks, as the diagonal BP strategy will not likely produce a pairing (too far from the preferred pairing closer to the diagonal). This effect is evident in fig. 4A-4C: fig. 4A includes 29 historic allocations (complete set), fig. 4B samples 21 historic allocations, and fig. 4C only samples 12.

Fig. 4D illustrates a schematic representation of a task allocation strategy expected performance estimate 400D in accordance with an embodiment of the present disclosure. The expected performance estimate 400D shows a correlation curve when the expected performance (e.g., expected result to sample size ratio) is plotted against varying degrees of threshold fit closeness. The correlation curve starts near the top left and falls off to the bottom right, indicating that the performance of the sample increases as the threshold fit proximity narrows towards a preferred task assignment strategy (e.g., pairing tasks with agents that fall near the diagonal).

In a real-world task allocation system using the BP task allocation strategy, the actual task allocation can be found in the entire pairing space, where many pairs are relatively close to the optimal diagonal and other pairs are relatively far from the optimal diagonal. Fitting the proximity distribution assignments with varying degrees of threshold can help estimate how well the BP task assignment strategy achieves peak performance. Comparing this realistic distribution (e.g., BP "On" data) to the expected performance estimate 400D may allow for increasingly accurate estimates of overall expected performance. In some embodiments, the expected performance may be a simple average of the performance of the samples across different degrees of closeness-of-fit. In other embodiments, the expected performance may be a weighted average of the performance of the samples weighted according to the distribution of pairs found in the set of "On" data across different degrees of closeness of fit and when the task distribution system uses a preferred pairing strategy (e.g., BP strategy). In other embodiments, other formulas may be applied to estimate the expected performance or gain in the task distribution system using the correlation curve of the expected performance estimate 400D.

As more and more historical tasks are excluded from narrower and narrower samples, the amount of data available to determine the estimation gain decreases, and therefore the accuracy of the estimation also decreases. Indeed, at some sufficiently narrow band, all historical allocation data may be excluded, resulting in an empty sample with indeterminate ("0/0") expected performance.

Thus, in some embodiments, it may be advantageous to calculate an expected accuracy (e.g., standard error) of the expected performance estimate for each band. In these embodiments, accuracy or error information may be used to help assess the reliability of a given expected performance estimate.

As described above, in these embodiments, any number of estimates and precision/error measurements may be made for narrower and narrower bands, excluding more and more historical tasks as the bands get closer to the optimal pairing until no more remain.

The examples shown in fig. 4A-4C use a single set of historical data. In some embodiments, the verification process may be repeated for any number of different sets of historical data. In some embodiments, each of the one or more historical allocation sets may be a "holding set" or a "validation set" that is completely or at least partially different from the "training set" used to create the pairing strategy. In other embodiments, some or all of the historical allocation sets may be "in-sample sets" taken from a larger historical allocation set used to create the pairing policy.

In some embodiments, for a retention set or an intra-sample set, these samples may be oversampled, counting a single historical allocation multiple times across multiple samples for situations where such oversampling may be used to determine a more accurate expected performance estimate. As a simple example, if the retention set contains 5 historic allocations labeled A-E, the first sample may contain historic allocations A, B, and C; the second sample may contain historical allocations B, C, and D; the third sample may contain historical assignments C, D, and E, among others. In this example, the oversampled historical allocation, B, is included in at least the first and second samples. In another example, such as when close oversampling is fitted to the reduction threshold, the first (largest) sample may contain historical allocations a-E; the second (smaller) sample may contain historical allocations B-D, and so on, until the smallest sample may contain only historical allocation B.

Embodiments of the present disclosure may use one or more of a variety of different verification techniques, such as k-fold, random, time, random roll, offline data, training on/off, training off, life curve, and so forth. In some embodiments, the verification technique may also include normalization to time-based changes in the call center system environment. Generalized k-fold verification techniques and some of these other techniques are described in detail, for example, in James et al, An Introduction to Statistical Learning (2013), pages 29-33 and 176-86 and Hastieet al, The Elements of Statistical Learning,2d ed. (2008), pages 219-57, The entire contents of which are incorporated herein by reference.

In the previous example, the basic task allocation policy may be a FIFO, which results in a relatively evenly distributed historical allocation throughout the possible pairing space. However, in some task allocation systems, different task allocation policies may be used that introduce utilization deviations to agents and/or tasks. Fig. 4E and 4F show examples of agent utilization deviations caused by PBR, and fig. 4G and 4H show examples of task utilization deviations due to task optimization strategies (e.g., optimized or "VIP" call routing in a call center).

Fig. 4E shows a schematic representation of a task allocation strategy expected performance estimate 400D in accordance with an embodiment of the present invention. In this example, the majority of historical allocations are clustered to the right of the graph, surrounding agents with higher percentile ranks. In this example, the agent percentile is proportional to the agent program performance, and this agent utilization deviation is due to the use of the PBR as the base pairing policy for the history allocation. In other embodiments or contact center environments, percentiles may correspond to metrics other than performance. For example, the agent percentile may be proportional to the capabilities of each agent to affect the outcome of the call, regardless of the overall performance of the agent (e.g., the amount of revenue generated).

Thus, a lower percentile of agents may obtain a smaller proportion of data. Thus, if left uncorrected, the bias may also skew or bias the expected performance estimate, as explained below with reference to fig. 4F.

Fig. 4F illustrates a schematic representation of a task allocation strategy expected performance estimate 400E in accordance with an embodiment of the present disclosure. The expected performance estimate 400E illustrates the same historical allocation as the expected performance estimate 400D, wherein out-of-band, off-diagonal regions have been excluded.

The subset of historical jobs that are included also has a biased distribution, favoring higher ranked agents. A naive approach to estimating the expected performance of the band would indicate that there are 9 expected results in a total of 11 historical agent tasks. However, many of these desired results are clustered around highly performing agents, and thus may result in unrealistically high expected performance assessments.

Thus, in some embodiments, expected performance may be more accurately estimated by weighting each vertical time slice of historical allocations proportionally according to the number of historical allocations found within a given time slice. Re-weighting the subset of historical assignments (i.e., "agent percentile correction" or "AP correction") in this manner may eliminate bias from the basic PBR policy, resulting in a more accurate estimate of expected performance.

Fig. 4G illustrates a schematic representation of a task allocation strategy expected performance estimate 400F in accordance with an embodiment of the present disclosure. In this example, most of the historical allocations are clustered to the top of the graph, surrounding the tasks with the higher percentile rank. In this example, the task utilization bias is due to the use of a task prioritization policy as the base pairing policy for history allocation.

Thus, the data available for the lower percentile tasks is proportionally reduced. Thus, if left uncorrected, the bias may also skew or bias the expected performance estimate, as explained below with reference to fig. 4H.

Fig. 4H illustrates a schematic representation of a task allocation strategy expected performance estimate 400G in accordance with an embodiment of the present disclosure. The expected performance estimate 400G illustrates the same historical allocation as the expected performance estimate 400F, wherein out-of-band regions away from the diagonal have been excluded.

The included subset of historical assignments also has a biased distribution, favoring higher ranked tasks. A naive approach to estimating the expected performance of the band would indicate that there are 9 expected results in a total of 11 historical agent tasks. However, many of these desired results are clustered around highly performing agents, thus leading to unrealistically high expected performance estimates.

Thus, in some embodiments, expected performance may be more accurately estimated by weighting each horizontal time slice of historical allocations proportionally according to the number of historical allocations found within a given time slice. Re-weighting a subset of the historical allocations in this manner (i.e., "task percentage correction" or "TP correction," or for a contact center context, "contact percentile correction" or "CP correction") can eliminate bias from the basic task/contact prioritization policy, yielding a more accurate estimate of expected performance.

In some embodiments, it may be used to measure how well (or how poor) the basic task allocation strategy performs relative to the task allocation strategy optimization under validation. Fig. 4I and 4J illustrate an example of one such technique applied to a diagonal BP pairing strategy.

Fig. 4I illustrates a schematic representation of a task allocation strategy expected performance estimate 400H in accordance with an embodiment of the present disclosure. The x-axis is labeled "achievement of peak performance" and proceeds from "high" near the origin to "low" to the right. For the diagonal BP model, a high achievement of peak performance indicates that the sampling history allocation is relatively close to the ideal best diagonal at which the hypothetical peak performance can be achieved. A low achievement of peak performance indicates a sampling history allocation in a wider band spread away from the diagonal.

The y-axis is labeled "sample size" and proceeds from 0 near the origin to the top n (here, the size of the complete set of n historical allocations). For the diagonal BP model, the sample size shrinks as the band size becomes narrower. As the band size approaches peak performance along the diagonal, the sample size eventually drops to 0. As the band size approaches low performance, the sample size eventually reaches n, encompassing the entire historical allocation set.

If the basic task allocation strategy is FIFO, whereby the historical allocation can be distributed relatively evenly across each band, it is often the case that as the width decreases towards peak performance, the sample size decreases in proportion to the square of the width of the band. Thus, the FIFO is a neutral strategy to represent a baseline level of performance within the task distribution system. In the expected performance estimate 400H, the dashed curve represents the baseline performance level for each sample size.

In this example, the solid curve curves "over" the dashed baseline performance curve, causing a relatively high achievement (relatively narrow band) of sample size to peak performance to increase rapidly. In these environments, the historical allocation appears to be proportionally closer to the diagonal cluster. In these environments, the base strategy that generates the historical assignments may have achieved a relatively high implementation of peak performance when compared to the task assignment strategy being validated. Therefore, the expected gain of the basic (existing) strategy may be relatively low compared to the FIFO.

Fig. 4J illustrates a schematic representation of a task allocation strategy expected performance estimate 400I according to an embodiment of the present disclosure. The expected performance estimate 400I is similar to the expected performance estimate 400H (fig. 4I), except that in this example, the curve curves "below" the dashed baseline performance curve, such that the sample size slowly increases for a relatively high achievement (relatively narrow band) of peak performance. In these environments, historical allocations appear to be proportionately clustered in regions away from the diagonal. In these environments, the basic strategy of generating historical assignments may perform less well than a FIFO, and mathematically may be equivalent to executing a strategy that performs less well than randomly. Therefore, the expected gain of the basic (existing) strategy may be relatively high compared to the FIFO.

In some embodiments, it may be used to estimate or otherwise predict which band or ranges are most likely to simulate a "real world" task allocation system. In real-world task distribution systems, the task distribution system will experience different degrees of possible choice over time, as the supply of available agents to meet the needs of waiting tasks constantly fluctuates. The real-world distribution of choices may peak between the ideal best choice (peak performance) and a fully constrained limited choice (e.g., always one-to-one or "L0").

FIG. 4K illustrates a technique for visualizing the freedom-of-choice distributions found in the underlying historical allocation data to improve the real-world expectation of the estimated expected performance of the task allocation strategy being validated.

Fig. 4K illustrates a schematic representation of a task allocation strategy expected performance estimate 400J in accordance with an embodiment of the present disclosure. In these embodiments, information about the distribution or frequency of selection to varying degrees may be associated with historical allocation information, and a band or range of bands may be determined to more closely simulate a real-world task allocation system than other bands/samples. After identifying the band or band range, the embodiments may output an estimated expected performance and/or standard error for the identified band or band range as the most likely expected performance.

In some embodiments, a workforce recommendation may be made to increase or decrease agent staffing to increase the average selection amount of pairing strategies available to select an agent or task, respectively. In some embodiments, the pairing strategy may delay pairing when the amount of available selections is deemed too low, thereby providing time for more tasks and/or agents to become ready for pairing.

The above examples mainly discuss the diagonal BP strategy. In some embodiments, the expected performance may be estimated with other types of BP policies, such as network flow BP policies.

FIG. 5 shows a schematic representation of a task allocation spending matrix 500 according to an embodiment of the present disclosure. Similar to the task assignment model 300A (FIG. 3A) and the contact pairing model 300B (FIG. 3B), the spending matrix 500 shows three agents a₀-a₂And three task types t₀-t₂. Each cell contains an interaction item g (a) for a given agent and task type assignment_i,t_k) The value of (c).

In this example, three preferred pairs have the expected value of 1 and are diagonally downward, while other less preferred pairs have an expected value of 0.5 or 0. Unlike the example of the contact pair model 300B (fig. 3B), the expected value of a pair does not continue to decrease as the pair is farther from the diagonal. For example, task t₁With seat a₂Is paired (expected value 0) to task t₂With seat a₂The pair of (expected value 0.5) is closer to the diagonal.

In some embodiments, the diagonal pairing strategy may provide excellent performance improvement and balanced agent utilization even for environments such as the highly simplified illustrative example of the task allocation disbursement matrix 500. In other embodiments, such as to an environment similar to the task allocation spending matrix 500 in which no agent and ordering of tasks provide a consistent increase in expected values as the pair distance from the diagonal becomes shorter (i.e., the fit closeness is closer) -different pairing strategies, such as network flow (or linearly programmed) BP strategies, may provide better performance optimization while still achieving a desired balance of agent utilization. The task allocation spending matrix 500 may be used in conjunction with a network flow BP strategy, as described, for example, in U.S. patent application No.15/582,223, the entire contents of which are incorporated herein by reference.

As with the diagonal BP policy, historical allocation data may also be used to validate the network flow BP policy. However, because there is no geometric analogy to the network flow BP policy, such as distance from the diagonal in the diagonal BP policy, it is not intended that these verification techniques be illustrated in the figures as in fig. 4A-4H. In some embodiments, historical assignments may be sampled (or oversampled) repeatedly at one or more approximations of the peak performance of the network flow BP policy, given a desired or optimal selection amount. For example, the historical allocation sample size may be determined to be 3 (or 4 or greater).

For example, a sample of historical allocations may include a₀To t₂Preferred pairing with desired result, a₁To t₁Non-preferred pairing with the desired result, and a₂To t₀Non-preferred pairing with an undesired result. The base performance of the sample can be determined as the proportion of the expected result in the sample, or 2/3 ≈ 67%.

Assuming a selection of 3 as in sample size 3, non-preferred pairings according to network flow 500B may be excluded. Thus, only one historical allocation, a with desired results₁To t₁In this sample, or 1/1 ═ 100%.

In these embodiments, the amount of data available in any one sample is highly constrained (e.g., only 3 historical allocations) compared to tens or thousands of historical allocations that can be found in the corpus. Therefore, it is particularly important to repeatedly sample historical assignments to determine the frequency of preferred pairs with desired outcomes and the weighted proportion of those preferred pairs.

As with the diagonal BP strategy, it is useful to determine whether the base strategy has been partially optimized and/or to correct an excessive proportion or deviation of agents and/or tasks in the historical allocation data.

Although the techniques vary in detail depending on the type of task allocation policy being created and verified (e.g., diagonal BP policy verification techniques as compared to network flow BP policy verification techniques), these techniques may involve sampling or weighting historical allocation data. In some embodiments, these techniques may involve oversampling, bias correction, approximation of peak performance or best choice, expected performance estimation, expected gain estimation (compared to basic historical performance), accuracy/error measurement, expected real-world gain estimation (e.g., compared to basic distributions of available choices), and so forth. Fig. 6A-9 illustrate flow charts of the various performance estimation approaches described above.

Fig. 6A illustrates a flow diagram of an expected performance estimation method 600A in accordance with an embodiment of the present disclosure. At block 610, the performance estimation method 600A may begin.

At block 610, a plurality of historical assignments may be received. In some embodiments, the plurality of historical assignments may be received from a historical assignment module, such as historical assignment module 150 (FIG. 1) or historical contact pairing module 250 (FIG. 2)). In some embodiments, the plurality of historical assignments may be within a sample of a plurality of historical assignments, which are also used to build, generate, construct, train, refine, or otherwise determine a task assignment (pairing) model or task assignment (pairing) strategy. In other embodiments, the plurality of historical allocations are off-sample (retention set, validation set, etc.) that include historical allocations that are not used for model building. After receiving the plurality of historical assignments, the method 600A of performance estimation in anticipation may proceed to block 620A.

At block 620A, a sample of a plurality of historical allocations based on a policy for pairing agents with tasks may be determined. In some embodiments, the samples may be determined by an expected performance estimation module, such as the expected performance estimation module 160 (fig. 1 and 2). In some embodiments, the sample may be determined by analyzing a plurality of historical assignments, including historical assignments in the sample that are sufficiently likely to occur using the task assignment policy, and excluding historical assignments from the sample that are unlikely to occur. For example, in the case of a diagonal BP strategy, pairs that fall within a band or a certain distance from the diagonal are included. After determining a plurality of historically assigned samples based on the pairing strategy, the expected performance estimation method 600A may proceed to block 630A.

At block 630A, an expected performance of the sample-based pairing strategy may be determined. In some embodiments, the determination or estimate of expected performance may be calculated as a ratio of the expected result (e.g., a positive result for a binomial variable, or a sufficiently positive result for a polynomial or continuous variable) to the total number of historical pairings in the sample. In other embodiments, the expected performance estimate may be corrected or otherwise adjusted to account for deviations present in the basic historical allocation data, such as an excessively high proportion of a subset of agents (e.g., under PBR) or an excessively high proportion of a subset of task types (e.g., under a task prioritization policy). In some embodiments, the expected performance estimate may take into account the frequency of over-sampling some historical allocations (e.g., multiple counts within the same sample, according to a sample determination technique). After determining the expected performance of the pairing strategy, the expected performance estimation method 600A may proceed to block 640.

At block 640, the determined or estimated expected performance may be output. In some embodiments, the expected performance estimation module 160 may output the estimated expected performance to the task distribution module 110 (fig. 1), the switch 210 (fig. 2), or another component or module within the task distribution system, or otherwise communicatively coupled to the task distribution system.

In some embodiments, after outputting the estimated expected performance determined at block 630, the expected performance estimation method 600A may return to block 620 to determine a different sample. In other embodiments, the expected performance estimation method 600A may return to block 610 to receive a different plurality of historical assignments (e.g., within different samples or different retention sets). In other embodiments, the expected performance estimation method 600A may end.

Fig. 6B illustrates a flow diagram of an expected performance estimation method 600B in accordance with an embodiment of the present disclosure. The expected performance estimation method 600B is similar to the expected performance estimation method 600A except that historical assignments may be weighted instead of being included or excluded in the sample.

The expected performance estimation method 600B may begin at block 610. At block 610, a plurality of historical assignments may be received, as in the expected performance estimation method 600A. After receiving the plurality of historical assignments, the method 600B of performance estimation in anticipation may proceed to block 620B.

At block 620B, a weighting of at least one of a plurality of historical allocations based on a policy for pairing agents with tasks may be determined. In some embodiments, a weighting may be assigned to all historical assignments received at block 610. In some embodiments, a zero weighting may be assigned to some historical assignments. In these embodiments, zero weighting may be similar to excluding historical pairs from the samples as in block 620A (fig. 6A) of the expected performance estimation method 600A. In some embodiments, the weighting may indicate a likelihood that a historical allocation will occur under the verified pairing policy. After determining the weighting for at least one of the plurality of historical assignments, the expected performance estimation method 600B may proceed to block 630B.

At block 630B, the expected performance of the weighting-based policy may be determined. For example, the expected performance may be calculated as a weighted average of the values associated with each weighted historical assignment. As in block 630A of the expected performance estimation method 600A, the expected performance estimate may be adjusted in various embodiments and situations, such as to correct for deviations in the basic historical allocation data. After determining the expected performance, the expected performance estimation method 600B may proceed to block 640.

At block 640, the expected performance may be output, as in the expected performance estimation method 600A. In various embodiments, the method 600B of performance estimation expected may return to block 610 for the next multiple historical assignments, to 620B for the next weighted determination, or the method 600B of performance estimation expected may end.

Fig. 7 illustrates a flow chart of an expected performance estimation method 700 in accordance with an embodiment of the present disclosure. The expected performance estimation method 700 may begin at block 710.

At block 710, a plurality of historical assignments may be received. After receiving the plurality of historical assignments, the method 700 may proceed to block 720.

At block 720, the plurality of historical assignments may be divided into a training subset and a validation subset (retention subset, out of sample). After partitioning the plurality of historical allocations, the method 700 of performance estimation in anticipation may proceed to block 730.

At block 730, a pairing strategy may be generated based on the training subsets. For example, historical assignments may be used to automatically identify patterns within a task distribution system, and/or historical assignments may be used to calculate values/costs of a pairing model or an expense matrix. After generating the pairing strategy, the expected performance estimation method 700 may proceed to block 740.

At block 740, an expected performance of the pairing strategy may be determined based on the verification subset. In some embodiments, the expected performance may be determined using any of the validation techniques described herein, such as the expected performance estimation methods 600A and 600B (e.g., determining samples or weights, respectively, for the validation set; determining the expected performance based on the samples or weights, respectively, and in some embodiments, repeatedly sampling or weighting historical assignment data). After determining the expected performance of the pairing strategy, the expected performance estimation method 700 may proceed to block 750.

At block 750, the expected performance may be output. After outputting the expected performance, some embodiments may return to block 720 to obtain the next validation set or to block 740 to determine the next expected performance based on the validation subset. In other embodiments, the expected performance estimation method 700 may end.

Fig. 8 illustrates a flow chart of a method 800 of expected performance estimation in accordance with an embodiment of the present disclosure. The expected performance estimation method 800 is similar to the expected performance estimation method 700 (fig. 7), except that it explicitly describes embodiments in which some blocks may be repeated for multiple validation sets. At block 810, the expected performance estimation method 800 may begin.

At block 810, a plurality of historical assignments may be received. After receiving the plurality of historical assignments, the method 800 of predictive performance estimation may proceed to block 820. At block 820, a plurality of historical assignments may be partitioned into a training subset and a validation subset. After partitioning the plurality of historical allocations, the method 800 of expected performance estimation may proceed to block 830. At block 830, an expected performance of the pairing strategy may be determined based on the validation set. After determining the expected performance, the expected performance estimation method 800 may proceed to block 840.

At block 840, a determination may be made as to whether to repeat a portion of the method for the next verification subset. If so, the expected performance estimation method 800 may return to, for example, block 820 to divide the next validation subset and/or the next training subset, and to block 830 to determine the expected performance based on the next validation subset. If not, the expected performance estimation method 800 may proceed to block 850.

At block 850, a combined expected performance (e.g., an average expected performance or a weighted average expected performance) for at least two of the plurality of verification subsets may be output. After outputting the combined expected performance, the expected performance estimation method 800 may end.

Fig. 9 illustrates a flow chart of an expected performance estimation method 900 in accordance with an embodiment of the present disclosure. The expected performance estimation method 900 is similar to the process described with reference to fig. 4A-4H and 5A-5B, wherein the determining of samples is repeated to estimate the expected performance at a gradual approximation of an ideal peak policy (e.g., a gradually narrowing band of a diagonal BP policy or a gradually increasing sample size of a network flow BP policy). At block 910, the expected performance estimation method 900 may begin.

At block 910, a plurality of historical assignments may be received. After receiving the plurality of historical assignments, the method 900 of performance estimation in anticipation may proceed to block 920.

At block 920, a plurality of historically assigned samples may be determined based on the degree of achievement of the peak performance relative to the pairing strategy (e.g., a particular band size or distance from diagonal of the BP strategy or a particular sample size of the BP strategy of the network flow). After determining the samples, the expected performance estimation method 900 may proceed to block 930.

At block 930, the expected performance of the pairing strategy may be determined based on the sample using any of the previously described techniques. After determining the expected performance, the expected performance estimation method 900 may proceed to block 940.

At block 940, a determination may be made whether to repeat for the next degree of implementation. If so, the expected performance estimation method 900 may return to block 920 to determine a next sample based on the next degree of implementation (e.g., a narrower band in a diagonal BP policy, or a larger sample size in a network flow BP policy), and proceed to block 930 to determine another expected performance based on the next sample and the next degree of implementation. If not, the expected performance estimation method 900 may proceed to block 950.

At block 950, an expected performance based on the plurality of samples and the expected degree of achievement relative to the peak performance may be output. For example, a chart may be provided plotting the estimated expected performance along the y-axis for each degree of achievement on the x-axis. In other embodiments, the expected performance estimates may be combined (e.g., averaged or weighted averaged). In other embodiments, a determination of a "real world" implementation of peak performance may be evaluated, and an expected performance of the real world extent of the implementation may be output. After outputting the expected performance, the expected performance estimation method 900 may end.

At this point, it should be noted that estimating the expected performance in a task distribution system according to the present disclosure as described above may involve, to some extent, processing input data and generating output data. The input data processing and output data generation may be implemented in hardware or software. For example, certain electronic components may be used in an expected performance estimation module or similar or related circuitry for implementing functions associated with estimating expected performance in a task distribution system according to the present disclosure as described above. Alternatively, one or more processors operating with instructions may implement the functions associated with estimating expected performance in a task distribution system according to the present disclosure as described above. If this is the case, it is also within the scope of the disclosure that the instructions may be stored on one or more non-transitory processor-readable storage media (e.g., a magnetic disk or other storage medium) or transmitted to the one or more processors via one or more signals embedded in one or more carrier waves.

The present disclosure is not to be limited in scope by the specific embodiments described herein. Indeed, other various embodiments of the present disclosure and modifications thereof, in addition to those described herein, will be apparent to those of ordinary skill in the art from the foregoing description and accompanying drawings. Accordingly, such other embodiments and modifications are intended to fall within the scope of this disclosure. Moreover, although the present disclosure is described herein in the context of at least one particular implementation in at least one particular environment for at least one particular purpose, those of ordinary skill in the art will recognize that its usefulness is not limited thereto and that the present disclosure may be beneficially implemented in any number of environments for any number of purposes. Accordingly, the following claims should be construed in view of the full breadth and spirit of the present disclosure as described herein.

Claims

1. A method for estimating expected performance in a task distribution system, comprising:

receiving, by at least one switch configured to perform task allocation operations in a task allocation system, a plurality of historical agent-task allocations;

dividing, by at least one computer processor communicatively coupled to a task allocation system and configured to operate in the task allocation system, the plurality of historical agent-task allocations into a first subset of historical agent-task allocations and a maintained subset of historical agent-task allocations that is different from the first subset;

determining, by the at least one computer processor, a sample of the plurality of historical agent-task assignments by sampling the plurality of historical agent-task assignments according to a policy that pairs agents with tasks, wherein the policy is generated based on the first subset, wherein the sample is a subset of the retained subset;

determining, by the at least one computer processor, an expected performance of the policy based on the sample, wherein determining the expected performance is based on the retention subset; and

outputting, by the at least one computer processor, the expected performance for use in pairing an agent with a task in the task distribution system based on the policy;

wherein the actual performance of the task distribution system is optimized by using the strategy based on the expected performance.

2. The method of claim 1, wherein the task assignment system is a contact center, and wherein the policy assigns contacts to contact center agents.

3. The method of claim 1, further comprising:

oversampling, by the at least one computer processor, the plurality of historical agent-task assignments, wherein at least one of the plurality of oversampled historical agent-task assignments is included in each of the plurality of samples.

4. The method of claim 1, further comprising:

determining, by the at least one computer processor, a deviation in the sample; and

taking into account, by the at least one computer processor, the deviation in the expected performance.

5. The method of claim 4, wherein the deviation is attributable to a too high proportion of a subset of agents in the sample or a too high proportion of a subset of task types in the sample.

6. The method of claim 1, wherein determining the expected performance comprises:

determining, by the at least one computer processor, a plurality of samples of the plurality of historical agent-task assignments.

7. The method of claim 1, further comprising:

such that a historical agent-task allocation ratio attributable to a second policy for pairing agents and tasks that is different from the policy is prohibitively high in the retention subset.

8. The method of claim 1, wherein determining the expected performance is based on a plurality of retention subsets.

9. The method of claim 8, further comprising determining a standard error associated with said expected performance based on said plurality of retention subsets.

10. The method of claim 1, wherein the first and second light sources are selected from the group consisting of a red light source, a green light source, and a blue light source,

wherein an amount of difference in performance between an expected performance of the sample and a peak performance of the strategy is associated with the sample.

11. A method for estimating expected performance in a task distribution system, comprising:

determining, by the at least one computer processor, a weighting of at least one of the plurality of historical agent-task assignments by weighting the plurality of historical agent-task assignments according to a policy that pairs agents with tasks, wherein the policy is generated based on the first subset;

determining, by the at least one computer processor, an expected performance of the policy based on the weighting, wherein determining the expected performance is based on the retention subset; and

wherein the policy is used to optimize the actual performance of the task distribution system based on the expected performance.

12. The method of claim 11, wherein the task assignment system is a contact center, and wherein the policy assigns contacts to contact center agents.

13. The method of claim 11, wherein the at least one weight is zero or a minimum value.

14. The method of claim 11, further comprising:

15. The method of claim 14, wherein the deviation is attributable to an overly high proportion of a subset of agents or an overly high proportion of a subset of task types.

16. The method of claim 11, wherein determining the expected performance comprises combining at least two weighted results corresponding to at least two weighted pairs in the sample.

17. A system for estimating expected performance in a task distribution system, comprising:

at least one computer processor communicatively coupled to a task distribution system and configured to perform task distribution operations in the task distribution system, wherein the at least one computer processor is configured to:

receiving a plurality of historical agent-task assignments at a switch configured to perform task assignment operations in the task assignment system;

dividing the plurality of historical agent-task assignments into a first subset of historical agent-task assignments and a maintained subset of historical agent-task assignments that is different from the first subset;

determining a sample of the plurality of historical agent-task assignments by sampling the plurality of historical agent-task assignments according to a policy that pairs agents with tasks, wherein the policy is generated based on the first subset, wherein the sample is a subset of the retained subset;

determining an expected performance of the policy based on the sample, wherein determining the expected performance is based on the retention subset; and

outputting the expected performance for use in pairing an agent with a task in the task distribution system based on the policy;

18. The system of claim 17, wherein the task assignment system is a contact center, and wherein the policy assigns contacts to contact center agents.

19. The system of claim 17, wherein the at least one computer processor is further configured to:

oversampling the plurality of historical agent-task assignments, wherein at least one of the plurality of oversampled historical agent-task assignments is included in each of the plurality of samples.

20. The system of claim 17, wherein the at least one computer processor is further configured to:

determining a deviation in the sample; and

a deviation of the expected performance is considered.

21. The system of claim 20, wherein the deviation is attributable to a too high proportion of a subset of agents in the sample or a too high proportion of a subset of task types in the sample.

22. The system of claim 17, wherein determining the expected performance comprises determining a plurality of samples of the plurality of historical agent-task assignments.

23. A non-transitory computer readable medium storing instructions for estimating expected performance in a task distribution system, wherein,

the instructions are configured to be readable from the medium by at least one computer processor communicatively coupled to the task distribution system and configured to perform task distribution operations in the task distribution system, thereby causing the at least one computer processor to operate to:

outputting the expected performance for use in pairing an agent with a task in the task distribution system based on the policy; and

24. The non-transitory computer readable medium of claim 23, wherein the task assignment system is a contact center, and wherein the policy assigns contacts to contact center agents.

25. The non-transitory computer readable medium of claim 23, wherein the at least one computer processor is further caused to operate to:

oversampling the plurality of historical agent-task assignments, wherein at least one of the plurality of oversampled historical agent-task assignments is included in each of a plurality of samples.

26. The non-transitory computer readable medium of claim 23, wherein the at least one computer processor is further caused to operate to:

determining a deviation in the sample; and

a deviation of the expected performance is considered.

27. The non-transitory computer readable medium of claim 26, wherein the deviation is attributable to a too high proportion of a subset of agents in the sample or a too high proportion of a subset of task types in the sample.

28. The non-transitory computer readable medium of claim 23, wherein determining the expected performance comprises determining a plurality of samples of the plurality of historical agent-task assignments.

29. A method for estimating expected performance in a task distribution system, comprising:

receiving, by at least one computer processor communicatively coupled to a task allocation system and configured to perform task allocation operations in the task allocation system, a first plurality of historical agent-task allocations;

determining, by the at least one computer processor, a fitted proximity of each of the first plurality of historical agent-task assignments to a preferred task assignment policy for validating the preferred task assignment policy;

determining, by the at least one computer processor, a threshold fit proximity of each of the first plurality of historical agent-task assignments to the preferred task allocation policy;

determining, by the at least one computer processor, an expected performance of a task allocation system using the preferred task allocation strategy based on a subset of the first plurality of historical agent-task allocations within the threshold fit proximity;

outputting, by the at least one computer processor, the expected performance for use in pairing an agent with a task in the task allocation system based on the preferred task allocation policy; and

establishing, by the at least one computer processor, a connection between an agent and a task in a switch of the task distribution system based on the expected performance to achieve a first amount of performance gain of the task distribution system attributable to the preferred task distribution strategy, wherein actual performance of the task distribution system is optimized by using the verified preferred task distribution strategy based on the expected performance.

30. The method of claim 29, further comprising:

determining, by the at least one computer processor, a plurality of thresholds of closeness-to-fit to the preferred task allocation policy.

31. The method of claim 30, further comprising:

determining, by the at least one computer processor, a relevance of a performance of the preferred task allocation strategy to a degree of change in an achievement of the preferred task allocation strategy.

32. The method of claim 31, wherein the expected performance is further based on a relevance of performance.

33. The method of claim 32, further comprising:

receiving, by the at least one computer processor, a second plurality of historical agent-task assignments generated using the preferred task allocation policy; and

analyzing, by the at least one computer processor, a relevance of the performance based on the second plurality of historical agent-task assignments.

34. The method of claim 33, wherein the second plurality of historical agent-task assignments covers a second date range that is different from the first date range of the first plurality of historical agent-task assignments.

35. The method of claim 33, wherein each of the second plurality of historical agent-task assignments is paired using the preferred task allocation policy, and wherein each of the first plurality of historical agent-task assignments is paired using a different task allocation policy.

36. The method of claim 33, wherein the second plurality of historical agent-task assignments comprises some historical agent-task assignments from the first plurality of historical agent-task assignments.

37. The method of claim 33, wherein each of the second plurality of historical agent-task assignments is different from each of the first plurality of historical agent-task assignments.

38. The method of claim 29, wherein the task allocation system is a contact center system, and wherein the preferred task allocation policy allocates contacts to contact center agents.

39. The method of claim 29, wherein the plurality of historical agent-task assignments is a maintained subset of a set of historical agent-task assignments.

40. The method of claim 29, wherein the subset of the plurality of historical agent-task assignments that are within the threshold fit proximity are samples of the retention set.

41. A system for estimating expected performance in a task distribution system, comprising:

at least one computer processor communicatively coupled to a task distribution system and configured to perform task distribution operations in the task distribution system, wherein the at least one computer processor is further configured to:

receiving a first plurality of historical agent-task assignments;

determining a fit proximity of each of the first plurality of historical agent-task assignments to a preferred task assignment policy for validating the preferred task assignment policy;

determining a threshold fit proximity of each of the first plurality of historical agent-task assignments to the preferred task allocation policy;

determining an expected performance of a task allocation system using the preferred task allocation strategy based on a subset of the first plurality of historical agent-task allocations within the threshold fit proximity;

outputting the expected performance for use in pairing agents with tasks in the task distribution system based on the preferred task distribution policy; and

establishing, in a switch of the task distribution system, a connection between an agent and a task based on the expected performance to achieve a first amount of performance gain of the task distribution system attributable to the preferred task distribution strategy, optimizing actual performance of the task distribution system by using the verified preferred task distribution strategy based on the expected performance.

42. The system of claim 41, wherein the at least one computer processor is further configured to:

determining a plurality of thresholds of closeness of fit to the preferred task allocation policy.

43. The system of claim 42, wherein the at least one computer processor is further configured to:

determining a relevance of a performance of the preferred task allocation strategy to a degree of change in an achievement of the preferred task allocation strategy.

44. The system of claim 43, wherein the expected performance is further based on a relevance of performance.

45. The system of claim 44, wherein the at least one computer processor is further configured to:

receiving a second plurality of historical agent-task assignments generated using the preferred task allocation policy; and

analyzing the relevance of the performance based on the second plurality of historical agent-task assignments.

46. The system of claim 45, wherein the second plurality of historical agent-task assignments covers a second date range that is different from the first date range of the first plurality of historical agent-task assignments.

47. The system of claim 45, wherein each of the second plurality of historical agent-task assignments is paired using the preferred task allocation policy, and wherein each of the first plurality of historical agent-task assignments is paired using a different task allocation policy.

48. The system of claim 45, wherein the second plurality of historical agent-task assignments comprises some historical agent-task assignments from the first plurality of historical agent-task assignments.

49. The system of claim 45, wherein each of the second plurality of historical agent-task assignments is different from each of the first plurality of historical agent-task assignments.

50. The system of claim 41, wherein the task allocation system is a contact center system, and wherein the preferred task allocation policy allocates contacts to contact center agents.

51. The system of claim 41, wherein the plurality of historical agent-task assignments is a maintained subset of a set of historical agent-task assignments.

52. The system of claim 41, wherein the subset of the plurality of historical agent-task assignments that are within the threshold fit proximity is a sample of the retention set.

53. A non-transitory computer readable medium storing instructions for estimating expected performance in a task distribution system, wherein,

receiving a first plurality of historical agent-task assignments;

establishing, in a switch of the task distribution system, a connection between an agent and a task based on the expected performance to achieve a first amount of performance gain of the task distribution system attributable to the preferred task distribution strategy, wherein actual performance of the task distribution system is optimized by using the verified preferred task distribution strategy based on the expected performance.

54. The non-transitory computer readable medium of claim 53, wherein the at least one computer processor is further caused to operate to:

55. The non-transitory computer readable medium of claim 54, wherein the at least one computer processor is further caused to operate to:

56. The non-transitory computer readable medium of claim 55, wherein the expected performance is further based on a relevance of performance.

57. The non-transitory computer readable medium of claim 56, wherein the at least one computer processor is further caused to operate to:

58. The non-transitory computer readable medium of claim 53, wherein the task allocation system is a contact center system, and wherein the preferred task allocation policy allocates contacts to contact center agents.