HK1238037B - Systems and methods for maintaining network service levels - Google Patents

Description

System and method for maintaining network service levels

Background Art

Information is transmitted across a computer network. This information is represented as bits grouped into packets. These packets are passed from network device to network device, such as switches and routers, to propagate the information across the computer network. Each packet is transmitted from its source to a destination specified by the corresponding packet's header information. The packet's source and destination may be in different parts of the network, each operated by a different party. There may be multiple possible routes between the source and destination.

Wide area networks ("WANs") such as the Internet can include multiple subnetworks called autonomous systems ("ASs"). An AS is a portion of a network that appears to the rest of the network as having unified management of a single routing policy and presents a consistent image of reachable network destinations to the rest of the network, for example, as the network address space reachable through the AS. In some cases, an AS can be identified by an autonomous system number ("ASN") that is unique within the network. Typically, the operator of an AS has an agreement with a third party to allow data to be carried on one or more ASs controlled by the respective third party, typically under a "clearance" agreement for billable transportation or as a "clearanceless" peer-to-peer agreement. Data can then be transferred from one AS to another within the scope of the agreement between the operators of the ASs at, for example, peering points, multi-homed network devices, or Internet exchange points ("IXPs").

Summary of the Invention

In some aspects, the present disclosure relates to a method for maintaining a network service level. The method includes identifying a first plurality of network incidents that occurred during a first portion of a measurement period, and identifying a second plurality of network incidents that occurred during a second portion of the measurement period and occurred after the first portion of the measurement period. The method includes determining a plurality of remaining incident tolerances based on the impact of the first and second plurality of network incidents on corresponding sets of incident tolerances for the measurement period. The method includes generating a severity metric for at least a subset of the second plurality of network incidents based on an aggregated impact characteristic of one or more of the second plurality of network incidents weighted by the remaining incident tolerance associated with each of the second network incidents in the subset of the second network incidents. The method includes then selecting at least one of the incidents in the subset of the second network incidents for remediation.

In some aspects, the present disclosure relates to a system for maintaining a network service level. The system includes a computer-readable memory storing a record of network incidents; and one or more processors configured to access the computer-readable memory and execute instructions that, when executed by the processors, cause the processors to: use the record of network incidents stored in the computer-readable memory to identify a first plurality of network incidents that occurred during a first portion of a measurement period, and further identify a second plurality of network incidents that occurred during a second portion of the measurement period and occurred after the first portion of the measurement period. The instructions, when executed, further cause the processors to: determine a plurality of remaining incident tolerances based on the impact of the first and second plurality of network incidents on corresponding sets of incident tolerances for the measurement period; generate a severity metric for at least a subset of the second plurality of network incidents based on an aggregated impact characteristic of one or more of the second plurality of network incidents weighted by the remaining incident tolerance associated with each of the second network incidents in the subset of the second network incidents; and select at least one of the incidents in the subset of the second network incidents for remediation.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and related objects, features and advantages of the present disclosure will be more fully understood by referring to the following detailed description taken in conjunction with the accompanying drawings, in which:

FIG1 is a block diagram of an example network environment;

2A and 2B are block diagrams illustrating how communications may be redirected around a network failure;

FIG3A is an example table showing service level incident records;

FIG3B is an example table representing an aggregation of service level incident records;

FIG4 is a flow chart illustrating an example method for maintaining network service levels;

FIG5 is a Venn diagram illustrating the intersection of filters used to prioritize incidents;

FIG6 is a block diagram of a network device suitable for use in the various described implementations; and

7 is a block diagram of a computing system suitable for use in the various described implementations.

For clarity, not every component may be labeled in every drawing. The drawings are not intended to be drawn to scale. Like reference numerals and labels indicate like elements throughout the drawings.

DETAILED DESCRIPTION

Computing devices communicate over network routes that can span multiple autonomous systems (“AS”). AS networks, or portions of AS networks referred to as subnets, provide services to various network customers in a variety of environments, including but not limited to data service networks, access networks, transport networks, and multi-tenant networks (e.g., computing “clouds,” hosted computing services, and networks as a service). Network managers make promises to their customers that a certain level of service will be provided by the network. These service level agreements (“SLAs”) define one or more service level objectives (“SLOs”) for network uptime and quality (e.g., bandwidth, latency, etc.). Typically, an SLA is a contractual limit on tolerance for incidents that will inevitably occur causing network service to be interrupted or degraded. However, it can be difficult to determine whether a network incident or a set of incidents has a sufficient impact on the service level objectives to violate the SLA until the SLA has already been violated.

As described herein, a network administrator can use a service monitor to track service level incidents ("SLIs"). For example, an administrator can use an SLO-related tool that identifies an incident in which a software-defined network ("SDN") controller refuses to allocate resources to a new communication flow, for example, because network capacity is insufficient to support the requirements of the new flow at the time of the request. Any one such refusal is unlikely to result in an SLA violation. However, repeated refusal may result in an SLA violation. In some implementations, an SLI occurs whenever a communication flow encounters network congestion. Some communication protocols implement a congestion notification protocol, and a service monitor can detect or be notified of flows in which the congestion notification protocol indicates congestion. For example, the Transmission Control Protocol ("TCP") has a header bit reserved for explicit congestion notification ("ECN"), and the monitor can record an SLI whenever a flow includes a packet with an ECN bit set to indicate congestion. As another example, in some implementations, an SLA includes minimum requirements for the values of one or more metrics of communication quality averaged over a fixed or sliding period of time. Communication through the network can be measured using one or more metrics, including, for example, bandwidth, throughput, and actual throughput, as described in more detail below. Service level incidents can include, for example, incidents in which a network link is unavailable, a network flow is denied or interrupted, a network flow encounters network congestion, and/or incidents in which a value of a metric of network communication quality exceeds or falls below a threshold.

SLA violations can be predicted and prevented before they occur by closely monitoring SLIs. For example, as described herein, a monitor or network analyzer can collect SLI records from one or more monitoring tools, filter out some of the records based on various criteria, and identify significant SLIs from the remaining records. In some implementations, for example, each SLI is assigned a significance weight based on the impact of the corresponding incident. For example, in some implementations, an SLA includes a tolerance level for network failures over a period of time. If a service-level incident occurs near the end of that period, the service-level incident can be weighted higher or lower based on whether previous incidents have impacted the tolerance level. For example, if a particular SLA allows seven hours of downtime per month, a few seconds of downtime near the end of the month can be weighted lower if there has already been minimal downtime, and higher if there has been significant downtime approaching or exceeding the seven tolerated hours in that month. The monitor or network analyzer can then identify one or more specific incidents where corrective action would have the highest benefit in preventing SLA violations. The causes of these specific incidents can be identified and remediated. Predicting and preventing these SLA violations can improve network operations by maintaining network service levels.

FIG1 is a block diagram of an example network environment 100. In broad outline, FIG1 depicts a plurality of terminal nodes 120 configured to communicate with respective master nodes 160 via a network 110. While the Internet is a good example of a large network 110, this specification is equally applicable to other networks. As shown, the terminal nodes 120 access the network 110 via a network portion 112, which may be, for example, a portion of the network 110, an access network (e.g., an Internet service provider ("ISP")), a transport network, or any other network that facilitates communication between the terminal nodes 120 and the master nodes 160. As shown in FIG1 , the network 110 includes network portions 114 and 116. Each of the network portions 112, 114, and 116 is an illustrative network area and may be part of the same autonomous system, may be a different autonomous system, or may include multiple autonomous systems. The network portion 114 includes various network nodes 140 through which data is transferred between the terminal nodes 120 and the master nodes 160. Some implementations may benefit from the use of software-defined networks ("SDNs") in which data forwarding devices are remotely controlled by one or more network controllers. Thus, although not required in some implementations, network portion 114 is illustrated as an SDN, where network nodes 140 are data forwarding devices remotely controlled by one or more network controllers 146. Network portion 116 includes master nodes 160, which represent master devices within, for example, one or more data centers or service centers.

1 also illustrates a network monitor 180 that is located within the network 110 and has the capability to directly or indirectly monitor service level incidents (“SLIs”) occurring within the scope of the network 110. The network monitor 180 uses one or more storage devices 188 to store records of the SLIs. Although only one network monitor 180 is shown, some implementations use multiple network monitors 180 distributed throughout the network 110. In some such implementations, the distributed network monitors 180 share the one or more storage devices 188. A network analyzer 190 accesses and analyzes the SLI records stored in the one or more storage devices 188. In some implementations, the network monitor 180 is or includes the network analyzer 190. In some implementations, the network analyzer 190 is separate and distinct from the network monitor 180.

The terminal nodes 120 and master nodes 160 shown in FIG1 are participants in various data communications through the network environment 100. The terminal nodes 120 and master nodes 160 can each be, for example, a computing system 910 as shown in FIG7 and described below. For example, the master node 160 can be a computing system that provides a service, and the terminal node 120 can be a computing system that consumes the service. The master node 160 can transmit data to the terminal node 120, which then serves as a destination for the transmitted data. Similarly, the terminal node 120 can transmit data to the master node 160, which then serves as a destination for the transmitted data. The terminal nodes 120 and master nodes 160 can alternate between sending and receiving data. For example, the terminal node 120 can send a request for data to the master node 160, and the master node 160 can respond to the request by providing data. In some cases, multiple terminal nodes 120 and/or multiple master nodes 160 can participate in the exchange of data. The master node 160 can act as an intermediary between multiple terminal nodes 120, for example, as a communication facilitator. Each terminal node 120 and master node 160 can fill any number of roles. However, in each such capacity, the terminal node 120 and master node 160 participate in communications transmitted via the network environment 100. The communications between the terminal node 120 and the master node 160 can be constructed as a stream of data packets, for example, in the form of data packets according to an Internet protocol such as IPv4 or IPv6. The stream can use, for example, an open systems interconnection ("OSI") layer-4 transport protocol, such as Transmission Control Protocol ("TCP") or Stream Control Transmission Protocol ("SCTP"), transmitted via the networks 110, 112, 114, and 116 via IP.

The terminal node 120 can be a laptop, desktop, tablet, electronic board, personal digital assistant, smartphone, video game device, television, television accessory box (also known as a "set-top box"), kiosk, portable computer, or any other such device. The terminal device 120 is capable of presenting content to a user or facilitating the presentation of content to a user. In some implementations, the terminal device 120 runs an operating system that manages the execution of software applications on the terminal device 120. In some such implementations, the manufacturer or distributor provides the terminal device 120 with the operating system. Applications execute within a computing environment controlled by the operating system, i.e., "on top" of the operating system. Applications can be installed natively with the operating system or later, for example, by a distributor or user. In some implementations, the operating system and/or applications are embedded within the terminal device 120, for example, encoded in read-only memory.

A master node 160 can be a computer that provides services to other master nodes 160 or to terminal nodes 120. For example, a master node 160 can be an email server, a file server, a data cache, a name server, a content server, a data relay, a web server, or any other network service host. In some implementations, one or more master nodes 160 are part of a content delivery network ("CDN"). Although shown only in network portion 116, master nodes 160 can be distributed throughout network environment 100.

The network environment 100 includes network parts 110, 112, 114 and 116, through which the terminal node 120 and the master node 160 exchange information. The network parts 110, 112, 114 and 116 can be under unified control, for example, as part of the same AS network, or can be under different controls. Each network part 110, 112, 114 and 116 is composed of various network devices (e.g., network nodes 140), which are linked together to form one or more communication paths (e.g., data links 142) between the participating devices. For example, a network node 140 is illustrated in the network part 114, wherein the interconnection link 142 forms a data plane. Each network node 140 includes at least one network interface for transmitting and receiving data by the connected data plane link 142, forming a network together. In some implementations, the network device 730 shown in Figure 6 and described below is suitable for use as the network node 140.

The network environment 100, including various network segments 110, 112, 114, and 116, can be composed of multiple networks, each of which can be any of the following: a local area network (LAN), such as a company intranet; a metropolitan area network (MAN); a wide area network (WAN); an internetwork, such as the Internet; or a peer-to-peer network, such as an ad hoc WiFi peer-to-peer network. The data links between the devices can be any combination of wired links (e.g., fiber, mesh, coaxial, Cat-5, Cat-5e, Cat-6, etc.) and/or wireless links (e.g., radio, satellite, or microwave-based). Network segments 112 and 114 are illustrated as being part of a larger network segment 110, consistent with the example in which network monitor 180 is responsible for network 110; however, network segments 110, 112, 114, and 116 can each be public, private, or any combination of public and private networks. The networks can be any type and/or form of data network and/or communication network.

In some implementations, one or more of network portions 110, 112, 114, or 116 are implemented using network function virtualization ("NFV"). In an NFV network, some network functions typically implemented in network device 140 are implemented as software executing on a processor (e.g., a general-purpose processor). In some implementations, the virtualized network functions include one or more of load balancing, access control, firewall, intrusion detection, and routing. Other network functions may also be virtualized in this manner. In some implementations, the virtualized network functions include functionality for reporting network metrics, network outages, and other indications of SLIs to network monitor 180.

In some implementations, one or more network portions 110, 112, 114, and 116 are software-defined networks ("SDNs") in which data forwarding devices (e.g., network nodes 140) are controlled by remote network controllers 146 that are separate from the data forwarding devices, as shown, for example, with respect to network portion 114. In some such implementations, SDN network nodes 140 are controlled by one or more SDN controllers 146 via control plane links 148, which are distinct from and therefore out-of-band relative to data plane links 142. In some implementations, SDN network nodes 140 are controlled via in-band data plane links 142 or via a hybrid combination of in-band data plane 142 and out-of-band control plane links 148. In some implementations of SDN networks, multiple packet data transmissions flow through the network on assigned routes. When an SDN data forwarding device receives a packet for an unrecognized flow, the data forwarding device assigns or requests the controller to assign a route to the new flow. Each subsequently received packet for the flow is then forwarded by the data forwarding device along the same route. In some implementations, the SDN controller 146 selects a route for the new flow based on criteria associated with the new flow, such as requirements associated with the OSI layer 7 application protocol identified in the flow. For example, a flow for Voice over IP (“VoIP”) may require low network latency, while a flow for File Transfer Protocol (“FTP”) may tolerate higher latency, and the controller 146 will therefore prioritize VoIP traffic over a low-latency route. In some implementations, if the controller 146 cannot identify a route suitable for the new flow, it rejects the flow. Rejecting the flow may constitute a service level incident. In some implementations, the controller 146 reports the rejection of the new flow to the network monitor 180, for example, via link 186. In some implementations, link 186 is part of the control plane. In some implementations, link 186 is part of the data plane. In some implementations, the SDN controller 720 shown in FIG. 6 and described below is suitable as the network controller 146 .

The network environment 100 includes one or more network monitors 180. In some implementations, the network monitor 180 is a hardware device that includes one or more computing processors, a memory device, a network interface, and a connection circuit. For example, in some implementations, the network monitor 180 is a computing device, such as the computing device 910 shown in Figure 7 and described below. Each network monitor 180 is located in the network 110 or communicates with the network 110 and has the following capabilities: directly or indirectly monitor service level incidents ("SLIs") that occur within the scope of the network 110. In some implementations, the network controller 146 reports service level incidents to the network monitor 180. In some implementations, communication participants such as the master node 160 report service level incidents to the network monitor 180. In some implementations, the network monitor 180 detects service level incidents. For example, in some such implementations, the network monitor 180 periodically transmits a probe packet (for example, an Internet Control Message Protocol ("ICMP") packet) and uses the characteristics of the network response to the probe packet to determine the network status. The network monitor 180 records information representing each SLI in one or more storage devices 188. In some implementations, each SLI is represented as a record in a data structure stored in the one or more storage devices 188 for analysis. In some implementations, each SLI is represented as an entry in a database, for example, as a set of entries or rows of a table in a relational database. The record for each SLI can be stored in any suitable format. In some implementations, the one or more storage devices 188 are internal to the network monitor 180 or in the same location as the network monitor 180. In some implementations, the one or more storage devices 188 are external to the network monitor 180, for example, as a discrete data server, network attached storage ("NAS"), or storage area network ("SAN"). In some implementations, the network monitor 180 further includes a network analyzer 190.

Storage device 188 is a data storage device within network monitor 180 or external to network monitor 180 but accessible to network monitor 180. Storage device 188 may include any device or collection of devices suitable for storing computer-readable data. Suitable data storage devices include volatile or non-volatile storage, network attached storage ("NAS"), and storage area networks ("SAN"). The data storage device may include one or more mass storage devices, which may be co-located or distributed. Suitable devices for storing data include semiconductor memory devices such as EPROM, EEPROM, SDRAM, and flash memory devices. Suitable devices for storing data include magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, optical, and other such higher-capacity formats. The data storage device may be virtualized. The data storage device may be accessed via an intermediate server and/or via a network. The data storage device may organize data into files, data blocks, or collections of blocks. The data storage device may provide error recovery using, for example, redundant storage and/or error recovery data (e.g., parity bits). Storage device 188 may host a database, such as a relational database. In some implementations, data is stored as entries in one or more database tables in a database stored in a data store. In some such implementations, data is accessed using a query language, such as Structured Query Language ("SQL") or a variant such as PostgreSQL. Storage device 188 may host a file storage system. Data structured as a knowledge base may be stored. Data may be stored in an encrypted form. Access to stored data may be restricted by one or more authentication systems.

In some implementations, an SLI may occur when communications over a network degrade below a specific quality level. For example, an SLA may include minimum or maximum thresholds for the average values of one or more network communication quality metrics, such as throughput, bandwidth, and latency. Throughput is the amount of information, e.g., the number of bits, transmitted over a portion of a network in a fixed period of time. Bandwidth is the maximum potential throughput, where the limitation is physical or artificial (e.g., policy-driven). Congestion occurs when a network device attempts to achieve a greater throughput than the available bandwidth can accommodate. Actual throughput is the throughput of information content, excluding other traffic such as network configuration data, protocol control information, or repeated transmissions of lost packets. Latency is the amount of time that elapses between a sender transmitting a packet and the intended recipient processing it, i.e., the delay attributable to transmission. Hysteresis is the result of delay, e.g., the perception of delay from the perspective of a communication participant. Hysteresis may occur, for example, when latency exceeds a certain tolerance threshold, e.g., when the delay becomes noticeable to the end user or fails to meet the Quality of Service ("QoS") requirements of a communication protocol. Although hysteresis can also occur when packets are lost or corrupted in transit, it is generally considered synonymous with latency. Delay (and hysteresis) can be measured in terms of one-way transmission or as the round-trip time between a packet transmission and a subsequent response or acknowledgment. In some cases, latency is measured in terms of path length, i.e., the number of intermediate network devices ("hops") in a route. Each hop can contribute to the overall latency of a route, so paths with a lower number of hops are expected to have less latency and less chance of forwarding failures. Packet delay variation (i.e., transmission jitter) is the variation in latency over time, for example, when packets arrive in bursts or with inconsistent delays. Transmission errors can cause poor throughput, high latency or hysteresis, and undesirable delay variation. Metrics of transmission errors include counts of packet retransmissions, the ratio of packet retransmissions to first transmissions, and congestion-related transmissions, such as packets with the Explicit Congestion Notification ("ECN") flag set. SLI can be recorded for each such transmission error or when the value of one or more metrics of transmission errors exceeds or falls below corresponding thresholds.

The network analyzer 190 is responsible for analyzing the SLI records identified by the network monitor 180. In some implementations, the network analyzer 190 is a component or module of the network monitor 180. In some implementations, the network analyzer 190 is a hardware device that includes one or more computing processors, a memory device, a network interface, and connecting circuitry. For example, in some implementations, the network analyzer 190 is a computing device, such as the computing device 910 shown in Figure 7 and described below. The network analyzer 190 reads the SLI records from the storage device 188 and prioritizes the service level incidents represented. In some implementations, the network analyzer 190 identifies one or more specific service level incidents represented by the SLI records as high priority. The network administrator can then further investigate the high priority incident and take action to address the root cause of the incident.

Figures 2A and 2B are block diagrams illustrating how communications can be redirected around a network failure 214. In some cases, an SLI can be the result of a seemingly unrelated network failure. For example, an SLI on a network link can be the result of an excess of that link caused by a failure on a different network link elsewhere in the network. Figures 2A and 2B illustrate this example.

In broad overview, Figures 2A and 2B illustrate a network environment 200 that includes three distinct network regions 240 _(A) , 240 _(B) , and 240 _(C) . The illustrated network environment 200 includes a data path 210 between regions 240 _(A) and 240 _(C) , a data path 220 between regions 240 _(A) and 240 _(B) , and a data path 230 between regions 240 _(B) and 240 _(C). The illustrated data paths 210, 220, and 230 are shown as single lines representing any form of network path, including, for example, a single direct link, multiple links, link aggregations, network fabrics, networks of links, and the like. In Figure 2A, data can flow directly 216 from network region 240 _(A) to 240 _(C) via data path 210. However, in FIG. 2B , a fault 214 along data path 210 blocks flow 216 from region 240 _(A) to region 240 _(C) .

In FIG2B , data from region 240 _(A) to region 240 _(C) flows through region 240 _(B) to circumvent failure 214. That is, direct data flow 216 is replaced by data flow 226 from region 240 _(A) to region 240 _(B) and data flow 236 from region 240 _(B) to region 240 _(C) . If the combined capacity of paths 220 and 230 (i.e., paths) is equal to or greater than the capacity of failed path 210, then failure 214 will not result in a service level incident (“SLI”). However, redirecting traffic from region 240 _{( A) to region 240 (C} ) through region 240 _(B) using an alternate path may affect other traffic along one or both of component paths 220 and 230, which may result in an SLI. For example, if each path 210, 220, and 230 has the same capacity and is at the same utilization, then failure 214 will double the utilization of the paths 220 and 230. If the utilization is initially greater than 50%, doubling the utilization will exceed the capacity and result in an SLI.

If the resulting load on the network path approaches capacity, then SLI may be present for the subsequent increase in traffic. Even if the initial utilization of one of the paths is low, if the resulting combined load exceeds the capacity of the data path, then SLI may be present. For example, if the combined load on network path 230 approaches full utilization after redirecting traffic around failure 214, then a subsequent increase in traffic from region 240 _(B) to region 240 _(C) will fail along network path 230. The resulting SLI record will be associated with the traffic along network path 230. However, the actual root cause is failure 214 along network path 210.

2A and 2B are simplified. In reality, traffic routed around a failure will impact various alternative paths and trigger service level incidents in potentially unexpected locations. But analysis of the SLI records in the aggregate can help identify the underlying cause. Thus, in some implementations, the network analyzer 190 can look for a collection of service level incidents on various paths that indicates—in the aggregate—where remedial action is warranted. In some implementations, the network analyzer 190 identifies related incidents associated with different services, making it more likely that the cause is network, rather than a cause more directly related to the services themselves.

In some cases, a service level incident ("SLI") occurs when the content of a packet in a flow cannot be propagated through a network or a portion of a network, or when the content of a packet in a flow is not propagated in a manner that satisfies one or more network quality metrics. For example, an SLI may occur when network resources cannot be allocated to a flow, when a network flow experiences congestion, or when the value of one or more network communication metrics exceeds or falls below a corresponding threshold. A service level agreement ("SLA") may allow for a certain number of incidents or a certain aggregate impact of incidents during a particular measurement period. For example, an SLA may tolerate up to 1% of the flows being interrupted or denied on a weekly basis. In some implementations, the period has fixed start and end times, for example, a week may be defined as from midnight Sunday to 11:59 PM the following Saturday. In some implementations, the period is a sliding time window, for example, a week may be defined as a window of 7 consecutive days or any window of 168 hours. In some implementations, the measurement period may be a discrete time block or a sliding time window specified by the SLA. The incident tolerance of the measurement period decreases with each incident that occurs. That is, when an incident occurs, the remaining incident tolerance for the measurement period covering the incident is reduced by the impact of the incident. If the SLA allows or tolerates 10 hours of downtime per month, 2 hours of downtime leaves 8 hours of remaining incident tolerance for the remainder of the month. If the SLI exceeds the incident tolerance of the SLA, the SLI is an SLA violation.

In some implementations, of two comparable incidents that are similar in importance or network impact, when one SLI has a greater impact on the remaining tolerance limit of the SLA than the other SLI, the SLI with the greater impact on the remaining tolerance limit is prioritized over the other SLI. In some implementations, an SLI that causes the remaining tolerance limit of the SLA to be less than a threshold is considered a more severe incident than a comparable SLI that does not cause the remaining tolerance limit of the SLA to be less than the threshold. In some implementations, when one SLI has a greater impact on the remaining tolerance limit of the SLA than the other SLI, the SLI with the greater impact on the remaining tolerance limit is prioritized over the other SLI, even if other factors may suggest prioritizing the other SLI based on importance or network impact. In some implementations, multiple factors are used to identify which SLI to prioritize. For example, Figure 5, discussed below, illustrates an example of using multiple filters to identify a set of priority incidents.

Figures 3A and 3B are example tables representing service-level incidents. In some implementations, service-level incidents are represented by network incident records. In some implementations, network incident records include only information sufficient to identify the corresponding incident. In some implementations, network incident records include additional information, such as information that may be helpful for subsequent diagnosis. In some implementations, network incident records include at least the time and date of the incident, the routing information for the data, and a description or classification of the services affected by the incident.

Figure 3A illustrates an example of a table 300 of service level incident ("SLI") records, wherein each SLI is represented by a corresponding row 372 containing a data entry for the affected flow (e.g., a flow that cannot be assigned a route or for which an assigned route has become unavailable). As shown, each row 372 includes data entries for the source 312 and destination 316 of the affected flow, the service level objective ("SLO") 322 of the service supported by the affected flow, a service identifier 332, and the event time 352 when the flow was affected. The source 312 and destination 316 are represented by identifiers of the participating ends of the flow, such as a network address, network name, address range, domain name, or any other such identifier. The SLO 322 is represented by an identifier of the SLO, such as a name or number. In some implementations, the SLO 322 name is a descriptive string. In some implementations, the SLO 322 name is a group classification identifier. In some implementations, the SLO 322 name is a descriptive characteristic of the objective, e.g., a maximum incident tolerance level. The service identifier 332 identifies the service or group of services affected by the SLI. For example, if the flow is associated with a particular service hosted by one or more master nodes 160, the service identifier 332 may be a string that identifies the service. The event time 352 is a timestamp that indicates when the SLI occurred or when the SLI record was entered (which may generally correspond to when the SLI occurred, but may not be its precise moment). Although shown as a single table 300, the information represented in FIG3A may be stored as multiple tables or in a non-relational database structure. Table 300 is provided as an example of how service level incidents may be represented in the data store 188; alternative data structures are used in some implementations.

FIG3B illustrates an example table 305 for representing an aggregation of service-level incident records. In example table 305, each set of SLI records is represented by a corresponding row 374 containing aggregated data entries corresponding to the SLI records for various affected flows (e.g., as shown in table 300 shown in FIG3A). As shown, each row 374 includes: data entries for the source region 314 and destination region 318 of the flows affected by the one or more represented sets of incidents; the aggregate SLO level 324 and aggregate service or service class identifier 334 for the services supported by the affected flows; a count 340 of SLI records in the set; and the start 354 and end 356 of the event time range when the flows were affected. The source 314 and destination 318 ranges are represented by identifiers of the participating ends of the flows, such as network addresses, network names, address ranges, domain names, or any other such identifiers. The SLO level 324 is represented by a generalized identifier for the SLO, such as a name or number. In some implementations, the represented set of SLIs may have the same SLO, in which case SLO level 324 may be equivalent to SLO 322. In some implementations, the represented set of SLIs may have a shared SLO characteristic, and this shared SLO characteristic is used as SLO level 324. In some implementations, SLO level 324 is a generalization of the target for the flow represented by the set. Similarly, service or service class identifier 334 identifies the service or service group affected by the represented SLI. In some implementations, table 305 may be sorted by count 340. Event time range start 354 and 356 are timestamps representing the start and end of event time range 352 for the represented set of incidents. Although shown as a single table 305, the information represented in FIG3B may be stored as multiple tables or in a non-relational database structure. Table 305 is provided as an example of how a set of service-level incidents may be represented in data store 188; alternative data structures are used in some implementations.

In some implementations, the data represented in table 305 shown in FIG3B is generated by applying one or more filters or aggregate queries to the data represented in table 300 shown in FIG3A. For example, in some implementations, queries are used to identify similar SLIs that occurred along various network corridors within a particular time range, where a network corridor is a set of network paths between two sets of terminal nodes or regions. Network corridors can be characterized, for example, by parallel network paths, shared network paths, collaborative network paths, link aggregation, and other such redundancies. The ends of a network corridor can be geographic areas, network service areas, co-located computing devices, data centers, proximate address ranges, and the like. In some implementations, the network analyzer 190 represents a network corridor using a pair of network address sets, where each of the network address sets identifies or defines a terminal node at a respective end of the network corridor.

A query or set of queries can be used to identify SLI records for frequently occurring incidents affecting different services along a particular network corridor, which can indicate a problem in that network corridor. In some implementations, the network analyzer 190 identifies a set of SLI records using a count 340 above a certain minimum threshold, which can be, for example, a preconfigured number or percentage.

FIG4 is a flow chart illustrating an example method 400 for maintaining network service levels. In a broad overview of method 400, network analyzer 190 identifies, at stage 410, a first plurality of network incidents that occurred during a first portion of a measurement period. At stage 420, network analyzer 190 identifies a second plurality of network incidents that occurred during a second portion of the measurement period, occurring after the first portion. At stage 430, network analyzer 190 determines a plurality of remaining incident tolerances based on the impact of the first and second plurality of network incidents on corresponding sets of incident tolerances for the measurement period. At stage 440, network analyzer 190 generates a severity metric for at least a subset of the second plurality of network incidents based on an aggregated impact characteristic of one or more of the second plurality of network incidents weighted by the remaining incident tolerance associated with each second network incident in the subset of the second network incidents. Furthermore, at stage 450, network analyzer 190 selects at least one incident in the subset of the second network incidents for remediation.

Referring to FIG. 4 in more detail, at stage 410, network analyzer 190 identifies a first plurality of network incidents that occurred during a first portion of a measurement period. Network analyzer 190 identifies network incidents by accessing records stored in data store 188 by network monitor 180. In some implementations, network analyzer 190 queries data store 188 to identify and/or retrieve SLI records for incidents that occurred during the first portion of the measurement period. In some implementations, network analyzer 190 uses a query (e.g., an SQL query) to identify the records and simultaneously filters out or aggregates the records based on criteria that limit the scope of the analysis. For example, the criteria may eliminate records for incidents that occurred in apparent isolation or for unrelated services. In some implementations, the criteria may limit the identified records to a specific network corridor. In some implementations, the criteria may identify clusters of incidents, for example, in time, geography, or network topology. In some implementations, the query returns only a set of records for incidents that occurred more than a threshold number of times during the first portion of the measurement period. In some implementations, multiple queries or filters are used to identify incident records to be included in the first plurality of network incidents. FIG. 5 , presented below, illustrates a Venn diagram for identifying a set of priority incidents 540 using a combination of queries or filters 510 , 520 , and 530 .

Still referring to stage 410 of FIG. 4 , the first portion of the measurement period provides historical context for analyzing events in a later portion of the measurement period, such as the second portion of the measurement period. The first portion of the measurement period may, for example, begin at the start of the measurement period and end at a percentage of the measurement period, such as half or 67% of the measurement period. The first portion may, for example, begin at the start of the measurement period and end at the time of access to data store 188 in stage 410. In some implementations, the first portion of time begins at the start of the measurement period and ends at the start of the second portion of time, which in turn ends at the end of the measurement period or at the time of the last service-level incident. In some such implementations, the end of the first portion of time is offset from the end of the measurement period, for example, so that the second portion of time is a fixed length, such as the last six hours of the measurement period, or a preconfigured percentage of the measurement period, such as the last 10% of the measurement period. In some implementations, the end of the first portion of time is selected so that the second portion of time is the shorter of the fixed length or the preconfigured percentage of the measurement period. That is, for example, in some implementations, the first portion of time ends 6 hours before the end of the measurement period or 90% of the measurement period, whichever is shorter (where 6 hours and 90% are example numbers - other lengths may also be suitable).

At stage 420, network analyzer 190 identifies a second plurality of network incidents that occurred during a second portion of the measurement period, occurring after the first portion of the measurement period. Network analyzer 190 identifies the network incidents by accessing records stored by network monitor 180 in data store 188. In some implementations, network analyzer 190 queries data store 188 to identify and/or retrieve SLI records for incidents that occurred during the second portion of the measurement period. In some implementations, network analyzer 190 uses a query (e.g., an SQL query) to identify records based on criteria that limit the scope of the analysis. For example, the criteria may select records for incidents that are related or associated with the incidents identified in stage 410. In some implementations, network analyzer 190 uses the same query and filters used in stage 410, applied to the second portion of the measurement period.

In some implementations, the second portion of the measurement period is continuous with the first portion of the measurement period, as described above. The second portion of the measurement period can, for example, be a period that begins at the end of the first period and ends at the end of the measurement period. The second portion can, for example, be a period that begins at the end of the first period and ends at the time of access to data store 188 in stage 420. In some implementations, the second portion overlaps with or encompasses the first portion. Typically, the first portion of the measurement period provides context for analyzing network performance during the second portion of the measurement period. Service level incidents that occur during the second portion of the measurement period can then be identified by comparing the context provided by the first portion of the measurement period, or the first and second portions of the measurement period.

At stage 430, the network analyzer 190 determines a plurality of remaining incident tolerance limits based on the corresponding sets of incident tolerance limits for the first and second pluralities of network incidents for the measurement period. For each SLA affected by a service level incident in the first and second pluralities of network incidents, the network analyzer 190 identifies the corresponding incident tolerance limit and the impact on the corresponding incident tolerance limit, e.g., resulting in a remaining incident tolerance limit for the measurement period.

At stage 440, network analyzer 190 generates a severity metric value for at least a subset of the second network incidents based on the aggregated impact characteristics of one or more of the second plurality of network incidents weighted by the remaining incident tolerance associated with each second network incident in the subset of the second network incidents. Each SLI may be assigned a score based on one or more metrics, the score representing the severity of the incident. In some implementations, the metric considers the count of corresponding incidents during the measurement period. In some implementations, the metric includes a priority value associated with the affected network path. In some implementations, the metric assigns different values to different services, for example, assigning a higher severity score to incidents affecting higher-priority services. The score, i.e., the severity metric value, is then adjusted, i.e., weighted, by a factor representing the corresponding remaining incident tolerance for the SLA affected by the incident. In some implementations, the weighting factor increases as the remaining incident tolerance approaches zero. In some implementations, the severity metric includes the frequency of incidents during the measurement period.

At stage 450, network analyzer 190 selects at least one of the incidents in the subset of the second network incidents for remediation. In some implementations, network analyzer 190 identifies one or more incidents using a severity metric value that is greater than a threshold. In some implementations, network analyzer 190 identifies one or more incidents using a severity metric value that is in the upper percentile (e.g., upper 75th percentile or upper 90th percentile, etc.). In some implementations, network analyzer 190 identifies one or more incidents using the highest severity metric value of the incidents that occurred during the second portion of the measurement period. In some implementations, remediation of incidents with high severity metric values is more likely to improve overall network conditions than remediation of lower-ranked incidents.

FIG5 is a Venn diagram illustrating the intersection of filters used to prioritize incidents. In some implementations of method 400, network analyzer 190 selects incidents from a subset of the second network incidents for remediation based on multiple incident filters. As shown in FIG5 , in some implementations, network analyzer 190 identifies a ranked set of high-priority incidents by identifying the intersection of filters 510, 520, and 530. First filter 510, for example, uses one or more queries of service-level incident records stored in storage 188 to identify incidents with the highest frequency of occurrence. In some implementations, this query selects clusters of similar incidents and ranks them by the corresponding count of incidents in each cluster. Second filter 520 identifies incidents associated with the largest clusters. For example, in some implementations, incidents are clustered by shared or similar attributes. In some implementations, incidents are clustered by affected network links, routes, end nodes, or end node regions, such that the resulting clusters group incidents that affect the same network corridor. Clustering filter 520 identifies the largest clusters and allows network analyzer 190 to prioritize incidents associated with the largest clusters. A third filter 530 identifies incidents with the highest weighted impact scores. In some implementations, incidents are assigned a value for one or more impact metrics that measure the impact of the incident on network quality. This value is weighted by one or more factors, including, for example, the remaining tolerance level for a corresponding service level objective or service level agreement. In some implementations, network analyzer 190 identifies a set of prioritized incidents based on the intersection 540 of filters 510, 520, and 530. In some implementations, other filters are used. In some implementations, additional filters are used.

In some implementations, the network analyzer 190 generates a report identifying the selected network incidents. In some implementations, the report is provided to one or more system operators via email, SMS text message, automated phone call, instant message, or any other available medium for communication.

FIG6 is a block diagram of an example network device 730. According to one illustrative implementation, the example network device 730 is suitable for use in implementing the intermediate network device described herein. The computing system 910 described below with reference to FIG7 may also be suitable for use as the network device 730. For example, using network function virtualization ("NFV"), a network function that is typically implemented in hardware circuitry is implemented as software executed on a processor (e.g., a general-purpose processor). In a broad overview, the network device 730 includes a control module 744 and a memory 736, which is used, for example, to store device configuration and routing data. The network device 730 includes a forwarding engine 734 that uses the device configuration and routing data stored in the memory 736 to manage data traffic at the network interface 738. In some implementations, the network device 730 is implemented for use in a software-defined network ("SDN"), wherein the network device 730 is controlled by an external SDN controller 720, for example, via a control plane link 712. The SDN controller 720 includes a control module 742 and a memory 726. The computing system 910 described below with reference to FIG7 may also be suitable for use as the SDN controller 720. In some implementations, one or more functional components of the network device 730 or the SDN controller 720 are implemented as software components executed by a general-purpose processor.

6 , in more detail, network device 730 includes a set of network interfaces 738. Each network interface 738 can be connected to one or more external devices via one or more links, forming a network (e.g., network 110 shown in FIG1 ). External devices send data packets to network device 730 via these links, arriving via an ingress interface (e.g., network interface 738 _(a) ). Network device 730 forwards received data packets to the appropriate next hop via an egress interface (e.g., network interface 738 _(c) ). In some implementations, forwarding engine 734 determines which network interface 738 to use for forwarding each received data packet.

The forwarding engine 734 uses the configuration and routing data in the memory 736 to manage data traffic at the network interface port 738. The configuration and routing data in the memory 736 are controlled by the control module 744. In some implementations, the forwarding engine 734 updates the packet header before forwarding the packet to the egress network interface 738. For example, the forwarding engine 734 can update the ECN, TTL, or checksum information in the packet header. In some implementations, the incoming packet contains routing instructions embedded in the header of the incoming packet, and the forwarding engine 734 forwards the packet based on the embedded instructions.

Memory 736 can be any device suitable for storing computer-readable data. Examples include, but are not limited to, semiconductor memory devices such as EPROM, EEPROM, SRAM, and flash memory devices. In some implementations, memory 736 of network device 730 includes memory dedicated to storing patterns used to identify packet flows, such as, for example, a ternary content addressable memory ("TCAM"). In some implementations, memory 736 of network device 730 includes memory dedicated to buffering packet flows as they traverse network device 730. Network device 730 can have any number of memory devices 736.

The control module 744 manages the performance of the network device 730. In some implementations, the control module 744 receives instructions from an external control device. For example, in a software-defined network ("SDN"), the control module 744 can receive control instructions from an SDN controller 720 external to the network device 730. In some implementations, the control module 744 processes routing information packets (i.e., control plane packets) and updates the memory 736 with modifications to the routing table used by the forwarding engine 734. In some implementations, the control module 744 reads data arriving at the egress interface 738 into a buffer stored in the memory 736. The control module 744 can be implemented using a general-purpose processor or dedicated logic circuitry, such as an application-specific integrated circuit ("ASIC").

FIG7 is a block diagram of an example computing system 910. According to one illustrative implementation, the example computing system 910 is suitable for implementing the computerized components described herein. In broad overview, the computing system 910 includes at least one processor 950 for performing actions according to instructions, and one or more memory devices 970 or 975 for storing instructions and data. The illustrated example computing system 910 includes one or more processors 950 communicating with a memory 970 via a bus 915; at least one network interface controller 920 having a network interface 922 for connecting to a network device 924 (e.g., for accessing a network); and other components 980, such as input/output (“I/O”) components 930. Typically, the processor 950 executes instructions received from the memory. The illustrated processor 950 includes a cache 975 or is directly connected to the cache 975. In some cases, instructions are read from the memory 970 into the cache 975 and executed by the processor 950 from the cache 975.

More specifically, processor 950 can be any logic circuit that processes instructions, such as instructions retrieved from memory 970 or cache 975. In many embodiments, processor 950 is a microprocessor unit or a dedicated processor. Computing device 910 can be based on any processor or set of processors capable of operating as described herein. Processor 950 can be a single-core or multi-core processor. Processor 950 can be a plurality of different processors. In some implementations, processor 950 is implemented as a circuit on one or more "chips."

Memory 970 can be any device suitable for storing computer-readable data. Memory 970 can be a device with fixed storage or a device for reading removable storage media. Examples include all forms of non-volatile memory, media, and storage devices, semiconductor memory devices (e.g., EPROM, EEPROM, SDRAM, and flash memory devices), magnetic disks, magneto-optical disks, and optical disks (e.g., CD-ROM, DVD-ROM, or (Blu-ray) disks). Computing device 910 can have any number of memory devices 970.

The cache 975 is a form of computer memory that is typically placed close to the processor 950 for fast access times. In some implementations, the cache 975 is part of the processor 950 or on the same chip as the processor 950. In some implementations, there are multiple levels of cache 975, such as L2 and L3 cache layers.

The network interface controller 920 manages data exchange via a network interface 922 (sometimes referred to as a network interface port). The network interface controller 920 handles the physical and data link layers of the OSI model for network communications. In some implementations, some of the network interface controller's tasks are handled by one or more of the processors 950. In some implementations, the network interface controller 920 is incorporated into the processor 950, for example as a circuit on the same chip. In some implementations, the computing system 910 has multiple network interfaces 922 controlled by a single controller 920. In some implementations, the computing system 910 has multiple network interface controllers 920. In some implementations, each network interface 922 is a connection point for a physical network link (e.g., a Cat-5 Ethernet link). In some implementations, the network interface controller 920 supports wireless network connections, and the interface 922 is a wireless (e.g., radio) receiver/transmitter (e.g., for any of the IEEE 802.11 protocols, near field communication (NFC), Bluetooth, BLE, ANT, or any other wireless protocol). In some implementations, the network interface controller 920 implements one or more network protocols, such as Ethernet. Typically, the computing device 910 exchanges data with other computing devices via a physical or wireless link through the network interface 922. The network interface 922 can be directly linked to another device or linked to another device via an intermediate device, such as a network device such as a hub, bridge, switch, or router, which connects the computing device 910 to a data network such as the Internet.

The computing system 910 may include, or provide an interface for, one or more input or output ("I/O") components 930. Input devices include, without limitation, keyboards, microphones, touch screens, foot pedals, sensors, MIDI devices, and pointing devices such as a mouse or trackball. Output devices include, without limitation, video displays, speakers, refreshable Braille terminals, lights, MIDI devices, and 2-D or 3-D printers.

Other components 980 may include I/O interfaces, external serial device ports, and any additional coprocessors. For example, computing device 910 may include an interface (e.g., a Universal Serial Bus ("USB") interface) for connecting input devices, output devices, or additional memory devices (e.g., a portable flash drive or external media drive). In some implementations, computing device 910 includes additional devices 980, such as coprocessors. For example, a math coprocessor may assist processor 950 with high-precision or complex calculations.

The subject matter and implementation of the operations described in this specification may be implemented in digital circuits or in computer software implemented on tangible media, firmware, or hardware, which includes the structures disclosed in this specification and their structural equivalents or one or more combinations thereof. The implementation of the subject matter described in this specification may be implemented as one or more computer programs implemented on tangible media, that is, one or more modules of computer program instructions, which are encoded on one or more computer storage media to be executed by a data processing device or used to control the operation of a data processing device. A computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these. A computer storage medium may also be or be included in one or more discrete components or media (e.g., multiple CDs, disks, or other storage devices). A computer storage medium is tangible and stores data such as computer-executable instructions in a non-transient form.

A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled languages, interpreted languages, declarative languages, and procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files that store one or more modules, libraries, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers that are located in one location or distributed in multiple locations and interconnected by a communication network.

The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and devices may be implemented as, special-purpose logic circuitry, such as a field programmable gate array ("FPGA") or an application-specific integrated circuit ("ASIC"). Such a special-purpose circuit may be referred to as a computer processor, even though it is not a general-purpose processor.

Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the claimed content, but should be interpreted as descriptions of features specific to a particular implementation of a particular invention. Certain features described in this specification in the context of discrete implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations or in any appropriate sub-combination. Moreover, although features may be described above as functioning in a certain combination and even initially claimed as such, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variant of the sub-combination.

Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequence, or that all illustrated operations be performed, in order to achieve the desired results. In certain circumstances, multitasking and parallel processing may be beneficial. Moreover, the separation of various system components in the above-described implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

References to "or" can be interpreted as inclusive, such that any term described using "or" can refer to any of a single, more than one, and all of the terms described. Labels "first," "second," and "third," etc. are not necessarily meant to indicate an order and are generally used only to distinguish between similar or analogous items or elements.

Thus, particular implementations of the subject matter have been described. Other implementations are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In some implementations, multitasking or parallel processing may be employed.

Claims (18)

1. A method for maintaining network service levels, the method comprising: Identify a first plurality of network incidents that occur in the first part of the measurement period, and designate the first plurality of network incidents as the first network incident; Identify a second plurality of network incidents that occur after the first portion of the measurement period and during the second portion of the measurement period, the second plurality of network incidents being designated as a second network incident; Multiple remaining incident tolerance limits are determined based on the impact of the first network incident and the second network incident on the corresponding set of incident tolerance limits for the measurement period, wherein the incident tolerance limit is the tolerance level for network incidents over a time period, and wherein the remaining incident tolerance limits are obtained by reducing the incident tolerance limit as each incident occurs. Generate a severity metric for each second network incident, wherein each second network incident is assigned a score representing the severity of the incident based on one or more metrics, and wherein the severity metric is weighted based on the remaining incident tolerance limit associated with each second network incident; and At least one incident in the second network incident is selected for remediation based on a comparison of the severity metrics. 2. The method of claim 1, wherein the identified first plurality of network incidents and second plurality of network incidents are represented by network incident records stored in a computer-readable storage medium. 3. The method according to claim 2, wherein the network incident log includes at least: (i) Information on the time and date of the accident. (ii) the routing information for the occurrence of the aforementioned incident, and (iii) A description or classification of the services affected by the incident. 4. The method of claim 1, further comprising limiting a subset of the second network incidents to network incidents that meet the selection criteria. 5. The method of claim 4, further comprising selecting a subset of the second network incidents based on the fact that the count of network incidents satisfying the selection criteria exceeds a threshold. 6. The method of claim 1, further comprising limiting the subset of the second network incident to network incidents affecting network flows, each network flow having at least one corresponding terminal node in a shared geographical area. 7. The method of claim 1, further comprising selecting a subset of the second network incidents, wherein selecting a subset of the second network incidents comprises: identifying network incidents affecting network flow between the first terminal node and the second terminal node based on a first terminal node and a second terminal node, the first terminal node having network addresses within a first set of network addresses, and the second terminal node having network addresses within a second set of network addresses. 8. The method of claim 7, further comprising: selecting the first set of network addresses and the second set of network addresses based on a shared network link between nodes addressed in the first set of network addresses and nodes addressed in the second set of network addresses. 9. The method according to claim 1, wherein the measurement period is a rolling time window. 10. A system for maintaining network service levels, the system comprising: Computer-readable storage device for storing records of network incidents; and One or more processors are configured to access the computer-readable memory and execute instructions that, when executed by the processor, cause the processor to: The records of network incidents stored in the computer-readable storage are used to identify a first plurality of network incidents that occurred during a first portion of the measurement period, the first plurality of network incidents being designated as a first network incident. Identify a second plurality of network incidents that occur after the first portion of the measurement period and during the second portion of the measurement period, the second plurality of network incidents being designated as a second network incident; Multiple remaining incident tolerance limits are determined based on the impact of the first plurality of network incidents and the second plurality of network incidents on the corresponding set of incident tolerance limits for the measurement period, wherein the incident tolerance limit is a tolerance level for network incidents over a time period, and wherein the remaining incident tolerance limits are obtained by reducing the incident tolerance limit as each incident occurs. Generate a severity metric for each second network incident, wherein each second network incident is assigned a score representing the severity of the incident based on one or more metrics, and wherein the severity metric is weighted based on the remaining incident tolerance limit associated with each second network incident; and At least one incident in the second network incident is selected for remediation based on a comparison of the severity metrics. 11. The system of claim 10, wherein the identified first plurality of network incidents and second plurality of network incidents are represented by network incident records stored in the computer-readable storage medium. 12. The system of claim 11, wherein the network incident log includes at least: (i) Information on the time and date of the accident. (ii) the routing information for the occurrence of the aforementioned incident, and (iii) A description or classification of the services affected by the incident. 13. The system of claim 10, wherein the instructions, when executed by the processor, cause the processor to limit the subset of the second network incidents to network incidents that satisfy the selection criteria. 14. The system of claim 13, wherein the instruction, when executed by the processor, causes the processor to select a subset of the second network incidents based on the fact that the count of network incidents satisfying the selection criterion exceeds a threshold. 15. The system of claim 10, wherein the instructions, when executed by the processor, cause the processor to limit the subset of the second network incidents to network incidents affecting network flows, each network flow having at least one corresponding terminal node in a shared geographical area. 16. The system of claim 10, wherein when the instruction is executed by the processor, the processor causes the processor to select a subset of the second network incident, wherein selecting a subset of the second network incident comprises: identifying network incidents affecting network flow between the first terminal node and the second terminal node based on a first terminal node and a second terminal node, the first terminal node having a network address in a first set of network addresses, and the second terminal node having a network address in a second set of network addresses. 17. The system of claim 16, wherein, when executed by the processor, the instructions cause the processor to select the first set of network addresses and the second set of network addresses based on a shared network link between nodes addressed in the first set of network addresses and nodes addressed in the second set of network addresses. 18. The system of claim 10, wherein the measurement period is a rolling time window.