CN116615898A - Maintaining quality of service handling of packets using security parameter index values - Google Patents

Abstract

Techniques for load balancing encrypted traffic based on a Security Parameter Index (SPI) value of a packet header and a set of five tuple values of the packet header are described herein. Further, techniques for including quality of service (QoS) type information in an SPI value field of a packet header are described herein. The QoS type information may indicate a particular traffic class from which the packet is to be processed. In addition, techniques for pre-configuring a back-end host such that encrypted traffic may migrate from another back-end host to the back-end host without causing a temporary service disruption are also described herein.

Description

Quality of service handling of maintenance packets using security parameter index values

related application

This application claims priority to U.S. Utility Patent Application No. 17/171,604, filed February 9, 2021, which claims priority to U.S. Provisional Patent Application No. 63/124,317, filed December 11, 2020, The entire contents of both applications are incorporated herein by reference.

technical field

The present disclosure generally relates to improved techniques for load balancing encrypted traffic using security parameter index (SPI) values in packet headers.

Background technique

Building a cloud-delivered software-as-a-service (SaaS) product involves creating a distributed system that is delivered to users in the cloud. Typically, traffic is sent into these services according to one or more routing policies, such as Equal Cost Multipath (ECMP) routing. ECMP and other routing strategies allow fixed flows based on "5-tuples" to send packets to specific backend instances. A grouped quintuple generally refers to a set of five distinct values comprising a Transmission Control Protocol/Internet Protocol (TCP/IP) connection. The set of values for the 5-tuple includes the source IP address, source port number, destination IP address, destination port number, and the particular protocol being used.

However, because ECMP and other routing strategies use quintuples, they do not take into account the individual components of encrypted connections such as Internet Protocol Security (IPsec) connections, which include Internet Key Exchange (IKE) traffic and Encapsulating Security Payload (ESP) traffic. flow. Additionally, encrypted connections (such as IPsec) are difficult to provide traffic classification for due to their encrypted nature. Once the packets are encrypted and encapsulated, it becomes nearly impossible to enforce any form of Quality of Service (QoS).

Additionally, in networked environments where a load balancer is placed in front of a pool of worker nodes responsible for handling encrypted traffic, when a worker node goes offline, the encrypted sessions assigned to the worker node must migrate to one or more other hosts. This typically results in a temporary service interruption while (one or more) new host worker nodes and clients negotiate new encrypted connections.

Description of drawings

A detailed description will be made below in conjunction with the accompanying drawings. In the figures, the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical items. The systems depicted in the figures are not to scale and components in the figures may be depicted not to scale with each other.

1 shows a schematic diagram of an example system architecture of a networking environment including a tunneled communication session that includes separate control plane and data plane traffic flows.

2 shows a schematic diagram of an example traffic flow in which a load balancer node sends traffic to downstream nodes according to one or more routing policies.

3 illustrates a data flow diagram of an example traffic flow between various nodes and/or devices for establishing a load-balanced communication session of traffic using SPI values of packet headers.

4A and 4B illustrate data flow diagrams of example traffic flows between various nodes and/or devices of a communication session for indicating QoS type information in SPI values in packet headers.

5A-5C collectively illustrate a schematic diagram of example data flows associated with performing encrypted tunnel migration.

6 illustrates a logic flow diagram of an example method for maintaining QoS handling of packets by using SPI values.

7 illustrates a logic flow diagram of an example method for load balancing traffic based on SPI values in packet headers.

8 illustrates a logic flow diagram of an example method for performing encrypted tunnel migration.

9 illustrates a logic flow diagram of another example method for performing encrypted tunnel migration.

10 shows a schematic diagram of an example computer hardware architecture for implementing network nodes and/or devices (e.g., load balancers, control nodes, data nodes, etc.) that can be used to implement the various aspects presented herein. all aspects of technology.

Detailed ways

overview

Various aspects of the invention are set out in the independent claims and preferred features are set out in the dependent claims. Features of one aspect may apply to each aspect alone or in combination with the other aspects.

This disclosure describes systems and methods that improve techniques related to load balancing encrypted traffic by using security parameter index (SPI) values in packet headers. By way of example and not limitation, methods according to various techniques described in this disclosure may include receiving from a client device and at a network device of the network establishing an encrypted tunnel through the network such that data plane traffic can be transmitted between the client device and the network device via the encrypted tunnel. Requests that flow between services. The method may also include generating an SPI value to be used by the client device for data plane traffic and sending an indication of the SPI value to the client device. Additionally, the method can include receiving, at the load balancer, the data packet including the SPI value, and based at least in part on the SPI value, determining to send the data packet to a server of the set of servers supporting the service. Accordingly, the load balancer can send data packets to the servers.

In some cases, the method may additionally or alternatively include determining that data plane traffic belongs to a particular traffic class within a set of traffic classes. Certain traffic classes can be associated with certain Quality of Service (QoS) performance metrics. Accordingly, the method may include generating an SPI value to be used by the client device for data plane traffic. SPI values may correspond to specific traffic classes. In this manner, the load balancer can receive data packets of data plane traffic that include SPI values, and based at least in part on the data packets that include SPI values, the load balancer can send the data packets over the network such that processing according to a particular QoS performance metric Data grouping.

In an additional or alternative example, the method can include receiving, at the load balancer and from the client device, the first data plane traffic having the first SPI value and the set of quintuple values. Based at least in part on the first SPI value and the set of 5-tuple values, the load balancer can send the first data plane traffic to the first node. The method may also include receiving, at the load balancer, an indication that at least a portion of the first data plane traffic is to be sent to the second node. Based at least in part on the indication, the load balancer can prompt the second node to provide one or more interfaces such that at least a portion of the first data plane traffic can be sent to the second node. In this manner, the load balancer can receive second data plane traffic from the client device having the second SPI value and the set of 5-tuple values. Based at least in part on the second SPI value and the set of 5-tuple values, the load balancer may determine that the second data plane traffic includes at least the portion of the first data plane traffic, and in response, send the second data plane traffic to the second node.

Additionally, the techniques described herein can be performed as a method and/or by a system having a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, perform the techniques described herein .

example embodiment

As mentioned above, traffic is typically sent to various services according to one or more routing policies, such as equal-cost multi-path (ECMP) routing. However, because these routing policies use quintuples, they do not take into account individual flows of encrypted connections such as Internet Protocol Security (IPsec) connections, which include Internet Key Exchange (IKE) traffic and Encapsulating Security Payload (ESP) traffic. This means that the entropy of these flows may be much smaller than would be achievable if the per-tunnel entropy was provided by Security Associations (SAs) for IPsec IKE and ESP flows. For example, IPsec IKE and ESP traffic contain additional identifiers called Security Parameter Indexes (SPIs). The SPI value is used to uniquely identify the established IPsec SA.

Accordingly, one aspect of the present disclosure provides techniques for utilizing SPI values to allow load balancing and pinning each IPsec IKE and ESP flow to a specific backend. By implementing these techniques, several advantages can be realized, including the ability to terminate the same encrypted tunnel/SA on multiple systems, which allows scaling capacity. Additionally, higher performance can be achieved by having finer-grained control over where control-plane and data-plane traffic sessions may land on backend nodes/servers. Furthermore, different traffic can be handled accordingly (e.g. performance levels, allowing customers to have their own backend).

Additionally, encrypted connections (such as IPsec) are difficult to provide for traffic classification due to their encrypted nature. Once the packets are encrypted and encapsulated, it becomes nearly impossible to enforce any form of Quality of Service (QoS). For example, when looking specifically at building an IPsec cloud-delivered virtual private network (VPN) product, you're dealing with ESP in IP packets and/or ESP in UDP packets. Accordingly, another aspect of the present disclosure includes techniques for encoding and mapping QoS type information into SPI values so that QoS can be performed on packets even after the packets are encrypted and encapsulated. Executing these techniques can allow the classification of encrypted traffic, allowing DiffServ-style resource allocation on data nodes, making it possible to allocate individual SAs on data nodes more appropriate for that traffic class, and also to adjust Traffic shaping rules within a data node to adjust network throughput among the streams assigned to it.

Additionally, in networked environments where a load balancer is placed in front of a pool of worker nodes responsible for handling encrypted traffic, when a worker node goes offline, the encrypted sessions assigned to the worker node must migrate to one or more other hosts. This typically results in a temporary service interruption while (one or more) new host worker nodes and clients negotiate new encrypted connections. Accordingly, yet another aspect of the present disclosure includes techniques for adding support for backend worker nodes (e.g., data nodes) to signal a load balancer to indicate that a worker node is about to be removed from the backend worker node queue . In this way, these techniques can reduce the impact of planned or unplanned shutdowns by migrating encrypted tunnels out of a worker node when it goes into an unhealthy state or is set to be replaced by another node. Additionally, these techniques can reduce the impact of rebalancing load across server pools.

Therefore, according to various techniques described in this disclosure, improvements in computer-related techniques can be realized. As mentioned earlier, the entropy for an encrypted stream may be much smaller than would be achievable if per-tunnel entropy was provided. For example, most routing policies (like ECMP) use a set of five-tuple values for hashing. However, exploiting the packet's SPI value may allow the use of six-tuple logic for better distribution of flows to headend nodes. In addition, even if the packet has been encapsulated, the SPI value can be further used to indicate the QoS type information of the data packet, so that a separate SA can be allocated on the data node that is more suitable for this traffic class. These are just a few examples of the many improvements that can be made in accordance with the techniques described in this disclosure. These and other improvements will be readily apparent and understandable to those of ordinary skill in the art.

By way of example and not limitation, methods according to various techniques described in this disclosure may include receiving a packet from a client device indicating establishment of an encrypted tunnel through the network such that data plane traffic may pass between the client device and the service via the encrypted tunnel Flow requests. In some examples, the request packet may be received at the network by a load balancer or router of the network and the load balancer or router may send the request to a control node of the network. Additionally, the load balancer or router can send the request packet to the control node based at least in part on an equal cost multipath (ECMP) routing policy and/or a 5-tuple associated with the request packet. In some examples, the request to establish an encrypted tunnel may include a request to establish an IPsec connection and/or a request to establish an IPsec SA or sub-SA.

In some examples, a network may be configured such that the network includes separate control nodes and data nodes. In other words, the network can be configured to split the processing of control plane traffic (e.g., IKE traffic) and data plane traffic (e.g., ESP traffic) to different nodes (e.g., a control node or "IKE " nodes and data nodes for processing the data plane). This may allow the network to scale each node type individually and/or independently. Control nodes and data nodes may include head-end servers associated with the service. In some examples, the control nodes can operate on a first set of computing resources associated with the network, and the data nodes can operate on a second, different set of computing resources associated with the network.

In some examples, the method can include determining that data plane traffic belongs to a particular traffic class of a set of traffic classes. Certain traffic classes can be associated with certain Quality of Service (QoS) performance metrics. In some examples, the classifier may be invoked prior to establishing the SA in order for the control node to create an SPI value that matches the correct traffic class. This can be done in a number of different ways. For example, a load balancer or router can invoke a classifier and inject the class information as a header before control plane traffic is forwarded to the control node. This can be done by using currently unused fields in the IP header (such as the DSCP field) or by creating new fields. Additionally, or alternatively, the control node may invoke the classifier when it initiates an SA. By any of these means, the control node can be provided with class information so that the control node can create an SPI value corresponding to the traffic class.

In some examples, the method can include generating an SPI value to be used by the client device for data plane traffic. An SPI value may include a bit combination that identifies a particular SA. In some cases, multiple SPI values may be generated, and each individual SPI value of the multiple SPI values may identify a corresponding SA. Additionally, SPI values can be generated by the control node.

As noted above, in various examples, the SPI value may include QoS type information (e.g., Differentiated Services (DiffServ) Type Information, Type of Service (ToS), Differentiated Services Code Point (DSCP) Type Information, and/or Experimental Bit (EXP ) type information) indicating the specific traffic class according to which the packet is processed. Thus, in some examples, generating the SPI value may include generating a first bit combination representing a particular traffic class according to which the packet is to be processed, generating a second bit combination representing a particular SA, and masking or combining the first bit combination and the second bit combination. The two bits are combined such that the first bit combination includes a first part of the SPI value and the second bit combination includes a second part of the SPI value. For example, the SPI field of a packet is defined as an arbitrary value of 32 bits, and the range of 0-255 is defined as reserved. This leaves values from 256 (0x00000100) to 4294967295 (Oxffffffff) to be used as SPI values. Thus, in some examples, the first part (e.g., the "front" part) of the 32-bit field can be used for QoS mapping, and the first hex digit can be "reserved" by offsetting the SPI value by 4 bits for map. For example, using the hexadecimal values 0x[0]3ec7b2a to 0x[f]3ec7b2a, the hexadecimal digits [0] to [f] could represent the QoS mapping, and 0x3ec7b2a could represent the actual SPI offset of the consumed 4 bits . That is, the first part of the SPI value (for example, hexadecimal digits [0] to [f]) may represent a specific traffic class QoS mapping, and the second part of the SPI value (for example, hexadecimal digits 3ec7b2a ) can identify a specific SA. This results in 15 mapped values being used. Additionally or alternatively, the already established 802.1q Class of Service (CoS) or Multiprotocol Label Switching (MPLS) EXP to DSCP bitmaps may be followed since they have similar bit sizes.

After generating the SPI, the method may include sending an indication of the value of the SPI to the client device. In some examples, the control node may perform a direct server return (DSR) to send the indication to the client device. In some examples, data packets including SPI values may be received by a load balancer. The data packets may include data packets of data plane traffic. That is, the protocol associated with the data packets may correspond to a data plane traffic protocol, such as ESP. In some examples, a data packet may include a set of five-tuple values. For example, a set of 5-tuple values for a data packet may include a source IP address value, a source port value, a destination IP address value, a destination port value, and a protocol associated with the data packet. As described herein, a set of six-tuple values may be used to refer to an SPI value and a set of five-tuple values. That is, a set of six-tuple values may include a source IP address value, a source port value, a destination IP address value, a destination port value, a protocol associated with the data packet, and an SPI value. However, different values can be used.

In some cases, a load balancer may determine to send a data packet to a server (eg, a data node) in a set of servers or nodes that support the service. For example, based at least in part on the SPI value and/or the set of 5-tuple values, the load balancer can determine to send the data packet to the server. In some examples, the load balancer can receive data representing an association between the SPI value and the set of 5-tuple values associated with the client device, and determining to send the data packet to the server can be further based at least in part on the data. That is, the load balancer can be updated with a map indicating the association between SPI values and quintuple values. In some examples, determining to send the data packet to the server may be based at least in part on computing a hash value representing the SPI value and/or the set of 5-tuple values. The load balancer can use a hash function to calculate the hash. Additionally or alternatively, the load balancer may determine to send the data packet to the server based at least in part on one or more routing policies (eg, ECMP).

In some examples, the method can include sending the data packet to the server. Additionally, data packets may be sent over the network such that the data packets are processed according to a particular QoS performance metric and/or traffic class. For example, if the SPI value includes an indication of a particular traffic class and/or QoS performance metric according to which the data packet is to be processed, the load balancer may send the packet over the network according to the particular traffic class and/or QoS performance metric.

In an additional or alternative example, the method can include generating a second SPI value for use by the client device for data plane traffic. The second SPI value may identify a second SA. The second SPI value may be generated by the control node. In some examples, generating the second SPI value may be based at least in part on classifying the request packet to determine a traffic class associated with the request packet, as described above. After generating the second SPI, the method may include sending an indication of the second SPI value to the client device. In some examples, the control node may perform a direct server return (DSR) to send an indication of the second SPI value to the client device.

In some examples, the method can include receiving, at the load balancer, a second data packet including a second SPI value. Additionally, the second data packet may comprise the set of 5-tuple values, a portion of the set of 5-tuple values, or a new set of 5-tuple values. In some cases, the set of 5-tuple values may indicate that the second data packet was sent by the client device or a different client device. Based at least in part on the second data packet including the second SPI value and/or the set of 5-tuple values, the load balancer may send the second data packet to a second server (e.g., a second data node) in the server group . In some examples, the first portion of the second SPI value may correspond to a second traffic class associated with a second QoS performance metric. As such, based at least in part on the second data packet including the second SPI value, the load balancer can send the second data packet over the network such that the second data packet is processed according to the second QoS performance metric.

As noted above, an aspect of the techniques described herein may also include adding support for backend worker nodes (such as data nodes) to signal the load balancer that a worker node is about to be removed from the backend worker node queue. remove. Accordingly, in an additional or alternative example, the method can include receiving, at the load balancer, from the client device, the first data plane traffic having the first SPI value and the set of quintuple values. The first SPI value may identify a first security association (SA) between the client device and the first node.

In some examples, the method can include sending the first data plane traffic to a first node of a set of nodes. In at least one example, the first node may include a first data node in a set of data nodes. The first node may be associated with a first encrypted tunnel (eg, IPsec SA). In some examples, sending the first data plane traffic to the first node may be based at least in part on the first SPI value and the set of 5-tuple values. For example, the load balancer may compute a hash value (eg, a six-tuple) representing the first SPI value and the set of five-tuple values. Based at least in part on the hash value, the load balancer can send the first data plane traffic to the first node according to the ECMP routing policy.

In various examples, the method can include receiving, at the load balancer, information that additional data plane traffic to be received from the client device is to be sent to a second node of the set of nodes. In at least one example, the second node may include a second data node in the set of data nodes. The second node may be associated with a second encrypted tunnel (eg, a second IPsec SA). In some cases, the indication may include an indication that at least a portion of the first data plane traffic is to be sent to the second node. That is, the indication can inform the load balancer that it needs to adjust where it sends data plane traffic. For example, the load capacity associated with the first node may meet or exceed a threshold load capacity. Additionally or alternatively, the indication may inform the load balancer that the first node is about to be removed from the set of nodes (eg, taken offline, serviced, etc.).

In some examples, a controller associated with the network may send or cause the indication to be sent to the load balancer. For example, a controller may receive telemetry data from the set of nodes. Based at least in part on the telemetry data, the controller may determine that the load balancer will adjust where it sends data plane traffic and/or control plane traffic. For example, telemetry data may indicate load capacity associated with respective nodes of the set of nodes. Additionally or alternatively, the telemetry data may indicate a state associated with a respective node of the set of nodes (eg, whether a node is unhealthy, hung, or crashed, whether a host is about to be rotated, etc.). Accordingly, the controller may send an indication to the load balancer and/or, in some examples, a notification to the first node prompting the first node to send the indication to the load balancer. In some examples, the controller may include a distributed system that includes a key-value store.

Based at least in part on the indication, in some examples, the load balancer and/or controller may prompt the second node to provide one or more resources such that a portion of the first data plane traffic may be sent to the second node. One or more resources may include interfaces, channels, computing resources, and the like. In this way, by prompting the second node to provide one or more resources, the second node can "warm up" before data plane traffic is sent to the second node. In examples where part of the data plane traffic is redirected from a first node to a second node, warming up or pre-configuring the second node can help reduce downtime and/or temporary service disruption while the second node and client devices negotiate A new encrypted connection. In at least one example, prompting the second node to provide the one or more resources may include generating and/or sending an empty Encapsulating Security Payload (ESP) packet to the second node. An empty ESP packet may include an Internet Protocol (IP) address and port associated with the client device, and in some cases other five-tuple values.

In some examples, the load balancer and/or the controller may send a request to the third node for the third node to generate the second SPI value. The load balancer and/or controller can send the request based at least in part on the indication. In at least one example, the third node includes a first control node (eg, an IKE node) in a set of control nodes. Additionally, the request may include a "rekey" request. That is, the request may be for the third node to create a second SA between the client device and the second node in place of the first SA between the client device and the first node. Accordingly, in some examples, the method may include receiving, at the load balancer, an indication of the second SPI value. The indication of the second SPI value may include an indication of an association between the second SPI value and the set of 5-tuple values.

In some examples, the method can include receiving, at the load balancer and from the client device, second data plane traffic having the second SPI value and the set of 5-tuple values. The second data plane traffic may include some (eg, a portion) or all of the first data plane traffic that the load balancer previously sent to the first node. Based at least in part on the second data plane traffic having the second SPI value and the set of quintuple values, the method may include, in some cases, determining that the second data plane traffic includes some (e.g., part) or all. For example, the load balancer may not know the second SPI value, and the load balancer may keep track of all SPI values associated with a particular set of quintuple values. Once the load balancer issues a rekey request, it may start monitoring for new/unknown SPIs associated with this set of quintuple values and send all data plane traffic with new/unknown SPIs to the second node.

In some examples, based at least in part on the second SPI value and the set of 5-tuple values, the method can include sending the second data plane traffic to the second node. Additionally, in some cases, the method may include removing a first association between a first SPI value and the set of quintuple values and/or storing a second association between a second SPI value and the set of quintuple values .

Certain embodiments and examples of the present disclosure will now be described more fully below with reference to the accompanying drawings, in which various aspects are shown. Aspects may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein. For example, while many of the examples herein are described with respect to ECMP routing, it should be understood that other routing strategies may be used. Additionally, while many examples are shown as distributed systems, it should be understood that the various processes and methods described may be performed by more or fewer devices. As described herein, the present disclosure encompasses variations of the examples. Like numbers refer to like elements throughout.

FIG. 1 shows a schematic diagram of an example system architecture 100 including a networked environment 102 of tunneled communication sessions that include separate control-plane and data-plane traffic flows. Generally, networked environment 102 may include equipment housed or located in one or more data centers 104, which may be located in different physical locations. For example, networked environment 102 may be supported by a network of devices in a public cloud computing platform, a private/enterprise computing platform, and/or any combination thereof. One or more data centers 104 may be physical facilities or buildings located over a geographic area designated to store networked devices that are part of networked environment 102 . Data center 104 may include various networking equipment, as well as redundant or backup components and infrastructure for power, data communication connections, environmental controls, and various security equipment. In some examples, data center 104 may include one or more virtual data centers, which are pools or collections of cloud infrastructure resources specifically designed for enterprise needs and/or cloud-based service provider needs. In general, data centers 104 (physical and/or virtual) may provide basic resources such as processors (CPUs), memory (RAM), storage (disks), and networks (bandwidth). However, in some examples, devices in networked environment 102 may not be located in well-defined data center 104, but may be located in other locations or buildings.

Networked environment 102 may be accessed by client devices 106 over one or more networks 108 . Networked environment 102 and network 108 may each include one or more networks implemented by any feasible communication technology (eg, wired and/or wireless modalities and/or technologies), respectively. Networked environment 102 and network 108 may each include a personal area network (PAN), a local area network (LAN), a campus network (CAN), a metropolitan area network (MAN), an extranet, an intranet, the Internet, a short-range wireless communication network such as ZigBee , Bluetooth, etc.), Virtual Private Network (VPN), Wide Area Network (WAN) - centralized and/or distributed - and/or any combination, permutation and/or aggregation thereof. Networked environment 102 may include devices, virtual resources, or other nodes that relay packets from one network segment to another through nodes in a computer network.

In some examples, networked environment 102 may provide one or more services 110, host one or more services 110, provide a connection to or otherwise support one or more services 110 for client Device 106 is connected and used. Client device 106 may include any type of device configured to communicate over network 108 using various communication protocols (eg, VPN, SSL, TLS, DTLS, and/or any other protocol). For example, client devices 106 may include personal user devices (e.g., desktop computers, laptop computers, phones, tablets, wearable devices, entertainment devices such as televisions, etc.), network devices (e.g., servers, routers, switches, access point, etc.) and/or any other type of computing device.

In some examples, networked environment 102 may include edge routers 112(1) and 112(2) (hereinafter collectively referred to as "edge routers 112"), load balancers 114(1)-114(N) (hereinafter collectively referred to as "load balancer 114") (where N represents any number greater than or equal to one), data nodes 116(1)-116(N), control nodes 118(1)-118(N), firewall nodes 120(1)-120 (N), a key-value storage device 122 and a controller 124 . In various examples, various systems/devices/nodes of networked environment 102 may communicate with each other via a management plane and/or a message bus associated with networked environment 102 . For example, a common message bus associated with networked environment 102 may enable a data node to signal to a load balancer that it is about to be removed from the data node queue, that the load balancer needs to adjust where it will flow to, and so on. Additionally, a message bus associated with networked environment 102 may enable any devices/systems/nodes of networked environment 102 to communicate directly with each other.

In some examples, edge routers 112 and load balancers 114 may use ECMP, which is a policy in which next-hop packet forwarding to a single destination may occur over multiple "best paths" between Tied for first place in routing metric calculations. Additionally, edge routers 112 and load balancers 114 may use any routing strategy in conjunction with or instead of ECMP routing, such as Open Shortest Path First (OSPF), Intermediate System to Intermediate System (ISIS), Enhanced Interior Gateway Routing Protocol (EIGRP), domain name System (DNS) load balancing and/or Border Gateway Protocol (BGP). Although shown in FIG. 1 as separate entities, it is to be understood that in some cases edge router 112 and load balancer 114 may reside on the same hardware device and/or node.

In some cases, edge router 112 may balance traffic 126 based on a hash of the network quintuple in order to route packets to load balancer 114 . Traffic 126 may include control plane traffic 128 and data plane traffic 130 . Additionally, load balancer 114 may balance traffic 126 based on the hash of the network six-tuple to route control plane traffic 128 to control node 118 and data plane traffic 130 to data node 116 . The packet's network six-tuple may include the packet's SPI value, source IP address, source port, destination IP address, destination port, and protocol.

As shown, networked environment 102 may include data nodes 116(1)-116(N) (collectively hereinafter "data nodes 116") (where N represents any number greater than or equal to one). In some examples, data nodes 116 may process data plane traffic 130 on behalf of networked environment 102 . Data plane traffic 130 may include ESP traffic associated with IPsec connections. In some examples, data node 116(1) of data nodes 116 may be associated with one or more IPsec security associations. Additionally, data nodes 116 may forward data plane traffic 130 to one or more downstream nodes and/or devices, such as firewall nodes 120(1)-120(N) (hereinafter collectively referred to as "firewall nodes 120") (where N represents any number greater than or equal to one). In some examples, a first one of data nodes 116 may be associated with a first traffic class, a second one of data nodes 116 may be associated with a second traffic class, and so on. Additionally or alternatively, the first interface of the first data node in the data nodes 116 may be associated with the first traffic class, the second interface of the first data node in the data nodes 116 may be associated with the second traffic class, etc.

Networked environment 102 may also include one or more control nodes 118(1)-118(N) (collectively hereinafter "control nodes 118") (where N represents any number greater than or equal to one). In some examples, control node 118 may handle control plane traffic 128 on behalf of networked environment 102 . Control plane traffic 128 may include IKE traffic associated with IPsec connections.

As shown, both data node 116 and control node 118 may perform direct server return (DSR) to send return traffic 132 back to client device 106 . That is, data nodes 116 and control nodes 118 may send return traffic 132 to client device 106 via edge router 112 ( 1 ), bypassing load balancer 114 . Additionally or alternatively, data nodes 116 and control nodes 118 may send return traffic 132 directly to client devices, bypassing edge router 112(1).

The networked environment 102 may also include a key-value store 122 and a controller 124 . Key-value store 122 may include one or more databases that are accessible by various nodes and devices of networked environment 102 . In some examples, load balancers 114 , data nodes 116 , control nodes 118 , and other nodes and/or devices of networked environment 102 may read data from and write data to key-value store 122 . Key-value store 122 may store associations between SPI values and SAs, sets of SPI values and quintuple values, and the like. In some examples, controller 124 may receive telemetry data from data nodes 116 and/or control nodes 118 and determine states associated with each of data nodes 116 and/or control nodes 118 based at least in part on the telemetry data . For example, controller 124 may receive telemetry data indicative of load capacity associated with data node 116(1). Controller 124 may also determine whether the load capacity meets or exceeds a threshold load capacity, and if so, controller 124 may prompt data node 116(1) to send a notification to load balancer 114(1) to request that load balancer 114(1) ) adjusts where it sends data plane traffic 130 to.

Although depicted in FIG. 1 as separate hardware components, it should be understood that edge routers 112, load balancers 114, data nodes 116, control nodes 118, firewall nodes 120, key-value stores 122, and/or controllers 124 may be at least part of A software component that resides in memory. In this manner, one or more processors may execute instructions that cause the one or more processors to perform the processing described herein with respect to edge router 112, load balancer 114, data node 116, control node 118, firewall node 120, key-value All operations described by the storage device 122 and/or the controller 124. In some cases, edge routers 112, load balancers 114, data nodes 116, control nodes 118, firewall nodes 120, key-value stores 122, and/or controllers 124 may be resident on stand-alone appliances or systems of stand-alone appliances separate hardware and/or software components. Additionally or alternatively, edge routers 112, load balancers 114, data nodes 116, control nodes 118, firewall nodes 120, key-value stores 122, and/or controllers 124 may comprise any type of network equipment, such as servers, switches, , routers, hubs, bridges, gateways, modems, repeaters, access points, etc.

2 shows a schematic diagram of an example traffic flow 200 in which load balancer 114(1) sends traffic to downstream nodes according to one or more routing policies. For example, load balancer 114(1) may receive incoming tunneled traffic 202 from client devices (eg, one or more client devices 106). In some cases, incoming tunnel traffic 202 may include control plane traffic 128 and/or data plane traffic 130 . Additionally, incoming tunnel traffic 202 may include an SPI value and a set of 5-tuple values.

In some examples, when load balancer 114(1) receives incoming tunnel traffic 202, load balancer 114(1) may calculate a hash value representing the SPI value of incoming tunnel traffic 202 and the set of five-tuple values . Load balancer 114(1) may then determine the particular node of data nodes 116 or control nodes 118 to which incoming tunnel traffic 202 is to be sent based at least in part on the hash value and using ECMP routing policies. For example, if incoming tunnel traffic 202 includes control plane traffic 128 (eg, IKE traffic), load balancer 114(1) may send control plane traffic 128 to one of control nodes 118 based at least in part on the hash value. Likewise, if incoming tunnel traffic 202 includes data plane traffic 130 (eg, ESP traffic), load balancer 114(1) may send data plane traffic 130 to one of data nodes 116 based at least in part on the hash value.

3 shows a data flow diagram of an example traffic flow 300 between various nodes and/or devices for establishing a load-balanced communication session of traffic using SPI values of packet headers. Example traffic flow 300 includes client 302 , router/load balancer 304 , first headend 306 , and second headend 308 . In an example, the first headend 306 and the second headend 308 may include data nodes, control nodes, servers, and the like. For example, the first head-end 306 may include a control node and the second head-end 308 may include a data node.

To begin the example traffic flow 300 , the client 302 sends a connection request packet 310 to the router/load balancer 304 . Connection request packet 310 may indicate a request to establish an encrypted tunnel so that traffic may flow from client 302 to second headend 308 . Connection request packet 310 may include a set of five-tuple values. After receiving the connection request packet 310 , the router/load balancer 304 can send the connection request packet 310 to the first headend 306 . Router/load balancer 304 may determine to send connection request packet 310 to first headend 306 based at least in part on computing a hash value representative of the set of quintuple values included in connection request packet 310 . Additionally or alternatively, router/load balancer 304 may determine to send connection request packet 310 to first headend 306 based at least in part on an ECMP routing policy.

After receiving the connection request packet 310 , the first headend 306 can establish an IKE session 314 with the client 302 . In this way, IKE traffic can flow between client 302 and first headend 306 . In some cases, establishing the IKE session 314 may include authenticating the user 302 associated with the client, such as by determining the identity of the user. Once the IKE session is established, the first headend may send a reply packet 316 to the client 302 . Reply packet 316 may indicate that an IKE session has been established.

Client 302 may then send ESP traffic 318 to router/load balancer 304 , and router/load balancer 304 may forward this ESP traffic 318 to second headend 308 . After receiving ESP traffic 318, second headend 308 may generate SPI value 320 for use by client 302 in sending data plane traffic over the ESP channel. The second head-end 308 may further associate the SPI value with the set of 5-tuple values. In this manner, second headend 308 can update router/load balancer 304 with quintuple and SPI mapping 322 . In some cases, quintuple and SPI map 322 may include a hash value. Additionally or alternatively, the quintuple and SPI map 322 may indicate that future data plane packets including certain sets of quintuple values and certain SPI values are to be sent to the second head-end 308 . The second head-end 308 may then send a reply packet 324 back to the client 302 . Reply packet 324 may indicate that client 302 may begin sending data plane traffic 326 using the ESP channel or encrypted tunnel.

After receiving the reply packet 324, the client 302 may start sending data plane traffic 326 over the ESP channel. When router/load balancer 304 receives data plane traffic, router/load balancer 304 may calculate a hash value and SPI value 328 representing the network quintuple. For example, packets of data plane traffic 326 may include SPI values and network quintuples. Based at least in part on computing the hash, router/load balancer 304 may send data plane traffic 326 to second headend 308 . For example, router/load balancer 304 can send data plane traffic 326 to second headend 308 based at least in part on quintuple and SPI map 322 .

4A and 4B illustrate example traffic flows 400(1) and 400 between various nodes and/or devices of a communication session for indicating an SPI value in a packet header and/or QoS type information in an SPI value field. (2) The data flow diagram. Example traffic flows 400 ( 1 ) and 400 ( 2 ) may include client 302 , router/load balancer 304 , IKE node 402 , and classifier 404 .

Referring to FIG. 4A , client 302 may send connection request packet 406 to router/load balancer 304 . In some examples, connection request packet 406 may include an IKE SA INIT request packet. Connection request packet 406 may indicate a request to establish an encrypted tunnel (eg, an IPsec connection) for client 302 to send data to and/or receive data from the service. Router/load balancer 304 , upon receiving connection request packet 406 , may invoke classifier 404 to determine the traffic class associated with connection request packet 406 . For example, connection request packet 304 may indicate the type of traffic that client 302 wishes to send and/or receive (eg, voice, video, audio, network, etc.), and classifier 404 may be configured to determine what type of traffic that is. Additionally or alternatively, connection request packet 406 may include a request to establish multiple connections, each connection being associated with a different traffic class and/or priority. In some examples, invoking classifier 404 may include sending, by router/load balancer 304 , connection request packet 406 to classifier 404 .

In some examples, classifier 404 may operate on data packet 408 to determine a traffic class associated with connection request data packet 406 . For example, classifier 404 may determine that connection request packet 406 includes a request to establish one or more of a voice traffic channel, a video traffic channel, an audio traffic channel, a network traffic channel, and the like. Classifier 404 may send classification packet 410 indicating the traffic class associated with connection request packet 406 after determining the traffic class. Classifier 404 may send classified packet 410 to router/load balancer 304 . In turn, router/load balancer 304 may inject an indication of the traffic class classification into the packet header of connection request packet 406 . In this manner, connection request packet 406 may include an updated connection request packet 414 that includes an indication of traffic class classification information in its packet header.

Router/load balancer 304 may send updated connection request packet 414 to a control node, such as IKE node 402 . IKE node 402 may receive updated connection request packet 414 . IKE node 402 may generate one or more SPI values 416 based at least in part on the traffic class classification information included in the packet header of updated connection request packet 414 . One or more SPI values 416 may indicate, in whole or in part, a traffic class according to which data plane traffic is to be processed. That is, a particular SPI value may include a unique bit combination, and a first bit combination (e.g., the first portion of the SPI value) of the unique bit combination may indicate a traffic class, and a second bit of the unique bit combination The combination (eg, the second portion of the SPI value) can identify a security association between client 302 and one or more hosts associated with the encrypted tunnel connection. In other words, the SPI value field of the data plane packet header may include a combination of first bits indicating the traffic class and a second bit identifying a security association between the client 302 and one or more hosts associated with the encrypted tunnel connection combination.

After generating one or more SPI values 416 , IKE node 402 may send a response packet 418 . In some cases, response packet 418 may include an IKE INIT response packet. Additionally or alternatively, response packet 418 may include some or all of one or more SPI values 416 . In this manner, client 302 may use a first SPI value of one or more SPI values 416 to send first data plane traffic according to a first traffic class associated with a first QoS metric and may use one or more SPI The second SPI value in value 416 is to send the second data plane traffic according to the second traffic class associated with the second QoS metric.

Referring to FIG. 4B , client 302 may send connection request packet 406 to router/load balancer 304 . In some examples, connection request packet 406 may include an IKE SAINIT request packet indicating a request by IKE node 402 to establish an IPsec security association. Connection request packet 406 may indicate a request to establish an encrypted tunnel (eg, an IPsec connection) for client 302 to send data to and/or receive data from the service. Router/load balancer 304 may send connection request packet 406 to IKE node 402 after receiving connection request packet 304 . In some examples, router/load balancer 304 may calculate a hash value representative of the network quintuple included in connection request packet 406 and, based at least in part on the hash value, send connection request packet 406 to the IKE node 402. For example, router/load balancer 304 may determine to send the connection request packet to IKE node 402 using an ECMP routing policy and based at least in part on the hash value.

In order for IKE node 402 to generate one or more SPI values that match the correct traffic class, IKE node 402 may invoke classifier 404 before establishing a connection for client 302 . For example, IKE node 402 can send connection request packet 406 or a portion of connection request packet 406 to classifier so that classifier 404 can determine a traffic class associated with connection request packet 406 . For example, connection request packet 406 may indicate the type of traffic that client 302 wishes to send and/or receive (eg, voice, video, audio, network, etc.), and classifier 404 may be configured to determine what type of traffic that is. Additionally or alternatively, connection request packet 406 may include a request to establish multiple connections, each connection being associated with a different traffic class and/or priority.

In some examples, classifier 404 may operate on connection request packet 408 to determine a traffic class associated with connection request packet 406 . For example, classifier 404 may determine that connection request packet 406 includes a request to establish one or more of a voice traffic channel, a video traffic channel, an audio traffic channel, a network traffic channel, and the like. Classifier 404 may send classification packet 410 indicating the traffic class associated with connection request packet 406 after determining the traffic class. Classifier 404 may send classified packet 410 to IKE node 402 .

In some examples, IKE node 402 may receive classified packet 410 . Based at least in part on a traffic class associated with connection request packet 406 , IKE node 402 may generate one or more SPI values 416 . One or more SPI values 416 may indicate, in whole or in part, a traffic class according to which data plane traffic is to be processed. That is, a particular SPI value may include a unique bit combination, and a first bit combination (e.g., the first portion of the SPI value) of the unique bit combination may indicate a traffic class, and a second bit of the unique bit combination The combination (eg, the second portion of the SPI value) can identify a security association between client 302 and one or more hosts associated with the encrypted tunnel connection. In other words, the SPI value field of the data plane packet header may include a combination of first bits indicating the traffic class and a second bit identifying a security association between the client 302 and one or more hosts associated with the encrypted tunnel connection combination.

After generating one or more SPI values 416 , IKE node 402 may send a response packet 418 to client 302 . In some cases, response packet 418 may include an IKE INIT response packet. Additionally or alternatively, response packet 418 may include some or all of one or more SPI values 416 . In this manner, client 302 may use a first SPI value of one or more SPI values 416 to send first data plane traffic according to a first traffic class associated with a first QoS metric and may use one or more SPI The second SPI value in value 416 is to send the second data plane traffic according to the second traffic class associated with the second QoS metric.

5A-5C collectively illustrate a schematic diagram of an example data flow 500 associated with performing encrypted tunnel migration. At "1," load balancer 114(1) may receive traffic 502 from one or more client devices 106 and forward traffic 502 to one or more backend nodes 504(1)-504(N) (hereinafter Collectively referred to as "backend nodes 504) (where N represents any number greater than or equal to one). Traffic 502 may include a first traffic 502(1) to be sent to node 504(1), a first traffic to be sent to node 504(2) and the Nth traffic 502(N) to be sent to node 504(N) (where N represents any number greater than or equal to one). Additionally, traffic 502 may include data plane traffic and/or Control plane traffic.

In some examples, load balancer 114(1) may determine that first traffic 502(1), second traffic 502(2), and Nth traffic 502(N) are to be sent to node 504, respectively, based at least in part on an ECMP routing policy. (1), 504(2), and 504(N). ECMP routing policies may use six-tuple logic to determine which of backend nodes 504 to send individual packets of traffic 502 . The six-tuple logic may include an SPI value for a single packet and a set of five-tuple values (source address, destination address, source port, destination port, and protocol) for a single packet. For example, each packet of first flow 502(1), second flow 502(2), and Nth flow 502(N) may each include a corresponding SPI value and a corresponding set of quintuple values, and load balancer 114 (1) A corresponding hash value may be calculated for each individual group, and the respective hash value represents a corresponding SPI value and a corresponding set of five-tuple values of each individual group. In this manner, each respective hash value of each individual packet of traffic 502 may indicate to which of backend nodes 504 the individual packet is to be sent.

At “2,” controller 124 may receive telemetry data 506 associated with backend node 504 . For example, node 504(1) may send first telemetry data to controller 124, node 504(2) may send second telemetry data to controller 124, and node 504(N) may send Nth telemetry data to controller 124 . In some examples, telemetry data 506 may indicate a load capacity associated with each of backend nodes 504 . That is, telemetry data 506 may indicate that node 504(1) is operating at 27% capacity, node 504(2) is operating at 100% capacity, and node 504(N) is operating at 17% capacity. In some examples, the load capacity associated with the backend node may include a tunnel load capacity associated with the backend node, an amount of hardware resources available/used by the backend node, an amount of virtual computing resources available/used by the backend node, etc. one or more.

At "3," controller 124 may send an indication 508 for load balancer 114(1) to adjust the data flow. That is, adjust where load balancer 114(1) sends various portions of traffic 502 (eg, which of backend nodes 504). For example, based at least in part on telemetry data 506, controller 124 may determine that the load capacity of node 504(2) exceeds a threshold load capacity. The threshold load capacity may include, for example, a percentage value (eg, 80%, 85%, 90%, 100%, etc.). Additionally, the threshold load capacity may be dynamic and change (eg, from 80% to 90%) depending on the time of day, day of the week, current demand, etc. In some cases, controller 124 may send the indication directly to load balancer 114(1). Additionally or alternatively, controller 124 may send an indication to node 504(2), as shown in FIG. 5B.

At “4,” node 504(2) may send or forward indication 508 to load balancer 114(1) based at least in part on receiving the indication from controller 124. Indication 508 may be configured to prompt load balancer 114(1) to perform one or more actions to adjust where it sends data streams. Thus, at "5," load balancer 114(1), based at least in part on receiving indication 508, may send indication 510 to node 504(N) to prompt node 504(N) to prepare one or more interfaces for the second traffic A portion of 502(2) may be sent/redirected to node 504(N). In at least one example, node 504(N) may include a data node for processing ESP traffic and indication 510 may include an empty ESP packet including a client device associated with one or more client devices 106 source IP address and port. In this manner, when node 504(N) receives a null ESP packet, node 504(N) may begin setting up the interface in preparation for receiving IPsec security associations. Additionally, load balancer 114(1) may send a key update request to the control node responsible for the corresponding IKE session associated with the IPsec security association.

At "6," load balancer 114(1) may begin sending additional traffic 512 to nodes 504(N). Additional traffic 512 may include at least a portion of second traffic 502(2) that was previously sent to node 504(2). In this manner, the load capacity of node 504(2) may be reduced (eg, to 75%) and the load capacity of node 504(N) may be increased (eg, to 42%). In some examples, load balancer 114(1) may begin sending additional traffic 512 to nodes 504(N) based at least in part on receiving indication 508 of adjusted data flow. Furthermore, load balancer 114(1) may determine that additional traffic 512 is to be sent to node 504(N) based at least in part on the SPI value included in a separate packet of additional traffic 512. For example, in some examples, based at least in part on the load balancer 114(1) sending the key update request, the load balancer 114(1) may not be aware of the SPI value included in the separate packet because the IKE node may have issued The new SPI value to be used by the client device. Accordingly, load balancer 114(1) may identify a set of quintuple values included in a separate packet and determine additional traffic 512 based at least in part on identifying the set of quintuple values and based at least in part on issuing an update key request will be sent to node 504(N). In addition, load balancer 114(1) may store an association between new/unknown SPI values and the set of 5-tuple values.

6, 7, 8, and 9 illustrate logic flow diagrams of various example methods associated with the techniques presented herein for load balancing encrypted traffic based on SPI values. The logical operations described herein with reference to FIGS. 6, 7, 8, and 9 may be implemented (1) as a series of computer-implemented acts or program modules running on a computing system and/or (2) as Interconnected machine logic circuits or circuit modules.

Implementation of the various components described herein is a matter of choice depending on the performance and other requirements of the computing system. Accordingly, the logical operations described herein are referred to variously as operations, structural devices, acts or modules. These operations, structural devices, acts and modules may be implemented in software, firmware, special purpose digital logic and any combination thereof. It should also be understood that more or fewer operations may be performed than shown in FIGS. 6 , 7 , 8 , and 9 and described herein. Operations may also be performed in parallel, or in an order different from that described herein. Some or all of these operations may also be performed by components other than those specifically identified. Although the techniques described in this disclosure are described with reference to specific components, in other examples the techniques may be implemented with fewer components, more components, different components, or any configuration of components.

6 illustrates a logic flow diagram of an example method 600 for maintaining QoS handling of packets using SPI values. Example method 600 begins at operation 602, which includes receiving, from a client device and at a network device of the network, a request to establish an encrypted tunnel through a network so that data plane traffic may flow between the client device and a service via the encrypted tunnel. For example, load balancer 114(1) and/or control node 118(1) may receive a request from a first client device of one or more client devices 106. Additionally, in some examples, the request may include an IKE SA INIT request packet to establish an IPsec SA between the first client device and the first data node 116(1), such that the data plane traffic 130 may flow between the client device and the service Flow between 110.

At operation 604 , the example method 600 includes determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes, the particular traffic class being associated with a particular quality of service (QoS) performance metric. In some examples, control node 118(1) may invoke a classifier to determine a particular traffic class. Additionally or alternatively, load balancer 114(1) may invoke a classifier to determine a particular traffic class.

At operation 606 , the example method 600 includes generating a Security Parameter Index (SPI) value to be used by the client device for data plane traffic, the SPI value corresponding to a particular traffic class. For example, control node 118(1) may generate an SPI value to be used by a first client device of one or more client devices 106 . The SPI value and/or a portion of the SPI value field may correspond to a particular traffic class. That is, in some examples, an SPI value may include a unique bit combination, and some bits of the unique bit combination may correspond to a particular traffic class.

At operation 608, the example method 600 includes sending an indication of the SPI value to the client device. For example, control node 118(1) may send an indication of the SPI value directly to a first of one or more client devices 106 by performing a direct server return (DSR) to bypass at least load balancer 114 . In some examples, the indication may include an IKEINIT response packet indicating that an IPsec SA has been established for a first client device of the one or more client devices 106 .

At operation 610 , example method 600 includes receiving, at a load balancing node associated with the network, a data packet of data plane traffic that includes the SPI value. For example, load balancer 114(1) may receive data packets for data plane traffic 130 from edge router 112(1), or in the case where the load balancing technique may be performed by edge router 112(1), directly from the first client The device receives data packets of data plane traffic 130 . In some cases, edge router 112(1) may enforce an ECMP routing policy and determine to send the data packet to load balancer 114(1) based on computing a hash value representing a set of network quintuple values included in the data packet. ).

At operation 612 , example method 600 includes sending the data packet over the network based at least in part on the data packet including the SPI value such that the data packet is processed according to the particular QoS performance metric. For example, load balancer 114(1) may send the data packet to data node 116(1) based on enforcing an ECMP routing policy that includes computing a representative SPI value and the set of network quintuple values included in the data packet the hash value of . Additionally, data node 116(1) may be associated with a traffic class, and load balancer 114(1) may determine to send the data packet to the data packet based at least in part on the hash value and/or determining that the SPI value is associated with the traffic class. Node 116(1).

7 illustrates a logic flow diagram of an example method 700 for load balancing traffic based on SPI values in packet headers. The example method 700 begins at operation 702, which includes receiving, from a client device and at a network device of the network, a request to establish an encrypted tunnel through a network so that data plane traffic may flow between the client device and a service via the encrypted tunnel. For example, load balancer 114(1) and/or control node 118(1) may receive a request from a first client device of one or more client devices 106. Additionally, in some examples, the request may include an IKE SA INIT request packet to establish an IPsec SA between the first client device and the first data node 116(1), such that the data plane traffic 130 may flow between the client device and the service Flow between 110.

At operation 704 , example method 700 includes generating a Security Parameter Index (SPI) value to be used by the client device for data plane traffic. For example, control node 118(1) may generate an SPI value to be used by a first client device of one or more client devices 118(1). In some examples, the SPI value may identify an IPsec SA between the first client device and one or more data nodes 116 and/or one or more interfaces of data nodes 116 .

At operation 706, the example method 700 includes sending an indication of the SPI value to the client device. For example, control node 118(1) may send an indication of the SPI value directly to a first of one or more client devices 106 by performing a direct server return (DSR) to bypass at least load balancer 114 . In some examples, the indication may include an IKEINIT response packet indicating that an IPsec SA has been established for a first client device of the one or more client devices 106 .

At operation 708 , the example method 700 includes receiving, at the load balancer, a data packet including the SPI value. For example, load balancer 114(1) may receive data packets for data plane traffic 130 from edge router 112(1), or in the case where the load balancing technique may be performed by edge router 112(1), directly from the first client The device receives data packets of data plane traffic 130 . In some cases, edge router 112(1) may enforce an ECMP routing policy and determine to send the data packet to load balancer 114(1) based on computing a hash value representing a set of network quintuple values included in the data packet. ).

At operation 710 , example method 700 includes determining, by the load balancer and based at least in part on the SPI value, to send the data packet to a server of a set of servers supporting the service. In some examples, determining the server in the set of servers to send the data packet to may also include determining an encrypted tunnel between the load balancer and the server that will be used to send the data packet to the server. For example, load balancer 114(1) may send the data packet to data node 116(1) based on enforcing an ECMP routing policy that includes computing a representative SPI value and the set of network quintuple values included in the data packet the hash value of .

At operation 712, the example method 700 includes sending the data packet to the server. For example, load balancer 114(1) may send data packets of data plane traffic 130 to data node 116(1), such that data node 116(1) may forward the data packets to firewall node 120(1), which in turn 120(1) can forward the data packet downstream to service 110.

FIG. 8 illustrates a logic flow diagram of an example method 800 for performing encrypted tunnel migration. Example method 800 begins at operation 802, which includes receiving, at a load balancer and from a client device, first data plane traffic having a first security parameter index (SPI) value and a set of quintuple values. For example, load balancer 114(1) may receive first data plane traffic from edge router 112(1) or, where load balancing techniques may be performed by edge router 112(1), directly from a first client device First data plane traffic. In some cases, edge router 112(1) may enforce an ECMP routing policy based on computing a first hash value representing the set of network quintuple values included in the first data plane traffic, and determine that the first data plane traffic Plane traffic is sent to load balancer 114(1).

At operation 804 , the example method 800 includes sending the first data plane traffic to a first node based at least in part on the first SPI value, the first node being associated with the first encrypted tunnel. For example, load balancer 114(1) may send first data plane traffic 130 to data node 116(1). For example, the first SPI value may identify an IPsec SA between the client device and data node 116(1). In some examples, sending the first data plane traffic to the first node may be based at least in part on computing a second hash value representing the first SPI value and a set of network quintuple values of the first data plane traffic.

At operation 806 , example method 800 includes receiving, at the load balancer, an indication that additional data plane traffic received from the client device is to be sent to a second node, the second node being associated with the second encrypted tunnel. For example, the indication may indicate that the first node is operating at maximum load capacity or that the first node is about to lose connection, leave, be served, etc. The load balancer may receive indications from a controller such as controller 124 and/or a node such as one of control node 118 or data node 116 . In some examples, the indication may prompt the load balancer to send one of the control nodes 118 a rekey request, eg, to establish a new IPsec SA for the client device. Additionally or alternatively, the indication may prompt the load balancer to send a null ESP packet to the second data node (e.g., data node 116(N)), the null ESP packet including the client device's IP address and port such that the second data node Nodes can start setting up interfaces to receive additional data plane traffic.

At operation 808 , the example method 800 includes receiving, at the load balancer and from the client device, second data plane traffic having the second SPI value and the set of 5-tuple values. For example, load balancer 114(1) may receive second data plane traffic from edge router 112(1) or, where load balancing techniques may be performed by edge router 112(1), directly from a first client device Second data plane traffic. In some cases, edge router 112(1) may enforce the ECMP routing policy based on computing a third hash value representing the set of network quintuple values included in the second data plane traffic. Because the data packet includes the set of network quintuple values, the third hash value may be equal to the first hash value and edge router 112(1) may forward the second data plane traffic to load balancer 114(1).

At operation 810 , example method 800 includes sending the second data plane traffic to the second node based at least in part on the second data plane traffic having the set of quintuple values. For example, load balancer 114(1) may send the second data plane traffic to data nodes 116(N). In some examples, load balancer 114(1) may send second data plane traffic 130 to data node 116(N) based at least in part on not identifying the second SPI value. Additionally, load balancer 114(1) may send second data plane traffic 130 to data node 116(N) based at least in part on identifying that second data plane traffic 130 includes the set of network quintuple values. For example, load balancer 114(1) may not have an association stored between the second SPI value and the set of network quintuple values. However, because load balancer 114(1) may have issued a key update request, load balancer 114(1) may associate a second SPI value with the set of network quintuple values. In other words, because load balancer 114(1) issues a key update request, when load balancer 114(1) receives a data plane load balancer 114(1) may associate the new/unknown SPI value with a known set of network quintuple values and send the data plane traffic to data node 116(N) accordingly.

FIG. 9 illustrates a logic flow diagram of another example method 900 for performing encrypted tunnel migration. The example method 900 begins at operation 902, which includes receiving, at a load balancer and from a client device, first data plane traffic having a first security parameter index (SPI) value and a set of quintuple values.

At operation 904 , example method 900 includes sending first data plane traffic to the first node based at least in part on the first SPI value and the set of 5-tuple values. For example, load balancer 114(1) may send first data plane traffic 130 to data node 116(1). For example, the first SPI value may identify an IPsec SA between the client device and data node 116(1). In some examples, sending the first data plane traffic to the first node may be based at least in part on computing a hash value representing the set of network quintuple values of the first SPI value and the first data plane traffic.

At operation 906 , example method 900 includes receiving, at the load balancer, an indication that at least a portion of the first data plane traffic is to be sent to the second node. For example, the indication may indicate that the first node is operating at maximum load capacity or that the first node is about to lose connection, leave, be served, etc. The load balancer may receive indications from a controller such as controller 124 and/or a node such as one of control node 118 or data node 116 . In some examples, the indication may prompt the load balancer to send one of the control nodes 118 a rekey request, eg, to establish a new IPsec SA for the client device. Additionally or alternatively, the indication may prompt the load balancer to send a null ESP packet to the second data node (e.g., data node 116(N)), the null ESP packet including the client device's IP address and port such that the second data node The node may begin setting up the interface to receive the portion of the first data plane traffic.

At operation 908 , example method 900 includes, based at least in part on the indication, prompting the second node to provide one or more interfaces such that at least the portion of the first data plane traffic may be sent to the second node. For example, load balancer 114(1) may send a null ESP packet to data node 116(N). Null ESP data packets may include an IP address and/or port associated with the client device such that data node 116(N) may begin providing one or more interfaces for the portion of the first data plane traffic.

At operation 910 , example method 900 includes receiving, at the load balancer and from the client device, second data plane traffic having the second SPI value and the set of 5-tuple values. The second SPI value may include a new or unknown SPI value. That is, the load balancer may not have associated the second SPI value with the set of 5-tuple values. At operation 912 , example method 900 includes determining, based at least in part on the second SPI value and the set of quintuple values, that the second data plane traffic includes at least the portion of the first data plane traffic. For example, load balancer 114(1) may track all SPI values associated with the set of 5-tuple values. In this manner, if load balancer 114(1) issues a key update request, load balancer 114(1) may begin monitoring new/unknown SPI values associated with the set of quintuple values to determine which key is included. A second data plane traffic of the portion of the data plane traffic comprising the set of 5-tuple values and a second (new/unknown) SPI value.

At operation 914, the example method 900 includes sending the second data plane traffic to the second node. For example, load balancer 114(1) may send the second data plane traffic to data nodes 116(N). In some cases, sending the second data plane traffic to the second node may be based at least in part on determining that the second data plane traffic includes the portion of the first data plane traffic. Additionally or alternatively, sending the second data plane traffic to the second node may be based at least in part on an ECMP routing policy and calculating a hash value representative of the second SPI value and the set of quintuple values.

10 shows a schematic diagram of an example computer hardware architecture for implementing network nodes and/or devices (e.g., load balancers, control nodes, data nodes, etc.) that can be used to implement the various aspects presented herein. all aspects of technology. The computer structure is shown in Figure 10, which shows a traditional server computer, network device, workstation, desktop computer, laptop computer, tablet computer, network device, e-reader, smart phone and/or other computing devices, and can be used to execute any of the software components presented herein. Computer 1000 may include networking devices such as servers, switches, routers, hubs, bridges, gateways, modems, repeaters, access points, and the like.

The computer 1000 includes a base board 1002 or "motherboard," which is a printed circuit board that can be connected to a number of components or devices through a system bus or other electrical communication paths. In one illustrative configuration, one or more central processing units (“CPUs”) 1004 operate in conjunction with chipset 1006 . CPU 1004 may be a standard programmable processor that performs the arithmetic and logical operations necessary for the operation of computer 1000 .

CPU 1004 performs operations by transitioning from one discrete physical state to the next by manipulating switching elements that distinguish and change the states. A switching element typically includes electronic circuitry, such as a flip-flop, that maintains one of two binary states, and includes electronic circuitry, such as a logic gate, that provides an output state based on a logical combination of the states of one or more other switching elements. These basic switching elements can be combined to create more complex logic circuits, including registers, adder and subtractors, arithmetic logic units, floating point units, and more.

Chipset 1006 provides the interface between CPU 1004 and the remaining components and devices on substrate 1002 . Chipset 1006 may provide an interface to RAM 1008 used as main memory in computer 1000 . Chipset 1006 may further provide an interface to a computer-readable storage medium, such as read-only memory (“ROM”) 1010 or non-volatile RAM (“NVRAM”), for storing The computer 1000 and the basic routines that transfer information between various components and devices. Depending on the configuration described herein, ROM 1010 or NVRAM may also store other software components necessary for the operation of computer 1000 .

Computer 1000 may operate in a networked environment using logical connections through a network (eg, network 108 and/or network 1024 ) to remote computing devices and computer systems. Chipset 1006 may include functionality for providing network connectivity through NIC 1012, such as a Gigabit Ethernet adapter. NIC 1012 is capable of connecting computer 800 to other computing devices over a network. It should be appreciated that multiple NICs 1012 may be present in computer 1000, connecting the computer to other types of networks and remote computer systems. In some examples, NIC 1012 may be configured to perform at least some of the techniques described herein and may include components for performing the techniques described herein.

The computer 1000 can be connected to a storage device 1018 that provides non-volatile storage for the computer. The storage device 1018 can store an operating system 1020, programs 1022, and data, which have been described in greater detail herein. Storage device 1018 may be connected to computer 1000 through storage controller 1014 connected to chipset 1006 . Storage device 1018 may consist of one or more physical storage units. The storage controller 1014 may interface with the physical storage unit via a Serial Attached SCSI (“SAS”) interface, a Serial Advanced Technology Attachment (“SATA”) interface, a Fiber Channel (“FC”) interface, or other type of interface for use in Used to physically connect and transfer data between a computer and a physical storage unit.

Computer 1000 may store data on storage devices 1018 by transforming the physical state of the physical storage elements to reflect the information being stored. The specific transition in physical state may depend on various factors in different embodiments of the present description. Examples of such factors may include, but are not limited to, the technology used to implement the physical storage unit, whether the storage device 1018 features primary or secondary memory, and the like.

For example, computer 1000 may store information to storage device 1018 by issuing instructions via storage controller 1014 to change the magnetic properties of a specific location within a disk drive unit, the reflective or refractive properties of a specific location in an optical storage unit element, or a specific capacitor , transistors, or the electrical characteristics of other discrete components in solid-state memory cells. Other transformations of physical media are possible without departing from the scope and spirit of the present description, and the foregoing examples are provided for ease of description only. Computer 1000 may further read information from storage device 1018 by detecting a physical state or characteristic of one or more specific locations within the physical storage unit.

In addition to mass storage device 1018 described above, computer 1000 may have access to other computer-readable storage media to store and retrieve information such as program modules, data structures, or other data. Those skilled in the art should understand that a computer readable storage medium is any available medium that provides non-transitory storage of data and that can be accessed by the computer 1000 . In some examples, operations performed by system architecture 100 and/or any components included therein may be supported by one or more devices similar to computer 1000 . In other words, some or all operations performed by system architecture 100 and/or any components included therein may be performed by one or more computing devices 1000 operating in a cloud-based arrangement.

By way of example, and not limitation, computer readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, Erasable Programmable ROM (“EPROM”), Electrically Erasable Programmable ROM (“EEPROM”), flash memory or other solid-state storage technologies, compact disc ROM (“CD -ROM"), digital versatile disc ("DVD"), high-definition DVD ("HD-DVD"), BLU-RAY or other optical storage device, tape cartridge, tape, magnetic disk storage device or other magnetic storage device or any other A medium that can be used to store desired information in a non-transitory manner.

As noted above, the storage device 1018 may store an operating system 1020 for controlling the operation of the computer 1000 . According to one embodiment, the operating system includes a LINUX operating system. According to another embodiment, the operating system includes Microsoft® from Microsoft Corporation of Redmond, Washington. SERVER operating system. According to a further embodiment, the operating system may comprise the UNIX operating system or one of its variants. It should be understood that other operating systems may also be used. The storage device 1018 may store other system or application programs and data used by the computer 1000 .

In one embodiment, storage device 1018 or other computer-readable storage medium is encoded with computer-executable instructions that, when loaded into computer 1000, transform the computer from a general-purpose computing system into a computer-based computer capable of implementing embodiments described herein. dedicated computer. As noted above, these computer-executable instructions transform computer 1000 by specifying how CPU 1004 transitions between states. According to one embodiment, computer 1000 may have access to a computer-readable storage medium storing computer-executable instructions that, when executed by computer 1000, perform the various processes described above with respect to FIGS. 1-9 . Computer 1000 may also include a computer-readable storage medium having stored thereon instructions for performing any of the other computer-implemented operations described herein.

Computer 1000 may also include one or more input/output controllers 1016 for receiving and processing input from a variety of input devices, such as a keyboard, mouse, touch pad, touch screen, electronic stylus, or other types of input device. Similarly, input/output controller 1016 may provide output to a display, such as a computer monitor, flat panel display, digital projector, printer, or other type of output device. It should be appreciated that computer 1000 may not include all of the components shown in FIG. 10, may include other components not explicitly shown in FIG. 10, or may utilize an entirely different architecture than that shown in FIG.

As described herein, computer 1000 may include one or more of a data node, a control node, a firewall node, an edge router, and/or a key-value store. Computer 1000 may include one or more hardware processors 1004 (processors) configured to execute one or more stored instructions. Processor 1004 may include one or more cores. Additionally, computer 1000 may include one or more network interfaces (eg, NIC 1012 ) configured to provide communication between computer 1000 and other devices over a network (eg, networks 108 and 1024 ). Network interfaces may include devices configured to couple to personal area networks (PANs), wired and wireless local area networks (LANs), wired and wireless wide area networks (WANs), and the like. For example, network interfaces may include devices compatible with Ethernet, Wi-Fi ^™ , and the like.

Program 1022 may include any type of program or process to perform the techniques described in this disclosure to load balance encrypted traffic based on SPI values in packet headers, and use SPI values to indicate QoS and migrate encrypted connections to different hosts.

In summary, this document describes techniques for load balancing encrypted traffic based on a Security Parameter Index (SPI) value of a packet header and a set of 5-tuple values of the packet header. Additionally, techniques are described herein for including quality of service (QoS) type information in the SPI value field of a packet header. The QoS type information may indicate a specific traffic class according to which the data packets are processed. Additionally, this article describes techniques for pre-configuring a backend host so that encrypted traffic can be migrated from another backend host to the backend host without causing a temporary service interruption.

While the invention has been described with respect to specific examples, it should be understood that the scope of the invention is not limited to these specific examples. For example, while many examples are described with respect to the IPsec protocol, it should be understood that the techniques described are applicable to other protocols. Since other modifications and variations as appropriate to particular operating requirements and circumstances will be apparent to those skilled in the art, the present invention is not to be considered limited to the examples chosen for purposes of disclosure and covers All changes and modifications are true in spirit and scope.

Although this application describes embodiments having specific structural features and/or methodological acts, it is to be understood that claims are not necessarily limited to the specific features or acts described. Rather, the specific features and acts are merely illustrative of some embodiments that are within the scope of the claims of the present application.

Claims (23)

1. A method comprising: receiving, from a client device and at a network device of the network, a request to establish an encrypted tunnel through the network such that data plane traffic flows between the client device and a service via the encrypted tunnel; determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes, the particular traffic class being associated with a particular quality of service (QoS) performance metric; generating a Security Parameter Index (SPI) value to be used by the client device for the data plane traffic, the SPI value corresponding to the particular traffic class; sending an indication of the SPI value to the client device; receiving a data packet of the data plane traffic comprising the SPI value at a load balancing node associated with the network; and Based at least in part on the data packet including the SPI value, the data packet is sent over the network such that the data packet is processed according to the particular QoS performance metric. 2. The method of claim 1, wherein generating the SPI value comprises: generating a first bit pattern representative of the particular QoS performance metric of the data packet to be processed based at least in part on the particular traffic class; generate a second combination representing the security association; and The first bit combination and the second bit combination are masked such that the first bit combination includes a first portion of the SPI value and the second bit combination includes a second portion of the SPI value. 3. The method of claim 2, wherein the first bit combination is represented by a first hexadecimal digit and the second bit combination is represented by a plurality of hexadecimal digits. 4. The method of any one of claims 1 to 3, wherein a first part of the SPI value is a first identifier corresponding to the particular traffic class, and a second part of the SPI value is A second identifier corresponding to the security association for the network. 5. The method according to any one of claims 1 to 4, wherein said data packet is a first data packet, said SPI value is a first SPI value, said specific traffic class is a first traffic class, And the specific QoS performance metric is a first QoS performance metric, the method further comprising: receiving a second data packet at the load balancing node, the second data packet including a second SPI value corresponding to a second traffic class, the second traffic class being associated with a second QoS performance metric; and Based at least in part on the second data packet including the second SPI value, the second data packet is sent over the network such that the second data packet is processed according to the second QoS performance metric. 6. The method of any one of claims 1 to 5, wherein sending the data packet over the network comprises using equivalent multipathing based at least in part on the SPI value and a quintuple of the data packet (ECMP) routing algorithm to send the data packets through the network. 7. The method of any one of claims 1 to 6, wherein generating the SPI value comprises generating a plurality of SPI values to be used by the client device for the data plane traffic, the plurality of SPI values Each of the values corresponds to a respective traffic class, each respective traffic class being associated with a respective QoS performance metric. 8. A system comprising: one or more processors; and One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising: receiving from a client device a request to establish an encrypted tunnel over a network such that data plane traffic flows between the client device and a service via the encrypted tunnel; determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes, the particular traffic class being associated with a particular quality of service (QoS) performance metric; generating a Security Parameter Index (SPI) value to be used by the client device for the data plane traffic, the SPI value corresponding to the particular traffic class; sending an indication of the SPI value to the client device; receiving a data packet of the data plane traffic from the client device comprising the SPI value; and Based at least in part on the data packet including the SPI value, the data packet is sent over the network such that the data packet is processed according to the particular QoS performance metric. 9. The system of claim 8, wherein generating the SPI value comprises: generating a first bit pattern representative of the particular QoS performance metric of the data packet to be processed based at least in part on the particular traffic class; generate a second combination representing the security association; and The first bit combination and the second bit combination are masked such that the first bit combination includes a first portion of the SPI value and the second bit combination includes a second portion of the SPI value. 10. The system of claim 9, wherein the first bit combination is represented by a first hexadecimal digit and the second bit combination is represented by a plurality of hexadecimal digits. 11. The system of any one of claims 8 to 10, wherein the first part of the SPI value is a first identifier corresponding to the particular traffic class, and the second part of the SPI value is A second identifier corresponding to the security association for the network. 12. The system of any one of claims 8 to 11, wherein the data packet is a first data packet, the SPI value is a first SPI value, and the specific traffic class is a first traffic class, And the specific QoS performance metric is a first QoS performance metric, the operations further comprising: receiving a second data packet comprising a second SPI value corresponding to a second traffic class associated with a second QoS performance metric; and Based at least in part on the second data packet including the second SPI value, the second data packet is sent over the network such that the second data packet is processed according to the second QoS performance metric. 13. The system of any one of claims 8 to 12, wherein sending the data packet over the network comprises using equivalent multipathing based at least in part on the SPI value and a quintuple of the data packet (ECMP) routing algorithm to send the data packets through the network. 14. The system of any one of claims 8 to 13, wherein generating the SPI value comprises generating a plurality of SPI values to be used by the client device for the data plane traffic, the plurality of SPI values Each of the values corresponds to a respective traffic class, each respective traffic class being associated with a respective QoS performance metric. 15. A non-transitory computer-readable medium storing instructions that, when executed by one or more computing devices, cause the computing devices to perform operations, the operations comprising: receiving from a client device a request to establish an encrypted tunnel over a network such that data plane traffic flows between the client device and a service via the encrypted tunnel; determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes, the particular traffic class being associated with a particular quality of service (QoS) performance metric; generating a security parameter index (SPI) value to be used by the client device for the data plane traffic, wherein a first portion of the SPI value corresponds to the particular traffic class; sending an indication of the SPI value to the client device; receiving a data packet of the data plane traffic from the client device comprising the SPI value; and Based at least in part on the data packet including the SPI value, the data packet is sent over the network such that the data packet is processed according to the particular QoS performance metric. 16. The non-transitory computer readable medium of claim 15 , wherein generating the SPI value comprises: generating a first bit pattern representative of the particular QoS performance metric of the data packet to be processed based at least in part on the particular traffic class; generate a second combination representing the security association; and The first bit combination and the second bit combination are masked such that the first bit combination includes a first portion of the SPI value and the second bit combination includes a second portion of the SPI value. 17. The non-transitory computer readable medium of claim 16 , wherein the first bit combination is represented by a first hexadecimal digit, and the second bit combination is represented by a plurality of hexadecimal digits . 18. The non-transitory computer readable medium of any one of claims 15 to 17, wherein the second portion of the SPI value is a second identifier corresponding to a security association for the network. 19. The non-transitory computer readable medium of any one of claims 15 to 18, wherein the data packet is a first data packet, the SPI value is a first SPI value, and the specific traffic class is the first traffic category, and the specific QoS performance metric is the first QoS performance metric, the operations further include: receiving a second data packet comprising a second SPI value, a first portion of the second SPI value corresponding to a second traffic class associated with a second QoS performance metric; and Based at least in part on the second data packet including the second SPI value, the second data packet is sent over the network such that the second data packet is processed according to the second QoS performance metric. 20. The non-transitory computer readable medium of any one of claims 15 to 19, wherein generating the SPI value comprises generating a plurality of SPI values to be used by the client device for the data plane traffic , each of the plurality of SPI values corresponds to a corresponding traffic class, and each corresponding traffic class is associated with a corresponding QoS performance metric. 21. A device comprising: means for receiving, from a client device and at a network device of the network, a request to establish an encrypted tunnel through the network such that data plane traffic flows between the client device and a service via the encrypted tunnel; means for determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes, the particular traffic class being associated with a particular quality of service (QoS) performance metric; means for generating a Security Parameter Index (SPI) value to be used by said client device for said data plane traffic, said SPI value corresponding to said particular traffic class; means for sending an indication of the SPI value to the client device; means for receiving, at a load balancing node associated with the network, a data packet of the data plane traffic comprising the SPI value; and Means for: sending the data packet over the network based at least in part on the data packet including the SPI value such that the data packet is processed according to the particular QoS performance metric. 22. The apparatus of claim 21, further comprising means for carrying out the method of any one of claims 2-7. 23. A computer program, computer program product or computer readable medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method as claimed in any one of claims 1 to 7.